See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/319562141

The Sensorama Revisited: Evaluating the Application of Multi-sensory Input

on the Sense of Presence in 360-Degree Immersive Film in Virtual Reality

Chapter · January 2018

DOI: 10.1007/978-3-319-64027-3_13

CITATIONS READS

38 1,657

2 authors:

Sarah Jones Steve Dawkins

University of Gloucestershire Coventry University
13 PUBLICATIONS 232 CITATIONS 11 PUBLICATIONS 75 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Sarah Jones on 26 July 2019.

The user has requested enhancement of the downloaded file.

 Metadata of the chapter that will be visualized in
SpringerLink
Book Title Augmented Reality and Virtual Reality
Series Title
Chapter Title The Sensorama Revisited: Evaluating the Application of Multi-sensory Input on the Sense of Presence in
360-Degree Immersive Film in Virtual Reality
Copyright Year 2018
Copyright HolderName Springer International Publishing AG
Corresponding Author Family Name Jones
Particle
Given Name Sarah
Prefix
Suffix
Division School of Media and Performing Arts
Organization Coventry University
Address Coventry, UK
Email sarah.jones@coventry.ac.uk
Author Family Name Dawkins
Particle
Given Name Steve
Prefix
Suffix
Division School of Media and Performing Arts
Organization Coventry University
Address Coventry, UK
Email

Abstract It has been suggested that 360-degree immersive film viewed in virtual environments, does not allow for a
sense of presence owing to the lack of interactivity, agency and realism. This paper outlines the findings of
a research project to evaluate how such a sense of presence can be enabled through the introduction of
multi-sensory input to the viewing experience. Using an original 360-degree film that was shot in Hong
Kong’s Chungking Mansions as a basis for research, this paper interrogates Ryan’s assertion that Virtual
Reality (VR) that combines interactivity, immersion and narrativity is an example of the “total art” that VR
producers need to aspire to. By adding changes in heat and scent to the viewing experience, the extent to
which those sensory stimuli, which would not normally be part of a viewing experience, lead to an
increased feeling of presence is evaluated. In doing so, we suggest that the viewing experience may not
need all three elements of total art to be equivalent in order for a meaningful viewing experience to occur.
Keywords Presence - 360-degree film - Multi-sensory - Virtual reality
(separated by '-')
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 1/15
Author Proof

1 The Sensorama Revisited: Evaluating

2 the Application of Multi-sensory Input

F
3 on the Sense of Presence in 360-Degree
Immersive Film in Virtual Reality

OO
4

5 Sarah Jones and Steve Dawkins

PR
6 Abstract It has been suggested that 360-degree immersive film viewed in virtual
7 environments, does not allow for a sense of presence owing to the lack of inter-
8 activity, agency and realism. This paper outlines the findings of a research project to
9 evaluate how such a sense of presence can be enabled through the introduction of
10 multi-sensory input to the viewing experience. Using an original 360-degree film
11 that was shot in Hong Kong’s Chungking Mansions as a basis for research, this
12

13
D
paper interrogates Ryan’s assertion that Virtual Reality (VR) that combines inter-
activity, immersion and narrativity is an example of the “total art” that VR pro-
TE
14 ducers need to aspire to. By adding changes in heat and scent to the viewing
15 experience, the extent to which those sensory stimuli, which would not normally be
16 part of a viewing experience, lead to an increased feeling of presence is evaluated.
17 In doing so, we suggest that the viewing experience may not need all three elements
18 of total art to be equivalent in order for a meaningful viewing experience to occur.
EC

20
19 Keywords Presence
360-degree film
Multi-sensory
Virtual reality

21 1 Introduction
R

22 With the emergence of affordable and accessible VR viewing technology and

OR

23 spherical cameras, there has been a growing move towards the production of
24 360-degree film that can be argued to provide a more immersive, embodied
25 experience of moving images than 2D film (Jones 2016; de la Pena et al. 2010).
26 This has led some to define VR as being the “ultimate empathy generator” (De La
27 Pena et al. 2010), with the creator of one innovative example of a 360-degree
documentary, Clouds Over Sidra (2015), describing the film as an “empathy
C

28

29 machine” (Milk 2015). For storytellers, focusing on stories that build a stronger
UN

S. Jones (&)
S. Dawkins

School of Media and Performing Arts, Coventry University, Coventry, UK
e-mail: sarah.jones@coventry.ac.uk

© Springer International Publishing AG 2018 1

T. Jung and M.C. tom Dieck (eds.), Augmented Reality and Virtual Reality,
Progress in IS, DOI 10.1007/978-3-319-64027-3_13
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 2/15

2 S. Jones and S. Dawkins

Author Proof

30 connection between the audience and the subject could, potentially at least, lead to a
31 significantly different relationship between the audience and the text. AQ1

32 One of the main concerns for VR filmmakers is in maximizing the experience in

F
33 order to sustain this connection and so this research starts with a deceptively simple
34 question: does adding multi-sensory input to the viewing experience enhance the

OO
35 sense of presence in 360-degree film and, if so, to what extent?
36 It is clear that, on first experiencing 360-degree film, there is a ‘wow’ moment
37 and a sense of wonder because this experience is profoundly different to viewing in
38 a traditional flat manner. However, even on a second experience, this wears off and
39 some of the inadequacies of the technology begin to be apparent. One reason for
40 this might be that current technologies of production, or the way they have been

PR
41 used, mean that interaction and agency is limited in 360-degree film (Smith 2015).
42 There are also wider concerns over audience compassion fatigue within media
43 (Höijer 2004, Nikunen 2016) that consequently suggest the limitations of VR and
44 360-degree film to sustain the emotional impact within immersive storytelling.
45 The challenge of creating and maintaining presence in the virtual world in order
46 to enable maximum engagement is widely acknowledged as being the goal for VR
47

48
D
creators. As Pimental and Texeira noted during the first wave of VR in the 1990s:
the question isn’t whether the created world is as real as the physical world, but
whether the created world is real enough for you to suspend your disbelief for a
TE
49

50 period of time (Pimentel and Texeira 1993).

51 Lombard and Ditton (1997) describe presence as “the illusion that a mediated
52 experience is not mediated”. Creating a 360-degree filmed experience for immer-
53 sion that with is indistinguishable from reality is challenging when the user is not
actively involved in the story. Ryan’s concept of “total art” (2015) suggests that
EC

54

55 such a simple suspension of disbelief is not enough as that is what many existing
56 forms of media already enable. VR has to aspire to total art in order to differentiate
57 itself. Although 360-degree gaming appears to offer Ryan’s three pre-requisites for
58 total art, 360-degree film currently does not allow for the interactivity and infinite
worlds that VR needs to create this total sense of presence.
R

59

60 This paper analyses that central predicament. Using an original film, ‘Rapid
61 Passage through Various Ambiences’ (2016), as a basis for new research, to create
OR

62 the experience of being within the Chungking Mansions in Hong Kong. It adds in a
63 range of sensory stimuli to evaluate the correlation between presence and
64 cross-modal work. Through interviews with participants, the layers of immersion
65 are analysed to understand the impact of sensory experiences to create presence for
66 the user within 360-degree film.
C

2 Literature Review
UN

67

68 Despite being established as an emerging technology as far back as the 1950s with
69 Heilig’s Sensorama and the “Experience Theatre”, the re-emergence and accessi-
70 bility of VR since 2014 means that is has established itself as a growing medium for
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 3/15

The Sensorama Revisited: Evaluating the Application … 3

Author Proof

71 storytelling. Much has been written about CGI within the virtual world and its
72 applications across industries including health (Hsu et al. 2013) and education
73 (Huang et al. 2010) but the inclusion of the more accessible spherical narrative film

F
74 in these studies is limited. There are early academic studies into spherical panorama
75 still images which demonstrated that they could offer a “proper point of view”, as

OO
76 Barker’s 1787 patent suggests. Uricchio (2011) argued that Barker’s work on the
77 panorama was one of the earliest ideas that suggested the immersive opportunities
78 of virtual reality within the still image and that it produces a “second order reality”
79 (Otto 2007) highlighting that film imagery could offer presence but there have been
80 limited studies since. From this, we can deduce that the opportunities for
81 360-degree film can enable us to place someone in a virtual environment and

PR
82 making the user “feel as if really on the very spot” (Uricchio 2011).
83 The aim of VR to transport you to another time and place has been well doc-
84 umented but the scope to extend this to film-making and immersive storytelling
85 fails due to the lack of definition and agency that often exists within this medium.
86 The concept of VR as a “total art” can give our best understanding of where the
87 potential for 360-degree film lies. Ryan (2015) defines total art as being charac-
88

89
D
terised by three main elements: narrativity, immersion and interactivity. For her,
total art could be likened to art forms, such as the opera, where a state of immersion
capture the different sensory experiences that are able to transport the audience to
TE
90

91 another time and place. Her argument is that VR should go a few steps further than
92 the previous total art form of opera, by combining sound, graphics, text (spoken or
93 written), movement (both by the interactor and the objects within the virtual
94 environment), olfactory effects, and haptic sensations” (1997: np). It is at this point
EC

95 that this happens that we can begin to define VR as total art.

96 Examples of VR as total art are clear where the combination of narrativity,
97 immersion and interactivity gives the sense of presence in the virtual environment.
98 By presence, we follow Pimentel and Texeira’s (1993) definition of creating an
99 environment that enables the experiencer to suspend all belief. Lombard and Ditton
(1997) recognised that in a virtual environment, the user would be aware of using a
R

100

101 head-mounted display but still, “to some degree, her perceptions overlook that
102 knowledge and objects, events, entities, and environments are perceived as if the
OR

103 technology was not involved in the experience”. Presence is the single defining
104 characteristic of virtual reality and defines the moment that the mind tricks the body
105 into feeling that it is somewhere else. Ideas of presence have long been studied
106 within neuroscience (Reiner 2011).
107 Content analysis of interactive virtual reality content carried out by Dolan and
Parets (2015) identified four narrative relationships within virtual experiences. In
C

108

109 their study, they isolated four types of possibility based on the user’s experience
110 and the influence that they can exert on the narrative. A user can exist either as an
UN

111 observer or a participant depending upon how active they are in the experience (See
112 Fig. 1). They develop this to categorise the influence, identifying when the viewer AQ2

113 is actively making decisions and in control of the experience. When we apply this to
114 360-degree narrative film, actions are limited within the virtual environment.
115 A narrative is often prescribed by the director and although a user can experience
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 4/15

4 S. Jones and S. Dawkins

Author Proof

Fig. 1 Narrative
relationships within virtual
experiences

F
OO
PR
116 the environment as they choose, they are not free to make the decision to go to
117 another space and consequently the experience is that of a participant. Dolan and
118 Parets’ research described this as being “relatable to traditional media where you do
119 not exist in the world or influence the story” (2016). The storyteller retains complete
120 control of the action and what is presented to the viewer.
121 VR has long focused on the idea of the body. Fuchs et al. (2006) identified the
122

123
D
idea of VR as one that allowed the subject “to remove himself from physical reality
in order to virtually change the time, place and or type of interaction” (2006, p. 7). AQ3
TE
124 Through the combination of immersion and interactivity with multimodal experi-
125 ences, we can see how the body can be transformed to a virtual being. Concepts of
126 the virtual body and technological embodiment have long been established within
127 film, theatre and art research (Featherstone and Burrows 1996; Bouko 2011).
128 Studies in theatre allow research into self-representation to cross over into mixed
media formats, which can then be applied to immersive film. To achieve a sense of
EC

129

130 virtual embodiment, Bouko suggests that the weighting of the self needs to be
131 lessened to create a new virtual body. By applying this idea to immersive film, we
132 can develop the sensory experiences in a way that that they can simulate a new state
133 of presence. The importance of the bodily experience is clear within the literature of
R

134 one that is intertwined with the representation of the virtual (Bouko 2011).
135 The idea of the virtual embodiment is magnified when we add the application of
136 non-visual interfaces. The research findings develop a new idea concerning sub-
OR

137 jective body ownership illusion (Bergstrom et al. 2016) that show that immersive
138 virtual reality with multi-sensory experiences, can replicate the subjective, physi-
139 ological and cognitive effects of the body in a virtual world. It explores the extent to
140 film which film and moving image material can be, and needs to be, supplemented
141 by other sensory experiences—smell, heat, touch—to be fully immersive and
explores the extent to which actual knowledge of the space in the film enabled a
C

142

143 more fully-embodied experience. Friedman et al. studied the notion of time travel in
144 the virtual world concluding that if a sense of presence can be achieved, then a user
UN

145 can change their own perceptions of events and have a sense of body ownership and
146 agency over the virtual body (Friedman et al. 2014).
147 The nature of immersive storytelling means that you want to be able to reach out
148 and touch what is around you and feel that you are in the heart of the action.
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 5/15

The Sensorama Revisited: Evaluating the Application … 5

Author Proof

149 Personal space in virtual reality has long been discussed (Hall 1963; Bailenson et al.
150 2001; Wilcox et al. 2006) with research showing intrusion can cause significant
151 negative reactions. This can be exploited to enhance the emotional connection and

F
152 “evoke real audience discomfort when faced with a virtual ‘closetalker’ (Wilcox
153 et al. 2006). It is also said to cause more discomfort when the close proximity is to

OO
154 people, rather than objects (Bailenson et al. 2001).
155 Studies in cross-modal work usually focus on haptics and are limited to touch.
156 The importance of olfactory stimuli have been identified as adding presence within
157 healthcare with Krueger (1996) suggesting that the development of virtual surgical
158 training systems would be limited until odors were present in the environment. Heat
159 lamps were used within the University of Virginia VR group to replicate the

PR
160 environment of a fire-breathing dragon so that you would feel the heat sensitivities
161 on your skin. Evaluations of these multimodalities are limited in literature, how-
162 ever, the value to adding presence has been recognised in a number of studies.
163 Munyan et al. (2006) found that the more senses that were added, the greater the
164 level of presence would be. This reflected on the use of presence connected to
165 memory of the virtual environment and focused on olfactory stimuli. In an earlier
166

167
D
1999 study that surveyed 322 participants on the effects of a tactile, olfactory, audio
and visual cues in relation to memory and objects in the virtual world, it was clear
that increasing modalities enhanced both presence and memory (Dinh et al. 1999).
TE
168

169 The depth of the visual didn’t have an impact on presence but the findings pointed
170 to “the more sensory cues that were added, the greater the sense of presence”.
171 These studies have all been carried out in CGI virtual environments. By using
172 these ideas to create a multi-sensory experience, the application of non-visual
EC

173 interfaces can be added to test the layers of immersion and use techniques to disrupt
174 the experience of being in the virtual environment within a 360-degree film.

175 3 Methodology
R

176 There is currently a gap in literature concerning 360-degree films and their con-
OR

177 tribution to immersive technologies and virtual reality. With the current limitations
178 that do not allow 360-degree to achieve total art status, this research fills that gap by
179 exploring the extent to which the experience of 360-degree film can be enhanced by
180 the addition of multi-sensory input into the experience. In doing so, it explores how
181 the sense of presence and immersion can be enhanced and attempts to discover
whether, if interactivity is currently not possible in 360-degree narrative film, such
C

182

183 an enhanced sense of presence moves it closer towards Ryan’s notion of total art.
184 An experimental 360-degree film, entitled Rapid Passage through Various
UN

185 Ambiences, was produced around the Chungking Mansions in Hong Kong in June
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 6/15

6 S. Jones and S. Dawkins

Author Proof

186 2016. Famous as the setting for Wong Kar-Wai’s Chungking Express (1994), the
187 Mansions is a seventeen storey commercial and residential complex, described as “a
188 world hub of low-end globalisation” (Mathews 2011). On the ground floor, the

F
189 building is a complex maze of touts, and tourists that is home to an estimated 4000
190 people and is visited by approximately 10,000 people each day. Hostels, providing

OO
191 some of the cheapest accommodation in Hong Kong, are situated on the upper
192 floors of the Mansions.
193 The title of the film is lifted directly from Guy Debord’s definition of the dérive:
194 a mode of walking that emphasises curiosity, drifting, exploration and wonder
195 (Debord 1996). The film was shot over a two-day filming period embracing the
196 concept of ‘drifting’ and with no pre-production deliberately to capture the chaos of

PR
197 the activity and the diverse communities that make up the Mansions. The film was
198 intended to be experiential; it was directed to capture the essence of the experience
199 of being there. The intention of the editing was to break newly-established filming
200 conventions for 360-degree film—stitch lines from the cameras are visible, camera
201 operators are in shot, and characters break the personal space filming line—in order
202 to create the sense of uneasiness that a first-time visitor to the Mansions may feel
203

204
(Fig. 2) D
On the first viewing of the film in a head-mounted display (HMD), it was clear
that although there was a level of immersion that enabled the viewer to have the
TE
205

B & W IN PRINT
R EC
OR

Fig. 2 A screenshot from rapid passage through various ambiences (2016) with the invasion of
personal space to create the experience of uneasiness
C
UN
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 7/15

The Sensorama Revisited: Evaluating the Application … 7

Author Proof

Fig. 3 The viewing tent

B & W IN PRINT
F
OO
PR
206 ‘wow moment’ and to start to understand what it was like to be situated in the film
207 world, there was no sense of the chaos and confusion and bodily sensations that
D
208 replicated the physical experience of being in the real environment. The film felt
209 ‘empty’.
TE
210 The aim of this research project is to examine whether a 360-degree film, such as
211 Rapid Passage Through Various Ambiences, needs multiple layers of non-visual
212 interfaces to create a more “total art” immersive experience and consequently,
213 presence. The evaluation provides an understanding of the extent in which the
214 experience of immersion might influence experiences within a virtual world. In
EC

215 short, what needs to be ‘added’ to the film itself to create a more totally-immersive
216 experience and if such an experience is, indeed, possible.

Fig. 4 Testing rapid passage

R

B & W IN PRINT
through various ambiences
(2016)
C OR
UN
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 8/15

8 S. Jones and S. Dawkins

Author Proof

221 In this exploratory study, twenty participants of mixed genders, ages and eth-
222 nicities watched the film on a HMD through a mobile device. The participants were
223 selected randomly through a call out to staff and students located within the

F
224 University, ensuring a diverse pool of respondents. The testing occurred while in
225 the controlled space of a tent in a room with no other external stimuli. The par-

OO
226 ticipants put on the HMD outside the tent and were then led into it.
227 1. The first viewing was a simple view with a HMD (see Fig. 4).
228 2. The second viewing, approximately one week after the first, added in ‘layers of
229 immersion’ at the start of the experience. A temperature-controlled environment
230 was created so that users would feel the heat intensities of being in the envi-

PR
231 ronment and the blast of a fan as they moved through the market and smells
232 appropriate to the market environment were introduced (see Fig. 3).
233

234 Immediately after viewing, participants were asked to fill in a form rating a series of
235 statements to get them to reflect upon the experience and to focus in on the areas
236 being examined in the research. This was followed by a semi-structured interview
237 with each participant which was designed to establish their understanding of the
238

239
D
experience of the space and how immersed they felt within the virtual environment.
The framework for interviews with audiences was developed from previous studies
around VR examining the levels of immersion and understanding of stories, pri-
TE
240

241 marily questions regarding presence, focus on place, illusion, plausibility and
242 co-presence (Banakou et al. 2013).
243 The statements that participants rated were as follows:
244 1. I had the sensation of being in Chungking
EC

245 2. There were times when Chungking was more real for me than where I was
246 watching the film
247 3. I felt like I could respond to the people as if they were real people.
248 4. Even though the virtual body wasn’t me, I had the sensation that I was there.
249 5. I felt that the virtual body was someone else.
R

250 6. I felt like I was moving around Chungking.

251 7. I felt immersed in the environment.
8. My body responded to the environment.
OR

252

253 9. I was lost in the environment.

254 10. I felt like I could interact with the environment and the people.
255 11. My body was interacting with the environment.
256 12. I could choose my experience in ChungKing.
257

Once focussed, the participants were immediately interviewed. Table 1 displays the
C

258

259 questions for each interview related to the areas on the rating form.
UN
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 9/15

The Sensorama Revisited: Evaluating the Application … 9

Author Proof

Table 1 Interview questions

Interview 1 Interview 2
1. Have you ever seen a VR film before? 1. Sum up, if you can, how you felt the first

F
How did you feel on first watching one? time that you saw the film
2. When watching this film, did you forget 2. Describe how the second viewing felt

OO
that you were in a tent? How long did it take different (positive or negative)? How
you to forget? If you didn’t forget, what was significantly different did it feel and why?
it that made you not forget? 3. Describe the sensation of being in
3. Describe the sensation of being in the Chungking with the external stimuli
place? Did it feel like it was your body there? (smell/heat)? Did it feel more like it was your
Why? If not, why not? body there than the first viewing? Why? If
4. Describe how you felt about the people in not, why not?

PR
the film. Who did you identify with most and 4. Did:
why? Were there moments when you felt • the smells make a difference to the viewing
uncomfortable? Why? experience? Describe what that difference
5. Did it feel like you were moving around was and how significant it was.
chungking? Did you feel active or passive in • the heat make a difference to the viewing
that movement? Why? Did you feel like you experience? Describe what that difference
could interact with the environment? was and how significant it was
Why/why not?
6. Did you feel ‘immersed’ in the
environment? Did you feel any bodily
D
5. Did the external stimuli have an effect on
how you felt about your interaction with the
environment? Did it affect how active or
sensations (fear etc.)? What was it, passive you felt? Why?
TE
specifically, that made you feel that? 6. Did the external stimuli make you feel
7. Do you have any other observations about more ‘immersed’ in the environment than the
the experience of watching the film? first viewing? Why/why not?
7. Did the external stimuli have an effect on
any bodily sensations you felt (fear etc.)?
EC

8. What elements do you think could to be

added to VR film to take it up to the next
level of immersivity or embodiment?
Real-time body parts (such as own hands)
9. Do you have any other observations about
the experience of watching the film with the
external stimuli?
R

4 Findings
OR

260

261 The interviews took place directly after the experience. They followed the same
262 structure focusing on initial reactions, the different sensory experiences, interaction
263 in the environment, presence and embodiment.
264 The initial response to being inside an environment that added in thermoceptive
C

265 (heat) and olfactory (smell) stimuli was that it created a more immersive experi-
266 ence. Previous studies (Ischer et al. 2014) indicated the importance in setting the
UN

267 right environment to carry out experiments in presence where standard laboratories
268 rarely replicated the “complexity of real world experiences”. By replicating the
269 correct environment, participants said they felt “significantly more immersed”
270 when there was sensory stimuli;
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 10/15

10 S. Jones and S. Dawkins

Author Proof

271 I got into the tent and straight away I could smell the different smells and the heat that was
272 coming into the tent as well, it made it feel a lot more realistic that I was there.

There was a varied response to the addition of olfactory stimuli. Previous literature

F
273

274 has recognised its impact on presence and memory and, although it enhanced
275 presence, the smells were not distinct enough to elicit full immersion. For this

OO
276 study, it was not possible to add a range of different smells at different times in the
277 experience. More sophisticated technology is needed to do this, which will be the
278 focus of a further study. Some participants said the smells seemed to stabilise in the
279 experience because there wasn’t a range of smells that were linked to the scenes.
280 For example, two scenes in the film showed laundry and rubbish/sewers. Most
participants said they wanted to be able to smell these. However the variety of

PR
281

282 smells in the market scenes, which dominate the film, could be said to have a small
283 impact on presence.
284 When you are walking through the streets, there’s all the different parts of the market. You
285 can imagine you are there and someone sat in a café next you and you can smell what
286 they’re eating.

287

288
D
The impact of heat sensitivities had a much stronger impact on presence. Nearly all
participants responded positively to the element of heat being added to the envi-
ronment. They all spoke about the “expectation” of the environment being hot.
TE
289

290 A crowded market place in a country like Hong Kong is expected to be hot and
291 crowded so there is the feeling that it is necessary to experience the heat if there is
292 to be a sense of presence. A cold room would not create the right conditions for
293 presence in the environment to be achieved. Participants spoke of the “expectation
that it is hot” and the “passive heat all around you”. They said, “it just makes you
EC

294

295 feel like you’re there and your body responds to it really well as if you’re there”.
296 With the heat increasing throughout the experience, it felt more real to participants:
297 The heat once got almost a bit too hot but that’s what would happen in that situation. It
298 added to it massively.
R

299 With Lombard and Ditton defining presence as being when the mind forgets the
300 elements of technology, it can be argued that when actions in the virtual word are
OR

301 mirrored in the real world, that sense of presence is achieved. Through adding in
302 heat, participants responded in the manner in which they would if they were in the
303 Chungking Mansions. One said, “I felt like I wanted to take my jacket off because I
304 thought I was in a different country so why would I have my jacket on?”. This was
305 further drawn out and discovered during questions focusing on interactivity.
306 Through adding in heat and smells, one participant described it as “the film really
C

307 responds to you”.

308 One participant noted that they didn’t feel like they could interact in the envi-
ronment but that wasn’t down to a lack of immersion. Instead they stated that they
UN

309

310 wouldn’t choose to interact with people in this environment in the real world.
311 Instead they would walk through and observe but not try and connect. This again
312 suggests that moment of presence where their behaviour was the same in both
313 environments.
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 11/15

The Sensorama Revisited: Evaluating the Application … 11

Author Proof

314 Half of participants spoke about virtual embodiment and how the added senses
315 can create this extended level of immersion adding to the impact of presence. This
316 was pointed to as being a result of the senses.

F
317 There was definitely more of a sense of embodiment.…because your senses are reactive to
318 what is around you and visually as well you can see the same thing as last week but because

OO
319 you can smell and feel different things you feel like you are much more of a part of it.

320 The heat also enhanced this from other participants. The environment was set-up
321 with a heater in one area of the tent which blasted the air in in one position.
322 Although it was noted that it would be better to have the air circulating in different
323 areas, for the majority of participants it was either seen as anything worth noting, or

PR
324 the air created the feeling of movement helping the case for virtual embodiment.
325 It felt like you were moving round and passing you and you did feel more embodied in the
326 space in that way.

327 This study focused on enhancing presence through multi-modalities work, but it
328 also seemed to play a role in the interactivity of the environment.
329

330
331
D
With my senses being alert I looked into a lot more detail about what was going on.
Whether that’s with the second time watching it or because my body was responding in a
different way. I was looking at more finer details. I was looking at where the smell could be
TE
332 coming from.

333 Some ambiguities were identified during the interviews. In the first viewing, one
334 participant reported moments of motion sickness, which is accentuated in this film
335 due to the high level tracking shots meaning a disconnect between the body and
EC

336 mind. Although a common concern within virtual reality (Ohyama et al. 2007),
337 technologies and established filming conventions for 360-degree film are helping to
338 remove this problem. However, in this film specifically, it was designed to create
339 that sense of uneasiness to replicate the experience of being in the environment and
340 so tracking shots and flash frame sequences were used to add distortion. Only one
participant reported motion sickness. In the second week, the participant didn’t
R

341

342 suffer, as reported:

343 I didn’t seem to experience it at all. I don’t know if that is that it’s not just your eyes telling
OR

344 you are in a different place and moving but the hot air the smell and the whole ambiance of
345 it all.

346 Further studies on this are needed. This data suggests that by adding in different
347 senses, it can enhance the trickery of your mind that you are in this environment and
348 mean that it is a more authentic experience so you would expect the motion.
C

349 A common statement in the interviews focused on the issue of the sensory
350 stimuli being added to a film that they had already watched. This was necessary to
be able to measure the impact on presence but it did point to a concern. Around half
UN

351

352 of the participants spoke about the experience as being less immersive, which was
353 suggested that an element of the narrative is lost when you have experienced it
354 before. One participant said that they weren’t as “curious” as they were when they
355 watched it the first time. Another said, “this time I knew what I was expecting so
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 12/15

12 S. Jones and S. Dawkins

Author Proof

356 I didn’t go straight there”, signifying a drop in the level of presence. To understand
357 better then, a number of participants watched the film for the first time with the
358 added sensory stimuli. The responses clearly signify a more intense experience of

F
359 feeling presence in the environment. A participant, who had experienced VR pre-
360 viously, stated;

OO
361 The sensory side of it was unbelievable so that was amazing to have the sensory side of the
362 smell and heat which helped immerse you in it. But the movement, I actually felt like I was
363 moving around in the environment in a more natural way than I did before so I felt like I
364 could almost control where I was rather than just being led by the environment.

365 It indicates a quicker response to feeling presence and more authenticity to being in

PR
366 the environment. Although there is no interactivity in the 360-degree film, where
367 you can choose different pathways, narratives or move around in different direc-
368 tions. The respondent here felt like there was movement and agency in the envi-
369 ronment given the added senses.
370 The senses added impact to the uneasiness that was created in the film. One
371 scene focuses on a man who passes through the market and then spots the camera.
372

373

374
D
He is curious as to what it is and comes close to the camera to investigate (Fig. 1).
Many respondents spoke about the uneasiness in this scene and how it made them
jump or feel that their personal space has been invaded. The participant who only
TE
375 watched the film with the added senses described it with a lot more intensity in the
376 language that was used.
377 Once I got scared in there when a big guy came up to me and looked and for a minute I felt
378 threatened by him and thought he was going to hit me. I actually felt…this is how
379 immersive it is… as I really felt he was in my space and I felt out of control totally as I
EC

380 thought I really don’t know what he was going to do. I was really tense. I had in the back of
381 my head that I could, if I wanted, to come out of it and take the thing of. I didn’t like him
382 and I was on the edge.

383 The same scene was also discussed by other participants with the heat leading to
384 more “agitation”. One participant said that “heat puts you on edge more” so this
R

385 scene made you feel more uneasy. Another participant had a comparable
386 experience,
OR

387 Generally the heat itself makes you feel more uncomfortable in there so it adds to the
388 element of people coming closer to you. Your mind tricks you into thinking it’s the heat
389 coming off him, rather than the general heat in the area adding to the uncomfortability (sic.).

390 However, there were moments when the suspension of disbelief and sense of
391 presence were reduced. Paradoxically, it was in the above example. One respondent
C

392 was so scared that they tried to push the person away but, clearly could not. As they
393 put it: “It goes against the laws of physics” not being able to touch the person and,
394 at that point, the overwhelming sense of presence disappeared.
UN

395 From this we can deduce that specific ideas that are being explored in the film, in
396 this case the sense of uneasiness in the virtual environment, can be heightened and
397 accentuated through adding in multisensory modes at specific intervals in the
398 360-degree film.
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 13/15

The Sensorama Revisited: Evaluating the Application … 13

Author Proof

399 5 Conclusion

The added sensory stimuli to 360-degree film enhances the level of presence in a

F
400

401 virtual environment, bringing us closer to the sense of total immersion that has been
significant within CGI virtual environments. Heat and olfactory additions create a

OO
402

403 more immersive experience within 360-degree film and create the sense of inter-
404 action and embodiment that has been lacking within simple spherical film. To create
405 virtual worlds that allow the user to suspend disbelief, 360-degree films need to
406 work that bit harder and utilise the opportunities that multimodalities bring to
407 enhance presence in the virtual environment.

PR
408 However, our research also suggests that any answer needs to be a little more
409 nuanced around three main areas and that further research needs to be carried out in
410 all of these areas to fully determine the extent to which it does:
411 1. The experience of viewing 360-degree film is profoundly different to traditional
412 forms of screen-based media: it is experiencing rather than viewing. As a result,
413 it is argued that traditional terms for the users of such films—such as audience
414

415
D
or viewer—is replaced with a new term: ‘experiencer’. From this research, it is
clear that the experience of 360-degree film is enhanced by the addition of
multi-sensory input and, from this very small study, would argue that there is a
TE
416

417 hierarchy of affect: visual, auditory, thermoceptive and olfactory, in that order.
418 Further studies might explore this and the extent that other forms of sensory
419 input, such as the “movement and haptic sensations” that Ryan advocates, might
420 have on the experience.
EC

421 2. The spaces that multi-modality need to be experienced in have to be relatively

422 sophisticated to enhance the sense of presence fully. There are numerous current
423 examples of ‘4D’ film where, in a stable and controlled environment, such as a
424 theme park, multi-sensory input is added to the experience. This research sug-
425 gests that the addition of stimuli needs to be carefully controlled for it to achieve
maximum effect. Although respondents mentioned that heat could be a constant
R

426

427 in relation to our film, they all mentioned that olfactory stimuli needed to be
428 appropriate to the narrative and to be variable in intensity. Currently, such
OR

429 sophistication and expense precludes true multi-modality from happening in a

430 domestic setting which has profound implications for who is able to access it,
431 when and how and, more importantly, who is excluded from it.
432 3. That new means of experiencing might involve technologies that both enhance
433 the sense of presence while, paradoxically limiting it: for example, the use of
helmets. The VR experience relies on a complex set of circumstances being just
C

434

435 right. If one, such as the fit of the HMD, is not, the experience is diminished to
436 the extent where disbelief cannot be suspended. This also applies to the addition
UN

437 of other sensory input.

 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 14/15

14 S. Jones and S. Dawkins

Author Proof

438 References

Bailenson, J. N., Blascovich, J., Beall, A. C., & Loomis, J. M. (2001). Equilibrium theory

F
439

440 revisited: Mutual gaze and personal space in virtual environments. Presence, 10(6), 583–598.
441 Banakou, D., Groten, R., & Slater, M. (2013). Illusory ownership of a virtual child body causes

OO
442 overestimation of object sizes and implicit attitude changes. Proceedings of the National
443 Academy of Sciences of the United States of America, 110, 12846–12851. doi:10.1073/pnas.
444 1306779110.
445 Bergstrom, I., Kilteni, K., & Slater, M. (2016). First-Person Perspective Virtual Body Posture
446 Influences Stress: A Virtual Reality Body Ownership Study. PLoS ONE, 11(2).
447 Bouko, C., & Slater, M. (2011). Identity, otherness and the virtual double. Technoetic Arts, 9(1),
448 17–30.

PR
449 Debord, G. (1996). Theory of the derive. In Andreotti, L. & Costa, X. Theory of the dérive and
450 other Situationist writings on the city. Museu d’art contemporani de Barcelona.
451 De la Peña, N., Weil, P., Llobera, J., Giannopoulos, E., Pomés, A., Spanlang, B., et al. (2010).
452 Immersive journalism: immersive virtual reality for the first-person experience of news.
453 Presence: Teleoperators and Virtual. Environment, 19(4), 291–301.
454 Dinh, H. Q., Walker, N., Hodges, L. F., Song, C., & Kobayashi, A. (1999, March). Evaluating the
455 importance of multi-sensory input on memory and the sense of presence in virtual
456
457

458
D
environments. Proceedings Virtual Reality, 1999, IEEE, 222–228.
Dolan, D. & Perets, M. (2016). Redefining the Axiom of Story: The VR and 360 Video Complex.
Retrieved December, 2016, from https://medium.com/@devon.michael/redefining-the-axiom-
of-story-the-vr-and-360-video-complex-bee3c20d69df#.94ngjcddg.
TE
459
460 Featherstone, M., & Burrows, R. (1996). Cyberspace/cyberbodies/cyberpunk: Cultures of
461 technological embodiment. London: Sage.
462 Friedman, D., Pizarro, R., Or-Berkers, K., Neyret, S., Pan, X., & Slater, M. (2014). A method for
463 generating an illusion of backwards time travel using immersive virtual reality-an exploratory
464 study. Frontiers in Psychology, 5, 943.
Fuchs, P. & Moreau, G. (2006). Introduction à la réalité virtuelle/Introduction to Virtual Reality.
EC

465

466 In Fuchs et al. (Eds.), Le Traité de la réalité virtuelle. Volume 2: L’interfaçage, l’immersion et
467 l’interaction en environnement virtuel/The Treatise of Virtual Reality. Volume 2: Interfacing,
468 immersion and interaction in virtual environment (3–30), Paris: ENSMP Presses.
469 Hall, E. T. (1963). A system for the notation of proxemic behavior. American Anthropologis, 65
470 (5), 1003–1026.
Heilig, M. (1998). Beginnings: Sensorama and the Telesphere Mask. In Dodsworth, C. (Ed.)
R

471
472 Digital Illusion (343–351). New York:ACM Press.
473 Höijer, B. (2004). The Discourse of Global Compassion: The Audience and Media Reporting of
474 Human Suffering. Media, Culture and Society, 26, 513–531.
OR

475 Hsu, E., Li, Y., Bayram, J., Levinson, D., Yang, S., & Monahan, C. (2013). State of virtual reality
476 based disaster preparedness and response training. PLoS Currents, 5, 1.
477 Huang, H., Ulrich, R., & Liaw, S. (2010). Investigating learners’ attitudes toward virtual reality
478 learning environments: Based on a constructivist approach. Computers & Education, 55(3),
479 1171–1182.
480 Ischer, M., Baron, N., Mermoud, C., Cayeux, I., Porcherot, C., Sander, D., et al. (2014). How
C

481 incorporation of scents could enhance immersive virtual experiences. Frontiers in Psychology,
482 5, 736.
483 Jones, S. (2016) Disrupting the Narrative: Immersive Journalism in Virtual Reality, Journal of
UN

484 Media Practice (forthcoming)

485 Lombard, M., & Ditton, T. (1997). At the Heart of It All: The Concept of Presence. Journal of
486 Computer-Mediated Communication. doi:10.1111/j.1083-6101.1997.tb00072.x.
487 Mathews, G. (2011). Ghetto at the Center of the World: Chungking Mansions. Hong Kong
488 University of Chicago Press.
 Layout: T1 Standard Book ID: 436951_1_En Book ISBN: 978-3-319-64026-6
Chapter No.: 13 Date: 15-7-2017 Time: 7:20 pm Page: 15/15

The Sensorama Revisited: Evaluating the Application … 15

Author Proof

489 Milk, C. (2015). “How virtual reality can create the ultimate empathy machine”. Paper Presented at
490 Ted2015.
491 Munyan, B.G., Neer, S.M., Beidel, D.C., Jentsch F. (2006) Olfactory Stimuli Increase Presence in

F
492 Virtual Environments..In Scavone, C. (Ed). PLoS ONE. doi:10.1371/journal.pone.0157568
493 Nikunen, K. (2016). Media, passion and humanitarian reality television. European Journal of
494 Cultural Studies, 19(3), 265–282.

OO
495 Otto, P. (2007). Between the Virtual and the Actual: Robert Barker’s Panorama of London and the
496 Multiplication of the Real in late eighteenth-century London. Romanticism on the Net, 46, 1.
497 Ohyama, N., Watanabe, M., Akizuki, T., & Harada, T. (2007). Autonomic responses during
498 motion sickness induced by virtual reality. Auris, Nasus, Larynx, 34(3), 303–306.
499 Pimentel, K., & Texeira, K. (1993). Virtual Reality: Through the New Looking-Glass. London:
500 McGraw Hill.
501 Reiner, M. (2011). Presence: Brain, virtual reality and robots. Brain Research Bulletin, 85(5), 243–

PR
502 244.
503 Ryan, M. L. (1997). Interactive drama: Narrativity in a highly interactive environment. Modern AQ4
504 Fiction Studies, 43(3), 677–707.
505 Ryan, M. L. (2015). Narrative as Virtual Reality 2: Revisiting Immersion and Interactivity in
506 Literature and Electronic Media. JHU Press.
507 Smith, W. (2015). Stop calling Google’s Cardboard 360-degree videos VR. Retrieved December,
508 2016, from https://www.wired.com/2015/11/360-video-isnt-virtual-reality/.
509
510

511
iterations. Early Popular Visual Culture, 9(3), 225–238.
D
Uricchio, W. (2011). A ‘proper point of view’: The panorama and some of its early media

Wilcox, L., Allison, R., Elfassy, S., & Grelik, C. (2006). Personal space in virtual reality. ACM
TE
512 Transactions on Applied Perception (TAP), 3(4), 412–428.
R EC
C OR
UN
Author Proof

Author Query Form

F
123

OO
Book ID : 436951_1_En
Chapter No : 13
the language of science

Please ensure you fill out your response to the queries raised
below and return this form along with your corrections.

PR
Dear Author,
During the process of typesetting your chapter, the following queries have
arisen. Please check your typeset proof carefully against the queries listed
below and mark the necessary changes either directly on the proof/online
grid or in the ‘Author’s response’ area provided below D
TE
Query Refs. Details Required Author’s Response
AQ1 Please confirm if the inserted city name is correct. Amend if necessary.
AQ2 “Dolan and Parets 2015” and “Krueger 1996” are cited in the text but are not
given in the reference list. Please provide full references or delete the
citations from the text.
AQ3 “Fuchs et al.” “Mathews 2014” and “Munyan et al. 2016” were changed to
“Fuchs et al. 2006” “Mathews 2011” and “Munyan et al. 2006” to match the
EC

reference list. Please confirm.

AQ4 “Ryan 1997” is not cited in the text. Please provide the citation.
R
C OR
UN
 MARKED PROOF
Please correct and return this set
Please use the proof correction marks shown below for all alterations and corrections. If you
wish to return your proof by fax you should ensure that all amendments are written clearly
in dark ink and are made well within the page margins.

Instruction to printer Textual mark Marginal mark

Leave unchanged under matter to remain
Insert in text the matter New matter followed by
indicated in the margin or
Delete through single character, rule or underline
or or
through all characters to be deleted
Substitute character or
through letter or new character or
substitute part of one or
more word(s) through characters new characters
Change to italics under matter to be changed
Change to capitals under matter to be changed
Change to small capitals under matter to be changed
Change to bold type under matter to be changed
Change to bold italic under matter to be changed
Change to lower case Encircle matter to be changed
Change italic to upright type (As above)
Change bold to non-bold type (As above)
or
Insert ‘superior’ character through character or
under character
where required
e.g. or
Insert ‘inferior’ character (As above) over character
e.g.
Insert full stop (As above)
Insert comma (As above)
or and/or
Insert single quotation marks (As above)
or

or and/or
Insert double quotation marks (As above)
or
Insert hyphen (As above)
Start new paragraph
No new paragraph
Transpose
Close up linking characters

Insert or substitute space through character or

between characters or words where required

Reduce space between between characters or

characters or words words affected

View publication stats