0% found this document useful (0 votes)

101 views26 pages

Extended Reality (XR) Environments - August 2021

Uploaded by

Avinash Keats

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

101 views26 pages

Extended Reality (XR) Environments - August 2021

Uploaded by

Avinash Keats

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 26

Trim Size: 8in x 10in k

Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 782

Handbook of Human Factors and Ergonomics, Fifth Edition. Gavriel Salvendy and Waldemar Karwowski.
© 2021 John Wiley & Sons, Inc. Published 2021 by John Wiley & Sons, Inc.

Downloaded from https://onlinelibrary.wiley.com/doi/ by Iowa State University Library, Wiley Online Library on [05/11/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
CHAPTER 30
EXTENDED REALITY (XR) ENVIRONMENTS
Kay M. Stanney, Hannah Nye, and Sam Haddad
Design Interactive, Inc.
Orlando, Florida

Kelly S. Hale
Draper
Cambridge, Massachusetts

Christina K. Padron
Dynepic, Inc.
Orlando, Florida

Joseph V. Cohn
Defense Health Agency
Washington, DC

1 INTRODUCTION 782 5 EXTENDED REALITY USABILITY ENGINEERING 799

5.1 XR Inspection Methods 800
2 SYSTEM REQUIREMENTS 783
5.2 XR User Testing Methods 800
2.1 Hardware Requirements 783
2.2 Software Requirements 787 6 APPLICATIONS 803
6.1 Training 803
k 3 DESIGN AND IMPLEMENTATION STRATEGIES 788
6.2 Education 803
k
3.1 User-Centered Design in XR 789
6.3 Operations 804
3.2 XR Design Principles 792
6.4 Entertainment 805
3.3 Engagement Design Strategies 797
6.5 Sales and Marketing 806
3.4 Cybersickness Remediation Design Strategies 797
7 CONCLUSION 807
4 HEALTH AND SAFETY ISSUES 798
4.1 Cybersickness, Adaptation, and Aftereffects 798 ACKNOWLEDGMENTS 807
4.2 Social Impact 799
REFERENCES 807

1 INTRODUCTION “Link Trainer” (Jeon, 2005) or Morton Heilig’s “Sensorama”

Moore’s law, the observation that the number of transistors in an (Heilig, 1962). Yet, understanding the biological and cognitive
integrated circuit doubles approximately every two years, has processes underlying human perception, memory, and action;
become an almost ubiquitous metaphor for the rapid pace of extent to which the core technologies have become smaller,
innovation in any number of technical domains. Unpacking this more powerful, and more readily available; and, degree to
metaphor, one could easily argue that advances in any single which designers, developers, and users are now able to interact
technical domain are enabled by a range of contributing factors collaboratively in real time, have combined to place XR at a
that extend well beyond the four walls of the laboratory. Changes tipping-point of becoming a common-place tool for training and
in social and cultural attitudes, shifts in policy, the elasticizing educating, collaborating and partnering, networking and enter-
of borders, and boundaries—both real and perceived—across taining, and other applications which have yet to be envisioned
private and public as well as commercial and non-commercial and realized.
spaces, all combine with scientific and technical innovation in This chapter provides an overview of advances in five spe-
unique ways to lay the foundation for these rapid advances. cific areas that collectively empower XR to branch into the very
eXtended Reality (XR), a technical capability that offers fabric of our everyday lives, professionally, socially, and per-
a range of computer-generated immersive experiences that sonally. The chapter begins with a perspective on the technol-
mirror reality to varying degrees, provides an excellent example ogy, providing an overview of the basic system requirements for
of how advances across many different domains continue XR, discussing how advances in hardware, software, and related
to drive innovation—and in turn, open up new avenues of capabilities continue to enable ever more realistic and immersive
application. The basic requirements for developing effective experiences—and make them available to an increasingly larger
XR solutions—hardware, software, and user-centered design and broader audience. The chapter then shifts to the perspective
(UCD) principles—remain unchanged since as far back as the of the user, focusing on how UCD principles can and should be
782

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 783

REAL AUGMENTED MIXED VIRTUAL

WORLD REALITY REALITY REALITY

Figure 1 Virtuality continuum.

applied to maximize the immersive experience while minimiz- System requirements for XR solutions include hardware
ing perceptual dissonance. The chapter then moves to specific tools consisting primarily of:
issues affecting the utility and use of these systems—and how to
manage them—focusing on health, safety, and social concerns. • interface devices, oftentimes head worn, used to present
The chapter ends with a review of several different domains for multimodal information and sense the XR world;
which XR is readily suited. • interaction technology that allows users to navigate
through and interact with the XR world;
• tracking devices used to identify head, hand, and limb
2 SYSTEM REQUIREMENTS position, orientation, and location.
eXtended Reality solutions are computer-generated immersive
environments that provide a spectrum of experiences (see And software tools including:
k Figure 1), including (Mann, Furness, Yuan, Iorio, & Wang, k
2018; Milgram & Kishino, 1994): • modeling software to develop 3D models, which consti-
tute XR content assets;
• Augmented Reality (AR), which overlays virtualized • developer software used to generate XR content;
content onto the real world; • communication networks used to support multiuser XR
• Mixed Reality (MR), which in addition to augmenting experiences.
the real world with virtualized content, allows the virtual
content to be aware of and interact with the real world; or
vice versa with real objects in a virtual world, i.e., Aug- 2.1 Hardware Requirements
mented Virtuality (AV), which is another form of MR; XR experiences require higher-end hardware as compared
• Virtual Reality (VR), which fully immerses users in an with traditional PC or mobile software experiences (Xoticpc,
entirely simulated virtual world; 2019). For non-XR applications, high-end requirements gener-
• XR Blended, which is the blending of all three form ally require “gaming” hardware capable of rendering realistic
factors, including the point-of-need overlays of AR, virtual imagery and auditory cues at real-time frame rates of
immersive contextualization of VR, and physical 30 frames per second (FPS) or greater. With XR, however,
interaction afforded by MR, to achieve a currently additional concerns must be accounted for. In XR, 30 FPS is
unparalleled immersive experience. not acceptable, as any frame rate less than 60 FPS can break
immersion and even render users sick, with 75 FPS or greater
Advances in powerful hardware chipsets, such as mobile mul- preferred (Oculus, 2017). In addition, typical PC or mobile
ticore Central Processing Units (CPU), Graphics Processing software experiences only render to a single display device,
Units (GPU), and even artificial intelligence (AI)-dedicated such as a monitor or TV. While this is true for monocular
neural chipsets, have made possible the high frame rates and based-AR on mobile devices, stereoscopic head-worn displays
performance necessary to create seamless experiences across (HWDs) have two displays, typically at HD resolution, that
this virtuality continuum (Milgram, Takemura, Utsumi, & must render the virtualized imagery at 60 FPS or greater,
Kishino, 1994). Advances in tracking technologies have seen effectively doubling the workload of the GPU. Additionally,
expensive, bulky external sensors now being integrated into XR tracking systems, which must be maintained by the CPU
headset technology itself. Improvements in software and hard- at just as high of a refresh rate, are more sophisticated than
ware allow these tracking technologies to maintain refresh the traditional mouse, keyboard, or game controller typically
rates necessary to seamlessly correlate virtual and real-world associated with high end gaming platforms. While advances in
imagery, thereby providing an engaging immersive experience mobile chipsets, such as the Qualcomm Snapdragon XR2 and
(see Table 1). Increasing battery life, mobile bandwidth, and Apple A13 Bionic, are starting to realize these hefty require-
app ecosystem compatibility, coupled with decreasing data ments in mobile platforms and HWDs, many XR platforms
latency, price point, and social inertia are collectively providing are still tethered to high-powered desktop-class CPUs, such as
the foundation for mass adoption of XR solutions (Cook, Jones, Intel’s 10th generation Core Processors, and GPUs, such as the
Raghavan, & Salf, 2018). Nvidia’s RTX series (Burek & Stobing, 2019).

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 784

Table 1 Technology Component Comparison Across XR Platforms

XR platform
Component Augmented Reality Mixed Reality Virtual Reality
Environment In AR, the environment is the real In MR, augmented content is In VR, the environment is entirely
world and virtualized content is overlaid onto the real world, but virtually simulated via 3D models
overlaid onto the real-world additionally it interacts with the or 360 video
location real world, e.g., a virtualized
character walking along a
real-world desk
Hardware In AR, hardware is typically a mobile In MR, hardware requires some In VR, hardware is typically a HWD
device or an HWD that allows for perception of the real world, which that occludes the real-world view
view of the real world. Sensors needs depth sensing capabilities from the wearer; Sensors for
could be as simple as an RGB via a depth capable camera, or tracking are integrated or external
camera to a depth-capable advanced computer vision
camera algorithms that can detect
real-world planes via a normal
camera
Interaction In AR, interactions are done primarily The interactions for MR are largely In VR, interactions are done primarily
via gestures—touch on mobile, similar to AR with position and/or rotation
and hand gestures with HWD; and tracked game-controllers, but also
can also include movement and can include gestures, head
speech. For HWD platforms, head tracking, movement, and speech
tracking can additionally fuel
interaction. Some platforms
employ controllers
Software In AR, high frame rates are typical, Similar to AR, MR systems are In VR, high frame rates are typical,
but the hardware itself is typically usually self-contained, but as many VR platforms are
self-contained and not as powerful additional consideration needs to currently tethered to high-powered
as VR PCs, so visuals are typically be given to physics and collision PCs, they oftentimes also feature
not as high resolution resources for real/virtual high-resolution visualizations
interactions

k k

2.1.1 Augmented Reality and Mixed Reality Advanced tracking AR solutions incorporate localized track-
Technology ing combined with odometry-based navigation systems, such
Augmented reality and mixed reality involve overlaying virtual that pose data (i.e., location and orientation) can be accurately
content onto the real world (Cipresso, Giglioli, Raya, & Riva, perceived across large distances where GPS data are limited or
2018; Mann et al., 2018). This is typically delivered spatially unavailable (Laney et al., 2019). This capability expands the
and in real time, using either a mobile device such as a smart- effective working space for a mobile solution, allowing accurate
phone or tablet, or using an HWD. In AR, the visible natural placement and viewing of virtual objects beyond a single room
environment is simply overlaid with a layer of digital content. without the need for excessive fiducial markers—supporting
In MR, augmented content is localized and affixed to the real larger indoor spaces than would previously have been known to
world using either computer-vision-based methods to determine the AR system. Using this technology, AR systems can build
the position of the augmented content, or infrared-based depth pose data as a user moves throughout an unmapped space,
sensors that spatially map out the surrounding environment. building a “breadcrumb” trace of where they have traveled so
Tablet- and smartphone-based AR/MR solutions are often that placed AR markers can be viewed accurately at a later
delivered using typical commercial-off-the-shelf (COTS) time. Applications for this tracking and registration capability
devices, such as an iPhone or Android tablet or smartphone include disaster response, underground operations, and indoor
(Mann et al., 2018). These systems use SDKs, such as Apple’s navigation and beyond.
ARKit (see Figure 2) or Google’s ARCore, to provide immer- For HWD-based AR/MR systems, such as the HoloLens 2
sive experiences (Shavel, 2018). These solutions work by and Magic Leap One, similar spatial mapping algorithms are
leveraging onboard camera systems on mobile devices and executed, but are typically more accurate than their mobile coun-
computer vision algorithms to predict, map out, and track a terparts (Brigham, 2017). This is because HWD-based systems
spatial understanding of the area to be augmented. Because are outfitted with infrared depth sensors that can provide a much
these cameras do not have true-depth mapping capability, more accurate spatial mapping system than what can be provided
the algorithms are optimized to detect vertical and horizontal using a simple camera (see Figure 3). These infrared sensors are
planes. These planes are then tracked and maintained relative sometimes colloquially called ‘inside out’ because the sensors
to the camera’s position in real time. Developers can then affix are mounted to the HWD and track the outside world relative to
augmented content to these planes, and as long as the tracking the headset (Cook et al., 2018). This is in contrast to older HWD
system is maintained, augmented content remains fixed to the systems that did not contain sensors, such as the Canon MREAL,
planar anchor. This allows users to either place AR content into that required mounting infrared sensors to the HWD and exter-
the world at a fixed position or extend their AR application into nally, in a fixed area such as a 10 × 10 room, tracking the HWD
an MR application that actually interacts with the real world, position. The more modern ‘inside out’ sensors allow users to
e.g., making a virtualized character that walks around the planar escape the restrictions associated with sensor- and camera-filled
area detected on a real floor. rooms.

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 785

Figure 4 Fiducial marker being used to anchor a virtual plant to

a physical image.

to recognize a particular image and calculate its location, scale,

and orientation in real time with respect to a camera’s posi-
tion (Romero-Ramirez, Muñoz-Salinas, & Medina-Carnicer,
2018). This allows developers to affix these markers to par-
ticular locations, such as specific pieces of equipment, for
accurate anchoring of virtual content to an associated image
Figure 2 Floor and desk horizontal planes detected using and subsequent real-world equipment locations.
ARKit. In addition to affixing augmented visual content to the real
world, AR developers can affix spatialized sound (see Chapter
18 by Casali in this volume) to the environment (Bauer et al.,
2019). Spatial sound is best instantiated in HWD-based AR solu-
tions, since the primary tracking target for such systems is the
user’s head rather than a handheld tablet. While, in the past,
k headphone spatialization required expensive, specialized hard- k
ware to achieve real-time rates, modern multicore processors
have made it possible to render complex audio environments
over headphones. With binaural rendering, a sound can be placed
in any location, right or left, up or down, near or far, via the
use of a head-related transfer function (HRTF) to represent the
way sound sources change as a listener moves his or her head
(Meshram et al., 2014). While for optimal results, HRTFs must
be personalized for each individual user, fairly accurate generic
models are typically provided and perform satisfactorily (Berger
et al., 2018). Most of the mainstream AR/MR HWD manufactur-
Figure 3 HoloLens spatial mapping overlaid onto a real-world
ers provide optimized sound SDKs, such as the HoloLens’ Unity
space.
plug-in (Microsoft Spatializer; Microsoft, 2019c), which allow
for content developers to implement spatialized sounds whose
source is a location in three-dimensional space, or volumetric
HWD systems are typically outfitted with high resolution sound areas, thereby isolating or playing particular sounds when
binocular projectors that can render AR/MR content onto a a user enters particular areas of the augmented world.
see-through optical display (Cipresso et al., 2018; Mann et al.,
2018). These modern HWDs allow users to see the real world
directly rather than through a pass-through camera system. The 2.1.2 Virtual Reality Technology
previous generation video pass-through systems had to integrate
both virtual imagery and a camera feed into a single image in Virtual Reality technology involves simulating an entire virtual
real time, which proved to be both costly and demanding in world, including assets and the world itself, and presenting
terms of hardware resources, thereby often resulting in a low that in an HWD such that the users’ view and perception of
quality view of the real world, as the view was observed through the real world is obstructed (Cipresso et al., 2018; Mann et al.,
a digital capture of a camera. Some AR HWD systems, such 2018). Similar to AR/MR technology, VR technology involves
as the Vuzix M400, provide a monocular rather than binocular rendering realistic virtualized imagery in real time to users
display system. While these can be useful to display 2D-fixed using advanced hardware. These systems can take three main
information or even images over the real world, without stereo- configuration forms (Greenwald, 2019; see Figure 5):
scopic projection capability these systems typically are not used
to place augmented content spatially, and function in a way • tethered to high-powered ‘VR-Ready’ PCs using head-
more similar to projection-based heads-up displays (HUDs) sets, such as the HTC Vive Cosmos, Oculus Rift S, Valve
than AR devices. Index, or Windows Mixed Reality (see Figure 5(a));
Both mobile- and HWD-based AR/MR systems can use • tethered to mobile device-driven headsets, such as
image recognition, also called fiducial markers (see Figure 4), the Google Daydream, Nintendo Labo VR Kit,

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 786

(a) (b) (c)

Figure 5 (a) Oculus Rift; (b) Samsung Gear VR; and (c) All-In-One Oculus Quest System.

Qualcomm-Compatible XR Viewers, or Samsung for a particular VR application, the tethered form factor is
GearVR (see Figure 5(b)); likely most suitable. Also, the battery life on self-encapsulated
• wireless and self-encapsulated standalone systems, such systems tends to be a limiting factor, although they can be run
as the Oculus Quest, Pico Neo 2, Oculus Go, Lenovo continuously while tethered to a power adapter.
Mirage Solo with Daydream, or HTC Vive Focus (see
Figure 5(c)).
2.1.3 Interaction and Tracking Technology
The tethered VR configurations (see Figure 5(a)) are currently While previous generation computer systems used a mouse, key-
the most powerful form factor for delivering VR solutions. board, or game controller, current XR systems afford a large
These headsets offload expensive CPU and GPU calculations variety of techniques for interaction. Interaction in XR is typi-
to modern desktop PCs equipped with multi-core CPUs and cally accomplished using one or a combination of the following
GPUs, with high RAM. This allows lightweight headsets to methods:
simply contain projection systems and sensor systems to track
positions, while offloading heavy lifting computer calculations • gestures
to the tethered computer’s hardware. These systems, however, • controllers
have the disadvantage of always having to maintain the tether, • head tracking
which is not readily viewable when immersed in a VR envi-
ronment and can cause users to trip or use less locomotion to • movement
explore the virtual world than they would with a tether-less • speech
form factor. Further, while these systems do provide 6 degrees
of freedom (DOF), the tether makes this cumbersome for users Gesture recognition is one of the primary forms of interaction
k and limits their exploration potential. in XR. Gesture recognition is the process by which human k
The second form of VR configuration consists of a mobile position and movement are tracked and interpreted to recognize
smartphone inserted into a lightweight chassis equipped with semantically meaningful gestures (Turk, 2014). Gestures can be
binocular optics (see Figure 5(b)). These solutions use local used to specify and control objects of interest, direct navigation,
computational resources to interactively render VR content via manipulate the environment, and issue meaningful commands.
CPU/GPU on the smartphone (Lai et al., 2017). Since mobile In the past, gesture tracking devices were worn (e.g., gloves,
devices are typically not as powerful as desktop PCs, virtualized bodysuits) but significant advances have been made with pas-
imagery is typically not as realistic in appearance as compared sive techniques (e.g., computer vision), which provide more
to other VR form factors. These solutions also lack advanced natural, non-contact, and less obtrusive solutions than those
sensors to track position and only track head rotation, offering that must be worn. These advances have extended into AR
3DOF potential. This limits their use to seated/stationary VR mobile and headset environments. For example, the Magic
applications and precludes their use for physical locomotion. Leap One and HoloLens 2 headsets provide full real-time hand
Another disadvantage of this type of VR headset is its reliance tracking and gesture recognition (Stein, 2019). VR headset
on a mobile device being mounted into the display for the manufacturers, such as Oculus and HTC, also are integrating
duration of the immersive experience. This can be cumbersome hand tracking into some of their platforms. Apple’s ARKit 2
and make scalability and deployment of this type of VR sys- provides real-time body tracking without the need for obtrusive
tems difficult since the mobile device is typically an externally suits, sensors, or a wired PC (Lang, 2019a). Currently, how-
obtained smartphone. While these systems are less powerful, ever, hand-gesture models and libraries are not uniform across
of lower immersive quality, and less deployable than their PC platforms, which presents usability problems in terms of learn-
counterparts, they do allow users to be immersed in a tether-less ability, memorability, and ease of use (Case, 2019). There is a
VR environment, which can provide usability advantages. At need to standardize gesture interaction in XR platforms, just as
the time of this writing, however, the Google Cardboard, Google WIMPs (windows, icons, menus, pointing device) standardized
Daydream, and Samsung GearVR smartphone VR platforms interaction in Windows-based applications.
appear to be phasing out (Greenwald, 2019). Controllers are another form of interaction used in XR
The final VR configuration form is self-encapsulated sys- (Case, 2019). Unlike past controllers that simply consisted of
tems (Greenwald, 2019; Robertson, 2019; see Figure 5(c)). buttons and analog sticks, XR controllers are outfitted with
These “all-in-one” headsets contain all necessary CPU, GPU, sensors that allow for some form of tracking in 3D space. On
and tracking hardware in one system. While they are slightly the lower end, these can be simple 3DOF controllers such as
heavier than their tethered counterparts, they are still lightweight the Oculus Go’s controller, which allows users to “point” their
and designed for prolonged periods of use. These systems are controller at virtual content and interact with it, to advanced
also easier to set up because they are not bound to a specific 6DOF controllers, such as the HTC Vive or Oculus Touch con-
room or PC setup. They also provide 6DOF wireless tracking trollers. While 3DOF XR applications tend to render the virtual
in real time. A disadvantage of these systems compared with controller alone, 6DOF XR applications often render hands in
desktop VR systems is that they do not have as extensive CPU place of the virtual controllers or hands with virtual controllers
and GPU capabilities, so if ultra-realistic imagery is important (XR Association, 2019). The more advanced 6DOF controllers

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 787

provide real-time position and rotational updates that have a approach one of the boundaries, a virtual representation of
low enough latency and accurate enough tracking to allow for the obstacle is overlaid, providing a cue to users that they’ve
the creation of a new generation of full body interaction, such reached the limit of the occupiable real space. Typically, at
as that used in the fast-paced game Beat Saber (Lang, 2019b). this point users can either turn around and go back into their
Controllers are more predominately found on VR platforms. For current virtual/real-mapped space, or they can use a mechanism
AR mobile platforms and AR headsets, controllers are typically typically dubbed ‘teleportation’ to designate with an input
secondary to gestures, although depending on the manufacturer device (typically a controller) where they would like to teleport
they may offer a controller across the 3DOF to 6DOF spectrum. to in the virtual space. The virtual world’s frame of reference
XR HWD platforms typically offer 3DOF or 6DOF is then teleported to that location, which provides users a new
head-tracking (Lang, 2013). 3DOF head-tracking tracks the location in the virtual/real-mapped space that allows them to
rotational movements of the head but cannot track locomo- traverse the desired area. This type of system is realized, for
tion, such as walking forward and backward or standing up example, in the Oculus Guardian System (Oculus, 2019).
and sitting down, which is typically true of lower-end and Another interaction option is speech control. Google
mobile VR platforms and some AR headsets, such as the Epson (Google Developers, 2019), Apple (Martinmitrevski, 2018),
Moverio BT series (Greenwald, 2019). Other HWDs offer and Microsoft (2019c) all have sophisticated speech recognition
6DOF head-tracking, such as the Oculus Quest and Microsoft libraries available for developers to integrate into XR applica-
HoloLens 2. These solutions allow users to interact with tions. These systems typically work best with pre-programmed
augmented content by using their head. For example, an AR keywords, but also contain dictation engines that can process
application could require a medical trainee to move their head large volumes of verbalized speech into interactions that can
close to a virtualized patient’s mouth to teach students how to be processed in real time and deployed into XR applications.
These dictation engines, however, often require internet access
position themselves to check for proper breathing.
for processing, which may or may not necessarily be possible
6DOF HWDs also enable movement and locomotion, which
in all locations where XR systems are deployed. This presents a
is another form of interaction. With the advent of inside-out challenge if text input is used in place of speech, which can be
tracking and portable wireless computer systems in all-in-one highly cumbersome in XR platforms.
XR HWDs, such as the Magic Leap, Microsoft HoloLens, and To support natural and intuitive interaction, a variety of
Oculus Quest (Greenwald, 2019), users can now easily move interaction techniques can be coupled. For example, combining
through immersive environments, whether VR or AR. Even speech interaction with nonverbal gestures and motion inter-
when using fully occluded VR platforms, users can safely faces can provide a means of interaction that closely captures
traverse an area that contains obstacles. This is made possible the way we interact with the real world.
by “boundary systems” present in many VR platforms (Ding-
man, 2019). These systems allow users to map out boundaries
around real-world obstacles prior to being fully immersed 2.2 Software Requirements
in the virtual world. These obstacles include such items as Software development of XR systems has evolved from tra-
k couches and coffee tables, walls in the room, or anything that ditional backgrounds of both game development, as well as k
could prove hazardous to users. Once all obstacles are defined, computer programming. In a unique way, both of these once
the boundary box becomes invisible, allowing users to roam similar but separate fields have now been combined to create
freely and safely in the virtual world (see Figure 6). If users realistic XR worlds driven by sophisticated back ends that

Figure 6 Boundary box.

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 788

model realistic data (e.g., biological processes associated with popular SDK that at face value targets Microsoft’s products, but
a given anatomical system in a medical XR training system; this SDK is actually designed to be platform agnostic and XR
Sadeghnejad et al., 2019). Integrated Developer Environments device independent (Jackson, 2018). This allows developers to
(IDEs), such as Unity3D and Unreal, initially developed for take advantage of MR Toolkit’s large library of XR interactions
game developers have now become the primary platforms and XR user interfaces. A developer could, for instance, use
for development of XR content (Unity, 2019). XR Standards, MR Toolkit to develop an AR application for HoloLens, Oculus
such as OpenXR by Khronos Group (Orland, 2019), are being Rift, and even tablet-based AR platforms by using ARKit and
developed to make cross-platform XR development more sim- only write minimal additional code to support all three form
plified and streamlined. Primers are also evolving to guide XR factors.
application development (XR Association, 2019). After 3D content is developed and the software has been
Software development of XR systems involves creating 3D written to drive that content, the final step is deploying the
models to represent the virtualized world and augmented content content to the different XR platforms. For desktop XR applica-
to be rendered, developing software code to drive the state and tions this involves creating a simple executable that is run like
behavior of the virtual world and its content, and then deploying any normal application. These executables can be delivered to
these assets onto XR hardware platforms that may be hosted by clients using a variety of methods (physical or digital delivery)
ultra-broadband networks (Cook et al., 2018). and, provided the required XR hardware is available, clients
will be able to run these applications in the same manner as
any other desktop application. For mobile-based AR, such as
2.2.1 Modeling for XR iOS and Android, XR applications may need to be prepack-
Modeling XR worlds and their associated content has evolved aged into a store-delivery format and then submitted to the
from the creation of 3D models optimized for CAD or device’s respective store. Store-based AR apps typically require
post-processed computer-generated imagery used in films, additional instructions due to the wide variety and availability
to those that are optimized for real-time rendering at high FPS of deployment and to avoid confusion for users who may be
in order to maintain immersion and avoid causing cybersickness unfamiliar with XR technology. Human interface guidelines are
(Caserman, Martinussen, & Göbel, 2019; Cook et al., 2018). available to assist developers in making the transition to XR
Thus, content may need to contain relatively low polygon easier and more intuitive for users (Apple Developer, 2019; XR
counts and texture sizes in order for XR systems to maintain Association, 2019).
acceptable rendering rates, even on higher end hardware. These
3D models are commonly created with COTS tools, such as
3DS Max and Maya (Gaget, 2019), and come in a variety of 2.2.3 Networks
formats, such as FBX (Filmbox) and OBJ. XR developers need 10–20x the storage capacity as compared
Exciting advances are on the horizon, which promise a to requirements for playing traditional standard HD video files
more specific XR-oriented portable 3D model format. Two (Cook et al., 2018). Cloud XR services can help meet some of
k such formats now available are known as USDZ (Universal these requirements by allowing for processing of XR-related k
Scene Description) and glTF (Graphics Library Transmission content to be delegated to significantly more powerful server
Format) (Ramamirtham, 2018). USDZ, which was developed PCs rather than handled locally on mobile devices and HWDs.
by Apple and Pixar, allows developers to make 3D models in This can mean significant battery and CPU power savings,
the traditional manner, but additionally provide real-world scale which leads to longer XR content usage, and more local
and other factors specifically optimized for XR. These USDZ resources dedicated to other tasks, such as improved content
models can then be downloaded directly onto an XR device fidelity. Cloud XR services also allow for unique synchroniza-
and presented in the real world, maintaining their scale and tion of content across multiple devices and multiple users. For
position once placed. glTF provides an efficient, interoperable example, Microsoft Azure’s Spatial Anchors allow content to
format that minimizes the size of 3D assets, as well as the be anchored and persisted in the real world (e.g., a park play-
runtime processing needed to unpack and use those assets in ground) and then consumed on other AR devices (Microsoft
real time. Azure, 2019). This is the backbone of the game Minecraft
Earth, which allows players to augment the real world with
2.2.2 Software Development for XR virtualized Minecraft worlds using their AR-enabled handheld
smartphones or tablets (Minecraft, 2019).
Although the sophistication of XR applications can vary from While many urban areas are now outfitted with high-speed
very detailed and immersive VR solutions to AR applications fiber optic internet connectivity from the hundreds of Megabits
that consist of only overlaying simple 3D modeled content onto to even Gigabit per second (Gbps) range, there is great promise
the real world, most XR applications require software engineers in wireless connectivity in the near future obtaining similar or
to develop interactive content (XR Association, 2019). This even faster speeds with the advent of 5G, which boasts theoreti-
typically involves importing 3D models into an Integrated cal speeds of up to 20Gbps (Qualcomm, 2019). Combined with
Development Environment (IDE), such as Unity, and exercising its wireless connectivity and intended integration into mobile
software engineering development skills to bring the content to devices, internet speed may present a much less or even miti-
life. To assist software developers, SDKs have been developed gated bottleneck in the propagation of XR Cloud services in the
that are optimized for XR and provide low-level tools and near future.
functionality needed to develop XR applications. Typically,
aggregate libraries that support multiple platform specific
SDKs, such as Apple’s ARKit and Google’s ARCore, are
wrapped into collective SDKs such as Unity’s AR Foundation, 3 DESIGN AND IMPLEMENTATION STRATEGIES
which allows software developers to target both platforms with While many conventional human–computer interaction tech-
the same code. Exclusive features to specific platforms are still niques can be used to design and implement XR systems, there
available and engineers can properly architect their code to take are unique design and implementation considerations that must
maximum advantage of specific platform capabilities, while be addressed. In particular, given the multi-space realm of
creating a common code base for common XR functionality. XR applications, users need to be provided with natural and
For example, Microsoft’s Mixed Reality (MR) Toolkit is a intuitive designs that are ubiquitous with their environment.

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 789

Designing for the goal of UCD will first inform the types that one can either virtually (VR) or physically (AR) embody
of user activities and behavioral responses from the system perception and manipulate content in 3D space (Johnson-
that need to be created. Then, specific XR design principles Glenberg, 2018). Results are mixed with regard to whether
can be considered to ensure naturalistic XR interaction is or not these immersive benefits enhance learning, with some
achieved. In particular, designers must consider the appropriate studies demonstrating enhanced human performance with
spatial and behavioral interactions, conversation, multi-space, higher levels of immersion (Bowman & McMahan, 2007),
navigation and wayfinding guidance, and spatialized auditory while others suggest there may even be an adverse effect on
information design principles that will lead to the creation of human information processing (see Chapter 5 by Wickens and
an engaging experience that promotes presence and immer- Carswell in this volume), perhaps because of distractions from
sion, while minimizing cybersickness. XR design principles the real world (Oh, Herrera, & Bailenson, 2019). Thus, before
and guidelines are ultimately focused on taking advantage adopting an XR solution, it is important to determine if an
of the multi-dimensionality of XR in such a manner that a immersive environment is appropriate.
user-centered, intuitive, engaging, and non-sickness-inducing Once immersion has been determined to make sense for a
immersive experience is realized. given application, the second XR-specific UCD phase is to derive
the requirements of use in order to determine which type of
immersion is most appropriate, as VR and AR/MR systems offer
3.1 User-Centered Design in XR different experiences.
The goal of UCD for immersive XR environments is to cre- In VR, users are transported to and completely immersed
ate a solution that is ubiquitously effective and engaging for in a virtual environment (see Figure 7). For VR, there needs
target users. In The Design of Everyday Things, Don Norman to be justification for over-riding the real world (e.g., cost,
states a ubiquitous design derives from “an approach that puts safety, inaccessibility, etc.). For example, a specific training
human needs, capabilities, and behavior first” (1990, p. 8). environment that is out of reach physically or too dangerous to
These basic principles for UCD need to be carefully instan- directly train in could be reproduced in VR in order to match the
tiated in XR applications if these solutions are to reach mass operational environment (Champney, Carroll, & Surpris, 2014).
adoption. Specifically, the first phase of XR-specific UCD is For instance, Walmart’s diversity, inclusion, and situation VR
to analyze the target context of use including users, goals, trainer provides a well-suited use case, as it allows trainees
tasks, and environments (de Clerk, Dangelmaier, Schmierer, & to experience non-typical situations (e.g., Black Friday, dis-
Spath, 2019) to determine if there is justification for immersing criminating situations with fellow employees/customers) and to
users in an extended reality. When a user is technologically practice their reactions without having to recreate the situation
engulfed in and surrounded by a vivid, multi-sensory alter- at a larger scale or with live actors (Akhtar, 2018). Another
native reality that transports them away from their physical use case is the application of VR in treating phobias. Stud-
reality, it is known as immersion (Witmer & Singer, 1998). ies have found successful treatments of spider phobias when
Immersion may lead to presence, which is a subjective psy- users confronted their fears in the virtual world (Miloff et al.,
k chological experience of actually “being” in an alternative 2019). After receiving this treatment, participants experienced k
reality. Key benefits of immersion and presence relate to significant reductions in behavioral avoidance and self-reported
XR’s more realistic spatial cues (e.g., stereopsis, motion par- fear. In general, VR is expected to fill use cases in the areas of
allax) that support higher levels of spatial understanding as entertainment and video games (Burt, 2019), social networking
compared to standard, non-immersive displays (Ragan, Sown- and collaboration (Cook et al., 2018), manufacturing design
dararajan, Kopper, & Bowman, 2010), as well as the ability (Mamiit, 2016), employee training (Sonalkar et al., 2020),
to take advantage of physical, whole-body interactions such education (Markowitz et al., 2018), therapy (Bailey et al., 2017;

Figure 7 Virtuality continuum in context.

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 790

Bullock et al., 2019; Rizzo, 2019), rehabilitation (Koenig, Krch, outcome. Their AR application allows users to preview virtual
Lange, & Rizzo, 2019), customer interaction support (Akhtar, options overlaid onto a physical showcase kitchen. This left
2018), and more. users feeling confident in the product and informed about their
In MR, users remain in the real world and non-real, virtual decision (Akhtar, 2018). Overall, in pre-purchase situations
sensory information, such as sound, video, graphics, haptics, (e.g., “trying on” clothes before going to a store; International
and olfaction, is “smartly” overlaid onto the real world, thereby Data Corp., 2017), retail showcasing (e.g., foreseeing how a
allowing users to physically interact with and manipulate both living room would look with new decorations; Dacko, 2016),
real and virtual items and environments (see Figure 7). A pri- experience augmentation (e.g., visitors of a museum viewing
mary benefit of MR is that it can provide contextually anchored digital information [history, opinions, etc.] or hearing sound-
information, which allows the system and user to react based scapes (Janer, Roma, & Kersten, 2016; Sikora, Russo, Derek &
on the context of the real world. GPS location, environmental Jurčević, 2018) superimposed onto exhibits they are viewing),
surfaces, real-world textures, and real-world objects all provide assistive technology (e.g., supporting day-to-day affairs, such
contextual channels to information (Ajanki et al., 2011). Thus, as navigating public transportation; Oesterreich & Teuteberg,
MR alters and adds to what one perceives in the real-world envi- 2018), and process manufacturing job aids (e.g., putting a
ronment, allowing users to interact hands-free in contextualized device into location to show how it fits into an overall mecha-
scenarios and at real-world scale. For instructional applications nism; de Souza Cardoso, Mariano, & Zorzal, 2019; Palmarini
that do not require simulated environmental factors for training et al., 2018), which benefit from contextualization of 2D and
(e.g., a large-scale accident, a fire, or other virtual contexts that 3D animations, vivid instructions/information, and ability to
are difficult, dangerous, or costly to recreate in the real world), annotate, AR technology is a strong use case.
MR provides the opportunity to engage with real-world objects While a primary benefit of both VR and AR/MR is that they
in augmented space, walking around, interacting with, and allow users to have virtual embodiments, which enhances both
physically exploring and learning from both the real world and ease of user interaction and presence, particularly by providing
augmented entities. This was possible for doctors at Cleveland an anchor for visuomotor tasks (Borrego, Latorre, Alcañiz, &
Clinic who, after teaching anatomy in MR, saw a significant Llorens, 2019), they do so in different ways and thus one must
increase in recall and performance compared to those who did carefully determine which form of embodiment meets the needs
not receive MR training (Akhtar, 2018). Overall, for onsite of a given XR application (Matamala-Gomez et al., 2019). In
assembly support, safety operations support, and process man- general, there are three specific subcomponents that can create
ufacturing, and other forms of hands-on training, which benefit a sense of embodiment: sense of self-location, sense of agency,
from contextualization of information, embodiment of actions and sense of body ownership (Kilteni, Groten, & Slater, 2012)
and hands-free operation, MR technology is a strong use case (see Figure 8).
(International Data Corp., 2017).
In AR, location-based digital content is overlaid on users’ • Sense of location (i.e., a sense of residing inside a body)
real-world surroundings in real time to inform or guide (see can be fostered by providing a first-person visual per-
k Figure 7). AR can enhance everyday experiences by aiding perspective. k
ception (see Chapter 3 by Proctor and Proctor in this volume), • Sense of agency (i.e., a sense of being able to initiate
comprehension, and interaction with the real world. Lowe’s and control actions) is fostered when users are able
Home Improvement utilized the contextualization that AR to interact with and control their movements in active
offers after they identified a pain point with their customers: XR scenarios, especially when they are pursuing a goal
the inability to commit to remodeling without seeing the and there is a possibility of achieving that goal through

(a) (b) (c)

Figure 8 Three components of embodiment: (a) sense of location; (b) sense of agency; (c) sense of body-ownership.

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 791

self-directed physical movement even though their body be the prominent view; (3) designing content at proper scale,
is not physically in the virtual space (Gorisse, Christ- where AR scene content is rendered exactly where it would be
mann, Houzangbe, & Richir, 2019; Piccione, Collett, & in the physical world to real-life size and depth, encourages
De Foe, 2019). users to explore and experience spatial and temporal concepts,
• Sense of body ownership (i.e., a sense of self-attribution provides a means of emphasizing contextual relationships
of a body) can be fostered by increasing the sensory cor- between real and virtual objects, and provides intuitive inter-
relations between the actual physical stimulation of one’s action via a contextually valid dynamic, first-person point of
biological body and the visually seen stimulation of an view. Interactions at real-world size/scale can help to create
avatar’s body (see Chapter 29 by Duffy in this volume) a physical mental model of user actions, such as the optimal
or body part and ensuring the avatar’s body obeys struc- embodied manner to complete a task. It is important to note that
tural and morphological constraints such that it appears ‘scale’ is a multi-dimensional construct and involves design
human-like (Braun et al., 2018). consideration across personal (within arm’s reach), action
(within 30 m), and vista (beyond 30 m) spaces (Cutting &
In VR, embodiment is generally achieved by using a self-avatar Vishton, 1995).
to represent and replicate a user’s body posture and motions Thus, XR exists on a spectrum, where on one end is the real
using body-tracking systems (Gonzalez-Franco & Lanier, 2017; environment and on the other is the completely virtual. Connect-
Spanlang et al., 2014). Lee (2004, p. 376) explains that “In ing the two ends, augmented reality transitions into mixed, with
the case of a psychologically assumed virtual self … a virtual the biggest contrast being the overlay to anchoring of virtuality
environment reacts to users as if they were in there.” While in onto reality.
VR, the real body is dissociated from the virtual representation, The third XR-specific UCD phase is to apply design prin-
embodiment can be effectively achieved for given body parts ciples that ensure immersive design solutions satisfy require-
by factoring in the three components: (1) first-person visual ments of use identified in the first two phases. UCD principles
perspective; (2) being able to initiate and control actions; and for AR/MR versus VR experiences vary to some degree.
(3) mapping users’ physical movements to avatar actions, as AR/MR-specific design principles include:
much as possible. For example, Gilbers (2017, p. 46) demon-
strated “the possibility to experience ownership over a third • create a reality enhancing system—one that is “transpar-
virtual hand, as more than 25% of the participants (in their ent and more a part of the users’ perceptive system than
study) experienced that they had three hands.” VR-based a separate entity in itself” (Nilsson, 2010, p. 147);
embodiment is most appropriate when users need to experience • consider AR/MR technology and user’s limited field of
presence and body ownership. A user’s sense of location can be view in relation to their space; ensure users have enough
altered when they are provided a rich environment to experience room to properly view augmented content;
and create a dynamic viewpoint for themselves; “Embodiment • determine critical versus non-critical information and
performs a spatial narrative of the landscape and the story” display these information elements with a correct amount
k (GitHub, 2020). Body ownership of a different persona allows a of obscurity/opacity. k
user to gain perspective due to: (1) the emotional connection that
is built when inhabiting a different body; and (2) experiencing VR-specific design principles include:
the virtual world react to the user’s actions while embodying a
different being. Body ownership can build strong feelings that • minimize cybersickness within VR by providing active
can affect intrinsic motivation. viewpoint control, reducing sensory conflicts, providing
leading indicators, and providing rest frames (Stanney,
If one feels embodied in a virtual body, insults or praise Fidopiastis, & Foster, 2020);
regarding this body, referring to properties that would • consider the 360 environment and provide appropriate
not be true for the biological body, could cause emotional visual and spatial audio cues to define distance (Loyola,
arousal … one would manifest strong reactions to a 2017; Rana, Ozincar, & Smolic, 2019);
threat in the case where it refers to one’s own body, • provide boundary markers (see Figure 6) to ensure user
compared to when there is no sensation of ownership. safety within the real environment (Dingman, 2019;
(Kilteni et al., 2012, p. 382) Hartmann, Holz, Ofek, & Wilson, 2019).

In AR/MR, embodiment (Rosa et al., 2019) is generally General immersive design principles include:
achieved by allowing users to see their real body and providing
virtual hands, extended bodies, or extra limbs when interaction • rely on sound to provide a well-rounded, situated, and
occurs outside of personal space (Cutting & Vishton, 1995). spatially oriented experience, (Janer et al., 2016; Sikora
Embodiment in AR/MR is most effectively achieved by: (1) pro- et al., 2018), as when there is a congruence between
viding users with control of virtual and real objects. Users are visual and auditory stimuli users experience a height-
more engaged when their actions impact the environment, as ened sense of presence (Hruby, 2019), are more adept
“overlapping of cause and event creates a substantive relation- at navigation (Bormann, 2005), and more effective at
ship … thus establish[ing] a greater awareness of the role of search and other such tasks (Rumiński, 2015);
the body … producing a more intuitive and intimate experience” • consider user’s field of view and understand perceptual
(GitHub, 2020); (2) using sensors to track the position of user’s limitations (e.g., distortion due to the structure of the
hands in relation to virtual objects, thereby enabling users environment and its interaction with augmentations;
to pick up virtual objects, manipulate them, and move them issues associated with headset field of view, resolution,
about. This embodied interaction with virtual content has been display size, and contrast; content capturing issues such
demonstrated to enhance learning (Santos et al., 2014). Further, as image resolution and capture frame rate; issues with
presentation of virtual objects and real-world elements must be the design, layout, and registration of content and aug-
choreographed in such a manner as to create interactions that mentations; issues associated with users and their ability
best determine when users need to have their hands/body visible to perceive and interact with the virtual world; Kruijff,
within the field of view versus when AR/MR elements should Swan, & Feiner, 2010);

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 792

• provide natural interactions that mimic real life by perception–action couplings if the full range of expected sen-
creating intuitive affordances, as intuitiveness relies on sory cues is not supported or if it is supported in a manner that is
experiences with the physical environment (e.g., using not contiguous with real-world expectations. Such pitfalls can
our hands to directly manipulate virtual objects). Pay be avoided through consideration of the coordination between
special attention to the degree of indirection (i.e., spatial sensing and user command and the transposition of senses in
or temporal offset between interaction elements and the feedback loop (Stanney, Samman, Reeves, et al., 2004;
manipulated objects, seeking minimal offset), degree Stone, Bisantz, Llinas, & Paquet, 2009). Specifically, command
of integration (i.e., ratio of degrees of freedom of coordination considers user input as primarily monomodal and
task elements versus their input devices, seeking a 1:1 feedback to the user as multimodal. Designers need to consider
ratio), and degree of compatibility (i.e., correspondence (Wickens, Hollands, Banbury, & Parasuraman, 2016):
between physical actions of user interactions and those
of manipulated objects, seeking direct manipulation; • which input modalities are most appropriate to support
Poor et al., 2016). execution of a given task within the XR environment
(e.g., time-sensitive information is best presented
Other design considerations for this UCD phase will be consid- auditorily, while spatial information is best presented
ered in Section 3.2, XR Design Principles. visually);
The fourth XR-specific UCD phase is to carry out an eval- • how these modalities can be supported (e.g., either
uation to ensure resulting designs are effective, intuitive, and directly or via perceptual illusions) and where that
satisfying to use, without causing undue adverse discomfort support should occur (e.g., focal or ambient);
during use or adverse aftereffects upon post-exposure. Gabbard • if there is any need for redundant user input (e.g., add
(1997) developed a taxonomy of usability characteristics for redundant audio cues to reduce visual demands);
immersive environments, from which Stanney, Mollaghasemi, • whether or not users can effectively handle such parallel
Reeves, Breaux, and Graeber (2003) developed the MAUVE input.
(Multi-Criteria Assessment of Usability for Virtual Environ-
ments) approach, which organizes usability characteristics into Further, cross-modal integration rules can be used to consider
two primary usability attributes (XR system usability and XR aspects of multimodal interaction, including (Johnson-Glenberg,
user considerations); four secondary attributes (interaction, 2018; Stanney et al., 2004; Storms, 2002):
multimodal system output, engagement, and side effects);
and 11 tertiary attributes (navigation, user movement, object • temporal and spatial coincidence to ensure natural per-
selection and manipulation, visual output, auditory output, ception of perception–action couplings;
haptic output, presence, immersion, comfort, sickness, and • working memory capacity to ensure users are not over-
aftereffects). MAUVE is an inspection method that can be used whelmed by the demands of interaction;
at various stages in the development lifecycle, from initial sto- • intersensory facilitation effects (e.g., a visual display
k ryboard design to final evaluation and testing. Beyond usability, enhanced by the addition of an auditory display) that can k
the cost-effectiveness of XR systems should also be evaluated enhance perception in XR environments;
(Wang, Wu, Wang, Chi, & Wang, 2018). With these aspects • congruency in associated sensorimotor areas (i.e., intu-
considered, developers can evaluate if XR technology offers itive match between design of feedback from different
performance and financial advantages over current practices modalities);
or technologies. This is an essential determination if XR tech-
nology is to achieve mass adoption both commercially and in • inverse effectiveness (i.e., multisensory inputs interact
research domains. synergistically when the unimodal stimuli occur at
relatively the same time and place and are relatively
weak).
3.2 XR Design Principles
Additional multimodal design guidelines have been provided by
Immersive XR environments need to be designed to provide Hale et al. (2009), who have outlined how a number of sensory
users with engaging experiences that allow for direct manip- cues may effectively be used to enhance specific SA compo-
ulative and intuitive multisensory interaction (Bullinger et al., nents (i.e., object recognition, spatial, temporal) within an XR
2001), with the ultimate goal of achieving human–human environment, with the goal of optimizing SA development.
communication/human–system interaction that is as natural as Multimodal interaction design guidelines continue to evolve
possible (Reeves et al., 2004). If designed effectively, engage- and offer techniques to enhance the user experience of immer-
ment in such immersive multimodal XR experiences can lead sive XR environments (cf. Almeida, Teixeira, Silva, & Ketsmur,
to high levels of situation awareness (SA) and in turn high 2019), including perfecting the design of fundamental XR inter-
levels of human performance (Hale et al., 2009); however, mul- action use cases (Ravasz, 2016):
timodal interaction within XR must be appropriately designed
to lead to this enhanced awareness. To achieve effective and Role of ground. Users must be able to relate to and have
engaging XR applications, there are several design factors to a sense of orientation within an immersive XR environ-
consider, including: natural interaction design, conversation ment, thus the ground-to-horizon relationship is critically
design, behavior and interaction fidelity, perceptual illusions, important (Gibson, 2015). In VR, a fixed horizon can pro-
multi-space design, navigation and wayfinding design, and vide such orientation, while also reducing cybersickness
spatial sound design. (Prothero & Parker, 2003). In AR, interaction with vir-
tual objects is affected by their perceived distances, which
may be biased if these objects are not perceived as mak-
3.2.1 Natural Interaction Design ing contact with the ground plane (Rosales et al., 2019).
To design natural XR interaction, it is essential to consider Thus, for AR it is beneficial to use shadows and inter-
rules governing integration of multiple sources of sensory reflections (i.e., light reflecting from an object’s surface
feedback. People have adapted their perception–action systems to the ground surface) to provide cues regarding whether
to “expect” a particular type of information flow in the real or not an object is in contact with the ground or other
world (Poor et al., 2016); XR runs the risk of breaking these support surface.

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 793

Atmosphere. Providing aerial or atmospheric perspective which requires designing a persona that can represent a familiar
(i.e., using color changes to create the illusion of depth entity while providing natural communication based on the user.
by simulating atmospheric hue changes at different dis- Coined by Grice in 1975, the cooperation principle is a natural
tances) can support users in scaling an XR environment, form of communication in which both participants in a conver-
thereby making the experience feel more intuitive and sation are believed to want to cooperate (Grice, 1975). This is a
natural. Specifically, objects in action or vista space helpful standard when creating sample dialogs. After creating a
(Cutting & Vishton, 1995) would have more air and persona, Google’s conversation guidelines recommend drafting
particles to look through to view them, making these sample conversations between the persona (system) and user
distant objects look less sharp and blurrier than close to experiment and inform component driven prompts. Scripts
objects (Goldstein, 2013). In general, gradually fading can follow the process of natural conversation turn-taking: User
the landscape from personal to vista space provides a utterance, persona situation identification, persona response,
clear cue for depth and distance. and persona conversational turn ending prompt. For advanced
Terrain features. Systematically using terrain features, systems, components may include several variants to struc-
including ground, path, obstacle, barrier, water margin, tured dynamic, rule-based sentences. Effective principles for
brink, step, and slope (Gibson, 2015) as the building conversation design include:
blocks of an XR environment will result in a natural and
intuitive interaction design guided by human intuition. • rely on existing models of human-to-human conversation
when designing conversations;
Approach to the environment through soundscape. Users can
be gently immersed into an XR environment through the • anticipate and build the system to understand multiple
use of an ambient soundscape that helps the user to build user utterances, including providing feedback for those
utterances;
a mental model of the environment before being fully
immersed into it. Further, in XR environments, the way • limit the amount of information provided to users to
sound propagates should be altered by the structure of a avoid information overload and user frustration, provid-
space, its architecture, textures, and general organization, ing no more than three different options per interaction.
thus directing how a space is perceived (Herzer, 2014; Group together a longer list and prompt users to explore
Janer et al., 2016; Sikora et al., 2018). more options if necessary (via voice response).
Wayfinding with objects. Although having an XR environ- • provided that a screen is not available, inform users of
ment cluttered with a bunch of virtual elements might functionality that they are currently accessing;
break user’s immersion, such objects can be highly • similar to natural conversation, avoid technical jargon to
effective in guiding users through the environment if help reduce cognitive load;
they are contextually relevant (e.g., users can be guided • provide sequential numbering of results, to act as verbal
via directional AR prompts overlaid onto the real-world “handles,” from which users will easily recall and select;
k setting to guide them to the most efficient route). In • create effective and memorable interactable words for k
general, egocentric-updating of location is crucial for users;
navigating and wayfinding in unfamiliar environments,
• after a persona has responded, set a response time limit
and immersive XR environments can facilitate this pro-
that, once reached, creates another response prompt to
cess through the provision of prominent features (e.g.,
verify if users still want to continue the conversation.
landmarks, well-structured paths, regions with varying
sensory characteristics, signs at decision points) present Following these design standards, conversation design in XR
along the route (Sharma et al., 2017). can allow users to comfortably speak and know what to say
Contextual reticle. To orient users in space, a contextual intuitively.
reticle can be used to show the center of focus, direct
gaze, and support movement and interaction with objects. 3.2.3 Behavior and Interaction Fidelity
Different states of the user, such as idle state, movement
state, and interaction state demand different interactions Studies have shown that in XR, the critical key to creating a real-
from the reticle. istic experience is to provide object and cue recognitions that are
Interactive objects. When users are in the XR environment, based on familiar behaviors that users can understand. Behavior
interactive objects should draw attention with subtle fidelity can be created by designing reactions of familiar objects
changes, such as with minor shading in the color of the and cues to follow real-world physics, gravity, and optical laws
object or even a subtle sound depicting its behavior, or (Jerald, 2015). Behavior fidelity of objects can be realized in
with the help of a contextual reticle. XR according to (Maier & Fadel, 2009; Poor et al., 2016; XR
Association, 2019) by:

3.2.2 Conversation Design • using physics collisions to indicate interactions between

different virtual objects;
Conversation and existing voice user interfaces are a good
fit for XR systems, particularly for AR, given the hands-free • using physics colliders that indicate to users they have
capabilities that voice interaction can provide and the desire for moved into an object (e.g., a virtual wall) even if the
‘reality-based’ interaction (Girouard, Shaer, Solovey, Poor, & object is outside of the user’s current field of view;
Jacob, 2019). With the multimodal capability of XR systems, • carefully designing two-handed interactions in XR, as
a “say what you see” convention can help guide users to say they are difficult to convey unless users’ hand represen-
the correct prompts and constrain user input (Yankelovich, tation can be veridical in the virtual world;
1996). Voice is an efficient input modality, allowing hands-free • conveying the weight of virtual objects by, for example,
interactions; in contrast, visuals are an efficient output modality, providing resistance to the raycast used to select them,
allowing presentation of a vast amount of information at the such as by having the raycast bend a small amount for
same time, which in turn reduces the burden on users’ mem- light objects and larger amounts for heavier objects,
ory (Whitenton, 2017). Google’s (2020) conversation design thereby creating the illusion that virtual objects have
guidelines outline how to build a VUI (Voice User Interface), mass;

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 794

• infusing audio into 3D objects and/or adding attenuation world) is known to be enhanced via a number of dis-
to sound effects by allowing the audio to fade and stop play factors, including a wide field of view and high
when users are no longer interacting with the objects; spatial frequency content (Hettinger, 2002), moving
• using affordances to subtly communicate the properties sounds (Riecke, Väljamäe, & Schulte-Pelkum, 2009),
of an artifact to afford its potential uses to users. and viewpoint oscillation (Kim & Khuu, 2014; Kitazaki,
Onimaru, & Sato, 2010).
In addition, interaction fidelity (i.e., action realism, wherein • change blindness (i.e., failing to notice alterations in a
users have the ability to directly control and manipulate the visual scene) can be used to apply subtle manipulations
XR environment) can be created by designing user actions in to the geometry of a virtual world and direct movement
XR to resemble real-world actions in terms of biomechanical behavior, such as redirecting a user’s walking path
similarity, input, and control. Guidelines for interaction fidelity throughout a virtual environment (Hartmann et al., 2019;
include (Rogers et al., 2019): Suma et al., 2010);
• acoustic illusions (e.g., a fountain sound; Riecke et al.,
• provide high interaction fidelity to support object manip- 2009; Riecke, 2016) could be used to create a sense of
ulation tasks in XR; vection in an XR, even when no such visual motion is
• provide moderate interaction fidelity for whole-body provided;
movements in XR, as abstractions that reduce physical • haptic illusions (Collins & Kapralos, 2019; Hayward,
demands or intricate movement may increase usability 2008) could be used to provide users with an impres-
and ease of use; sion of actually feeling virtual objects when they are
• provide designs that encourage exploration in XR such in fact touching real-world props or traveling along
that users interact with the virtual space (e.g., novel forms a trajectory path that may even vary in size, shape,
of navigation and orientation; use of search-based game weight, or surface from their virtual counterparts with-
mechanics); out users perceiving these discrepancies (e.g., feel
• consider onlooker effects, which can engender feelings an illusory bump when actually touching a flat surface
of self-consciousness during XR whole-body or, contra- (Robles-De-La-Torre & Hayward, 2001); feel an illusory
rily, decrease feelings of isolation to XR interaction; sharp edge when hand actually travels along a smooth
trajectory (Portillo-Rodríguez et al., 2006)).
• for smartphone- and tablet-based AR, as visual localiza-
tion of objects can be very dependent on how the device One solution to current technological shortcomings, sensorial
is being held, provide indicators to guide object localiza- transposition, occurs when a user receives feedback through
tion, such as blur, contour scale, or vocal hints; senses other than those expected, which may occur because
• for smartphone- and tablet-based AR, provide interac- a command coordination scheme has substituted available
tions that consider building for use of two hands and pro- sensory feedback for those that cannot be generated within a
k vide clear instructions that illustrate the role of the hands virtual environment. According to Stiles and Shimojo (2015),
k
during those interactions; cross-modal mappings are ubiquitous and include such things
• consider haptic substitutions, such as tapping two fin- as the intuitive matching of high frequency sounds to vertically
gers to each other for sensory replication or by providing elevated locations, scan time to horizontal/left-right coordi-
visual/audio feedback, to increase immersion; nates, brightness to loud sounds, and the very basic more to
• provide substitutions (e.g., perceptual illusions) and up/less to down. Such multimodal mappings may be critical to
approximations for physically challenging whole-body ensure sensory substitution devices are intuitive.
movements (e.g., holding a button to simulate holding Sensorial substitution schemes may be one-for-one (e.g.,
on to a virtual object), which may increase interaction visual for force) or more complex (e.g., visual for force and
fidelity. auditory; visual and auditory for force). If designed effectively,
command coordination and sensory substitution schemes can
provide multimodal interaction that allows for better user
3.2.4 Perceptual Illusions control of XR environments. On the other hand, if designed
When sensorial transpositions are used, there is an opportunity poorly, these solutions may in fact exacerbate interaction prob-
for perceptual illusions to occur. With perceptual illusions, lems. Some design principles to consider when implementing
certain perceptual qualities perceived by one sensory system sensorial substitution schemes include (Kristjánsson et al.,
are influenced by another sensory system (e.g., ‘feel’ a squeeze 2016):
when you see your hand grabbing a virtual object). Such illu-
• ensure sensorial substitution schemes only convey criti-
sions could simplify and reduce the cost of XR development
cal information to avoid overloading human information
efforts (Debarba et al., 2018; Nilsson, 2018; Storms, 2002).
processing (see Chapter 5 by Wickens and Carswell in
For example, when attending to a visual image coupled with
this volume);
a low-quality auditory display, auditory–visual cross-modal
perception allows for an increase in perceived quality of the • ensure sensorial substitution schemes are tasks-focused,
visual image. Thus, in this case, if the visual image is the focus as bandwidth differences between sensory systems
of the task, there may be no need to use a high-quality auditory severely constrain the nature and amount of information
display. that can be conveyed;
There are several types of perceptual illusions that can be • consider more than one sensory feedback modality, as the
used in the design of XR environments (Debarba et al., 2018; design of multisensory integration may not necessarily
Nilsson, 2018; Steinicke & Willemsen, 2010): be confined to one source of feedback;
• assess the nature of spatiotemporal continuity for dif-
• visual illusions can be used to replace missing pro- ferent senses, as perception is a continuous process (see
prioceptive and vestibular senses, as vision usually Chapter 3 by Proctor and Proctor in this volume) and
dominates these senses. For example, vection (i.e., a thus sensorial substitution schemes must consider how
compelling illusion of self-motion throughout a virtual perception changes over time.

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 795

3.2.5 Multi-Space Design & Fidopiastis, 2004). Navigation cues are also important
Designing for a multidimensional space has been an artform for in personal space, specifically as they relate to guiding
decades. Whether layering a painting (i.e., with a foreground, users in where to orient their heads such that their FOV
mid-ground, and background) or creating forced perspective on is positioned to take in priority information.
a theater stage (i.e., as developed during the Italian Renaissance, • As our bodies contribute to the perception of what is real
where principles of perspective allow designers to create vistas (Lakoff & Johnson, 1999), for personal space to be per-
with objects decreasing in size toward a “vanishing point” on ceived as authentic it is important to foster embodied per-
the horizon, such as conveyed via a proscenium “picture frame” ception. This can be achieved by integrating many senses
stage), the foundation for the design of space and depth has into the design of user interactions, such as touch, posi-
evolved over time. Cutting and Vishton (1995) suggest three tional awareness, balance, sound, and movement through
separate spaces around a moving observer that are important which an embodied knowing of complex behaviors can
to layout and distance based on their ordinal depth-threshold be derived within the virtual space (Wilson & Golonka,
functions, including personal space, action space, and vista 2013).
space. In virtual XR spaces, design principles can be created • Consider experiential qualities of virtual spaces in addi-
specifically for these three separate, user interaction spaces (see tion to visual properties, such as the ability of the body
Table 2). to “read” the environment via haptic cues, which can be
Personal space is described as the “zone immediately sur- derived by requiring users to move through space, engage
rounding the observer’s head, generally within arm’s reach with it, and have memorable physical experiences along
and slightly beyond” (Cutting & Vishton, 1995, p. 100). When the way (Lyndon & Moore, 1994), as well as providing a
immersed in an XR environment, users will perform the bulk feeling of heat and humidity, momentum of gait and body
of their interactions within this space. In the real world, this inertia as one traverses a virtual or augmented space, and
surround is typically regarded by a user as their “own” space echo of footfalls, all of which convene to define tactile
and is portable, morphing in terms of size and proportion as one resilience of place-form (Frampton, 1983);
interacts with the environment (Sommer, 1969). The same must • Foster spatial stability (Dessing, Crawford, & Meden-
be achieved in XR spaces if they are to be natural and intuitive. dorp, 2011), despite continuous changes in sensory and
Thus, within an XR environment, personal space should not motor inputs owing to movement of eyes, head, and body,
be designed as a passive “out there” structure; it should be to foster haptic understanding (O’Neill, 2001);
created to support a complex and dynamic exchange in which • Avoid locking UI elements into the corners of an XR
the virtual space informs human knowledge and affords user headset, rather, position content either in 3D space with
interaction, thereby shaping the manner in which users come to a body-locked behavior (tag-along) or in a world-locked
know the virtual space. Some principles by which an authentic position (Microsoft, 2019a). Such UI elements are a type
personal space can be achieved include: of information source that can support interaction in per-
k sonal and action space (see Table 2). k
• Design visual cues carefully as they are critical in per-
ceiving personal space, including (from most critical It is essential that personal space be designed to accommodate
to least) occlusion, binocular disparity, relative size, action. If a user is situated in a location that does not accom-
accommodation and convergence, and motion paral- modate action, the system should automatically identify such
lax generated by head movement and manipulation of physical limitations within this space and recommend the user
objects (Cutting & Vishton, 1995). It is important to note move to a better location before beginning the experience (i.e.,
that depth order from occlusion can rely on contrast and a feature many VR games offer today). This type of contextual
on fine details (as low as 1 arc min), which may not be cuing from the real-world environment is useful within personal
perceivable at certain sizes and distances, thus needing space because of the ability to detect the real world and notify
to be altered depending on the XR HWD (Rolland, Ha, the user of physical space requirements.
Action space extends from the end of personal space to about
30 m—“Because the utility of disparity and motion perspective
Table 2 Information Source Types Within Areas of XR decline to our effective threshold value of 10% at about 30 m
Space … this effectively delimits space at 30 m” (Cutting & Vishton,
1995, p. 101). Within this space, users can walk and run. Dream-
Personal space Action space Vista space works suggests in their VR Storyboard Convention model that
Occlusion, X X X this is the space narrative occupies. The mobility of users sug-
interposition gests that it is critical to provide area boundaries in this space to
avoid physical collision. In addition, in this space, cues that must
Relative size X X X carefully be designed are occlusion, height in the visual field,
Motion parallax/ X X motion perspective, and relative size, however, the small field of
perspective view and low resolution of many XR HWDs may compromise
Height in visual field X X some of these cues. Accommodation, convergence, and binoc-
Ariel perspective X ular disparity cues within an XR HWD will likely not support
Texture gradients X
human depth perception in action space. Some of the principles
by which an authentic action space can be achieved include:
Accommodation X
and convergence
• Use an invisible “boundary system” to map out action
Binocular disparity, X space, including boundaries around real-world obstacles
stereopsis so users can be warned if they are about to collide with
Contextual cues X X X an object or have reached the limit of the occupiable real
Navigation cues X X X space (Dingman, 2019; Hartmann et al., 2019; XR Asso-
Boundary area X
ciation, 2019). XR headsets, such as the HoloLens, can
provide spatial mapping to assist in boundary creation.

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 796

• Use subtle contextual cues to help set user expectations knowledge to users, helping to enrich their cognitive
for how far they can roam within action space, such as map (cf. Minocha & Hardy, 2016).
by using a virtual floor that changes material or texture, • Avoid placing critical information in vista space, as dif-
fog that gets denser as users roam toward the edges of ficulty in translating information in vista space can cause
active space, or grid lines that show the actual bound- misinterpretation (National Research Council, 1997).
aries of action space (Hartmann et al., 2019; XR Associ- • Given the amount of likely movement in vista space, pro-
ation, 2019). vide users with macro navigation (i.e., selecting routes
• Support object manipulation within action space (Poor over long distances) cues that do not obscure the envi-
et al., 2016; XR Association, 2019) by doing the follow- ronment.
ing:
• Set maximum placement distance defaults to help
ensure users place objects at comfortable view- 3.2.6 Navigation and Wayﬁnding Design
ing distances, have objects maintain realistic scale To support interaction in multi-space XR environments, it is
as users drag them around the action space, pro- essential to carefully design for navigation and wayfinding.
vide visual indicators for user’s awareness about Effective XR interaction design can be impeded if navigational
system-interpreted user gaze (e.g., raycast), and complexities arise. Navigation is the aggregate of wayfinding
use visual indicators for destination points (e.g., (e.g., cognitive planning of one’s route) and physical movement
shadows) to help users understand where objects that allows travel throughout a virtual environment (Darken &
will be placed on a detected surface. Peterson, 2002). A number of tools and techniques have been
• Anticipate proximity as an attempt by users to inter- developed to aid wayfinding in virtual worlds, including maps,
act with objects. If a user makes a gesture in proxim- landmarks, trails, and direction finding. These tools can be
ity of an object, consider providing feedback regard- used to display current position, current orientation (e.g., com-
less of the type of gesture. pass), log movements (e.g., breadcrumb trails), demonstrate or
• Provide virtual objects with surface detection so that access the surround (e.g., maps, binoculars), or provide guided
objects don’t appear to be levitating. movement (e.g., signs, landmarks) (Chen & Stanney, 1999).
For example, Burigat and Chittaro (2007) found 3D arrows to
• Given the amount of likely movement in action space, be particularly effective in guiding navigation throughout an
provide users with micro navigation (i.e., fine adjust- abstract virtual environment. A number of principles concerning
ments needed to orient users to desired location) cues how best to design and use navigation and wayfinding tools
that do not obscure the environment. in XR have been developed (cf. Minocha & Hardy, 2016),
Finally, vista space is beyond 30 m from users. Daum including:
and Hecht (2009) suggest that vista space should be
divided into near vista space (i.e., up to 100 m) and far • Within smaller XR environments, ensure navigation
k vista space (i.e., >100 m) because underestimation and design takes into account the proximity of users to k
spatial compression in near vista space are expected physical and virtual objects, as field of view limita-
to be roughly proportional, whereas in far vista space tions within the HWD can present significant issues for
overestimation and expansion of the visual field are typi- micro-navigation.
cally over-proportional; thus, different design principles
are needed in these sectors. Vista space is a limited • Provide bottom-up spatial cues that increase user naviga-
interaction zone, wherein perspective of imagery should tion, such as those provided by an omnidirectional atten-
replicate the real world. Occlusion, texture gradients, tion funnel (Biocca, Tang, Owen, & Fan, 2006).
height in the visual field, relative size, and aerial per- • Progressively disclose navigation, to avoid information
spective (Cutting & Vishton, 1995) are several cues overload.
that can be utilized in this space to build critical depth • Depending on users’ information-seeking goal (locating
information, however, field of view and HWD display vs exploring), provide users with information that is spe-
resolution may compromise the effectiveness of these cific to their tasks (e.g., for locating tasks, avoid extra-
cues. Contextualized spatial audio in this space can serve neous information that can confuse users; for exploring
as a helpful cue to indicate distance and provide subtle tasks, provide information about possible areas to navi-
world building information. Some principles by which gate to).
an authentic vista space can be achieved include: • Consider use of virtual landmarks to improve familiarity
• Similar to a painting, provide aerial perspective by of surroundings for users, helping to solidify orientation
replicating differences in color and opacity that occur (Foltz, 1998; Minocha & Hardy, 2016).
naturally from the atmosphere to help users intuitively • For path providence, create semi-translucent route paths
understand variations of distance (Cutting & Vishton, instead of directional arrows to eliminate ambiguity.
1995).
• In systems where users’ SA is a priority for location
• Based on the amount of effort that users may perceive based interactions, use a World-in-Miniature view in
a vista space to impose, consider changes in terrain to conjunction with AR markers to enhance users’ exocen-
achieve realism. For example, if users are tired, they may tric view (Ball & Johnsen, 2017; Elvezio, Sukan, Feiner,
perceive a path up a hill to be a longer distance (Daum & & Tversky, 2017; Stoakley, Conway, & Pausch, 1995).
Hecht, 2009).
• Consider how weather, color, and type of terrain can be If effectively applied to XRs, these principles should lead to
used to affect distance estimation for users. “Soldiers reduced disorientation and enhanced wayfinding in both large-
tend to underestimate distances when the targets are and small-scale XR environments.
clearly visible, the air is clear, the sun is at their back,
or the terrain is not fully visible or uniform” (Daum &
Hecht, 2009, p. 1129). 3.2.7 Spatial Audio Design
• Consider how global landmarks within vista space (such Audio within XR, as with films and video games, has proven
as hills and skyline) can be used to provide spatial to be important in creating an immersive experience. Schell

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 797

(2008, p. 4) explains, “sound is what truly convinces the mind • Create challenges that provide an appropriate level of
(it) is in a place, in other words, hearing is believing.” Spatially, stimulation according to a user’s skills and knowl-
environmental audio cues can live within XR spaces to provide edge level; increasing or decreasing as necessary in an
environmental awareness. Placing audio within virtual spaces adaptive manner (O’Brien & Toms, 2008).
will allow users to determine the properties of objects and to • Provide explanations for system inconsistencies to avoid
distinguish location and distance. An example of such a property breaking the suspension of disbelief.
is when you speak near a wall and feel an echo (Cortes, 2016).
• Encourage perceived user control by providing users
For users simply experiencing a soundscape, spatial audio
with multiple paths to accomplish a goal and control
can provide navigation guidance and a mental model support.
over the inconsequential (O’Brien & Toms, 2008).
Microsoft’s system, Soundscape, provides users, particularly
those with sight loss, 3D spatial audio to enrich their navigation • Create a high concept that will provide users with an
experience (Gartenberg, 2018). Some of the principles by which internal “premise” (story) that sets expectations for
an authentic spatial audio design can be achieved include (Janer system behavior.
et al., 2016; Sikora et al., 2018): • Based on user analysis, provide activities and opportuni-
ties that are relevant to users, which will increase likeli-
• Design audio to originate in the appropriate direction to hood of internal motivation to perform actions.
correlate the sound with objects, thus creating binaural
hearing to enhance presence (Witmer & Singer, 1998).
• Provide as much haptic information about an object’s 3.4 Cybersickness Remediation Design
material as possible via audio. For example, the sound Strategies
of a metal object interacting with another metal object Exposure to an XR system often produces unwanted side effects
should make a distinct sound compared to a metal object that could render users incapable of remaining immersed in the
interacting with carpet. XR environment or functioning effectively upon return to the
• When visuals are cluttered or overloaded, provide audio real world (see Section 4.1). These adverse effects may include
as an alternate solution for information transfer (National nausea and vomiting, postural instability, visual disturbances,
Research Council, 1997). and profound drowsiness (Stanney, Salvendy, et al., 1998).
• Audio should contain realistic fall-off curves that model As users subsequently take on their normal routines, unaware
loudness with distance, similar to the real world. of these lingering effects, their safety and well-being may be
compromised. XR systems and associated usage protocols
• Provide audio as a source of sensory substitution, partic-
ularly for interactions that would naturally require haptic should be designed to maximize comfort, minimize risks,
feedback, for example, the click of a button. warn users about potential aftereffects, monitor users during
exposure, assess users’ risk, and debrief users after exposure.
• Provide audio cues, such as tones, for communication of There are design strategies that can be implemented to support
k simple information and speech cues for more intrinsic cybersickness remediation, such as: k
types of information (National Research Council, 1997).
• Provide multisensory feedback that reduces sensory
3.3 Engagement Design Strategies conflicts:
Engagement involves both system-driven involvement (i.e., an • Add concordant physical motion to match visual
XR system “provides” immersion) and user-driven involvement motion (e.g., a motion base) to reduce visual-
(i.e., a user “feels” presence) (Stanney, Mollaghasemi, Reeves, vestibular conflicts (Bos, 2015).
Breaux, & Graeber, 2003). The engagement process starts • Ask users to actively align their head/body to the
with the system, as immersion is increased by encompassing behavior of the virtual motion they are experiencing
users into an XR system with as many relevant multisen- to reduce visual-vestibular conflicts (Wada et al.,
sory sensations as possible (Mestre, 2005). It is the job of 2012).
the designed system to keep users in an engagement activity • Provide users with active viewpoint control by
loop, which passes users through activities, accomplishments, using low latency head tracking (Stanney & Hash,
and resulting affect by presenting meaningful information for 1998).
novelty, motivation, and interest. It is during this activity loop
that users eventually perceive a sense of presence in the XR • Provide rest frames from which users can judge vir-
environment. Presence has been described as “the psycholog- tual motion, with an independent visual background
ical perception of ‘being in’ or ‘existing in’ the virtual world that is in alignment with a viewer’s inertial cues
in which one is immersed” (Mestre, 2005, p. 2) and “as the likely being a most effective rest frame (Stanney
subjective perception of experiencing oneself as being in a et al., 2020).
computer-generated environment rather than in one’s actual • When sensory conflicts cannot be designed out, provide
physical location” (Stanney, Salvendy, et al., 1998, p. 459). visual motion cues that match users’ vestibular system
Presence in XR is valuable because it can reproduce sensation, (Prothero & Parker, 2003), such as:
perceptions, and emotional responses (see Chapter 9 by Feng • A fixed-horizon.
Zhou et al. in this volume) that are similar to those experienced • A stable vehicle dashboard.
during human–human interaction (Lombard & Ditton, 1997).
Disengagement can happen based on a number of factors (and • Provide users with the ability to anticipate impending
should be avoided), including: usability challenges, positive motion via a leading indicator, which can be created with
or negative affect, perceived time lags and/or interruptions multisensory cues, including:
(O’Brien & Toms, 2008), or absence of meaningful informa- • Visually, such as via a trajectory line that shows the
tion. Some principles by which engagement can be realized in direction of movement, which is a simple but effec-
XR include: tive visual leading indicator to support movement
anticipation (Lin, Abi-Rached, & Lahav, 2004).
• Provide users with meaningful information throughout For example, in Along the Trail, a VR application
the system interaction to maintain intrinsic motivation. that features a data-generated landscape, users are

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 798

provided an orange trajectory line that acts as a ‘rail’ 4.1 Cybersickness, Adaptation, and Aftereffects
and is in high contrast against a darker landscape Motion-sickness-like symptoms and other adverse aftereffects
(Panoptic Lab, 2015). (e.g., balance disturbances, visual stress, altered hand–eye coor-
• Physically, such as via physical leading indicators dination) are potential unwanted byproducts of XR exposure
that are devised to link a passive user’s hand (e.g., (Stanney et al., 2020; Stanney & Kennedy, 2008). The sickness
when a passive passenger) to an active control related to XR systems is commonly referred to as ‘cybersick-
handle to receive indication of forthcoming motion ness’ (McCauley & Sharkey, 1992). Some of the most common
(Lackner, 2014). symptoms exhibited include dizziness, drowsiness, headache,
• Aurally, such as via 3D audio soundscapes or 3D nausea, visual fatigue, and general malaise (Kennedy et al.,
audio beacons and directional audio, that inform 1993). The level of adverse effects is more pronounced in VR
users where they are in relation to other locations versus AR/MR.
(Gartenberg, 2018). In VR, more than 80% of users will experience some level of
• Implement layered teleportation design that combines disturbance, with approximately 12% ceasing exposure prema-
several unique techniques: peripheral blur, dynamic field turely due to this adversity (Stanney et al., 2003). Of those who
of view, blink, and rapid movement. drop out, approximately 10% can be expected to have an emetic
response (e.g., vomiting), however, only 1–2% of all VR users
• Peripheral blur can be designed to support locomo- will have such a response. In general, those exposed to VR tend
tion. Coined as “teleblur” by Big Immersive (Milik, to experience more disorientation (D) than neurovegative (N)
2018), such blur can reduce the amount of detail users symptoms, and least of oculomotor-related (O) disturbances,
perceive in their HWD FOV, reducing potential thus having a D > N > O symptom profile (Stanney, Salvendy
cybersickness triggers driven by visual-vestibular et al., 1998). These adverse effects are known to increase in
mismatch (Fernandes & Feiner, 2016). incidence and intensity with prolonged VR exposure duration
• Dynamic field of view is a technique by which the (Kennedy et al., 2000). While most VR users will experience
HWD FOV is modified based on visual parameters, some level of adverse effects, symptoms vary substantially from
such as speed and angular velocity, reducing poten- one individual to another as well as from one system to another
tial cybersickness triggers driven by visual-vestibular (Kennedy & Fowlkes, 1992). These effects can be assessed via
mismatch (Fernandes & Feiner, 2016). the Simulator Sickness Questionnaire (Kennedy et al., 1993),
• Blink teleportation is highly effective at eliminating with values above 20 requiring due caution (e.g., warn and
cybersickness because users see minimal motion. observe users) (Stanney et al., 2005).
With this technique, as users commence movement, In AR/MR, users tend to experience less severe adverse
signs of motion briefly appear, then the visual scene symptoms than those exposed to VR experiences (Vovk, Wild,
fades out, and then users are relocated to a new area Guest, & Kuula, 2018). This is likely because the real-world
as imagery fades back in (Neimark, 2019). While provides ample sources that can be used as rest frames from
k this is a successful cybersickness remediation, users which to judge motion against, thus minimizing the impact k
are completely taken out of the XR for a signifi- of visual-vestibular mismatches between virtual motion of
cant amount of time, thereby potentially reducing augmented entities and real- world inertial cues. AR displays
immersion and leading to disorientation. do, however, still impose vergence-accommodation conflicts,
• Rapid movement between two location points from and likely put a physiological load on the human visual sys-
a first-person perspective in XR has proven to be tem. Thus, in general, those exposed to AR are expected to
successful in maintaining immersion, while signifi- experience more oculomotor-related (O) disturbances than
cantly reducing motion sickness. Studies found that disorientation (D), and least neurovegative (N) symptoms, with
“rapid movement in very short bursts (<300 ms) the expectation being that these systems will present with an
doesn’t produce any greater feelings of motion sick- O > D > N symptom profile, with no dropouts nor emesis
ness than teleportation” (Habgood, Moore, Wilson, (Hughes et al., 2020).
& Alapont, 2018, p. 8). To overcome adverse effects, individuals generally undergo
physiological adaptation during XR exposure. This adapta-
tion is the natural and automatic response to an intersensorily
Used together, these cybersickness remediation techniques imperfect XR and is elicited due to the plasticity of the human
enhance the XR experience and decrease the likelihood of nervous system (Welch, 1978). Due to technological flaws
adverse effects. Batman Arkham VR (Kohler, 2016) is an (e.g., visual-vestibular mismatches; vergence-accommodation
example that utilizes these techniques. When transporting conflicts), users of XR systems may be confronted with one or
between locations, users experience 400 ms of locomotion cou- more intersensory discordances (e.g., a disparity between seen
pled with peripheral blur. Once users arrive at their destination, and felt limb position). In order to perform effectively in the XR,
the FOV “blinks” to 90% black opacity and then slowly fades they must compensate for these discordances by adapting their
back in. This method of combining multiple remediation tech- psychomotor behavior or visual functioning. Once interaction
niques offers users several layers of cybersickness protection, with XR is discontinued, these compensations may persist for
while reducing the potential of losing presence. some time after exposure, leading to adverse aftereffects, which
pose a safety concern.
Once XR exposure ceases and users return to their natural
environment, they are likely unaware that interaction with the
4 HEALTH AND SAFETY ISSUES
XR has potentially changed their ability to effectively interact
The health and safety risks associated with XR exposure compli- with their normal physical environment (Stanney & Kennedy,
cate usage protocols and lead to products liability concerns. It is 1998). Several different kinds of aftereffects may persist for
thus essential to understand these issues when utilizing XR tech- prolonged periods following XR exposure (Szpak, Michalski,
nology. There are both physiological and psychological risks Saredakis, Chen & Loetscher, 2019). For example, hand–eye
associated with XR exposure, the former being related primarily coordination can be degraded via perceptual–motor distur-
to sickness and aftereffects and the latter primarily being con- bances (Lee & Park, 2019), postural sway can arise (Tychsen
cerned with the social impact. & Foeller, 2020), as can changes in the vestibulo-ocular reflex

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 799

(VOR) or one’s ability to stabilize an image on the retina, mental conditioning, personality, worldview); (3) circumstances
and cognitive abilities, such as choice reaction time (Mittel- of exposure (e.g., private experience, family, religion, spiritual);
staedt, Wacker, & Stelling, 2019). The implications of these and (4) effects of exposure on psychological, interpersonal, or
aftereffects are: social function.
• XR exposure duration may need to be minimized.
• Highly susceptible individuals or those from clinical pop-
5 EXTENDED REALITY USABILITY ENGINEERING
ulations (e.g., those prone to seizures) may need to avoid
exposure. Usability engineering involves quantifying experiences of users
• Users may need to be monitored during XR exposure. interacting with systems, with regard to understandability, effi-
• Users’ activities may need to be monitored for a period ciency, error rates, and user satisfaction (Downey & Laskowski,
of time post-exposure to avoid personal injury or harm. 1997; also see Chapter 38 by Lewis and Sauro in this volume).
The International Organization for Standardization (ISO) has
• Design strategies should be implemented that support released ISO 9241-11: 2018(en), which defines usability as the
cybersickness remediation (see Section 3.4). “extent to which a system, product or service can be used by
specified users to achieve specified goals with effectiveness,
4.2 Social Impact efficiency and satisfaction in a specified context of use,” and pro-
eXtended reality technology, like other interactive technologies vides guidance on designing and evaluating human interaction
(e.g., video games, computers), has the potential for negative with systems, including regular, ongoing use, infrequent use,
social implications through misuse, particularly in terms of learning to use, and maintenance of such systems (ISO, 2019).
the possibility for behavioral addictions and hindered social In addition, the standard aims to ensure a system can be used by
dynamics (Kenwright, 2018). Thailand, Vietnam, China, and people with the widest range of capabilities, while minimizing
South Korea have all instituted shutdown laws that ban young risks and undesirable consequences of use errors (ISO, 2019). A
people from playing more traditional online games between related standard, ISO 9241-210:2019, Human-centered Design
certain hours of the day (Király, et al., 2018). The concern is for Interactive Systems, focuses on ensuring systems are usable
that overuse may cause functional and psychological impair- and useful by “focusing on users, their needs and requirements,
ments for gamers, such as impaired interpersonal relationships and by applying human factors/ergonomics, and usability
and decreased work or educational performance (Király, knowledge and techniques” (p. 1).
Nagygyörgy, Griffiths, & Demetrovics, 2014). Further, to date, More recently, user experience (UX) has emerged as an
policies that have attempted to address such social issues evaluation field, with a goal of expanding beyond “traditional”
(e.g., shutdown policies, parental controls, warning messages), task-based evaluations of usability to incorporate a broader
have not proven to be effective. The problem is so pervasive scope of metrics that capture dynamic, context-dependent,
that ‘Internet Gaming Disorder’ (IGD) has been included in and subjective indicators of an individual’s experience when
k Section III of the latest edition of the Diagnostic and Statistical interacting with a system (Scapin, Senach, Trousse, & Pallot, k
Manual of Mental Disorders (DSM-5; American Psychiatric 2012).
Association, 2013). Such multidimensional evaluation approaches are particu-
Currently, it is not clear whether or not XR exposure will larly valuable for XR technology, but progress has been slow.
pose more significant adverse social impacts than its ancestors; For example, according to Dey et al. (2018), less than 10%
early research, however, is not reassuring. Users who engage in of AR papers published annually between 2005 and 2014
what seems like harmless violence in the virtual world experi- included a user study. From those that did include a user study,
ence higher levels of presence and body ownership than more evaluations focused most heavily on user performance (61%)
traditional (television) counterparts (Wilson & McGill, 2018), and questionnaire or survey-based findings (50%). Similarly,
which may desensitize players to violence (Grizzard, Tamborini, Cavalcanti et al. (2018) reported that rehabilitation AR usability
Sherry, & Weber, 2017). Thus, XR overuse may pose even more evaluations lacked heuristic evaluations, and most often focused
severe concerns than current gaming disorders. Grizzard et al. on task performance (51%), followed by user experience (35%),
(2017) found that after playing violent video games for five con- and few got to the level where perception and cognition were
secutive days, habituation occurred in that the ability of the game evaluated, which is where many of the benefits of XR are
to elicit guilt was decreased with repeated exposure, thereby expected to lie (14%). As Dunser and Billinghurst (2011) note,
demonstrating the potential for video game play to lead to emo- one must go beyond traditional usability methods to provide
tional desensitization. There is concern that such desensitiza- meaningful evaluations of XR applications, as they are distinctly
tion to negative stimulation may then subsequently be channeled different from traditional desktop applications. For example,
into real-world activities. Further, what happens if desensitiza- XR systems can support multidimensional object selection
tion is coupled with more immersive and embodied XR games; and manipulation in 3D space, multimodal system output (i.e.,
will violent augmented and virtual reality experiences normal- incorporation of visual, audition, and haptics), and collaboration
ize antisocial behaviors (Franks, 2017)? The ultimate concern is of users sharing the same space. Furthermore, traditional meth-
that XR immersion may potentially be a more powerful percep- ods do not consider assessing presence and adverse aftereffects,
tual experience than past, less immersive gaming technologies, which are of importance in more immersive XR applications
thereby increasing the negative social impact of this technology. (Stanney, Mollaghasemi, Reeves, Breaux, & Graeber, 2003).
Further, if societal lines are crossed in XR, its visceral nature As XR devices reach mass adoption, there is a need to expand
may challenge the lines the law draws between virtual, physical, and improve upon evaluation methods to ensure the resultant
and psychological harm (Lemley & Volokh, 2018). A proac- innovative solutions are highly usable, effective, efficient, and
tive approach is needed, which weighs the risks and potential engaging. Usability evaluation of XR systems must thus go
consequences associated with XR exposure against the bene- beyond the initial focus of task-based assessments to incorpo-
fits. Waiting for the onset of harmful social consequences should rate additional considerations, such as stimulation (i.e., personal
not be tolerated. Koltko-Rivera (2005) suggests that a proac- growth, increased knowledge and skills), identification (i.e.,
tive approach would involve determining (1) types and degree self-expression, interaction with relevant others), and evoca-
of XR content (e.g., aggressive, sexual); (2) types of individu- tion (i.e., self-maintenance, memories) (Hassenzahl, 2003).
als or groups exposed to this content (e.g., their mental aptitude, Bowman, Gabbard, and Hix (2002) and Ramli and Zaman

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 800

Table 3 Usability Evaluation Methods content and real-world entities in real-time, the need to interact
Inspection methods: User testing: on the go, hand gestures, 3D navigation and wayfinding, immer-
does not require users requires users sion, presence, cybersickness, and more. While some provide
a manageable list of generalized guidelines (Sutcliffe & Gault,
Qualitative Heuristic evaluation Think-aloud protocols 2004; Murtza et al., 2017), others provide a comprehensive list
Cognitive walkthrough Observation of issues that are more suitable to application by experienced
Interviews designers (Gabbard, 1997; Stanney et al., 2003). Table 4 pro-
Focus groups vides a consolidated list of heuristics for XR usability based
Questionnaires on previous literature that can readily be applied by evaluators
Quantitative Performance models Formative evaluations with varied experience.
(e.g., Fitts’ Law; Summative evaluations Beyond heuristics, models of user interaction and human
GOMS, KLM) cognitive processing have been developed, which can be used
to conduct a predictive analysis of user interaction for a given
system design. Fitts’ Law describes the relationship between
(2011) provide frameworks for such multifactor usability meth- the time it takes to reach a target based on the travel distance
ods applicable to XR, categorizing methods into qualitative and target size (Fitts, 1954). This relationship can be used to
or quantitative, and whether they require users or not. Within design various interaction spaces within XR, ensuring that the
these frameworks, the general flow of evaluation activities is: speed of movement required is within human capabilities for a
(1) to start with task analysis to understand target tasks and given distance and target interaction space size. For example,
related interactions and information requirements; (2) to follow Fitts’s law was applied to investigate effects of geometry
with expert-directed inspection methods to characterize the displacement and texture rendering on input performance in
specific manner in which XR is being designed to enhance VR; results failed to show evidence that changing depth cues
human performance and the user experience; (3) to follow increased or decreased a user’s input performance (Schwind,
with qualitative user testing to characterize users’ thought pat- Leusmann, & Henze, 2019). This result suggests that the pro-
terns and perceptions of XR interaction; and (4) finally, to use prioceptive system and coordinated control of hand and eye
quantitative—both user-centered formative and comparative movement seem not to require all depth cues of one’s own
summative—evaluations to assess impacts of the XR system in body. Such predictive modeling techniques can be used to
terms of gains in human performance, understanding, and/or inform design of XR applications such that they support human
user engagement (Gabbard & Swan, 2008). A simplified version performance.
of these frameworks is presented in Table 3. This section is Another available engineering modeling approach is the
organized based on this framework and discusses two main Goals, Operators, Methods and Selection Rules (GOMS) model
usability evaluation methods: inspection methods and user of human cognitive processing associated with human-system
testing. tasks (Card, Moran, & Newell, 1983). In this approach, each
k cognitive action is associated with a given processing time, and k
5.1 XR Inspection Methods one can predictively assess the time required to conduct various
interaction methods based on designs. The parameters and time
Inspection methods provide an evaluation of a system against estimates included in the original GOMS model were based on
a set of design guidelines (i.e., heuristics) or performance traditional desktop-based system interaction and may not be
models, which do not require user involvement. In terms of the directly applicable within an XR environment. This, and other
former, when heuristics are not met, a system is deemed less parametric models such as the Keystroke-Level Model, have yet
usable. Often, evaluators set out to complete a set of tasks that to be extended to address XR interactions (Al-Megren, Khabti,
represent user interactions with the system (based on a task & Al-Khalifa, 2018). Further research is required in this area to
analysis as mentioned above) and compare each task step to develop and validate engineering models for human interaction
a list of heuristics (also known as a cognitive walkthrough). within XR spaces and systems. In particular, there is a need
Violations of heuristics are recorded and submitted back to the to extend the scope of such modeling approaches to tangible,
design team for consideration and redesign. It is recommended physical, voice- and gesture-controlled XR interfaces (McIntee,
that 3–5 usability experts complete the heuristic review, as 2016).
little additional information is gained from additional evalua-
tors (Nielsen, 1994). Usability heuristics were first published 5.2 XR User Testing Methods
by Molich and Nielsen (1990), based primarily on desktop
computer environments. These were later updated by Nielsen User testing involves representative users interacting with a
(1994) to generate general guidelines applicable to evaluation system while their experience is measured in some way. Allow-
of most interactive systems designed for human use. These ing users to interact with a system provides designers with (1)
heuristics, while initially developed prior to prolific use of XR, insights into how effective and efficient users can interact with
are applicable to many XR applications due to their general the system, (2) the ability to identify unrecoverable errors that
nature. For example, Wang, Cheng, and Guo (2019) found should be considered for redesign, and (3) the chance to gauge
that 77% of 672 issues identified with a function-oriented VR interest and user satisfaction from the experience. Interviews
application could be mapped to Nielsen’s 10 heuristics. and focus groups provide a guided discussion on system design
Many other heuristic lists specific to XR applications with users that provide qualitative data for designers. Formative
have been created over the past few decades in an attempt to studies evaluate users while interacting with a system, and look
capture usability concerns unique to such immersive environ- to quantify user interactions (e.g., speed, accuracy, errors), and
ments (Altarteer, Charissis, Harrison, & Chan, 2017; Endsley provide hints to when users may fail to achieve desired perfor-
et al., 2017; Gabbard, 1997; Guimaraes & Martins, 2014; mance. Summative studies are used to directly compare two or
Kalalahti, 2015; Ko, Chang, & Ji, 2013; Murtza, Monroe, more systems/experiences using quantitative performance data
& Youmans, 2017; Pinelle, Wong, & Stach, 2008; Stanney, of user interaction. Within formative and summative evalua-
Mollaghasemi, Reeves, Breaux, & Graeber, 2003; Sutcliffe tions, there are many user testing methods available that may
& Gault, 2004; Wetzel, Blum, Broll, & Oppermann, 2011). be utilized based on desired outcomes and resources available
These lists were developed to address aspects of XR interaction (Table 5). In a review of methods used in evaluating mobile AR
capabilities that go beyond usability factors typically related environments, 69% relied on self-report questionnaires, while
to desktop computing, such as tracking and merging of virtual only 3% used performance metrics (28% used a combination

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 801

Table 4 XR Usability Principles

Nielsen’s 10 heuristics Additional VR heuristic considerations Additional AR heuristic considerations
Visibility of system Carefully design scenes with superimposed 3D objects and determine how camera should be
status—always keep user pointed in XR scenes (de Almedia Pacheco, Guimarães, Correa, & Farinazzo Martins, 2018);
informed about what is Provide a frame designating input space (Ko et al., 2013)
going on
Minimize scanning and effort required to access information by maintaining information accessibility
within display screen itself, or just outside its perimeter and accessible via a short head rotation at
a body-referenced location (e.g., as if virtually mounted to a display attached to the wrist; Wickens,
Dempsey, Pringle, Kazansky, & Hutka, 2018)
Minimize frame-of-reference transformations, for example by superimposing a 3D AR grid directly
onto the forward real world view (Wickens et al., 2018)
Match between system and User should always be aware of own location Ensure virtual and real-world entities are
real world—speak users’ and spatial relations between self and virtual combined in an elegant manner, with
language, with words, objects in the surround appropriate 3D registration and seamless
phrases and concepts Use visualizations and metaphors that have real-time interaction when relevant
familiar to the user meaning within physical and task Real world should be basis of
environment, which are presented to support application—follow real-world conventions,
natural engagement and have information appear in natural and
logical order
Size of objects should be appropriate to match
real world; Switching attention between
application and real world should be smooth
Adapt virtual content to environmental
conditions (lighting/sound)
Keep information sources that need to be compared (e.g., system map view and forward real world
view) in close proximity (Wickens et al., 2018)
Ensure persistence of positioning superimposed elements onto the real world regardless of camera
motion speed (de Almedia Pacheco et al., 2018)
Provide multimodal cues, such as spatialized sound and visual elements, when task-relevant
information is provided (Ko et al., 2013)
User control and Should be possible for users to choose between interaction methods—co-usage should be clear
k freedom—need clearly Allow users to customize settings k
marked “exit” to leave Effect of user actions on virtual objects should be immediately visible and match expected behavior
unwanted state without
having to go through Means of exiting virtual space should be explicitly communicated
extended dialogue Often controls are outside of the field-of-view in XR displays, and thus techniques to locate and
orient to controls are necessary (de Almedia Pacheco et al., 2018)
Consistency and Keep interaction simple and provide consistent Manipulation should be natural—match
standards—follow platform responses to user actions characteristics of object and mental model
conventions of users
Display elements in HWDs should be located, as much as possible, in the same relative locations
with the same layout as in the real world (Wickens et al., 2018)
Error prevention Design seamlessly and for disconnection Ensure system is useful and usable from a
Ensure users can undo incorrect actions variety of viewing angles, distances and
movements
Recognition rather than Virtual objects should not occlude information
recall—instructions for that impacts usage of application
using visible or easily
retrievable cues whenever
appropriate
Flexibility and efficiency of Allow users to skip non-playable and frequently repeated content; Minimize distraction and overload
use—experienced users
provided shortcuts while
novices are fully supported
(continued overleaf)

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 802

Table 4 (continued)
Nielsen’s 10 heuristics Additional VR heuristic considerations Additional AR heuristic considerations
Aesthetic and minimalist Allow form to communicate function Too many virtual objects should not be visible
design at one time to maximize legibility (Wickens
et al., 2018)
Provide intuitive and customizable input
mappings
Clearly separate virtual content from
background in different usage situations
(brightness/contrast)
Help users recognize, Design to avoid mistakes and prevent undesired actions
diagnose, and recover from Ensure system indicates problems precisely and makes suggestions in constructive manner
errors
Help and System should be intuitive and require minimal instructions and help
documentation—provide Position tracking: Tracking must be fast and reliable to maintain position in real scene and minimize
instructions, training and adverse effects; Consider accessibility of off screen objects
help
Presence: Virtual representation of self (if present) should allow users to act and explore in a natural
way; Users’ perception of being in XR space should be natural
Physical comfort of use/ergonomics: Device should not be too heavy, difficult to handle, cause
physical load/discomfort, or limit normal physical actions; Device should not cause sickness or
adverse aftereffects
System setup: Keep environment setup (e.g., sensors/cameras, fiducial markers) as simple as
possible
System safety: Consider physical space constraints and appropriate design awareness cues for
users; Ensure users are aware of own location and spatial relations between self and virtual objects
in the surround

Table 5 User Testing Methods and Metrics for XR Evaluation

k Category Method Metrics k
Explicit Focus groups/ interviews Guided discussion of system components/aspects and
associated usability and user experience parameters
Formative and summative Self-report questionnaires:
evaluations – System Usability Scale (SUS)
– Post-study System Usability Questionnaire (PSSUQ)
– Usability Metric for User Experience (UMUX)
– Presence Questionnaire (PQ)
– Workload Questionnaire (e.g., NASA TLX)
– Sickness Questionnaire (e.g., SSQ)
– Questionnaire for User Interaction Satisfaction (QUIS)
– Smart Glasses User Satisfaction (SGUS) Questionnaire
Overt performance (e.g., reaction time, accuracy/ errors)
tracking and/or observation
Think-aloud protocol—have users verbalize what they are
doing and their thought process throughout interaction
Implicit Formative and summative Eye tracking—visual search/attention
evaluations that use Facial expressions via camera
neurophysiological measures Linguistic analyses (e.g., voice intonation/ cadence/
or ‘unobservable’ behavior amplitude, language used)
Cardiovascular response (e.g., electrocardiography,
respiration measures)
Musculoskeletal response (e.g., electromyography,
accelerometers)
Neural activity (e.g., electroencephalography)
Gastrointestinal response (e.g., electrogastrography)
Stress response (e.g., electrodermal response)

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 803

of metrics) (Lim, Selamat, Alias, Krejcar, & Fujita, 2019). recent years, gaining traction in enterprise training and support
Obtaining user feedback early and often throughout design, applications for maintenance, warehouse operations, as well as
development, and deployment of XR systems can enhance sys- expanded entertainment offerings. These applications provide
tem effectiveness, while reducing cost and development time, adaptable, deployable, and safety-conscious solutions at a mod-
which taken together can realize end results that better meet est cost, create game-based and learning virtual experiences that
user expectations and needs. For example, Langlois, That, and would otherwise be difficult to explore, and offer rehabilitation
Mermillod (2016) used user testing to systematically compare and medical applications that reach far beyond conventional
a real and augmented heads-up display on the basis of depth computer-based options. The viral outbreak of COVID-19 in
perception features. The user testing was able to benchmark early 2020 and the associated lockdown drove adoption of XR
AR designs to determine which ones ensured good congruence (Mathur, 2020). XR technology began to be used to close the
to real objects and thus would be expected to enhance SA and physical gap caused by social distancing, while connecting
human performance. people in a variety of application domains (Papagiannis, 2020;
Using self-report questionnaires during XR user testing Petrock, 2020; Redohl, 2020). XR has even been seen as key
provides a quick and easy method to gather user feedback. to maintaining military readiness during “Black Swan” events
Questionnaires such as the System Usability Scale (SUS; such as COVID-19 (Blades, 2020). This section outlines com-
Brooke, 1996, 2013), the Post-System Usability Questionnaire mon and emerging applications of XR technology that will
(PSSUQ; Lewis, 1992), and the Usability Metric for User Expe- predictably surge over the coming years.
rience (UMUX; Finstad, 2010) are quick to administer and can
be useful for uncovering problems, however, they cannot isolate 6.1 Training
what the problems actually are Other self-report questionnaires
can provide insight into XR-relevant constructs, such as pres- XR technologies are being explored and/or deployed in a
ence, how challenging a system interface/interaction may be variety of training domains. Given the value of full-scale
and whether any adverse effects were experienced. Commonly simulation-based training (e.g., full motion flight simulators,
used questionnaires include the Presence Questionnaire (PQ; high fidelity medical manikins; see Chapter 16 by Bisbey et al.
Witmer & Singer, 1998), the NASA Task Load Index (Hart & in this volume) that has been realized over the past few decades,
Staveland, 1998) and the Simulator Sickness Questionnaire the lower cost and mobility of AR and VR hardware platforms
(SSQ; Kennedy, Lane, Berbaum, & Lilienthal, 1993). These have allowed training developers to increase the return on
have been used extensively to assess XR experiences and training investment by providing a similar or better training
experience for a lower overall cost (thedanse.com, 2015).
can provide useful data for assessing the impact of a system
One area where XR technologies have gained a lot of traction
on user experience. Further, many satisfaction-based ques-
in recent years is in maintenance training (see Figure 9). In an
tionnaires have been developed—often tailored for a specific
era where many professionals and amateurs alike are employ-
situation—that consider likability, pleasure, comfort, trust, flex-
ing internet videos to learn or refresh their memory on how to
ibility in use, and safety of the user experience (Bevan, 2008).
k diagnose and repair maintenance issues, AR provides an oppor- k
Measures of satisfaction, such as the Questionnaire for User tunity for maintainers to have videos, pictures, virtual models
Interaction Satisfaction (QUIS; Chin, Diehl, & Norman, 1988) (or holograms) of equipment, and other types of media in their
and the Smart Glasses User Satisfaction (SGUS) Questionnaire field-of-view while remaining hands-free to perform their work.
(Xue, Sharma, & Wild, 2019), have been the most often used Palmarini et al. (2018) found that AR is being used for aviation,
metrics for evaluating mobile AR solutions (Lim et al., 2019). automotive, train, plant, and general mechanical maintenance.
During formative (Chu, Chen, Hwang, & Chen, 2019) and Similarly, AR and VR technologies are being used in the con-
summative (da Silva, Teixeira, Cavalcante, & Teichrieb, 2019) struction industry for safety training and beyond, given that they
evaluations, a number of explicit and implicit metrics may be provide an immersive experience without some of the challenges
used to quantify an XR user experience. Explicit measures that can accompany on-the-job training, such as schedule con-
include task performance indicators, such as speed and accu- flicts, potential hazards, weather situations, and need for safety
racy. Users can be asked to talk aloud while interacting with an and liability (Li et al., 2017).
XR experience such that insights may be gained regarding their The medical domain is another area that is investing in using
approach to interaction, frustrations, or points of confusion, as AR and VR for training (see Figure 9). Some examples of ways
well as points of enjoyment, excitement, or enlightenment. This that VR is used include building virtual representations of organs
provides qualitative data to support quantitative data captured and tissues to be viewed in depth in VR, learning to communi-
during an evaluation. In recent years, implicit measures have cate more effectively with patients, providing training on how to
gained use in the usability community. Such measures pro- diagnose and treat diseases, and surgical simulations (Hsieh &
vide insights into users’ cognitive processing and interaction Lee, 2018). For example, surgical teams that are not co-located
techniques using unobtrusive measures of visual attention, can train together by having the expert and trainee each wearing
facial expressions, voice analysis, cardiovascular responses, a HWD, such as the Microsoft HoloLens, and networked into
musculoskeletal responses, and neural activity. These are par- the same scenario such that they are seeing the exact same visual
ticularly relevant to XR environments, which are designed to representations. As the trainee simulates the surgical operation
create immersive experiences where users feel and respond required by the scenario, the expert physician can provide feed-
physiologically as a result. Having quantifiable data of such back and guidance. Seymour et al. (2002) found that training
physiological responses can provide comparator data from residents in VR resulted in gallbladder dissections performed
which to benchmark XR experiences against their real-world 29% faster than residents who did not receive training in VR.
counterparts. There were also significantly less errors made by the VR-trained
group.

6 APPLICATIONS 6.2 Education

XR environments have been adopted by an ever-growing num- Similar to training, the immersive nature, ability to provide
ber of domains. VR, while initially used primarily in training visualizations in 3D space while keeping users safe, small form
and entertainment, has since expanded into diverse areas, such factors, and reduced cost are making XR technologies more
as system design and assembly. AR has seen rapid growth in and more popular as educational tools. Studies have shown

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 804

(a) (b)
Figure 9 XR training application examples: (a) maintenance; (b) medical.

that content learned via XR technologies can improve stu- without having to travel to the actual places (Nikoletta et al.,
dents’ problem-solving skills, long-term memory, motivation, 2008). Similarly, an Honor Flight was filmed and implemented
collaborative abilities (Alakärppä et al., 2017; Hung et al., in VR allowing veterans in a hospice and not able to make
2017; Tobar-Muñoz et al., 2017), and learning performance the trip to experience it as though they had been able to travel
(Pellas et al., 2017; Wei et al., 2015; Zhang et al., 2014). XR (Walters, 2019).
game-based learning experiences have been used to teach
different Science, Technology, Engineering, and Mathematics 6.3 Operations
(STEM) topics, such as natural, physical, and social sciences
k (Pellas et al., 2017). For example, a mobile AR application has While XR environments have been developed over the past k
been used to simulate molecule reactions for middle school half century, they have gained popularity as operational solu-
student education (Ewais & De Troyer, 2019). tions in the last decade. In particular, XR enterprise based
Teaching anatomy, physiology, and pathology of the human solutions are now being developed in key areas such as design,
body is another great example (see Figure 10). There are a assembly, maintenance and repair, warehouse operations
variety of AR and VR applications that render 3D stereoscopic and remote expert access (Fink, 2019). As early as 1996,
visual effects to allow a user to visualize the different compo- companies reported on their research into XR solutions for
nents and systems within the human body, often being able to assembly tasks (c.f. Mizell, 1997). The automobile industry
add and take away layers of different systems to see how they is using VR to design full-scale car models for design evalu-
interact with each other, allowing the user to zoom in on parts ation and test, allowing engineers, designers, and researchers
or rotate them, and allowing users to walk into virtual models from around the world to evaluate a design prior to prototype
to visualize what is happening inside (Hsieh & Lee, 2018). development (Schreiber, 2019). This application takes advan-
tage of two key VR value drivers—reducing design time and
Outside of the classroom, VR experiences are being devel-
increasing workforce collaboration (Deloitte, 2018). XR tech-
oped to replicate real places, such as museums and monuments,
nologies are also being utilized in the design of spaces, such as
to allow students to see and learn about history, art, etc.

(a) (b)
Figure 10 XR education application examples: (a) anatomy, physiology, and pathology; (b) museum. Source: Adobe Stock/©ake1150
©romaset. Reproduced with permission of Adobe Stock.

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 805

(a) (b)
Figure 11 XR Operational application examples: (a) architecture blueprints; (b) trucking operational support. Source: Adobe
Stock/©zapp2photo. Reproduced with permission of Adobe Stock.

architecture, interior design, and plant design. For example, Live View was made broadly available in beta form in 2019,
architects are able to use AR to create a 3D model of a building allowing users to see augmented directional cues overlaid on a
from blueprints (see Figure 11), share it with a large team mobile phone’s camera view of the real world (Warren, 2019).
of architects, plumbers, electricians, real estate investors, and The travel and tourism industry will continue to expand its use
construction workers so that all can quickly and easily test how of AR apps, which allows travelers to navigate cities, identify
equipment will fit and communicate design decisions (urdesign, and learn more information about landmarks and city sites,
2017). Further, interior designers are able to test how elements, have up-to-date schedules and times of various transportation
such as furniture, paint color, etc. will look and fit into a room, options and events, and translate foreign language signs for
as well as share this vision with clients and architects using AR. comprehension (Shah, 2019).
While VR solutions are providing benefits, it is expected that
AR solutions will surpass VR utility within operational settings.
By 2017, AR solutions were being tested in manufacturer 6.4 Entertainment
assembly facilities and demonstrating performance improve-
ments of 25–34% on first use (Abraham & Annunziata, 2017; The entertainment industry has leveraged unique characteris-
Boeing, 2018; Capgemini Research Institute, 2018). Enterprise tics of XR experiences, providing dynamic and exciting expo-
k solutions for large-scale maintenance (e.g., aircraft, trucking sures in a multitude of forms. XR entertainment applications k
industry; see Figure 11) benefit from the AR advantages of have found their way into games, sports, movies, art, online com-
hands-free operation while accessing support documentation munities, location-based entertainment, theme parks, and other
via digital, context- and spatially-relevant content within an venues (Badiqué et al., 2002; Burt, 2019; Nakatsu et al., 2005).
HWD as opposed to paper manuals. This capability offers XR technologies provide a more immersive medium for enter-
a more efficient and effective opportunity for maintaining tainment, as compared to more traditional entertainment media
equipment. In a similar light, inspections at construction sites (e.g., film, play), through use of artificial virtual characters (i.e.,
are being supported by AR by having to-scale virtual models avatars), engaging narrative, and dynamic interactive control to
of building components superimposed on real buildings for a create engaging experiences.
quick and easy visual indication of any differences (Li et al., VR gaming and entertainment accounted for US$4.15 billion
2017). Further, many systems are incorporating reach back in 2018 spending and is expected to grow to over US$70 billion
capabilities to experts who can access a maintainer’s view of by 2026 (Fortune Business Insights, 2019). Gaming accounts for
a system and provide real-time support as needed to guide about 50% of the VR software market, with adventure, action,
diagnostic and repair actions (Howard, 2019; Soldatos, 2019). and simulation games being most popular (Petrov, 2019).
Warehouse operations are benefiting from AR capabilities. For Actively engaging with online communities allows VR gamers
example, picklist processes showed a 46% gain in efficiency to immerse into interactive worlds, seen from a first-person
when using AR compared to standard process (Abraham & perspective. Often, game controllers are incorporated to support
Annuziata, 2017). Utilizing HWD AR solutions allow workers typical game-based interactions (pointing commands) and
to locate items and track movement without having to use motion detectors are used to sense gamer’s movements and
handheld tracking systems, freeing hands to focus on picking replicate within the XR game. Interaction gloves are entering
and placing items. Mobile phone AR apps are also available, the entertainment market, which support more detailed hand
providing context-relevant content and reach back support for movement control and potential for haptic feedback beyond
improved maintenance performance. “buzzing” indicators of contact (Ochanji, 2020; Zhu et al.,
Surgeons are using AR to plan surgeries to increase the 2020). With these enhanced, multimodal experiences, embod-
efficiency and effectiveness of surgeries and reduce the chance iment with XR characters is often more than that experienced
of errors (Hsieh & Lee, 2018). Pratt et al. (2018) used AR to with traditional PC/laptop games (Wilson & McGill, 2018;
overlay computed tomography angiography (CTA) images over see Figure 12).
a patient’s body to demonstrate subsurface vascular anatomy XR entertainment centers have gained popularity, where
before any incisions were made, aiding navigation and accurate immersive experiences are provided as a service. DisneyQuest,
dissection. Further, because AR headsets can be controlled which opened in 1998, was an early version of such a center,
through gestures and voice commands rather than touch, sterile where users could experience VR (among other immersive
conditions in operating rooms can be maintained. experiences), such as riding a simulated roller coaster and
For the general public, AR navigational solutions continue navigating down rapids in a raft. Since then, more immersive,
to expand and provide enhanced wayfinding and information “location-based” VR experiences have been created where users
discovery as one navigates their environment. Google Maps enter and engage in an interactive world to achieve a mission

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 806

(a) (b)
Figure 12 XR entertainment application examples: (a) fully embodied VR; (b) Smartphone AR gaming apps. Source: (a) Shutterstock
/©Kit8. Reproduced with permission of Shutterstock; (b) Adobe Stock/©Freer. Reproduced with permission of Adobe Stock.

k k

(a) (b)
Figure 13 XR Sales and marketing applications: (a) virtual test drive; (b) in-store purchase support. Source: Adobe Stock/©ake1150
©stockphoto-graf. Reproduced with permission of Adobe Stock.

(Rubin, 2019). Similarly, VR game arcades allow users to expe- now available in various applications, such as Instagram, Boo!,
rience a broad spectrum of systems and games without having SNOW, Camera360, and Face Swap Live (Moreau, 2019), to
to purchase XR equipment (Castillo, 2018). In addition, popular personalize characters and share photos/videos with friends.
roller coaster experiences have incorporated VR technology, With the plethora of self-altering filters, there are concerns
such that riders can experience traveling through a virtual world regarding self-image and self-esteem, particularly in teenage
while on the coaster (Michaels, 2016). girls (Ehmke, 2020). Rauschnabel (2018) developed a theoreti-
With the release of Pokemon Go! in 2016, AR games that cal model of gratification expectations from AR enhancements
overlay augmented content on to the real world surged in to the real world. Further research to better understand the
popularity (see Figure 12). This application provided a massive societal impact of such enhancements, which are becoming the
multiplayer online experience using smartphones that pro- “new normal” in self-imaging, is needed to optimize this avail-
vided competition, collaboration, and familiar characters in an able technology to support human growth and entertainment
interactive experience, tracking user location relative to virtual while minimizing negative impacts related to mental health
elements placed around the globe. Since that time, a number concerns (Kenwright, 2018).
of AR games have been released and popularized in various
user communities (Long, 2019). With this enhanced, connected
entertainment opportunity comes risks, primarily focused on 6.5 Sales and Marketing
security and safety (Banister & Hertel, 2018). The industry Companies are also beginning to discover the value of transition-
must carefully consider and account for potential issues in these ing from common video-based marketing experiences to fully
areas to continue to see sustained growth and utility. immersive XR experiences. In fact, 75% of Forbes World’s Most
Simplistic, widely available AR experiences have emerged Valuable Brands have developed some form of XR experiences
with smartphone technology, particularly via facial filters or for customers or employees (Korolov, 2015). Brands such as
lenses. In 2015, Snapchat was one of the first applications allow- Lowe’s, IKEA, BMW, Audi, and Porsche have integrated XR
ing users to add AR content to images/videos that were captured experiences into their marketing strategy. Examples of VR use
on their phone. This capability utilized facial recognition tech- for marketing include a VR tour of an airline business cabin, a
nology to pinpoint users within a scene and offer various AR 360-degree panoramic view of a fashion runway show during
characteristic add-ons. Filters have become widespread and are London Fashion Week, a VR experience to test drive cars, and

k
Trim Size: 8in x 10in k
Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 807

advances have made design of environments rendered in XR,

and development of the content that populates those environ-
ments, more sophisticated, further increasing the immersive
quality of these systems.
From a design perspective, XR poses unique challenges.
The user population drives design considerations, making a
“one-size-fits-all” approach impractical, requiring consideration
of a range of factors that include: identifying the best manner
in which to convey specific types of information; determining
the best approaches for allowing users to control the flow of
the XR experience; and, specifying the level of interactive
fidelity necessary for an effective immersive experience that
minimizes cybersickness. These user design considerations are
best considered in terms of a tradespace driven by technology
considerations. Critical boundaries on solutions deriving from
this tradespace analysis, which should be considered when
developing XR applications, include ensuring the health and
Figure 14 Immersive realtor applications. Source: Adobe safety of all users as well as the social consequences of making
Stock/©ArchiVIZ ©sharplaninac. Reproduced with permission these systems available. Designed correctly, user testing can
of Adobe Stock. provide deep insight into these and other critical boundaries.
The applications for XR continue to evolve as the underly-
supporting in-store furniture purchases by examining different ing technologies and designs advance. As recently as a decade
material swatches (mbryonic, 2019; see Figure 13). ago, the cost of the infrastructure required to design, develop,
Realtors are also using VR to allow potential buyers to expe- and maintain these systems was a critical barrier to entry for
rience a “virtual walk-through” of a house or building in first many users. Today, these core technologies can be had relatively
person that can be viewed either on a mobile device such as inexpensively. As a result, XR is now more accessible for educa-
a smartphone or tablet, or in a VR headset. They accomplish tion and training, collaborating and partnering, networking and
this by taking a series of pictures with a 3600 camera that are entertaining and more uses than ever before. Advances in net-
then integrated to provide an immersive experience. Realtors are working infrastructure, increased access to the internet through
using AR to promote home and business sales by incorporating wireless technologies and the advent of cloud computing and
virtual models of furniture into real rooms, allowing potential storage continue to democratize XR, ensuring that new uses and
buyers to visualize how furniture can fit into their space without applications will continue to emerge far into the future.
the added cost and time required for staging (urdesign, 2017; see
k Figure 14). k
In general, AR techniques are becoming popular with in pre- ACKNOWLEDGMENTS
purchase sales situations (e.g., fashion “try-ons,” International We would like to thank Riz Sharzaman and C.J. Boger for their
Data Corp, 2017; retail showcasing such as decorating spaces, assistance with the graphics. This material is based upon work
Dacko, 2016) and eCommerce in general (e.g., previewing prod- supported in part by the Office of Naval Research (ONR) under
uct modifications, exploring features), as they have been demon- Grants No. N68335-18-C-0535 and N68335-20-F-0155, Con-
strated to vastly increase sales conversion rates (Saleem, 2019). gressionally Directed Medical Research Programs (CDMRP)
These benefits of AR sales and marketing are thought to derive Joint Program Committee-1/Medical Simulation and Informa-
from a heightened sense of ownership when engaged with dig-
tion Sciences (JPC-1/MSIS) Award No. MTEC-17-08-Multi-
ital products, immersive storytelling, wonder, and social status
Topic-0304, Program Executive Office Simulation, Training
associated with crowd-sourced product reviews.
and Instrumentation (PEO STRI) Phase I SBIR Contract
No. W900KK-16-C-0040, Phase I Option SBIR Contract
7 CONCLUSION No. W900KK-18-C-0012, and Phase II SBIR Contract No.
W900KK-18-C-0029, and Army Research Laboratory (ARL)
eXtended Reality represents the next stage in the evolution of Simulation and Training Technology Center (STTC) Contract
a capability rooted in an early twentieth-century requirement No. W911NF-18-C-0031. The views, opinions and/or findings
for safely training individuals to master a challenging skill contained in this research/presentation/publication are those of
set—manned flight (Chihyung, 2015). Yet, in one sense, the the author(s)/ company and do not necessarily reflect the views
concept of simulating reality dates back thousands of years, to of the Department of Defense and should not be construed as
the ancient Egyptians, who used physical models to simulate an official DoD/ ONR, CDMRP JPC-1/MSIS, PEO STRI, and
war planning, and the ancient Chinese, who built elaborate ARL STTC position, policy or decision unless so designated by
“automata” to train soldiers in combat (Amico, 2009). While it other documentation. No official endorsement should be made.
would be a stretch to say that these earliest efforts directly trace Reference herein to any specific commercial product, process,
to today’s capabilities, it is accurate to suggest that the primary or service by trade name, trademark, manufacturer, or other-
challenge faced by XR—developing solutions that effectively wise, does not necessarily constitute or imply its endorsement,
mimic reality—is similar to that encountered by those early recommendation, or favoring by the U.S. Government.
efforts. What have advanced over time are the approaches
available to creating these solutions. This chapter has provided
an overview of these advances, from the technology, design,
and application perspectives. REFERENCES
From a technical perspective, advances in the underlying
hardware, ranging from headsets, to tracking, to interaction, Abraham, M. & Annuziata, M. (2017). Augmented reality is already
have made XR solutions less expensive, more powerful—hence improving worker performance. Harvard Business Review.
of higher immersive quality—and more accessible to a broader https://hbr.org/2017/03/augmented-reality-is-already-improving-
range of users. Paralleling these hardware advances, software worker-performance

NCOI Annotations Form For Teacher II Applicant
100% (10)
NCOI Annotations Form For Teacher II Applicant
6 pages
Virtual Herbal Garden Review Paper 2
No ratings yet
Virtual Herbal Garden Review Paper 2
15 pages
Ethical Considerations in Civic Engagement
80% (5)
Ethical Considerations in Civic Engagement
2 pages
Research Paper On Extended Reality
No ratings yet
Research Paper On Extended Reality
10 pages
ARVR Student Handbook Level2
No ratings yet
ARVR Student Handbook Level2
39 pages
Extendedrealitybasicintroduction 240303054546 93d03d84
No ratings yet
Extendedrealitybasicintroduction 240303054546 93d03d84
21 pages
The Challenges of Entering The Metaverse An Experi
No ratings yet
The Challenges of Entering The Metaverse An Experi
23 pages
Information 14 00252 With Cover
No ratings yet
Information 14 00252 With Cover
18 pages
Phishing Attack Prevention Using AI
No ratings yet
Phishing Attack Prevention Using AI
19 pages
A Review of Interaction Techniques For Immersive Environments
No ratings yet
A Review of Interaction Techniques For Immersive Environments
19 pages
Applsci 13 01292
No ratings yet
Applsci 13 01292
18 pages
Extended Reality Seminar Presentation
89% (9)
Extended Reality Seminar Presentation
15 pages
AR and HMI
No ratings yet
AR and HMI
14 pages
A Deep Reinforcement Learning-Based Offloading Scheme For Multi-Access Edge Computing-Supported Extended Reality Systems
No ratings yet
A Deep Reinforcement Learning-Based Offloading Scheme For Multi-Access Edge Computing-Supported Extended Reality Systems
11 pages
Towards 6G Metaverse
No ratings yet
Towards 6G Metaverse
9 pages
Center For Immersive Experiences
No ratings yet
Center For Immersive Experiences
38 pages
Extended Reality
No ratings yet
Extended Reality
6 pages
Varjo Industrial Design Whitepaper 2021
No ratings yet
Varjo Industrial Design Whitepaper 2021
29 pages
ARVR Student Handbook Level1 (2) Compressed Organized Organized
No ratings yet
ARVR Student Handbook Level1 (2) Compressed Organized Organized
12 pages
Extended Reality XR Report
No ratings yet
Extended Reality XR Report
3 pages
XR Today MarketGuide 2022
No ratings yet
XR Today MarketGuide 2022
48 pages
SLR
No ratings yet
SLR
3 pages
Digital Health - VR AR Metaverse
No ratings yet
Digital Health - VR AR Metaverse
25 pages
XR Study Content
No ratings yet
XR Study Content
17 pages
Extended Reality
No ratings yet
Extended Reality
2 pages
Accessibility Needs of Extended Reality Hardware: A Mixed Academic-Industry Reflection
No ratings yet
Accessibility Needs of Extended Reality Hardware: A Mixed Academic-Industry Reflection
5 pages
Brochure MIT xPRO ARVR 04-08-23 - V26
No ratings yet
Brochure MIT xPRO ARVR 04-08-23 - V26
15 pages
Brochure MIT xPRO ARVR 30-08-22 V21
No ratings yet
Brochure MIT xPRO ARVR 30-08-22 V21
15 pages
Webxr Asset Management in Developing Virtual Reality Learning Media
No ratings yet
Webxr Asset Management in Developing Virtual Reality Learning Media
9 pages
Extended Reality (XR)
No ratings yet
Extended Reality (XR)
19 pages
Extended Reality
100% (4)
Extended Reality
19 pages
MR Research Paper
No ratings yet
MR Research Paper
48 pages
Research Paper On Extended Reality
No ratings yet
Research Paper On Extended Reality
15 pages
Extended Reality
No ratings yet
Extended Reality
21 pages
Xcu9dp6vu13n7py Compressed
No ratings yet
Xcu9dp6vu13n7py Compressed
10 pages
XR Doc Main
No ratings yet
XR Doc Main
21 pages
Gfmam The Maintenance Framework First Edition English Version
100% (1)
Gfmam The Maintenance Framework First Edition English Version
24 pages
Introduction To Extended Reality
No ratings yet
Introduction To Extended Reality
9 pages
Extended Reality
No ratings yet
Extended Reality
13 pages
Bhavya Ar Vs VR
No ratings yet
Bhavya Ar Vs VR
14 pages
History of Extended Reality
No ratings yet
History of Extended Reality
5 pages
Digital Technology Radar 2025 Immersive Reality VR AR XR
No ratings yet
Digital Technology Radar 2025 Immersive Reality VR AR XR
3 pages
Extended Reality in The Pharmaceutical Industry
No ratings yet
Extended Reality in The Pharmaceutical Industry
10 pages
Extended Reality
No ratings yet
Extended Reality
10 pages
Exploring The Evolving Landscape of Extended Reality XR Technology
No ratings yet
Exploring The Evolving Landscape of Extended Reality XR Technology
6 pages
Extended Reality Exploring The Immersive Future
No ratings yet
Extended Reality Exploring The Immersive Future
10 pages
Meta
No ratings yet
Meta
4 pages
Darood
No ratings yet
Darood
22 pages
Chapter 5 - Immersive Technologies
No ratings yet
Chapter 5 - Immersive Technologies
51 pages
Report
No ratings yet
Report
15 pages
Extened Reality (XR)
No ratings yet
Extened Reality (XR)
8 pages
Chapter 5
No ratings yet
Chapter 5
25 pages
Report
No ratings yet
Report
11 pages
Extended Reality
No ratings yet
Extended Reality
26 pages
Extended Reality Seminar Report
No ratings yet
Extended Reality Seminar Report
27 pages
Chapter 5 AR
No ratings yet
Chapter 5 AR
25 pages
Unit 1
No ratings yet
Unit 1
115 pages
Extended Reality
No ratings yet
Extended Reality
4 pages
Jayaram Ts
No ratings yet
Jayaram Ts
18 pages
The Jhānas
No ratings yet
The Jhānas
4 pages
Extended Reality Seminar Report PDF
No ratings yet
Extended Reality Seminar Report PDF
25 pages
U2000 Northbound Performance File Interface Developer Guide (NE-Based)
No ratings yet
U2000 Northbound Performance File Interface Developer Guide (NE-Based)
79 pages
Extened Reality
No ratings yet
Extened Reality
21 pages
METERIAL REQUISITION FIL 2015-2016
No ratings yet
METERIAL REQUISITION FIL 2015-2016
329 pages
Comment: Materials and Devices For Immersive Virtual Reality
No ratings yet
Comment: Materials and Devices For Immersive Virtual Reality
3 pages
Establishing OPC UA Connectivity With Rockwell Automation® Integrated Architecture
No ratings yet
Establishing OPC UA Connectivity With Rockwell Automation® Integrated Architecture
3 pages
Trainng report-BSNL: Monday, July 2, 2007
No ratings yet
Trainng report-BSNL: Monday, July 2, 2007
40 pages
1.1 How-To-Use-This-Competency-Based-Learning-Material
No ratings yet
1.1 How-To-Use-This-Competency-Based-Learning-Material
2 pages
Course Module: Course Module Code Topic Coverage Reference/s Duration Learning Outcomes
No ratings yet
Course Module: Course Module Code Topic Coverage Reference/s Duration Learning Outcomes
3 pages
1 s2.0 S0263224113006519 Main
No ratings yet
1 s2.0 S0263224113006519 Main
11 pages
CPAP-HFNC - Medin - NC3 Ops - Manual Book
No ratings yet
CPAP-HFNC - Medin - NC3 Ops - Manual Book
59 pages
Scherfi Gsvej 8, DK-2100 Copenhagen Ø, Denmark Tel.: +45 39 17 17 17. Fax: +45 39 17 18 18. E-Mail: Postmaster@euro - Who.int Web Site: WWW - Euro.who - Int
No ratings yet
Scherfi Gsvej 8, DK-2100 Copenhagen Ø, Denmark Tel.: +45 39 17 17 17. Fax: +45 39 17 18 18. E-Mail: Postmaster@euro - Who.int Web Site: WWW - Euro.who - Int
205 pages
Weekly Topical Test 1 Trigonometry
No ratings yet
Weekly Topical Test 1 Trigonometry
3 pages
Evidence Claim Assessment Worksheet
No ratings yet
Evidence Claim Assessment Worksheet
3 pages
Sharif Abushaikha Hotel General Manager
No ratings yet
Sharif Abushaikha Hotel General Manager
2 pages
TIẾNG ANH CHUYÊN NGÀNH 2
No ratings yet
TIẾNG ANH CHUYÊN NGÀNH 2
12 pages
Agam
No ratings yet
Agam
12 pages
Facebook Marketing For Tourism
No ratings yet
Facebook Marketing For Tourism
20 pages
Fundamental Rights
No ratings yet
Fundamental Rights
23 pages
SPM Time Card Management
No ratings yet
SPM Time Card Management
12 pages
Central Govt - Part II
No ratings yet
Central Govt - Part II
30 pages
Local Govt
No ratings yet
Local Govt
22 pages
DPSP
No ratings yet
DPSP
25 pages
Dept. of Chemistry, Rajabazar Science College, 92-Acharya Prafulla Chandra Road, University of Calcutta, Kolkata - 700009, West Bengal, India
No ratings yet
Dept. of Chemistry, Rajabazar Science College, 92-Acharya Prafulla Chandra Road, University of Calcutta, Kolkata - 700009, West Bengal, India
6 pages
Constitutional
No ratings yet
Constitutional
15 pages
For Ex Project
No ratings yet
For Ex Project
64 pages
(英文) HART规范：SPEC 127 通用命令00 19 PDF
No ratings yet
(英文) HART规范：SPEC 127 通用命令00 19 PDF
29 pages
Peace and Conflict Studies
No ratings yet
Peace and Conflict Studies
18 pages
Explosion Proof Pressure Transmitter: Model PT124B-282 Intelligent Type
No ratings yet
Explosion Proof Pressure Transmitter: Model PT124B-282 Intelligent Type
2 pages
Crystal Structures I
No ratings yet
Crystal Structures I
28 pages
#6 Adding File Upload To A Form
No ratings yet
#6 Adding File Upload To A Form
10 pages
Samuel Maldonado Setu - CV & Ijazah
No ratings yet
Samuel Maldonado Setu - CV & Ijazah
4 pages
TSR Notes
No ratings yet
TSR Notes
6 pages

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.

Extended Reality (XR) Environments - August 2021

Uploaded by

Extended Reality (XR) Environments - August 2021

Uploaded by

Trim Size: 8in x 10in k

Salvendy636083 c30.tex V1 - 09/07/2021 12:43 A.M. Page 782

1 INTRODUCTION 782 5 EXTENDED REALITY USABILITY ENGINEERING 799

1 INTRODUCTION “Link Trainer” (Jeon, 2005) or Morton Heilig’s “Sensorama”

REAL AUGMENTED MIXED VIRTUAL

Figure 1 Virtuality continuum.

Table 1 Technology Component Comparison Across XR Platforms

Figure 4 Fiducial marker being used to anchor a virtual plant to

to recognize a particular image and calculate its location, scale,

(a) (b) (c)

Figure 6 Boundary box.

Figure 7 Virtuality continuum in context.

(a) (b) (c)

3.2.2 Conversation Design • using physics collisions to indicate interactions between

Table 4 XR Usability Principles

Table 5 User Testing Methods and Metrics for XR Evaluation

6 APPLICATIONS 6.2 Education

advances have made design of environments rendered in XR,

You might also like

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.