Newton Lee

Editor

Encyclopedia of
Computer
Graphics and
Games
Encyclopedia of Computer Graphics and
Games
Newton Lee
Editor

Encyclopedia of
Computer Graphics and
Games

With 906 Figures and 116 Tables

Editor
Newton Lee
Institute for Education, Research, and Scholarships
Los Angeles, CA, USA
Vincennes University
Vincennes, IN, USA

ISBN 978-3-031-23159-9 ISBN 978-3-031-23161-2 (eBook)

https://doi.org/10.1007/978-3-031-23161-2
© Springer Nature Switzerland AG 2024
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or
part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way,
and transmission or information storage and retrieval, electronic adaptation, computer software, or
by similar or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are exempt
from the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
herein or for any errors or omissions that may have been made. The publisher remains neutral with
regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG.
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Paper in this product is recyclable.

To all my students, past, present, and future!
Preface

The Encyclopedia of Computer Graphics and Games (ECGG) is an authori-

tative reference work covering the history, technologies, and trends of com-
puter graphics and games catered to industry professionals and academic
communities worldwide. The breadth and depth of topics covered by the
encyclopedia benefit a wide diversity of readers including researchers, practi-
tioners, teachers, and students who seek general as well as specific knowledge
in computer graphics and games.

Los Angeles, USA Newton Lee

November 2023

vii
Acknowledgements

The Encyclopedia of Computer Graphics and Games (ECGG) is a labor of

love by more than 365 contributing authors, peer reviewers, academic and
industry co-chairs, editorial board members, and volunteers. I would like to
extend my heartfelt thanks to everyone including Springer Nature editors and
project coordinators for making this encyclopedia a reality.

ix
 List of Topics

3D Visualization Animation(Facial)
3D Modelling Through Photogrammetry in Face Beautification in Antiage
Cultural Heritage
3D Printing, History of
3D Visualization Interface for Temporal Analysis Applying Artificial Intelligence to Virtual
of Social Media Reality and Intelligent Virtual Environments
3D-Rendered Images and Their Application in the
3D Room Layout System Using IEC (Interactive
Interior Design
Evaluational Computation)
Deep Learning Algorithms for 3D Reconstruction
Emotion-Based 3D CG Character Behaviors
Tactile Visualization and 3D Printing for
Genetic Algorithm (GA)-Based NPC Making
Education
Technologies for the Design Review Process
Art and Design
Animation Artistic Data Visualization in the Making
Color: Pixels, Bits, and Bytes
Abstraction and Stylized Design in 3D Animated
Computer Graphics, Video Games, and
Films: Extrapolation of 2D Animation Design
Gamification Impacting (Re)Habilitation,
Animation and Neurocinematics: Visible
Healthcare, and Inclusive Well-Being
Language of E-motion-S and Its Magical
Imagineering Ceramic Pottery Using Computer
Science
Graphics
Challenges Facing the Arab Animation Cinema
Character Animation Scripting Environment
Artificial Intelligence
Exploring Innovative Technology: 2D Image
Based Animation with the iPad Automated Game Design Testing Using Machine
Motion Matching: Data-Driven Character Learning
Animation Systems Character Artificial Intelligence
Pipeline of 2D Vector Animation in Television Classical Learning Method in Digital Games
Series Computer Games and Artificial Intelligence
Preserving the Collective Memory and Re- Computer Go
creating Identity Through Animation Constructing Game Agents Through Simulated
Teaching Computer Graphics by Application Evolution
Vector Graphics Game Player Modeling

xi
xii List of Topics

Machine Learning for Computer Games Integrating Virtual Reality and Augmented
Meta Artificial Intelligence and Artificial Reality into Advertising Campaigns: History,
Intelligence Director Technology, and Future Trends
Monte-Carlo Tree Search Interacting with a Fully Simulated Self-Balancing
Navigation Artificial Intelligence Bipedal Character in Augmented and Virtual
Overview of Artificial Intelligence Reality
Quality Assurance-Artificial Intelligence Interaction with Mobile Augmented Reality
RTS AI Problems and Techniques Environments
Skull and Roses Card Game Interactive Augmented Reality to Support
StarCraft Bots and Competitions Education
World Representation in Artificial Intelligence Key Early Verticals: Challenges and Limitations
in Implementation of Augmented Reality
Life-Size Telepresence and Technologies
Audio Live Texture Mapping in Handheld Augmented
Reality Coloring Book
Adaptive Music Mixed Reality
Audio and Facial Recognition CAPTCHAs for Potential of Augmented Reality for Intelligent
Visually Impaired Users Transportation Systems
Audiogame Virtual Reality and Robotics
Dynamic Music Generation: Audio Analysis-
Synthesis Methods Augmented Reality and Virtual Reality
Emotional Congruence in Video Game Audio
Overview of Virtual Ambisonic Systems Artificial Reality Continuum
Procedural Audio in Video Games Brain Signals as a New Biometric Authentication
Spatial Audio and Sound Design in the Context of Method Using Brain-Computer Interface
Games and Multimedia Collaborative Environments for Augmented and
Virtual Reality Applications
Color Detection Using Brain Computer Interface
Construction Management Processes in a Digital
Augmented Reality
Built Environment, Modelling
3D Selection Techniques for Distant Object Digital Images Using Heuristic AdaBoost Haar
Interaction in Augmented Reality Cascade Classifier Model, Detection of
Augmented and Gamified Lives Partially Occluded Faces
Augmented Learning Experience for School Engaging Dogs with Computer Screens: Animal-
Education Computer Interaction
Augmented Reality Entertainment: Taking Fingerprint Verification Based on Combining
Gaming Out of the Box Minutiae Extraction and Statistical Features
Augmented Reality for Human-Robot Interaction Gaming Control Using BCI
in Industry Immersive Technologies for Accessible User
Augmented Reality for Maintenance Experiences
Augmented Reality in Image-Guided Surgery Immersive Visualizations Using Augmented
Augmented Reality Ludo Board Game with Reality and Virtual Reality
Q-Learning on Handheld Interactive Augmented Reality Pop-Up Book
Conceptual Model of Mobile Augmented Reality with Natural Gesture Interaction for Handheld
for Cultural Heritage Object Manipulation Using Real Hand Gesture
Enhanced Visualization by Augmented Reality for Augmented Reality Interior Design
Gamification and Social Robots in Education Position-Aware 3D Facial Expression Mapping
History of Augmented Reality Using Ray Casting and Blendshape
List of Topics xiii

Smart Calibration Between RGB-D and Thermal Education

Cameras for ROI Detection and Tracking in
Cross-cultural Game Studies
Physiological Monitoring
PBL-Based Industry-Academia Game
Tabletop Storytelling
Development Education
Tracking Techniques in Augmented Reality for
Handheld Interfaces
Engine Architecture
Virtual Human for Assisted Healthcare:
Application and Technology Game Engine
Virtual Reality Proton Beam Therapy Unit: Case Game Loop and Typical Implementation
Study on the Development Game Physics Engine, Overview
Volumetric Filmmaking Interactive Computer Graphics and Model-View-
Controller Architecture
Cinematics Panda3D
Physical, Virtual, and Game World Persistence
Movie-Making of Spatiotemporal Dynamics in
Unity, a 2D and 3D Game Engine
Complex Systems
Unreal Engine, a 3D Game Engine
Postproduction in Game Cinematics
Virtual World, a Definition Incorporating
Distributed Computing and Instances
Computer Vision
American Sign Language Detection Game and other Media
Experiential Media: Using Machine Vision and
Animal Crossing: A Causal Game
Sensor-Input to Create Dynamic Real-Time
Anti-phishing Attacks in Gamification
Generated Media
Assassin’s Creed, an Analysis
Fall Risk Detection in Computer Vision
Bayonetta 2, an Analysis
Healthcare Robots with Islamic Practices
Call of Duty Franchise, an Analysis
Locomotion and Human Tracking in Healthcare
Children Privacy Protection
Robots
Children’s Games, from Turtle to Squirtle
Computer Games and the Evolution of Digital
Data Visualization
Rights
2-Simplex Prism as a Cognitive Graphics Tool for Computer Games in Education
Decision-Making Counter-Strike Global Offensive, an Analysis
Cognitive Processing of Information Dark Souls III, an Analysis
Visualization Dark Souls RPG Through the Lens of Challenge
Indigenous Knowledge for Mental Health, Data Destiny and Destiny 2, an Analysis of an FPS
Visualization Digital Games for Animals
Indigenous Language Revitalization with Stories Diversity in Gaming and the Metaverse
and Games Dōjin Game
Mixed Reality and Immersive Data Visualization Facial Recognition and Emotion Detection in
Multivariate Visualization Using Scatterplots Environmental Installation and Social Media
Scalable Techniques to Visualize Spatiotemporal Applications
Data Fantasy XVI Online, a Massively Multiplayer
Stress Reduction, Relaxation, and Meditative Online Role-Playing Game
States Using Psychophysiological Fire Emblem Fates, Conquest
Measurements Based on Biofeedback Systems First-Person Shooter Games, a Brief History
via HRV and EEG Five Nights at Freddy’s, a Point and Click Horror
Tensor Field Visualization Game
xiv List of Topics

Fortnite: A Brief History Game Design and Development

Games and the Magic Circle
3D Game Asset Generation of Historical
Gamification of Modern Society: Digital Media’s
Architecture Through Photogrammetry
Influence on Current Social Practices
3D Puzzle Games in Extended Reality
Gardenscapes and Homescapes, Casual Mobile
Environments
Games
Academic and Video Game Industry “Divide”
God of War (2018), an Action-Adventure Game
Among Us and Its Popularity During COVID-19
God of War, an Analysis
Pandemic
Hypermedia Narrative as a Tool for Serious
Analog Prototyping for Digital Game Design
Games
Animal Crossing: New Horizons and
Itch.io, History of
Its Popularity During COVID-19
King of Fighters, a Brief History
Pandemic
Kingdom Hearts (2002): An Analysis
Challenge-Based Learning in a Serious Global
Mario Kart, an Analysis of Its Absence from
Game
Esports
Comic Arts in Games, Asset Production, and
NBA 2K, a Brief History
Rendering
NFT Games
Dark Souls Through the Lens of Essential
On Computer Games About Cooking
Experience
Origin of Games
Dead Space Through the Lens of Resonance
Overwatch: Team-Based Multiplayer First-Person
Design Framework for Learning to Support
Shooter Game
Industry 4.0
Pervasive Games
Design of Alienation in Video Games
Pokémon and World Championships
Domain-Specific Choices Affecting Design Effort
Professional Call of Duty Player Matthew
in Gamification
“Nadeshot” Haag: An e-Sports Case Study
Educational Game Abzû and the Lens of Fun
Public Health Education via Computer Games
Learning
Rocket League: An Analysis
Emotion in Games
Smart Toys
Game Design and Emotions: Analysis Models
Social-, Mobile-, and Multi-Player-Games and
Game Development Leadership Tips
Their Impact on Today’s Online Entertainment
Game Prosumption
Industry
Game Thinking X Game Design Thinking
Sociality of Digital Games
Game Writer’s Dilemma: Context vs. Story
Sonic Adventure 2, an Analysis
Game-Based Interventions in Public Health:
Speedrunning
Exploiting the Engaging Factor of Gameplay
Speedrunning in Video Games
Games and Active Aging
Star Wars Battlefront (2004), an Analysis
Games in Science
STEM Learning Through Video Games
Gamification
Super Mario Galaxy: An Overview
Gamification and Serious Games
Super Smash Bros. Ultimate and E-sports
Gamification Ethics
Super Smash Bros.: A Brief History
Gamification in Crowdsourcing Applications
The Elder Scrolls V Skyrim
Hades: An Analysis
Toy Computing
Hearthstone: A Collectable Card Game Through
Video Game Culture in Cape Town, South Africa
the Lens of Problem Solving
World of Tanks, MMO Strategy Freemium Game
Incremental Games
World of Warcraft, a MMORPG with Expansions
Indie Game
List of Topics xv

Legend of Zelda Breath of the Wild and the Lens Underground Design of Kaizo Games
of Curiosity Video Game Storytelling Fundamentals: Setting,
Madden NFL and Infinite Inspiration Power Status, Tone, and Escalation
MEEGA+, Systematic Model to Evaluate Video Game Trolls and Dopamine Withdrawal
Educational Games Videogame Engagement: Psychological
Motion Planning in Computer Games Frameworks
Narrative Design Visual Accessibility in Computer Games
Narrative in Video Games Visual Novel
New Super Mario Bros. Wii, an Analysis
Nursing Education Through Virtual Reality: Holography
Bridging the Gap
Holography as an Architectural Decoration
Online Gaming Industry Evolution,
Holography, History of
Monetization, and Prospects
Image Quality Evaluation of a Computer-
Online Players: Engagement, Immersion, and
Generated Phase Hologram
Absorption Across Secondary Worlds
Parasocial Phenomena in Video Games
Interaction
Persona 3 and the Lens of Surprise
Player Experience, Design and Research Biosensing in Interactive Art: A User-Centered
Player Personas and Game Choice Taxonomy
Political Game Design Computer Games for People with Disability
Post-Digital Graphics in Computer Games Player-Avatar Link: Interdisciplinary
Principle Structure to Create a 2D Game Level Embodiment Perspectives
Editor Video Games and Accessibility: A Case Study of
Protection Korona: A Game Design on Covid-19 The Last of Us II
Psychological Game Design
Query-by-Gaming Miscellaneous
Redesigning Games for New Interfaces and
Contemporary Computer Shogi
Platforms
Loot Boxes: Gambling-Like Mechanics in Video
Rehabilitation Games
Games
Resident Evil 2, History of
Minkowski-Lorentz Spaces Applications:
ROP-Skill System: Model in Serious Games for
Resolution of Apollonius and Dupin Problems
Universities
Mobile Persuasive Applications
Secure Gaming: Cheat-Resistant Protocols and
Player Abusive Behavior Detection
Game History Validation
Puyo Puyo
Semiotics of Computer Games
Shadow Shooter: All-Around Game with e-Yumi
Serious Online Games for Engaged Learning
3D
Through Flow
Theory of Minkowski-Lorentz Spaces
Spatio-temporal Narrative Framework for
Underwater Enhanced Detail and Dehaze
Architecture in Video Games
Technique (UEDD) for Underwater Image
Strategies for Design and Development of Serious
Enhancement
Games: Indian Perspective
Symbolic Planning in Computer Games
Modeling and Texturing
The Sims Franchise, a Retrospective of Racial
Representation and Skin Tones B-Splines
Timed Automata for Video Games and Interaction Constrained Edges and Delaunay Triangulation
Transformational Games Delaunay Triangulation
xvi List of Topics

Modeling and Mesh Processing for Games Crowd Evacuation Using Simulation Techniques
Pencils of Spheres in the Minkowski-Lorentz Crowd Simulation
Spaces Fluid Simulation
Planetary Generation in Games Lattice Boltzmann Method for Diffusion-
Poisson-Disk Sampling: Theory and Applications Reaction Problems
Shape Deformation Models Lattice Boltzmann Method for Fluid Simulation
Sketch-Based Posing for 3D Animation Lattice Gas Cellular Automata for Fluid
Spheres, AABB, and OOBB as Bounding Volume Simulation
Hierarchies Position Based Dynamics
The New Age of Procedural Texturing Simulation and Comparison of AODV and DSDV
UV Map Generation on Triangular Mesh Protocols in MANETs
Simulation of Emotional Crowd and Applications
Networked Games
Platform
Area of Interest Management in Massively
Multiplayer Online Games Game Venues and Platforms
Client/Server Gaming Architectures Ludii General Game System for Modeling,
Cloud for Gaming Analyzing, and Designing Board Games
Cognitive Psychology Applied to User
Experience in Video Games Rendering
Detecting and Preventing Online Game Bots in
High-Performance Many-Light Rendering
MMORPGs
Ray Tracing in Video Games
Disney Toontown Online, a Massively
Rendering Equation
Multiplayer Online Role-Playing Game
Distributed Simulation and Games
User Interface
Game Bot Detection on Massive Multiplayer
Online Role-Playing Games (MMORPGs) Automated Image Captioning for the Visually
Systems Impaired
Griefing in MMORPGs Data Gloves for Hand and Finger Motion
IPv6 Common Security Vulnerabilities and Tools: Interactions
Overview of IPv6 with Respect to Online Decoupling Game Tool GUIs from Core Editing
Games Operations
Mobile Cloud Gaming Game Interface: Influence of Diegese Theory on
Online Gaming Architectures the User Experience
Online Gaming Scalability Human Interaction in Machine Learning (ML) for
Peer-to-Peer Gaming Healthcare
Toxic Behaviors in Online Gaming Plug-in-Based Asset Compiler Architecture
Tangible Surface-Based Interactions
Open-Source Code Unified Modeling Language (UML) for Sight
Loss
Open Source 3D Printing, History of User Interface (UI) in Semiautonomous Vehicles

Physics and Simulation Virtual Reality

Bounding Volume Hierarchies for Rigid Bodies 3D Avatars in Virtual Reality Experience
Cellular Automata Methods Accessibility of Virtual Reality for Persons with
Collision Detection Disabilities
Computational Steering for Computational Fluid Collaborative Engineering and Virtual
Dynamics Prototyping Within Virtual Reality
List of Topics xvii

Cybersickness Redirected Walking in Virtual Reality

Deep Reinforcement Learning in Virtual Social Virtual Reality
Environments Sonic Interactions in Virtual Environments
Educational Virtual Reality Game Design for Film Sound Spatialization
and Animation Spatial Perception in Virtual Environments
EEG as an Input for Virtual Reality Storytelling in Virtual Reality
Everyday Virtual Reality Substitutional Reality
Eye Tracking in Virtual Reality Training Spatial Skills with Virtual Reality and
Foundations of Interaction in the Virtual Reality Augmented Reality
Medium Trustworthy Embodied Virtual Agents
History of Virtual Reality Uncanny Valley in Virtual Reality
Immersive Auralization Using Headphones User Acoustics with Head-Related Transfer
Immersive Technologies for Medical Education Functions
Immersive Virtual Reality Serious Games User-Centered Design and Evaluation
Information Presentation Methods in Virtual Methodology for Virtual Environments
Reality Virtual Hand Metaphor in Virtual Reality
Interactive Virtual Reality Navigation Using Cave Virtual Pointing Metaphor in Virtual Reality
Automatic Virtual Environment Technology Virtual Reality Applications in Education
Locomotion in Virtual Reality Video Games Virtual Reality as New Media
Making Virtual Reality (VR) Accessible for Virtual Reality Exercise and Rehabilitation
People with Disabilities Virtual Reality Game Engines
Mindfulness, Virtual Reality, and Video Games Virtual Reality Retailing
Mixed Reality, Gamified Presence, and Virtual Reality Stereo Post-Conversion After
Storytelling for Virtual Museums Effects Workflow
Multi-user Virtual Environments for Education Virtual Reality System Fidelity
Natural Walking in Virtual Reality Virtual Reality Systems, Tools, and Frameworks
Origin of Virtual Reality Virtual Reality Therapy
Perceptual Illusions and Distortions in Virtual Virtual Reality: A Model for Understanding
Reality Immersive Computing
Presence and Immersion in Virtual Reality Virtual Reality-Based Daily Scrum Meetings
Raycasting in Virtual Reality
About the Editor

Newton Lee is the founding president of the

501(c)(3) nonprofit Institute for Education,
Research, and Scholarships based in Los Angeles,
California, a former Disney and Bell Labs engi-
neer, and a 2021 graduate of the FBI Citizens
Academy.
Serving as an FBI Ambassador, Prof. Lee
expounds on social media, campus safety, student
mental health, cybersecurity, and counterterrorism
as portrayed in the highly acclaimed Total Infor-
mation Awareness book series published by
Springer Nature. The Total Information Awareness
trilogy has garnered rave reviews from Newsweek,
The Daily Beast, ACM Computing Reviews,
AdWeek, and Choice Magazine, among others.
Veteran Staff Sergeant Andrew Price of the United
States Air Force (USAF) remarked, “I am inspired
by the prospect of world peace. I’d fully recom-
mend following the author’s steps, reaching
beyond our borders, making friends outside our
norm, and helping to foster world peace and a
better tomorrow.”
Prof. Lee has co-developed over 100 online
games at The Walt Disney Company, and
12 bestselling and award-winning interactive titles
including The Lion King Animated Storybook and
Winnie the Pooh and the Honey Tree that were
featured in the Billboard Magazine. He has also
executive produced original songs that have
played on American Idol and charted on US Bill-
board, UK Music Week, and US iTunes HOT
100 (Electronic).
He earned a B.S. and M.S. in Computer Science
from Virginia Tech as well as an A.S. in Electrical
Engineering and an honorary doctorate in Com-
puter Science from Vincennes University. He has
xix
xx About the Editor

been honored with a Michigan Leading Edge

Technologies Award, two community develop-
ment awards from the California Junior Chamber
of Commerce, and four volunteer project leader-
ship awards from The Walt Disney Company.
Editorial Board Members

Academic Co-Chairs
Shlomo Dubnov
Department of Music and Computer Science and
Engineering
University of California San Diego
San Diego, CA, USA

Patrick C. K. Hung
University of Ontario Institute of Technology
Oshawa, ON, Canada

Jaci Lee Lederman

Vincennes University
Vincennes, IN, USA

xxi
xxii Editorial Board Members

Industry Co-Chairs
Shuichi Kurabayashi
Cygames, Inc. & Keio University
Kanagawa, Japan

Xiaomao Wu
Gritworld GmbH
Frankfurt am Main
Hessen, Germany

Editorial Board Members

Leigh Achterbosch
School of Science, Engineering, IT and Physical
Sciences
Federation University Australia Mt Helen
Ballarat, VIC, Australia

Ramazan S. Aygun
Department of Computer Science
Kennesaw State University
Marietta, GA, USA
Editorial Board Members xxiii

Barbaros Bostan
BUG Game Lab
Bahçeşehir University (BAU)
Istanbul, Turkey

Anthony L. Brooks
Aalborg University
Aalborg, Denmark

Guven Catak
BUG Game Lab
Bahçeşehir University (BAU)
Istanbul, Turkey

Alvin Kok Chuen Chan

Cambridge Corporate University
Lucerne, Switzerland
xxiv Editorial Board Members

Anirban Chowdhury
Department of User Experience and Interaction
Design, School of Design (SoD)
University of Petroleum and Energy Studies
(UPES)
Dehradun, Uttarakhand, India

Saverio Debernardis
Dipartimento di Meccanica
Matematica e Management
Politecnico di Bari,
Bari, Italy

Abdennour El Rhalibi
Liverpool John Moores University
Liverpool, UK

Stefano Ferretti
Department of Computer Science and
Engineering
University of Bologna
Bologna, Italy

Han Hu
School of Information and Electronics
Beijing Institute of Technology
Beijing, China
Editorial Board Members xxv

Ms. Susan Johnston

Select Services Films Inc.
Los Angeles, CA, USA

Chris Joslin
Carleton University
Ottawa, Canada

Sicilia Ferreira Judice

Department of Computer Science
University of Calgary
Calgary, Canada

Hoshang Kolivand
Department of Computer Science
Faculty of Engineering and Technology
Liverpool John Moores University
Liverpool, UK
xxvi Editorial Board Members

Dario Maggiorini
Department of Computer Science
University of Milan
Milan, Italy

Tim McGraw
Purdue University
West Lafayette, IN, USA

George Papagiannakis
ORamaVR S.A.
Heraklion, Greece

FORTH-ICS
Heraklion Greece University of Crete
Heraklion, Greece

Florian Richoux
Nantes Atlantic Computer Science
Laboratory (LINA)
Université de Nantes
Nantes, France
Editorial Board Members xxvii

Andrea Sanna
Dipartimento di Automatica e Informatica
Politecnico di Torino
Turin, Italy

Yann Savoye
Institut fur Informatik
Innsbruck University
Innsbruck, Austria

Sercan Şengün
Wonsook Kim School of Art
Illinois State University
Normal, IL, USA

Ruck Thawonmas
Ritsumeikan University
Shiga, Japan
xxviii Editorial Board Members

Vinesh Thiruchelvam
Asia Pacific University of Technology &
Innovation
Kuala Lumpur, Malaysia

Rojin Vishkaie
Amazon
Seattle, WA, USA

Duncan A. H. Williams
Digital Creativity Labs
Department of Computer Science
University of York
York, UK

Sai-Keung Wong
National Chiao Tung University
Hsinchu, Taiwan
Editorial Board Members xxix

Editorial Board Intern

Sam Romershausen
Vincennes University
Vincennes, IN, USA
Contributors

Nur Ameerah Abdul Halim Mixed and Virtual Reality Research Lab,
Vicubelab, Johor Bahru, Malaysia
School of Computing, Faculty of Engineering, Universiti Teknologi Malaysia,
Johor, Malaysia
Aref Abedjooy Faculty of Science, Ontario Tech University, Oshawa, ON,
Canada
Juliana Aida Abu Bakar Institude of Creative Humanities, Multimedia &
Innovation, School of Creative Industry Management & Performing Arts,
Universiti Utara Malaysia, Sintok, Kedah, Malaysia
Faris Abuhashish Animation & Multimedia, University of Petra, Amman,
Jordan
Arab Open University, Amman, Jordan
Leigh Achterbosch Faculty of Science and Technology, Federation Univer-
sity Australia, Mt Helen, Ballarat, VIC, Australia
Ali Adjorlu Aalborg University, Copenhagen, Denmark
Ahmad Hakim Ahmad Rahman Universiti Teknologi Malaysia, Skudai,
Malaysia
Zahra Ahmadi Ontario Tech University, Oshawa, Canada
Ahmed Sabah Ahmed Business of Information Technology(BIT), College
of Business Informatics, University of Information Technology and Commu-
nications, Baghdad, Iraq
Ecehan Akan Digital Game Design, Bahçeşehir University Faculty of
Communication, Istanbul, Turkey
Ryuya Akase Graduate School of Information Science and Electrical
Engineering, Kyushu University, Fukuoka, Japan
Kemal Akay Unity Technologies, Copenhagen, Denmark
Mohamad Yahya Fekri Aladin Mixed and Virtual Reality Research Lab,
Vicubelab, Universiti Teknologi Malaysia, Johor Bahru, Malaysia
School of Computing, Faculty of Engineering, Universiti Teknologi Malaysia,
Johor Bahru, Malaysia
xxxi
xxxii Contributors

Mac Alexander Vincennes University, Vincennes, IN, USA

Aisha Al-Hamar Department of Computer Science, Loughborough Univer-
sity, Loughborough, UK
Yousef Al-Hamar Department of Computer Science, Faculty of Engineering
and Technology, Liverpool John Moores University, Liverpool, UK
Itimad Raheem Ali Information System Management(ISM), College of
Business Informatics, University of Information Technology and Communi-
cations, Baghdad, Iraq
Dhiya Al-Jumeily Liverpool John Moores University, Liverpool, UK
Mohammed Hazim Alkawaz Faculty of Information Sciences and
Engineering, Management and Science University, Shah Alam, Malaysia
Mansour Alqarni Ontario Tech University, Oshawa, ON, Canada
Tariq Alrimawi Graphic Design Department, University of Petra, Amman,
Jordan
Chan Kok Chuen Alvin Cambridge Corporate University, Lucerne,
Switzerland
Luís Aly FEUP, University of Porto, Porto, Portugal
Inês Amaral Instituto Superior Miguel Torga, Coimbra, Portugal
University of Minho, Braga, Portugal
Christos-Nikolaos Anagnostopoulos Intelligent Multimedia and Virtual
Environments Lab, Department of Cultural Technology and Communication,
University of the Aegean, Mytilene, Greece
Jaime Arias Université Paris 13, Sorbonne Paris Cité, LIPN, CNRS UMR
7030, Villetaneuse, France
Metin Arıca Nowhere Studios, Istanbul, Turkey
Sylvester Arnab Disruptive Media Learning Lab, Coventry University,
Coventry, West Midlands, UK
Shiva Asadianfam Department of Computer Engineering, Qom Branch,
Islamic Azad University, Qom, Iran
Sima Asadianfam Department of Computer Engineering, Zanjan Branch,
Islamic Azad University, Zanjan, Iran
Ramazan S. Aygun Computer Science Department, University of Alabama
in Huntsville, Huntsville, AL, USA
Abdullah Bade Mathematics, Graphics and Visualization Research Group
(MGRAVS), Faculty of Science and Natural Resources, Universiti Malaysia
Sabah, Kota Kinabalu, Sabah, Malaysia
Jean-François Baffier Graduate School of Information Science and
Technology, The University of Tokyo, Tokyo, Japan
Contributors xxxiii

Steve Bakos Ontario Tech University, Oshawa, ON, Canada

Tamilesh Balasuntharam Ontario Tech University, Oshawa, ON, Canada
Shakti Banerjee Immersive Media Design, MIT Institute of Design, MIT
Art, Design & Technology University, Pune, Maharashtra, India
Mason Bates Creative Technologies Program, Illinois State University,
Normal, IL, USA
Stefania Bau Init Esports, Lewes, DE, USA
Jean-Paul Bécar LAMAV-CGAO, FR CNRS 2956 EA 4015, Valenciennes,
France
Nicola Bellotto School of Computer Science, University of Lincoln, Lincoln,
UK
Geraldine Anna Berger Department of Business Information Systems,
Friedrich-Schiller University, Jena, Germany
Ross Berger Vistance Consulting, Los Angeles, CA, USA
Mehmet Ilker Berkman Communication Design, Bahçeşehir University
Faculty of Communication, Istanbul, Turkey
Gilberto Bernardes INESC TEC and University of Porto, Faculty of
Engineering, Porto, Portugal
Stéphanie Bertrand Foundation for Research and Technology Hellas,
Heraklion, Greece
Vineetha Bettaiah Computer Science Department, University of Alabama in
Huntsville, Huntsville, AL, USA
Mark Billinghurst School of Information Technology and Mathematical
Sciences, University of South Australia, Mawson Lakes, SA, Australia
Muhammad Anwar Bin Ahmad Universiti Teknologi Malaysia, Skudai,
Malaysia
João Ricardo Bittencourt Universidade do Vale do Rio dos Sinos
(UNISINOS), Porto Alegre, Brazil
Jeremy Blackburn Department of Computer Science, University of
Alabama at Birmingham, Birmingham, AL, USA
A. Seugnet Blignaut Technology Enhanced Learning for Innovative Teach-
ing and Training South Africa (TELIT-SA), North-West University,
Vanderbijlpark, South Africa
Nicholas Bode Ontario Tech University, Oshawa, ON, Canada
Dominik Borer ETH Zurich, Zürich, Switzerland
Adriano Ferreti Borgatto Department of Informatics and Statistics (INE),
Federal University of Santa Catarina (UFSC), Florianópolis, SC, Brazil
Daniel N. Boulos University of Hawai’i Manoa, Honolulu, HI, USA
xxxiv Contributors

Evren Bozgeyikli School of Information, University of Arizona, Tucson,

AZ, USA

Lal “Lila” Bozgeyikli School of Information, University of Arizona,

Tucson, AZ, USA

Anthony L. Brooks Aalborg University, Aalborg, Denmark

Douglas Brown Games Academy, Falmouth University, Cornwall, UK

Cameron Browne Maastricht University, Maastricht, The Netherlands

Jon Ram Bruun-Pedersen Department of Architecture, Design & Media

Technology, Aalborg University Copenhagen, Copenhagen, Denmark

Petyo Budakov New Bulgarian University, Sofia, Bulgaria

Natasha Bulatovic Trygg Department of Art History, University of Turku,

Turku, Finland

Alyssa Bump Creative Technologies Program, Illinois State University,

Normal, IL, USA

Dawson Bundren Vincennes University, Vincennes, IN, USA

Haley Burch Vincennes University, Vincennes, IN, USA

John Burnett Department of Music, Sonic Arts Research and Development

Group, Qualcomm Institute, University of California, San Diego, La Jolla,
CA, USA
Bernard Butler Emerging Networks Laboratory, Telecommunications Soft-
ware & Systems Group, Waterford Institute of Technology, Waterford, Ireland

Josh Bycer Game-Wisdom, Cherry Hill, NJ, USA

Marco Antonio Martínez Cano Ontario Tech University, Oshawa, ON,

Canada

Emanuele Carlini ISTI-CNR, Pisa, Italy

Inma Carpe The Animation Workshop, VIA University College, Viborg,

Denmark
Polytechnic University of Valencia, Valencia, Spain

Michal Čertický Agent Technology Center at Czech Technical University in

Prague, Prague, Czech Republic

Alan Chalmers WMG, University of Warwick, Coventry, UK

Kompalli Jwala Seethal Chandra Microsoft Corporation, Redmond, WA,

USA

Arindam Chaudhuri Samsung R & D Institute Delhi, Noida, India

Min Chen School of Computer Science and Technology, Huazhong Univer-

sity of Science and Technology, Wuhan, China
Contributors xxxv

Yingjie Chen Department of Computer Graphics Technology, Purdue

University, West Lafayette, IN, USA
Yinsheng Chen Liverpool John Moores University, Liverpool, UK
Vien Cheung Leeds Institute of Textiles and Colour, University of Leeds,
Leeds, UK
Sanmitra Chitte World University of Design, Dean, School of Management,
Sonepat, Haryana, India
Arsh Chowdhry Ontario Tech University, Oshawa, ON, Canada
David Churchill Computing Science Department, University of Alberta,
Edmonton, AB, Canada
Murat Çınar Department of Computer Education and Instructional Technol-
ogy, Hacettepe University, Ankara, Turkey
David M. Clark Vincennes University, Vincennes, IN, USA
Matthew Clark Vincennes University, Vincennes, IN, USA
Diogo Cocharro INESC TEC and University of Porto, Faculty of Engineer-
ing, Porto, Portugal
Brandon Coker Vincennes University, Vincennes, IN, USA
Bryce Coleman Vincennes University, Vincennes, IN, USA
Shane Conley Vincennes University, Vincennes, IN, USA
Marco Corbetta Cloud Imperium Games, London, UK
Brody Corenflos Vincennes University, Vincennes, IN, USA
Serhan Cosar School of Computer Science, University of Lincoln, Lincoln,
UK
Peter I. Cowling Digital Creativity Labs, Department of Computer Science,
University of York, York, UK
Claire M. Culver Ontario Tech University, Oshawa, ON, Canada
Sabrina Culyba Schell Games, Pittsburgh, PA, USA
Fabrizio Cutolo Department of Information Engineering, University of Pisa,
Pisa, Italy
Gabriele D’Angelo Department of Computer Science and Engineering,
University of Bologna, Bologna, Italy
Igor Dall’Avanzi Computing, Goldsmiths College, University of London,
London, UK
Girish Dalvi IDC School of Design, Indian Institute of Technology, Bombay,
India
xxxvi Contributors

Sarah Dashti Cardiff School of Technologies, Cardiff Metropolitan Univer-

sity, Cardiff, UK
Jeffery Jonathan Joshua (‫ )ישוע‬Davis The Embassy of Peace, Whitianga,
New Zealand
Manoj Dawarwadikar SP Jain School of Global Management, Sydney,
Bangalore, India
Francesco De Pace Dipartimento di Automatica e Informatica, Politecnico di
Torino, Turin, Italy
Sébastien Deguy Allegorithmic, Venice, CA, USA
Fatemeh Dehghani Faculty of Science, Ontario Tech University, Oshawa,
ON, Canada
Myriam Desainte-Catherine Univ. Bordeaux, Bordeaux INP, CNRS,
LaBRI, UMR5800, Talence, France
Surojit Dey Design Programme, Indian Institute of Technology, Kanpur,
Uttar Pradesh, India
Malay Dhamelia IDC School of Design, Indian Institute of Technology,
Bombay, India
Roberto Dillon James Cook University, Singapore, Singapore
Kosmas Dimitropoulos Information Technologies Institute, CERTH,
Thessaloniki, Greece
Simena Dinas Departamento de Electrónica y Ciencias de la Computación,
Pontificia Universidad Javeriana Cali – Colombia, Cali, Valle, Colombia
Simena Dinas Facultad de Ingeniería, Universidad Santiago de Cali, Cali,
Colombia
Dilek Doğan Department of Informatics, Ankara University, Ankara, Turkey
Oğuz Orkun Doma Architectural Design Computing, Istanbul Technical
University, Istanbul, Turkey
Matt Dombrowski University of Central Florida, College of Arts & Human-
ities, School of Visual Arts & Design, Orlando, FL, USA
Joshua Dove-Henfrey Department of Computer Science, Faculty of Engi-
neering and Technology, Liverpool John Moores University, Liverpool, UK
Jennifer Coleman Dowling Communication Arts Department, Framingham
State University, Framingham, MA, USA
Giannis Drossis Computer Science Department, University of Crete,
Heraklion, Greece
Foundation for Research and Technology Hellas, Heraklion, Greece
Lucie Druoton Arts et Métiers, University of Burgundy-Franche-Comté,
Dijon, France
Contributors xxxvii

Shlomo Dubnov Department of Music and Computer Science and Engineer-

ing, University of California San Diego, San Diego, CA, USA

Tammuz Dubnov Zuzor, Tel Aviv, Israel

Adam Dubrowski Disciplines of Emergency Medicine and Pediatrics and

the Marine Institute, Memorial University of Newfoundland, St. Johns,
Canada
Elena Dzardanova Department of Product and Systems Design Engineer-
ing, University of the Aegean, Ermoupoli, Greece
Department of Product and Systems Design Engineering, University of the
Aegean, Mytilene, Greece

Anwar Yahya Ebrahim University of Babylon, Babylon, Iraq

Mehran Ebrahimi Faculty of Science, Ontario Tech University, Oshawa,
ON, Canada

Brian Edington Vincennes University, Vincennes, IN, USA

Gabriel Elvery University of Glasgow, Glasgow, Scotland

Chris Exton University of Limerick, Limerick, Ireland

Fazliaty Edora Fadzli School of Computing, Faculty of Engineering,
Universiti Teknologi Malaysia (UTM), Johor, Malaysia
Mixed and Virtual Environment Research Lab (mivielab), ViCubeLab,
Universiti Teknologi Malaysia, Johor, Malaysia

Marcelo Fantinato School of Arts, Sciences and Humanities, University of

Sao Paulo, Sao Paulo, Sao Paulo, Brazil

Sehar Shahzad Farooq Department of Computer Science and Engineering,

Sejong University, Seoul, South Korea
Stefano Ferretti Department of Computer Science and Engineering, Univer-
sity of Bologna, Bologna, Italy
Mikhail Fiadotau Centre of Excellence in Media Innovation and Digital
Culture (MEDIT), Tallinn University, Tallinn, Estonia

Frederico Fonseca Instituto Superior Miguel Torga, Coimbra, Portugal

Brandon Ford Vincennes University, Vincennes, IN, USA

Felipe Oviedo Frosi UniRitter Laureate International Universities, Porto

Alegre, Brazil

Laurent Fuchs Université de Poitiers, Chasseneuil, France

Seth Gaither Vincennes University, Vincennes, IN, USA

Robert John Gandy Liverpool Business School, Liverpool John Moores
University, Liverpool, UK
xxxviii Contributors

Tom Alexander Garner VIA Research, University of Portsmouth, Ports-

mouth, UK
Lionel Garnier Arts et Métiers, University of Burgundy-Franche-Comté,
Dijon, France
Damianos Gavalas Department of Product and Systems Design Engineer-
ing, University of the Aegean, Ermoupoli, Greece
Michele Geronazzo Department of Architecture, Design, and Media
Technology, Aalborg University, København, Denmark
Efstratios Geronikolakis Computer Science Department, University of
Crete, Heraklion, Greece
Foundation for Research and Technology Hellas, Heraklion, Greece
David Gibson Curtin University, Perth, WA, Australia
Gilson A. Giraldi National Laboratory of Scientific Computing (LNCC),
Petrópolis, Brazil
Curtis Gittens Department of Computer Science, Mathematics and Physics,
University of the West Indies, Cave Hill Campus, Bridgetown, Barbados
Sam Godby Creative Technologies Program, Illinois State University,
Normal, IL, USA
Eg Su Goh Media and Game Innovation Centre of Excellence, Institute of
Human Centered Engineering, Universiti Teknologi Malaysia, Johor Bahru,
Johor, Malaysia
Rafael Gonzales Creative Technologies Program, Illinois State University,
Normal, IL, USA
Randy Goodman Vincennes University, Vincennes, IN, USA
Alexandros Gouvatsos Hibbert Ralph Animation, London, UK
National Centre for Computer Animation, Bournemouth University, Dorset,
UK
Paweł Grabarczyk Center for Computer Games Research, IT University of
Copenhagen, København, Denmark
Alonso Gragera Aguaza Graduate School of Information Science and
Technology, The University of Tokyo, Tokyo, Japan
Nikolaos Grammalidis Information Technologies Institute, CERTH,
Thessaloniki, Greece
Athina Grammatikopoulou Information Technologies Institute, CERTH,
Thessaloniki, Greece
Fabien Gravot SQUARE-ENIX, Tokyo, Japan
Jordan Greenwood Federation University, Mt Helen, VIC, Australia
Contributors xxxix

Christiane Gresse von Wangenheim Department of Informatics and Statis-

tics (INE), Federal University of Santa Catarina (UFSC), Florianópolis, SC,
Brazil
Martin Guay Disney Research, Zürich, Switzerland
Simone Guggiari ETH Zurich, Zürich, Switzerland
Bouilly Guillaume Advanced Technology Division, SQUARE ENIX CO.,
LTD., Tokyo, Japan
Chen Guo School of Media Arts & Design, James Madison University,
Harrisonburg, VA, USA
Adan Häfliger Cygames Research, Cygames, Inc., Tokyo, Japan
Hooria Hajiyan Faculty of Science, Ontario Tech University, Oshawa, ON,
Canada
D. Fox Harrell MIT Comparative Media Studies Program and CSAIL,
Cambridge, MA, USA
Adrian R. G. Harwood Research IT, IT Services, The University of Man-
chester, Manchester, UK
Jian He Department of Computer Science, University of Texas at Austin,
Austin, TX, USA
Benjamin Hebgen NEC Research Labs Europe, Heidelberg, Germany
Laura L. Henderson The Honourable Society of Lincoln’s Inn, London, UK
The City Law School, City, University of London, London, UK
Ryan Hilderbrand Vincennes University, Vincennes, IN, USA
Hanno Hildmann Departamento de Ingenieria de Sistemas y Automatica,
Universidad Carlos III de Madrid, Leganes/Madrid, Spain
Jule Hildmann The University of Edinburgh, Edinburgh, UK
Takuya Hiraoka HEROZ, Inc., Osaka, Japan
Fatemeh Hirbodvash Faculty of Science, Ontario Tech University, Oshawa,
ON, Canada
Dylan Hires Vincennes University, Vincennes, IN, USA
Benjamin Hirsch Emirates ICT Innovation Center (EBTIC) / Khalifa
University, Abu Dhabi, United Arab Emirates
Aaron Hitchcock Computing and Software Systems, University of
Washington Bothell, Bothell, WA, USA
John Hoback Vincennes University, Vincennes, IN, USA
Celia Hodent Epic Games, Cary, NC, USA
Eric Hodgson Miami University, Oxford, OH, USA
xl Contributors

Kunihito Hoki Department of Communication Engineering and Informatics,

The University of Electro-Communications, Chofu, Tokyo, Japan
Jin Huang Zhejiang University, Hangzhou, China
Patrick C. K. Hung Faculty of Business and IT, Ontario Tech University,
Oshawa, ON, Canada
William Hurst Department of Computer Science, Liverpool John Moores
University, Liverpool, UK
Wolfgang Hürst Utrecht Center for Game Research, Utrecht University,
Utrecht, The Netherlands
Abir Hussain Liverpool John Moores University, Liverpool, UK
Sami Hyrynsalmi Pervasive Computing, Tampere University of Technol-
ogy, Pori, Finland
Sara Al Hajj Ibrahim Faculty of Science, Ontario Tech University, Oshawa,
ON, Canada
Kokolo Ikeda School of Information Science (Department of Information
Science & Artificial Intelligence), Japan Advanced Institute of Science and
Technology, Nomi, Ishikawa, Japan
Jouni Ikonen Software engineering, Lappeenranta University of Technol-
ogy, Lappeenranta, Finland
Farkhund Iqbal College of Technological Innovation, Zayed University,
Abu Dhabi, United Arab Emirates
Leah Irving Charles Sturt University, Wagga Wagga, NSW, Australia
Setsuko Ishii Independent Artist, Bunkyo-ku, Tokyo, Japan
Ajune Wanis Ismail Mixed and Virtual Reality Research Lab, Vicubelab,
School of Computing, Faculty of Engineering, Universiti Teknologi Malaysia,
Johor Bahru, Malaysia
Ismahafezi Ismail Universiti Sultan Zainal Abidin, Gong Badak, Malaysia
Takeshi Ito Department of Communication Engineering and Informatics,
The University of Electro-Communications, Chofu, Tokyo, Japan
Masahiko Itoh Institute of Industrial Science, The University of Tokyo,
Tokyo, Japan
Ranjit Jagtap Symbiosis Institute of Design, Pune, India
Christian F. Janßen Institute for Fluid Dynamics and Ship Theory, Hamburg
University of Technology, Hamburg, Germany
Brendan Jennings Emerging Networks Laboratory, Telecommunications
Software & Systems Group, Waterford Institute of Technology, Waterford,
Ireland
Contributors xli

Nancy Johnson Vincennes University, Vincennes, IN, USA

Taeya Johnson Vincennes University, Vincennes, IN, USA
Susan Johnston Select Services Films Inc., Los Angeles, CA, USA
Sicilia Ferreira Judice Faculty of Technical Education State of Rio de
Janeiro, FAETERJ Petropolis, Petropolis, Brazil
Oytun Kal Game Design, Bahcesehir University, Istanbul, Turkey
Devkan Kaleci Department of Computer Education and Instructional Tech-
nology, Faculty of Education, Inonu University, Malatya, Turkey
Kosuke Kaneko Cyber Security Center, Kyushu University, Fukuoka, Japan
Kamen Kanev Research Institute of Electronics, Shizuoka University,
Hamamatsu, Japan
Yowei Kang Kainan University, Taoyuan, Taiwan
Howard Kaplan University of South Florida, Tampa, FL, USA
Bill Kapralos Software Informatics Research Centre, University of Ontario
Institute of Technology, Oshawa, Canada
José Karam-Filho National Laboratory of Scientific Computing (LNCC),
Petrópolis, Brazil
Vlasios Kasapakis Department of Cultural Technology and Communication,
University of the Aegean, Mytilene, Greece
Daniel Kasnick Vincennes University, Vincennes, IN, USA
Yihao Ke Department of Computer Science, Sun Yat-sen University, Guang-
zhou, China
Rhiannon Kelly Wonsook Kim School of Art, Illinois State University,
Normal, IL, USA
Grady Kestler Department of Physics, Sonic Arts Research and Develop-
ment Group, Qualcomm Institute, University of California, San Diego,
La Jolla, CA, USA
Umair Azfar Khan School of Science & Engineering, Habib University,
Karachi, Sindh, Pakistan
Raksha Khandelwal Symbiosis Institute of Design, Pune, India
Huy Kang Kim Graduate School of Information Security, Korea University,
Seongbuk-Gu, Seoul, Republic of Korea
School of Cybersecurity, Korea University, Seoul, South Korea
Kyung-Joong Kim Department of Computer Science and Engineering,
Sejong University, Seoul, South Korea
Kai K. Kimppa Turku School of Economics, University of Turku, Turku,
Finland
xlii Contributors

Antti Knutas Lappeenranta University of Technology, Lappeenranta,

Finland
Hartmut Koenitz Professorship Interactive Narrative Design, HKU Univer-
sity of the Arts, Utrecht, The Netherlands
Hoshang Kolivand Department of Computer Science, Faculty of Engineer-
ing and Technology, Liverpool John Moores University (LJMU), Liverpool,
UK
Stacey A. Koornneef Ontario Tech University, Oshawa, ON, Canada
Aki Koponen School of Economics, University of Turku, Turku, Finland
Mehmet Kosa Department of Information Systems, Middle East Technical
University, Ankara, Turkey
Robert Kozma CLION, Department Mathematical Sciences, University of
Memphis, Memphis, TN, USA
Fabrizio Lamberti Dipartimento di Automatica e Informatica, Politecnico di
Torino, Torino, Italy
Eike Langbehn Human-Computer Interaction, Universität Hamburg,
Hamburg, Germany
Andrea Lanzi Università degli Studi di Milano, Milan, Italy
Kieran Latham Department of Computer Science, Liverpool John Moores
University, Liverpool, UK
Vicki Lau Seyenapse, Los Angeles, CA, USA
Chung V. Le Center for Simulation and Visualization, Duy Tan University,
Da Nang, Vietnam
Jaci Lee Lederman Vincennes University, Vincennes, IN, USA
Jong Weon Lee Department of Digital Contents, Sejong University, Seoul,
South Korea
Newton Lee Institute for Education, Research, and Scholarships, Los
Angeles, CA, USA
Vincennes University, Vincennes, IN, USA
Sangho Lee School of Cybersecurity, Korea University, Seoul, South Korea
Richard Levy University of Calgary, Calgary, AB, Canada
Peter R. Lewis Ontario Tech University, Oshawa, ON, Canada
Yong Li Department of Electronic Engineering, Tsinghua University,
Beijing, China
Fotis Liarokapis Masaryk University, Ponava, Brno, Czech Republic
Thomas Lilge gamelab.berlin, Humboldt University Berlin, Berlin,
Germany
Contributors xliii

Song Shuli Lily School of Digital Art and Animation, Communications

University of China, Beijing, P.R. China
Feng Lin China-Singapore International Joint Research Institute, Singapore,
Singapore
School of Computer Science and Engineering, Nanyang Technological
University, Singapore, Singapore
Jan Linxweiler Center for Mechanics, Uncertainty and Simulation in Engi-
neering, Technische Universität Braunschweig, Braunschweig, Germany
Yen-Hung Liu Faculty of Business and IT, Ontario Tech University, Oshawa,
ON, Canada
Nathan Lloyd Ontario Tech University, Oshawa, ON, Canada
Jaime Lochner Licensed Mental Health Counselor, Assistant Team Lead,
CAT Team, Aspire Health Partners, Orlando, FL, USA
Borja Barinaga López Universidad Francisco de Vitoria de Madrid, Madrid,
Spain
Victor M. López-Menchero Spanish Society of Virtual Archaeology,
Seville, Spain
Danny Ngo Lung Yao Faculty of Science and Natural Resources, Universiti
Malaysia Sabah, Kota Kinabalu, Malaysia
Jonathan A. Ly Vincennes University, Vincennes, IN, USA
Lizhaung Ma Shanghai Jiao Tong University, Shanghai, China
Krystina S. Madej Georgia Tech, Atlanta, Georgia, USA
Nadia Magnenat-Thalmann MIRALab, University of Geneva, Geneva 4,
Switzerland
Adnan Mahmood Emerging Networks Laboratory, Telecommunications
Software & Systems Group, Waterford Institute of Technology, Waterford,
Ireland
Communications Research Group, Faculty of Engineering, Universiti
Malaysia Sarawak, Sarawak, Malaysia
Amol D. Mali Computer Science Department, University of Wisconsin-
Milwaukee, Milwaukee, WI, USA
Kazuko Manabe Square Enix Co., Ltd., Tokyo, Japan
Federico Manuri Dipartimento di Automatica e Informatica, Politecnico di
Torino, Turin, Italy
Raphaël Marczak Univ. Bordeaux, Bordeaux INP, CNRS, LaBRI,
UMR5800, Talence, France
George Margetis Foundation for Research and Technology Hellas,
Heraklion, Greece
xliv Contributors

Ilaria Mariani Department of Design, Politecnico di Milano, Milan, Italy

Dianne Marsh Liverpool John Moores University, FET, SBE, Liverpool,
UK
Hector J. Martínez Universidad del Valle, Cali, Colombia
Moreno Marzolla Department of Computer Science and Engineering,
University of Bologna, Bologna, Italy
Deepak John Mathew Indian Institute of Technology, Hyderabad, India
Wilfred Matipa Liverpool John Moores University, FET, SBE, Liverpool,
UK
Ken S. McAllister Rhetoric, Composition, and the Teaching of English,
University of Arizona, Tucson, AZ, USA
Kyle McCarter Vincennes University, Vincennes, IN, USA
Tim McGraw Purdue University, West Lafayette, IN, USA
Michael Mcloughlin Centre for Digital Music, Queen Mary University of
London, London, UK
Ryan P. McMahan University of Texas at Dallas, Richardson, TX, USA
Michael McMillan Vincennes University, Vincennes, IN, USA
Nannapat Meemongkolkiat Faculty of Business and IT, Ontario Tech
University, Oshawa, ON, Canada
Felipe M. Megale Ontario Tech University, Oshawa, ON, Canada
Devi Meghana Department of Design, Indian Institute of Technology Hyder-
abad, Hyderabad, India
Grant Meredith Federation University, Mt Helen, VIC, Australia
Tristan Michael Simmons Wonsook Kim School of Art, Illinois State
University, Normal, IL, USA
Risto Miikkulainen Department of Computer Science, University of Texas
at Austin, Austin, TX, USA
Hidenori Mimura Research Institute of Electronics, Shizuoka University,
Hamamatsu, Japan
Xiaoping Min College of Computer and Information Science, Northeastern
University, Boston, MA, USA
Silvia Mirri University of Bologna, Bologna, Italy
Youichiro Miyake Square Enix Co., Ltd., Tokyo, Japan
Yuta Mizuno SQUARE ENIX Co., Ltd., Tokyo, Japan
M. Mohan School of Design, University of Petroleum and Energy Studies,
Dehradun, Uttarakhand, India
Contributors xlv

Norhaida Mohd Suaib Universiti Teknologi Malaysia, Skudai, Malaysia

Géraldine Morin Laboratoire IRIT, Université Paul Sabatier, Toulouse,
France
Brooke Morrill Schell Games, Pittsburgh, PA, USA
Christos Mousas Visual Computing Lab, Department of Computer Science,
Dartmouth College, Hanover, NH, USA
Martin Müller University of Alberta, Edmonton, AB, Canada
Damian T. Murphy Department of Electronic Engineering, University of
York, York, UK
Jamila Mustafina Kazan Federal University, Kazan, Russia
Devon Myers Vincennes University, Vincennes, IN, USA
Ricardo Nakamura Polytechnic School, University of São Paulo, São
Paulo, Brazil
Eugene Nalivaiko School of Biomedical Sciences and Pharmacy, University
of Newcastle, Callaghan, NSW, Australia
Andrés Adolfo Navarro-Newball Electronics and Computer Science,
Pontificia Universidad Javeriana, Cali, Colombia
Nicusor Nedelcu 7thFACTOR Entertainment Studios, Brasov County,
Romania
Keith Nesbitt School of Electrical Engineering and Computing, University
of Newcastle, Callaghan, NSW, Australia
Kim J. L. Nevelsteen Immersive Networking, DSV, Stockholm University,
Kista, Sweden
Philip W. S. Newall Experimental Gambling Research Laboratory, School of
Health, Medical and Applied Sciences, CQUniversity, Sydney, NSW,
Australia
Tho L. Nguyen Center for Simulation and Visualization, Duy Tan Univer-
sity, Da Nang, Vietnam
Rune K. L. Nielsen Center for Computer Games Research, IT University of
Copenhagen, København, Denmark
Niels Christian Nilsson Aalborg University Copenhagen, Copenhagen,
Denmark
Muhammad Nur Affendy Nor’a Mixed and Virtual Reality Research Lab,
Vicubelab, Universiti Teknologi Malaysia, Johor Bahru, Malaysia
Chris North CVG Group, Crytek, Frankfurt am Main, Germany
Fátima L. S. Nunes School of Arts, Sciences and Humanities, University of
São Paulo, São Paulo, Brazil
xlvi Contributors

Darryl O’Hare Imagin3D, Office 4, CTH, Sci Tech Daresbury, Daresbury,

UK
Yoshihiro Okada Department of Informatics, ISEE, Graduate School of
Information Science and Electrical Engineering, Kyushu University Library,
Kyushu University, Nishi-ku, Fukuoka, Japan
Innovation Center for Educational Resource, Kyushu University, Nishi-ku,
Fukuoka, Japan
Danielle Marie Olson Computer Science Artificial Intelligence Laboratory
(CSAIL), Massachusetts Institute of Technology (MIT), Cambridge, MA,
USA
Prasad S. Onkar Department of Design, Indian Institute of Technology
Hyderabad, Hyderabad, India
Santiago Ontañón Computer Science Department, Drexel University,
Philadelphia, PA, USA
Ozan Özkan Augmea Simulation Technologies A.S., Istanbul, Turkey
Department of Information Technologies, Marmara University, Istanbul,
Turkey
Selcen Ozturkcan Jönköping International Business School, Jönköping
University, Jönköping, Sweden
Faculty of Communication, Bahcesehir University, Istanbul, Turkey
Francesco De Pace Institute of Visual Computing and Human-Centered
Technology, TU Wien, Wien, Austria
Marlene Palafox Mexico City, Mexico
Swapnil Pande Product Designer (UI/UX), Delhi, India
George Papagiannakis Computer Science Department, University of Crete,
Heraklion, Greece
Foundation for Research and Technology Hellas, Heraklion, Greece
Gianluca Paravati Dipartimento di Automatica e Informatica, Politecnico di
Torino, Torino, Italy
Nikolaos Partarakis Foundation for Research and Technology Hellas,
Heraklion, Greece
Maria Pateraki Foundation for Research and Technology Hellas, Heraklion,
Greece
Emily Peed Institute for Education, Research, and Scholarships, Los
Angeles, CA, USA
Ulka Chandini Pendit Department of Virtual Reality, Faculty of Creative
Multimedia, Multimedia University, Cyberjaya, Selangor, Malaysia
Rui Penha INESC-TEC and FEUP, University of Porto, Porto, Portugal
Contributors xlvii

Giani Petri Department of Informatics and Statistics (INE), Federal Univer-

sity of Santa Catarina (UFSC), Florianópolis, SC, Brazil
Jaden C. Pfoff Vincennes University, Vincennes, IN, USA
Michael Phillips Vincennes University, Vincennes, IN, USA
Krzysztof Pietroszek Game Research Lab, School of Computing and
Design, California State University Monterey Bay, Seaside, CA, USA
Immesive Designs, Experiences, Applications and Stories (IDEAS) Lab,
School of Communication, American University, Washington, DC, USA
Éric Piette Maastricht University, Maastricht, The Netherlands
Louis Pisha Department of Electrical and Computer Engineering, Sonic Arts
Research and Development Group, Qualcomm Institute, University of
California, San Diego, La Jolla, CA, USA
Zac Pitcher Vincennes University, Vincennes, IN, USA
Vaughan Powell VIA Research, University of Portsmouth, Portsmouth, UK
Wendy Powell VIA Research, University of Portsmouth, Portsmouth, UK
Rachel Power Init Esports, Lewes, DE, USA
Edmond Prakash Cardiff School of Technologies, Cardiff Metropolitan
University, Cardiff, UK
Catia Prandi University of Bologna, Bologna, Italy
Mike Preuss Information Systems and Statistics, Westfälische Wilhelms-
Universität Munster, Münster, Germany
A. Protopsaltis University of Western Macedonia, Kozani, Greece
ORamaVR S.A., Heraklion, Greece
Anna Pyayt University of South Florida, Tampa, FL, USA
John Quarles Department of Computer Science, University of Texas at San
Antonio, San Antonio, TX, USA
Rupendra Raavi Faculty of Business and IT, Ontario Tech University,
Oshawa, ON, Canada
Laura Rafferty Faculty of Business and IT, University of Ontario Institute of
Technology, Oshawa, ON, Canada
Madhusudan Rao NTT Data Services Pvt. Ltd, Bangalore, India
Shawkh Ibne Rashid Faculty of Science, Ontario Tech University, Oshawa,
ON, Canada
Werner Ravyse Faculty of ICT and Chemical Engineering, Futuristic Inter-
active Technologies, Turku University of Applied Sciences, Turku, Finland
Technology Enhanced Learning for Innovative Teaching and Training South
Africa (TELIT-SA), North-West University, Vanderbijlpark, South Africa
xlviii Contributors

Manuel Rebol American University, Washington, DC, USA

Graz University of Technology, Graz, Austria
Carol Luckhardt Redfield Computer Science Department, St. Mary’s
University, San Antonio, TX, USA
Brayden Rexing Vincennes University, Vincennes, IN, USA
Laura Ricci Department of Computer Science, University of Pisa, Pisa, Italy
Florian Richoux Nantes Atlantic Computer Science Laboratory (LINA),
University of Nantes, Nantes, France
David W. Rick Rhetoric, Composition, and the Teaching of English,
University of Arizona, Tucson, AZ, USA
Erin Elizabeth Ridgely Creative Technologies Program, Illinois State
University, Normal, IL, USA
Nicholas Ries Vincennes University, Vincennes, IN, USA
Jorge Roa Research and Development Center of Information Systems Engi-
neering (CIDISI), Universidad Tecnológica Nacional – Facultad Regional
Santa Fe (UTN-FRSF), Santa Fe, Santa Fe, Argentina
Sergio Rodríguez Valdunciel Department of Drawing, Universidad
Politécnica de Valencia, Valencia, Spain
Jake Romershausen Indiana University, Bloomington, IN, USA
Sam Romershausen Vincennes University, Vincennes, IN, USA
Nina Rosa Utrecht Center for Game Research, Utrecht University, Utrecht,
The Netherlands
Satyaki Roy Design Programme, Indian Institute of Technology, Kanpur,
Uttar Pradesh, India
Judd Ethan Ruggill School of Social and Behavioral Sciences, Arizona
State University, Tempe, AZ, USA
Sun Ruoqi Shanghai Jiao Tong University, Shanghai, China
Pensyl William Russell College of Arts, Media and Design, Northeastern
University, Boston, MA, USA
Nur Syafiqah Safiee Mixed and Virtual Reality Research Lab, Vicubelab,
Universiti Teknologi Malaysia, Johor Bahru, Malaysia
School of Computing, Faculty of Engineering, Universiti Teknologi Malaysia,
Johor Bahru, Malaysia
Parisa Salmani Ontario Tech University, Oshawa, Canada
Paola Salomoni University of Bologna, Bologna, Italy
Yavuz Samur Computer Education and Instructional Technologies,
Bahcesehir University, Istanbul, Turkey
Contributors xlix

Isidro Moreno Sánchez Universidad Complutense de Madrid, Madrid,

Spain
Andrew Sands Imagin3D, Office 4, CTH, Sci Tech Daresbury, Daresbury,
UK
Andrea Sanna Dipartimento di Automatica e Informatica, Politecnico di
Torino, Turin, Italy
Stevie C. F. Sansalone Ontario Tech University, Oshawa, ON, Canada
Pratiti Sarkar Design Programme, Indian Institute of Technology, Kanpur,
Uttar Pradesh, India
Yukiko Sato Cygames, Inc., Shibuya, Tokyo, Japan
Daiki Satoi SQUARE ENIX Co., Ltd., Tokyo, Japan
Yann Savoye Institut fur Informatik, Innsbruck University, Room 3M11,
Innsbruck, Austria
Manuel Schmidt University of Innsbruck, Innsbruck, Austria
Ferdinand Schober Xbox Advanced Technology Group, Microsoft Corpo-
ration, Redmond, WA, USA
Jacob Schrum Department of Mathematics and Computer Science, South-
western University, Georgetown, TX, USA
Florian Schübeler The Embassy of Peace, Whitianga, New Zealand
John Scott Vincennes University, Vincennes, IN, USA
Katy Scott Curtin University, Perth, WA, Australia
Hock Soon Seah School of Computer Science and Engineering, Nanyang
Technological University, Singapore, Singapore
Sercan Şengün Wonsook Kim School of Art, Illinois State University,
Normal, IL, USA
Massachusetts Institute of Technology CSAIL, Cambridge, MA, USA
Stefania Serafin Multisensory Experience Lab, Aalborg University Copen-
hagen, Copenhagen, Denmark
Diğdem Sezen Faculty of Communications, Istanbul University, Istanbul,
Turkey
Tonguc Ibrahim Sezen Faculty of Communication and Environment, Rhine-
Waal University of Applied Sciences, Kamp-Lintfort, Germany
Raj Shah Liverpool John Moores University, FET, SBE, Liverpool, UK
Yuanyuan Shen Liverpool John Moores University, Liverpool, UK
Isabel Cristina Siqueira da Silva UniRitter Laureate International Univer-
sities, Porto Alegre, Brazil
l Contributors

Adalberto L. Simeone KU Leuven, Department of Computer Science,

Heverlee, Belgium
Cansu Nur Simsek Kadir Has University, Istanbul, Turkey
Tulasii Sivaraja Mathematics, Graphics and Visualization Research Group
(MGRAVS), Faculty of Science and Natural Resources, Universiti Malaysia
Sabah, Kota Kinabalu, Sabah, Malaysia
Petter Skult Faculty of Arts, Psychology and Theology, Åbo Akademi
University, Turku, Finland
Jouni Smed Department of Future Technologies, University of Turku,
Turku, Finland
Ross T. Smith School of Information Technology and Mathematical
Sciences, University of South Australia, Mawson Lakes, SA, Australia
Dennis J. N. J. Soemers Maastricht University, Maastricht, The Netherlands
Chaitanya Solanki Indian Institute of Technology, Hyderabad, India
Christian Stein gamelab.berlin, Humboldt University Berlin, Berlin,
Germany
Frank Steinicke Human-Computer Interaction, Universität Hamburg,
Hamburg, Germany
Constantine Stephanidis Computer Science Department, University of
Crete, Heraklion, Greece
Foundation for Research and Technology Hellas, Heraklion, Greece
Conor Stephens University of Limerick, Limerick, Ireland
Matthew Stephenson Maastricht University, Maastricht, The Netherlands
Lauren Elizabeth Stipp Creative Technologies Program, Illinois State
University, Normal, IL, USA
Mahadeo Sukhai ARIA Team, Canadian National Institute for the Blind,
Toronto, ON, Canada
CNIB Foundation, Kingston, ON, Canada
Robert W. Sumner Disney Research, Zürich, Switzerland
Mohd Shahrizal Sunar Media and Game Innovation Centre of Excellence,
Institute of Human Centered Engineering, Universiti Teknologi Malaysia,
Johor Bahru, Johor, Malaysia
Kelvin Sung Computing and Software Systems, University of Washington
Bothell, Bothell, WA, USA
Vorapong Suppakitpaisarn JST, ERATO Kawarabayashi Large Graph
Project, National Institute of Informatics, Tokyo, Japan
Elif Surer Department of Modeling and Simulation, Graduate School of
Informatics, Middle East Technical University, Ankara, Turkey
Contributors li

Elisabeth Ainsley Sutherland Mediate VR, Cambridge, MA, USA

Erkki Sutinen Department of Future Technologies, University of Turku,
Turku, Finland
Christian Swinford Vincennes University, Vincennes, IN, USA
Stella Sylaiou Hellenic Open University, Patras, Greece
Gabriel Synnaeve Cognitive Science and Psycholinguistics (LSCP) of ENS
Ulm, Paris, France
Mostafa Tajdini Staffordshire University, Stoke on Trent, UK
Takenobu Takizawa Faculty of Political Science and Economics, Waseda
University, Tokyo, Japan
Sin Ying Tan Liverpool John Moores University, Liverpool, UK
Kimmo Tarkkanen Faculty of ICT and Chemical Engineering, Futuristic
Interactive Technologies, Turku University of Applied Sciences, Turku,
Finland
Huda Kadhim Tayyeh Information System Management(ISM), College of
Business Informatics, University of Information Technology and Communi-
cations, Baghdad, Iraq
Sam-Odusina Temitope Liverpool John Moores University, FET, SBE,
Liverpool, UK
Tansel Tepe Department of Computer Education and Instructional Technol-
ogy, Muallim Rıfat Education Faculty, Kilis 7 Aralık University, Kilis, Turkey
Burak Tezateşer Nowhere Studios, Istanbul, Turkey
Daniel Thalmann Institute for Media Innovation, Nanyang Technological
University, Singapore, Singapore
Mattia Thibault Interdisciplinary Center of Research on Communication,
University of Turin, Turin, Italy
Bruce H. Thomas School of Information Technology and Mathematical
Sciences, University of South Australia, Mawson Lakes, SA, Australia
Özhan Tıngöy Department of Information Technologies, Marmara Univer-
sity, Istanbul, Turkey
Romero Tori Polytechnic School, University of São Paulo, São Paulo, Brazil
Rodrigo Torres Instituto Tecnológico y de Estudios Superiores de Monter-
rey, Mexico City, Mexico
Róbert Tóth Faculty of Informatics, University of Debrecen, Debrecen,
Hungary
Jolanda G. Tromp Center for Simulation and Visualization, Duy Tan
University, Da Nang, Vietnam
lii Contributors

Tsung-Chih Tsai Fashion Tech, Nogle, Taipei, Taiwan

Michael Tsioumas Hellenic Ministry of Culture and Sports, Service of
Modern Monuments and Technical Works of Central Macedonia,
Thessaloniki, Greece
Cetin Tuker Graphic Design Department, Mimar Sinan Fine Arts University,
Istanbul, Turkey
Hakan Tüzün Department of Computer Education and Instructional Tech-
nology, Faculty of Education, Hacettepe University, Ankara, Turkey
Alberto Uriarte Computer Science Department, Drexel University,
Philadelphia, PA, USA
Alvaro Uribe-Quevedo Software Informatics Research Centre, University
of Ontario Institute of Technology, Oshawa, Canada
Ahmet Uysal Department of Psychology, Middle East Technical University,
Ankara, Turkey
Jukka Vahlo School of Economics, University of Turku, Turku, Finland
Peter Vamplew Faculty of Science and Technology, Federation University
Australia, Mt Helen, Ballarat, VIC, Australia
Martha Vassiliadi Department of Philology, School of Philosophy, Aristotle
University of Thessaloniki, Thessaloniki, Greece
Carolina Padilla Velasco Ontario Tech University, Oshawa, ON, Canada
Remco Veltkamp Utrecht Center for Game Research, Utrecht University,
Utrecht, The Netherlands
Anja Venter Applied Design, Cape Peninsula University of Technology,
Cape Town, South Africa
Rojin Vishkaie College of Communication, Information, and Media, Ball
State University, Muncie, IN, USA
G. Stewart Von Itzstein School of Information Technology and Mathemat-
ical Sciences, University of South Australia, Mawson Lakes, SA, Australia
Evan Lee Wagoner Vincennes University, Vincennes, IN, USA
Isaac Wake Vincennes University, Vincennes, IN, USA
Min Wang Shanghai Jiao Tong University, Shanghai, China
Tong Wang Cygames, Inc., Tokyo, Japan
Shuang Wei Department of Computer Graphics Technology, Purdue Univer-
sity, West Lafayette, IN, USA
Peter Werkhoven Utrecht Center for Game Research, Utrecht University,
Utrecht, The Netherlands
Contributors liii

Duncan A. H. Williams Digital Creativity Labs, Department of Computer

Science, University of York, York, UK

Mark H. M. Winands Department of Data Science and Knowledge Engi-

neering, Maastricht University, Maastricht, The Netherlands

Inon Wiratsin Faculty of Business and IT, Ontario Tech University, Oshawa,
ON, Canada

Sai-Keung Wong National Chiao Tung University, Hsinchu, Taiwan

Ji Young Woo Department of Big Data Engineering, Soonchunhyang

University, Asan-si, South Korea

Jiyoung Woo Graduate School of Information Security, Korea University,

Seongbuk-Gu, Seoul, Republic of Korea

Di Wu Department of Computer Science, Sun Yat-sen University,

Guangzhou, China

Leon Y. Xiao The Honourable Society of Lincoln’s Inn, London, UK

School of Law, Queen Mary University of London, London, UK
Center for Computer Games Research, IT University of Copenhagen,
København, Denmark

Zhidong Xiao National Centre for Computer Animation, Bournemouth

University, Dorset, UK

Rebecca Ruige Xu Syracuse University, Syracuse, NY, USA

Shahrokh Yadegari Department of Music, Sonic Arts Research and Devel-

opment Group, Qualcomm Institute, University of California, San Diego,
La Jolla, CA, USA

Junzi Yang Mixed and Virtual Reality Research Lab, Vicubelab, School of
Computing, Faculty of Engineering, Universiti Teknologi Malaysia, Johor
Bahru, Malaysia

Kenneth C. C. Yang The University of Texas at El Paso, El Paso, TX, USA

Anna Yankovskaya Tomsk State University of Architecture and Building,

Tomsk, Russia

Benjamin Yankson Faculty of Business and IT, University of Ontario Insti-

tute of Technology, Oshawa, Ontario, Canada

Masasuke Yasumoto Department of Information Media, Kanagawa Institute

of Technology, Kanagawa, Japan

Matthew Yee-King Computing, Goldsmiths College, University of London,

London, UK

Murat Yilmaz Virtual Reality Research and Development Laboratory,

Department of Computer Engineering, Cankaya University, Ankara, Turkey
liv Contributors

Han Kyu Yoo Department of Digital Contents, Sejong University, Seoul,

South Korea
Hiroshi Yoshikawa Department Computer Engineering, College of Science
and Technology, Nihon University, Funabashi, Chiba, Japan
Kazuki Yoshizoe Graduate School of Frontier Sciences, The University of
Tokyo, Tokyo, Japan
Fares Yousefi Department of Computer Science, Faculty of Engineering and
Technology, Liverpool John Moores University (LJMU), Liverpool, UK
Fangyi Yu Ontario Tech University, Oshawa, ON, Canada
Shigang Yue School of Computer Science, University of Lincoln, Lincoln,
UK
Müberra Yüksel Faculty of Applied Sciences, Department of International
Trade, Boğaziçi University, Istanbul, Turkey
Cik Suhaimi Yusof Mixed and Virtual Reality Research Lab, Vicubelab,
Universiti Teknologi Malaysia, Johor Bahru, Malaysia
Paulo Zaffari Hoplon Infotainment S.A., Florianopolis, Brazil
Syamsul Bahrin Zaibon Institude of Creative Humanities, Multimedia &
Innovation, School of Creative Industry Management & Performing Arts,
Universiti Utara Malaysia, Sintok, Kedah, Malaysia
Cagri Hakan Zaman MIT CSAIL, Cambridge, MA, USA
Hushairi Zen Communications Research Group, Faculty of Engineering,
Universiti Malaysia Sarawak, Sarawak, Malaysia
YanXiang Zhang Department of Communication of Science and Technol-
ogy, University of Science and Technology of China, Hefei, Anhui, China
Feng Zhao School of Information Science and Technology, University of
Science and Technology of China, Hefei, China
QingQing Zhao Department of Communication of Science and Technology,
University of Science and Technology of China, Hefei, Anhui, China
Marianna Zichar Faculty of Informatics, University of Debrecen, Debrecen,
Hungary
A

2D Animation 2-Simplex Prism as a

Cognitive Graphics Tool for
▶ Pipeline of 2D Vector Animation in Television Decision-Making
Series
Anna Yankovskaya
Tomsk State University of Architecture and
Building, Tomsk, Russia
2D Game

▶ Principle Structure to Create a 2D Game Level Synonyms

Editor
2-Simplex prism; Cognitive graphics tool;
Decision-making; Geoinformation system; Intel-
ligent dynamic system; Justification; n-simplex
2D Game Engines

▶ Game Engine Definition

The following definitions from the papers

(Yankovskaya 2011; Yankovskaya et al. 2015a)
2-Dimensional Animation are used:
A cognitive graphics tool (CGT) visually
▶ Exploring Innovative Technology: 2D Image reflects a complex object, phenomenon, or pro-
Based Animation with the iPad cess on a computer screen, enabling the users to
form a new decision, idea, or hypothesis based on
the visuals.
2-Simplex is an equilateral triangle.
2-Simplex Prism 3-Simplex is a regular tetrahedron.
2-Simplex prism is a triangular prism which
▶ 2-Simplex Prism as a Cognitive Graphics Tool has identical equilateral triangles (2-simplexes) in
for Decision-Making its bases.

© Springer Nature Switzerland AG 2024

N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
2 2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making

The height of the 2-simplex prism hypothesis). An important feature of CGT is

(Yankovskaya et al. 2015a) in intelligent dynamic targeted influence on the intuitive imaginative
systems corresponds to the dynamic process time thinking mechanisms.
interval under consideration. It is divided into a Cognitive graphics tools are used in a variety
number of time intervals. The number of time of intelligent systems for analysis of information
intervals corresponds to the number of diagnostic structures of knowledge and data, revealing
or predictive decisions. of different kinds of regularities in data and
The distance between two adjacent knowledge, and decision-making and its justifi-
2-simpleces is proportional to the time interval cation. Possible areas of application are medicine
between adjacent 2-simpleces. (diagnosis of diseases, treatment and preventive
In this case the distance between two adjacent measures, rehabilitation of patients as well
2-simpleces corresponds to the distance between as solving organizational and managerial
two points on a map. problems), education, geology, engineering,
Learning axis is an independent direction of electronics, sociology, psychology, psychiatry,
student evolution, e.g., theory knowledge level, ecobiomedicine, ecogeology, etc. Dynamism
ability to practical tasks solving, ability of electric represented by CGT can effectively reflect the
circuit design, and research. dynamics of the patterns of objects under inves-
Test result is a set of coefficients, numerically tigation in time.
representing the student’s assessment achieved.
Each coefficient corresponds to a particular
learning axis. Problem Background and Mathematical
Prediction result is a set of coefficients, numer- Basics of Intelligent Systems
ically representing the student’s assessment which Construction
he is likely to achieve in a preassigned time
interval. We suggest the following classifications of CGT
by ways of representation (Yankovskaya and
Galkin 2009; Yankovskaya 2011):
Introduction
1. Naturalistic CGT, familiar graphic images of
Cognitive graphics tool (CGT) was developed as real objects. For example, visualization of tree
part of AI research in the 1980s of the twentieth data structure of pathological processes in
century in the works of Pospelov (1992) and bronchus (bronchial asthma, Fig. 1a).
Zenkin (1991). A large number of CGT have 2. Abstract CGT, CGT that does not have a map-
been created and developed, including cognitive ping in the ordinary reality. For example, visu-
maps (Axelrod 1976), cognitive graphics system alization of current level of students’
(Albu and Khoroshevskiy 1990), medical CGT knowledge by applying 2-simplex (Fig. 1b).
(Kobrinskiy 1996), radar plots (Saary 2008),
Chernoff faces (Raciborski 2009), the ternary dia- We use test pattern recognition to create the
gram (Wang et al. 2012), 2-simplex (Yankovskaya intelligent systems (Yankovskaya 2011), based
2011), and others (Yankovskaya et al. 2015a; on nontraditional (unusual) matrix method of
Yankovskaya 2017). CGT allows representing data and knowledge representation; deep opti-
the content of the object or process under investi- mizing logic-combinatorial transformations in
gation on a computer screen. CGT visually and feature space; logic-combinatorial and logic-
clearly reflects the essence of a complex object probabilistic, logic-combinatorial-probabilistic,
(phenomenon, process) and is also capable of and genetic methods of test pattern recognition;
providing a fundamentally new decision (idea, methods of complex decision-making; cognitive
2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making 3

2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making, Fig. 1 Examples of naturalistic-based CGT
(a) and CGT that does not have mapping in the ordinary reality (b)

2-Simplex Prism as a z1 z2 z3 z4 z5 z6 z7 z8 z9 z10 z11 k1 k2

Cognitive Graphics Tool 1 4 6 3 2 2 1 2 3 4 1 1 1 2 1
for Decision-Making, 4 4 5 2 3 2 7 8 3 4 1 2 1 2 1
Fig. 2 Matrix model for 3 4 5 3 3 2 4 5 3 4 1 3 1 2 1
the data and knowledge 3 4 4 1 4 4 2 3 1 5 1 4 2 1 2
representation Q= 2 4 2 1 6 3 4 5 2 3 1 5 R= 2 1 R'= 2
2 4 5 1 3 - 4 3 1 2 1 6 2 1 2
1 4 3 2 5 2 1 2 3 4 1 7 1 3 3
3 4 2 2 6 2 2 3 3 2 1 8 1 3 3
5 4 2 2 6 3 5 6 2 4 1 9 3 2 4
4 4 6 1 2 5 5 6 1 4 2 10 3 2 4

means of decision-making; and justification of The element qij of the matrix Q determines the
decision-making results. value j-th feature for i-th object if the value is
We use unusual matrix model (Yankovskaya defined and “–” if the value is undefined.
2011) for the data and knowledge representation, Each row of R is corresponded to the row of the
example of which is given in Fig. 2. matrix Q having the same index.
Q – integer descriptions matrix. Columns of R are corresponded to
z1, z2, . . ., z11 – characteristic features. distinguishing levels that represent classification
R – integer distinguishing matrix. features.
k1, k2 – classification features. The set of all nonrepeating rows of the matrix
R0 – one-column matrix whose elements are R is compared to the number of selected patterns
the numbers of patterns. Each pattern is associated presented by the one-column matrix R0 .
with the final decision. The distinguishing matrices can be of three
Rows of Q are mapped to objects from learning types (R1, R2, and R3). The rows of Q are put in
sample of a problem field. correspondence with the rows of R1, R2, and R3
Columns of Q are mapped to characteristic and the levels of distinguishing (classification)
features, which describe each object. features, with the columns of these matrices. R1
4 2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making

2-Simplex Prism as a
Cognitive Graphics Tool
for Decision-Making,
Fig. 3 A block diagram of
regularities definition

represents the included classification mecha- patterns (classes at the fixed mechanism of classi-
nisms. R2 determines the sequence of actions fication). The rows of U0 are associated with char-
which must be performed for each object. R3 acteristic features, and its columns, with the results
represents independent classification mechanisms of comparison of all possible pairs, pattern–pattern,
corresponding, for instance, to the opinions of object–pattern, and object–object, from different
different experts. patterns.
The matrix of transition P (Yankovskaya et al. A diagnostic test (DT) is a set of features that
2001; Yankovskaya 2011) is intended for the rep- distinguishes any pair of objects that belong to
resentation of dynamic knowledge about the inves- different patterns and constructed on the base of
tigated objects. Its rows are associated with the rows irredundant implication matrix with application of
of Q, and columns are associated with the instants logical-combinatorial algorithms (i.e., column
(intervals) of time or control actions (Fig. 2). coverings findings).
The weight coefficients of the features charac- The DT is called “unconditional” if all features
terizing their individual contribution to the distin- of the investigated object included in test are used
guishability of objects from certain patterns simultaneously in decision-making process.
(Yankovskaya 2011) and the information weight Decision-making on belonging of object under
defined on the subset of tests used for a final study to one or another pattern for every
decision-making (Yankovskaya 2011) with use irredundant unconditional DT (IUDT) is
the definition from Zhuravlev and Gurevitch performed out with use of threshold value of
(1990) is also regarded as regularities. conditional degree of proximity of the object
The following approaches are used when under study to the patterns.
revealing regularities (Yankovskaya 2011): Mixed DT (MDTs) present a new paradigm of
(1) with the construction of the irredundant matrix development of intelligent systems based on test
of implications; (2) with the partial construction of methods of pattern recognition (Yankovskaya 1996).
the irredundant matrix of implications; and MDT is a compromise between unconditional
(3) without the construction of the irredundant and conditional components which are used for
matrix of implications. decision-making in intelligent systems as well as
An irredundant matrix of implications (U0 ) is in blended education and training.
constructed on the base of matrices Q and R and Respondent is a person participating in learn-
defines distinguishability of objects from different ing and testing.
2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making 5

Definition of a membership object x under (i ¼ 1, 2, . . ., n + 1) is the distance from this point

study to pattern k is defined based on the coeffi- to the i-th side (Yankovskaya 1991).
cient of conditional degree of proximity (ak) of Coefficient hi, (i ¼ 1, 2, . . ., n + 1) represents A
object under study to the pattern (class) k, calcu- the degree of conditional proximity of the object
lated by the formula (1): under investigation to the i-th pattern. The advan-
tage is that n-simplex possesses the property of the
Sxk constancy of the sum of distances (h) from any
ak ¼ ,
Sk point to each side and the property of ratios pres-
ervation h1: h2: . . .: hn + 1 ¼ a1: a2: . . .: an + 1.
where x – object under study The main function of n-simplex is a represen-
tation of an investigated object disposition among
k – pattern (class), other objects of a learning sample (Yankovskaya
Sk – coefficient of interclass similarity (similarity 2011).
of objects inside the pattern (class)),
Skx– coefficient of similarity object x under study Construction of Cognitive Graphics Tools:
with pattern (class) k. 2-Simplex, 3-Simplex, and 2-Simplex Prism
We construct CGT 2-simplex and 3-simplex to visu-
The rules for decision-making are constructed alize objects of 3 (4) patterns and the object under
using coefficients of conditional degree of prox- investigation using the formulas (1) given below.
imity ai (Yankovskaya 2011). For 2-simplex (3-simplex), distances h1, h2,
Applied intelligent system is constructed on and h3 (h1, h2, h3, h4) are calculated on the basis
the base of intelligent instrumental software (IIS) of coefficients ai and normalization operation
IMSLOG (Yankovskaya et al. 2003). Purposes of from the following relations:
IIS IMSLOG are formation of knowledge about For 3-simplex distances h1, h2, h3, and h4 are
objects in concrete problem or interdisciplinary calculated on the basis of coefficients ai and nor-
area; regularities revealing in data and knowledge; malization operation from the following relations:
decision rules construction on the basis of
revealed regularities; recognition of object under 4
investigation; and decision-making and its justifi- h¼ hi
cation with CGT. i¼1
4
h¼a ai ð1Þ
i¼1
Mathematical Basis for Construction of h1 h2 h3 h4
¼ ¼ ¼
Cognitive Graphics Tools a1 a2 a3 a4

Theorem for the Construction of a Cognitive by the formula

Graphics Tool n-Simplex
In 1991, Yankovskaya (1991), it was proposed h  ai
hi ¼ 4
ð2Þ
using cognitive graphics tool 2-simplex for
ai
decision-making about object under investigation i¼1
in the characteristic feature space to a certain
pattern in intelligent systems for decision-making. Edges of 2-simplex (equilateral triangle edges),
shown in Fig. 1b, are associated with three patterns
Theorem Suppose a1, a2, . . ., an + 1 is a set of (classes); the circle with big radius is the investi-
simultaneously non-zero numbers where n is the gated object, and small circles are sample objects.
dimension of a regular simplex. Then, there is one Each pattern corresponds with some color. The
and only one point that following condition h1: h2: distance from an object to a side is directly propor-
. . .: hn + 1 ¼ a1: a2: . . .: an + 1 is correct, where hi tional to the object proximity to the pattern
6 2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making

3
2

1 2-Simplex Prism as a Cognitive Graphics Tool for

Decision-Making, Fig. 5 3-Simplex with learning sam-
2-Simplex Prism as a Cognitive Graphics Tool for ple objects and the object under investigation
Decision-Making, Fig. 4 2-Simplex with learning sam-
ple objects and the object under investigation

where H0 – length of 2-simplex prism preassigned

corresponding to the side. The distance for the by a user and corresponded to the investigation
investigated object is displayed as color perpendic- duration
ular lines to 2-simplex sides. Color of each line
corresponds to the pattern color. An object color is Ti – timestamp of features fixation of object under
mapped with an associated pattern (the nearest investigation for i-th examination
pattern or pattern determined by an expert. Tmin – timestamp of features fixation of object
We use dashed 2-simplexes to illustrate the under investigation for the first examination
accuracy of the pattern recognition on the Fig. 4. Tmax – timestamp of features fixation of object
The edges of 3-simplex, shown in Fig. 5, are under investigation for the last examination.
associated with four patterns. We demonstrate in
Fig. 5 the objects of four patterns (A, B, C, D) and Since 2-simplex prism construction is based on
the investigated object related to pattern A within 2-simplex, then the description of all 2-simplex
3-simplex. Objects related to pattern A are red, to objects is also correct for 2-simplex prism.
pattern B are yellow, to pattern C are brown, and For intelligent geoinformation systems
to pattern D are green. (GIS), the height corresponds to the distance
The results of each of the diagnostic, predictive from initial point to final destination. In this
decisions are shown in the form of points in case the distance between two adjacent
2-simplexes disposed on cross sections of 2-simpleces corresponds to the distance between
2-simplex prism. two points on a map.
Distance from the basis of the prism to i-th Learning trajectory construction via 2-simplex
2-simplex hi0 corresponds to the fixation prism is given in Fig. 6.
moment of object under investigation features, The edges of the 2-simplexes correspond to
and it is calculated based on the following assessments satisfactory (red), good (yellow),
formula: and excellent (green).
We demonstrate learning path in the 2-simplex
T i  T min prism in Fig. 6. Height H0 corresponds to time
hi 0 ¼ H 0  ,
T max  T min interval of learning, Ti (i ϵ {1, 2, 3, 4}).
2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making 7

2-Simplex Prism as a
Cognitive Graphics Tool
for Decision-Making,
Fig. 6 Learning trajectory A
construction via 2-simplex
prism cognitive tool

Software implementation of these models for 2. The reproduction of the material in modified
intelligent system (IS) includes development of form.
corresponded mathematical apparatus for trans- 3. Extraction of new knowledge based on the
formation feature space to pattern space. studied material.
4. Problem-solving, etc.

Using 2-Simplex Prism and 3-Simplex for It should be noticed that the different set of
Dynamic Diagnostic, Forecasting these parameters (a1, a2, a3) can be transformed
Processes and Dynamic Geoinformation in same distances h1, h2, h3 in case when sums of
Systems ai for different sets are equal. So for that and
similar cases, it is necessary to introduce the new
Application in Learning-Testing Systems parameter: a color saturation of point
Until 2015, we used 2-simplex and 3-simplex to corresponded to the sum of ai : a1 + a2 + a3.
make and justify decisions for dynamic processes Example of 2-simplex prism usage for
visualization, modeling, and prediction learning-testing systems is given in Fig. 6.
(Yankovskaya 1997, 2011; Yankovskaya et al. 2-Simplex prism allows to represent dynamics
2003, 2015b; Yankovskaya and Semenov 2012). for ability development of a respondent or a group
In 2015 we started to use 2-simplex prism for of respondents. But it should be noticed that rep-
these purposes in various problem areas. resentation of test results of a big group of respon-
This section describes visualization of testing dents with the usage of 2-simplex prism can be too
knowledge result in learning-testing system with complex and inconvenient.
estimation coefficients usage (Yankovskaya et al.
2016a, b, 2017a, 2018). In learning-testing system Prediction of Students’ Learning Results and
developed by us, respondent, after studying Cognitive Graphics Tools for Its Visualization
selected discipline, should pass MDT. During We represent 3-simplex that visualizes dynamical
solution of this test, respondent actions map process of learning and subdivides the respon-
(RAM) is forming, which determines how well dents into two subgroups with close levels of
the respondent cope with different tasks based on learning abilities in Fig. 7.
the following abilities (skills): One of the main research tasks is a fast creation
of a simple prediction model prototype for a learn-
1. Storage and reproduction of the material in ing process. It allows to solve the following prob-
unmodified form. lems: (1) remove obstacles for solving other tasks
8 2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making

skills to solve problems, and laboratory perfor-

mance skills. A point in the space of these compo-
nents at a particular time instant corresponds to the
current evaluation of the above components combi-
nations. In the process of learning, the point position
can vary depending on the students’ values correc-
tion. Preferably, this position should be within the
tolerance area.
The prediction model has the following inputs:
(1) a history of previous student testing results,
which are obtained by MDT, described before,
and (2) a timestamp of prediction. The output of
the prediction model is a predicted result which
student should get at testing in preassigned
timestamp.
2-Simplex Prism as a Cognitive Graphics Tool for In the current research for a prediction, a sim-
Decision-Making, Fig. 7 3-Simplex for dynamic pro- ple extrapolation polynomial function is used.
cesses cognitive visualization Polynomial degree p is configurable and can be
used for its influence estimation on a prediction
of the current research; (2) get an estimation of an quality. For a polynomial function, the last
influence of learning process trajectory and its p + 1 results of already performed tests by a
prediction visualization given for students on the student for each axis are used. A system of linear
speed of their learning; and (3) integrate a visual- equations is constructed based on them and is
ization library in the Web-based learning platform solved via Gauss method. Confidence region pre-
(Moodle). As a result, the main attention is diction is calculated as delta between predicted
focused on inputs and outputs of this model. and real result for the last performed test.
The prediction model has the following inputs: An example of student learning trajectory visu-
(1) a history of previous student testing results, alization and different prediction models is given
which are obtained by MDT, described before, in Fig. 8. By varying different models in the
and (2) a time stamp of prediction. The output of prediction process, it was found that for the major-
the model is a predicted result which student ity of the respondents, the best prediction is
should get at testing in preassigned time stamp. achieved by using a linear polynomial model
Learning intelligent system (Yankovskaya (the polynomial of the first power). In most cases
et al. 2017b) is a good example of intelligent this simple prediction model shows better quality
dynamic system. Control and prediction of stu- of prediction than any other investigated model
dents’ learning results are the ones of the “hottest” (using quadratic polynomial).
problems in modern learning process. The respon-
dents assess their results within a particular course Decision Support for Diagnostics and
module and explore the prospects for further intel- Correction of Psychosomatic Disorders with
lectual development based on assignment perfor- Usage of 2-Simplex Prism
mance and test results at each stage of learning Diagnosis and intervention of organization stress
cycle. The system implements the learning result with the usage of 2-simplex prism is another inter-
prediction and suggests to students the next devel- esting example of CGT usage (Yankovskaya and
opment directions: (1) research, (2) practical, Yamshanov 2016). The idea of diagnosis is based
(3) teaching, and (4) management activities. on a three-step diagnostics for each stage of orga-
We use 2-simplex prism to visualize and justify nization stress (1, intenseness; 2, adaptation; 3,
students’ learning trajectories. The sides of prism exhaustion) on the base of G. Selye conception.
correspond to the theoretical knowledge evaluation, This scenario is interesting because there are four
2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making 9

2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making, Fig. 8 Influence of the polynomial degree on
the prediction quality

2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making, Fig. 9 Results of diagnostic tests for
organizational stress revealing using 2-simplex prism

patterns used (0 – for absence of organization stage (pattern 2) to the absence of stress
stress) which is more that can be visualized and (pattern 0).
justified in 2-simplex prism. So two 2-simplex The fifth test (T5) reveals the absence of the
prisms can be used: one for the beginning of stress organization.
intervention (Fig. 9a) and another one for the It should be noticed that the cognitive property
ending of intervention (Fig. 9b). of color is used in 2-simplex prism to represent
The first test (T1) reveals a level between the dangerous diagnoses and patterns.
stage of exhaustion and the resistance stage and
prepotency of the stage of exhaustion over the Cognitive Modeling
resistance stage. The second test (T2) reveals that The cognitive modeling of a decision-making in
illness is decreasing from the exhausted stage the intelligent systems is one of the most impor-
(pattern 3) to the resistance stage (pattern 2). The tant directions for creating intelligent systems
third test (T3) reveals that illness is decreased to a (IS) in some priority areas of science researches
level between the resistance stage (pattern 2) and and developing as medicine, psychology, sociol-
alarm stage (pattern 1). The fourth test (T4) reveals ogy, environmental protection, energetics, sys-
prepotency of the alarm stage (pattern 1). tems of transport and telecommunication, control
The second 2-simplex prism (Fig. 5) represents systems, etc. Manipulating some parameters of an
the transformation process from the resistance object under investigation and using cognitive
10 2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making

tools of decision-making and its justifications in The cognitive modeling of decision-making is

IS, we can perform a cognitive modeling based on based on mathematical and computer methods
the different kinds of knowledge representations. with applying n-simplex family that allows opti-
An example of the modeling result on the base mizing a choice of influence of investigated object
of prediction and therapeutic intervention with the in accordance with a dynamical model of param-
use of 2-simplex prism (Yankovskaya and eter change.
Yamshanov 2016) is presented in Fig. 10. After
the first patient examination (T1), the diagnosis is
revealed – stage 3 (exhausted) of organization Using 2-Simplex Prism, 2-Simplex, and
stress – and strategy of recovering is changed. 3-Simplex for Geoinformation Systems
With usage of mathematical model of a patient
recovery process, it is possible to predict progress GIS (Ryumkin and Yankovskaya 2003) are
of patient recovery, which is shown in Fig. 10 as widely applied practically in all spheres of
polyline of a light-blue color. human activity.
After the second examination (T2), a progress There are different scenarios for CGT usage in
of psychology stress recovering is diagnosed – GIS. First common scenario is combination of
organization stress is moved from stage 3 to ordinary reality objects (naturalistic CGT) and
stage 2 (resistance), but a real progress is worse CGT, for example, map with real object and
than the predicted one. At this moment the doctor CGT overlay. Intelligent system of road-climatic
has two different strategies for continuation of zoning of territories (Yankovskaya and
recovering: purple and blue. The doctor uses this Sukhorukov 2017; Yankovskaya and Yamshanov
cognitive representation of different recovering 2014) is good example for this scenario. The free-
strategies. He can choose one which gives the distributed open-street maps with information
best result in the future. At this moment a strategy layer overlay for the presentation of common
associated with a polyline of a purple color is information are used. Information layer presents
more reasonable, because this strategy is applied road regions with borders and some information
to a patient. The model predicts full patient recov- about its. This information is a number of zone
ery until the moment of the next examination (T3). and subzone which are determined for road
region. A CGT for GIS visualization is presented
in Fig. 11. Note that for the mapping of decision-
making results with the usage of CGT, we use
3-simplex for the zone representation and
2-simplex for subzone representation in case
when the number of subzones equals 3. The infor-
mation layer is denoted by number 1. It is a
transparent layer over the map. The thin black
lines separate the different road regions. The dif-
ferent color tones are used for labeling the differ-
ent zones (red color tone is used for zone 2; blue
color tone is used for zone 3). Each color of the
road region in every zone is unique color grada-
tion from zone base color given from color trans-
formation in the hue-saturation-bright palette
(HSB palette). The wide black lines are used to
separate the different zones. Hatching over road
region shows subzone type. Only three hatching
2-Simplex Prism as a Cognitive Graphics Tool for
Decision-Making, Fig. 10 Visualizing of a modeling types are used and only two types from them
result in 2-simplex prism presented in Fig. 11.
2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making 11

2-Simplex Prism as a
Cognitive Graphics Tool
for Decision-Making,
Fig. 11 Visualization tool A
for representation of the
map with zoning results

2-Simplex Prism as a
Cognitive Graphics Tool
for Decision-Making,
Fig. 12 Results of
diagnostic tests for
intelligent geoinformation
systems. L1, L2, L3, L4 –
distances between
geographical points

Another interesting scenario of usage is the The most important advantage for the information
investigation of dependency of some object visualization in 2-simplex prism is the opportunity to
parameters on the base of distance from some analyze in dynamics the object under investigation. It
point. It is also reasonable to use 2-simplex allows users to make decisions, justify them, and
prism for decision-making and its justification in analyze changes of object parameters.
intelligent geoinformation systems (Yankovskaya The new approach to the prediction of stu-
2017). An example of health problem diagnosis dents’ learning results based on MDT and
on the base of distance is given in Fig. 12. 2-simplex prism is examined. Simple prototype
of the prediction model for a learning process is
given. Specificity of confidence region visualiza-
Conclusions and Discussion tion for CGT 2-simplex and 2-simplex prism are
described. The mixed DT (MDT) tools were ver-
Cognitive graphics tool (CGT) is one of the direc- ified on third year students’ test results of special-
tions of artificial intelligence which is reasonable ity “Electrical Power Engineering.” Variation of
to use for any problem solution. CGT described in prediction polinome degree shows that for the
the paper are efficient for decision-making and its majority of students the prediction results
justification for pattern recognition problem in big obtained using linear polynome is better than
number of software systems. that obtained using square polynome. On this
12 2-Simplex Prism as a Cognitive Graphics Tool for Decision-Making

basis the following hypothesis can be formulated: Proceedings, Russian Federation, Novosibirsk,
the second derivative is not constant for a learning pp. 202–205, 2003
Saary, M.J.: Radar plots: a useful way for presenting mul-
process and cannot be used for correct prediction. tivariate health care data. J. Clin. Epidemiol. 61(4),
A lot of intelligent systems constructed on the 311–317 (2008)
base of intelligent instrumental system IMSLOG Wang, B., Feng, X., Chu, K.H.: A novel graphical proce-
(IIS IMSLOG) shows that CGT usage is very dure based on ternary diagram for minimizing refinery
consumption of fresh hydrogen. J. Clean. Prod. 37,
reasonable for big number of problem and inter- 202–210 (2012)
disciplinary areas. Yankovskaya, A.E.: Transformation feature space into pat-
The research was conducted in such universi- tern space on the base of logic-combinatorial methods
ties as Tomsk State University of Architecture and and properties of some geometric objects. In: Pattern
Recognition and Image Analysis, Minsk, pp. 178–181,
Building, Tomsk State University of Control Sys- 1991
tems and Radioelectronics, Tomsk State Univer- Yankovskaya, A.E.: Design of optimal mixed diagnostic
sity, Tomsk Polytechnic University, and (Tomsk test with reference to the problems of evolutionary
State Pedagogical University.) computation. In: Proceedings of the First International
Conference on Evolutionary Computation and Its
Applications (EVCA’96), Moscow, pp. 292–297,
1996
Yankovskaya, A.: Decision-making and decision-
Cross-References justification using cognitive graphics methods based
on the experts of different qualification. In: Russian
▶ Computer Graphics, Video Games, and Gamifi- Academy of Science Bulletin, Theory and Control Sys-
tems, vol. 5, pp. 125–128, 1997
cation Impacting (Re)habilitation, Healthcare, Yankovskaya, A.: Logical Tests and Means of Cognitive
and Inclusive Well-Being Graphics. LAP LAMBERT Academic Publishing,
▶ Overview of Artificial Intelligence Saarbrigge (2011)
▶ Teaching Computer Graphics by Application Yankovskaya, A.: 2-Simplex prism as a cognitive graphic
tool for decision-making and its justification in intelli-
gent dynamic and geoinformation systems. In: Pro-
Acknowledgment The research was funded by RFBR ceedings of 4th International Conference “Computer
grant (Project No. 16-07-0859a). Graphics and Animation,” p. 42, 2017
Yankovskaya, A., Galkin, D.: Cognitive computer based
on n-m multiterminal networks for pattern recognition
in applied intelligent systems. In: Proceedings of Con-
References ference GraphiCon’2009, MAKS Press, Moscow,
pp. 299–300, 2009. ISBN 978-5-317-02975-3
Albu, V.А., Khoroshevskiy, V.F.: COGR – cognitive Yankovskaya, A., Semenov, M.: Computer based learning
graphics system, design, development, application. In: by means of mixed diagnostic tests, threshold function
Russian Academy of Science Bulletin, Technical and fuzzy logic. In: Proceedings of the IASTED Inter-
Cybernetics, pp. 12–20, 1990 (in Russian) national Conference on Human–Computer Interaction,
Axelrod, R.M.: The Structure of Decision: Cognitive Maps Baltimore, pp. 218–225, 2012
of Political Elites. Princeton University Press, Yankovskaya, A.E., Sukhorukov, A.V.: Complex matrix
Princeton (1976) model for data and knowledge representation for
Kobrinskiy, B.A.: Why should we take in account imagi- road-climatic zoning of the territories and the results
nary thinking and intuition in medical expert systems. of its approbation. In: International Conference Infor-
In: Artificial Intelligence – 96, Proceedings of the 5th mation Technology and Nanotechnology, Samara,
National Conference with International Participation, pp. 264–270, 2017
pp. 207–210, 1996 (in Russian) Yankovskaya, A., Yamshanov, A.: Bases of intelligent
Pospelov, D.A.: Cognitive graphics – a window into the system creation of decision-making support on road-
new world. In: Software Products and Systems, climatic zoning. In: Pattern Recognition and Informa-
pp. 4–6, 1992 (in Russian) tion Processing (PRIP’2014): Proceedings of the 12th
Raciborski, R.: Graphical representation of multivariate International Conference, UIIP NASB, Minsk, vol.
data using Chernoff faces. Stata J. 9(3), 374 (2009) 340, pp. 311–315, 28–30 May 2014
Ryumkin, A., Yankovskaya, A.: Intelligent expansion of Yankovskaya, A., Yamshanov, A.: Family of 2-simplex
the geoinformation system. In: The 6th German- cognitive tools and their applications for decision-
Russian Workshop “Pattern Recognition and Image making and its justification. In: Computer Science and
Understanding” OGRW-6-2003, Workshop Information Technology (CS & IT), pp. 63–76, 2016
3D Avatars in Virtual Reality Experience 13

Yankovskaya, A.E, Gedike, A.I., Ametov, R.V.: Intelligent Zhuravlev, Y.I., Gurevitch, I.B.: Pattern recognition and
dynamic system. In: Knowledge-Dialog-Solution image analysis. In: Pospelov, D.A. (ed.) Artificial Intel-
(KDS-2001), Proceedings of International Science – ligence in 3 Books, Book 2: Models and Methods:
Practical Conference, vol. 2, Pub. “Lan,” Saint- Reference Book, p. 149191. Radio and Comm, Mos- A
Petersburg, pp. 645–652, 2001 cow (1990). in Russian
Yankovskaya, A.E., Gedike, A.I., Ametov, R.V., Bleikher,
A.M.: IMSLOG-2002 software tool for supporting
information technologies of test pattern recognition.
In: Pattern Recognition and Image Analysis, vol.
13, no. 4, pp. 650–657, 2003 3D Animation Visualization
Yankovskaya, A., Yamshanov, A., Krivdyuk, N.:
2-Simplex prism – a cognitive tool for decision-making
and its justifications in intelligent dynamic systems. In:
▶ Tabletop Storytelling
Book of Abstracts of the 17th All-Russian Conference
with International Participation: Mathematical
Methods for Pattern Recognition, Svetlogorsk, p. 83,
2015a
Yankovskaya, A., Dementyev, Y., Yamshanov, A.: Appli- 3D Avatars
cation of learning and testing intelligent system with
cognitive component based on mixed diagnostics tests. ▶ 3D Avatars in Virtual Reality Experience
In: Procedia – Social and Behavioral Sciences, Tomsk,
vol. 206, pp. 254–261, 2015b
Yankovskaya, A., Dementyev, Y., Yamshanov, A.,
Lyapunov, D.: Prediction of students’ learning results
with usage of mixed diagnostic tests and 2-simplex
prism. In: Intelligent Data Processing: Theory and 3D Avatars in Virtual Reality
Applications: Book of Abstracts of the 11th Interna-
tional Conference (Moscow/Barcelona), Torus Press, Experience
Moscow, vol. 238 pp. 44–45, 2016a
Yankovskaya, A., Dementyev, Y., Lyapunov, D., Manoj Dawarwadikar1 and Madhusudan Rao2
Yamshanov, A.: Intelligent information technology 1
SP Jain School of Global Management, Sydney,
in education. In: Proceedings of the 2016 Confer-
ence on Information Technologies in Science, Bangalore, India
2
Management, Social Sphere and Medicine NTT Data Services Pvt. Ltd, Bangalore, India
(ITSMSSM), Atlantis Press, Tomsk, vol.
51, pp. 17–21, 2016b
Yankovskaya, A.E., Dementev, Y.N., Lyapunov, D.Y.,
Yamshanov, A.V.: Learning outcomes evaluation Synonyms
based on mixed diagnostic tests and cognitive graphic
tools. In: Proceedings of the XVIIth International Con- 3D Avatars; Digital humans; Immersive technol-
ference on Linguistic and Cultural Studies: Traditions ogies; Virtual reality
and Innovations, Advances in Intelligent Systems and
Computing (LKTI 2017), Tomsk, vol. 677, pp. 81–90,
11–13 Oct 2017a
Yankovskaya, A.E., Shelupanov, A.A., Shurygin, Y.A., Definition
Dementiev, Y.N., Yamshanov, A.V., Lyapunov, D.
Y.: Intelligent learning and testing predictive system
with cognitive component. In: Open Semantic Tech- 3D Avatar is a representation of human users in a
nologies for Intelligent Systems, pp. 199–204, Virtual reality environment. It represents the body
2017b of the user, interaction of the user with the sur-
Yankovskaya, A., Dementyev, Y., Yamshanov, A., rounding environment, and other users in the vir-
Lyapunov, D.: Assessing student learning outcomes
using mixed diagnostic tests and cognitive graphic tual world to enhance the realism of the overall
tools. In: Open Semantic Technologies for Intelligent experience. It facilitates collaboration among var-
Systems: Proceedings of International Conference, ious users and the virtual environment in multi-
Minsk, vol. 2, pp. 351–354, 15–17 Feb 2018 (ISSN player games, remote collaboration meetings,
2415–7740. http://proc.ostis.net)
Zenkin, A.A.: Cognitive Computer Graphics. Nauka, Mos- virtual events, immersive learning classes, and
cow (1991). in Russian many such applications.
14 3D Avatars in Virtual Reality Experience

Introduction view of the physical world. It is achieved through

a handheld device such as a smartphone or spe-
Remote collaboration and remote interaction with cialized hardware such as Mixed Reality Glasses.
peers are rising steadily due to their apparent AR or MR enhance the surrounding reality with
advantages of cost-saving, convenience, and digital content but do not alter the physical reality.
scalability. Immersive technology such as virtual The interactions in AR remain restricted mainly to
reality transforms the remote collaboration expe- the digital overlay on the surroundings; however,
riences through its unique features such as pres- in VR, the interaction extends to the environment
ence and realistic interactions. and other users. This difference creates a need to
Virtual reality, typically called VR, is a simu- represent the users in a simulated VR environ-
lation of a new environment with visuals and ment, and 3D Avatars play a significant role in
interactions created through computing devices. this interaction.
It can be achieved through specialized hardware Virtual reality has several applications
such as a head-mounted device (HMD) and haptic depending on the need for interactivity, immer-
controllers. Vision and spatial audio rendering by sion, and presence. Sectors such as gaming, edu-
HMD and haptic feedback through controllers cation, manufacturing, healthcare, military, and
make the experiences feel real for users. Virtual real estate have seen a varied degree of adoption
reality replaces physical reality with computer- of VR for applications such as remote collabora-
generated content. It achieves realism through tions, live events, and immersive learning (Rao
information visualization, guiding the user, and and Dawarwadikar 2020).
making the environment interactive through Many of these experiences are individual expe-
mediums such as gestures, movements, and riences where the simulation and interaction limit
audio (Porter and Heppelmann 2017). to only virtual environments. Examples of these
Immersion, presence, and interactivity are the experiences are simulation games, movie- view-
characteristics of virtual reality which make it ing experiences, and immersive browsing appli-
suitable for remote collaboration with other users cations. However, in several experiences, the
in a simulated world. Immersion in a virtual envi- interaction extends to other users who share the
ronment is achieved by designing the surround- same virtual environment. Examples of these
ings and content around the user. Content formats experiences are multiplayer games, social net-
such as 360-degree photos, videos, and computer- work applications, live virtual event applications,
generated graphical environments make immer- remote learning, and professional collaboration
sion possible and productive. Presence in a virtual applications (Bredikhina et al. 2020).
environment for the user depends on the represen-
tation of the user and other users in the same Need for 3D Avatars in Virtual Reality
virtual environment. 3D Avatars are one of the Avatar, in simple terms, is the representation of a
representation methods to create a feeling of pres- player in a video game or artificially created vir-
ence for all the users. Interaction with the envi- tual world. Technically, it could be as simple as a
ronment and other users is the most effective way 2D shape or as complex as a realistic 3D model.
to make the entire experience productive and An Avatar enhances the gaming experience by the
enjoyable. Based on the capabilities of the VR player’s ability to identify themselves with their
HMD, the experiences could be noninteractive to avatar’s visual and behavioral characteristics.
highly interactive. One such feature is degree-of- This perception is significant in virtual reality
freedom, allowing users to move in the virtual applications where interaction with the environ-
environment and change perspectives differently ment and other users plays a crucial role (Wauck
(Pangilinan et al. 2019). et al. 2018). Research shows the psychological
Other immersive technologies such as Aug- significance of 3D Avatars in players treating
mented Reality (AR) and Mixed Reality their Avatar as an extended body in the virtual
(MR) superimpose digital content on a user’s world. However, it depends on the experience
3D Avatars in Virtual Reality Experience 15

and the player’s attitude toward the experience. movement, especially their head (tilting, nod-
Environmental and interactive feedback from ding), body (changing places), and hands
other users also influences the immersive experi- (waving, clapping). Oculus Venues or Oculus A
ence of users (Rosa et al. 2018). TV provides similar experiences to view an
3D Avatars can be created in multiple ways, immersive stage performance while interacting
such as 3D laser scanning, photogrammetry, and with others. PokerStars VR and Racket Fury
machine vision. With the help of 3D scanners or allow players to play Poker and Table tennis,
appropriate software tools, realistic 3D models respectively, with other online players. The inter-
can be created from as simple as a single photo- action stays effective due to the use of synthetic
graph to as complex as a full-body scan (Berdic Avatars. Social VR gatherings and events are
et al. 2017; Jo et al. 2017). There are broad cate- available in AltspaceVR, VRChat, and RecRoom,
gories of how Avatars are created and presented. where social groups can organize training, semi-
There could be entirely imaginative Avatars that nars, discussions, and many more such activities
do not resemble the player. Cartoonish-looking (Liu 2020).
Avatars can partially resemble a player. However,
realistic Avatars could resemble the appearance of What Makes an Avatar More Effective?
the player to a great degree (Čeliković et al. 2018). Presence – Avatar invokes body ownership
This broad range of realism of Avatar is used in among users, and research shows that presence
various applications based on need. The players’ is felt more effectively when players use a
perception of themselves enhances their ability to cartoon-like virtual Avatar that mimics players’
align themselves with their Avatar through outfits. Other possible ways to mimic player
customizations (Wauck et al. 2018). Realistic behavior are realistic Avatar created from photo-
Avatars invoke the illusion of virtual body own- grammetry of players and cartoon-like Avatars
ership characterized by acceptance control and sketched by the artists (Jo et al. 2017).
change in behavior of the player in the virtual Familiarity – Avatars feel most realistic to
world. The uncanny valley (i.e., the relationship players when they are created with appropriate
between Avatar and the emotional response) is shapes and sizes mimicking the reality. Virtual
characterized by humanness, eeriness, and attrac- body parts such as hands play a crucial role in
tiveness. The realistic Avatars help in realizing VR experiences, and the appropriate size of hands
self-presence as well as the copresence of other and other objects play a critical role in creating
players. Avatars help build rapport between realism (Ogawa et al. 2018). In some VR applica-
players sharing the same virtual environment tions, the entire body of the Avatar is customiz-
(Latoschik et al. 2017). able, whereas, in some cases, only a partial body is
The systems to create configurable and cus- visible and customizable. Figure 1 shows exam-
tomizable Avatars are technically complex and ples of 3D Avatar customizations available in
require a certain level of expertise for users to some VR applications. Table 1 summarizes the
create them (Čeliković et al. 2018). The creation customization options available for physical
of an Avatar requires a lot of time and effort on the appearances in popular VR applications.
artistic and technical sides. The effort is assessed Expressions – Mimicking players’ facial
based on the need of the application. Research expressions on their Avatar strengthens the feeling
shows that there is little to no effect on player’s of presence and interaction as nonverbal commu-
performance in a virtual environment and player’s nication plays a significant role in human commu-
subjective experience, based on the quality of nication. Latest developments in hardware
their Avatar in the experience (Wauck et al. 2018). sensors and technology are accelerating the
Virtual reality entertainment applications such enhancement of Avatars through facial expression
as BigScreen allow multiple people to watch a mapping in real time (Suzuki et al. 2017). While
movie in the virtual cinema hall. The interaction mapping players with Avatar’s enfacement
among the viewers happens through their Avatar (similarity of player’s face with Avatar’s face)
16 3D Avatars in Virtual Reality Experience

3D Avatars in Virtual Reality Experience, Fig. 1 Examples of 3D Avatar customization in VR applications

3D Avatars in Virtual Reality Experience, Table 1 Comparison of VR applications for 3D Avatar customizations
Name of VR
application Avatar physical appearance customizations available
Body Head Face Clothing and accessories
AltSpace vr Body shape, skin Hair, hair dye, hat, hat Eyebrows, eyes, Top, top accent, jacket, jacket
tone, and nail accent, and jaw shape mouth, nose, facial accent, and bottom
polish hair, and eyewear
Spatial VR Body type Shirt color
(gender), skin
tone
Engage VR Body type Hair Eyes, nose, and Tops, bottoms, shoes, and
(gender), skin mouth glasses
tone, and body
weight
Oculus Body, skin tone Hair Face shape, face Eyewear, headwear, bindi, ear
Venues markings, and face piercing, and nose piercing
lines
RecRoom Skin color, body Face, hair style, hair Eyes, mouth, ears Clothing, shoulder,
colors, beard style, necklaces, belts, hats, glasses,
and beard colors earrings, and hand gloves
BigScreen Skin color, body Hair, hair color, Lips, eye shape, and Glasses, shirt, shirt color, and
VR moustache, sideburns, eye color hat
beard, and brows
3D Avatars in Virtual Reality Experience 17

and lip sync while talking enhance the embodi- cases. Utilitarian applications of virtual reality
ment significantly. Real-time communication with include remote maintenance support, virtual
Avatars through speech using the onboard micro- tours of factories, remote design/product reviews, A
phone of VR hardware also enriches interactions and specialized applications such as remotely
among players (Gonzalez-Franco et al. 2020). assisted surgeries. The feeling of presence
Control – Avatar’s sense of ownership is sig- expressed through 3D Avatars enhances the qual-
nificantly influenced by self-control, i.e., control- ity of the collaboration experiences, yet there are
ling Avatar’s movements and actions through the several areas of improvement. Generating realistic
player’s body movements. The player’s point of Avatars of multiple users, representing them in the
view in a virtual environment (first-person view virtual environment to identify them uniquely, and
vs. third-person view) determines the realism of navigating the virtual environments without over-
the Avatar for players interacting in groups. These lapping multiple Avatars at the same place are
factors often take precedence over the appearance some of the challenges faced by current platforms
of the Avatar alone (Fribourg et al. 2020). such as VRChat, SpatialVR, and GlueVR (Liu
2020). Creating 3D Avatars professionally by
scanning the user in 360 degrees surely enhances
Applications of 3D Avatars in Virtual the usability and performance of professional col-
Reality Experiences laboration applications (Kolkmeier et al. 2018)
(Fig. 2).
Multiplayer Games
Games have been the most popular category of Virtual Events
virtual reality applications since the launch of Virtual events are gaining popularity due to their
commercial VR headsets. In multiplayer games, cost and convenience benefits. Though VR hard-
interaction with the environment and interaction ware availability is the major hindrance in orga-
with other players are two distinct parts. For nizing large-scale virtual events such as
interacting with the environment, the embodiment conferences, stage shows, and live performances,
and presence of hands play a pivotal role. Avatars the replacement of physical events into virtual
do not influence perception much. However, in events is on the rise. 3D Avatars and the feeling
multiplayer games interacting with other players of presence in the virtual event venue provide an
involves using Avatars to create the feeling of opportunity for participants to interact and net-
presence. work with fellow participants in events such as
In role-playing games (RPG) such as rescue the entrepreneurship and innovation summit orga-
games, puzzle-solving games, or combat games, nized on the VirBELA VR platform (Jauhiainen
the realism of other players’ Avatars enhances the 2021). The interactions among groups can range
feeling of presence, and own Avatar influences the from attending a seminar or workshop to enjoying
players’ social reputation. Often players choose the after-event parties. Based on the situations, 3D
their ideal self over their actual self while choos- Avatars and their realism play different roles in
ing customizations in the Avatar such as appear- enhancing interactions. Additional factors such as
ance, skin tones, and clothing (Wauck et al. 2018). virtual environment design and network speed for
Most fitness games and esports simulations, such live experiences play a more significant role in
as virtual boxing games, do not require full-body overall user experience in such social events
Avatars of the players as they project the first- (Kreskowski et al. 2020).
person view to the players. The only Avatar of
the opponent is considered relevant in such cases. Immersive Learning
Remote immersive learning has gained traction in
Professional Remote Collaboration the last few years, fueled by the availability of
Remote collaboration at the professional level has standalone VR hardware such as Google Card-
increased over the last few years with various use board and Oculus Quest. VR solutions and apps
18 3D Avatars in Virtual Reality Experience

3D Avatars in Virtual Reality Experience, Fig. 2 Virtual reality group meeting with 3D Avatars of the users (App: Big
Screen VR on Oculus Quest)

intercepted learnings at various levels, such as today technologically, we are still far away from
professional, personal, and academic, and various the realism that matches the cinematic quality of
age groups such as schools, colleges, and corpo- characters. The costs and effort involved in
rate training. Professional skills such as technical improving the realism and features of a 3D Avatar
skills and design skills can be taught in an are disproportionate to the benefits at the con-
immersive virtual environment facilitated by vir- sumer levels; hence, most VR applications still
tual instructors and the presence of other class- use synthetic or cartoonish Avatars, which are
mates remotely. During such interactions, Avatars widely accepted in current applications. So far,
of participants play a pivotal role in engaging in creating a realistic Avatar from a user image is
the learning experience. Hand movement, head the most popular and cost-effective technique.
movement, and voice interactions make the The alternate to 3D Avatars is holographic pro-
group- learning experience engaging. Personal jections, which can simulate the humans’ appear-
skills such as public speaking are possible on a ance and their realistic size and expressions.
virtual stage with a simulated audience created Technology for holographic projections and
through 3D Avatars. Such virtual learning experi- superimposing real human wrappers to create
ences enhance motivation as well as the under- realistic cinematic characters is developing rap-
standing of the learners (Gomes de Siqueira idly and will be incorporated into virtual worlds
et al. 2021). soon (Balamurugan 2017).

Limitations of 3D Avatars Conclusion

Though 3D Avatars are an excellent way of 3D Avatars play an essential role in virtual reality
representing humans in the virtual world, and experiences while interacting with other users and
there is a varying degree of realism available occasionally with the environment. Though the
3D Avatars in Virtual Reality Experience 19

technology to create realistic 3D Avatars is IEEE Trans. Vis. Comput. Graph. 26,
not fully mature and cost-effective, some 2062–2072 (2020). https://doi.org/10.1109/TVCG.
2020.2973077
workarounds are widely adopted across VR solu- Gomes de Siqueira, A., Feijóo-García, P.G., Stuart, J., Lok, A
tions. Depending on the experience, full-size body B.: Toward facilitating team formation and communi-
Avatars or only specific body parts such as hands cation through avatar based interaction in desktop-
or heads are used in VR experiences. VR users’ based immersive virtual environments. Front Virtual
Real. 2, 1–18 (2021). https://doi.org/10.3389/frvir.
presence and immersive experience largely 2021.647801
depend on their perception of themselves in vir- Gonzalez-Franco, M., Steed, A., Hoogendyk, S., Ofek, E.:
tual Avatar and their social reputation, influencing Using facial animation to increase the enfacement illu-
their customization choices. As more and more sion and avatar self-identification. IEEE Trans. Vis.
Comput. Graph. 26, 2023–2029 (2020). https://doi.
social experiences and new-age VR hardware org/10.1109/TVCG.2020.2973075
penetration are increasing, the need to develop Jauhiainen, J.S.: Entrepreneurship and innovation events
3D Avatars is growing over the last few years. during the COVID-19 pandemic: the user preferences
However, alternates such as virtual humans pro- of VirBELA virtual 3D platform at the SHIFT event
organized in Finland. Sustain. 13 (2021). https://doi.
jected through holographic projections will org/10.3390/su13073802
replace most synthetic 3D Avatars in the long run. Jo, D., Kim, K., Welch, G.F., et al.: The impact of avatar-
Advances in other technologies such as artifi- owner visual similarity on body ownership in
cial intelligence, networks (5G/6G), and pro- immersive virtual reality. Proc ACM Symp Virtual
Real Softw Technol VRST Part, F1319 (2017).
cessing hardware (GPUs) will fast track adoption https://doi.org/10.1145/3139131.3141214
of realistic Avatars in VR experiences. Kolkmeier, J., Reidsma, D., Harmsen, E., et al.: With a
little help from a holographic friend: the
OpenIMPRESS mixed reality telepresence toolkit for
remote collaboration systems. Proc ACM Symp Virtual
Cross-References Real Softw Technol VRST. (2018). https://doi.org/10.
1145/3281505.3281542
▶ Immersive Visualizations Using Augmented Kreskowski, A., Beck, S., Froehlich, B.: Output-sensitive
Reality and Virtual Reality avatar representations for immersive telepresence.
IEEE Trans. Vis. Comput. Graph. X, 1–13 (2020).
▶ Player-Avatar Link: Interdisciplinary Embodi- https://doi.org/10.1109/TVCG.2020.3037360
ment Perspectives Latoschik, M.E., Roth, D., Gall, D., et al.: The effect of
avatar realism in immersive social virtual realities. Proc
ACM Symp Virtual Real Softw Technol VRST Part,
F1319 (2017). https://doi.org/10.1145/3139131.
References 3139156
Liu, Q. Contextual Group Walkthrough: Social VR Plat-
Balamurugan, C.R.: Hologram based three dimensional form Comparison and Evaluation (2020)
projection. 6th Int Conf Res Trends Eng Appl Sci Ogawa, N., Narumi, T., Hirose, M.: Object size perception
Manag, 763–768 (2017) in immersive virtual reality: avatar realism affects the
Berdic, N., Dragan, D., Mihic, S., Anisic, Z.: Creation and way we perceive. In: 25th IEEE Conf Virtual Real 3D
usage of 3D full body avatars. Ann Fac Eng User Interfaces, VR 2018 – Proc, pp. 647–648 (2018).
Hunedoara. 15, 29–34 (2017) https://doi.org/10.1109/VR.2018.8446318
Bredikhina, L., Kameoka, T., Shimbo, S., Shirai, A.: Ava- Pangilinan, E., Lukas, S., Mohan, V.: Creating Augmented
tar driven VR society trends in Japan. In: Proceedings – and Virtual Realities: Theory and Practice for Next-
2020 IEEE Conference on Virtual Reality and 3D User Generation Spatial Computing, First. O’Reilly Media,
Interfaces, VRW 2020, pp. 497–503 (2020) Inc (2019)
Čeliković, D., Batalo, B., Radisavljević, D., Dragan, D., Porter and Heppelmann: A Manager’s guide to augmented
Anišić, Z.: 3D avatar platform — a unique configurator reality. Harv. Bus. Rev., 85 (2017)
for 3D figurine customization. In Proceedings of the Rao, M., Dawarwadikar, M.: Immersive visualizations
8th International Conference on Mass Customization using augmented reality and virtual reality. In: Ency-
and Personalization in Central Europe MCP-CE clopedia of Computer Graphics and Games. Springer
pp. 19–21. (2018) International Publishing, Cham (2020)
Fribourg, R., Argelaguet, F., Lécuyer, A., Hoyet, L.: Avatar Rosa, N., Hürst, W., Veltkamp, R., Werkhoven, P.: Player-
and sense of embodiment: studying the relative prefer- avatar link: interdisciplinary embodiment perspectives.
ence between appearance, control and point of view. Encyclopedia of Computer Graphics and Games,
20 3D Board Game

1–6 (2018). https://doi.org/10.1007/978-3-319-08234- from stones to mountains, bricks to monu-

9_110-1 ments. They can include every 3D object
Suzuki, K., Nakamura, F., Otsuka, J., et al.: Recognition
and mapping of facial expressions to avatar by embed- that can go into a game.
ded photo reflective sensors in head mounted display.
Proc – IEEE Virtual Real, 177–185 (2017). https://doi.
org/10.1109/VR.2017.7892245 Introduction
Wauck, H., Lucas, G., Shapiro, A., et al.: Analyzing the
effect of avatar self-similarity on men and women in a
search and rescue game. Conf Hum Factors Comput A large spectrum of domains like video game
Syst – Proc 2018-April, 1–12 (2018). https://doi.org/ production, virtual reality simulations, and aug-
10.1145/3173574.3174059 mented reality applications make intensive use of
3D-generated assets. There are many graphics
suites that already offer a large set of tools and
functionalities to manage the creation of such
3D Board Game content, although they are usually characterized
by a steep learning curve.
▶ Augmented Reality Ludo Board Game with In games, the use of photogrammetry can
Q-Learning on Handheld enable the developer to build 3D objects that can
be developed to look close to the original. Game
developers have already adopted the method of
photogrammetry to develop game objects and this
3D Game Asset Generation of entry highlights its use to accurately recreate his-
Historical Architecture torical representations in-game.
Through Photogrammetry

Chaitanya Solanki and Deepak John Mathew Resources Required for

Indian Institute of Technology, Hyderabad, India Photogrammetry

As photogrammetry uses images as input data to

Synonyms generate these models, hardware to record images
like a handheld camera or a phone camera is
Asset creation; Cultural heritage; Digital preser- mandatory to the process. Other requirements to
vation; Game assets; Image-based modeling generate models from photos are the image
stitching software like Trimble Inpho, Reality
Capture, Pix4D, AliceVision Meshroom, Agisoft
Definition Photoscan, and more. A few of these software, for
example, Reality Capture, can take photos as well
Photogrammetry is the method of generating as videos as input data. Recently, software like
accurate photorealistic 3D models through orga- Trnio and Sony 3D Creator allows the user to scan
nized assortments of photographs. Typically, pho- and produce 3D models with a smartphone
togrammetry is associated with the making of as well.
maps from aerial photographs but when the same After the model is generated, modeling soft-
system of algorithms are applied to documenting a ware like Blender can be used to refine it by
3D object, it can be used to create photo accurate editing out any extra unwanted meshes or by
3D models of any building, object, or landscape making up for incomplete data. Following that,
desired. mesh manipulation software like ZBrush can be
3D assets in game-building refer to the used to reduce the mesh count of the 3D object so
three-dimensional models that the game that it becomes functionally lighter while
employs as objects; the assets can range retaining its visual aesthetics and intricacies.
3D Game Asset Generation of Historical Architecture Through Photogrammetry 21

Digitizing Cultural Heritage Through digitally using the available database online and
Photogrammetry some panoramic image-based documentation.
Their research exploits the potential of multi- A
In a study by Yilmaz and team (2007), the image reconstruction to model the destroyed tem-
researchers reported on a two-story building that ple of Bel, one of the heritage monuments of Syria
had burnt down twice and the conservation (Wahbeh et al. 2016), see Fig. 1a. This is also in
office of Turkey had needed to reconstruct the line with the image-based modeling system pre-
building through a restoration project. There sented by Snavely et al. (2006) in their research,
happened to be no documentation available in where they were able to interactively browse and
the form of drawings or measurements as a refer- explore large unstructured collections of photo-
ence to go ahead. Photogrammetry was then used graphs of a scene using a novel 3D interface. Their
to document the building holistically and image-based modeling system could then produce
re-photographed after the second fire to note the a 3D model rendered solely from these collections
changes in the physicality of the structure. After of unstructured images, mainly retrieved from the
processing the images in the photo-modeling soft- Internet (Snavely et al. 2006), shown in Fig. 1b.
ware, architectural drawings could be obtained
through the 3D model produced. With the help
of the drawings, architects were able to estimate Using Cultural Heritage In-Game
the three-dimensional measurements of the origi-
nal building and with that, they were able to A famous recent example of preserving a cultur-
restore the building back to its original form ally important monument through digital docu-
(Yilmaz et al. 2007). This achievement of digitally mentation in a game is that of the burning of the
countering architectural wear and tear can help Notre Dame cathedral in Paris. Built-in a gothic
games in employing photogrammetry to complete French architectural style, the construction was
or re-create parts of worn down monuments into first started in the twelfth century and was not
their original pristine condition for enhanced finished until 1345. In 2019, the cathedral became
immersion. a victim of a fire breakout that damaged a signif-
Wahbeh et al. (2016) in their study go as far as icant part of its architecture. However, between
to show that historical architecture, if well photo- 2010 and 2014, Ubisoft, a company that makes
graphed even by tourists, can be recreated digital games, had intensively mapped the entire

3D Game Asset Generation of Historical Architecture Wahbeh et al. 2016); (b) Photogrammetric modeling of a
Through Photogrammetry, Fig. 1 (a) Photogrammet- heritage building using large data sets of unorganized
ric modeling of the “Bel” Temple in Syria. (Source: images. (Source: Snavely et al. 2006)
22 3D Game Asset Generation of Historical Architecture Through Photogrammetry

monument in an effort to recreate it in their game objects are not lost due to their complexities as
“Assassin’s Creed Unity” (Elbaz et al. 2020). This photogrammetry offers a more honest recreation
indicated positively that historical monuments when compared to 3D modeling. It has also
could play a huge role in level design in games. shown to help experts recreate old monuments
This is in line with Statham’s (2020) finding even when they have been the subject of partial
that models based on image-based modeling can wear and tear.
be used in digital games as assets. The study also The methods of photogrammetry can also be
revealed how many of the major commercial applied to creating a true-to-reality environment
games had also started employing photogramme- around the monuments. The landscapes can be
try at a large scale in an effort to render the game scanned thoroughly by imaging the surrounding
environment as close to real-life as possible objects like rocks, pillars, trees, etc., and they can
(Statham 2020) (Fig. 2). then be rendered into photogrammetric models
(Davis et al. 2017; Statham 2020). This can
increase the authenticity of the digital recreation
Conclusion and Discussion many folds and greatly help with level and envi-
ronment design in-game as currently there is
The photogrammetric digitization of cultural her- observed to be a lack of diverse environmental
itage is a noninvasive method to preserve and assets available that are representative of varying
archive historical monuments and sites. Although geographies.
photogrammetry has been adopted by the gaming It is also pertinent to observe and follow all the
industry to provide for realistic landscapes and local and national laws that the structure falls
objects, it can also be used to authentically recre- under. In many countries, flying a camera
ate historical sites and landscapes. Using this mounted drone near a culturally important monu-
method also ensures that the intricacies of art ment is not allowed. The process of

3D Game Asset Generation of Historical Architecture Unity. (Source: https://news.ubisoft.com/en-us/article/

Through Photogrammetry, Fig. 2 3D Modeling of the 2Hh4JLkJ1GJIMEg0lk3Lfy)
cathedral of Notre Dame in the game Assassin’s Creed
3D Modelling Through Photogrammetry in Cultural Heritage 23

documentation and the following usage of the

digital version of the monuments in-game can be 3D Modelling Through
prone copyright infringements and other viola- Photogrammetry in Cultural A
tions and should be taken care of accordingly. Heritage

Vlasios Kasapakis1, Damianos Gavalas2 and

Cross-References Elena Dzardanova2
1
Department of Cultural Technology and
▶ 3D Modelling Through Photogrammetry in Communication, University of the Aegean,
Cultural Heritage Mytilene, Greece
2
Department of Product and Systems Design
Engineering, University of the Aegean,
References Ermoupoli, Greece

Davis, A., Belton, D., Helmholz, P., Bourke, P., McDonald,

J.: Pilbara rock art: laser scanning, photogrammetry
and 3D photographic reconstruction as heritage man-
Synonyms
agement tools. Herit. Sci. 5(1), 25 (2017)
Elbaz, N., Kamel, S., Abdelmohsen, S.: Heritage building Photogrammetry; Terrestrial Photogrammetry
information modelling: towards a new era of interoper-
ability. In: Architecture and Urbanism: A Smart Out-
look, pp. 231–239. Springer, Cham (2020)
Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: Definitions
exploring photo collections in 3D. In: ACM Siggraph
2006 Papers, pp. 835–846 (2006) Photogrammetry is a technique for estimating
Statham, N.: Use of photogrammetry in video games: a
historical overview. Games Cult. 15(3),
the exact position of surface points of an object
289–307 (2020) by using multiple photographs. Aerial photo-
Supporting Notre-Dame de Paris (2017, April 17), grammetry is based on the acquisition of photo-
Retrieved February 18, 2021, from https://news. graphs of a certain area from the sky, commonly
ubisoft.com/en-us/article/2Hh4JLkJ1GJIMEg0lk3Lfy/
supporting-notredame-de-paris
by placing cameras on a plane, drone, or even a
Wahbeh, W., Nebiker, S., Fangi, G.: Combining public satellite, to create a topographical map or terrain
domain and professional panoramic imagery for the model. Terrestrial photogrammetry aims at taking
accurate and dense 3d reconstruction of the destroyed 3D measurements of an object, using photographs
bel Temple in Palmyra. ISPRS Ann. Photogramm.
Remote Sens. Spat. Inf. Sci. 3(5) (2016)
of that object taken from a camera positioned
Yilmaz, H.M., Yakar, M., Gulec, S.A., Dulgerler, O.N.: on the surface of the earth. Terrestrial photogram-
Importance of digital close-range photogrammetry in metry is also referred to as close-range photo-
the documentation of cultural heritage. J. Cult. Herit. grammetry when the photographs of the object
8(4), 428–433 (2007)
are taken at a close range.

Introduction
3D Game Engines
Photogrammetry is a manual process which
▶ Game Engine requires in-depth understanding since it involves
several aspects; for instance, the focal length
and position of the camera when taking each
photograph will impact the quality of the end
3D Interaction result. However, recent technological advance-
ments (increase of computational power, release
▶ Virtual Hand Metaphor in Virtual Reality of affordable, yet powerful, digital cameras, etc.),
24 3D Modelling Through Photogrammetry in Cultural Heritage

along with the availability of free, user-friendly tailored according to physical dimensions; thus,
software, have simplified the process of photo- monuments and archaeological sites are digitized
grammetry. Combined with high-quality compan- with the use of topographic techniques and aerial
ion software, digital close-range photogrammetry photogrammetry, whereas close-range photo-
is heavily used for the 3D digitization of cultural grammetry is a better-suited, cost-effective 3D
heritage. digitization solution for smaller-scale objects
The most widespread photogrammetry method (Pavlidis et al. 2007).
for creating 3D models of real objects is based
on structure-from-motion (SfM). SfM involves
the provision of overlapping images of an object, Using Photogrammetry to 3D Digitize
captured from multiple viewpoints (see Fig. 1). Cultural Heritage
Important data, such as the camera’s position and
orientation, are automatically calculated by spe- Several studies document best practices and
cialized software, which also facilitates the 3D demonstrate the advantages of employing either
model creation and texturing process, extracting close-range or aerial photogrammetry for the 3D
information directly from the set of overlapping digitization of cultural heritage. Some of these
images (Westoby et al. 2012). advantages are high accuracy of the produced
3D models and improvement of the safety
factor during the digitization of hazardous or inac-
3D Digitization of Cultural Heritage cessible areas (Remondino et al. 2005; Yilmaz
et al. 2007; Fassi et al. 2013).
The 3D digitization of cultural heritage is a com- Santagati et al. (2013) provide an example
mon practice for the generation of 3D models used of close-range photogrammetry in cultural heri-
for exhibition, conservation, and protection, 3D tage documentation, describing the 3D digitiza-
printing-based replication, dissemination through tion process of a small chapel using Autodesk
web and mobile application channels, digital res- 123D Catch (recently rebranded to Recap https://
toration of damaged parts, and monitoring of www.autodesk.com/products/recap/), a wide-
overtime alterations (e.g., when cultural assets spread, cloud-based, free photogrammetry soft-
are exposed to open environment) (Pieraccini ware (see Fig. 2a and b ). The study revealed
et al. 2001). The 3D digitization process is that close-range photogrammetry requires short

3D Modelling Through
Photogrammetry in
Cultural Heritage,
Fig. 1 Multiple
overlapping photos of an
object acquired for SfM
photogrammetry
3D Modelling Through Photogrammetry in Cultural Heritage 25

3D Modelling Through Photogrammetry in Cultural Santagati et al. 2013); (c) the Clifden Castle in Ireland;
Heritage, Fig. 2 (a) Chapel; (b) the chapel’s 3D model (d) the Clifden Castle 3D model created using aerial pho-
created using close-range photogrammetry. (Source: togrammetry. (Copyright: Pix4D - Measure from Image)

image processing times while offering high products for structured photo datasets (Santagati
accuracy and quality even when nonprofessional et al. 2013), and the holes found on 3D objects
cameras are used. Moreover, the metric accuracy when the physical object is not properly photo-
for small objects (e.g., statures) is in the order of graphed (Remondino et al. 2005).
mm, while for medium to large objects and archi-
tectural buildings, it is in the order of cm.
On the other hand, aerial photogrammetry Conclusion
can be used to accurately digitize monu-
ments (Grussenmeyer et al. 2008). An example The 3D digitization of cultural heritage may
is shown in Fig. 2c and d, where a castle’s 3D be carried out through the manual creation of
model has been produced using photographs 3D models, by analyzing architectural plans, or
taken with a camera mounted on a drone, there- via automated methods, such as laser scanning
after fed to a popular aerial photogrammetry soft- and photogrammetry. Even though there is an
ware, Pix4Dmodel https://pix4d.com/product/ ongoing debate regarding comparative advan-
pix4dmodel/. tages of photogrammetry to laser scanning and
Even if photogrammetry appears suitable for vice versa, for 3D digitization in cultural heri-
3D cultural heritage digitization, it does not come tage, complementary application of methods is
without limitations. The most common ones are considered as the most appropriate solution for
the long processing times to attain high accuracy digitization projects (Grussenmeyer
(Fassi et al. 2013), the requirement of software et al. 2008).
26 3D Object Manipulation

References
3D Printing, History of
Fassi, F., Fregonese, L., Ackermann, S., De Troia, V.: Com-
parison between laser scanning and automated 3D
Emily Peed1 and Newton Lee1,2
modelling techniques to reconstruct complex and exten- 1
sive cultural heritage areas. Int. Arch. Photogramm. Institute for Education, Research, and
Remote. Sens. Spat. Inf. Sci. 5, W1 (2013) Scholarships, Los Angeles, CA, USA
Grussenmeyer, P., Landes, T., Voegtle, T., Ringle, K.: 2
Vincennes University, Vincennes, IN, USA
Comparison methods of terrestrial laser scanning,
photogrammetry and tacheometry data for recording
of cultural heritage buildings. Int. Arch. Photogramm.
Remote. Sens. Spat. Inf. Sci. 37, 213–218 (2008) Synonyms
Pavlidis, G., Koutsoudis, A., Arnaoutoglou, F.,
Tsioukas, V., Chamzas, C.: Methods for 3D digitization
of cultural heritage. J. Cult. Herit. 8, 93–98 (2007)
Additive manufacturing; Direct digital
Pieraccini, M., Guidi, G., Atzeni, C.: 3D digitizing of manufacturing; Rapid manufacturing; Rapid pro-
cultural heritage. J. Cult. Herit. 2, 63–70 (2001) totyping; Solid-free form technology
Remondino, F., Guarnieri A., Vettore, A.: 3D modeling of
close-range objects: photogrammetry or laser scan-
ning?. Videometrics VIII. Vol. 5665. International
Society for Optics and Photonics 2005 Definitions
Santagati, C., Inzerillo, L., Di Paola, F.: Image-based
modeling techniques for architectural heritage 3D 3D Printing is a form of manufacturing that adds
digitalization: limits and potentialities. Int. Arch.
Photogramm. Remote. Sens. Spat. Inf. Sci. 5. w2,
materials together in layers to form an object. This
555–560 (2013) is in direct contrast to subtraction manufacturing,
Westoby, M., Brasington, J., Glasser, N., which cuts away at a material to form an object.
Hambrey, M., Reynolds, J.: ‘Structure-from-Motion’ Primarily using plastics and/or metal, this form of
photogrammetry: a low-cost, effective tool for geoscience
applications. Geomorphology. 179, 300–314 (2012)
manufacturing is rapidly developing and handling
Yilmaz, H.M., Yakar, M., Gulec, S.A., Dulgerler, O.N.: new, exotic materials. Its increasing adoption rate
Importance of digital close-range photogrammetry in will have a big impact on the processes of distri-
documentation of cultural heritage. J. Cult. Herit. 8, bution and production.
428–433 (2007)

Introduction

3D Object Manipulation Beginning in the 1980s, this technology has had

an interesting development as it has reached its
▶ Object Manipulation Using Real Hand Gesture more mainstreamed status. The individuals that
for Augmented Reality Interior Design contributed to this interesting industry are varied
across the globe. First being referenced as Rapid
Prototyping, this industry donned the title “3D
Printing” in the 1990s and has slowly become an
3D Pointing almost ubiquitous household term today.

▶ Raycasting in Virtual Reality

▶ Virtual Pointing Metaphor in Virtual Reality The History of 3D Printing

The 1980s is known for its retina-burning bright

spandex pants, go-go dancers, and massively
3D Printing teased hair, but lesser known is the fact that it
was the same decade that 3D printing was born.
▶ Tactile Visualization and 3D Printing for It flew under the radar of the public for several
Education decades in almost complete obscurity as it
3D Printing, History of 27

incubated. During that time they were more often table (Tomioka and Okazaki 2014). Despite his
to fall under the title of Rapid Prototyping tremendous personal success with the concept,
(RP) technologies. Actually, 3D printing is inter- Mr. Kodama felt disheartened and thought his A
changeable with quite a few terms. There is the concept was nothing more than a novelty, rather
previously mentioned Rapid Prototyping but there than the beginning of something revolutionary.
is also Rapid Manufacturing (RM), Additive Reflecting now, he says that: “I should have
Manufacturing (AM) technologies, Solid-Free worked harder to make people understand the
Form technology (SFF), or Direct Digital significance of my research results”(Tomioka
Manufacturing (DDM). and Okazaki 2014).
What is so fundamentally different about 3D Other individuals, Alain Le Mèhautè, Olivier
printing is that it uses an additive manufacturing de Witte, and Jean Claude Andrè, filed a patent
approach, which is where precise amounts of in 1984 on the Stereolithographic process but
materials are bound together in exact layers to was abandoned by the French General Electric
form an object, versus the conventional subtrac- Company (now Alcatel-Alsthom) and CILAS, a
tive manufacturing method, which removes mate- subsidiary of the European Aeronautic Defense
rials to create products. Much like the varied titles and Space Consortium for a “lack of business
that can be referenced to this form of manufactur- perspective” (Mendoza 2016). It seemed that
ing, so too are the hands that formed it. The his- this technology was doomed in its cradle as
tory of 3D Printing is varied and expands the many overlooked its potential, while the inven-
narratives of some very eclectic people. tors that produced the machines languished
Hideo Kodama of Nagoya Municipal Indus- knowing something was brewing beneath the
trial Research Institute published the first working surface.
account of a photopolymer additive manufactur- This is not to say that the people who contrib-
ing system in 1981 (Tomioka and Okazaki 2014). uted did not receive credit for some of the foun-
Mr. Kodama’s inspirational moment struck him dational research and thought processes that were
over a year earlier while he was riding a bus home, fundamental to this industry. In 1995, Hideo
reflecting on an exhibition in Nagoya where he Kodama was chosen to receive the Rank Prize, a
was able to observe a machine capable of making privately funded British award for inventions, and
letters utilizing liquid resin applied to a glass was credited with creating the first of the key
surface. The machine he witnessed at work was technologies for unlocking the rapid prototyping
targeted towards the newspaper industry; how- industry (Tomioka and Okazaki 2014). He shared
ever, upon reflection Kodama realized that he this award with Charles Hull, our next inventor to
could harness this to create three-dimensional highlight.
objects and began work on his concept (Tomioka A few years after Kodama on the other side of
and Okazaki 2014). the world we have Charles Hull, or Chuck, as he
It was in early 1980 that he set off to work often goes by. This man is considered the father of
applying what he saw at the exhibition to a new 3D printing and is often where you hear the story
way of creating items. After a viable system was of 3D printing begin. In the 1980s, he was work-
brought to fruition, he began to show his peers his ing for an ultraviolet lamp company that added
innovative concept while starting the process of a layer of hard plastic onto surfaces, such as tables
filing a patent; however, without the interests or and countertops. After gaining permission from
support of his peers, doubt soon overcame him his superiors, Hull began to tinker after hours with
and he quietly disregarded the effort. In a regret- a way to use the UV light to create tangible 3D
table move, he did not complete a review period objects from a Computer Aided Design (CAD)
necessary for receiving the patent. Kodama is said software, primarily utilizing the materials and sci-
to have created a two-story miniature house the ence familiar to him through his daily work. He
size of a human palm by manipulating thin would experiment with photopolymers to lay the
layers of resin; impressively, the 3D model held foundations of what would later cement him as the
rooms, a spiral staircase, and even a dining room Father of 3D Printing. To which, there has been
28 3D Printing, History of

contention to, and not just those hailing Hideo astute judgments, from projecting the amount of
Kodama or the French as originators of the time the technology would require to become
concept. well-known to his imaginative nature that enabled
The material that Hull decided to use is called him to contrive his invention, had helped to
a Photopolymer. These materials are a type of cement him as the father of this industry.
plastic which harden and soften under different There were some, just like for Hideo Kodama,
intensities of UV light. After a few design itera- who felt that Charles Hull has been given too
tions, Hull created a finalized machine that manip- much credit for the creation of this industry
ulated minuscule plastic layers of photopolymer while inadvertently leaving others in the dark;
and each of these small layers combined to form however, it takes a combination of belief in your
the entire object. A simple cup, only a few inches product or process, dedication to create a com-
tall, was the first item to be fabricated and pany, and a little luck in the right market opportu-
represented a fundamentally different approach nity to make as large and noticeable of an impact
to our general notion of production (Davis 2014). as he has. As the saying goes: “to the victor go the
Through those long nights of toil, he invented spoils” – this is no less for Charles Hull. There are
the Stereolithographic (SL) process and filed for more people involved in the beginning of this
a patent in 1984, receiving it in 1986. He actually industry than just he. So, we are going to wind
filed the patent 3 weeks after the French team and back the hand of time a bit to capture our next
even further behind Kodama, but due to technical inventor in the right light.
requirements and timing Charles Hull became the The 1970s was a time of expansion and explo-
first person to patent and create a usable 3D print- ration for many. For William (Bill) Masters, this
ing method. What also lends to this standing as was especially true. During this decade, he is said
an industry founder is the creation of the Standard to have first speculated his form of 3D printing
Tessellation Language (STL) file, which are still technology. From the South Caroline: A History
widely used today. of 3D Printing website, Bill Masters reflects on
In the early 1990s, the Stereolithographic a kayaking trip he took where he recalls the
Apparatus, or SLA-1 machine, was created by moment when inspiration struck him:
3D Systems with Charles Hull as one of the foun-
ders. Although with some error, this machine It was on a river trip. Back when I had the whole
showed that complex parts could be built over- nine yards, the Volkswagen Van [and] the long hair,
so when you’re on the side of the river. . . and
night or within a few hours using this method. you’re all laying there and looking up there at the
Primarily driven by tremendous cost, these sky at all those little dots up there – they call them
cumbersome contraptions were more useful for stars. Why can’t you make things in outer space?
transportation and other larger commercial indus- That’s how it started; it started by looking at a star.
(The Father of 3D Printing n.d.)
tries. It was said that Charles Hull was obsessed
with helping Detroit regain its competitive advan- While he may have conceived his device early
tage as an influx of higher quality Japanese on, there were sets of personal circumstances that
imports convoluted the market and moved inhibited him from pursuing his musings of this
manufacturing jobs away (Davis 2014). concept until a few years down the road. Coinci-
Charles Hull was anticipatory of the gestation dently enough, when he did, he found it was
period that this type of technology would require around the same time that Charles Hull was also
while reaching full market awareness and public- submitting his patent materials. That year was
ity. In early interviews, he would project that it 1984, July 7th of 1984 to be exact. The date is
would take 20–30 years for the technology to find special for William Masters as it was the date
itself into more mainstream applications; how- for which he filed the patent for his Ballistic
ever, it is with a combination of surprise and Particle Manufacturing technology – patent
excitement that he now sees how this technology #4665492 (USPTO Patent Full Text and Image
has grown in its capabilities (Davis 2014). His Database n.d.-a). Most particularly, it is also a full
3D Printing, History of 29

month ahead of Charles Hull’s patent #4575330, crosshairs. Although many of these individuals
which was filed on August 8, 1984 (USPTO were in completely different areas of the country,
Patent Full Text and Image Database n.d.-b). each seemed to be enraptured by automating the A
Charles Hull’s patent application was accepted creation process behind three dimensional objects
and published before the patent that Masters had from the computer; however, just like their unique
submitted. To be realistic, it takes much more than locations, each approached the concept in very
a patent filing date and speculation of a concept different ways.
to receive the credit of founding an entire industry. To say that Carl Deckard was young when he
Albeit, Masters contends that the birthplace began his pursuit of this technology is an under-
of 3D printing is in North Carolina and also attri- statement. A freshman in Mechanical Engineering
butes the fact that Hull created a company along at the University of Texas in Austin, he spent
his patent almost immediately as a reason for his his nights working in a metal shop that relied on
success. This is in contrast to Masters, who was the new, at the time, technology of Computer
not actively participatory until 1988 when he cre- Aided Design (CAD) and began to daydream.
ated Perception Systems, which later became Bal- He chose his major of Mechanical Engineering
listic Particle Manufacturing (BPM) Technology. because he found that it was: “The closest thing to
It was not until 1991 that his company, then newly majoring in inventing” (Selective Laser Sintering
titled as BPM Technology, obtained funding from and Birth of an Industry 2012).
Palmetto Seed Capital to create the machine based Deckard was working for a facility in 1981
on his patent (The Father of 3D Printing n.d.). called TRW Mission, which crafted parts using
This was years after Charles Hull and other inven- CAD software; however, many parts were created
tors had already forged a path ahead of him by from castings, or the castings themselves came
creating different machines, processes, and were from handcrafted casting patterns and he began
already selling machines based on their concepts. to see that there was a potentially large market for
In addition, it is distinct to note that even if the creating casting patterns out of CAD Models. He
technology that Masters had created was to be envisioned lasers tracing themselves over fine
as widely adopted as Hull’s then there is doubt layers of dust to bind together materials. By
that it would have fared as well because it only the time his senior year rolled around the only
creates structurally weak, hollow models. While thing he needed to make his musings a reality
Bill Masters may not have revolutionized 3D were the parts to do so. An Associate Professor
printing industry as a whole, he still filed a by the name of Dr. Joseph Beaman took the young
respectable amount of patents for technologies student under his wing, then described as young
outside of 3D printing and also within this indus- and hungry, and the two began a trek to create this
try. Some of these patents include extruding fluent new form of machinery manufacturing.
materials, 3D printing using pin arrays, and for the Deckard began his transition into graduate
use of fluent material droplets (The Father of 3D school and, as luck would have it, the Mechanical
Printing n.d.). Engineering Department was also moving to a
new building, meaning that the budget had room
for some equipment purchases. Beaman and
Selective Laser Sintering and Fused Deckard took advantage of the opportunity and
Deposition Modeling together they submitted a budget for the $30,000
worth of materials required to bring the idea to
As previously mentioned, there were many life. They affectionately dubbed the early stages
machines that also proliferated during this time of the Selective Laser Slithering (SLS) machine
as we come crashing into our next process. During “Betsy,” as it developed through the mid- to late-
the 1980s, there seems to have been an itch in 1980s. Although at first it was slightly crude
many inventors’ mind for machinery such as this, method of production, as Deckard refilled a
as our next innovator Carl Deckard comes into our small box with powder by hand and ran the
30 3D Printing, History of

computer which powered the scanner on top of the 1980’s and 1990’s, one of which actually involved
table. The first parts that were created were simply William Masters from the then Perception Sys-
hunks of plastic to demonstrate that the concept tems, Nova Automation received funding from
could actually work. As he labored, though, more the Goodrich Corporation. With this funding
precise parts began to be produced by regulating they were able to keep Blair on board and they
the laser with the computer by writing code; upon renamed themselves the DTM Corporation. It was
evaluation, it was found that the parts produced a reference to the term Desk Top Manufacturing
were beginning to be of usable quality. At that or some have said it is a reference to the words
point, Deckard called upon Beaman to write it up “Deckard, Texas, and McClure”.
for his Master’s degree because he had just created Unfortunately, even after their hard efforts the
an entirely new and viable machine. He was none technology was seen as a reflection of an industry
the wiser that this would lay the foundation of still in its infancy and did not fare well. The majority
some incredible processes in 3D printing technol- shares were sold to a group of private investors,
ogies in the decades to follow. who then turned around and sold the company and
Deckard stayed at the University of Texas to concepts to 3D Systems, which allowed them to
continue refining the idea, receiving a grant from acquire key patent rights to the SLS technology. For
the National Science Foundation. They enclosed 3D Systems to now hold the rights to SLS and SL
the rudimentary aspects of the machine into an technologies has played to the company’s power
electrical box, added a counter-rotating roller to position as the “world’s leading provider of additive
level the powder between layers of the laser manufacturing technologies.”
sintering (which was being done by hand before), When it comes to market dominance, Stratasys
and after the method had been fully polished the is one of the few companies to challenge 3D
parts began to coming off the machine at a higher Systems when it comes to their lion’s share of
quality. It was then that the machine started to control. Scott Crump, founder of Stratasys, pat-
show the makings of something more versatile ented the Fused Deposition Modelling (FDM)
than just creating casting patterns. technology in 1989 (Perez 2013). Not only is it
Paul Forderhase, another graduate student, the most familiar form of 3D printing for the
joined the efforts as the concept matured from public, it was actually a pursuit with adorable
being an undergraduate dream to a graduate project roots. In 1988, Crump decided that it would be a
and was now gaining enough momentum to seek wonderful idea for him to make a toy frog for his
becoming a commercial company. An Austin busi- young daughter using a glue gun loaded with
ness owner by the name of Harold Blair paired with a mixture of polyethylene and candle wax (Perez
an Assistant Dean of Engineering and occasional 2013). With the support of his wife, and several
adjunct professor by the name of Dr. Paul burnt plastic pans later, he soon became obsessed
F. McClure. They had become interested in the and took his project to the garage – where he
technology and the company was named Nova devoted many long weekends to it. He invested
Automation – after Blair’s existing company called into digital-plotting equipment (which cost about
Nova Graphics International Corporation. Deckard $10 K) to help automate the process and the first
estimated that they would need $75,000 in startup prototypes of the toy began to be churned out.
capital to get off the ground, which was doubled by His wife prodded and pushed for him to either
Beaman, and then doubled again by those oversee- turn this affixation of his into a viable company
ing the project bringing their estimated startup cost or give it up because he had already spent tens of
to $300,000 – this was just to keep the interests of thousands of dollars to produce one supposed toy
Blair and McClure (Selective Laser Sintering and for his daughter (Perez 2013). By that time
Birth of an Industry 2012). though, it had evolved into a larger project, a
After a few hit and miss opportunities for mission, a higher calling than simply creating a
funding with several companies through the plastic trinket toy for his beloved. He saw the
3D Printing, History of 31

potential of a machine like this as he clacked away see the light of potential. At the time he was
many nights in his garage. working for a company called General Scanning,
The first of the Stratasys kits were $130,000 which evaluated additive manufacturing technol- A
and not viable for the regular consumer market, ogies at a project level, and they decided not to
nor really even for small businesses (Perez 2013). invest into developing the technology. Langer
Much like 3D Systems, their efforts were revised remained unconvinced of their decision. He firmly
and they began to focus selling his machines to believed that the technology would be the future.
larger corporations that had the funds necessary to He formed his own company in 1989 and set off to
fuel his refrigerator-sized machines. They liqui- create a new and viable industry. By the amount of
dated all of their family assets and poured every- sheer success that he has encountered, time has
thing into their company to get it to that point. shown that his judgment was sound.
However, to even fulfill the first orders they would If the previously mentioned words “Laser
require the support of venture capitalist – they Sintering” were familiar, you were keen. The
found a company willing to invest in the concept technology was originally created in the United
called Battery Ventures. The company took a 35% States and had an interesting pathway to this Ger-
stake in the company for $1.2 million (Perez man-based company. When Carl Deckard’s com-
2013). pany failed and 3D Systems gained the U.S
Since then, Stratasys has evolved to be one of patents on the SLS technology, they also entered
the largest companies in the world for 3D Print- an agreement with EOS where 3D Systems would
ing – often battling for glory alongside 3D Sys- purchase a product line from EOS, which was
tems. Scott Crump is a formal Mechanical directed at SL technologies, while EOS would
Engineer who heads this company and he is be able to take over global patent rights on the
credited with Charles Hull et al. as one of the SLS technology. This included other interesting
founders of the 3D printing industry. In 2013, developments of Laser Sintering, such as applica-
Stratasys strategically bought out MakerBot – tions of metal manufacturing.
who has become a household name for the home Under the same umbrella that Carl Deckard
desktop 3D printer. worked under, Suman Das also developed applica-
tions for SLS technology at the university, except
he used metal powders for his Master’s and Ph.D
EOS and the Evolution of Selective Laser studies. It may come as a surprise to no one that he
Sintering was also under the supervision of Joe Beaman. If
that name sounds familiar it is because he was the
This machinery was a global phenomenon during same man who aided Carl Deckard in his
its development, Hans Langer formed Electro pioneering in the original Laser Sintering concept.
Optical Systems (EOS) GmbH in Germany Under the Defense Advanced Research Project
around the same time that patent applications for Agency (DARPA), Office of Naval Research
the first forms of this technology began flying (ONR), and Air Force Research Laboratory
around the United States in 1989. Still true to (AFRL) sponsorship, Suman designed and built
this day, EOS machines are recognized for their two additive manufacturing machines and aided
superior quality of output that utilizes the Laser in co-inventing two laser-based additive
Sintering (LS) process. Their first ‘Stereos’ manufacturing processes in metal for specific use
machines were sold in the 1990’s, making them in high performance aerospace components (Selec-
the first European provider of high-end rapid pro- tive Laser Sintering, Birth of an Industry).Thus,
totyping systems. enabling the ideas from a Texan student to reach
As to what motivated Langer to strike it off into the global market through a German company.
unknown territory such as this was actually the The development in technology in Germany
cloudy doubt cast by others which enabled him to can be dizzying due to massive amount of cross
32 3D Printing, History of

collaboration between key companies. An off- wake. The decades will blur by and the innova-
shoot process to SLS called Selective Laser Melt- tion will exceed our expectations, as they did for
ing was initially developed in 1995 at the Charles Hull. 3D printing is set to carve an inter-
Fraunhofer Institute for Laser Technology (ILT), esting niche for itself, uniquely performing
two Doctors by the names of Dr. Dieter Schwarze where traditional methods fail, while also turning
and Dr. Matthias Fockele, who then formed F&S the table of development. While it may truly
Sterelithographietechnik GmbH. never replace mass manufacturing, the changes
Around the same time, a company by the name it makes in our supply chain and the way we view
of TRUMPF Group began to work with their own manufacturing as a whole can cause us to change
brand of this technology based on the ILT as creators and consumers. This is where the
research. What also makes them a powerful con- power of 3D printing lays, in its ability to enable
tender in the 3D printing market today is their creativity and innovation in places we thought it
extensive history of precision laser systems, and was stagnate, to bring increased individuality to
the fact that they also happen to hold exclusive the products we consume, and rethink our
rights to ILT DMLS patent portfolio. DMLS is manufacturing process now that we are brought
Direct Metal Laser Sintering, which combines more intimately to it.
SLS and SLM properties. This technology was
created in 2002 with a collaborative agreement
between EOS and TRUMPF where they decided References
to share key technology with the goal of enabling
more growth between the methods, based partially Davis, A.: Layer-by-layer: The evolution of 3D printing.
Retrieved 15 Dec 2015, from http://theinstitute.ieee.
on the research that TRUMPF Group leveraged
org/tech-history/technology-history/layerbylayer-the-
from the ILT. evolution-of-3d-printing (2014, November 14)
Always eyeing advancement, in 2008 EOS/ Evans, J. DMLS: A bumpy road in history. Retrieved
TRUMPF announced an agreement with a com- December 15, 2015, from https://designandmotion.
net/design-2/manufacturing-design/dmls-a-little-his
pany called MCP (who was itself partnered with
tory/ (2014, November 10)
F&S Sterelithographietechnik) for patent licenses GCRI Interview. Retrieved January 15, 2016, from http://
that expanded EOS and TRUMPF laser-sintering www.germaninnovation.org/docs/GCRIInterview-Langer.
patents, specifically the machinery that enabled pdf (2013, July 7)
Mendoza, H. R.: Alain Le Méhauté, The man who sub-
titanium and aluminum powder manipulation;
mitted patent for SLA 3D printing before Chuck
however, the patents did not extend to North Hull. Retrieved 28 Dec 2017, from https://3dprint.
American and when MCP attempted to sell a com/65466/reflections-alain-le-mehaute/ (2016,
machine, the Realizer, in the USA through 3D March 16)
Perez, B.: 3D printing pioneer Scott Crump’s kitchen
Systems they landed in some hot water with
experiment. Retrieved 15 Dec 2015, from http://
EOS. They had technically performed patent www.scmp.com/business/companies/article/1287961/
infringement and EOS pursued a lawsuit against 3d-printing-pioneer-scott-crumps-kitchen-experiment
them. After settling the lawsuit for an untold (2013, July 22)
Selective Laser Sintering, Birth of an Industry: Retrieved
amount of money, MCP (now MTT) went through
28 Dec 2015, from http://www.me.utexas.edu/news/
some hand changing and consolidation to reform news/selective-laser-sintering-birth-of-an-industry (2012,
itself as a private company under SLM Solutions December 06)
GmbH, who now sells their own brand of SLM The Father of 3D Printing: Retrieved 28 Dec 2017, from
http://billmasters3d.com/father-of-3d-printing/ (n.d.)
machinery. Tomioka, S., Okazaki, A.: Japanese 1st to seek patent for
3-D printing back in 1980s, but still lost out – AJW by
The Asahi Shimbun. Retrieved 15 Dec 2015, from
Conclusion http://ajw.asahi.com/article/behind_news/social_affairs/
AJ201409150037 (2014, September 15)
USPTO Patent Full Text and Image Database: Retrieved
3D printing is bounding forward quickly, 28 Dec 2017, from http://patft.uspto.gov/netacgi/nph-
crafting vehicles, organs, and even homes in its Parser?Sect1¼PTO1&Sect2¼HITOFF&d¼PALL&
3D Puzzle Games in Extended Reality Environments 33

p¼1&u¼%2Fnetahtml%2FPTO%2Fsrchnum.htm& Introduction
r¼1&f¼G&l¼50&s1¼4665492.PN.&OS¼PN%
2F4665492&RS¼PN%2F4665492 (n.d.-a). Retrieved
from United States Patent Office numeric search Games bring the attention and indulges a person A
function in an activity irrespective of the age. Games have
USPTO Patent Full Text and Image Database: Retrieved been around humans since the dawn of history and
28 Dec 2017, from http://patft.uspto.gov/netacgi/nph- they are inextricably intertwined with the human
Parser?Sect1¼PTO1&Sect2¼HITOFF&d¼PALL&
p¼1&u¼%2Fnetahtml%2FPTO%2Fsrchnum.htm& evolution. In recent times, as the digital technolo-
r ¼1&f¼G&l¼50&s1¼4575330.PN.&OS¼PN/ gies evolved, video games became more popular
4575330&RS¼PN/4575330 (n.d.-b). Retrieved from among all age groups of people. In that, puzzle
United States Patent Office numeric search function video games were primarily designed to keep the
player/s engaged by testing the ingenuity or
knowledge. The games were designed with
increasing levels of complexities to cater to the
tendency of the players to move to higher com-
3D Puzzle Games in Extended plexities and challenges. Thus, once the player
Reality Environments solves or reaches the end of a level, it gave the
sense of satisfaction which in turn enhanced the
Prasad S. Onkar and Devi Meghana engagement of the player with the games. Dis-
Department of Design, Indian Institute of sected or jigsaw puzzles are spatial rearrangement
Technology Hyderabad, Hyderabad, India games which involves visuospatial thinking and
problem-solving. John Spilsbury, a London-based
cartographer, is considered as the creator of jigsaw
Synonyms puzzles (Williams 2004). In 1760s, he first created
such a puzzle by pasting the map of different
Augmented reality; Extended reality; Mixed kingdoms in Europe on a wooden plank and dis-
reality; Spatial interactions; Virtual environments; sected it into the maps of individual kingdoms.
Virtual reality Such maps were marketed as tools for geography
education for children. These puzzles are chal-
lenging and also enhances the motor ability and
Definitions constructional praxis in children. These games
have come off the age through different transfor-
Augmented reality (AR): It can be defined as a mation. The early transformations include “three-
medium where the virtual models are aug- dimensional jigsaw puzzles” which was patented
mented onto a real-time objects, or a virtual by Hammer Willie in 1961. Host of interesting
information overlayed on top of a real-world geometric puzzle designs have been explored by
objects or space. Stewart Coffin (2006). Many researchers have
Virtual reality (VR): It can be stated as a medium explored algorithmic analysis and synthesis of
which provides three-dimensional immersive such puzzles (Song et al. 2012). Though the com-
environments which allow the user to interact puters can create interesting puzzles in the digital
with the digitally created objects. environment, they are unable to actively engage
Mixed reality (MR): As the name suggests, it is with the players, unless the interaction medium
the medium where virtual, i.e., digital objects supports this complex perceptual task. Toward
and real-time physical objects coexist and this, a host of digital interactive technologies
interact with each other. have been developed to make seamless interaction
Extended reality (XR): It is an umbrella with the virtual digital world. The advent of tech-
term which encompasses technologies like nologies like virtual reality (VR), augmented real-
VR, AR, MR, etc. and their combined ity (AR), and mixed reality (MR) have shown a
experiences. great promise to facilitate such puzzle games in
34 3D Puzzle Games in Extended Reality Environments

the modern era. Such interactive gaming modali- technologies facilitate physical puzzle interac-
ties have the potential to solve the present-day tions with digital overlay of information to sup-
challenges. This article presents the role and pos- port the game play. Augmented reality system
sibilities of 3D puzzle games in the context of integrates computer-generated information with
spatial, immersive interactive technologies like the help of sensors to create 3D virtual models,
VR, AR, MR, etc. which are collectively known which has to be calculated along with the user’s
as extended reality (XR) technologies. position and orientation. These technologies are
also ubiquitous because it can be deployed on the
smartphones, and utilize the functionalities of the
VR in Gaming with Puzzles AR applications (Roberto et al. 2019). This tech-
nology has potential to greatly transform interac-
Virtual reality (VR) immerses the human’s mind tive experiences of 3D puzzle games.
into another realm, thus by deceiving the human
senses, which aligns the brain with alternative
reality. In VR, stereoscopic displays, software Interactions in XR Media for Puzzles
platform, motion-tracking hardware, etc. are
used to generate a virtual environment. In VR Physical puzzle games make the users to think
systems, the user is unaware of the physical logically by interacting with the pieces to solve
(real) surrounding. This also implies that there is the puzzle. A game’s physical aspect consists of
a higher degree of engagement with the virtual interactions such as touch and feel, and empha-
environment and enhanced cognitive involve- sizes on physical skills of a player in solving it
ment. Perception of spatial arrangement is essen- with emotional cues. Computer games have limi-
tial for virtual games with puzzle metaphor. Such tation of physical interactions to the players’ use
games developed for applications like spatial per- of interface hardware to play the game (Nilsen
ception (Rasheed et al. 2015) and anatomy edu- et al. 2004). Puzzles require users to randomly
cation (Pohlandt et al. 2019). To facilitate greater arrange the elements, thus entailing the cognitive
immersion into the virtual environment, the map- activities such as exploring all the possibilities
ping between the human sensorial inputs and improving the strategy development. 3D puz-
corresponding with the spatial transformation of zles help users to think differently than the tradi-
virtual objects in the scene should be consistent. tional 2D puzzle pieces. As 3D puzzle would have
Thus, it is the responsibility of the VR content an extra dimension to it and becomes more chal-
developers to achieve realistic immersion into the lenging to solve it than a 2D puzzle set. Using 3D
virtual environment. puzzles as a metaphor can greatly enhance the
interactions in XR medium. The major potential
of XR is in altering how we perceive the virtual
AR/MR in Gaming Puzzles world with the help of computer-mediated user
experience. This disruptive behavior in leading
AR technology can help in transforming a physi- to paradigm shift in the applications developed
cal space through spatial interaction with the vir- in various domains like health care, education,
tual data overlapped to create an engaging entertainment, military applications, etc. Thus,
environment. In the context of assembling a 3D such user interfaces must have a richer visual
puzzle, user gets continuous feedback based on content and seamless interaction between real
their actions while assembling the puzzle pieces. and virtual worlds with the help of intuitive sens-
One such example is demonstrated in using ing, tracking, and feedback technologies. One of
projection-based technique (Kitagawa and the important factors in facilitating interactions
Yamamoto 2011), in which the user is guided by through XR media is to have reliable spatial inter-
projected image on the tabletop. These actions. The 3D puzzle metaphor is extremely
3D Puzzle Games in Extended Reality Environments 35

useful to achieve these seamless interactions as Object Recognition

described in the framework presented in the fol- To augment any physical task with the support of
lowing section. XR technologies, it is required to capture the A
physical scenario into digital format. Thus, object
recognition technologies will help in capturing the
Framework for Interactions in XR physical scenario. There are multiple aspects of
identifying the object or object recognition such
Based on the prior experience of developing XR as the perspective views and the angles related to
applications, a framework is derived which would the object. Image-based object recognition tech-
be useful for developing applications for any rou- nologies are useful for 2D planar scenario. In 3D,
tine task which can be made more engaging and capturing and identification of objects is complex.
convenient for the user with the help of XR Though there are many technologies like 3D scan-
technologies. ning (Tucker et al. 2014), depth cameras
Assembling the components of a puzzle con- (Langmann 2014) etc. to reconstruct a 3D space,
sists of various steps and should be done in a but still the challenge is to solve it in real time.
systematic sequence. Hence, it involves an
architecture integrating software and hardware. Movement or Orientation
The architecture of the system is shown in Fig. 1. When a successful object recognition is done, the
Here, when the user gazes at a three-dimensional next aspect is to facilitate movement or the orien-
object through a hardware such as Microsoft tation of a particular 3D object. When the
HoloLens ® which has camera embedded to it, dynamic object moves in the space, one can
the spatial information of the object is captured understand the transformation components such
by the camera through the markers and this as translation, rotation, scale, etc. covering all the
information is provided to the software to pro- three axes.
cess and identify the object. Once the objects are
detected, the corresponding animations will be Interaction Between the Physical and the
triggered and information explaining the process Virtual Objects
of interactions of physical objects will be pro- Interaction can be defined as communication and
vided. Further actions are triggered depending reciprocal actions between two entities to reach
on the user feedback, and this process will con- a common functional goal. The interaction
tinue until the task is completed successfully or between a physical object and the virtual object
any other conditions of termination of the inter- can be defined as both representations are
action. Robust implementation of this frame- interdependent. The interaction of bimanual
work for creating a better user experience motor skills enables the user to perform interac-
depends on the components of the framework tions that can be through narrative essentials such
described below. as hand gestures, voice, or gaze.

3D Puzzle Games in Extended Reality Environments, Fig. 1 Framework of interaction in XR system with voice
interaction
36 3D Puzzle Games in Extended Reality Environments

Voice Interaction components. Puzzles can be interesting metaphors

Voice is an important aspect as it is a part of for VR system designs. Especially in VR games,
interaction by which one can quickly get puzzles help in developing mechanics of the
responses from the system. The role of virtual games (Nyyssönen and Smed 2021). Thus, the
voice assistant can be seen as an embodiment. use of technology such as XR can help in guiding
As mentioned earlier about the continuous feed- the novice users to a precise point/dimension by
back, the role of voice is essential when the inter- detecting points on the physical objects. It helps in
actions are bimanual and also when the user has to building a virtual space to visualize content and
perform multiple interactions. solve problems such as tedious job of assembly
into an interesting one.

Applications
Cross-References
Virtual reality can provide seamless immersive
experience in interactive 3D medium. The essence ▶ Augmented Learning Experience for School
of virtual gaming can be put into various fields Education
such as education and training, which can be very ▶ Augmented Reality for Maintenance
engaging and enhance users’ expertise (Rasheed
et al. 2015).
Mixed reality has various entities such as References
specialized equipment and efficient software,
and with the recent advancements in technology Coffin, S.: Geometric Puzzle Design. CRC Press (2006)
DaValle, A., Azhar, S.: An investigation of mixed reality
many devices have emerged as potential inter-
technology for onsite construction assembly. MATEC
faces, such as Microsoft HoloLens ®. Hence this Web Conf. 312, 06001 (2020)
technology has become ubiquitous and can be Kitagawa, M., Yamamoto, T.: 3D puzzle guidance in aug-
used in various sectors such as education mented reality environment using a 3D desk surface
projection. In: 2011 IEEE Symposium on 3D User
(Rasheed et al. 2015), training and simulations Interfaces (3DUI), pp. 133–134 (2011)
(Mujber 2004), product repair and maintenance Langmann, B.: Depth camera assessment. In: Wide Area
(Paravati 2017), entertainment, health, and mil- 2D/3D Imaging, pp. 5–19. Springer Vieweg, Wiesba-
itary applications. Majority of MR’s potential den (2014)
Mujber, T.S.: Virtual reality applications in manufacturing
can be used in product assembly and the execu-
process simulation. J. Mater. Process. Technol. 155,
tion of industrial tasks. MR is also used in con- 1834–1838 (2004)
struction industry as a tool to BIM (building Nilsen, T., Linton, S., Looser, J.: Motivations for aug-
information modelling), to understand the mented reality gaming. Proc. FUSE. 4, 86–93 (2004)
Nyyssönen, T., Smed, J.: Exploring virtual reality mechan-
three-dimensional models (DaValle and Azhar ics in puzzle design. Comp. Game J. 1–23,
2020). 65–87 (2021)
3D puzzles are one of the best examples of an Paravati, G.: Augmented reality for maintenance. In: Lee
assembly. The user has to carefully place the N. (eds) Encyclopaedia of computer graphics and
games. Springer, Cham. https://doi.org/10.1007/978-
pieces of the puzzle in such a way that it makes 3-319-08234-9_91-1 (2017)
it as a single unit. Often, users fail to assemble the Pohlandt, D., Preim, B., Saalfeld, P.: Supporting anatomy
parts of a device or a piece of furniture. Thus, it education with a 3D puzzle in a virtual reality environ-
results in failure of the product and have to end up ment. In: Mensch und Computer 2019-Tagungsband.
ACM, New York (2019)
losing money for repair. Solving 3D puzzles are Rasheed, F., Onkar, P., Narula, M.: Immersive virtual real-
highly engaging task as it plays with the third ity to enhance the spatial awareness of students. In:
dimension as compared to a traditional 2D jigsaw Proceedings of the 7th International Conference on
puzzle. Such interfaces are extremely relevant to HCI, pp. 154–160 (2015)
the applications like manual assembly of machine
3D Room Layout System Using IEC (Interactive Evaluational Computation) 37

Roberto, P., Emanuele, F., Primo, Z., Adriano, M., Jelena, Definition
L., Marina, P.: Design, large-scale usage testing, and
important metrics for augmented reality gaming appli-
cations. ACM Trans. Multimed. Comput. Commun. IEC is the interactive optimization system incor- A
Appl. 15, 1–18 (2019) porating human tasks. 3D room layout system
Roy S., Sarkar P., Dey, S.: Augmented learning experience using IEC is the application of IEC, and it evolves
for school education. In: Lee N. (eds) Encyclopaedia of layout according to the user preferences.
computer graphics and games. Springer, Cham. https://
doi.org/10.1007/978-3-319-08234-9_88-1 (2017)
Song, P., Fu, C.W., Cohen-Or, D.: Recursive interlocking
puzzles. ACM Trans. Graph. 31(6), 1–10 (2012)
Tucker, C.S., Saint John, D.B., Behoora, I., Marcireau, A.: Introduction
Open source 3D scanning and printing for design cap-
ture and realization. In: International Design Engineer-
ing Technical Conferences and Computers and Designers usually build renderings to create a
Information in Engineering Conference, 46292, new layout, and they reorganize it to fit a
V01BT02A013 (2014) customer need. Furthermore, customers can
Williams, A.D.: The Jigsaw Puzzle: Piecing Together a
History. Berkley Publishing Group, New York (2004)
understand shapes intuitively if they provide
the 3D room layout. Numerical optimization
approaches that optimize parameters
constructing the 3D room layout can automate
3D Reconstruction these works. However, it is difficult to create a
model equation that emulates human thoughts
▶ Deep Learning Algorithms for 3D because it is a subjective personal preference.
Reconstruction Therefore, optimization systems incorporate the
human tasks that evaluate the fitness of solu-
tions manually. These systems usually use inter-
active evolutionary computation (IEC). This
3D Rendering approach is similar to the process of improve-
ment in animal and crop varieties. Some evolu-
▶ Panda3D tionary computing algorithms implement
IEC. The most famous algorithm is interactive
genetic algorithm (IGA), and some studies use
parallel distributed interactive genetic algorithm
3D Room Layout System (PDIGA), interactive differential evolution
Using IEC (Interactive (IDE), interactive particle swarm optimization,
Evaluational Computation) and interactive genetic programming (Takagi
et al. 1998, 2009). This entry focuses on
Ryuya Akase1 and Yoshihiro Okada2 IEC and interactive 3D room layout, and it espe-
1
Graduate School of Information Science and cially treats the system that uses IGA. The
Electrical Engineering, Kyushu University, remainder of this entry organizes as follows.
Fukuoka, Japan Section “Algorithms for IEC” gives the algo-
2
Innovation Center for Educational Resource, rithms for IEC and examples of some applica-
Kyushu University, Nishi-ku, Fukuoka, Japan tions. Section “Layout Generation” shows the
studies that generate 3D layout interactively.
Section “Quantitative Evaluation Techniques
Synonyms and Problems of the IEC-Based Systems”
describes quantitative evaluation techniques
Interactive design; Interactive genetic algorithm; and problems of the IEC-based system. Finally,
Interactive room layout section “Summary” summarizes this paper.
38 3D Room Layout System Using IEC (Interactive Evaluational Computation)

Algorithms for IEC and mutation elevate individuals. IGA replaces

Richard Dawkins famed for The Selfish Gene the fitness function with the user evaluation to
began the study of IEC (Dawkins 1986). After- incorporate user preference and knowledge.
ward, various fields such as arts, engineering, Table 2 shows the pseudo-code for typical IG-
education, and entertainments incorporate IEC A. The user rates the evolved individuals, and
(Bentley 1999). Takagi organized the applications IGA generates new individuals according to the
of IEC between 1986 and 2000 (Takagi 2001). user evaluations. The following are specific
Table 1 lists the major application. The following procedures:
are specific advantages of IEC:
1. Initialization: This process generates the initial
1. Personalize applications based on user individuals. Typical IGA creates genes config-
preferences. uring an individual randomly.
2. Incorporate knowledge and heuristics of users 2. Evaluation: This is the human task. The user
to the system. rates individuals based on his/her subjective
3. Aid creativity of users. preference. IGA uses these evaluations as
4. Provide user-friendly applications that need the fitness values to evolve the current
not special skills and knowledge. individuals.
5. Analyze user preferences by using optimized 3. Selection: This process selects some highly
solution. rated individuals as parents to create a new
generation. The following are specific selec-
Interactive Genetic Algorithm tion methods:
Genetic algorithm (GA) is a heuristic search algo- • Roulette selection: This method selects par-
rithm, and it bases on the Darwinian theory of ent individuals according to a rate that is
evolution. It finds the optimum solution by gen- proportional to the fitness values. Typical
erating individuals that can be the optimum solu- roulette selection uses the selection boxes
tion. Each individual that is in the population that have room according to the fitness
develops through the fitness function that deter- values and random numbers, and it selects
mines the ability to solve problems, and crossover the individuals by checking the random
numbers that are in the selection boxes.
3D Room Layout System Using IEC (Interactive
The following is the probability selecting
Evaluational Computation), Table 1 Examples of ith individual, where n is the number of
IEC applications individuals and f is a fitness value.
Comfortable
Creativity design Personalize fi
pi ¼ n
Biomorph Lighting Hearing aid f
k¼1 k
Face Web page Vision aid
image
Screen Vehicle Image retrieval
saver
• Tournament selection: This method selects
2D/3D CG Building Knowledge
parent individuals by using knockout
transmission competition.
Modeling Interior layout Knowledge • Elitist selection: Roulette selection and
acquisition tournament selection have the potential to
Animation Clothing Robot arm control lose the best individual because they are
Music Schedule Educational aid probabilistic methods. This method
Voice Game Advertising bequeaths the elite individual to the next
Artificial Story Seasoning generation, and it does not apply crossover
life composition adjustment
and mutation to the elite individual.
3D Room Layout System Using IEC (Interactive Evaluational Computation) 39

3D Room Layout System Using IEC (Interactive the early stage, and solutions lose the
Evaluational Computation), Table 2 Pseudo-code for diversity.
typical IGA
IGA ( )
A
IGA iterates these operations (2–5) until the
generation ¼ 0; user obtains a satisfied result or the number of
create initial group of individuals;
iterations exceeds the predefined constant. The
REPEAT
performance of IGA depends on the operators.
fitness ¼ USER (current group of individuals);
There are many methods other than listed above
IF user is satisfied THEN RETURN the best
individual; (Koza 1992) (Fig. 1).
i ¼ 0;
REPEAT Gene Expressions IGA has many types of gene
select two individuals based on fitness; expressions as with GA (Back 1996). An individ-
perform pcrossover; ual has the genotype and phenotype. GA operators
perform pmutation; operate genotype, and the user evaluates pheno-
insert two offspring into new group of type. The following are typical genotype
individuals; expressions:
i ¼ i + 2;
UNTIL i > predefined number of individuals;
(a) Binary code: This method expresses genes as
generation ¼ generation + 1;
binary codes.
UNTIL generation > predefined number of
generations;
(b) Gray code: This method expresses genes as
STOP the codes that maintain the difference of adja-
cent nodes at 1 bit, and it facilitates the local
search. The following are the conversion
equations of binary codes and gray codes,
4. Crossover: This process transposes gene where k is the bit location and n is the most
sequences of two individuals. The following significant bit:
are specific crossover methods:
• Single-point crossover: This method splits a
bn ð k ¼ nÞ
gene sequence into two halves. Children gk ¼
inherit the parent gene sequences half-and- bkþ1 bk ðotherwiseÞ
half.
n
• Multi-point crossover: This method has bk ¼ gi ðmod 2Þ
some split-off points. Children inherit the i¼k
parent gene sequences alternately.
• Uniform crossover: Children inherit the par- (c) String: This method expresses genes as
ent gene sequences according to a randomly strings. Mutation operator changes a gene ele-
generated mask. ment within the predefined character set.
5. Mutation: This is a way to change a part of (d) Real number: Real-coded GA uses real num-
gene in an individual randomly with a fixed bers directly. The IGA that emphasizes con-
probability. It is a useful way to prevent the tinuous values uses this coding. It can
initial convergence. Typical mutation uses the generate children around the parents. How-
reciprocal of a gene length as the mutation rate. ever, real-coded GA needs the special cross-
• If a specific individual generated in an early over operators such as unimodal normal
stage has an extremely high fitness value, distribution crossover (UNDX). It generates
IGA may select it as an optimal solution. children according to the normal distribution
This is the initial convergence, and it is a obtained from three parents (Ono et al. 2000)
problem in IGA. It converges exploration in (Table 3).
40 3D Room Layout System Using IEC (Interactive Evaluational Computation)

generation = 0
create initial group of individuals

fitness = USER ( current group of individuals )

user is satisfied or yes

generation > predefined END
maximum number of generations
no

i=0

select two individuals based on fitness

perform crossover

perform mutation

insert two offspring into

new group of individuals

i=i +2

i > predefined yes generation =

maximum number of individuals generation + 1

no

3D Room Layout System Using IEC (Interactive Evaluational Computation), Fig. 1 The flowchart of typical IGA

3D Room Layout System Using IEC (Interactive Evaluational Computation), Table 3 Typical genotype
expressions
Phenotype Binary code Gray code String Real number
1.0 000 000 “1.0” 1.0
1.5 001 001 “1.5” 1.5
2.0 010 011 “2.0” 2.0
2.5 011 010 “2.5” 2.5
3.0 100 110 “3.0” 3.0

Parallel Distributed Interactive Genetic Algorithm to the multiuser preferences. It can generate the
Miki et al. proposed parallel distributed interac- new solutions combined with other user evalua-
tive genetic algorithm (PDIGA) (Miki et al. 2003, tions, used in consensus building system. PDIGA
2006). Table 4 shows the pseudo-code for typical inherits the performance of PDGA that can reduce
PDIGA. This algorithm extends IGA to parallel the computation time and avoid the initial conver-
distributed model, optimizing solutions according gence of GA. PDGA connects each computer and
3D Room Layout System Using IEC (Interactive Evaluational Computation) 41

3D Room Layout System Using IEC (Interactive 2. Efficiency: It needs a small number of times of
Evaluational Computation), Table 4 Pseudo-code for evaluation.
typical PDIGA
PDIGA ( )
A
Takagi et al. explained advantages to use DE
generation ¼ 0; for IEC as follows (Takagi et al. 2009):
create initial group of individuals;
REPEAT
1. The user can use the paired comparison
receive an elite individual;
evaluation.
insert the elite individual into current group of
individuals; 2. It has a good convergence characteristic in a
fitness ¼ USER (current group of individuals); small population.
send a new elite individual;
IF user is satisfied THEN RETURN the best DE has some formats, and it expresses as
individual; DE/base/num/cross. The following are specific
i ¼ 0; formats:
REPEAT
select two individuals based on fitness; (i) Base: The selection method of a base vector.
perform crossover; • Rand: Select a vector from a parent group
perform mutation;
of individuals randomly.
insert two offspring into new group of
individuals;
• Best: Select the best vector from a parent
i ¼ i + 2;
group of individuals. DE/best has better
UNTIL i > predefined number of individuals; convergence performance than DE/rand.
generation ¼ generation + 1; However, it needs parallel comparison
UNTIL generation > predefined number of of all individuals to select the best
generations; vector.
STOP • Gravity: Select a centrobaric vector
from a parent group of individuals.
DE/gravity has almost the same conver-
runs IGA on those computers. PDIGA uses the gence performance as DE/best, and it
migration that sends and receives the elite indi- needs not parallel comparison of all
vidual each user selected, incorporating other individuals (Funaki et al. 2011). How-
individuals. ever, the convergence performance will
They also proposed global asynchronous dis- deteriorate if the centrobaric vector is
tributed interactive genetic algorithm (GADIGA). quite different from the global optimum
It compiles the elite individuals in a database so solution.
that each computer can migrate them without syn- • Moving: This selection method works
chronism. Each computer gets the elite individual with another selection method. It makes
from the database, incorporating it in own group a moving vector that accumulates the dif-
of individuals. ference between a target vector and trial
vector, and it adds the moving vector to
Interactive Differential Evolution the base vector to accelerate the conver-
Storn et al. proposed differential evolution gence performance.
(DE) that is a population-based descent method (ii) Num: The number of difference vectors.
for numerical optimization (Storn et al. 1997). (iii) Cross: The crossover method of a target vec-
The following are specific advantages: tor and trial vector.
• Bin: Use the binomial crossover that
1. Completeness: It searches comprehensive opti- performs a crossover with a fixed
mized solution. probability.
42 3D Room Layout System Using IEC (Interactive Evaluational Computation)

Population
(individuals)

Target Parameter Parameter Base

Parent
vector vector 1 vector 2 vector
Xi Xp1 - Xp2 Xb
Difference
vector
w(X p1 -Xp2 ) +
Mutant
vector Xm = Xb + w(Xp1-Xp2)
crossover
Initialization
Target Trial
Select target vector Xi
vector vector
Select parameter vectors Xp1, Xp2 and base vector Xb
Xi Xinew Make mutant vector Xm = Xb + w(Xp1-Xp2)
Cross Xi with Xm and get trial vector Xinew
Paired comparison
Choose Xi or Xinew
(user evaluation)
Overwrite target vector Xi
Xi or Xinew
Next individual
Child Target vector or Trial vector
Next generation

3D Room Layout System Using IEC (Interactive Evaluational Computation), Fig. 2 The conceptual diagram and
flowchart of IDE

• Exp: Use the exponential crossover that improve the quality of group of individuals
performs a crossover with the probability (Pei et al. 2013) (Table 5).
decreases in an exponential manner.
Examples of the IEC Applications
The user evaluates individuals in interac- Many researchers elaborate IEC applications,
tive differential evolution (IDE) as with IG- improving performance and extending application
A. Figure 2 illustrates the conceptual diagram ranges. This section introduces some applications
and flowchart of IDE. The user selects the developed in recent years.
target vector or trial vector, and he/she repeats
it until the number of evaluated target vectors Web Page Design Sorn et al. proposed the
exceeds the predefined number of individuals design system that generates a template for Web
to obtain a next generation. IDE reduces the page interactively using IGA (Sorn et al. 2013).
burden of user evaluation because it needs not Although the work of creating and designing Web
parallel comparison of all individuals except pages is increasing, they are time-consuming
for DE/best. However, the user needs to com- tasks. In addition, users have to learn the program-
pare four individuals in the evaluation stage ming languages such as JavaScript and usages of
when used in combination with the moving authoring tools so that they can create the partic-
vector. The user selects a vector from the ular Web page. The proposed system evolves
target vector, trial vector, combination of tar- HTML and CSS based on the user preferences,
get and moving vector, and combination of and genes represent the layout expressing a loca-
trial and moving vector in this case. In addi- tion of navigation bar and styles such as font color
tion, Pei et al. reported that incorporating the and size. It displays ten Web pages as individuals
opposition-based learning with IDE could in a generation, and the user gives ratings of one
3D Room Layout System Using IEC (Interactive Evaluational Computation) 43

3D Room Layout System Using IEC (Interactive Evaluational Computation), Table 5 Pseudo-code for typical
IDE/rand/1/bin
IDE ( )
A
w ¼ predefined weighting coefficient;
generation ¼ 0;
create initial group of individuals;
REPEAT
i ¼ 0;
REPEAT
x_i ¼ select a target vector from current group of individuals;
x_p1, x_p2 ¼ select parameter vectors from current group of individuals randomly;
x_b ¼ select a base vector from current group of individuals randomly;
x_m ¼ x_b + w * (x_p1 – x_p2);
x_new ¼ make a trial vector from x_i and x_m by using binomial crossover;
x_i ¼ USER (x_i, x_new);
IF user is satisfied THEN RETURN x_i;
i ¼ i + 1;
UNTIL i > predefined number of individuals;
generation ¼ generation + 1;
UNTIL generation > predefined number of generations;
STOP

(good) to five (bad). Furthermore, the user can interactively using IGA (Ghannem et al. 2013).
evaluate each section such as header and footer Refactoring is a technique that restructures
particularly so that the system reflects the specific existing models such as class diagrams in a soft-
user preferences. ware development cycle, and it improves design
quality while preserving its semantics. It is diffi-
Fashion Design Mok et al. proposed the design cult to automate a complex refactoring and eval-
system that generates a fashion sketch interac- uate the quality and integrity of refactored models
tively using IGA (Mok et al. 2012). The design because it needs empirical rules. The proposed
aid is useful for increasing productivity. The pro- system incorporates feedback from users into the
posed system has a sketch design model describ- optimization processing, and it uses the first
ing the characteristics of the design, a database knowledge obtained from examples of refactoring
storing sketch design models, and a multistage and the second knowledge obtained from users.
sketch design engine that builds the final design. The system analyzes the similarities of examples
Mok et al. demonstrated the system making the and inserts the second knowledge interactively,
skirts design. Genes represent the silhouette, and it displays applicable refactoring while opti-
shape, type of waist and hem, and decoration mizing a model. Genes represent the sequence of
such as dart, yoke, pleat, panel, gathers, slit, and refactoring operations such as “pull-up method,”
ruffles. The system generates initial population “rename attribute,” and “move attribute.” The
based on the waist level and skirt length the user system applies these operations to compose
decides at the beginning, and it evolves the skirt models and gets the evaluation the user scored
design. using the five-level scale.

Refactoring Ghannem et al. proposed the design 3D Motion Akase et al. proposed the design
system that generates a sequence of refactoring system that generates a 3D motion of the
44 3D Room Layout System Using IEC (Interactive Evaluational Computation)

Avatar interactively using IGA (Akase (Akazawa et al. 2005, 2006). The proposed sys-
et al. 2012). Movies and computer games use tem makes scenes by using a semantic database.
3D motions to make the motion of 3D char- The following are specific attributes of the seman-
acters. Generally, designers generate 3D tic database:
motions using motion capture systems. How-
ever, it is difficult to create various 3D 1. Face: Indicate the surface number of bounding
motions. The proposed system creates a box wrapping the 3D furniture object.
unique 3D motion by considering a physical 2. Occupancy distance: Indicate the minimum
limitation. It combines the inverse kinematics distance not to touch other objects.
with IGA to reduce the variables required for 3. Parent: Indicate the parent objects.
motion optimization, reducing the burden of 4. Constraint: Indicate whether a face should
user evaluation effort. The system optimizes touch a certain face of other object or not.
the trajectory of end effectors instead of the 5. Inside component: Indicate the objects placed
joint angles. Genes represent the traveling inside the object.
sequences in common with the gene expres- 6. Connection width: Indicate the width
sion of traveling salesman problem. As with number of objects to place them in the
the feature Sorn et al. introduced, the user can rectangular area.
evaluate each body part such as the head, 7. Connection depth: Indicate the depth
arms, and legs particularly so that the system number of objects to place them in the
reflects the specific user preferences. rectangular area.
8. Connection face: Indicate which faces of the
bounding box are connectable.
Layout Generation 9. Raito: Indicate the ratio of the number of
objects in a width direction to the number of
This section introduces the methodologies to cre- objects in a depth direction.
ate the interactive 3D layout generation system.
Similarly, Lap-Fai et al. proposed automatic
Automatic Layout Generation Based 3D layout systems based on ergonomics con-
on the Constraints straints (Lap-Fai et al. 2011). They defined ergo-
In recent years, designers create city design and nomics as follows:
room layout in a virtual world. For example, “City
Engine” can create cities including street layouts 1. Accessibility: Maintain the space for furniture
and buildings automatically. Parish and Muller to perform a basic furniture function.
proposed this system, and they use a procedural • For example, bookcases need extra space
approach based on L-systems to model cities for books.
(Parish et al. 2001). 2. Visibility: Maintain the sight for human to look
The layout work for 3D virtual rooms takes a viewing surfaces.
long time because 3D objects have six degrees of • For example, other objects should not block
freedom. In addition, 3D furniture objects have a television.
furniture-specific features. For instance, TV needs 3. Pairwise relationship: Maintain the semantic
some space for its watchers to watch, and desk and compatibility between objects.
chair need pairs generally. These features are asso- • For example, place a desk and chair in pairs.
ciated with ergonomic design. Akazawa
et al. proposed a system that generates 3D scene Their system executes the following processes
automatically based on contact constraints to arrange many objects:
3D Room Layout System Using IEC (Interactive Evaluational Computation) 45

(a) Create relationships between one object and Floor Layout

the other: System learns relationships from Garcia et al. proposed the IGA-based system that
preliminarily ordered 3D layout. generates the floor layouts of an ovine slaughter-
A
(b) Optimize the layout: Minimize the number house and recycling carton plant incorporating the
of inappropriate objects that violate the knowledge of expert designer and preference of
ergonomics constraint using simulated the user (Garcia et al. 2013). The system uses the
annealing. flexible bay structure that manages the floor lay-
outs by dividing floor into bay. Genes represent
Interactive Layout Generation Using the equipment number used to specify an equip-
the Language and the Real-World Information ment location and the position of a bay.
Calderon et al. proposed the interactive furniture
placement system using the constraint logic pro- 3D Room Layout
gramming (Calderon et al. 2003). The user can Akase et al. proposed the IGA-based system that
obtain knowledge and skills of the room layout generates the 3D room layouts by a combination
design by using the system. Similarly, Coyne of parent-child relationships (Akase et al. 2013,
et al. developed Words Eye that generates 3D 2014). The system saves the combinations to the
scenes using the natural language (Coyne
et al. 2001). Words Eye parses several sentences 3D Room Layout System Using IEC (Interactive
Evaluational Computation), Table 6 The build pro-
to extract the object information and tags the cess of combinations
corresponding label, and it depicts a 3D scene
maximum_number_of_furniture_objects ¼ 37;
from 2000 shape data and 3D texts.
maximum_number_of_patterns ¼ 8;
Alternatively, several studies use a 3D scanner patterns[maximum_number_of_furniture_objects]
and an RGBD camera to make 3D scene data from [maximum_number_of_patterns];
the real-world scene information. For instance, makePatternNumbers ( )
Nan et al. and Kim et al. proposed a new method i ¼ 0;
that makes 3D scenes using a 3D scanner (Kim REPEAT
et al. 2012; Nan et al. 2012). In addition, Shao object_i ¼ select a furniture objects;
et al. developed the system that searches similar pattern ¼ {};
3D shapes from a database using RGBD camera, //The system uses this number for genes
making the 3D scenes using them (Shao pattern_number ¼ 0;
et al. 2012). REPEAT
//designer’s task
Interactive Layout Generation Using IEC pattern.parent ¼ select a parent object;
//designer’s task
pattern.relative_distance ¼ set a
Color Design relative_distance;
Miki et al. proposed the office design system that //designer’s task
uses PDIGA (Miki et al. 2003). The system opti- pattern.relative_angle ¼ set a relative_angle;
mizes the color of office equipment and supplies patterns[i][pattern_number] ¼ pattern;
such as partitions, carpets, tables, personal com- pattern_number = pattern_number  1;
puters, and chairs. Genes represent the UNTIL
continuous-valued hue and tone, and the system pattern_number > maximum_number_of_patterns;
employs the real-coded GA method. The user i ¼ i + 1;
evaluates the presented color pattern in terms of UNTIL i > ¼
maximum_number_of_furniture_objects;
the preference and a style that suits the working
STOP
environment using the five-level scale.
46 3D Room Layout System Using IEC (Interactive Evaluational Computation)

3D Room Layout System Using IEC (Interactive database as with the system Akazawa
Evaluational Computation), Table 7 The build pro- et al. proposed. Table 6 explains the build process
cess of room layout
of combinations, and Table 7 shows the build
maximum_number_of_furniture_objects ¼ 37; process of room layouts. Genes represent the com-
//idx is the index number of an individual bination index called “pattern number.” Figure 3
gene_sequences[idx]
illustrates an example of gene sequences, and Fig. 4
[maximum_number_of_furniture_objects];
placeObjects (gene_sequences, patterns)
indicates a part of evolving process in order of top
i ¼ 0; left to bottom right. It optimizes 3D room layouts
REPEAT based on user evaluations.
//select a pattern number (gene element) from a gene
sequence
Quantitative Evaluation Techniques
pattern_number ¼ gene_sequences[idx] [i];
and Problems of the IEC-Based Systems
//get a pattern data
pattern ¼ patterns[i] [pattern_number];
Quantitative Evaluation of the IEC-Based
//The root parent is “Room”
IF pattern.parent is not placed THEN
Systems
i ¼ i + 1;
It is difficult to measure performance of the
continue; IEC-based system in a quantitative way because it
ENDIF includes the subjectivity of users. Many studies
//place a target object_i perform a perceptual study using actual users to
object_i.position ¼ pattern.parent.position + receive a questionnaire about the usability and logs
pattern.relative_distance; of fitness, and they evaluate the usefulness of the
object_i.rotation ¼ pattern.parent.rotation + pattern. IEC-based system applying a statistical test to them.
relative_angle; The following are specific assessment procedures:
i ¼ i + 1;
IF i > maximum_number_of_furniture_objects
THEN
1. Set the goal individual (concept) users should
i ¼ 0; create.
ENDIF 2. Conduct the experimental tests using actual
UNTIL all objects are placed; users.
STOP (A) Create the goal individual using an
authoring tool manually.

Generation 0

Individual 0 Individual 1 Individual 2

chair bed flower table chair bed flower table chair bed flower table
2 5 1 7 5 7 2 4 6 4 2 7

0 1 6 3 3 3 5 1 1 4 2 5
sofa pc tv clock sofa pc tv clock sofa pc tv clock

Individual 3 Individual 4 Individual 5

chair bed flower table chair bed flower table chair bed flower table
1 5 7 3 3 3 5 4 2 1 7 5

6 7 2 2 2 6 1 7 6 3 0 7
sofa pc tv clock sofa pc tv clock sofa pc tv clock

3D Room Layout System Using IEC (Interactive Evaluational Computation), Fig. 3 An example of gene
sequences
3D Room Layout System Using IEC (Interactive Evaluational Computation) 47

3D Room Layout System Using IEC (Interactive Evaluational Computation), Fig. 4 A part of evolving process in
order of top left to bottom right

3D Room Layout System

Using IEC (Interactive 4.5
Evaluational
Computation), 4
Fig. 5 Mixture Gaussian 3.5
function
3

2.5
z
2
1.5

1
3
0.5 2
1
0
–5 0
–4 –1 y
–3
–2 –2
x –1 –3
0
1 –4

(B) Create the goal individual using Takagi et al. proposed a performance evalua-
the IEC-based system interactively. tion method for IEC algorithms (Takagi
3. Subjects select A or B that let them create the et al. 2009). It represents a psychological pecu-
goal individual easily. liarity of an IEC user using a mixture Gaussian
4. Calculate statistically significant difference function consisting of four Gaussian functions.
between A and B. The following are specific function and parame-
ters, where n is the dimension number (Fig. 5):
48 3D Room Layout System Using IEC (Interactive Evaluational Computation)

2
4 n xij  mij
f ð x1 , . . . , xn Þ ¼ a
i¼1 i
exp  j¼1 2 s2ij

3:1
3:4
a¼ ,
4:1
3
1 1:5 2 2:5 1 1:5 2 2:5 1 1:5
0 2 3 1 0 2 3 1 0 2
m¼
2:5 2 1:5 3:5 2:5 2 1:5 3:5 2:5 2
2 1 1 3 2 1 1 3 2 1

1:5 1:5 1:5 1:5 1:5 1:5 1:5 1:5 1:5 1:5
2 2 2 2 2 2 2 2 2 2
s¼
1 1 1 1 1 1 1 1 1 1
2 2 2 2 2 2 2 2 2 2

Problems of the IEC-Based Systems first problem as the previous section

The maximum limit both of the number of indi- explained. This section introduces some sup-
viduals and generations is around 10–20 due to a port methods to reduce a burden of the user
burden of the user evaluation effort in the evaluation.
IEC-based systems (Takagi 2001; Takagi
et al. 2009). Therefore, developers have to con-
sider the following: Improvement of the IEC User Interface
The system Garcia et al. proposed clusters
1. Improve the convergence speed of group of individuals, and it displays some representa-
individuals. tive individuals to the user at least in the
• In general, it depends on an evolutionary early generations (Garcia et al. 2013). It
computing algorithm. employed the fuzzy c-means clustering algo-
2. Support the user evaluation effort. rithm (FCM) Bezdek et al. proposed (Bezdek
• Improve the IEC user interface and evalua- et al. 1984). Each individual can belong to
tion method. some clusters because the FCM allows over-
• Incorporate an agent that predicts the fitness lapping of clusters. This feature can help the
based on the characteristics of IEC users. user evaluation because the individuals that
the user did not evaluate can get scores from
The effective algorithms such as PDIGA, evaluated individuals belonging to a same
IDE, and their derived type are addressing the cluster.
3D Room Layout System Using IEC (Interactive Evaluational Computation) 49

3D Room Layout System Using IEC (Interactive 3D Room Layout System Using IEC (Interactive
Evaluational Computation), Table 8 The procedure Evaluational Computation), Table 9 The procedure
of proposed system of conjoint analysis
MAIN ( ) CONJOINT_ANALYSIS ( )s
A
p = predefined weighting coefficient; cards ¼ predefined renderings;
priorities, elite = CONJOINT_ANALYSIS (); REPEAT
generation ¼ 0; scores ¼ USER (subset of cards);
create initial group of individuals; averages ¼ calculate_average (subset of cards,
REPEAT scores);
//Function C checks the constraints associated with deviations ¼ calculate_sum_of_squared_deviation
priorities (subset of cards, scores, averages);
fitness = (1 – p) * priorities * C (current group of products ¼ calculate_sum_of_products (subset of
individuals) cards, scores, averages);
 p * USER (current group of individuals); solution ¼ solve_system_of_equations (deviations,
IF user is satisfied THEN RETURN the best products);
individual; priorities = calculate_priority (solution);
i ¼ 0; expectancies ¼ calculate_expectancy (cards,
REPEAT solution);
select two individuals based on fitness; elite_individual = select_best_card (cards,
expectancies);
//maintain upper two individuals
precision ¼ check_precision (scores, expectancies);
apply elitist selection;
UNTIL precision > ¼ 0.5;
//produce new two individuals
RETURN priorities, elite_individual;
perform crossover;
STOP
//produce new two individuals based on the elite
individual
perform crossover with elite;
perform mutation; the individual that fits the potential user prefer-
insert six offspring into new group of individuals;
ence analyzed by the conjoint analysis gets an
i ¼ i + 6;
additional score from the system automatically.
UNTIL i > predefined number of individuals;
These features can accelerate the convergence
generation ¼ generation + 1;
speed.
UNTIL generation > predefined number of
generations;
STOP
Summary

Analysis of the Characteristics of IEC Users This entry introduced the typical IEC algo-
The system Akase et al. proposed analyzes user rithms, generation methods of layout, and
preferences using a conjoint analysis so that it recent IEC applications. In addition, it indi-
can support user evaluations and reflect user cated the problems of the IEC-based systems
preferences effectively (Akase et al. 2014). The and some solutions for future directions.
conjoint analysis is an experimental design Although the burden of the user evaluation to
method that gets user evaluation values from optimize complex contents such as 3D room
some rendering and returns the characteristics layout that has many parameters is a problem,
of users. Tables 8 and 9 and Fig. 6 explain the many researchers are addressing it. An
procedures of the proposed system. The system IEC-based authoring tool could be a practical
generates offspring around the elite individual product if they resolved the problem
selected by the conjoint analysis. In addition, completely.
50 3D Room Layout System Using IEC (Interactive Evaluational Computation)

p = predefined weighting coefficient

priorities, elite = CONJOINT_ANALYSIS ( )

generation = 0
create initial group of individuals

fitness = ( 1 - p) * priorities* C ( current group of individuals )

+ p * USER ( current group of individuals )

user is satisfied or
yes
generation > predefined END
maximum number of generations
no
i=0

select two individuals based on fitness

apply elitist selection

perform crossover

perform crossover with elite

perform mutation

insert six offspring into

new group of individuals

i=i +6

i > predefined yes generation =

maximum number of individuals generation + 1
no

3D Room Layout System Using IEC (Interactive Evaluational Computation), Fig. 6 The flowchart of proposed
system

Cross-References of CISIS-2013, pp. 726–731. IEEE CS Press

(2013)
Akase, R., Okada, Y.: Web-based multiuser 3D room lay-
▶ Character Animation Scripting Environment out system using inter- active evolutionary computa-
▶ Genetic Algorithm (GA)-Based NPC Making tion with conjoint analysis. The 7th Int. Symposium on
▶ Teaching Computer Graphics by Application Visual Information Communication and Interaction
(VINCI-2014), pp. 178–187. ACM Press (2014)
Akazawa, Y., Okada, Y., Niijima, K.: Automatic 3D scene
generation based on contact constraints. Proc. Conf. on
References Computer Graphics and Artificial Intelligence,
pp. 593–598. (2005)
Akase, R., Nishino, H., Kagawa, T., Utsumiya, K., Okada, Akazawa, Y., Okada, Y., Niijima, K.: Interactive learning
Y.: An avatar motion generation method based on interface for automatic 3D scene generation. Proc. of
inverse kinematics and interactive evolutionary com- 7th Int. Conf. on Intelligent Games and Simulation,
putation. Proc. of the 4th Int. Workshop on Virtual pp. 30–35. (2006)
Environment and Network Oriented Applications Back, T.: Evolutionary Algorithms in Theory and Practice.
(VENOA-2012) of CISIS-2012, pp. 741–746. IEEE Oxford University Press, New York (1996)
CS Press (2012) Bentley, P.: Evolutionary Design by Computers, pp. 1–73.
Akase, R., Okada, Y.: Automatic 3D furniture lay- Morgan Kaufmann, San Francisco (1999)
out based on interactive evolutionary computa- Bezdek, J.C., Ehrlich, R., Full, W.: FCM: The fuzzy
tion. Proc. of the 5th Int. Workshop on Virtual c-means clustering algorithm. Comput. Geosci. 10,
Environment and Network Oriented Applications 191–203 (1984)
3D Selection Techniques for Distant Object Interaction in Augmented Reality 51

Calderon, C., Cavazza, M. Diaz, D.: A new approach to Shao, T., et al.: An interactive approach to semantic model-
virtual design for spatial configuration problems. Pro- ing of indoor scenes with an RGBD camera. ACM
ceedings. Seventh International Conference on Infor- Trans. Graph. 31(6), 136 (2012)
mation Visualization, pp. 518–523. (2003) Sorn, D., Sunisa, R.: Web page template design using A
Coyne, B., Sproat, R.: Words eye: An automatic text-to- interactive genetic algorithm. In: Computer Science
scene conversion system, ACM SIGGRAPH 2001. and Engineering Conference (ICSEC). 2013 Interna-
Proceedings of the 28th Annual Conference on Com- tional, IEEE, pp. 206–211. (2013)
puter Graphics and Interactive Techniques, Storn, R., Price, K.: Differential evolution – a simple and
pp. 487–496. (2001) efficient heuristic for global optimization over contin-
Dawkins, R.: The Blind Watchmaker. W.W. Norton, uous spaces. J. Glob. Optim. 11(4), 341–359 (1997)
New York (1986) Takagi, H.: Perspective on interactive evolutionary com-
Funaki, R., Takagi, H.: Application of gravity vectors and puting. J. Jpn. Soc. Artif. Intell. 13(5), 692–703 (1998)
moving vectors for the acceleration of both differential Takagi, H.: Interactive evolutionary computation: Fusion
evolution and interactive differential evolution. Int. of the capabilities of EC optimization and human eval-
Conf. on Genetic and Evolutionary Computing uation. Proc. IEEE 89(9), 1275–1296 (2001)
(ICGEC), pp. 287–290. (2011) Takagi, H., Pallez, D.: Paired Comparison Based Interac-
Garcia, H.L., Arauzo, A.A., Salas, M.L., Pierreval, H., tive Differential Evolution, Nature and Biologically
Corchado, E.: Facility layout design using a multi- Inspired Computing. pp. 475–480. India (2009)
objective interactive genetic algorithm to support the
DM, Expert Systems, pp. 1–14. (2013)
Ghannem, A., Ghizlane, B., Marouane, K.: Model
Refactoring Using Interactive Genetic Algorithm,
Search Based Software Engineering, pp. 96–110.
Springer, Berlin (2013)
Kim, Y., Mitra, N., Yan, D., Guibas, L.: Acquiring 3D 3D Selection Techniques for
indoor environments with variability and repetition. Distant Object Interaction in
ACM Trans. Graph. 31(6), 138 (2012)
Koza, J.R.: Genetic Programming: On the Programming of
Augmented Reality
Computers by Means of Natural Selection, vol. 1. MIT
press, Cambridge (1992) Nur Ameerah Abdul Halim1,2 and
Lap-Fai, Y., Sai-Kit, Y., Chi-Keung, T., Demetri, T., Tony, Ajune Wanis Ismail3
F.C., Stanley, O.: Make it home: Automatic optimiza- 1
tion of furniture arrangement. ACM Trans. Graph.
Mixed and Virtual Reality Research Lab,
30(4), 86 (2011) Vicubelab, Johor Bahru, Malaysia
2
Miki, M., Hiroyasu, T., Tomioka, H.: Parallel distributed School of Computing, Faculty of Engineering,
interactive genetic algorithm. Proc. Jpn. Soc. Mech. Universiti Teknologi Malaysia, Johor, Malaysia
Eng. Des. Syst. Conf. 13, 140–143 (2003) 3
Miki, M., Yamamoto, Y., Wake, S., Hiroyasu, T.: Global
Mixed and Virtual Reality Research Lab,
asynchronous distributed interactive genetic algorithm. Vicubelab, School of Computing, Faculty of
In: Systems, Man and Cybernetics. IEEE International Engineering, Universiti Teknologi Malaysia,
Conference on, vol. 4, pp. 3481–3485. IEEE, Taipei (2006) Johor Bahru, Malaysia
Mok, T.P., Wang, X.X., Xu, J., Kwok, Y.L.: Fashion sketch
design by inter- active genetic algorithms. AIP Confer-
ence Proceedings, pp. 357–364. (2012)
Nan, L., Xie, K., Sharf, A.: A search-classify approach for Synonyms
cluttered indoor scene understanding. ACM Trans.
Graph. 31(6), 137 (2012)
Ono, I., Kobayashi, S., Yoshida, K.: Optimal lens design
Augmented reality; Distant object; Selection
by real-coded genetic algorithms using UND- techniques; User interaction
X. Comput. Methods Appl. Mech. Eng. 186(2),
483–497 (2000)
Parish, Y., Muller, P.: Procedural modeling of cities. Pro-
ceedings of the 28th Annual Conference on Computer
Definition
Graphics and Interactive Techniques, pp. 301–308.
ACM (2001) Selection is a prerequisite task for object manipu-
Pei, Y., Takagi, H.: Triple and quadruple comparison-based lation in 3D user interfaces. The virtual object
interactive differential evolution and differential evolu-
tion. In: Proceedings of the Twelfth Workshop on
must obey specific conditions to be considered
Foundations of Genetic Algorithms XII, pp. 173–182. the targeted object for selection. This article
ACM, Australia (2013) explores the existing selection techniques for the
52 3D Selection Techniques for Distant Object Interaction in Augmented Reality

distant object in an augmented reality games. Still, it is now being used in a variety of
environment. areas, including education, healthcare, training,
remote collaboration, maintenance assembly, and
smart manufacturing (Barrie et al. 2019; Birt et al.
Introduction 2018; Peña-Ríos et al. 2012; Richert et al. 2019;
Stretton et al. 2018). Selection for the target object
Recently, augmented and mixed reality (AR/MR) in AR and MR will allow the interaction to be
application has become more popular among more precise and therefore enable a more complex
researchers, businesses, and developers in pre- interaction within the application.
senting the information. As current technologies In this context, this article presents and dis-
have improved and are capable of AR and MR, cusses the overview of the selection techniques
more interesting and improved applications are proposed by previous researchers for the target
developed to meet users’ needs. In AR and MR, object in MR to improve the interaction between
selection has been the common task in performing the users with the target object in the environment.
interaction with virtual objects that is being super-
imposed on the real-world environment.
According to Mine (1995), selection is considered Background
when an object is being pointed and validated
among other objects available. The selection is 3D interaction techniques have been extensively
important to be precise in order to select the researched in immersive virtual environments
targeted object in the scene correctly. Therefore, using HMDs and tracking devices, including
various approach has been taken by the previous data gloves and on desktop setups with a keyboard
researchers in order to overcome this issue. Mean- and mouse (Yin et al. 2019). Bowman et al. (2004)
while, according to Yin et al. (2019), the issue of investigated the interaction techniques commonly
defining suitable and propitious 3D interaction used in 3D user interfaces and created a taxonomy
techniques is still an agile area of study. of universal tasks for interacting with 3D virtual
Milgram continuum, a reality-virtuality contin- environments: selection and manipulation of vir-
uum, has sufficient reference for the framework tual targets; navigation and path findings inside a
regarding the classification of different immersive 3D environment; giving instructions using 3D
technologies. The continuum depicted MR as a menus: text, tags, and legends are examples of
transition between the actual and virtual worlds, symbolic input. Meanwhile, several studies have
including AR with augmented virtuality explored the attributes that improve the usefulness
(AV) placed in between (Milgram and Kishimo of 3D user interfaces, mainly for desktops and
1994). The blending of real and virtual worlds near-to-eyes displays. They have proposed a gen-
creates new surroundings and visualizations, allo- eral guideline to UI developers. Better use of
wing for an immersed user experience, which depth cues, especially occlusion, shadows, and
refers to the quality of the interactions given to perspectives, as well as considering the
the user. Meanwhile, AR enables the visualization constrained angle of the viewpoint position,
of virtual objects in conjunction with real-world object being in contrast with the surroundings,
scenes via mobile devices such as smartphones and other factors, are included among the
and tablets, including AR glasses and head- guidelines.
mounted displays (HMD) (Silva 2018). Therefore, accurate selection plays an impor-
With the current advanced technology, AR tant part in providing education and information
applications in mobile devices are more approach- exchange among academics, companies, and
able, allowing them to bloom and enable various developers. For a more pleasant overall experi-
applications widely. In the early stage of AR, AR ence, applications like entertainment and gaming
was first used for entertainment in the form of require more interaction for the user. While for
3D Selection Techniques for Distant Object Interaction in Augmented Reality 53

commercial uses such as healthcare or business pieces and rolls the dices using their brainwaves
training, there is a need for a feeling of physical as input. With the availability of such device/
presence to be more effective with the interaction technology, more possibilities of interaction or A
and enable accurate selection. other research intention can be explored in various
Several input alternatives currently exist and fields with the implementation of AR and MR.
has been proposed by previous researchers in There are various reasons on precise selection
performing selection and further interaction with for the distant object is needed. Since AR and MR
the virtual objects such as touchscreen-based are becoming more prevalent in data or informa-
inputs, gesture-based input, inertial device-based tion presentations, it is essential to provide an
input, speech-based inputs (Lee and Chu 2018; accurate selection for better interaction between
Mossel et al. 2013; Połap 2018; Su et al. 2018; the user and the object in the system provided. To
Yusof et al. 2020). However, gesture-based input define precise selection, the time taken for the
is described as the most natural and intuitive selection to be complete should be less, enabling
approach in performing selection to interact with a higher speed for the interaction to complete.
the virtual object. Interaction with natural gestures With accurate selection, a more precise result of
in free space enables possibilities for exploiting the interaction can be provided.
the interaction’s fullness and expressive-ness, However, previous researchers have addressed
allowing the users to manage several things at several challenges on achieving selection for
once with more degree of freedoms (DoFs) and smooth interaction with the virtual object.
utilize the familiar and real actions (Argelaguet Whitlock et al. (2018) have addressed interacting
and Andujar 2013). Furthermore, interaction with distant objects in AR. Aside from that,
using natural gestures is the most familiar action pointing accuracy may be affected by variations
for the users, which mimics the actions in of in-depth perception for real and virtual objects
daily life. (Whitlock et al. 2018). Ro et al. (2019) also high-
Besides that, the brain-computer interface light the issue of selection on a distant object with
(BCI) is another field that has sparked researcher’s limitations on devices such as see-through HMD
interest in enabling a new way of interaction in the (e.g., Hololens), where it requires the object to be
AR/MR environment (Si-Mohammed et al. within reach (limit distance reachable by the user)
2017). BCI is a system that takes a biosignal, for it to be selected. Other challenges are that
measured from a person, and predicts (in real- pointing to obscured target objects in noisy sur-
time) certain aspects of the person’s cognitive roundings may need nonlinear visual and spatial
state. BCIs could particularly contribute to mapping (Olwal and Feiner 2003). Therefore,
AR-based systems interaction means, especially previous researchers have proposed various
on visual selection tasks that can be done, as an approaches to address these issues.
example, via steady-state visually evoked poten- The following section discusses the
tials (SSVEP) or P300 (Gergondet et al. 2011). approaches from previous researchers on the
With the high computational power of current selection techniques for a target object in
computers and display devices, the goal to build AR/MR to enable and improve the interaction
a real-time adaptive system for AR has been made between the user and the target object in the
a lot more reachable (Vortmann 2019). Recently, environment.
the NextMind (www.next-mind.com) company
has developed an affordable brain device with a
novel type of SSVEP stimuli that is less straining 3D Selection Techniques
on the eye. For example, Pietroszek et al. (2021)
has utilized NextMind within their experiment According to the study by Argelaguet and
with AR HMD to allow for derogating experi- Andujar (2013), interaction is more physically
ence, where the player picks up and moves the challenging within the 3D environment and may
54 3D Selection Techniques for Distant Object Interaction in Augmented Reality

obstruct user tasks by requiring greater skill. Their both. However, the most common approach is
study has identified two main 3D selection meta- often by confirming selection with a button.
phors: virtual hand and virtual pointing. The vir- A study by Mossel et al. (2013) has presented
tual hand is a common metaphor for interacting DrillSample, which addresses the requirements
with objects in an immersive environment. It for precise selection with a one-handed handheld
enables interactions with objects, which are interaction in a dense AR surrounding. Other than
within the arm’s reach of the user. With virtual that, they also address issues of disambiguation
hand metaphor, objects are being interacted with and selection in a dense mobile AR environment
in a manner that is similar to touching and grasp- for highly occluded objects or with high visual
ing in real life. Virtual hands are often resemblance. In their approach to perform precise
implemented as a one-to-one mapping of selection in touch-based interaction, a single
the user interaction area (the region of space that touch action on the screen causes the coordinates
the user may reach, such as within arms reach of of the 2D screen to be projected back into the 3D
the user) and the control area (the space reachable space. It will trigger the ray to be cast from the
by a virtual hand in the immersive system). On the virtual camera’s position in the direction of the 3D
other hand, virtual hands are typically less accu- point into the handheld AR scene. The handheld
rate and result in delayed task completion than the device 6DOF stance, which is generally provided
alternative technique, virtual pointing, owing to in handheld AR, may be used to estimate the
existing technology implementations. direction.
Although the virtual hands technique is well- Meanwhile, Ramos et al. (2015) presented
known at the beginning since it directly maps the GyroWand, a 3D interaction with the implemen-
virtual tasks identically with the actual task, tation of raycasting technique in independent AR
which results in a more natural interaction. HMDs. The inertial measuring unit (IMU) on a
However, current researches have demonstrated handheld controller captures relative rotation data
that tackling real-world bodily constraints has utilized in the raycasting technique.
significant advantages, such as allowing the Another study by Ro et al. (2019) has proposed
selection on the out of reach objects to be a new user interface that utilizes raycasting using
performed by the user using raycasting, a the depth-variable method for selection on remote
pointing technique. According to a number of objects in an AR environment. The ray-depth
user analyses in the literature, virtual pointing information is included in addition to sensor infor-
techniques generally result in higher selection mation because ray-depth must be adjusted for
efficacy than rival 3D selection metaphors pointing in mid-air in a 3D space. Although
(Bowman et al. 2004). GyroWand is also utilizing IMU on a handheld
controller to capture the relative rotation data for
Ray Pointing the raycast for the selection, however unlike the
The raycasting technique can be implemented for AR pointer, it did not have depth information. The
touch-based interaction, gesture-based interac- GyroWand will go into the Disambiguation state,
tion, or hybrid interaction. Raycasting can also where users refine the actual target they want to
be implemented in various aspects, including select in-depth. Aside from that, the GyroWand
how the ray is being controlled. Raycasting was able to employ the ray as a line and convert it
extends the user’s reach by pointing an extended to a volumetric form if required. It will improve
ray outwards, starting at the user’s hand or the precision while choosing items at a distance.
designed cursor (Auteri et al. 2013). A starting
point and a trajectory are required to control the Flexible Pointer
ray. These two values may be obtained by moni- Apart from that, Olwal and Feiner (2003) have
toring the orientation and position of a controller, suggested a flexible virtual pointer that permits a
the hand of the user, head, or a combination of user in a 3D world to more readily point towards
3D Selection Techniques for Distant Object Interaction in Augmented Reality 55

completely or partly covered targets and further Hybrid Technique

clearly identify things to other users. The flexible Meanwhile, a study by Kim and Lee (2016) pro-
pointer reduces ambiguity by avoiding concealing posed a hybrid interaction technique that com- A
objects that might have been chosen if using tra- bines touch-based with real hand gesture-based
ditional raycasting techniques. The flexible interaction to perform a selection for further inter-
pointer is implemented as a quadratic Bézier action with objects in the environment. The selec-
spline, with three points (position, endpoint, and tion of the AR object is completed with a touch
control point) in 3D space controlling its location, gesture, while a real hand gesture is enabled for
length, and curvature. manipulation of the selected objects directly and
interactively. As a result of the hybrid interaction,
Go-Go Gesture Technique the user will be able to perform interactions such
Other than raycasting, Go-Go is another basic as manipulation on the AR targets in actual 3D
approach to perform selection on the target object space rather than 2D space.
(Poupyrev et al. 1996). Although Go-Go performs Due to calibration flaws and tracking system
poorly in dense settings when picking objects, it constraints, eye gazing may be quicker and more
can readily choose entirely occluded objects in a ergonomic for selecting the target object, but it has
single step. In the Go-Go technique, the selection poor precision. As a result, researchers have
is done by extending the virtual hand towards the developed multimodal techniques that improve
target object. A study by Jung et al. (2017) has eye-gaze selection using a supplementary input
proposed BoostHand in which they modify The modality that will refine the selection process.
Go-Go interaction technique into one that can be For example, the target selection technique pre-
switched using easy trigger actions. The Go-Go sented by Kytö et al. (2018) implements selection
interaction approach utilizes a preset mapping techniques with a multimodal approach, using
function to easily manipulate virtual objects with eye-gaze or head-motion. To achieve pinpoint
a virtual hand avatar. The interaction is intuitive precision, these techniques use rough pointing
and natural since it makes use of the natural hand, selection accompanied by a secondary, local refin-
requires little training, and performs the functions ing action. Each primary selection mode is able to
of selecting (grabbing) and manipulating combine with any refinement method. Their
(translation, rotation, and scale) in real-time. research is limited to 2D surfaces, such as those
A study by Yin et al. (2019) has addressed the implemented in selecting the menu, interactive
issue of selecting occluded target objects with visualization, and in-situ CAD application.
touch-based interaction. Touch-based interaction Furthermore, multimodal input is seen as a way
is one of the most natural and attractive input to increase the interactions of virtual and real-world
forms for handheld interfaces because it allows components. Because it allows for simultaneous
users to interact directly with the target object. interaction within the physical and virtual worlds,
Other than that, they also address the issue of the the interaction approach is suitable for AR applica-
small area for interaction, as the display screen is tions (Ismail and Sunar 2015). Meanwhile, speech
limited. Therefore, their study discusses four input is often implemented as a hybrid technique
novel 3D selection techniques with Go-Go and combined with gesture-based or gaze-based for
Raycasting techniques as the baseline to over- interaction with objects in AR and MR environ-
come these issues. After pointing at an object at ments. Speech has been shown to be an ideal
a distance (laser pointer metaphor) and choice for abstract actions, such as selecting a
intersecting the ray, the user may select the object device among many devices or interacting with
using other actions (button selection, gesture, or multi-object (Piumsomboon et al. 2014; Zaiţi
speech) (Ro et al. 2019). However, it becomes less et al. 2015). The advantages and drawbacks of the
intuitive and natural than the real hand manipula- 3D selection techniques of related researches
tion method. discussed in this article are summarized in Table 1.
56 3D Selection Techniques for Distant Object Interaction in Augmented Reality

3D Selection Techniques for Distant Object Interaction in Augmented Reality, Table 1 The summary of 3D
selection techniques of related researches
Researchers Proposed selection technique Advantage Drawback
Ro et al. AR Pointer – Able to specify Easy to learn for the user and The error rate is higher for
(2019) ray direction and adjustable fast completion time object rotation angle error (arc
ray’s depth to calculate the Able to register an object at a ball rotation method)
ray’s endpoint specific mid-air point in three
dimensions
Yin et al. Present four novels AR Although tested in a handheld The technique presented was
(2019) selection techniques for AR environment, it can also be only implemented for
handheld mobile devices. applied in other dense virtual selection using touch-screen
environments or other input
interactive interfaces that use
touch screen input
Kytö et al. Multimodal pointing Include two primary input Gaze calibration was typically
(2018) techniques for wearable AR modes, eye gaze and head less accurate for extreme
interfaces. Presented two pointing, and combine with a upper and lower targets, and
prototypes – GazeBrowser and refinement provided by a that overall accuracy tended to
SmartPupil handheld device degrade over time
Jung et al. BoostHand – a freehand, The user control space can be Tracking errors by Leap
(2017) distance-free object- expanded by emphasizing Motion can largely influence
manipulation system that intuitive and straightforward the system’s performance
supports simple trigger interaction
gestures using Leap Motion Able to provide better
performance at long distances
with fewer movements
Kim and Lee Hybrid interaction of touch- Enable natural and intuitive One hand needs to hold the
(2016) based and real hand gestures interaction with 3D AR objects device, which could lead to
for direct interaction with AR on handheld and mobile fatigue.
objects devices
Ramos et al. GyroWand – interprets the Initiate raycasting from other Does not investigate the
(2015) relative rotational values using spatial coordinates (chin is a performance and comfort of
a state machine and provide good candidate as the origin for the proposed approach against
three new disambiguation the raycast) the natural user interface
mechanisms
Piumsomboon G-SIAR – multimodal G-Shell required less effort, There are limits in the tracking
et al. (2014) interaction technique of frustration, mental, physical, resolution and speech and
gesture and speech and temporal demand, and gesture recognition accuracy
provided higher performances
Auteri et al. Presented a hybrid selection Enumerated different design The user felt a lack of control
(2013) technique that combines decisions involved blending when using PRISM
PRISM and Go-Go PRISM and Go-Go, resulting
in a huge improvement in task
precision
Mossel et al. DrillSample – two-step Develop guidelines to enable It was only tested for selection
(2013) selection technique with only precise selection in a single- on the occluded object within
one finger input in a dense handed handheld AR close range
handheld environment (high environment
visual similarities) The performance study clearly
revealed the strength of the
DrillSample technique
compared to related work in
the study
3D Selection Techniques for Distant Object Interaction in Augmented Reality 57

Conclusion Birt, J., Stromberga, Z., Cowling, M., Moro, C.: Mobile
mixed reality for experiential learning and simulation in
medical and health sciences education. Information.
Over the last decade, the topic of selection on the 9(2), 31 (2018) A
target object has been in the interest of researchers Bowman, D., Kruijff, E., LaViola Jr., J.J., Poupyrev, I.P.:
and developers from various fields of studies in 3D User interfaces: theory and practice, CourseSmart
AR and MR. It shows that selection still plays a eTextbook. Addison-Wesley (2004)
Gergondet, P., Druon, S., Kheddar, A., Hintermüller, C.,
vital role in interaction with the target object in Guger, C., Slater, M.: Using brain-computer interface to
AR or MR environment. steer a humanoid robot. IEEE ROBIO., 192–197 (2011)
Selection as the basic task in performing inter- Hincapié-Ramos, J. D., Özacar, K., Irani, P. P., Kitamura,
action with the virtual object must be as precise as Y.: Gyro wand: IMU-based raycasting for augmented
reality head-mounted displays. In: SUI 2015 – Pro-
possible. With preciseness, it will allow a faster ceedings of the 3rd ACM Symposium on Spatial User
and smoother interaction between the user and the Interaction, pp. 89–98 (2015)
virtual content. Selection techniques can be pro- Zaiţi, I.-A., Pentiuc, Ş.-G., Vatavu, R.-D.: On free-hand TV
vided in several techniques such as raycasting, control: experimental results on user-elicited gestures
with Leap Motion. Pers. Ubiquit. Comput. 19(5),
Go-Go, or hybrid techniques. Raycasting is a 821–838 (2015)
technique for aiming objects from afar with limits Ismail, A.W., Sunar, M.S.: Multimodal fusion: Gesture and
for object manipulations like translation and rota- speech input in augmented reality environment. In:
tion. Meanwhile, Go-Go produces better results Ismail, A.W., Sunar, M.S. (eds.) Computational intelli-
gence in information systems, vol. 331. Springer Inter-
when selecting the occluded target object. How- national Publishing (2015)
ever, a hybrid technique such as combining touch- Jung, W., Cho, W., Kim, H., Woo, W.: BoostHand :
based and gesture-based for selection and further DDistance-free Object Manipulation System with
interaction with the target object could offer an Switchable Non-linear Mapping for Augmented Real-
ity Classrooms. In: Adjunct Proceedings of the 2017
intuitive and regular interaction in the AR sur- IEEE International Symposium on Mixed and Aug-
roundings. While in the MR interface, occlusion mented Reality, ISMAR-Adjunct 2017, pp. 321–325.
becomes an issue when real hands have covered (2017)
the virtual object. As an advanced selection, Kim, M., Lee, J.Y.: Touch and hand gesture-based interac-
tions for directly manipulating 3D virtual objects in
Aladin et al. (2020) has explored the occlusion- mobile augmented reality. Multimed. Tools Appl.
hand gesture in MR to perform a precise selection 75(23), 16529–16550 (2016)
technique using the real hand gesture in MR. Kytö, M., Ens, B., Piumsomboon, T., Lee, G. A.,
Billinghurst, M.: Pinpointing: Precise head- and eye-
based target selection for augmented reality. In: Con-
ference on Human Factors in Computing Systems –
References Proceedings, 2018-April, pp. 1–14. (2018)
Lee, C. J., Chu, H. K.: Dual-mr: Interaction with mixed
Aladin, M. Y. F., Ismail, A. W., Ismail, N. A., Rahim, reality using smartphones. In: Proceedings of the 24th
M. S. M.: Object selection and scaling using multi- ACM Symposium on Virtual Reality Software and
modal interaction in mixed reality. In: IOP Conference Technology (pp. 1-2) (2018)
Series: Materials Science and Engineering, vol. Mine, M.: ISAAC: A virtual environment tool for the
979, no. 1, p. 012004. (2020) interactive construction of virtual worlds. (1995)
Argelaguet, F., Andujar, C.: A survey of 3D object selec- Milgram, P., Kishimo, F.: A taxonomy of mixed reality.
tion techniques for virtual environments. Comput. IEICE Trans. Inf. Syst. 77(12), 1321–1329 (1994)
Graph. (Pergamon). 37(3), 121–136 (2013) Makhataeva, Z., Varol, H.A.: Augmented reality for robot-
Auteri, C., Guerra, M., Frees, S.: Increasing precision for ics: a review. Robotics. 9(2), 21 (2020)
extended reach 3D manipulation. Int. J. Virtual Reality. Mossel, A., Venditti, B., Kaufmann, H.: Drillsample: pre-
12(1), 66–73 (2013) cise selection in dense handheld augmented reality
Barrie, M., Socha, J. J., Mansour, L., Patterson, E. S.: environments. In: Proceedings of the Virtual Reality
Mixed reality in medical education: a narrative litera- International Conference: Laval Virtual, pp. 1–10.
ture review. In: Proceedings of the International Sym- (2013)
posium on Human Factors and Ergonomics in Health Olwal, A., Feiner, S.: The flexible pointer: an interaction
Care, vol. 8, no. 1, pp. 28-32. Sage/Los Angeles: SAGE technique for selection in augmented and virtual reality.
Publications (2019) Uist’03. 03, 81–82 (2003)
58 3D Simulation

Peña-Ríos, A., Callaghan, V., Gardner, M., Alhaddad, Yusof, C. S., Halim, N. A. A., Nor’A, M. N. A., Ismail,
M. J.: Remote mixed reality collaborative laboratory A. W.: Finger-Ray Interaction using Real Hand in
activities: Learning activities within the InterReality Handheld Augmented Reality Interface. In: IOP Con-
Portal. In: 2012 IEEE/WIC/ACM International Confer- ference Series: Materials Science and Engineering,
ences on Web Intelligence and Intelligent Agent Tech- 979(1) (2020)
nology, vol. 3, pp. 362-366. IEEE (2012)
Piumsomboon, T., Altimira, D., Kim, H., Clark, A., Lee,
G., Billinghurst, M.: Grasp-Shell vs gesture-speech:
A comparison of direct and indirect natural interaction
techniques in augmented reality. In 2014 IEEE Interna- 3D Simulation
tional Symposium on Mixed and Augmented Reality
(ISMAR), pp. 73-82. IEEE (2014)
Połap, D.: Voice control in mixed reality. In: Proceedings
▶ Nursing Education Through Virtual Reality:
of the 2018 Federated Conference on Computer Sci- Bridging the Gap
ence and Information Systems, FedCSIS 2018, 15(1),
pp. 497–500 (2018)
Poupyrev, I., Ichikawa, T., Billinghurst, M., Weghorst, S.:
The Go-Go Interaction Technique : Nonlinear Map-
ping for Direct Manipulation in VR. pp. 79–80 (1996) 3D Skills
Pietroszek, K., Agraraharja, Z., & Eckhardt, C.: The Royal
Game of Ur: Virtual Reality Prototype of the Board ▶ Training Spatial Skills with Virtual Reality and
Game Played in Ancient Mesopotamia. In 2021 IEEE
Conference on Virtual Reality and 3D User Interfaces
Augmented Reality
Abstracts and Workshops (VRW), pp. 647-648. IEEE
(2021)
Richert, A., Mai, V., Mengen, H., Wolf, S.: Mixed Reality
Games in Engineering Education. In 2019 5th Experi-
ment International Conference (exp. at’19),
3D User Interfaces
pp. 365-370. IEEE (2019)
Ro, H., Byun, J.H., Park, Y.J., Lee, N.K., Han, T.D.: AR ▶ Natural Walking in Virtual Reality
pointer: Advanced raycasting interface using laser
pointer metaphor for object manipulation in 3D aug-
mented reality environment. Appl. Sci. 9(15),
3078 (2019)
Silva, I. C. S.: Mixed reality and immersive data visualiza- 3D Visualization
tion. In: Encyclopedia of Computer Graphics and
Games, pp. 1–8 (2018)
Si-Mohammed, H., Sanz, F. A., Casiez, G., Roussel, N.,
▶ Technologies for the Design Review Process
Lécuyer, A.: Brain-computer interfaces and augmented
reality: a state of the art. In Graz Brain-Computer
Interface Conference. (2017)
Stretton, T., Cochrane, T., Narayan, V.: Exploring mobile
mixed reality in healthcare higher education: a system-
3D Visualization Interface for
atic review. Res. Learn. Technol. 26, 2131–2131 (2018) Temporal Analysis of Social
Su, G.E., Sunar, M.S., Andias, R., Ismail, A.W.: An inertial Media
device-based user interaction with occlusion-free
object handling in a Handheld Augmented Reality.
Int. J. Int. Eng. 10(6), 159–168 (2018)
Masahiko Itoh
Vortmann, L. M.: Attention-driven interaction systems for Institute of Industrial Science, The University of
augmented reality. In: 2019 International Conference Tokyo, Tokyo, Japan
on Multimodal Interaction, pp. 482-486 (2019)
Whitlock, M., Harnner, E., Brubaker, J. R., Kane, S.,
Szafir, D. A.: Interacting with distant objects in aug-
mented reality. In: 25th IEEE Conference on Virtual Definition
Reality and 3D User Interfaces, VR 2018 – Proceed-
ings, June, 41–48 (2018) 3D visualization interface for temporal analysis
Yin, J., Fu, C., Zhang, X., Liu, T.: Precise target selection
techniques in handheld augmented reality interfaces.
of social media is the interface for visual analyt-
IEEE Access. 7, 17663–17674 (2019) ics of various types of time varying media
3D Visualization Interface for Temporal Analysis of Social Media 59

contents using 3D information visualization images (Kitsuregawa et al. 2008; Chi et al. 1998;
techniques. Chi and Card 1999; Toyoda and Kitsuregawa
2005; Kehoe and Gee 2009). A
In this entry, I will introduce three kinds of 3D
Introduction information visualization systems for analyzing
temporal changes in: (i) link structure,
Social media such as blogs and microblogs has (ii) textural contents, and (iii) image contents on
become popular. It enables us to easily and rapidly social media.
publish information on our personal activities,
interests, and opinions through writing document,
creating links to other information resources, and State-of-the-Art Work for Visualizing
providing images and/or movies. It dynamically Temporal Changes in Social Media
reflects real movements in society. Contents
Many organizations have collected and
archived social media contents over the long Visualization for Temporal Changes in Link
term. Time series of archived data enable us to Structure
analyze temporal changes in trends in social We first introduce an interactive 3D visualization
media that reflect both real and virtual activities. system for the time series of web graphs (Itoh
Visual analytics for extracting trends and read- et al. 2010). It is to enable us to examine the
ing stories from time sequential data sets are evolution of web graphs by comparing multiple
important research domains. There has been graphs that have different timings and topics. To
much research on analyzing temporal changes in accomplish the system, it utilized interactive 3D
trends on social media through visualizing link components called TimeSlices that are 2D planes
structures, results of text analysis, or flows of to visualize web graphs in a 3D environment. We

3D Visualization Interface for Temporal Analysis of Social Media, Fig. 1 Example for visualizing changes in link
structure on blogs related to the term “working poor”
60 3D Visualization Interface for Temporal Analysis of Social Media

can interactively add new TimeSlices along the appeared after the “working poor II” had been
timeline, and they can manipulate them to animate broadcast. We can find that influencers shifts in
web graphs. Visualized web graphs on TimeSlices focus from the official pages of “working poor” to
are snapshots of different timings. “working poor II.”
Figure 1 shows the example for visualizing
changes in link structure on blogs related to the Visualization for Temporal Changes
term “working poor.” First peak appeared after the in Textural Contents
TV program called the “working poor” had been We next introduce an interactive 3D visualization
broadcast. Most blogs were linked to the official system for exploring temporal changes in
page of the TV program. The second peak bloggers’ activities and interests through

3D Visualization Interface for Temporal Analysis of Social Media, Fig. 2 Example for visualizing changes in
textural contents on blogs for comparing marketing effect of two telcos
3D Visualization Interface for Temporal Analysis of Social Media 61

visualizing phrase dependency structures (Itoh to Telco B” in most months by observing changes
et al. 2012). To accomplish the system, it utilizes in the structure and frequencies for events. (ii) We
two 3D components such as TimeSlices and can expand nodes related to “announce” and A
TimeFluxes. TimeFluxes enable us to visualize “release” to find details on announcements and
temporal changes in the attribute values of partic- products that were released, and we then find
ular nodes at every timing. The system visualizes that “Telco A” announced a “new price plan” in
dependency structures of phrases as a unified tree the first peak and released “product A” in the
representation in TimeSlices and enables us to second peak.
interactively navigate to the detailed information
by expanding nodes in the tree representation. Visualization for Temporal Changes in Image
Sliding operation for the TimeSlices along the Contents
timeline indicates changes in the structure and We finally introduce a 3D system for visualizing
frequencies of dependency relations. To compare visual trends on social media that chronologically
different timings and topics side by side, it pro- displays extracted clusters of images on blogs
vides multiple 2D planes. It also visualizes (Itoh et al. 2013). The system first adopts a histo-
changes in the frequencies of dependency rela- gram of images by stacking them on a timeline to
tions by using TimeFluxes. visualize the flow of various images at each timing
Figure 2 shows the example for visualizing to visually analyze trends. This design enables us
changes in textural contents on blogs for compar- to find the timing for the beginning of the topic,
ing marketing effect of two telcos. The upper changes in trends for the topic, bursting points,
TimeSlice shows a topic for “Telco A,” while the and a lifetime of the trends. Secondly, it arranges
lower one shows a topic for “Telco B.” (i) We can multiple histograms of images in a 3D space to
recognize events related to “change/switch to visualize images on different aspects. This design
Telco A” are more popular than “change/switch allows us to observe different situations between

3D Visualization Interface for Temporal Analysis of Social Media, Fig. 3 Example for visualizing changes in
clustered images related to “Prime Minister Hatoyama” extracted from blogs
62 3D-Rendered Images and Their Application in the Interior Design

different topics, sequences of trends, and events Toyoda, M., Kitsuregawa, M.: A system for visualizing
with the same timing on different topics. and analyzing the evolution of the web with a time
series of graphs. In: Proceedings of HYPERTEXT’05,
Figure 3 shows the example for visualizing pp. 151–160 (2005)
clusters of images related to “Prime Minister
Hatoyama” extracted from blog based on visual,
textual, and chronological similarities. The top
20 clusters are arranged from front to back 3D-Rendered Images and
according to their rankings. Images are aggre- Their Application in the
gated per week. We can read stories about Interior Design
“Prime Minister Hatoyama” by exploring the
movements of topics. Petyo Budakov
New Bulgarian University, Sofia, Bulgaria

Conclusion
Synonyms
This entry has introduced the 3D visualization
systems for analyzing social media that utilized Depth of field; Interior design; Length of view;
one dimension in a 3D space as a timeline. Lens; Real estate photography; Render; Virtual
Although they independently visualized temporal camera; V-ray, 3D studio max
changes in link structures, results of text analysis,
and image clustering for single medium, we can
combine these contents and mechanisms to con- Definitions
struct integrated 3D visualization systems for
intermedia analysis. Aperture This is the hole through which light
enters the camera. Its size can be
changed to control the brightness of
References the light allowed through to the
image sensor.
Chi, E.H., Card, S.K.: Sensemaking of evolving web sites
using visualization spreadsheets. In: Proceedings of Depth of It determines the depth of field as a
InfoVis’99, pp. 18–25 (1999) field range of scene depths that appear
Chi, E.H., Pitkow, J., Mackinlay, J., Pirolli, P., Gossweiler, (DOF) focused in an image.
R., Card, S.K.: Visualizing the evolution of web ecol- Field of It illustrates what angle is captured
ogies. In: Proceedings of CHI’98, pp. 400–407 (1998)
Itoh, M., Toyoda, M., Kitsuregawa, M.: An interactive view by the lens. The size of the FOV is
visualization framework for time-series of web graphs (FOV) designated in mm.
in a 3D environment. In: Proceedings of iV 2010, Rendering The process of image synthesis by
pp. 54–60 (2010) simulating light environment. The
Itoh, M., Yoshinaga, N., Toyoda, M., Kitsuregawa, M.:
Analysis and visualization of temporal changes in rendering process is performed by
Bloggers’ activities and interests. In: Proceedings of render plugins.
PVis 2012, pp. 57–64 (2012) Shutter The shutter speed setting determines
Itoh, M., Toyoda, M., Kitsuregawa, M.: Visualizing time- how long the shutter remains open
varying topics via images and texts for inter-media anal-
ysis. In: Proceedings of iV 2013, pp. 568–576 (2013) to expose the image sensor, e.g., it
Kehoe, A., Gee, M.: Weaving web data into a diachronic controls the length of the exposure.
corpus patchwork. Lang. Comput. 69(1), 255–279 V-ray It is a render plugin which adds
(2009) some additional features to the
Kitsuregawa, M., Tamura, T., Toyoda, M., Kaji, N.: Socio-
sense: a system for analysing the societal behavior from existing software such as: 3D Studio
long term web archive. In: APWeb; LNCS, vol. 4976, Max, Maya, Blender, Rhynoceros
Springer, pp. 1–8 (2008) 3D.
3D-Rendered Images and Their Application in the Interior Design 63

V-ray The camera is the tool that captures could be without any distortions or including
Physical virtual 3D image/s. some redundant details.
Camera In order to explore the application of A
3D-rendered images in the interior design is
Introduction important to explain the 3D rendering as a process.
In the real world, the light sources emit photons
Many interior designers aim to improve the effec- that travel in certain direction until they interact
tiveness of their workflow by reducing the work- with a surface. When the photon hits a side, it
load they spend to prototype certain products and might be absorbed, reflected, or transmitted.
to dramatically increase the “wow” effect of their Some of the photons that hit the retina of the
final output. The creation of a “dream home” is a viewer are converted into a signal that is perceived
major goal to every practitioner whose projects by the brain and thus – creating an image. The
reflect the clients living preferences by lining up creation of an image which is based on the same
shapes, forms, color palette, finishes, textures, and interaction between photons and a 3D environ-
lights. Our fast-paced digital life has changed the ment might be simulated on the computer. The
consumer’s perception and requirements by mak- process of image synthesis by simulating light
ing it much more sophisticated. The traditional environment is called rendering (Walia 2010).
two-dimensional hand-drawing sketches provide An array of research has been conducted, focused
certain restrictions regarding the client’s imagina- on the following problems:
tion by not being able to realistically recreate the
real space. • How to achieve high quality rendering that
However, the hand-drawing rendering induces the feeling of photo realism?
(sketches) laid the basis of the contemporary 3D • How to reduce the rendering time?
design and they facilitate the interior designers in
building the main foundation of their projects
Rendering Techniques
(Wang 2002, p.12): “The real understanding of
pencil sketching goes beyond the “state of the
However, the rendering techniques could be clas-
art” – it is about creative seeing, such as how to
sified into two main approaches (Verma and Walia
isolate or highlight details in a complex visual
2010):
scene.” Sketching plays a significant role in
designer’s eye-hand coordination (Wang 2002).
• Geometry-based rendering – the illumination
People naturally see, observe, and create their pro-
of a scene has to be simulated by applying
jects and the pencil sketching enable the designers
shading models. This method requires more
to conceive some fresh concepts and to come up
computing power.
with innovative ideas. Thus, sketching as a
• Image-based rendering – using images as a
method could be deemed as a vital prerequisite
modeling primitive.
stage of the workflow rather than an “obsolete”
and useless approach.
Recently conducted research study, published Principles in the Interior Design
by marketsandmarkets.com argues the 3D render-
ing is expected to grow in a frenetic rate: from The initial stage of project planning includes plan-
USD 1.06 billion in 2017 to accumulate USD 2.92 ning of partitions, furniture layouts, finishes,
billion by 2022, at a compound annual growth rate etc. This process is based on following some spe-
(CAGR) of 22.40% during the forecast period. cific rules which might seem quite universal for all
Many experts are convinced that the usage of 3D types of design, e.g., graphic design, fashion
visualizations refine the interior design projects as design, etc. These rules are determined by Erica
a real showcase of what actually the future home Swanson as: balance, emphasis, proportion and
64 3D-Rendered Images and Their Application in the Interior Design

scale, harmony and unity, the rhythm (Swanson By using 3D renders, interior designers would
2010). be able to simulate how different furniture, finish,
wall textures interact with the environment and
• Balance the main light sources.
• Emphasize 3D modeling and texturing enable interior
• Proportion and scale designers easily adjust the space, colors, lights
• Unity and harmony sources, and textures according to clients’ prefer-
• Rhythm ences. Furthermore, the 3D modeling and render-
ing allow relatively quick customization and
Experienced designers often break these rules modifications after receiving the clients review.
or create their own in terms of what they insist to This process significantly increases the effective-
achieve. However, in order to illustrate the final ness of the interior design by facilitating the rela-
outcome, the real estate marketing utilizes the tionships between designers and clients.
photography as an appropriate manner to present However, rendering algorithms are rather com-
the completed property and its features. The inves- plicated and the accomplishment of high realistic
tigator illustrates below the ten most essential and 3D images often could be a highly time-con-
vastly popular principles in real estate photogra- suming process. It might require some special
phy, determined by Lohrman (2014): hardware features, based on the fact that rendering
time is highly dependent on scene complexity. As
1. Clearly illustrate the features of the estate. a consequence, interior designers should be very
2. Use wide-angle lens. careful when choosing the right render engine.
3. Remove the clutter shadow.
4. Take a primary exterior shot.
5. Illustrate the interior as light and bright – it What Is V-Ray
refers to the exposure control settings.
6. The property’s walls must look straight. V-ray is a render engine, which means it adds
7. Vertical and horizontal lines must be also some extra features to the existing software such
straight. as: 3D Studio Max, Maya, Blender, Rhynoceros
8. Avoid the brightness of the windows which 3D, etc. by improving the render speed and qual-
causes distraction. ity. This render engine aims to create photo-
9. Simplify the color palette. realistic images for the needs of the interior and
10. Present the photo in the most effective way. exterior design, product design, and animation.
This entry exams the V-ray as a plugin to 3D
The usage of 3D software could facilitate the Studio Max and all examples are produced in 3D
implementation of these principles by assuring a Studio Max. However, the Render Setup menu
set of tools that allows designers create, modify, allows to set up the right render – Mental Ray,
insert objects and textures, adjust the lighting V-ray, or V-ray RT. The last one enables its users
scheme, point of view, and render the final to utilize both CPU (central processing unit) and
image in a resolution that is suitable for different GPU (graphic processing unit) hardware acceler-
purposes. Moreover, the 3D modeling and render- ation and to see the update to the render while they
ing could provide clients with more realistic per- edit the scene in real time (lights, materials, tex-
spective of the space, to let them decide whether tures, camera, etc.).
certain furniture fits with the space or not as well Regarding the functionality of V-ray, this ren-
as to identify certain flaws. Once these faults are der engine provides users with its own lights and
identified, the designers could discuss with their camera.
clients how to resolve the problem and come up V-ray lights include a set of lights that are
with an appropriate solution without losing time specially designed for application in 3D Studio
later. Max. The standard Ray Traced Shadow and
3D-Rendered Images and Their Application in the Interior Design 65

Shadow Map shadows are optimized to work with instance, painters use different types of composi-
V-ray. However, V-ray offers its users with a V-ray tions, colors, shapes, and forms in order to convey
Light Meter Utility that enables them to evaluate emotions and induce feelings. Photographers and A
the lighting in a scene and to make cinematographers use compositions, point of
certain adjustments in order to refine the view, exposure, and motion. These settings help
lighting. Depending on the effect designers the authors to create their visual language. The
want to create, they could use some of the follow- camera is an essential and powerful tool for all
ing lights, determined by CHAOS GROUP interior designers and animators which they use to
(http://docs.chaosgroup.com): visualize their projects. Taking advantage of its
wide variety of settings, interior designers would
• Area lights – V-ray light that can be used to be able to establish a full control of the perspective
create physically accurate area lights of differ- and as a consequence to enhance the quality of
ent shapes. Designers could choose some of their final output. There are some really vital ele-
the following types of shapes: plane light, disc ments that need to be considered in order to obtain
light, sphere light, mesh light, and dome light. a better understanding of how the V-ray Physical
• A Sun light – designed to work together, V-ray Camera works and how its settings could be used
Sun and V-ray Sky reproduce accurately the for the needs of the interior designers.
real-life Sun-Sky environment. The V-ray render engine is a complex plugin
• Photometric lights – it enables the users to that offers an infinite number of settings enabling
utilize .ies files – it is a file which contains users enhance the realism and avoid many prob-
specific features of a real-world light bulb. lems such as: noise, flickering, etc. Most of the
• Ambient light – this light does not come from V-ray menu options are highly dependent on each
specific directions and it could be used to other. The researcher will explore in-depth the
reproduce or simulate global illumination. most important V-ray Physical Camera abilities,
rather than explaining the most essential settings
The accurate usage of the above-explained in a top to bottom manner. In order to establish
lights would allow interior designers establish clarity, the author’s considerations are focused on
a full-fledged 3D scene and enable them to cus- the camera as an important element that has a
tomize precisely the lighting effects they intend to significant contribution to the quality of the final
create. render. Furthermore, the camera constitutes just
part of a complex render algorithm which is based
on the settings, provided by the main V-ray menu.
V-Ray Physical Camera This menu provides a large number of abilities
that improves the rendering and its performance,
However, a 3D photo realism could not be possi- such as: Global Illumination (GI), Image Sampler,
ble without using an appropriate camera. The Adaptive DMC, etc.
camera is the tool that captures the image/s and Thus, a parallel will be done between the real-
enables the cameraman to tell stories. Knowing world camera and the one provided by V-ray in
how to manipulate and control the camera settings 3D Studio Max, called V-Ray Physical Camera.
will make possible to achieve all of the above- This comparison aims to identify whether the
mentioned real estate photography principles and, V-ray Physical Camera has the same abilities and
thus, to engage deeply their clients. This entry settings that the real-world digital camera has, in
makes a parallel between the features provided order to achieve and incorporate the five princi-
by the real camera and the one V-ray provides in ples listed above.
3D Studio Max. Many experts claim (Obeo 2017), the real-
The V-ray Physical Camera in 3D Studio Max estate photography has some special requirements
is a powerful tool which aims to narrate a certain to the equipment, e.g., the photographers should
story through its infinite number of tools. For be provided with a digital single lens reflex
66 3D-Rendered Images and Their Application in the Interior Design

camera (DSLR) body, tripod, flash, and a full In photography, there are certain types of
gamut of lenses. V-ray in 3D Studio Max offers lenses that enable photographers to capture inte-
all of these facilities – an effective synergy rior scenes with different field of view. Some
between the V-ray Physical Camera, V-ray Lights, widely available lenses according to the standard
and some advance render settings. In addition, 35-mm cameras are the following:
many practitioners recreate the flash effect by
aligning a V-ray white-colored light with a very • Normal lens (50 mm, 85 mm). It reproduces
small amount of multiplier to the camera. fairly the same what the human eyes see.
The real-world camera (DSLR or a mirrorless • Wide-angle lens (10–42 mm). When comparing
interchangeable lens camera) enables their users with the normal lens, it captures relatively wider
to control a number of settings, some of which are angle. For instance, small-sized bathrooms,
quite important for incorporating all of the above- bedrooms, and cellars are difficult to shoot with-
mentioned design principles. The analysis will be out having such kind of lens. Many experts and
based on the following control settings: photographers claim that all real estate look
better when using a wide-angle lens. The type
• Control the field of view of lens should be selected carefully in order to
• Controlling exposure (shutter controls and avoid distortion and inaccurate perspective.
aperture controls) • Zoom lens (100–800 mm). Nevertheless, it is
rarely used for taking photos of real estate
scenes – it captures relatively narrower angle,
Field of View comparing with the normal lens. This lens
allows the users to adjust the focus ring in
The field of view is one of the most important order to change the focal length. This process
parameters to which designers should pay atten- is known as lens breathing. However, by using
tion. It illustrates what angle is captured by the this lens, the photographers should be aware
lens and thus it enables the interior designers to that they distort the depth of field (DOF).
incorporate principles: 1, 3, 6, and 7 (Lohrman • Depth of field – There is a plethora of terms
2014) in their projects: that are tossed around DOF. However,
The field of view is dependent of two main Nagahara et al. (2008) determine the depth of
factors, explained by Douglass Kerr in his study field as a range of scene depths that appear
“Equivalent Focal Length”: focused in an image. Furthermore, the strength
of DOF is highly dependable by the aperture.
• The focal length – it determines the angle of By successfully utilizing the DOF, the real
view or, in other words, how much of the estate photographers could emphasize certain
scene will be captured. The greater the focal focal point of the living space that should be in
length, the narrower the field of view is (Kerr focus and visually appears as the most obvious
2009). point spotted in the interior.
• The camera’s format size – it refers to the size • Fish-eye lens (7–16 mm) – this lens provides an
of the film frame or with the size of the digital immensely wide angle and it distorts the straight
sensor. In general, the smaller the format size, lines into curves. The interior photographers
the smaller the field of view. Nowadays, the most utilize it when it comes to creating some special
popular digital single reflex cameras are effects or unconventional point of view.
manufactured with sensors nearly equal to those
of 35-mm film. For example, regarding the dig- A comparison between the real-world camera
ital photography, a lens is deemed to be a “normal and the V-ray Physical Camera features, which is
lens” when its focal length is approximately based on the above-mentioned characteristics, is
equivalent to its sensor diagonal size (Kerr 2009). illustrated in Table 1.
3D-Rendered Images and Their Application in the Interior Design 67

3D-Rendered Images and Their Application in the

Interior Design, Table 1 Comparison between the
Field of View (FOV), provided by the Real-World Camera
and the V-Ray Physical Camera A
Real-
world
Field of view camera V-ray Physical Camera
Depth of Yes Yes (users can specify the
field DOF depending on the
film/sensor preset chosen)
Focal length Yes Yes (default is 40 mm and
could be customized)
The Yes Yes (default is 35 mm and
camera’s could be customized)
format size
Lens Yes Yes (default is 1.0 and
breathing could be adjusted)

Exposure Control

The most important exposure controls on your

camera are the shutter speed, aperture, and ISO,
because they affect the total amount of light hit-
ting the camera sensor (Curtin 2011). These
settings are also determined as an “exposure tri- 3D-Rendered Images and Their Application in the
angle,” by photographer Barry O Carroll (www. Interior Design, Fig. 1 V-ray Physical Camera menu
bocphotography.com). basic parameters. (Source: Screenshot from the interface,
The V-ray exposure control menu (Figs. 1 3D Studio Max 2016)
and 2) allows to adjust these settings and to
choose between different modes when applying
the exposure control. This menu offers a number In Fig. 3, there is a comparison between a
of settings which are quite essential (Table 2). photo and a V-ray-rendered image. The right part
of the image is rendered with V-ray, while the left
Based on this table, the comparison of the
exposure control settings will be like this, illus- is taken by DSLR camera.
trated in Table 3: Figures 4 and 5 illustrate an interior which is
rendered with V-ray Adv.3.03
It is also important to note that this menu
enables users to adjust the white balance – quite V-ray Physical Camera, ISO: 400
essential features that photographers use to Shutter speed: 1/60, f 8, Vignetting – off. Lens:
50 mm.
remove some unrealistic colors.
The researcher illustrates below some renders Resolution: 6666  5430
produced with V-ray 3.20.03, compared with photos The image above shows the quality of
realism, produced by V-ray Physical Cam-
which are taken with real-word digital camera.
era with Wide-angle lens, 28 mm and ISO
400.
Renders All renders are produced with Vignetting: off.
By enabling this setting, some dark areas appear at
The V-ray Physical Camera productions are illus- the image corners which could be considered from
trated below. the designers as a flaw.
68 3D-Rendered Images and Their Application in the Interior Design

3D-Rendered Images and Their Application in the

Interior Design, Table 2 V-ray exposure control menu
V-ray exposure control menu
Mode Allows to choose between different
modes, e.g., photographic
Camera node Specifies certain node from which the
camera settings could be taken
Exposure It controls the exposure and enables to
value (EV) adjust the image as brighter or darker
Shutter speed The same function as the shutter has in
the digital photography. The value is
illustrated on numbers, e.g. 1/200 in
V-ray is 200
f-number Controls the aperture with the same
step as they are designated in the
photography (1.4, 2, 2.8, 5.6, 8, 11,
16, 22)
ISO It determines the film sensitivity.
Smaller value makes the render image
darker, and larger value – brighter.
Unlike the digital photography, the
large amount of ISO does not increase
the noise in the picture

3D-Rendered Images and Their Application in the

Interior Design, Table 3 Comparison between the expo-
3D-Rendered Images and Their Application in the
sure control settings provided by the Real-World Camera
Interior Design, Fig. 2 V-ray Physical Camera menu
and the V-Ray Physical Camera
(Source: Screenshot from the interface, 3D Studio Max
2016) Exposure control Real-world V-ray Physical
settings camera Camera
Shutter speed Yes Yes
Conclusion and Discussion f-number Yes Yes
ISO Yes Yes
Taking into account all of the explained consider-
ations, it is clear that CGI has a strong and essen-
tial impact on the interior designer’s workflow.
Today’s clients have very high expectations and to create visually compelling, high-realistic
very short attention spans. Thus, the interior images. V-ray Physical Camera is unable to solely
designers face a challenge to constantly improve achieve the desired render quality – the CGI real-
the quality of their outcomes, by providing their ism is a result of complex algorithm that in general
consumers with top-quality projects. In order to includes the accurate use of materials, lighting,
achieve the expected results, the practitioners camera, and some generic render settings. Never-
should optimize the time they spend to enhance theless, the camera plays quite essential role when
their performance. Being aware of the abilities it comes to visualizing the project and the V-ray
that the contemporary 3D software and its related Physical Camera provides users with all abilities
plugins provide will significantly facilitate the that the real-world camera does. However, regard-
relations between interior designers and their cli- ing the parallel between these two elements: the
ents. The listed analysis and illustrative examples virtual and the real-world one, the V-ray Physical
above suggest that V-ray and its camera could be Camera has an advantage regarding the ISO. The
successfully adopted by the professionals in order real-estate photographers often suffer when using
3D-Rendered Images and Their Application in the Interior Design 69

3D-Rendered Images and Their Application in the Interior Design, Fig. 3 Comparison between photo and V-ray
render. (Source: Created by the author – Petyo Budakov, Ph.D. in 3D Studio Max 2016)

3D-Rendered Images and Their Application in the 3D-Rendered Images and Their Application in the
Interior Design, Fig. 4 Interior design, rendered with Interior Design, Fig. 5 V-ray-rendered image (Source:
V-ray. (Source: http://help.chaosgroup.com/vray/help/ Created by the author – Petyo Budakov, Ph.D. in 3D Studio
rhino/200R1/images/tutorials/interior/Final%20Render% Max 2016)
20002.png; Created by the author – Petyo Budakov, Ph.D.
in 3D Studio Max 2016)
vision. The high-realistic 3D virtual projects sig-
nificantly facilitate the sales of incomplete prop-
inaccurate ISO values that make their photos erties by providing buyers with a clear vision and
noisy or grainy. This problem is completely precise idea of their future homes. Obviously, the
resolved by the unique V-ray algorithm which economic benefits are also present – instead of
opens new horizons to achieve extremely high using some less attractive two-dimensional
realism in terms of interior design projects. Fur- sketches, wasting time until the property is
thermore, the application of the 3D-rendered accomplished, and then hiring a photographer to
images have its strong impact on the real-estate take the photos, the interior designers are capable
marketing – 3D models are used to promote the to do all this in advance. As a consequence, the
sale of property by providing lenders and inves- project becomes much more affordable and a
tors with a good understanding of the project’s recently conducted survey by Real Estate in a
70 Abstraction and Stylized Design in 3D Animated Films

Digital Age (2017), 50% of the buyers find the 3D Chaosgroup.: Retried from: https://docs.chaosgroup.com/
visualizations and virtual tours as a preferable d i s p l a y / V R AY 3 M A X / A r e a + L i g h t s + % 7 C
+VRayLight. Accessed: 01 Oct 2017
option that facilitates their final decision for Curtin, D.: Exposure Controls—The Shutter and Aperture.
purchasing. Retried from: http://www.shortcourses.com/use/
As a conclusion, the 3D-rendered images trans- using1-6.html. (2011). Accessed: 07 July 2017
form not only the interior design in general but also Visualization and 3D Rendering Software Market by
Application (High-End Video Games, Architectural
make a strong impact on the real estate marketing. and Product Visualization, Training Simulation, Mar-
keting and Advertisement), Deployment Type, End-
User, and Region- Global forecast to 2022.: Published
Cross-References by: marketsandmarkets.com. Publishing date: June
2017, Report Code: TC 4168. Accessed: 03 Sep 2017

▶ 3D Modelling Through Photogrammetry in

Cultural Heritage
▶ 3D Visualization Interface for Temporal Anal-
ysis of Social Media
6 DoF Interaction

▶ Virtual Hand Metaphor in Virtual Reality

References

Books
Lohrman, L.: What Real Estate Agents Need to Know AABB, Aligned-Axis Bounding
About Photography, 4th edn. Media LLC (2014) Boxes
Nagahara, H., Kuthirummal, S., Zhou, C., Nayar, S.: Flex-
ible Depth of Field Photography, European Conference ▶ Spheres, AABB, and OOBB as Bounding
on Computer Vision, pp. 60–73. Retrieved from: http://
www.cs.columbia.edu/~sujit/PAMI.pdf (2008) Volume Hierarchies
Swanson, E.: Interior Design 101. Retrieved from: http://
ericaswansondesign.com/wordpress/wp-content/
uploads/2010/05/E-Book-Final.pdf (2010)
Verma, V., Walia, E.: 3D Rendering – Techniques and
Challenges. International Journal of Engineering and Absorption
Technology. 2(2), 29–33 (2010). Retrieved from: http://
www.enggjournals.com/ijet/docs/IJET10-02-02-01. ▶ Videogame Engagement: Psychological
pdf Frameworks
Wang, T.: Pencil Sketching, 2nd edn. Wiley, New York
(2002)

Reports
Kerr, D.: Equivalent Focal Length, Issue 1, Retrieved from:
http://dougkerr.net/Pumpkin/articles/Equiv_Focal_ Abstraction and Stylized
Length.pdf (2009) Design in 3D Animated Films:
OBEO.: Real Estate Photography, Retrieved from: http:// Extrapolation of 2D
www.obeo.com/wp-content/themes/obeo/assets/
media/obeo-photography-ebook.pdf (2017)
Animation Design
Real Estate in a Digital Age.: National Association of
Realtors. Retrieved from: https://www.nar.realtor/ Daniel N. Boulos
sites/default/files/reports/2017/2017-real-estate-in-a- University of Hawai’i Manoa, Honolulu, HI, USA
digital-age-03-10-2017.pdf (2017). Accessed 05 Aug
2017

Websites Synonyms
Carroll, B.: (www.bocphotography.com), Published by:
Barry O Carroll. Accessed: 15 Sep 2017 Accord; Conform; Formalize
Abstraction and Stylized Design in 3D Animated Films 71

Stylization is at the heart of 2D animation design challenge of stylization in 3D animated feature

and is only recently being more fully explored in films. Animated films are most often driven by
3D animated films. In the early days of 3D ani- the concerns of narrative storytelling structure. A
mation, the push for realism in lighting, rendering, Although some subject matter in narrative story
and deformations displaced a pursuit of styliza- may benefit from a photo-realistic approach to 3D
tion in the quest to expand the capabilities of imagery, the desired effect of many films is to
computer graphics technology. With those techni- remove the audience from their daily experience
cal problems solved, 3D animation has more and provide immersion into visualizations that
recently embraced stylization in design and char- depart from realism. Sparavigna observes, “How-
acter movement. Stylization also can be ever, it is not obvious that a photorealistic pro-
interpreted by some as playfulness, and “play is cessing is always to be preferred. . .Hand drawn
at the heart of animation” (Powers 2012, p. 52). illustrations can better explain a scene than pho-
Nature can be seen as an “abstract visual phenom- tographic plates, because in illustrating complex
enon” (Beckman and Ezawa 2012, p. 101), and phenomena, they can omit unnecessary details
“the portrayal of hyperrealistic human characters and propose only fundamental objects”
in 3D animation can lead to the alienation of an (Sparavigna and Marazzato, p. 1). One benefit of
audience, as they may not accept them as being a departure from photorealism is the ability to
real” (Kaba 2013, p. 188). It is the ability of communicate effectively and efficiently. Visuali-
animation to “break with naturalistic representa- zation in narrative film structure provides an
tion and visual realism” (Linsenmaier 2011, p. 1) opportunity to reinforce story points, clarify
that is observed as one of the strengths of the art. what is taking place, and also enhance the emo-
This entry discusses the implications of stylized tional context of the screen experience for the
design and its use in 3D animated films while audience.
drawing important references to traditional hand- When examining the art of oil painting, where
drawn animation stylization processes that pose a the design and construct of imagery is of equal
challenge to modern 3D animation studios. importance, the movements of Postimpressionism
and Modernism exemplify this point. The visual
experience in Vincent Van Gogh’s Starry Night is
Definition entirely dependent upon his unique interpretation
of the observed phenomenon of the night sky.
Stylization is the process of depicting or treating a Without this process the work would lose its
subject in a nonrealistic approach according to an visual identity and much of its emotional content
artistic convention. Stylization in animation (Fig. 1).
includes the two- and three-dimensional graphic In Marcel Duchamp’s Nude Descending a
representations of characters and objects as well Staircase No. 2, we observe the power of styliza-
as the fourth dimension, stylization of timing, and tion in communicating motion and once again
movement. emotional context (Fig. 2).

Definition of Style
Background The Oxford Dictionary offers the following defi-
nition for style, “A distinctive appearance, typi-
“Traditionally, computer graphics pursued the cally determined by the principles according to
reproduction of real world. Consequently, many which something is designed” (Oxford 2015). The
efforts were devoted to the photorealistic term “distinctive” is helpful, as one benefit of
approach of rendering and processing images” stylization is a unique visual identity. The term
(Sparavigna and Marazzato, p. 1). This observa- “designed” implies the intent to implement a pro-
tion is important as it identifies a fundamental cess leading to visual identity.
72 Abstraction and Stylized Design in 3D Animated Films

Abstraction and Stylized Design in 3D Animated Films: Extrapolation of 2D Animation Design, Fig. 1 Van-
Gogh’s Starry Night to photo comparison

Abstraction and Stylized

Design in 3D Animated
Films: Extrapolation
of 2D Animation Design,
Fig. 2 Duchamp’s Nude
Descending a Staircase
No. 2 photo comparison

Stylization in the 2D Animation Feature Film generating more returns than the initial film the
Process design appeared in.
Stylization is at the heart of the animation indus- The production process of traditional 2D ani-
try. A photo-realistic design of an animal such as a mated feature films leveraged stylization at many
bear cannot be easily registered for copyright; points within the creation timeline. It was often
however, a highly stylized interpretation of a highly stylized representations which first visual-
bear such as Hanna and Barbera’s Yogi Bear is ized a story idea in the visual development phase
copyrightable as a creative work. Such character of preproduction. Stylization was a central part of
design copyright is at the foundation of animation the 2D layout process where line drawings for
merchandising. Stylized character designs are each background painting were carefully created.
highly profitable for animation studios often Following the instructions of an art director,
Abstraction and Stylized Design in 3D Animated Films 73

specific stylization concepts would be applied by approach to the effect of a smoke cloud. The
layout artists as they would interpret the setting of representation favors specific types of curvilinear
the film, designing backgrounds for all shots treatments and angular oppositions. The visual A
within the film. The supervising animators then interpretation stands in marked contrast to the
applied stylization, as they began the experimen- same event in the physical world (Fig. 4).
tal animation that would set the tone for their
characters’ performances. The final design of the Realism in Early Computer Graphics
character was informed by the stylistic theme of The value of computer-generated imagery as a
the film. In these images from Walt Disney’s 1959 visual effects element for live-action film was
feature Sleeping Beauty, it is evident that stylistic evident from its early use such as in the 1973
choices created for the background design in a film Westworld and the 1977 film Star Wars Epi-
shot flowed into the interpretation of line and sode IV. Throughout the 1980s computer graphics
form in the character design (Fig. 3). played an ever-increasing role in live-action
The architectonic styling of the gothic-inspired visual effects. Animation that had traditionally
backgrounds by stylist Eyvind Earle carry through been accomplished through stop-motion tech-
into the angular interpretation of the characters; a niques was soon replaced by 3D computer ani-
harmony is achieved between the stylization mated effects. Accordingly, 3D animation tools
applied to the props, set elements, and the evolved along a trajectory of photorealism. Life-
characters. like portrayals of light and shadow as well as color
However, the stylization process did not end were necessary for seamless compositing with
with the look and feel of the character designs, live-action elements. The need for realistic treat-
backgrounds, and props. Stylization was central to ments influenced the evolutionary path of 3D
movement in character animation; the animation technology during the same period when uses
of the characters themselves provided a richly for 3D graphics and animation were still being
stylized experience of real-world motion and defined. After Pixar’s great achievement and crit-
timing, much in the same way a ballerina stylizes ical success with Toy Story in 1994, it was
such mundane activities as walking or such unreal established that 3D animation could compose the
activities as flying like a swan. entirety of a film. Rather than a means to an end,
Finally stylization was also applied in the inter- as in the case of visual effects, 3D animation had
pretation of natural phenomenon. This example become a final product. It would take however
from Walt Disney’s Hercules shows a stylized

Abstraction and Stylized Design in 3D Animated Films: Extrapolation of 2D Animation Design, Fig. 3 Design
examples from Walt Disney Pictures’ Sleeping Beauty
74 Abstraction and Stylized Design in 3D Animated Films

Abstraction and Stylized Design in 3D Animated Films: Extrapolation of 2D Animation Design, Fig. 4 Walt
Disney Pictures' Hercules to photo smoke comparison

Abstraction and Stylized Design in 3D Animated Films: Extrapolation of 2D Animation Design,

Fig. 5 Example of thick ink lines from early television animation (1950s)

several years to break from the pursuit of realism vector animation. Flash vector animation popu-
and implement stylization on a broad scale. lated the web of the 1990s offering motion
graphics via the low-bandwidth Internet connec-
tion common for that time. Flash software was
The Influence of Technology adapted for television production as it offered
timesaving advantages previously impossible in
Early Television the limited animation repertoire. The characteris-
Early television broadcasts were of limited reso- tic use of line and shape from early web anima-
lution with two alternating fields of horizontal tions found its way into many television shows of
lines creating each final frame. The thin lines of the last decade. A clear visual parallel can be seen
hand-inked animation cells used in theatrical in the flat graphic character styling favored by
shorts did not display well, and at certain tangents early television animation of the 1950s and the
to the horizontal could be lost altogether during Flash-influenced television designs of the last
broadcast. The use of thick ink lines in television 15 years (Fig. 6).
animation produced a solution that was not only
functional but also a stylistic standard for the new
medium (Fig. 5). Filmmaking

Early Internet Stylization and Genre

Early web animation utilized Flash software and Stylizations themselves can be seen to splinter
introduced the use of stylized thick ink lines to through the prism of genre. For example, the
Abstraction and Stylized Design in 3D Animated Films 75

Abstraction and Stylized Design in 3D Animated Films: Extrapolation of 2D Animation Design,

Fig. 6 Examples of Flash television animation styling (2000s)

Abstraction and Stylized Design in 3D Animated Films: Extrapolation of 2D Animation Design,

Fig. 7 Example – genre stylistic differences in early television animation

stylistic conventions in Rocky & Bullwinkle con- The music-driven films were segmented into
trast against the variations seen in Johnny Quest, separate capsules more indicative of the animated
which was influenced by comic book illustration short format. Narrative-driven features had the
(Fig. 7). burden of clarifying complex character arcs,
Early Disney theatrical features, such as staying onscreen for extended periods. More
Bambi, can be seen in stylistic contrast to detailed facial treatments were dictated by these
musically derived works such as Fantasia genre-induced requirements (Fig. 9).
(1940) or Make Mine Music (1946). In For example, the white region of the eye is
Bambi the soft-edged painted treatments by important in subtle facial expressions, and a
stylist Tyrus Wong set the mood and look of more stylized treatment of an eye may not encom-
the forest in which the highly stylized design pass the same expressive range.
of the character Bambi, from animator Marc
Davis, performed. In Make Mine Music mini- The Role of Stylization in Storytelling
malized character and background treatments Stylization plays an important role in narrative
in the segment All the Cats Join In were a story telling as seen in such early live-action
significant departure from more detailed films as Murnau’s 1922 Nosferatu where the styl-
human characters in the narrative plot-driven ized uses of shadow and form set an eerie context
features (Fig. 8). or in animated sequences such as the stag fight
76 Abstraction and Stylized Design in 3D Animated Films

Abstraction and Stylized Design in 3D Animated Films: Extrapolation of 2D Animation Design,

Fig. 8 Background and character comparison of Bambi and Make Mine Music

Abstraction and Stylized Design in 3D Animated Films: Extrapolation of 2D Animation Design, Fig. 9 Facial
detail stylization – comparison of Bambi and Make Mine Music

near the end of the theatrical feature Bambi where Students in art programs are taught, as foun-
the stylistic treatments of color and form enhance dation, the importance of design principles and
the emotional drama of the battle (Fig. 10). their corresponding emotional impact on the
viewer. Often through slight changes in the align-
Design Principles and Screen Composition ment of forms in a visual field is balance
Stylization is born of visual design principles and achieved or such subtle linking through contour
gains effect from their successful implementa- continuation fully realized. In their early years
tion. The elements of visual design are the tools Walt Disney Studios was particularly sensitive to
of the stylist as they interpret objective visual the relationship art had to animation and endured
facts, into their final subjective states. Shape, great effort and expense to expose its employees
form, color, line, texture, etc. are manipulated to these concepts via art classes and seminars.
in tangible ways, employing design concepts Transcripts from the 1930s recorded artists in
such as theme and variation and contrast and such evening seminars isolating what are now
harmony while enhancing rather than reducing taught as animation principles (Johnston and
a film’s context. Thomas 1984, pp. 71–72).
Abstraction and Stylized Design in 3D Animated Films 77

Abstraction and Stylized Design in 3D Animated Films: Extrapolation of 2D Animation Design,

Fig. 10 Stylization examples from Nosferatu and Bambi

Trained artists filled the majority of positions achievements and a new emphasis now placed
in 2D animated production pipelines. The effect of on stylization in 3D character animation.
this foundational integration of artists in the ani-
mation process led to the formation of visual The Emergence of Stylization in 3D Character
identities within animation studios, often around Animation
a handful of celebrated stylists. Maurice Noble Many efforts were made toward stylization in the
had a profound impact on what came widely rec- early history of 3D feature film production. Some
ognizable as the “look” of a Warner Bros. ani- labors were rewarded more than others; however,
mated short. Similarly Mary Blair impacted the it can be clearly seen by the time of the Pixar film
look of Disney features and short subjects through The Incredibles (2004); the final technical hurdles
multiple stylistic interpretations as can be seen in had been overcome and stylization began to enter
Saludos Amigos (1942), Johnny Appleseed with greater impact. Not only did this film achieve
(1948), and Peter Pan (1953), three works that significant stylization in character design but nota-
demonstrate a wide range of visual bly in character movement as well. The clarity and
interpretations. exaggeration of animation poses became compa-
rable to 2D animation. A break from realism is at
the center of the appeal of Mrs. Incredible whose
Character Animation: Stylized Timing body stretches to outrageous lengths in the hall-
in 3D Features way sequence as she tries to find her husband.
A comprehensive stylistic aesthetic carried over
Stylization in movement is the realm of the char- from development art into prop design, set ele-
acter animator. There have been notable ments, and lighting.
78 Abstraction and Stylized Design in 3D Animated Films

Stylized character movement began to appear com 2005), as well as DreamWorks and Warner
more consistently outside of Disney/Pixar films, Bros. The resulting talent migrations had an
as is exemplified by the Sheriff Earl character in impact on several 3D animated feature
Sony Animation Studios’ film Cloudy with a productions.
Chance of Meatballs (2009), as the sheriff In the case of The Incredibles, it was the first
bounces and flips rather than walks through a full 3D feature animation effort for director Brad
crowd. It was clear that 3D animation directors Bird who had been brought to Pixar by John
were embracing nonliteral forms of movement. Lasseter. The Incredibles production saw the
The trend continued in Cloudy with a Chance of arrival of animation director Tony Fucile who
Meatballs II where the fraudulent guru, Chester, had been a supervising animator on the 2D film
twists, slides, and gyrates from pose to pose, The Lion King before working as animation
devoid of literal movement that could have orig- supervisor under Brad Bird on the mixed
inated with motion capture technology or other 2D-3D features at Warner Bros. In the case of
automated methods. The 2010 release Tangled Cloudy with a Chance of Meatballs, Sony Pic-
from Walt Disney Pictures stood in contrast to tures had been on a course of introducing 2D
character movement in early 3D animation talent within their 3D ranks as Roger Allers and
efforts as it exhibited the strong clear poses and Jill Culton, codirectors for Open Season (2006),
simplified exaggerated movements that had been and Chris Buck and Ash Brannon, codirectors for
indicative of high-quality 2D character anima- Surf’s Up (2007), all had their roots in 2D feature
tion. Finally, Walt Disney Pictures’ Wreck-It animation. Finally codirectors Jim Reardon and
Ralph (2012) is full of stylistic motion intended Rich Moore (Wreck-It Ralph, 2012) were gradu-
to mimic the movement of pixel-graphic video ates of the Character Animation Department at
games. Characters move in multiple styles within CalArts and also had their roots in 2D animation
single shots. One character may be devoid of techniques.
animation principles such as Arcs resulting in
stiff and unnatural movement, while the next
character may follow the standard principles of Challenges in the 3D Feature Film
movement resulting in a natural screen presence. Pipeline
3D character animation success stories such as
these contrast with automated processes such as There is no one-size-fits-all approach to styliza-
motion capture. Motion capture is a process that tion nor should there be in the quest for visual
conceives 3D character animation as a copy of identity among the scores of animated features
real-world movement, while keyframed 3D char- released each year. Although stylization is widely
acter animation builds on a foundation of 2D addressed in 3D animated features today, often it
character animation traditions such as exaggera- is not fully realized particularly in scene design
tion or simplification. Automated processes and screen composition, leaving many 3D fea-
negate the interpretive role of the animator as tures with a similar look, lacking visual appeal.
stylist, resulting in movements that are prepared Stylization is a key ingredient in film, affecting
rather than designed. qualitatively the dialog with the audience and
enhancing the narrative. It should be fully
The Influence of 2D Animators on the 3D implemented in animated film, where it is most
Pipeline readily available. It is unfortunate to see many 3D
These 3D stylistic trends can be associated with animated feature films offer little stylization and
the arrival of many traditional 2D animation art- routinely forgo the advantages that stylization
ists who have joined the ranks of 3D studios. brings. This phenomenon can be examined from
Many animation artists were displaced by the several perspectives, but here it is seen through a
closure of 2D feature animation production at close look at the 3D pipeline and the hundreds of
Disney Studios in 2005 (Forum.rottentomatoes. workers that create the final films.
Abstraction and Stylized Design in 3D Animated Films 79

Compartmentalization: Disintegration 3D imagery as it moves from concept to comple-

of the Design Process tion poses many challenges to stylization in 3D
As was explored earlier the 2D animated feature film. Even with the efforts of a dedicated produc- A
film pipeline relegated the bulk of stylistic control tion stylist, visual influence is diluted as stage by
to relatively few artists. All of it was found in stage a single shot is completed over a long period
preproduction or early in the production phase of of time.
the film. The art director and key development
artists along with supervising animators and the Demographics of the Production Team
layout department determined the bulk of stylistic The evolution of computer-generated technology
integration before the majority of the people was so strenuously focused on photo-realistic
involved in production would begin their work. achievement for such a long period that stylization
In their early and concentrated efforts, most con- developed little momentum, often weakly inte-
sequential decisions were made determining the grated or entirely absent. As 3D animated features
final use of color, form, and directionality in quickly supplanted the 2D animated format, the
screen composition. balance of production personnel changed in a
However, the 3D pipeline presents a more dramatic way. In the typical 2D animated feature
complex and compartmentalized process. film, more than 80 % of the preproduction and
Although most 3D feature films have the benefit production team came from an art background.
of both development artists and art directors, the The 2D production team required 300–500 people
difficulty is in the component processes collec- most of whom had to be able to paint or draw well.
tively resulting in the final color and composition As a result most crewmembers had attended art
of each shot. The work of the 3D development programs within universities or colleges before
artist gives way to the modeling artist who first contributing to the film. By contrast 3D animated
visualizes the characters, props, and set elements films require a more diverse range of skills. Com-
in 3D geometry. The texture artist further contrib- puter technology is central to all the elements in
utes in ways that directly impact stylization. The each shot of a 3D film. As a result technologists
composition of the scene falls to a set decorator or are as numerous as artists on many 3D features.
shot composer then continues on to layout artists While a 3D character animator will graduate from
who block in the camera movement affecting each an art program, a character technical director will
scene’s composition. The animator follows with graduate from a computer science background.
the keyframing of character elements, working The table below clarifies the proportional differ-
primarily in the roughhewn visual context of ence between the two respective production
low-polygon models. In a subsequent step further teams. The example compares individuals listed
animation is added through automated simula- in both the visual effects and animation categories
tions and dynamic options and processes. Final in the film credits for Inside Out (2015), the 3D
lighting is then applied to the scene, only begin- example, and The Little Mermaid (1989), the 2D
ning to address the integral role of shadow and example (Table 1). The crew totals for each film
light in screen composition, at a very late stage in are much higher; the sample was limited to the
the process. It is when each frame is finally ren- personnel involved in the production process for
dered that the shot design and composition arrive the two listed categories. It should be understood
at their final state. that the 43 people in the technical processes for
Implementation of stylization in the 3D pipe- 2D animation would have contributed most of
line involves many departments, beginning early their efforts in postproduction after the hands of
in preproduction and ending with final render. By the 307 artists have already touched the film.
comparison the 2D stylization process was in the The 3D film process is challenged by this
hands of few artists, heavily weighted toward the change in balance, as the production team moves
preproduction phase of the film process. The away from a common background in art to one
length and complexity of the production of the pulling heavily toward digital technology. Thus,
80 Abstraction and Stylized Design in 3D Animated Films

Abstraction and Stylized Design in 3D Animated Films: Extrapolation of 2D Animation Design,

Table 1 Demographic shift in the constituency of feature animation production units (Data retrieved from IMDB.com)
Inside Out (2015a) 3D animation The Little Mermaid (2015b) 2D animation
Animator or character designer: 25 Animator, painter, character design, layout, 2D effects: 307
Software, simulation or technical: 17 Xerographic or technical: 43
Percentage of sample that are artists: 59 % Percentage of sample that are artists: 87 %

the largely artistic process of stylization has not drawings set the stage before the 2D animator
only been spread across a much broader span of ever set pencil to paper. As many times as the 3D
production time but among a different set of con- pipeline is repeated, it ought to be reimagined; such
tributors. Without a common artistic background a young set of processes should be ripe with exper-
rooted in design principles and aesthetics, the imentation. These experiments should be guided
understanding of stylization and what constitutes by design principles and stylization ideals, which
successful implementation may be a point of con- ultimately trump technology as a film seeks its
flict within 3D production teams. emotional connection with the audience.

Education and Training

Conclusion It would be equally advisable to educate 3D pro-
duction teams in artistic processes in an aggres-
The Need for Visual Identity sive and meaningful way, with a desire to learn
Animated feature films are more abundant today how such processes were employed in 2D anima-
than at any other time in the history of film. The tion units. Educating the team and developing a
vast majority of these films are 3D animated common core design philosophy would help
features. assure that stylization goals are met at all stages
of production.
Restructuring the 3D Pipeline It is very hopeful that stylization will be fully
It may be advisable to revisit the 3D pipeline in an realized in the future of 3D animated films, as the
effort to identify alternate methods for gaining the existing tools and processes are capable of far
control necessary for significant and successful greater results in this quest.
stylization. There must be an effort to simplify
the pipeline or further empower the art director
over component parts of the process. It may be References
that one art director alone will not be able to track
the myriad simultaneous decisions, which are the Beckman, K., Ezawa, K.: Animation, Abstraction, Sam-
pling: Kota Ezawa in Conversation with Karen
standard for efficient 3D production. Perhaps
Beckman, University of Pennsylvania Scholarly Com-
there will be room to experiment with the art mons. http://repository.upenn.edu/hisart_papers/5
director as a team or creative unit rather than a (2012). From web 05 July 2015
sole individual. This team could be as many as Forum.rottentomatoes.com: Forum Posting on Closure of
Disney Animation Australia – Derived from Australian
20 or more persons; thereby, an art director for the
Broadcasting Corp., 07/27/2005. http://forum.
team would be able to monitor each shot through rottentomatoes.com/topicR/show/1216747 (2005).
every stage of the pipeline. From web 05 July 2015
It is further advisable that other methods such as IMDB.com: Database retrieval – credit list for inside out.
http://www.imdb.com/title/tt2096673/fullcredits/
the stop-motion animation pipeline be examined as (2015a). From web 12 Sept 2015
possible influences for change. Perhaps one day IMDB.com: Database retrieval – credit list for the Little
final lighting and texturing will precede character Mermaid. http://www.imdb.com/title/tt0097757/
animation in much the same way shaded layout fullcredits/ (2015b). From web 12 Sept 2015
Academic and Video Game Industry “Divide” 81

Johnston, O., Thomas, F.: Disney Animation the Illusion of Introduction

Life. Abbeville Press, New York (1984)
Kaba, F.: Hyper-realistic characters and the existence of the
uncanny valley in animation films. Int. Rev. Soc. Sci. Contemporary game design and development has A
Humanit. 4(2), 188–195 (2013) been impaired by poor communication between
Linsenmaier, T.: Nea Ehrlich – animated documentaries as related practitioners and academics (Passarelli
masking. http://journal.animationstudies.org/nea- et al. 2018; Greenwood et al. 2021). In academia,
ehrlich-animated-documentaries-as-masking/ (2011).
From web 05 July 2015 game design is contextualized within the study of
Oxford English Dictionary: Online dictionary – definition games, the act of playing them, and the cultures
of stylization. http://www.oed.com (2015). From web surrounding them (Engström 2020). In contrast, at
05 July 2015 the industry level, the focus on game design has
Powers, P.: Ludic toons the dynamics of creative play
in studio animation. Am. J. Play 5(1), 22–54 revolved around self-expression, overcoming
(2012) challenges, player experiences, and making a
Sparavigna, C., Marazzato, R.: Non-photorealistic image profit (Davis and Lang 2012). The reasons for
processing: an impressionist rendering. www.acade poor communication between both parties are
mia.edu, http://www.academia.edu/4703400/Non-
photorealistic_image_processing_an_Impressionist_ complex, focusing on the importance of creativity
rendering. From web 05 July 2015 and business management, and therefore attention
is mostly given to game mechanics, aesthetics,
and marketing strategies (Marchand and Hennig-
Thurau 2013). These differences in priorities and
resulting mismatch of understanding are known as
Abzû the “academic-industry divide” (Colusso
et al. 2017).
▶ Educational Game Abzû and the Lens of Fun
To further understand the divide, it must be
Learning
understood clearly that academics and practi-
tioners have differing motivations driving how
they approach the gaming discipline. Typically,
academics focus their research on how games
Academic and Video Game generate social and cultural meanings, and how
Industry “Divide” game mechanics and aesthetics can inform the
development of new games (Engström 2020).
Jordan Greenwood and Grant Meredith Whereas industry practitioners focus their atten-
Federation University, Mt Helen, VIC, Australia tion on providing immersive experiences for
players and ensuring a profitable outcome for
themselves (Davis and Lang 2012). A result of
Synonyms these differing motivations is that academics and
industry practitioners often think that they are
Divide; Gap; Rigor-relevance debate discussing the intricacies of the same situation/s,
when in fact they are not, due to not understanding
the importance and links between their
Definition related work.

The “divide” can be classified as the limited trans-

ference and dissemination of knowledge, and the How the Divide Is Appearing Within the
lack of collaborative projects between academics Gaming Industry
and their industrial counterparts. The extent to
which the divide exists between the games indus- Beginning in 2016, the European Commission–
try and academia lacks clarity and needs contin- funded project, Gaming Horizons (www.
ued research. gaminghorizons.eu), investigated video games
82 Academic and Video Game Industry “Divide”

and society from a social sciences perspective employees; and opportunities to work on alter-
(Passarelli et al. 2018, 2020). This study found native projects of commercial value and access
that there was a clear working divide between to recognized world experts within defined
gaming practitioners and academics. Extending areas.
on Passarelli and colleagues, Engström (2020)
suggested that the divide needs to be viewed Barriers of the Divide
from different perspectives before a proper assess- On the contrary, previous research into the divide
ment can be made about the extent to which it within other disciplines has identified many bar-
exists between the games industry and academia. riers. These barriers of the division are catego-
rized into three main groups: communication,
Viewpoints on the Divide knowledge transfer, and collaboration outlined
There are two opposing viewpoints concerning below:
the general academic-industry divide: the propo-
nents for bridging the divide and those who Communication barriers: The accessibility of
believe that it should not be bridged. research outside of academia and industry’s
ability to access and understand it.
The divide should be bridged: The ideal is that Knowledge transfer barriers: Relevance issues
by working with industry, academics can and academics viewed as being poor at com-
engage in joint sensemaking to generate new municating their projects outside of the aca-
and useful knowledge that can be used to demic field.
improve processes and products (Shani and Collaboration barriers: Funding and manage-
Coghlan 2014). ment issues; issues over project intellectual
The divide should not be bridged: Based on property, and perceived trust issues between
Luhmann’s (1995) system theory which out- the two communities.
lines the extreme difficulty in integrating
knowledge that has been generated in different Suggestions on How to Bridge the Divide
contexts of science and practice (Kieser and To assist with bridging the divide, suggestions
Leiner 2012). have been offered. The suggestions have been
categorized into three main groups: general, col-
Benefits of Bridging the Divide laboration enhancement, and knowledge transfer-
Within differing disciplines, studies into bridging ence outlined below:
the academic-industry divide have identified more
barriers than benefits. This could be because those General suggestions: Such as the development
studies were limited to identifying barriers of the of different documents for different audiences
divide or were more focused on why the divide and using the agile research network
exists and not on the outcomes of successful approach.
bridging attempts (Beck and Ekbia 2018). The Collaboration enhancement suggestions: Such
benefits for bridging the divide can be summed as academics need to learn about industry prac-
up as follows: tices and industry needs to learn about what
academics are doing.
The benefits for academics: access to current Knowledge transference suggestions: Such as
industry knowledge, data, and practices; academics developing libraries and transla-
employment for graduates; and opportunities tional resources and the development of an
to innovate and to access alternative funding online space to communicate.
streams from governments or industry bodies.
The benefits for industry: access to additional It must be highlighted that most of the sugges-
avenues to solve problems; access to skilled tions made to reduce the divide have not been
Academic and Video Game Industry “Divide” 83

implemented into practice or if they have, they ▶ Gamification of Modern Society: Digital
have not been as successful as one would hope Media’s Influence on Current Social Practices
(Norman 2010). ▶ Serious Online Games for Engaged Learning A
Through Flow
Communities of Practice
Research that studies communities of practice
have also shown that academia and industry References
have different core goals, views, and agendas
Beck, J., Ekbia, H.R.: The theory-practice gap as genera-
(Gray et al. 2014; Wallin et al. 2014; Colusso tive metaphor. In: Proceedings of the 2018 CHI Con-
et al. 2017). This has led researchers to show ference on Human Factors in Computing Systems,
them as two different sets of communities. For pp. 1–11 (2018)
example, academia’s goal is to seek and gener- Colusso, L., Bennett, C.L., Hsieh, G., Munson, S.A.:
Translational resources: Reducing the gap between
ate knowledge and to provide training for future academic research and HCI practice. In: Proceedings
careers, whereas industry is focused on build- of the 2017 Conference on Designing Interactive Sys-
ing products and the selling of goods to make a tems, pp. 957–968 (2017)
profit (Lameman et al. 2010; Wallin et al. Davis, R., Lang, B.: Modeling game usage, purchase behav-
ior and ease of use. Entertain. Comput. 3, 27–36 (2012).
2014). The question now is whether the two https://doi.org/10.1016/j.entcom.2011.11.001
communities can interact without support, or Engström, H.: Game Development Research. University of
if it is better to develop an ecosystem that sup- Skövde, Skövde (2020)
ports communication between the two Gray, C., Stolterman, E., Siegel, M.: Reprioritizing the
relationship between HCI research and practice:
communities. Bubble-up and trickle-down effects. In: Proceedings
At the 2021 Interactive Games and Entertain- of the 2014 conference on Designing interactive sys-
ment Association (IGEA) Education Summit, a tems, pp. 725–734 (2014)
panel discussed the forming of a community of Greenwood, J., Achterbosch, L., Stranieri, A., Meredith,
G.: Understanding the gap between academics and
practice within the Australian and New Zealand game developers: An analysis of Gamasutra blogs.
population. The panel also attempted to prompt Paper presented at the International Conferences Inter-
the values of academics and practitioners working faces and Human Computer Interaction, online (2021).
together to improve the education of future mem- https://www.ihci-conf.org/wp-content/uploads/2021/
07/02_202105L018_Greenwood.pdf
bers of the game development community, which Ingram, C., Chubb, J., Boardman, C., Ursu, M.: Generat-
is a good starting point for future discussions ing real-world impact from academic research: Expe-
(IGEA 2021). rience report from a University impact Hub. In:
With this knowledge will Australian and Proceedings of the IEEE/ACM 42nd International
Conference on Software Engineering Workshops,
New Zealand universities and video game practi- pp. 611–618 (2020)
tioners take up the torch that the IGEA are pro- Interactive Games & Entertainment Association: Graduate
posing, to start forming collaborations such as program guidelines (2021). Available via. https://igea.
developing programs like the Mighty Kingdom net/2021/07/mighty-kingdom-igea-graduate-program-
guidelines/. Accessed 20 Aug 2021
Graduate Program (IGEA 2021) or the Dc Labs Kieser, A., Leiner, L.: Collaborate with practitioners: But
Institute (Ingram et al. 2020). Only time will tell beware of collaborative research. J. Manag. Inq. 21,
where this revolution will lead. 14–28 (2012). https://doi.org/10.1177/1056492611411923
Lameman, B., El-Nasr, M., Drachen, A., Foster, W.,
Moura, D., Aghabeigi, B.: User studies: A strategy
towards a successful industry-academic relationship.
Cross-References In: Proceedings of the International Academic Confer-
ence on the Future of Game Design and Technology,
▶ Computer Graphics, Video Games, and pp. 134–142 (2010)
Luhmann, N.: Social systems. Stanford University Press,
Gamification Impacting (Re)habilitation, Stanford (1995)
Healthcare, and Inclusive Well-Being Marchand, A., Hennig-Thurau, T.: Value creation in the
▶ Gamification and Serious Games video game industry: Industry economics, consumer
84 Acceptance Gap

benefits, and research opportunities. J. Interact. Mark.

27, 141–157 (2013). https://doi.org/10.1016/j.intmar. Accessibility of Virtual Reality
2013.05.001
Norman, D.: The research-practice gap: The need for trans- for Persons with Disabilities
lational developers. Interaction. 17, 9–12 (2010).
https://doi.org/10.1145/1806491.1806494 John Quarles
Passarelli, M., Earp, J., Dagnino, F.M., Manganello, F., Department of Computer Science, University of
Persico, D., Pozzi, F., ... Perrotta, C. Library not
found: The disconnect between gaming research and Texas at San Antonio, San Antonio, TX, USA
development. In: CSEDU 2018-Proceedings of the
10th International Conference on Computer Supported
Education, pp. 134–141 (2018) Synonyms
Passarelli, M., Earp, J., Dagnino, F.M., Manganello, F.,
Persico, D., Pozzi, F., ... Perrotta, C.: The distant Hori-
zon: Investigating the relationship between social sci- Accessibility; Games; Rehabilitation; Virtual
ences academic research and game development. reality
Entertain. Comput. 34 (2020). https://doi.org/10.1016/
j.entcom.2020.100339
Shani, A.B., Coghlan, D.: Collaborate with practitioners:
An alternative perspective a rejoinder to Kieser and Definition
Leiner (2012). J. Manag. Inq. 23, 433–437 (2014)
Wallin, J., Isaksson, O., Larsson, A., Elfström, B.O.: Immersive virtual reality – i.e., completely
Bridging the gap between University and Industry:
Three mechanisms for innovation efficiency. Int. blocking out the real world through a virtual real-
J. Innov. Technol. Manag. 11 (2014). https://doi.org/ ity display – is not currently universally usable or
10.1142/S0219877014400057 accessible to many persons with disabilities, such
as persons with balance impairments.

Introduction
Acceptance Gap
Virtual reality (VR) has traditionally been too
▶ Uncanny Valley in Virtual Reality expensive for the consumer market, which has
constrained its applicability to high cost applica-
tions, such as soldier training, surgical training,
and psychological therapy. However, with the
Accessibility decreasing costs of head mounted displays
(HMD) and real-time tracking hardware, VR
▶ Accessibility of Virtual Reality for Persons may soon be in homes all over the world. For
with Disabilities example, HMDs such as the Oculus Rift (https://
▶ Audio and Facial Recognition CAPTCHAs for www.oculus.com/) for VR and Microsoft’s
Visually Impaired Users upcoming Hololens (https://www.microsoft.com/
▶ Making Virtual Reality (VR) Accessible for microsoft-hololens/) for augmented reality
People with Disabilities (AR) will change the way that users play games
▶ Unified Modeling Language (UML) for Sight and experience the surrounding real world,
Loss respectively. Moreover, VR and AR can now be
effectively enabled through smartphones at an
even lower cost with the simple addition of a
head mounted case, such as MergeVR’s headset
Accessibility in Games (http://www.mergevr.com/). That is, everyone
with a smartphone has virtual environment
▶ Video Games and Accessibility: A Case Study (VE) devices in their pockets right now. Thus,
of The Last of Us II VR will be available to consumers who may
Accessibility of Virtual Reality for Persons with Disabilities 85

have disabilities. However, there is minimal having fun in the case of VR games. In general,
research that highlights the special needs of research suggests that VR and VR games have
these diverse populations with respect to measurable benefits for rehabilitation effective- A
immersive VR. Thus, there is a significant amount ness (Sveistrup 2004; Eng et al. 2007; Ma
of research that must be conducted to make VR et al. 2007; Crosbie et al. 2008; Adamovich
accessible to persons with disabilities. This entry et al. 2009) and motivation (Betker et al. 2007;
reviews the recent efforts of the San Antonio Verdonck and Ryan 2008).
Virtual Environments (SAVE) lab to better under- Visual Feedback: Visual feedback is any kind
stand how persons with disabilities are affected by of feedback for rehabilitation delivered to the
VR accessibility. patient through the visual modality. This includes
mirrors, computer displays, and VR. Visual feed-
back has been shown to be effective in rehabilita-
Background tion (Sütbeyaz et al. 2007; Čakrt et al. 2010;
Thikey et al. 2011).
Most of the information that is known about the Gait Rehabilitation: Gait (i.e., walking pat-
accessibility of VR for persons with disabilities terns) rehabilitation is the main type of rehabilita-
comes from research on virtual rehabilitation. VR tion that requires navigation in a VE. Most
has been shown to have significant benefits to systems used a head mounted display (HMD) or
rehabilitation. A VE is not subject to the dangers a large LCD screen. Results with VR systems in
and limitations of the real world (Boian gait rehabilitation were positive (Fung et al. 2006;
et al. 2002; Burdea 2003; Wood et al. 2003; Tierney et al. 2007; Bardack et al. 2010).
Merians et al. 2006), which expands the types of Design Guidelines for VR Rehabilitation
exercises that patients can practice, while still Games: There has been much research on deriving

Accessibility of Virtual
Reality for Persons
with Disabilities,
Fig. 1 Game Cane. The
user leans forward to move
the character forward and
rotate the cane to steer. If
more weight is put on the
cane (as measured by the
force sensitive resistor), it
will disrupt the movement
of the character in the game
86 Accessibility of Virtual Reality for Persons with Disabilities

design guidelines for VR rehabilitation games virtual reality display (e.g., a head mounted dis-
based on results of empirical studies (Flynn play, a 3D projector). All VR systems have
et al. 2008). Alankus et al.’s guidelines include: latency in them and classically latency has been
simple games should support multiple methods of the enemy of VR, often significantly hindering
user input, calibrate through example motions, user performance.
ensure that users’ motions cover their full range, However, we hypothesized that in some cases,
detect compensatory motion, and let therapists extra latency can potentially be used for the user’s
determine difficulty (Alankus et al. 2010). There benefit in areas such as stroke rehabilitation. For
have been many other guidelines derived (Goude example, in a recent study (Samaraweera
et al. 2007; Broeren et al. 2008; Burke et al. 2015), we intentionally applied an extra
et al. 2009a, b) and there is a need for more 200 ms of latency to the user’s virtual body, but
focused game design research and development only half of the body, which made the unaffected
for specific populations (Flores et al. 2008). half of the user’s body try to compensate for the
latent half. In this study, participants were asked to
walk towards a virtual mirror in which they could
SAVE Lab’s Research in Immersive VR see their avatar (Fig. 2). Interestingly, participants
Accessibility did not perceive the latency effect. Based on her

Making Balance Games More Accessible

Many existing balance based games are not acces-
sible for many persons with balance impairments.
To address this issue Cantu et al. developed a novel
interface – Game Cane (Fig. 1) (Cantu et al. 2014).
Game Cane enables the user to control games and
play balance based games using the natural
affordances of a cane. The Game Cane project has
two goals: (1) make balance games more accessible
and (2) help users with balance impairments to
improve their balance.
Specifically, users control orientation through
rotating the cane and leaning in each direction to
control direction of movement. To meet the reha-
bilitation goal of reducing dependency on the
cane, putting weight on the cane will disrupt
movement (e.g., make a character run slower;
make a car more difficult to turn).
Results of a user study suggest that the Game
Cane is easy to use and serves as sufficient moti-
vation to depend less on the cane during game
play. In the future, we plan to study long term
effects of balance improvement using
Game Cane.

Latency
Accessibility of Virtual Reality for Persons
One of the major potential threats to accessibility
with Disabilities, Fig. 2 Benefits of Latency: a look
is latency. Latency is the time it takes between a into a virtual mirror where the avatar has 200 ms latency
user moving and the movement being shown on a applied to one side of the body
Accessibility of Virtual Reality for Persons with Disabilities 87

promising results, we are now conducting a study Raising Awareness About Persons
on the benefits of this one-sided latency for stroke with Disabilities
patients who commonly have increased weakness Virtual reality still has a long way to go before it A
on one side. The ultimate goal is to apply her can be considered accessible for persons with
technique to help rehabilitate asymmetric walking disabilities. To educate future VR designers and
patterns in these patients. engineers about accessibility in VR, it is important
to raise awareness about the needs of persons with
Accessibility for Children with Autism disabilities. One of the ways that the SAVE lab has
Motivation may be a factor in the accessibility of been raising awareness is through our Virtual
3D User Interfaces for children with Autism. It Reality Walk MS (SAVELab 2015b) (Fig. 4) and
has been shown that many children with Autism our Virtual Reality Walk for Autism (SAVELab
have very specific and individualized interests, 2015a). Using Unity3D (unity3d.com) and
many of which may be uncommon. To more Exitgames Photon (exitgames.com) for network-
effectively motivate children with Autism to prac- ing, the VR walks mimic the real fundraising
tice hand-eye coordination tasks, we created a walks that occur annually, effectively involving
virtual soccer game, Imagination Soccer (Fig. 3), potential participants who may not be able to
where the user played the role of a goalie and attend the real walk. The VR walks are run con-
he/she could customize a virtual human kicker currently with the real walks. Users can choose an
(Mei et al. 2015). We compared customizable avatar and virtually walk around a virtual AT&T
versus noncustomizable virtual humans. As center. Users who are remote are also able to
expected, we found that the participants preferred communicate with people at the real walk since
the customizable virtual humans. Surprisingly, the the software runs on mobile phones. However,
users also exhibited significantly improved task there are still many research problems to be solved
performance with the customizable virtual to make communication more natural and the
humans. This suggests that customization is a interface more transparent. This is an area where
plausible way to make interfaces more accessible new advances in augmented reality technology
for children with Autism. may help to address these issues.

Accessibility of Virtual Reality for Persons with Disabilities, Fig. 3 Imagination Soccer – a game for training hand-
eye coordination for children with Autism
88 Accessibility of Virtual Reality for Persons with Disabilities

Accessibility of Virtual
Reality for Persons
with Disabilities,
Fig. 4 Virtual Reality
Walk MS – a mobile,
multiplayer virtual
environment for raising
awareness about multiple
sclerosis

Conclusion and traumatic brain injuries. Phys. Ther. 87(10), 1389

(2007)
Boian, R., Sharma, A., Han, C., Merians, A., Burdea, G.,
The SAVE lab is trying to push the boundaries Adamovich, S., Recce, M., Tremaine, M., Poizner, H.:
of VR to make it accessible for all users, includ- Virtual reality-based post-stroke hand rehabilitation.
ing persons with disabilities. We have Medicine meets virtual reality 02/10: digital upgrades,
conducted fundamental research towards under- applying Moore’s law to health: 64 (2002). Los
Angeles, CA
standing how persons with disabilities interact Broeren, J., Bjorkdahl, A., Claesson, L., Goude, D.,
with VR and have identified techniques to make Lundgren-Nilsson, A., Samuelsson, H., Blomstrand,
VR more accessible. However, there is still a C., Sunnerhagen, K., Rydmark, M.: Virtual rehabilita-
significant amount of research to be done before tion after stroke. Stud. Health Technol. Inform. 136,
77–82 (2008)
immersive VR can truly be accessible to Burdea, G.: Virtual rehabilitation-benefits and challenges.
everyone. Methods Inf. Med. 42(5), 519–523 (2003)
Burke, J., McNeill, M., Charles, D., Morrow, P., Crosbie,
J., McDonough, S.: Optimising engagement for stroke
rehabilitation using serious games. Vis. Comput.
References 25(12), 1085–1099 (2009a)
Burke, J., McNeill, M., Charles, D., Morrow, P., Crosbie,
Adamovich, S., Fluet, G., Tunik, E., Merians, A.: Sensori- J., McDonough, S.: Serious Games for Upper Limb
motor training in virtual reality: a review. NeuroReh- Rehabilitation Following Stroke. IEEE Computer Soci-
abilitation 25(1), 29–44 (2009) ety (2009)
Alankus, G., Lazar, A., May, M., Kelleher, C.: Towards Čakrt, O., Chovanec, M., Funda, T., Kalitová, P., Betka, J.,
customizable games for stroke rehabilitation. CHI, Zvěřina, E., Kolář, P., Jeřábek, J.: Exercise with visual
ACM (2010), Atlanta, GA feedback improves postural stability after vestibular
Bardack, A., Bhandari, P., Doggett, J., Epstein, M., schwannoma surgery. Eur. Arch. Otorhinolaryngol.
Gagliolo, N., Graff, S., Li, E., Petro, E., Sailey, M., 267(9), 1355–1360 (2010)
Salaets, N.: EMG biofeedback videogame system for Cantu, M., Espinoza, E., Guo, R., Quarles, J.: Game cane:
the gait rehabilitation of hemiparetic individuals. an assistive 3DUI for rehabilitation games. In: 3D User
Thesis, in the Digital Repository at the University of Interfaces (3DUI), 2014 I.E. Symposium on, IEEE
Maryland, (2010) (2014). Minneapolis, MN
Betker, A., Desai, A., Nett, C., Kapadia, N., Szturm, T.: Crosbie, J., Lennon, S., McGoldrick, M., McNeill, M.,
Game-based exercises for dynamic short-sitting bal- Burke, J., McDonough, S.: Virtual reality in the reha-
ance rehabilitation of people with chronic spinal cord bilitation of the upper limb after hemiplegic stroke: a
Action Adventure Game 89

randomised pilot study. In: Proceedings of the 7th in gait rehabilitation. MODSIM World (2007). Rich-
ICDVRAT with ArtAbilitation, pp. 229–235. Maia mond, VA
(2008) Verdonck, M., Ryan, S.: Mainstream technology as an
Eng, K., Siekierka, E., Pyk, P., Chevrier, E., Hauser, Y., occupational therapy tool: technophobe or technogeek? A
Cameirao, M., Holper, L., Hägni, K., Zimmerli, L., Br. J. Occup. Ther. 71(6), 253–256 (2008)
Duff, A.: Interactive visuo-motor therapy system for Wood, S., Murillo, N., Bach-y-Rita, P., Leder, R., Marks,
stroke rehabilitation. Med. Biol. Eng. Comput. 45(9), J., Page, S.: Motivating, game-based stroke rehabilita-
901–907 (2007) tion: a brief report. Top. Stroke Rehabil. 10(2),
Flores, E., Tobon, G., Cavallaro, E., Cavallaro, F., Perry, J., 134–140 (2003)
Keller, T.: Improving Patient Motivation in Game
Development for Motor Deficit Rehabilitation. ACM,
New York (2008)
Flynn, S., Lange, B., Yeh, S., Rizzo, A.: Virtual reality
rehabilitation–what do users with disabilities want? in
the Proceedings of ICDVRAT 2008, Maia & Porto, Accessible Game
Portugal (2008) Development
Fung, J., Richards, C., Malouin, F., McFadyen, B.,
Lamontagne, A.: A treadmill and motion coupled vir-
tual reality system for gait training post-stroke. Cyber- ▶ Visual Accessibility in Computer Games
psychol. Behav. 9(2), 157–162 (2006)
Goude, D., Björk, S., Rydmark, M.: Game design in virtual
reality systems for stroke rehabilitation. Stud. Health
Technol. Inform. 125, 146 (2007)
Ma, M., McNeill, M., Charles, D., McDonough, S., Accommodation &
Crosbie, J., Oliver, L., McGoldrick, C.: Adaptive vir- Convergence
tual reality games for rehabilitation of motor disorders.
Universal Access in Human-Computer Interaction. ▶ Spatial Perception in Virtual Environments
Ambient Interaction, pp. 681–690. (2007). Bejing,
China
Mei, C., Mason, L., Quarles, J.: How 3D Virtual Humans
Built by Adolescents with ASD Affect Their 3D Inter-
actions. ASSETS, Lisbon (2015) Accord
Merians, A., Poizner, H., Boian, R., Burdea, G.,
Adamovich, S.: Sensorimotor training in a virtual real-
ity environment: does it improve functional recovery ▶ Abstraction and Stylized Design in 3D Ani-
poststroke? Neurorehabil. Neural Repair 20(2), mated Films: Extrapolation of 2D Animation
252 (2006) Design
Samaraweera, G., Perdomo, A., Quarles, J.: Applying
latency to half of a self-avatar’s body to change real
walking patterns. In: Virtual Reality (VR), 2015 IEEE.
IEEE (2015). Arles, France
SAVELab: VR Walk for Autism. From https://play.google. ACI
com/store/apps/details?id¼com.SAVELab.AutismWalk
&hl¼en (2015a)
SAVELab: VR Walk MS: San Antonio. From https://play. ▶ Engaging Dogs with Computer Screens: Ani-
google.com/store/apps/details?id¼com.SAVELab.MS mal-Computer Interaction
Walk&hl¼en (2015b)
Sütbeyaz, S., Yavuzer, G., Sezer, N., Koseoglu, B.: Mirror
therapy enhances lower-extremity motor recovery and
motor functioning after stroke: a randomized con-
trolled trial. Arch. Phys. Med. Rehabil. 88(5), Action Adventure Game
555–559 (2007)
Sveistrup, H.: Motor rehabilitation using virtual reality.
J. NeuroEng. Rehabil. 1(1), 10 (2004) ▶ Disney Toontown Online, a Massively Multi-
Thikey, H., van Wjick, F., Grealy, M., Rowe, P.: player Online Role-Playing Game
A need for meaningful visual feedback of lower ▶ Fantasy XVI Online, a Massively Multiplayer
extremity function after stroke. IEEE (2011). Dub- Online Role-Playing Game
lin, Ireland
Tierney, N., Crouch, J., Garcia, H., Walker, M., Van Lunen, ▶ God of War (2018), an Action-Adventure Game
B., DeLeo, G., Maihafer, G., Ringleb, S.: Virtual reality ▶ The Elder Scrolls V Skyrim
90 Action Role-Playing Game

Action Role-Playing Game Adaptive Music

▶ Dark Souls III, an Analysis Shlomo Dubnov

▶ Kingdom Hearts (2002): An Analysis Department of Music and Computer Science and
Engineering, University of California San Diego,
San Diego, CA, USA
Action RPG
Synonyms
▶ Dark Souls RPG Through the Lens of Challenge
Algorithmic composition; Dynamic music; Gen-
erative music; Interactive music; Nonlinear music
Action-Adventure composition; Organic music; Procedural music;
Video game music middleware
▶ God of War, an Analysis

Definition
Action-Adventure Game Adaptive Music refers to a computer entertain-
ment system for dynamically composing a music
▶ Assassin’s Creed, an Analysis
soundtrack in response to dynamic and
▶ Legend of Zelda Breath of the Wild and the
unpredictable actions and events happening in a
Lens of Curiosity
video game. Since its inception, the concept has
evolved into a general approach to composition
that takes into account the possibility of changing
the musical material during gameplay, sometimes
Active Learning termed “composing for opportunities.” The
underlying principle is that music in a game
▶ Games and Active Aging
could be initiated, triggered, mixed, or completely
generated by directing the system in a way that is
aesthetically appropriate and natural to the game
and the user’s actions. The common architecture
Active Videogames of such a system comprises a database of musical
segments or clips containing one or more musical
▶ Games and Active Aging
sequences that could be launched in response to
the game logic and looped as long as the scene is
playing. Additional clips could be added or mixed
Activist Games or subtracted/muted to change the overall contents
of the music. Special treatment is done to enable
▶ Political Game Design smooth transitions between clips, as well as
adding short musical or sound elements, often
called stingers, that might be triggered in response
to user actions or facilitated switching music in
Adaptation the case of transitions by inserting music for end-
ing or starting a musical segment. In some cases,
▶ Redesigning Games for New Interfaces and musical algorithms can be deployed to generate
Platforms the music materials directly by sending note or
Adaptive Music 91

musical playing instructions to a synthesis engine. beat, so that the rhythmical structure of the music
Technically this requires to have a synthesizer or will not be interrupted. In cases where such
virtual music instrument available in the game matching is not possible, other transition methods A
engine to produce the sounds, which is in addition are used, such as inserting fixed music segments
to the more common use of an adaptive playback that either create an ending of the previous music
and mixer of prerecorded music segments stored and starting of new materials, or create a gradual
as audio files. change, such as speeding or slowing down to
match the tempo difference from the source
(current) to the target (next) clip. Since such tran-
Adaptive Composition Methods sition clips are not played in a loop, they are often
treated as stingers, or singularly triggered musical
The main two common methods for adaptive events. Methods for automatically determining
music are horizontal resequencing and vertical valid transitions between points in pre-recorded
layering or remixing (Sweet 2014). The names music requires analysis of the audio for repetitions
imply operations performed by a game sound by extraction of musical features. Research on
system on pre-recorded musical materials that automatic transitions detection is discussed in
are arranged in a traditional multitrack view, the algorithmic composition section.
where time is represented horizontally, and the
different musical tracks are lined up horizontally.
According to such arrangement, each vertical jux- Music Versus Sound Effects
taposition of musical tracks corresponds to a valid
musical combination of simultaneous music mate- A distinction between music and sound effects
rials, often arranged according to the different was traditionally evident as music would be pro-
musical instruments performing the music. duced by musical instruments and composed in
The adaptive operations correspond to skip- some well-established musical style. Sound
ping forward or backward in time and muting or effects or so-called Foley effects, named after
changing the relative volume/balance of musical Jack Foley, were designed to create sounds that
playback between tracks. Since in practice the correspond to physical events that are nonmusical
skips cannot be done between random time-points in their nature and largely correspond to sounds
in a track, the music is segmented or arranged made by objects present in the game. Since the
ahead of time into fixed clips that can be repeated sound engines already in the early game consoles
or looped, or can be smoothly switched to differ- allowed both synthesis of simple tones and
ent clips, with transitions happening only between playback of short recordings, the same system
ending and starting points of the clips. Accord- was used to trigger sound effects and
ingly, in some modern software, the geometrical music. Historically, the earliest sounds in pin-
arrangement of musical data is flipped so that the ball machines were produced by physical devices,
different clips are stacked vertically, with each such as solenoids hitting bells, which could be all
horizontal arrangement corresponding to a spe- classified as sound effects. Music playback was
cific scene or level in the game, with tracks often made available in the early game arcades,
containing concurrent clips arranged horizontally. but mostly in order to register the games as juke-
Such grid/spreadsheet-like data arrangement cre- boxes rather than gambling machines that were
ates a correspondence between game scenes or illegal (Beep 2016). As the availability of both
game levels and the musical data arrangement, music generation and playback of sound effects
where changes in game scenes or levels corre- was historically present in sound games, the com-
spond to moving up or down on the musical bination of both types of sounds became part of
data grid. the sound palette available to music game com-
Special treatment is given to transitions, where posers, blending composition and sound design
switching between clips is done on the musical into one common compositional strategy. In
92 Adaptive Music

parallel, experimental musical practices that were provide multiple alternative combinations of
unrelated to video games have been explored, musical materials rather than creating a single
since the end of nineteenth and beginning of timeline of musical events. Accordingly, the
twentieth century, the possibilities for including term nonlinear composition is often used to
nonmusical sounds in musical compositions. With describe the branching structure of musical mate-
experimental musical genres moving to main- rials in contrast to a traditional single sequential
stream in classical and pop music, game com- arrangement of a linear musical form. This creates
posers today blend traditional instruments with a novel approach to planning a composition by
sound effect as part of their video game music creating a database of sounds and identifying one
and soundtrack design. or more musical sequences to have one or more
decision points. The decision points within the
database then comprise a composing decision
Music Role in Video Game tree, with the decision points marking places
where branches in the performance of the musical
A common distinction is between Diegetic and sequences may occur. A sound driver interprets
Non-Diegetic Music, a term common in film each decision point within musical sequences
music as well. Diegetic audio refers to sound depending on the unpredicted actions and events
that originates in the game events themselves, initiated by a directing system that is linked to
which could be either sound effects related to events in the game. The directing system may
physical events happening in the game, or also query the state of the sound driver to adjust
music that originates from a game scene, such the branching decisions, such as preventing too
as musical band or radio present in the scene. many loops of playing the same musical material
Non-Diegetic or extra-diegetic music refers to if the game activities do not progress at sufficient
composed music that accompanies the scene, pace, or initiating some branching or transitions
usually to establish atmosphere or identify a randomly, unrelated to the actions of the game, in
character or situation. A simple criterion to dis- order to create more variety in the music. Other
tinguish between Diegetic and Non-Diegetic direct commands may be initiated by the directing
Music is whether a character in the game would system for controlling the performance of the
have heard that music or sound. Another techni- sound driver or playback device, such as adding
cal difference between Diegetic and Non- or removing (mixing) multiple simultaneous
Diegetic music is that the first one is usually sounds synchronously or asynchronously with
localized and treated in the game using 3D another playing clip. The synchronous versus
audio engine, while Non-Diegetic music is ren- asynchronous combination is designated
dered stereo, also called 2D audio. Among the according to delaying playback of new sounds in
roles of Non-Diegetic music, aspects of emo- order to match the rhythmic or beat structure
tional subtext, intensifying the game narrative, between musical materials or triggering the new
informing about progress or level of gameplay, sound at will, respectively, as well as including
are often mentioned. Both Diegetic and Non- additional decisions about the fade settings,
Diegetic music are important for providing a pickup or count-in beats, or criteria for selection
sense of immersion in the game. of the target sounds. The design of the branching
logic allows for both horizontal resequencing and
vertical layering composition techniques to be
Logic of Music Triggering and implemented by creating a hierarchical or coarse-
Combinations to-fine tree structure with main branches
corresponding to transitions between collections
One of the distinctive features in Adaptive Music of sounds arranged into themes, and fine grained
is that the arrangement of musical material has to decisions for track selection and combination
Adaptive Music 93

controlled by game levels within a theme. The Algorithmic Composition Methods

details of logic handling are different across dif-
ferent systems. For example, in the Elias adaptive An alternative or complementary method to hori- A
music system requests to change level, use an zontal resequencing and vertical layering that
action preset, play a stinger, etc., react differently operates on pre-recorded materials, are so-called
for each track according to their own rules, such algorithmic methods that use computer code to
as if a string melody track is playing a long generate music procedurally (Collins 2009;
phrase, it might wait to change after that line is Duarte 2020). A possible distinction between
complete according to a rule set by the composer, algorithmic and generative methods is that algo-
while the percussion track might be allowed to rithms often refer to rule based, largely determin-
change right away. Elias has a small AI that istic operations that encode musical knowledge
listens for “better” places to change, such as if using formal grammars or other heuristic
the rule for the Trumpet track is to change on bar methods, while generative methods often refer to
5 beat 1 and the musician is slightly ahead of the random generation or stochastic processes.
beat, the system will adapt to that change point Between the two extremes, many algorithmic
automatically. composition methods combine both randomness
and rules to create musical materials. Some exam-
ples of such combined methods are Markov
MIDI and Audio models that may be constrained or learned from
examples based on some underlying musical
MIDI and Audio are the two most common rep- model, stochastic grammars, Augmented Transi-
resentation formats of music that are handled by tion Networks and Petri-nets, or abstract mathe-
sound engines. The distinction is often important matical processes such as genetic
for sound engines as the type of adaptive opera- algorithms, L-systems, and artificial life. One
tions that are possible in each representation are important aspect in using any type of procedural
very different. In pre-recorded music referred as musical generation is the mapping of computa-
Audio, the adaptive operations are based on play- tional operations to musical parameters. Separate
back and mixing of musical clips. In MIDI processes might be used to model rhythm versus
(Musical Instrument Digital Interface) that sound note or pitch structures.
is generated by sending performance instruction More recently Neural networks have become a
to an electronic synthesizer. The operation possi- focus of intensive research in music generation,
ble using MIDI includes control of individual where musical rules are learned implicitly from
notes and instruments, thus allowing change in large enough corpuses of examples. Common
the instruments that perform the music, as well methods for such learning and generation are
as adding or removing notes using algorithmic recurrent neural networks (RNN), or combination
methods. Historically, adaptive operations such of RNNs with variational models such as Varia-
as change in tempo or shifting the pitch of music tional AutoEncodersx1 (VAE), Generative Adver-
were reserved to MIDI, but developments in audio sarial Networks (GAN), and more recently
signal processing allow today to change the speed Transformers (Briot et al. 2019). These methods
of playback or change the pitch of an audio operate mostly in the symbolic domain. Auto-
recording without changing the speed of playback matic remixing of audio files can be done by
with minimal audible artifacts. Other music for- applying string matching methods to sequences
mats, such as mod files that are used in music of features extracted from the acoustic signal. One
engines known as musical trackers, are also com- common method is using a Factor Oracle algo-
mon in video games music. Mod files combine rithm to create an automaton structure according
some of the aspects of recorded clips and note to suffix links connecting similar subsequences in
triggering instructions. the feature representation (Assayag and Dubnov
94 Adaptive Music

2004). Horizontal remixing is done by skipping Historical Developments

the audio playback according to the suffix links
that assures that the transitions are sounding Interactive sound in video games existed since
smooth since they share common history. This Pong in 1972, mostly to emulate sounds of pinball
method is operating according to a principle sim- machines in arcade games. Generative music in
ilar to a Markov model, except that the condition- video games was introduced in Ballbazer in 1984
ing on the past can be of different length. Due to for producing music segments between games and
this property the audio remixing models are also menu (Plut and Pasquier 2020). The music was
known as Variable Memory Markov Models or generated by a combination of short musical riffs
Variable Markov Oracle (VMO) (Wang for melody, bass, chords, and drums, with rules for
et al. 2017). choosing the next riff, the speed, and volume of
music. The first fully interactive music system
iMuse was developed and used in the game Mon-
Middleware Software key Island 2 in 1991 (Land and McConnel 1994).
The system is based on symbolic music represen-
Audio and Music Middleware are software tools, tation and implements a branching structure of
often developed by third parties that are not adaptive music composition, where the decisions
related to the game engine software, whose pur- on which music to play and what transitions to
pose is to help game composers and sound make are based on the game situations and
designers to create and integrate audio into the designed transitions between the scenes.
game. Most modern game engines, such as DirectMusic by Microsoft is a low-level API that
Unity or Unreal Engine, provision basic audio supports composition and playback of dynamic
operations that allow direct integration into the musical soundtracks based on stored composi-
game by direct calls to playback and mixing oper- tional material. The system was first introduced
ations. The difficulty with working directly with in 1996 as part of the a DirectX library, and
game development software is that the composi- became part of Windows 98 Second Edition in
tional process is very different from the usual February 1999, but was deprecated since 2000
Digital Audio Workstation (DAW) software that with parts of it moving to other low level audio
is used for composing or producing music for API, while some of the MIDI functionality
non-game applications. Accordingly, one of the remained in later DirectX versions. DirectMusic
main advantages of Middleware is providing a engine can be considered as an improvement on
more natural composition interface and easy iMuse in terms of its ability to simultaneously
work method for composers. Other more technical play and manipulate musical events for vertical
advantages of using Middleware are easy integra- remixing, but it lacks support for the logical struc-
tion into major game engines without dependency ture for horizontal resequencing of iMuse. Many
on game programmers, high-quality audio with other systems have been developed over the years,
control of the audio footprint and support for providing different levels of horizontal or vertical
various compression formats, integrated sound adaptivity, with various generative abilities and
effect and support of audio plug-ins, multi- triggering logic control. For example, a recently
threaded platform specific optimization, and so introduced Adaptive Music System (AMS)
on. Some of the major names in the industry (Hutchings and McCormack 2020) uses a combi-
include FMOD, Wise-Audiokinetic, Tazman- nation of musical rules with agent heuristics for
Audio Fabric, Elias Software, and Criware composing and arranging music according to
(Nogueira 2019). In terms of adaptive music emotional categories, which are linked to game
tools, some of the substantial differences between object and game environment through a knowl-
the different middleware are their level of support edge graph that is activated based on game
for MIDI and sound design implementation. actions.
Additive Manufacturing 95

Other Uses of Adaptive Audio Synthesis and Creative Systems. Springer, Appleton,
Wisconsin (2019)
Collins, K.: An introduction to procedural music in video
The idea of opening musical composition to chance games. Contemp. Music. Rev. 28(1), 5–15 (2009) A
operations and more recently to interaction with Duarte, A.E.L.: Algorithmic interactive music generation
outside contextual information is not limited to in videogames. SoundEffects. 9(1) (2020)
video games. The field of music meta-creation Hutchings, P., McCormack, J.: Adaptive music composi-
tion for games. IEEE Trans. Games. 12(3), 270–
explores the use of higher level constraints to 280 (2020)
guide algorithmic and generative systems IEEE SA (Standards Association). https://standards.ieee.
according to user specifications (Pasquier et al. org/project/1599.html (2018)
2017). The field of Machine Improvisation com- Land, M.Z., McConnel, P.N.: Method and apparatus for
dynamically composing music and sound effects using
bines machine listening with generative systems to a computer entertainment system, assigned to
create a real-time interaction between a machine LucasArts Entertainment Company. US patent
and a musician (Wang et al. 2017). Automatic 5,315,057. (1994)
sound effect generation from video is being studied Nogueira, T., Audio Middleware: Why would I want it in
my game? Gamasutra, on 19 July 2019. https://www.
using deep neural networks that learn a mapping gamasutra.com/blogs/TheoNogueira/20190719/
from video frames to audio, and using a video 346915/Audio_Middleware_Why_would_I_want_it_
encoder to drive a neural network type of sound in_my_game.php
generator (Zhou et al. 2017). These works differ in Pasquier, P., Eigenfeldt, A., Bown, O., Dubnov, S.: An
introduction to musical metacreation. Comput. Enter-
terms of the specific types of objects they are tain. 14, 2:1–2:14 (2017)
modeling and level of synchronization between Plut, C., Pasquier, P.: Generative music in video games:
sound and image. Sensors and depth cameras, state of the art, challenges, and prospects. Entertain.
such as Kinect, have been used in experimental Comput. 33, 100337 (2020). https://doi.org/10.1016/j.
entcom.2019.100337
dance to generate sounds in response to movement Sweet, M.: Writing Interactive Music for Video Games: a
(Berg et al. 2012). Other applications of adaptive Composer’s Guide. Addison-Wesley Professional
music include theatre, circus, and various types of (2014)
interactive multimedia, some of which is addressed Wang, C., Hsu, J., Dubnov, S.: Machine improvisation
with variable Markov Oracle: toward guided and struc-
by the IEEE 1599 standard for Multilayer Repre- tured improvisation. Comput. Entertain. 14, 4:1–4:
sentation of Music Using eXtensible Markup Lan- 18 (2017)
guage (XML) (IEEE SA 2018). The focus of such Zhou, Y., Wang, Z., Fang, C., Bui, T., Berg, T.L.: Visual to
applications is largely in regards to synchronization Sound: Generating Natural Sound for Videos in the
Wild, arXiv:1712.01393 (2017)
and logic of controlling different media assets dur-
ing an execution of a complex but mostly linear
presentation, with less emphasis on generative,
algorithmic, or remixing (horizontal or vertical)
techniques that are dominant in video games. Adaptive Music Systems

▶ Dynamic Music Generation: Audio Analysis-

References Synthesis Methods

Assayag, G., Dubnov, S.: Using factor oracles for machine

improvisation. Soft. Comput. 8, 604–610 (2004)
Beep: A Documentary History of Game Sound, (2016).
https://www.gamessound.com/ Additive Manufacturing
Berg, T.L., Chattopadhyay, D., Schedel, M., Vallier, T.:
Interactive Music: Human Motion Initiated Music Gen- ▶ 3D Printing, History of
eration Using Skeletal Tracking By Kinect. SEAMUS
▶ Open Source 3D Printing, History of
(2012)
Briot, J.-P., Hadjeres, G., Pachet, F.-D.: Deep Learning ▶ Tactile Visualization and 3D Printing for
Techniques for Music Generation, Computational Education
96 Administration

Administration AI Motion

▶ Game Development Leadership Tips ▶ Navigation Artificial Intelligence

Adventure Game Algorithmic Composition

▶ Dark Souls III, an Analysis ▶ Adaptive Music

▶ Procedural Audio in Video Games

Advertising
Alienation
▶ Integrating Virtual Reality and Augmented
Reality into Advertising Campaigns: History, ▶ Design of Alienation in Video Games
Technology, and Future Trends

Allegory
Affect
▶ Narrative Design
▶ Emotional Congruence in Video Game Audio

Ambient-Embedded
Affective Computing Interaction Surfaces

▶ Emotion-Based 3D CG Character Behaviors ▶ Tangible Surface-Based Interactions

Affective Ludology Ambisonic Binaural Rendering

▶ Player Experience, Design and Research ▶ Overview of Virtual Ambisonic Systems

After Effects American Sign Language

(ASL)
▶ Virtual Reality Stereo Post-Conversion After
Effects Workflow ▶ American Sign Language Detection
American Sign Language Detection 97

Education Act was changed in 1993 to acknowl-

American Sign Language edge ASL and LSQ as languages of instruction for
Detection deaf children (Carbin and Smith 2013). A
Due to a language contact situation, ASL was
Rupendra Raavi and Patrick C. K. Hung born in the early nineteenth century at the Amer-
Faculty of Business and IT, Ontario Tech ican School for the Deaf (ASD) in West Hart-
University, Oshawa, ON, Canada ford, Connecticut, in the United States. Since
then, schools for the deaf community organiza-
tions have worked hard to spread the usage of
Synonyms ASL. Despite the widespread use of ASL, no
reliable count of its users has been conducted.
American Sign Language (ASL); Convolutional According to reliable estimates, there are
Neural Network (CNN); Mean Average Precision between 250,000 and 500,000 ASL users in the
(MAP); World Health Organization (WHO); You United States, including a substantial number of
Only Look Once (YOLO) children of deaf individuals (Carbin and Smith
2013).

Definition
ASL Machine Learning Model
American Sign Language (ASL) detection is
based on an object detection machine learning You Only Look Once (YOLO) is a famous object
algorithm, which helps deaf and hard of hearing detection algorithm with around 50 Mean Aver-
people in terms of communication. age Precision (MAP), which has a set of
Convolutional Neural Networks (CNN) to help
the algorithm extract the visual features of an
Motivation and Background image upon which it is being trained (Kuo 2016;
Lu et al. 2020; Redmon and Farhadi 2018). Then
World Health Organization (WHO) projected that with the help of these visual characteristics intro-
by the year 2050, there would be around 2.5 duced to the model, the model detects the image in
billion people with some degree of hearing loss real-time videos/pictures. In this scenario, ASL
(WHO 2021). alphabet images are trained, and then when these
Most deaf anglophones adopt American Sign alphabets are shown to the camera, the algorithm
Language (ASL), which is a common “continen- will detect them.
tal” language. On the other side, British Sign Deaf and hard of hearing people usually com-
Language (BSL) and French Sign Language municate with a person with the help of an ASL
(LSF) became obsolete as people started to use interpreter. Due to the Covid-19 pandemic, every-
ASL. The structure and rules of ASL were dis- thing turned out to be virtual, so using ASL inter-
covered in the 1970s and early 1980s. Manual preters appears to be more challenging. Still,
communication was revived in the classroom as hiring an ASL interpreter, for instance, is not
many artificial manual codes for spoken English always affordable. Hence, a machine learning
or French. Since the late 1980s and early 1990s, model for ASL detection can help the deaf and
deafer people have sought sign language instruc- blind communicate without the help of a human
tion. Some schools are multilingual interpreter. For example, there are 26 alphabet
(ASL/English) and bicultural (Deaf/Hearing). hand signs for 26 alphabets and some hand signs
For example, the deaf anglophone community in for some phrases. A machine learning model such
Canada uses ASL. Manitoba led the way in 1988, as the YOLO object detection algorithm can be
with Alberta following in 1990. Alberta added built to detect these signs in real-time by computer
ASL to its provincial resolution. The Ontario vision (Ullah and Ullah 2020).
98 American Sign Language Detection

American Sign Language Detection, Fig. 1 ASL detection process by YOLO

The accuracy metrics used for YOLO are based state-of-art object detection models (Liu et al.
on the intersection over union concept. The for- 2016). When the camera turns on, the machine
mula used to calculate it is the ratio in-between learning model in the background will try to draw
area of overlap and intersection area. Figure 1 several bounding boxes and check whether any
describes the working of the algorithm initially. hand signs of American sign language alphabets
The machine learning YOLO-based object detec- are detected. If the machine learning model
tion algorithm will look over all the frames com- detects any hand signs, it highlights the hand
ing through the live video, and then the frames signs with the bounding boxes and will display
will be split into the bounding boxes in the train- the alphabet of the detection. Next, the machine
ing phase. Then for every bounding box, the learning model highlights the hand signs with the
model will depict the output, for example, 4  bounding boxes and then will display alphabets of
4  18 (similar to the one in the training phase). that detection. Each alphabet that is detected is
Here 4  4 is the grid used for the rectangular attached to the top of the screen. Then the machine
bounding box detection, and it is modified as per learning model will repeat the same process on all
individual preference and application. The pri- the frames coming out of the camera and find if
mary nine values will be correlated with the any frame can match the hand signs of the ASL.
anchor box of 1. Here the immediate value will
be the likeliness of an object within the bounding
box. Values from 3 to 6 will be the bounding box Cross-References
coordinates for that image. The final three values
will disclose where the actual image belongs. The ▶ Machine Learning
coming nine values are for the anchor box of 2. At
last non-max suppression is enforced over the
depicted bounding boxes to get the individual References
depiction of the image (Sharma 2021).
Other object detection models include Carbin, C., Smith, D.: Deaf culture. In: The Canadian
Encyclopedia. Retrieved from https://www.thecanadia
mobilenetssd, inception, vgg16, and efficient net.
nencyclopedia.ca/en/article/deaf-culture (2013)
These models can be used to build a similar sys- Kuo, C.C.J.: Understanding convolutional neural networks
tem, but YOLO usually outperforms all the other with a mathematical model. J. Vis. Commun. Image
Among Us and Its Popularity During COVID-19 Pandemic 99

Represent. 41, 406–413 (2016). https://doi.org/10. reasoning to uncover a player’s hidden role in a
1016/j.jvcir.2016.11.003 role-based game.
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S.,
Fu, C.-Y., Berg, A.C.: SSD: single shot multibox detec- Murder-Mystery: A genre of fictional media A
tor. In: Computer Vision – ECCV 2016, vol. 9905, that typically involves a crime or death that
pp. 21–37. Springer International Publishing (2016). needs to be solved by the central character(s).
https://doi.org/10.1007/978-3-319-46448-0_2 RPG (Role-playing Game): A genre of video
Lu, Y., Zhang, L., Xie, W.: YOLO-compact: an efficient
YOLO network for single category real-time object games that allow the player to fill and act out
detection. 2020 Chinese Control and Decision Confer- certain roles the gameplay provides.
ence (CCDC). (2020). https://doi.org/10.1109/ Party Game: A genre of games that facilitates
ccdc49329.2020.9164580 social interactions among a group of members.
Redmon, J., & Farhadi, A. (2018). YOLOv3: An Incre-
mental Improvement. https://doi.org/10.48550/
ARXIV.1804.02767
Sharma, P.: YOLO framework: object detection using Introduction
YOLO. Analytics Vidhya. Retrieved Apr 25, 2022,
from https://www.analyticsvidhya.com/blog/2018/12/
practical-guide-object-detection-yolo-framewor- Among Us is an online multiplayer game that
python/ (2021, August 26) incorporates the formats of social deduction,
Ullah, M.B., Ullah, M.B.: CPU based YOLO: a real time murder-mystery, RPG, and party game genres.
object detection algorithm. 2020 IEEE Region 10 Sym- The game was developed and published by the
posium (TENSYMP). (2020). https://doi.org/10.1109/
tensymp50017.2020.9230778 Washington-based American studio Innersloth on
WHO: Vision impairment and blindness. World Health June 15, 2018, but did not reach popularity until
Organization. Retrieved Apr 25, 2022, from https:// mid-2020s (Carless 2020).
www.who.int/news-room/fact-sheets/detail/blindness- The game’s popularity and player base, as of
and-visual-impairment#:~:text¼Globally%2C%20at%
20least%202.2%20billion,uncorrected%20refractive% time of writing, continues to grow with the num-
20errors%20and%20cataracts (2021, October 14) ber of active players reaching 3.8 million by late
September 2020 (Lugris 2020). A recent update
reports that the game grew its user base 1600% in
Among Us and Its Popularity 8 months (Jain 2021). On Twitch, a platform
During COVID-19 Pandemic where people can host streams and allow others
to watch, Among Us is ranked third for the number
Alyssa Bump1 and Sercan Şengün2,3 of average viewers over the course of a week with
1 32,069,859 hours watched as of October 3, 2020
Creative Technologies Program, Illinois State
University, Normal, IL, USA (SullyGnome 2020).
2 Robinson (2021) reports on the elicited emo-
Wonsook Kim School of Art, Illinois State
University, Normal, IL, USA tions and engagement of the game and finds that
3 compared to more photo realistic games,
Massachusetts Institute of Technology CSAIL,
Cambridge, MA, USA Among Us created consistent emotional reac-
tions – dominantly excitement, curiosity, and
relief.
Synonyms

Murder-mystery; Party game; RPG; Social deduc- History

tion game
Released in June 2018, Among Us was meant to
be a local-multiplayer mobile game. Initially
Definitions dubbed “spacemafia” on the iOS AppStore, the
game had a poor marketing campaign and a very
Social Deduction Game: A game genre that small player count. However, this small player
focuses on the use of logic and deductive count was very vocal about the game’s
100 Among Us and Its Popularity During COVID-19 Pandemic

development, pushing the developers to continue in-game purchases, their “pet” that follows
to work on it even after the team’s attempts to them around the map. This differs from other
abandon the project. Throughout the year, social-deduction games as those characters are
Innersloth had released several updates leading not as customizable or do not offer in-game
up to the release of the game for PC in November interactivity.
2018 and making it available for cross play There are two roles: the role of the Crewmate
between PC and mobile. and the role of the Imposter. The objective of the
However, the game’s current popularity is game is for either role to win. If the Crewmates do
credited to the YouTuber and Twitch streamer all their tasks or vote off all the Imposters, then the
Sodapoppin. He was among the first streamers Crewmates win the game. If the Imposters man-
to publicly show the otherwise unknown age to kill more Crewmates than the number
game. Other streams quickly followed, further needed that can vote them off, or if a critical
boosting the game’s exposure. Through the emergency timer on a sabotage event counts
events of the COVID-19 pandemic forcing down to zero, then it is a victory for the Imposter
gamers to either watch or play more than (s). The number of Crewmates left surviving that
what they previously could, the amounts of would enable an Imposter win is one Crewmate to
viewers and players for the game continued one Imposter. You can have up to three Imposters
to skyrocket throughout the following months and a total of 10 players. The minimum number of
as it allowed socializing through the stay-at- players you can have in a game is four. Roles are
home orders and social distancing randomly assigned each game. Imposters have a
regulations. cool down timer for killing and for sabotaging the
facility.
Upon finding a dead Crewmate left behind by
Gameplay the Imposter, all players will come together to
discuss through an in-game chat client about
Among Us was heavily inspired by other social- who is or is not an Imposter in the duration of a
deduction games such as Mafia (created by limited time period. The Imposter’s role during
Dimitry Davidoff, 1986, as attributed by Haffner discussions is to lie their way through the game
1999) and Secret Hitler (created and published by until the timer counts down to zero. People
Goat, Wolf, & Cabbage LLC, 2016), and provides suspected to be Imposter will be voted off by the
an online multiplayer experience like other games majority vote. Those who are not voted off con-
of this genre like Town of Salem (developed and tinue into the next round and the process repeats
published by BlankMediaGames, 2018). How- again. Meetings can also be issued if the emer-
ever, a number of different traits make the gency meeting button is activated during
gameplay unique and entertaining for both the gameplay.
player and the viewer. Players who are not “alive” or were voted out
The game offers three maps to play on: a still have an ability to play, providing another
spaceship, a headquarters, and an alien planet interactive experience different from other
base (respectively named The Skeld, Mira HQ, social-deduction games. These players appear
and Polus), as well as a top-down view of the as ghosts and are able to roam freely around the
players and the map. The players can interact map and finish their tasks, which helps to aid in
with the map by exploring and doing various Crewmate victory. If the ghost has been an
tasks. Despite the top-down view, players have Imposter, they are still able to sabotage the
a limited scope of vision so they cannot see map, but are unable to kill any living player.
everything that other players do or where Deceased or voted off players are unable to com-
exactly they are. The game also offers custom- municate with the living but are able to commu-
ization to players, allowing them to change nicate with each other through the in-game chat
their color, outfit, and if they have made client.
Among Us and Its Popularity During COVID-19 Pandemic 101

Socialization not give anything away. Dead players can talk

freely but must mute themselves when discus-
The event of the COVID-19 pandemic in 2020 has sions are happening. There is a playing order A
caused a shift in social dynamics and institutions. when Discord is involved that is not present
With the advocacy for social distancing, individ- when players rely only on the in-game chat client –
uals were looking for other ways to communicate which does not utilize any voice activity and
and conduct play. A popular outlet for this is video instead focuses on text-based communication.
games and one of these video games is Among Us On Twitch streams and YouTube videos, evidence
(Kriz 2020). of one’s guiltiness is also much more effectively
There had been not as much pressure in previ- discussed on a Discord call than it is in a game
ous years for the need to socialize and interact lacking verbal communication. Without verbal
with other individuals. While games have been communication, members tend to form a band-
partially viewed negatively before, 2020 has wagon out of wild accusations just because some-
shifted opinions for many officials including the one said so with no questions asked.
World Health Organization (WHO). In 2013, Among Us has also made it into Internet meme
gaming disorder was added to the DSM-5, a col- culture and has given rise to a variety of artistic
lective medical source that describes a variety of renditions, ranging from animated videos to peo-
disorders and ways to diagnose them (Petry and ple making their own Crewmate personas.
O’Brien 2013). However, during the pandemic,
WHO strongly encouraged the use of video games
during the pandemic, allowing a gateway for Cross-References
stress and anxiety relief and an outlet for social
interaction (Snider 2020). ▶ Social-, Mobile-, and Multi-Player-Games and
Following this, other social games like Fall Their Impact on Today’s Online Entertainment
Guys (developed by Mediatonic and published Industry
by Devolver Digital in 2020) and Animal Cross-
ing New Horizons (developed and published by
Nintendo in 2020) grew in trend. It is no surprise, References
then, that Among Us has reached among the top
Carless, S.: Behind the dizzying ride to the top for among
trending games of 2020. In fact, The Guardian us. Gamasutra. https://www.gamasutra.com/blogs/
offered Among Us as the “the ultimate party game SimonCarless/20200910/369968/Behind_the_dizzy
of the paranoid Covid era” (Stuart 2020). Among ing_ride_to_the_top_for_Among_Us.php (2020,
Us strongly encourages socializing, and not just September 10)
Haffner, F.: Questions to Dimitry Davidoff about the crea-
through its in-game chat client. Discord, a real-life tion of Mafia. Jeuxsoc.fr. https://jeuxsoc.fr/?
chat client geared towards gamers, has hit a spike principal¼/jeu/mafid (1999, February 2)
in downloads since Among Us reached popularity Jain, P.: How among us grew its user base by 1600% in
since players are utilizing Discord’s voice calling 8 months [growth case study]. Moengage.com. https://
www.moengage.com/blog/among-us-user-growth-
system in order to play the game together and mobile-gaming/ (2021, June 1)
make the game livelier with human voices. This Kriz, W.C.: Gaming in the time of COVID-19. Simul.
encourages social interactions and social bonding Gaming. 51(4) (2020). https://journals.sagepub.com/
with one another, a very important staple to men- doi/full/10.1177/1046878120931602
Lugris, M.: Among us had 3.8 million concurrent players
tal health in 2020’s world. last weekend. TheGamer.com. https://www.thegamer.
Playing over Discord and playing voiceless com/among-us-3-8-million-concurrent-players-last-
provides different experiences and methods. For weekend/ (2020, September 29)
example, playing the game through a Discord Petry, N.M., O’Brien, C.P.: Internet gaming disorder and
the DSM-5. Addiction. 108, 1186–1187 (2013). https://
phone call will require you to time your speeches. onlinelibrary.wiley.com/doi/pdf/10.1111/add.12162
Most players end up muting their microphones Robinson, J.: Affective teamwork: a comparative study on
during gameplay until discussion meetings, as to the effectiveness of emotional interaction and
102 Analog Prototyping for Digital Game Design

collaboration between players in cooperative survival games; they are expected to be fun. As Ferrara
games. JoelRobinson.co.uk. https://joelrobinson.co.uk/ points out, games are “inherently negotiated expe-
files/Affective_Teamwork.pdf (2021)
Snider, M.: Video games can be a healthy social pastime riences; the designer normally just defines the
during coronavirus pandemic. USA Today. https:// parameters of play, within which the players
www.usatoday.com/story/tech/gaming/2020/03/28/ bring the game to life” (Ferrara 2012), meaning
video-games-whos-prescription-solace-during- the prediction of how a game will be experienced
coronavirus-pandemic/2932976001/ (2020, March 28)
Stuart, K.: Among Us: the ultimate party game of the by players is quite difficult without seeing it in
paranoid Covid era. The Guardian. https://www. action. To reduce the risk of not being fun – mean-
theguardian.com/games/2020/sep/29/among-us-the- ing failure – video games are in need of being
ultimate-party-game-of-the-covid-era (2020, tested from the earliest possible step on. As an
September 29)
SullyGnome: Among us – twitch statistics, channels & answer to this need, Salen and Zimmerman sug-
viewers. SullyGnome.com. https://sullygnome.com/ gest an iterative approach to game design, a
game/among_Us (2020, October 4) “cyclic process that alternates between pro-
totyping, playtesting, evaluation, and refinement”
(Salen and Zimmerman 2004). This methodology
gives designers the possibility to evaluate and
Analog Prototyping for Digital adjust their design at each new iteration. The
Game Design process may start with low-fidelity analog proto-
types right after the initial conceptualization and
Tonguc Ibrahim Sezen end with high-fidelity digital prototypes, which
Faculty of Communication and Environment, may even become the final product after refine-
Rhine-Waal University of Applied Sciences, ment and polish. This article focuses on the analog
Kamp-Lintfort, Germany prototyping phase of this process. Why and how
do digital game designers use analog prototypes?

Synonyms
Why Analog Prototyping?
Iterative game design; Paper prototyping;
Physical prototyping; Prototyping In video game design analog prototypes are usu-
ally built to test singled out game aspects which
can be implemented without the aid of compu-
Definition tation and are not expected to reflect other fea-
tures of the project. A successful analog
An analog game prototype is a nondigital prelim- prototype is built quickly and provides “enough
inary playable object built to test various aspects of an experience for someone to grasp the game
of a video game. Analog prototypes are especially [or tested components of it] and give feedback”
beneficial in testing the functionality and percep- (Fullerton 2014). Media-independent game
tion of core game ideas and mechanics in the early mechanics may be tested using paper prototypes
phases of preproduction. Analog prototypes may resembling card or board games (Rollings and
take various shapes such as board games, toys, Morris 2004), toy prototypes may focus on the
and street games. playfulness of core mechanics (Gray et al. 2005;
Macklin and Sharp 2016), and physical proto-
types which are played like traditional street
Introduction games may offer unique insides to the intended
game experience (Adams and Dormans 2012;
Unlike other software, functioning as intended Waern and Back 2017). Not every aspect of a
and being user friendly are not enough for video video game can be tested through analog
Analog Prototyping for Digital Game Design 103

prototyping, but certain aspects of every game forces designers to define game aspects, helps
can be tested by it. them understand the workings and perception of
the game system, and gives them the opportunity A
to change any rule easily if they do not function as
Analog Prototyping Process intended. Schell warns against the trap of falling
into the temptation of overbuilding a prototype. In
Analog prototypes can be created using a wide his words, a prototype “should be quick and dirty”
range on objects. Some more or less standardized (Schell 2014). They are test subjects created to be
components for paper prototyping are meeples, thrown away. According to Ham and Fullerton,
tokens, index cards, tile cards, different types of compared to digital prototypes, analog prototypes
dice, and of course pen and paper. Digital tools for are much easier for game designers to scrap,
creating printable graph paper or exporting data mainly because they require much less time and
from spreadsheets into preexisting card templates effort to build (Fullerton 2014; Ham 2015).
can be used to quickly generate prototypes using Some aspects of video games, such as game
data from early game documents. The tools for economy and resource management mechanics
creating toy and physical prototypes are only lim- (Adams and Dormans 2012; Moore 2011), puz-
ited by the goals and imagination of the designers. zles (Ferrara 2012; Moore 2011), and macro- and
Focusing on the iterative design of the overall microspatial gameplay (Totten 2014) are consid-
gameplay experience, Fullerton proposes a four- ered more suitable to be tested through analog
step analog prototyping process which can be prototypes. They also provide a platform to bal-
used in video game design (Fullerton 2014): ance statistics tough experimentation and to iden-
The first step following the initial conceptuali- tify and close possible player exploits (Trefay
zation is the “foundation” where the goal is the 2010). Other aspects such as sensory experiences,
definition and design of basic game objects and mechanics involving continuous space and time,
the key procedures, or the core gameplay. The and game physics on the other hand are much
second step is “structure” where the designer harder to test trough them. Ham proposes a series
starts building the framework of the game. By of methods, such as creation of flowcharts or state
defining the essential rules and their structural charts to simulate AI and the use of simple heu-
roles in supporting other features, the designer ristics to test motion, to translate video game
builds an unfinished but functional game sys- mechanics into analog game mechanics (Ham
tem. In the next iteration step, “formal details,” 2015). Yet he also warns designers to question
new rules and procedures are added to the sys- the usefulness of analog prototypes if such trans-
tem to reach a fully functional game. The last lations are required. Digital prototyping may be
step is “refinement” where the designers start to the most efficient way of exploring aspects requir-
fill the details of the rough but playable system. ing such translations.
After several iterations and answering key ques-
tions regarding the playability, designers can
begin implementing their solutions in digital Conclusion
format.
Analog prototyping is especially beneficial in
video game design education and in experimental
Strengths and Limits of Analog game design. It is an easy, quick, and cheap way
Prototyping of focusing on and experimenting with game ideas
without being distracted with complexities of the
Analog prototyping is a fast and inexpensive way medium of choice. Despite its limits, its flexibility
of testing game ideas by turning them into tangi- makes it applicable for the testing of a wide range
ble playable objects. Building an analog prototype of features of various types of games.
104 Animacy

Cross-References
Animal Crossing:
▶ Collaborative Engineering and Virtual Pro- A Causal Game
totyping within Virtual Reality
▶ Narrative Design Taeya Johnson2 and Newton Lee1,2
▶ Paper Prototyping 1
Institute for Education, Research, and
▶ Prototyping Scholarships, Los Angeles, CA, USA
▶ Psychological Game Design 2
Vincennes University, Vincennes, IN, USA
▶ Skull and Roses Card Game

Synonyms
References
Causal game; Real-time game
Adams, D., Dormans, J.: Game Mechanics: Advanced
Game Design. New Riders, Berkeley (2012)
Ferrara, J.: Playful Design: Creating Game Experiences in
Everyday Interfaces. Rosenfeld Media, Brooklyn Definitions
(2012)
Fullerton, T.: Gamedesign Workshop: A Playcentric Causal game ¼ A game that is designed to be
Approach to Creating Innovative Games, 3rd edn.
CRC Press, Boca Raton (2014)
played occasionally for a relatively short period
Gray, K., Gabler, K., Shodhan, S., Kucic, M.: How to of time without losing points or competitive
Prototype a Game in Under 7 Days. Gamasutra: The advantages
Art & Business of Making Games. https://www. Real-time game ¼ A game that uses real-world
gamasutra.com/view/feature/130848/how_to_proto
type_a_game_in_under_7_.php (2005). Accessed
time instead of virtual in-game time
27 Dec 2017
Ham, A.: Tabletop Game Design for Video Game
Designers. CRC Press, Boca Raton (2015) Introduction
Macklin, C., Sharp, J.: Games, Design and Play:
A Detailed Approach to Iterative Game Design.
Addison Wesley, Boston (2016) During the COVID lockdown around the world,
Moore, M.: Basics of Game Design. CRC Press, Boca many people had resorted to video gaming as an
Raton (2011) escape into a new world full of many possibilities.
Rollings, A., Morris, D.: Game Architecture and
Design: A New Edition. New Riders, Indianapolis
One of the most popular games was Animal
(2004) Crossing: New Horizons. This entry discusses
Salen, K., Zimmerman, E.: Rules of Play: Game Design the brief history of Animal Crossing and how
Fundamentals. The MIT Press, Cambridge (2004) Animal Crossing: New Horizons became hugely
Schell, J.: The Art of Game Design: A Book of Lenses,
2nd edn. CRC Press, Boca Raton (2014)
popular in 2020 during the pandemic.
Totten, C.W.: An Architectural Approach to Level Design.
CRC Press, Boca Raton (2014)
Trefay, G.: Casual Game Design: Designing Play for the History
Gamer in All of Us. Elsevier, Burlington (2010)
Waern, A., Back, J.: Experimental game design. In:
Lankoski, P., Holopainen, J. (eds.) Game Design Like Nintendo itself, the famous game Animal
Research: An Introduction to Theory & Practice, Crossing started in Japan with a man named
pp. 157–169. ETC Press, Pittsburgh (2017) Katsuya Egushi. In 1986, Egushi was able to
obtain a job at Nintendo located in Kyoto, where
he worked on many games and was known as a
level designer for the well-known game Super
Animacy Mario Bros. 3. Although he was making a name
for himself at Nintendo, he was still homesick due
▶ Uncanny Valley in Virtual Reality to the relocation for his job, which led to the
Animal Crossing: A Causal Game 105

creation of the first Animal Crossing being pandemic. While the world was in a lockdown
released in 2001 (Nintendo Life 2020). and people were stuck in the house, the new
In numerous interviews, Egushi stated that the release of Animal Crossing offered people a A
game was a way for him to be able to recreate the chance to escape reality and enter a relaxing vir-
feeling of being with family and friends: “Animal tual world. Although people were trapped in their
Crossing features three themes: family, friend- homes and seeing friends was difficult, the game
ship, and community. But the reason I wanted to allowed the players to experience being with
investigate them was a result of being so lonely friends and doing activities through a console.
when I arrived in Kyoto. . .When I moved there The game became so popular that it sold more
I left my family and friends... In doing so, than 13 million copies within the first six weeks
I realized that being close to them – being able after its release (Huddleston Jr. 2020).
to spend time with them, talk to them, play with Animal Crossing is known to be a relaxing
them – was such a great and important thing. game. Not only can you interact with friends
I wondered for a long time if there would be a online, but players are also able to customize
way to recreate that feeling, and that was the their characters and have endless tasks to help
impetus behind the original Animal Crossing” build their community. When the game is
(Newton 2011). launched for the first time, you see an empty
As many people around the world play Animal island with a character and a tent. Throughout
Crossing, there is a connection with being able to the game, the character works on many tasks,
play with friends and create a virtual world within such as building new tools and collecting certain
it. While Animal Crossing is a relaxing game, items, to slowly build the island. As the game
there is more to offer while playing the game progresses, new tasks are required, and the
with many different characters and the opportu- players can travel to new islands to collect new
nity to play online. materials, play with friends on their islands, and
creating vacation homes for visitors. Figure 1
shows how the island looks when the game
Gameplay starts. Figure 2 shows the progression and after-
math of designing a client’s dream vacation
Animal Crossing has been around since the early home.
2000s, but the new series of the game became Within the game, there are concepts that were
hugely popular years later during the COVID put in place to help the players.

Animal Crossing: A Causal Game, Fig. 1 The island when the game starts
106 Animal Crossing: A Causal Game

Animal Crossing: A Causal Game, Fig. 2 A dream vacation home

The first thing many people notice is that the pandemic, the game continued to gain popularity
game has a built-in clock that is accurate to the as more people enjoyed the gameplay and used
time in the real world. Not only does this help Animal Crossing as an escape or a comfort game.
players not lose track of time, it also helps move
the game along similar to the real world. For
instance, if something is being built or mailed Conclusion
in the game, the players would have to wait a
whole day until that task is complete. Along with In summary, the history of Animal Crossing
the clock, the weather and region are also similar started when a game developer left home and
to where players are located in the real world. became homesick. The developer was able to
This gives players a sense of reality within turn his feelings of missing his friends and family
the game. into one of the top-selling games. The global
pandemic further boosted its popularity and
cemented its legacy in video gaming (Carpenter
Reviews 2021; Claiborn 2020).

Animal Crossing: New Horizons was first intro-

duced in 2020. With the pandemic trapping every- Cross-References
one indoor, the game prospered in numbers and
sales. Many people loved the game right from the ▶ Video Games
beginning with all the new things and creations
within the game. Some said that after playing it for
a while, they ran into a loop which consisted of References
running around the island, digging for fossils, and
fishing on occasions. These chores seemed bor- Carpenter, N. Animal crossing: New horizons: The final
review. Polygon (2021, November 15). https://www.
ing, but it did not last long. With each new update,
polygon.com/22783019/animal-crossing-new-
new tasks kept players entertained. After the horizons-final-review.
Animal Crossing: New Horizons and Its Popularity During COVID-19 Pandemic 107

Claiborn, S.: Animal crossing: New horizons review. IGN and the game can be explored differently for all
(2020, March 16). https://www.ign.com/articles/ players. Games that do not fall into this category
animal-crossing-new-horizons-review-for-switch.
Huddleston Jr, T.: How ‘animal crossing’ and the corona- typically restrict the player to a strict linear pro- A
virus pandemic made the nintendo switch fly off gression during a playthrough.
shelves. CNBC (2020, June 2). https://www.cnbc. Open-world game: Open-world games have a
com/2020/06/02/nintendo-switch-animal-crossing- large, complex, and interesting environments for
and-coronavirus-led-t o-record-sales.html.
Newton, J.: Celebrating 10 years of animal crossing. the players to explore and travel throughout.
Nintendo Life (2011, December 14). https://www. These games typically do not restrict player
nintendolife.com/news/2011/12/feature_celebrating_ movement to a linear progression or path and
10_years_of_animal_crossing. allow the player to explore the openness of the
Staff, Nintendo Life. Animal crossing: A brief history. Nintendo
Life (2020, March 11). https://www.nintendolife.com/ world at their whim.
news/2020/03/feature_animal_crossing_a_brief_history.

Introduction
Animal Crossing: New
Horizons and Its Popularity Animal Crossing: New Horizons was developed
During COVID-19 Pandemic and published by Nintendo, and it was released in
March 2020. The game became extremely popu-
Tristan Michael Simmons1 and Sercan Şengün1,2
1 lar due to its long-awaited arrival, but it was also
Wonsook Kim School of Art, Illinois State the beginning of the COVID-19 pandemic issues
University, Normal, IL, USA which forced many people indoors. Many players
2
Massachusetts Institute of Technology CSAIL,
found solstice in the simple pleasures of this
Cambridge, MA, USA game, and it became one of the largest releases
in recent years (Bogost 2020). The game was
released solely for the Nintendo Switch gaming
Synonyms console. Animal Crossing: New Horizons is the
fifth installment in the Animal Crossing series.
Life simulation game; Nonlinearity; Open-world
The other games in the main series and the
game; Social game spinoffs are:

Definitions • Animal Crossing (2001, released for

Nintendo 64)
Social game: A social game (sometimes referred • Animal Crossing: Wild World (2005, released
to as social simulation games) is a game in which for Nintendo DS)
one of the core themes revolves around • Animal Crossing: City Folk (2008, released for
maintaining and having meaningful social inter- Nintendo Wii)
action with both NPCs (nonplayable characters) • Animal Crossing: New Leaf (2012, released
and other players if the game is multiplayer. for Nintendo 3DS)
Life simulation game: Life simulation games • Animal Crossing: Happy Home Designer
usually have a focus on gameplay that allows (2015, spinoff, released for Nintendo 3DS)
players to have meaningful and complex interac- • Animal Crossing: Amiibo Festival (2015,
tions with the world around them. This interaction spinoff, released for Nintendo Wii U)
with the game’s world usually addresses common • Animal Crossing: Pocket Camp (2017, spinoff,
life themes, such as farming, fishing, NPC social released for iOS and Android)
interaction, resource gathering, money-making,
and many other themes found in day-to-day life. The game has an ESRB rating of E (“Animal
Nonlinearity: Nonlinearity in games results in Crossing” n.d.) as it appeals to almost all audi-
said games not having a set path for the players, ences and contains no adult themes. The Animal
108 Animal Crossing: New Horizons and Its Popularity During COVID-19 Pandemic

Crossing series is known for its complex world Animal Crossing: New Horizons and Its Popularity
building through its life simulation gameplay During COVID-19 Pandemic, Table 1 Activities in
the game
(Kim 2014); however, Animal Crossing: New
Horizons builds upon this by adding an interactive Activity Detail
and widely accessible multiplayer aspect. The Collecting Fossils, fishing, catching bug and
insects, foraging plants, currency, and
core mechanic of the series is that the game time recipes
runs on real time which adds a more immersive Building Town buildings, changing island
layer of complexity to the series. topology, planting plants, and
designing home
Crafting Tools, furniture, clothing, and
accessories
Gameplay
Character Collecting and changing clothes, hair
customization styles, and accessories
The gameplay for Animal Crossing: New Hori- Travel Visiting vacation islands and visiting
zons is very simple and complex at the same time. the islands of other players
The pace of the game is very relaxed and slow NPCs Collecting/Inviting NPCs, gifting,
which allows for many things to be completed in trading, and building relationships
one session while not feeling too rushed. There is
a vast pool of activities that the player must
choose from. The activities allow diverse NPCs in their game have, and it allows for a more
gameplay that the player can decide upon. The diverse and customizable world-building
open-world aspect of the game also allows the experience.
players to complete most tasks or activities at
any point which prevents the game from feeling
too restrictive or linear. The activities and tasks Reception and Effects of COVID-19
available to the player range from collecting fos-
sils, fishing, catching and collecting rare insects, The reception of Animal Crossing: New Horizons
building relationships with player NPCs, forag- was overwhelmingly positive due to both the
ing, collecting resources, earning money, and quality of the game and the timeframe in which
much more (see Table 1). The main “questline” it was released. The game debuted on the market
of the game is progressively paying off loans to on March 20, 2020, and by the end of the month,
increase the size and storage space of your home. Nintendo had sold over 11 million copies. In
Relative to other life simulation games on the North America, the game became the best-selling
market, Animal Crossing: New Horizons uses title of March 2020 and the second-best-selling
common themes and activities for the player that title of 2020 as a whole (Grubb 2020). As of June
resonate throughout many other games in its 30, 2020, the title has sold more than 22 million
genre. However, the unique spin that makes the copies, making it the second-best-selling game on
game a one of a kind is its diverse and complex the Nintendo Switch system (Byford 2020).
villager NPC system. As of October 2020, there is Other than the quality of the game, the timing
a total pool of 391 different and unique villagers in in which it was released played a significant
the game. Villagers are sorted into different per- factor toward its success. The game was released
sonality categories that affect their dialogue and during the start of the worldwide COVID-19
interactions with both the player and other NPCs. pandemic. Many communities and players
Villagers will slowly come to your island, and throughout the globe started quarantining, and the
through the discovery of vacationing villagers on game provided an outlet for escape and relaxation
other islands, you also have the ability to ask them that many individuals desperately needed during
to join your island. This complex system allows these times (Buchanan 2020). Although some pre-
for the player to decide what character traits the vious versions of the game were offered to have
Animation and Neurocinematics: Visible Language of E-motion-S and Its Magical Science 109

mechanics linked to addiction (Scully-Blaker 2019 Leporati, G.: Inside Academia’s Growing Interest in ‘Ani-
on Animal Crossing: Pocket Camp), this specific mal Crossing’. The Washington Post. (2020, July 14).
https://www.washingtonpost.com/video-games/2020/
version of the game offered players “temporary 07/14/inside-academias-growing-interest-animal- A
escape” from the risks of the pandemic, as well as crossing/
a way to “get rid of loneliness” (Zhu 2020). Many Scully-Blaker, R.: Buying time: capitalist temporalities in
individuals found interesting ways to use the game animal crossing: pocket camp. Loading. 12(20),
90–106 (2019). https://doi.org/10.7202/1065899ar
for events such as business meetings, weddings, Zhu, L.: The psychology behind video games during
birthdays, and much more. The widespread popu- COVID-19 pandemic: a case study of animal crossing:
larity of the game resulted in academic interest in new horizons. Human Behav Emerg Technol,
its reception based on social sciences and human- 1–3 (2020). https://doi.org/10.1002/hbe2.221
ities research (Leporati 2020). These endless pos-
sibilities and calming atmosphere for escape
allowed for the game to become one of the most
popular and impactful games of its time. Animal-Computer Interface

▶ Engaging Dogs with Computer Screens: Ani-

mal-Computer Interaction
Cross-References

▶ Among Us and Its Popularity During COVID-

19 Pandemic
▶ Social-, Mobile-, and Multi-Player-Games and Animation
Their Impact on Today’s Online Entertainment
Industry ▶ Animation and Neurocinematics: Visible Lan-
guage of E-motion-S and Its Magical Science
▶ Preserving the Collective Memory and Re-cre-
References ating Identity Through Animation
▶ Super Mario Galaxy: An Overview
Animal Crossing: New Horizons. ESBR.org. (n.d.). https://
www.esrb.org/ratings/36764/Animal+Crossing%
3A+New+Horizons/
Bogost, I.: The Quiet Revolution of Animal Crossing. The
Atlantic. (2020, April 15). https://www.theatlantic. Animation
com/family/archive/2020/04/animal-crossing-isnt-
escapist-its-political/610012/
and Neurocinematics: Visible
Buchanan, K.: Animal Crossing Is the Perfect Way to Language of E-motion-S
Spend Quarantine. New York Times. (2020, March and Its Magical Science
31). https://www.nytimes.com/2020/03/31/arts/
animal-crossing-virus.html
Byford, S.: Animal Crossing Catapults Nintendo to Strato-
Inma Carpe
spheric Earnings. The Verge. (2020, August 6). https:// The Animation Workshop, VIA University
www.theverge.com/2020/8/6/21356750/nintendo- College, Viborg, Denmark
earnings-q1-2020-switch-animal-crossing-sales Polytechnic University of Valencia, Valencia,
Grubb, J.: March 2020 NPD: Animal Crossing Powers
March to Blockbuster Game Sales. Venture Beat.
Spain
(2020, April 21). https://venturebeat.com/2020/04/21/
march-2020-npd-animal-crossing-powers-march-to-
blockbuster-game-sales/ Synonyms
Kim, J.: Interactivity, user-generated content and video
game: an ethnographic study of animal crossing: wild
world. Continuum. 28(3), 357–370 (2014). https://doi. Animation; Emotional Intelligence; Neurosci-
org/10.1080/10304312.2014.893984 ence; Social emotional learning; Visual literacy
110 Animation and Neurocinematics: Visible Language of E-motion-S and Its Magical Science

Definitions we can explore it through animation. Studies from

Talma Hendler, Zack Jeffrey, or Uri Hasson evi-
Animation: is the emotional energy in motion of a dence of how watching movies activate specific
visual thinking. The illusion of life by making areas in our brain related to different emotions.
sequential drawings (or 3D representations) of a This new neuroscience of film making is known
continuing action and projecting their photo- as Neurocinematics (Hasson 2008). There are no
graphs onto the screen at a constant rate. specific studies regarding to the positive effects of
Neurocinematics: refers to the neuroscience of the creative process of animation, especially
film, term coined by Uri Hasson from Princeton focusing on how animation and emotions are
University, who studies the effects on the viewer’s connected during the art production.
brain activity when watching a given film. Labs such as Lotto Lab in the United Kingdom,
the Animated Learning Lab in Denmark, or the
Film Board of Canada have been working on this
Introduction issue by developing new paradigms of teaching
connected to sciences and film production. They
Why does it matter what we know about emotions include into their teaching human values such as
when making movies? Can an animation help us mindfulness, compassion, resilience, and emo-
to understand them better? Even if we naturally tional intelligence. We consider animation as a
learn by telling stories, people may forget words social emotional learning tool; animation is the
and events, but they will never forget how some- emotional energy in movement that provides the
thing makes them feel. Movies touch our hearts in illusion of life, a soul, and a fantasy world as real.
the same way they touch our minds. It is an artistic thinking-feeling media, which pro-
Whether we hope to spot concealed emotions or vides great opportunities to experiment, by
seek compassionate connection, our ability to see playing, with different perspectives, creativity
and respond to others’ often unspoken feelings is and innovation, and new worlds.
central. This ability can be trained. We provide the Before going into the current fascination with
tools. Paul Ekman
visual effects and the most advance technology in
The use of animation as a creative media to movies, we should recall how the art of film
enhance communication implies to study relation- making started with silent feature films such as
ships, how we connect with each other and how from Méliès, A trip to the Moon (1902) or
our brains make connections with the information Chaplin’s The Kid (1921). In these films as many
that collects. Neuroscientists acknowledge that others of that time, the main tools that directors
we humans need to make sense of our reality, for could count on, to show the audience an idea or
what we make relationships depending on our feeling, were the movement, action, and music
perception (Beau Lotto 2015). The dangerous before dialog appeared. Animation happened
and magic point of this fact is the brain does not even before those movies were produced, in the
distinguish between the imaginative perception Upper Paleolithic caves of Altamira or Chauvet
from the real perception. If our well-being (France, ca. 30000 BC). Those were the first
depends on how we see, perceive the inner/out attempts to express the idea or feeling of how an
world, we need to experiment and study how we animal look like, furthermore, in movement.
make stories in order to deconstruct them and get Some anthropologists and psychologists of
to observe from different angles, not just one neuroaesthetics as Marcos Nadal (University of
reality but may be many others. Vienna 2014) believed that those representations
Emotional regulation is extremely relevant were probably like plays where they could
since influences our decision-making and rehearse a situation like hunting. Those cavemen
problem-solving skills. Our well-being or happi- were learning through visual storytelling, most
ness depends on this mysterious ancient world of likely making their own sounds and pantomime,
emotions connected to our way of thinking, and before any language existed. They made
Animation and Neurocinematics: Visible Language of E-motion-S and Its Magical Science 111

A
Animation and Neurocinematics: Visible Language of E-motion-S and Its Magical Science, Fig. 1 Film
animated strip

Animation
and Neurocinematics:
Visible Language
of E-motion-S and Its
Magical Science,
Fig. 2 Textile art for a
character design.
Re-construction of thyself

associations between what they saw and the draw- simplest elements of visual composition, the
ings on the walls with a meaning. pure expression of feelings in movement.
Animation can be as abstract as in its origins In the following lines, we will briefly present
and go beyond the limits of the physicality of live the relationship between animation, neuroscience,
action movies, for which we usually see a hybrid and emotions, which we use during film produc-
of productions that need animation to recreate the tions at different levels. We obtained very positive
impossible scenarios that we imagine in our results that motivate us to share and ask scientists
brains, such as Avatar (James Cameron 2009) to keep working with artists, like Paul Ekman who
or possible recreations from the past, such as has explored with his studies the universal signs of
Jurassic Park (Steven Spielberg 1993). Directors emotions and facial expressions in different cul-
like Ray Harryhausen (United States) and tures. Antonio Damasio pointed out that scientists
Jan Švankmajer (Czeck Republic) were record life as it is; yet, artists express it as it may or
experimenting live action with animation before may not be. Artists can be the scientists of pre-
big companies like Disney produced the well- conceiving life.
known Mary Poppins (1964), Who Framed “The greatest scientists are artists as well” Einstein.
Roger Rabbit (1998), and The Three Caballeros (Calaprice 2000, 245)
(1945). Animation acts as the bridge between
This is our vision, how we can use the knowl-
reality and fantasy, and the imaginary perception
edge of producing movies to change our percep-
and the real perception. It makes us believe the
tion, to learn about life understanding our
most abstract forms or realistic dreams as real life,
emotions, so the relationships that we have with
thanks to the emotions which connect us. The
the self and the external world (Fig. 2). Rewire our
short movie The Dot and the Line: A Romance
brain with a tool that helps to rewrite our story to
in Lower Mathematics (Chuck Jones 1965) is a
very harmonic story where we see through the become fully alive and make sense of our lives.
112 Animation and Neurocinematics: Visible Language of E-motion-S and Its Magical Science

Methodology: Working Beliefs-Feelings- The most recent Pixar movie, Inside Out
Actions Through Animated Productions (2015), is an excellent sample of the importance
to understand the relationship between emotions
and thoughts with behavior. Furthermore, some
schools are using the movie to talk about and
identify the emotions. By watching this movie,
we learn to identify four from the primary six
emotions that Antonio Damasio classified in his
research (Damasio 1999). We get to know why
and how the characters behave, what is inside our
heads and what kind of feelings and actions
emerge when one emotion is in control. We under-
stand the importance of accepting and balancing
In this section, we share our observations and
negative and positive emotions because they need
work method during the creative process of mak-
each other. The same thing applies to the question
ing an animated film.
of being more rational or emotional; both go hand-
We collected data from the Animated Learn-
in-hand and work together as we can see in Rea-
ing Lab in collaboration with educational insti-
son and emotion (Disney 1943). Some great films
tutions from different countries, such as San
as Party Cloudy (Pixar 2009) explore feelings and
Carles Fine Arts in Valencia, Spain; indepen-
ideas, friendship and resilience, with a clear
dent artists such as George Mcbean (UNICEF)
reflection by using images over words. Luxo Jr.
and creative professionals, who have been
(Pixar 1986) was a revolutionary experiment
working on tailoring animated workshops for
using procedural animation, where John Lasseter
students of different ages, from toddlers to post-
applied the classical principles of animation to 3D
graduate students. This makes a richer experi-
characters in order to provoke emotions. Most
ence at the time to exchange and contrast ideas,
recently, in the independent film-making arena,
which shows us interesting ways in which ani-
we find a movie which has a program for teachers
mation is a very powerful tool for building
to share wisdom about life, The Prophet (Salma
social relationships and increasing creativity.
Hayek 2015).
To answer why we connect animation, emo-
The field of neuroplasticity explains how our
tions, and neuroscience, we will start explaining
brain is capable of change with every experience
the relationship between our brain and
and by repetition, creating new synapses and pat-
emotions.
terns that can determine new behaviors (Davidson
Scientists such as Richard Davidson
2008). Animation is all about repetition and focus;
(University of Wisconsin) Daniel Siegel
it is a transformative process where we work
(Mindsight Institute, California) or Joe Dispenza
connecting our minds to our bodies. During any
(DC) have been studying neuroplasticity and
creative process, the energy flows where our
researching the effects of meditation and emo-
attention is focused (Mihaly Csikszentmihalyi
tional regulation for an optimal learning and
2008); some people are more kinesthetic, others
well-being. As we mentioned before, emotions
are attracted to sound or are more verbal; these are
affect our decision-making, if we learn how to
some of the multiple intelligences that Howard
identify the emotions and regulate them, we will
Gardner (Harvard 2006) acknowledged, all can
be able to develop resilience and increase our
be explored during the creation of an animated
sense of fulfillment and contentment. We found
movie as if it were a mindfulness practice.
that animation can be an excellent media to learn
Animation can be a practice of mindfulness,
and regulate our emotions, understand different
since animators need to observe outside of them-
perspectives, and be more conscious about our
selves, in order to be able to understand the char-
feelings and beliefs.
acter that we must animate, or have enough
Animation and Neurocinematics: Visible Language of E-motion-S and Its Magical Science 113

information to be able to design a new original under 9–10 years old. Ateliers are taught in a
character, environment which has to be believable constructivism learning method (Vygotsky), to
and engaging to the audience. This engagement animate in a very intuitive way, by playing with A
happens because our empathy and mirror neurons the software and receiving small lessons and sup-
activate when a speaker and listener interact port from the tutors. We focus on four main ani-
(Hasson 2010) (Fig. 3). Despite very subtle differ- mation techniques: pixilation, cut out, clay, and
ences between cultures, is our basic human essence 2D, to keep them in touch with analog materials
to connect through empathy; Paul Ekman and avoid working just with computers. We
(Emotions Revealed 2012) has been working for encouraged the importance of using kinesthetic
major animation studios due to his relevant studies techniques when possible, since it helps to focus.
about facial expressions, emotions, and deception. The rewarding system of the brain activates when
Animators have been using his knowledge to better students see a physical and visual product after
understand the nuances of expressing feelings their learning experience.
within different contexts and situations. This is Animators develop the four main components
relevant to exaggeration and having gestures to that Daniel Goleman acknowledges in his defini-
entertain the audience, as the story unwraps. Our tion of emotional intelligence (2005): self-
understanding about the story itself is a reason for awareness, social awareness, self-management,
case study; to question beliefs, decide which emo- and relationship management.
tions intervene and what actions are going to hap- Naturally our brain is plastic and shapes itself
pen to resolve the conflict. Life is presented as a by experiences; it is always transforming and cre-
conflict to resolve, with a touch of imagination. ating new synapses, even as we get older. When
Walt Disney Studios used to test future animators we work on making movies, we put ourselves as
with an assignment where they had to express directors or animators in hypothetical situations
different feelings and emotions using a sack of that, either, are real memories or fantasies. In
flour. A clear sample of how we can learn about either case, they are an excuse to experiment in a
emotions from the inside out is by doing, as we do safe context, situations which we could be
from outside in, by watching movies. involved in, provoking: reactions and, inducing
To work on a production, we set up groups of feelings and ideas that we can question by
three or four students, depending on their age and reflecting, especially working in groups where
interests. Each group had to discuss an idea and different perspectives are factored in.
make a film out of it. Different roles were distrib- The creators have to think and feel the way
uted amongst themselves, if they are not children their characters must behave. During this process,

Animation and Neurocinematics: Visible Language of E-motion-S and Its Magical Science, Fig. 3 Illustration of
speaker-listener when telling a story
114 Animation and Neurocinematics: Visible Language of E-motion-S and Its Magical Science

they are not just passive observers but active this is a whole new area of study, the cognitive
protagonists. As a result, the learning experience neuroscience of film (Jeffrey Zack 2014).
is stronger and their effects are more intense
regarding comprehension to why a character acts Life is a tragedy when seen in close-up, but a
comedy in long-shot. Chaplin (1889–1977)
in a certain manner, and how they should express
ideas and feelings in accordance with that behav- In making movies, we must be aware of the
ior. The results in comprehension of emotional meaning and function that every artistic compo-
status are higher than when watching movies. nent adds to the film. In animation, we work with
While watching a film, each viewer makes a lec- what big studios call color script, which shows the
ture of the scene based on his perception (from film’s colors and lights visualized in key frames
his/her background). In other words, they bring from the primary scenes. Nowadays, we can find
their vision from what they personally have lived; the whole movie compressed as a barcode, pro-
by doing animation, they become the character, viding the whole spectrum in one image. Anima-
forcing them to get his mindset with all its traits. tors learned to evoke emotions by using different
Feelings are no longer a personal interpretation technical elements of composition such as color,
but a rehearsal for being the character, even warm for love and positive feelings, more blue or
though there may be certain level of subjectivity darker for sad ones: in Beauty and the Beast
while acting. (Disney 1992) the castle changes from very dark
Animation acts as a metaphor to transport the bluish colors during the spell, to bright and warm
ateliers to live other people lives, through their towards the end of the film. Round shapes are
minds and bodies. We usually hear“ You don’t more suitable for children; they are soft and calm
understand because it never happened to you” as we see them in Lilo ad Stich (Disney 2002),
and somehow this is true since the process of while more angular shapes convey a cold and
embodying feelings and ideas is always more aggressive feeling. More light is associated with
real when one has lived a similar personal experi- happy and relaxed situations, such as in Tangle
ence. The reason is there is a trace of that past (Disney 2010) where everything is inspired by the
event through all the senses, instead of being just a painting of Fragonard; the main colors are pink
hypothetical thought or imagination. In this mat- and soft with an especial glow. Music is extremely
ter, the creative process is more important than the relevant as well as many other elements, such as
quality of the final result, because it makes you rhythm. Animators and directors start to be more
gain knowledge by experiencing instead by just aware of the psychological and symbolic meaning
listening or viewing (Fig. 3). of these components as they work on productions.
At the end of the production students, learn Even camera movement can create and enhance
technical aspects related to animation and film different moods.
making than can be applied to reflect on real life. To summarize, life in animation is a safe game
Movies are audiovisuals expressions from reflec- where we play somebody else; we are free and
tions about life, a work in progress; and we create focus on what we do. Being aware of what hap-
our own stories as recreations of past, present, or pens brings a more peaceful status of relaxation to
future events. We become the architects of per- face problems and make decisions. We work on
sonal realities, by editing those moments and put- resilience and the relationships between the world
ting them together to make sense of what we live and the self, as well as the connections between
(Lev Kuleshov 1920). Our brain does not distin- our thoughts, feelings, and actions, in order to
guish what is real or fiction, and perception and reach balance. Animated movies can teach us
cognition are crucial in understanding emotions how to feel like children again and inspire us to
and getting an optimal communication within the become our better selves, even when we are
self and others. We edit and change our realities already grown-ups thanks to neuroplasticity. It is
due to personal perception and the fragmentation a chance to find ourselves in somebody else’s
that occurs in our brains when processing data; eyes, so we can meet others within ourselves by
Animation and Neurocinematics: Visible Language of E-motion-S and Its Magical Science 115

empathy, which is the key to our emotional sys- Conclusions and Discussion
tem. Animation is an excellent art form for self-
awareness and self-development, which we can Working with animation provides the tools to A
use for children and adults. Filmmakers and pro- train our minds and bodies thanks to
fessionals of other visual fields must take bigger neuroplasticity, by applying the emotional intel-
responsibility of their influence in people’s lives, ligence. It can be considered as a medium for
through their movies, especially children. Anima- cognitive therapies.
tion is more than an entertaining media; it is a It is a bridge between sciences and arts.
visual language of emotions and feelings, worth to Animation is an excellent medium to teach
research the sciences of its effects in how we make visual literacy and develop critical minds to
up stories in our minds in order to make meanings avoid manipulation.
and sense of our lives, starting by how we per- Within art therapy, it works as an excellent new
ceive the world. approach with autistic children and any other con-
dition that is an obstacle to communication.
It is a mindfulness media and tool to put it in
Results practice and bring consciousness from the
unconsciousness.
– Animation improves our cognitive functions The creative process of an animated movie
and awareness of being. helps to develop important social skills.
– Students or professionals learn about emotions Animated movies serve as metaphors to com-
and feelings (especially the difference) municate when language barriers are an impedi-
– It enhances social skills such as cooperation, ment. It can be an alternative language to
compassion, tolerance, listening, and linguistics.
teamwork. Animation should be considered a social emo-
– Animation provides a more natural method to tional learning tool to be incorporated in regular
reflect on actions by having fun, without curricula to implement knowledge about
judgment. emotions.
– Communication becomes better within the Animated productions open new ways of
teamwork by sharing and listening. communication, contributing to the creation of
– Expressing feeling through animation encour- happier communities with the necessary tools to
ages students to find their voice when there is obtain an optimal sense of resilience, in order to
some impediment or difficulty, physical or cope with life’s challenges, learning to be
psychological. humans. Animation is a social emotional learn-
– The students raise their self-esteem and feeling ing media, extremely powerful to study deeper
of reward by producing a final product. the cognitive effects in our brains and minds
– Learning skills, as concentration, focus during art production, which can bring us a better
improve considerably after a program is understanding of how we see the world from
complete. different perspectives.
– Students start to develop a personal mindset Animation can be an important storytelling
with a critical view to question audiovisual media to be aware of our thoughts and feelings,
information and beliefs systems. to reflect upon them and understand the stories
– Animation students get greater knowledge and that our brain creates, since apparently its default
comprehension of visual literacy when mode activity is story-making (Mehl-Madrona
watching or producing movies. 2010).
– Creativity increases in all cases, including the
I am convinced that animation really is the ultimate
most introverted animators. art form of our time with endless new territories to
explore. Glen Keane. http://www.awn.com/news/
glen-keane-creates-nephtali-short-paris-opera
116 Animation Scripting

References Moll, C.L.: L.S. Vygotsky and Education (Routledge Key

Ideas in Education), 1st edn. Routledge, New York
Ansorge N.: Animated films in psychiatry. The psychiatric (2014). ISBN 978–0415899499
clinic of the University of Lausanne. Cery Hospital, Morawetz, C., Kirilina, E., Baudewig, J. Heekeren, H.R.:
1962–1981. Animation World Magazine, Issue 3.2. Relationship between personality traits and brain
[online]. http://www.awn.com/mag/issue3.2/3.2pages/ reward responses when playing on a team. Department
3.2ansorgeteachingeng.html (1998) of Education and Psychology, Freie Universita¨t Berlin,
Berlin. PLoS One www.plosone.org
Butterfill, A.S.: Perceiving expressions of emotion: what
Price, A.D.: The Pixar Touch: The Making of a Company,
evidence could bear on questions about perceptual
1st edn. Knopf, New York (2008). 304 p. ISBN
experience of mental states? Elsevier. Consciousness
10:0307265757
and Cognition. [online]. www.elsevier.com/locate/
Ramachandran, V.S.: The Tell-Tale Brain, 1st edn. Wind-
concog (2015)
mill books, London (2012). ISBN 9780099537595
Chin, P.N.: Teaching critical reflection through narrative
Raz, Gal: E-motion pictures of the brain: Recursive paths
storytelling. Michigan J. Comm. Ser. Learn. University
between affective neuroscience and film studies. In
of Rochester Medical Center, US. Summer 2004,
Arthur P. Shimamura (Ed), Psychocinematics: Explor-
pp. 57–63. [online]. http://hdl.handle.net/2027/spo.
ing Cognition at the Movies, Oxford University Press,
3239521.0010.305
pp. 285–336 (2013). Retrieved October 11, 2016, from
Clark, R.: Storytelling & Neuroscience. [online]. Slideshare. research gate, https://www.researchgate.net/publica
Feb 23, 2015. [Online]. http://www.slideshare.net/ tion/270887194_E-Motion_Pictures_of_the_Brain_
theelusivefish/storytelling-andneuroscience-pcto Recursive_Paths_Between_Affective_Neurosci
Coplan, A.: Catching characters’ emotions: emotional con- ence_and_Film_Studies
tagion responses to narrative fiction film. Film Stud Rieber, P.L.: Seriously considering play: designing inter-
8(1), 26–38 (2006) active learning environments based on the blending of
Davidson, R.J.: The Emotional Life of Your Brain, microworlds, simulations, and games. Educational
Budah’s Brain. Plume/Penguin Group, New York Technology Research and Development 44(2), 43–58
(2012). 304 p. ISBN 10: 0452298881 (1996). https://doi.org/10.1007/BF02300540
Gardner, H.: The theory of multiple intelligences: as psy- Siegel, Dan J. Random House 2010. The New Science of
chology, as education, as social science. October Personal Transformation; W.W. Norton 2009, The
22, 2011. Multimedia-and-multiple-intelligences –The Healing Power of Emotion: Affective Neuroscience,
American prospect no. 29, November–December 1996. Development & Clinical Practice; Guilford Press
Art, Mind, and Brain: A Cognitive Approach to 2012, The Developing Mind, Second Edition: How
Creativity Relationships and the Brain Interact to Shape Who
Goleman, D.: The Brain and Emotional Intelligence: New We Are
Insights, 1st edn. More Than Sound LLC, Northampton Smith, M.: Engaging Characters: Fiction, Emotion and the
(2011). 72 p. ASIN: B004WG5ANA Cinema, pp. 73–109. Oxford University Press, Oxford
Grundmann, U.: The intelligence of vision: an interview (1995)
with Rudolf Arnheim. Cabinet Magazine. Issue 2. The Animation Workshop/VIA University College. Ani-
[online]: http://www.cabinetmagazine.org/issues/2/ mated Learning. Digital web. [online] http://www.
rudolfarnheim.php (2001) animatedlearning.dk/
Jhonson, O., Frank, T.: Disney Animation: The Illusion of Zacks, J.: Flicker. Your Brain on Moves, 1st edn. Oxford
life, 3rd edn. Abbeville Press, New York (1988). ISBN University Press (2015). ISBN-10: 0199982872
10: 0896596982 Zaidel, D.W.: Art and brain: insights from neuropsychol-
Joe, D.: Breaking the Habit of Being Yourself: How to ogy, biology and evolution. J. Anat 216, 177–183
Lose Your Mind and Create a New One, 1st edn. Hay (2010)
House, Inc., Carlsbad (2012). ISBN 978-1-4019-
3810-9
Lebell, S., Epictetus by (Author).: Art of Living: The
Classical Manual on Virtue, Happiness, and Effective-
ness, 1st edn. Harper Collins Publishers, New York Animation Scripting
(1995). 144 p. ISBN-10: 0062513222
Miall, S.D.: Emotions and the structuring of narrative
responses. Poetics Today 32(2), 323–348 (2011). ▶ Character Animation Scripting Environment
[online]. http://poeticstoday.dukejournals.org/content/
32/2/323.abstract. Consulted: 4 May 2015
Miller, G.: Cinematic cuts exploit how your brain edits
what you see [online]. Wired Science. Disponible
en internet: http://www.wired.com/2014/09/ Anomaly Detection
cinema-science-film-cuts/ (2014). Consulted:
4 May 2015 ▶ Fall Risk Detection in Computer Vision
Anti-phishing Attacks in Gamification 117

banking users will use their mobile devices to

Anti-cheat Protocols manage their current accounts over 2.3 billion
times – more than the cumulative total of desktop A
▶ Secure Gaming: Cheat-Resistant Protocols and PCs, branch, and telephone banking users (Heinze
Game History Validation et al. 2016). However, less than 33% of mobile
users have installed antivirus software on their
devices when compared to the 91% level seen
for laptop users. One reason may be that 45% of
Anti-phishing Attacks in mobile users do not see cyberattacks on their
Gamification mobile devices as a threat. The rise in the use of
smartphones, their limited security, and the lack of
Yousef Al-Hamar1, Hoshang Kolivand1 and end user knowledge increase the risks of victimi-
Aisha Al-Hamar2 zation. This directly affects the gaming applica-
1
Department of Computer Science, Faculty of tions while using smartphones.
Engineering and Technology, Liverpool John Phishing is one of the social engineering
Moores University, Liverpool (LJMU), UK attacks which is the psychological manipulation
2
Department of Computer Science, of individuals into revealing confidential informa-
Loughborough University, Loughborough, UK tion (Anderson 2008). Phishing is a common form
of identity theft and it is among the highest-ranked
cybercrimes, costing organizations an average of
Synonyms $3.7 million per year in USA (Greenberg 2016).
A study by a British government–backed cyber-
Social engineering attacks, Spear phishing, Phish- security firm found that phishing attacks cost Brit-
ing attacks, Cybersecurity attacks, Phishing and ish users £174.4 million in 2015 (Greenberg
game attacks. 2016). The most common phishing uses emails,
chats, or websites along with attacks game appli-
cations especially among online and network
Definition games to get valuable information. A phishing
attack disguises itself as a harmless request from
Phishing is a social engineering attack which aims a trusted sender that tricks its victim into sharing
to manipulate people and encourage them to personal information. It can be thought of as an
expose their confidential information. There are attack that uses social engineering to target the
many different types of phishing attacks such as gullibility of people. Social engineering is “the
spear phishing, whaling, vishing, and smishing. use of non-technical means to gain unauthorized
access to information or computer systems”
(Thompson 2006). Phishing attacks are in theory
Introduction a well-engineered social attack to extract valuable
personal information from individuals. Phishing
Cybercrime is becoming a widespread problem attacks tend to be challenging to detect by inex-
that is posing an increased risk due to the increas- perienced people. This is due to the fact that they
ing number of devices such as smartphones that pretend to be coming from well-established and
are connected to the Internet. There is an increase trustworthy senders (Thompson 2006). Most peo-
in the usage of smartphones for a variety of appli- ple assume that phishing attacks are aimed to
cations, such as browsing the Internet, gaming, financially damage individuals alone. That may
social networking, online banking, and attending be true; however, their damage is actually far more
to emails. The past three years have seen an deep rooted and long lasting. This is because,
increase in smartphone usage for email applica- individuals who have been harmed have their
tions by 180% (Heinze et al. 2016). Furthermore, trust significantly diminished in electronics and
118 Anti-phishing Attacks in Gamification

business from which the phishing email appears to risk of phishing. The solutions are based on the
have come (Robila and Ragucci 2006). This in latest studies in the field in addition to the best
turn means the reputation of that business will be practices and awareness in the field. E-mail phish-
damaged, causing loss of its loyal customers, ing attacks have evolved from being purely tech-
since those customers tend to associate the busi- nical, consequently highlighting the need for
ness with phishing attacks and ceases to use the defense mechanisms that go beyond purely tech-
services offered by that business (Sullins 2006). nical controls. Thus, security in this application
An investigation by McAfee illustrated that must be viewed using a holistic perspective to
employees working in finance, accounting, and integrate technology with aspects of human
human resources were among the worst in behavior.
detecting phishing attacks and responding appro- A survey of the market reveals that there are
priately to them (Cochin et al. 2014). This then many commercially available solutions tailored
raises the question of how such departments, with toward fighting fishing attacks. Despite their
so much access to sensitive data, should guard widespread availability, tools to detect automated
their organization against phishing attacks. Mul- e-mails have limited capabilities (Fette et al. 2007)
tiple sources from renowned scholars strongly (Zhang et al. 2007). conducted a study on ten anti-
believe that the best way to defend against such phishing tools and found that only one of them
attacks is by using a multilayered technical anti- was able to correctly detect 90% of automated
phishing defense system. Nevertheless, the big- emails in a certain trial. However, a further inves-
gest part of this defense mechanism still remains tigation showed a false classification accuracy of
well educating and training employees of an orga- 42%, implying that it incorrectly classified 42% of
nization to be able to recognize phishing attempts emails. The reason for this difficulty in detecting
and immediately respond by reporting them to the potential phishing attacks is that phishing attacks
members of the IT team (Butler 2007; Hong 2012; are constantly evolving to take more complex
Swapan 2012). forms. As a result of this, it is very difficult for
Due to the severity of cybercrime at personal universal anti-phishing tools to detect and protect
and national levels, many countries have intro- the user from all forms of attacks (Zhang et al.
duced cybercrime laws. In the UK, Sections 2007). Instead, the readily available tools merely
41–44 of the Act, which amends the Computer reduce the risk of phishing attacks without offer-
Misuse Act 1990 (Thompson 2006), stipulates ing a comprehensive and accurate detection of all
that offenders will face tougher penalties for com- possible attacks (Dodge Jr et al. 2007) (Dodge and
mitting cybercrimes intended to cause serious Ferguson 2006) (Downs et al. 2007). classifies the
damage. The US Computer Fraud and Abuse incorrect detection of legitimate and phishing
Act (CFAA) was enacted in 1986 and its penalties emails as being false positive and negative,
can be imprisonment for not more than five years respectively. They also found that anti-phishing
and/or a fine of not more than $250,000 for indi- tools can never give complete protection against
viduals (Robila and Ragucci 2006). either form of false detection of emails. Therefore,
users should make their final decision on the
potential classification of emails using their own
Gamification Solution knowledge and experience rather than solely rely-
ing on the anti-phishing tool. This can prove to be
Phishing has been a complex phenomenon and a challenge as user behavior and knowledge is
therefore it is not possible to single out a solution unpredictable and there is no systematic way to
to avoid it. Therefore, the risk of phishing can be remove the risk associated with a lack of appro-
reduced through the education and training of priate user knowledge on the matter (Dodge Jr
users and by suitable technology. et al. 2007). This shows that complete protection
There are numerous technical and nontechnical is only achievable by having an idea of the behav-
solutions that have been suggested to reduce the ioral response of a user, be it through education or
Anti-phishing Attacks in Gamification 119

knowledge of the best practices. It is thus impor- between a full game and parts, gaming and
tant to understand what exactly makes people playing.
susceptible to phishing attacks to help us develop A way to play a game while having fun, but it is A
all anti-phishing tools, most importantly aware- not a full game. The effectiveness of teaching
ness and education of the issue (Kumaraguru et al. procedural and conceptual knowledge using
2007; Sheng et al. 2007). Further research in the games has been discussed extensively in the liter-
vulnerability of users to phishing attacks is needed ature (Rittle-Johnson and Koedinger 2002; Gee
to develop complete anti-phishing protection 2003).
tools that can offer an end-to-end solution against Researchers have put forward several
a variety of phishing attacks. blacklist/whitelist-based and heuristics-based
Despite the reduction in the potential of phish- defense mechanisms to protect against phishing.
ing attacks offered by technological solutions, Entities such as PhishTank (PhishTank) and Anti-
user-level protection still proves to be the most Phishing Working Group (APWG) have compiled
effective form owing to their high vulnerability reports on authentication services and phishing.
due to a lack of knowledge in the field (Downs Over the years, many tools have been developed
et al. 2007). This claim is proved by many which are designed to safeguard against the most
researchers in the field who conclude that there common phishing attacks: browsing services like
is a definite need to improve user awareness on the Microsoft SmartScreen Filter, Norton Safe Web,
matter to increase the chances of success of fight- McAfee SiteAdvisor, and Google Safe Browsing
ing against the attacks (Downs et al. 2007) are just some of the tools developed to this end.
(Kumaraguru et al. 2007) (Sheng et al. 2007). Nonetheless, phishing practices have evolved in
A study conducted by (Forte 2009) finds that sophistication in tandem with defense systems,
relying solely on technological barriers against often staying one step ahead in the game of
phishing attacks offers limited protection as the avoiding notice and bypassing safeguards (Yue
attacks are typically designed to work directly on and Wang 2010). The struggle between phishers
users and exploit their lack of knowledge. This and anti-phishers is an enduring one.
shows that it is of utmost importance to educate
users on the importance of verifying the legiti- Anti-Phishing Phil
macy of phone calls and emails that they receive. Sheng et al. (Sheng et al. 2007) presented the
Phishing awareness has been discussed exten- development of an online game called “Anti-
sively in the literature, with some sources refer- Phishing Phil.” Anti-Phishing Phil is a game
ring to recommendations known as best practices developed to educate users about phishing. This
that users can follow to limit their exposure to game teaches users how to spot phishing attacks.
potential attacks. They also discuss how these The player (user) is playing as a fish named Phil.
best practices can be taught to a wide audience Phil is hungry and wants to eat worms, so he can
in an effective manner. become a bigger fish. The problem is that the
“Gamification is defined as a process that inte- worms are associated with URLs. These URLs
grates game elements into game fewer objects in can be either to legitimate websites or to phishing
order to have graceful characteristics” (Yohannis websites. The player needs to choose the right
et al. 2014), this means that gamification uses worms to eat before running out of time. Phil’s
game elements (badges, levels, time constraints, father is guiding the player by giving tips on how a
etc.) in a nongame environment to make the sys- user can detect phishing URLs, thus using enter-
tem have graceful characteristics, meaning it is tainment for the purpose of educating the users
not a full game. It only uses elements from about phishing. Anti-Phishing Phil uses rounds,
games. Deterding et al. (Deterding et al. 2011) scores, lives, and time for the purpose of enter-
say that gamification refers to the use, design, tainment. If the user chose a good URL, then the
elements, and characteristics of games in a non- user is going to achieve points. If the user chose a
game context. This means that gamification is in phishing URL, then the user is going to lose
120 Anti-phishing Attacks in Gamification

points. This is with the purpose of educating about avoidance behavior of individuals was heightened
phishing threats. by the individual’s own threat perception, per-
Users can be educated by means of a game to ceived severity, perceived susceptibility, self-
enhance the learning process (Sheng et al. 2007). efficacy, and safeguard effectiveness. This is
For instance, they can be presented with a series of while the cost of safeguarding provided an
URLs: they gain points for correctly identifying adverse impact.
safe URLs and lose points for clicking phishing
links. The results of the game carried out by “Smells Phishy?” Board Game
(Sheng et al. 2007) showed an increase of 60% On the front of anti-phishing, board games have
in the detection of phishing links, thereby show- been developed to increase the player’s awareness
ing the successful transfer of knowledge of phish- regarding online phishing scams. One particular
ing attacks onto users. Another example of a study board game, by Baslyman and Chiasson
that aims to educate users using a game is done by (Herzberg 2009), was used in the study which
(Zhang et al. 2007). This game gives users clues showed people who played the game now better
on how to identify URLs that can potentially be understand what phishing scams are and how they
used for phishing attacks, such as teaching users can best avoid falling into their traps
to rely as much as possible on the links on the first (Arachchilage and Hameed 2017).
page of search engines as these are typically the
most reliable ones. Despite increasing the user Embedded Training Email
awareness toward potentially harmful URLs, a Kumaraguru et al. (Kumaraguru et al. 2007) have
limitation of this approach is that it relies solely invented and developed a new technique called
on detecting the legitimacy of the URLs without embedded training email system, in the quest to
making reference to the detection of actual phish- teach people how to protect themselves against
ing messages. The outcome of the game was also phishing attacks, focusing on the use of email.
determined using a scoring system to increase Further experiments have illustrated that two
user interest in the issue. Teaching the concepts embedded training designs work much better
using this interactive approach (as shown by than the current use of sending security notices
(Sheng et al. 2007)) aims to educate users in a (Arachchilage and Hameed 2017).
dynamic manner where they aim to score the
highest score possible, thereby increasing user Automatic Content Generation
interest in the underlying concepts aimed to be Tseng et al. (Tseng et al. 2011) have focused on a
taught and overall retention of information (Quinn game project that turns the assessment of the
2005). content of a website into a game in order to
teach users how to detect phishing. This includes
Mobile Game describing stereotypes features of a phishing
A new mobile game, at a prototype level, is devel- attack by coming up with a hierarchy frame for
oped by Arachchilage et al. (Arachchilage and the phishing attacks. Furthermore, other proper-
Hameed 2017) with the aim to teach and train ties of the frame model, such as its instantiation
individual’s minds to defend themselves from and inheritance, enable the extension of the phish-
the different techniques used by phishing attacks. ing pages, which in turn increases the game con-
This game is centered around the purpose of tent. The test on the effectiveness of this technique
improving the user’s behavior toward subcon- was carried out by the use of an anti-phishing
scious detection of the threat of the phishing educational game, which resulted in most experts
attack and hence avoiding it. The study on the and participants satisfied with this system.
effectiveness of this mobile game on teaching
ordinary people to thwart phishing attacks has Self-Efficacy
shown a significantly promising result. The Archchilage et al. (Arachchilage and Love 2014)
study further brought to attention that the have intercalated both conceptual and procedural
Anti-phishing Attacks in Gamification 121

knowledge into their innovation approach into computer security by changing people’s under-
gamification. This focuses on their effect on the standing and discipline (Denning et al. 2013).
player’s self-efficiency to avoid phishing attacks. The players of the game simulate the employees A
Therefore, the aim is to add self-efficiency into the of Hackers Inc., which is a tech security com-
game frame in order to make the game better pany that performs advanced checks on the secu-
equipped for teaching anti-phishing techniques. rity of individuals by actively trying to hack it
However, this comes with the challenge of actu- and then provide consultation services to the
ally making a successful transplant of self- individual on whom to improve their security.
efficiency into the body of the game. This is For the evaluation of the game, a group of 22 edu-
achieved by teaching individuals how to well cators and 250 students were taking into account.
differentiate true URLs (uniform resource loca- Most of the educators indicated that the game
tors) from fake ones, using elements from a theo- was a great teaching tool, in that they would use
retical model (Lin et al. 2015). This model has it again, as it was greatly welcomed and enjoyed
continually shown that the interaction of concep- by the students, while it significantly increased
tual knowledge and procedural knowledge tend to their understanding of computer security. There-
produce a positive effect on an individual’s self- fore, some educators believed that they would
efficiency when it comes to avoiding phishing even recommend it to others. A secondary sup-
scams. plementary evaluation of 11 educators teaching
nonsecurity computer science courses showed
PicoCTF that their response similarly aligned with a larger
A new and innovative competition has emerged group, with some reporting that they would no
among high school computer science enthusiasts, longer teach any computer science course with
called PicoCTF (Chapman et al. 2014). Unlike the game.
conventional computer games, this competition
is aiming to broaden the student’s understanding
of computer security, through a series of web- Conclusion
based game challenges. PictoCTF is based on
the idea of capturing the flag. Teams race to In this article, the authors discussed and
solve computer security obstacles, looking for a referenced briefly phishing attacks focusing on
digital “flag,” encrypted in text or a binary pro- gamification which is actually social engineer-
gram and saved in an unknown server. Although ing. Then they categorized the existing strategies
the competition can last for a couple of days for a to tackle the issue and enhance the awareness of
team to win by discovering the exact location of users regarding the gamification solutions.
the flag, the many challenges faced by the com- Phishing manipulates individuals psychologi-
petitors and the array of solution they come up cally to reveal confidential information. Phishing
with will allow them to learn crucial skills in is a common form of identity theft and is consid-
computer forensics, cryptography, web security, ered a serious and server cybercrime. The most
reverse engineering, and binary exploitation. This common phishing uses emails, chats or websites,
game is both an entertaining, challenging yet legal and online games to get valuable information.
way for both students and tactically professional Education and training in the case of gaming
experts to practice and improve their skill in a have become very essential even for a common
computer breach, which in turn will give them a man to prevent phishing. In order to handle
better understanding of how breaches can happen phishing in gaming, some of the existing
and how to avoid them. methods have been discussed. Since organiza-
tions are emphasizing the education and training
Control-Alt-Hack of their employees for data protection, game-
Meanwhile, Control-Alt-Hack is a new design of based learning is one of the top methods to iden-
card game also aiming at increasing awareness of tify phishing attacks.
122 Anti-social Behavior

Cross-References Heinze, A., Fletcher, G., Rashid, T., Cruz, A.: Digital and
Social Media Marketing: a Results-Driven Approach.
Routledge (2016)
▶ Fingerprint Verification Based on Combining Herzberg, A.: Why Johnny can't surf (safely)? Attacks and
Minutiae Extraction and Statistical Features defenses for web users. Comput. Secur. 28(1–2),
63–71 (2009)
Hong, J.: The state of phishing attacks. Commun. ACM.
55(1), 74–81 (2012)
References Kumaraguru, P., et al.: Protecting people from phishing:
the design and evaluation of an embedded training
Anderson, R.: Security Engineering: A guide to building email system. In: Proceedings of the SIGCHI confer-
dependable distributed systems, 2nd edn. Wiley ence on Human factors in computing systems. ACM
(2008) (2007)
Arachchilage, N.A.G., Hameed, M.A.: Integrating self- Lin, C., et al.: Efficient spear-phishing threat detection
efficacy into a gamified approach to thwart phishing using hypervisor monitor. In: 2015 International
attacks. CoRR. abs/1706.07748 (2017) Carnahan Conference on Security Technology
Arachchilage, N.A.G., Love, S.: Security awareness of (ICCST). (2015)
computer users: a phishing threat avoidance per- Quinn, C.N.: Engaging learning: Designing e-learning
spective. Comput. Hum. Behav. 38, 304–312 simulation games. Wiley (2005)
(2014) Rittle-Johnson, B., Koedinger, K.R.: Comparing Instruc-
Butler, R.: A framework of anti-phishing measures aimed tional Strategies for Integrating Conceptual and Proce-
at protecting the online consumer's identity. Electron. dural Knowledge. (2002)
Lib. 25(5), 517–533 (2007) Robila, S.A., Ragucci, J.W.: Don't be a phish: steps in user
Chapman, P., Burket, J., Brumley, D.: PicoCTF: A Game- education. SIGCSE Bull. 38(3), 237–241 (2006)
Based Computer Security Competition for High School Sheng, S., et al.: Anti-Phishing Phil: the design and eval-
Students. in 3GSE. (2014) uation of a game that teaches people not to fall for
Cochin, C., et al.:, McAfee Labs Threats Report. phish. (2007. p. 88–99
(2014) Sullins, L.L.: Phishing for a solutions: Domestic and inter-
Denning, T., et al.: Control-Alt-Hack: the design and eval- national approaches to decreasing online identity theft.
uation of a card game for computer security awareness Emory Int. Law Rev. 20, 397–433 (2006)
and education. In: Proceedings of the 2013 ACM Swapan, P.: Phishing counter measures and their effective-
SIGSAC conference on Computer & communications ness – literature review. Inf. Manag. Comput. Secur.
security. ACM (2013) 20(5), 382–420 (2012)
Deterding, S., et al.: From game design elements to Thompson, S.T.C.: Helping the hacker? Library informa-
gamefulness: defining gamification. In: Proceedings tion. Security Soc. Eng. 25(4), 222–225 (2006)
of the 15th international academic MindTrek confer- Tseng, S.-S., et al.: Automatic content generation for anti-
ence: Envisioning future media environments. ACM phishing education game. In: Electrical and Control
(2011) Engineering (ICECE), 2011 International Conference.
Dodge, R.C., Ferguson, A.J.: Using phishing for user email IEEE (2011)
security awareness. In: IFIP International Information Yohannis, A.R., Prabowo, Y.D., Waworuntu, A.:
Security Conference. Springer (2006) Defining Gamification: From lexical meaning and
Dodge Jr., R.C., Carver, C., Ferguson, A.J.: Phishing for process viewpoint towards a gameful reality. In:
user security awareness. Comput. Secur. 26(1), Information Technology Systems and Innovation
73–80 (2007) (ICITSI), 2014 International Conference. IEEE
Downs, J.S., Holbrook, M., Cranor, L.F.: Behavioral (2014)
response to phishing risk. In: Proceedings of the anti- Yue, C., Wang, H.: BogusBiter: A transparent protection
phishing working groups 2nd annual eCrime against phishing attacks. ACM Transactions on Internet
researchers summit. ACM (2007) Technology (TOIT). 10(2), 6 (2010)
Fette, I., Sadeh, N., Tomasic, A.: Learning to detect phish- Zhang, Y., et al.: Phinding phish: Evaluating anti-phishing
ing emails. In: Proceedings of the 16th international tools. In: Proceedings of the 14th annual network and
conference on World Wide Web, pp. 649–656. ACM, distributed system security symposium (NDSS 2007).
Banff (2007) Citeseer (2007)
Forte, D.: Application delivery: pros and cons both
virtual and real. Netw. Secur. 2009(12), 18–20
(2009)
Gee, J.P.: What video games have to teach us about learn-
ing and literacy. Computers in Entertainment (CIE).
1(1), 20–20 (2003)
Anti-social Behavior
Greenberg, A.: Phishing costs average organization $3.7
million per year. SC Media: Online (2016) ▶ Griefing in MMORPGs
Area of Interest Management in Massively Multiplayer Online Games 123

AnvilNext Arab Filmmakers

A
▶ Assassin’s Creed, an Analysis ▶ Challenges Facing the Arab Animation Cinema

Any% Architectural Decoration

▶ Speedrunning in Video Games ▶ Holography as an Architectural Decoration

AODV Area of Interest

▶ Simulation and Comparison of AODV and ▶ Area of Interest Management in Massively

DSDV Protocols in MANETs Multiplayer Online Games

AOI Area of Interest Management

in Massively Multiplayer
▶ Area of Interest Management in Massively Online Games
Multiplayer Online Games
Laura Ricci1 and Emanuele Carlini2
1
Department of Computer Science
University of Pisa, Pisa, Italy
2
Applied Game ISTI-CNR, Pisa, Italy

▶ Hypermedia Narrative as a Tool for Serious

Games Synonyms

AOI; Area of interest

Applied Gaming
Definition
▶ Gamification
In a Massively Multiplayer Online Game, the
Area-Of-Interest (AOI) is that portion of the vir-
tual world of specific interest for a player.
Arab Animation

▶ Challenges Facing the Arab Animation Cinema Introduction

Users participating in a Massively Multiplayer

Online Game (MMOG) share a synchronous and
Arab Cinema persistent virtual world with each other through
Avatars, i.e., the players alter ego in the virtual
▶ Challenges Facing the Arab Animation Cinema world. In order to enable an engaging experience
124 Area of Interest Management in Massively Multiplayer Online Games

typical of MMOG, the state of entities in the referred to as AOI, and the awareness is mutual
virtual world, such as avatars and objects, has to such that if A is aware of B, B is also aware of
be replicated in avatar nodes in a timely fashion. A. Spatial-based AOIM mechanisms can employ
However, broadcasting all state changes to every a static and persistent implementation of the AOIs
node in a MMOG is not a practical solution. (Carlini et al. 2012), or dynamic, by adjusting the
Naturally, each avatar is interested to be updated AOI shape and size according to the events hap-
in only a subset of the whole virtual world, which pening in the virtual world (Ahmed and
is commonly referred to as Area of Interest (AOI). Shirmohammadi 2008).
The AOI management (AOIM), often referred to
also as Interest Management (2002), is a core Geographic AOIM
activity in the operations of a MMOG, and can Geographical AOIM exploits the subdivision of
be informally defined as the following: given an the virtual world into regions, which are then
avatar, identify its AOI and activate those opera- distributed to different servers. Geographical and
tions that support its timely update. AOIM is also spatial AOIM are often used in combination: the
referred to as spatial publish subscribe (Hu and spatial AOIM is used to select those, among the
Chen 2011). In this model, publishers perform regions provided by the geographical AOIM, that
action and interact with the virtual world (e.g., are of interest for the avatars. A coarse grained
perform some movements), whilst subscribers geographical AOIM is usually implemented by
manifest their interest of receiving updates for a large centralized MMOG by dividing the virtual
specific area of the virtual world. AOIM is a world into large regions and instances (Prodan
fundamental operation in both centralized and and Nae 2009). Normally, the nimbus of avatars
distributed MMOG architecture. In the centralized is much smaller than such regions, and therefore
ones, it is principally a mean to reduce the volume only one region (i.e., server) is selected, as typical
of messages exchanged by the client and the of centralized approaches.
server, as well as the amount of data stored and More fine grained subdivisions of the virtual
elaborated locally to the client. In distributed world are typical of decentralized MMOG archi-
MMOG architectures instead, how AOIM is tectures (Ricci and Carlini 2012). A common
performed greatly impacts on the whole architec- approach considers an uniform partitioning of
ture. In fact, AOIM often drives the design of the the virtual world into rectangles or hexagons,
whole decentralized architecture, forcing an orga- with the area of interest that can span more
nization of the connections between nodes so that regions. Static uniform partitioning approaches
they are only communicating with other nodes have the problem of properly defining the region
that have relevant entities. AOIM can be divided size. If the size is too large, an avatar could receive
into two main categories (Carter et al. 2012): state updates from entities not in its actual interest,
spatial and geographic. wasting resources. Otherwise, if the region is too
small, an avatar would need to manage multiple
Spatial AOIM region of interests and switch very frequently
Spatial AOIM is based on the concepts of aura and between them, generating a lot of overhead. In
nimbus (Boulanger et al. 2006). The aura is the order to overcome these problems, in dynamic
spatial area for which an entity is perceived by partitioning, the size of the region can be opti-
others, whilst the nimbus is the spatial area for mized according to various parameters, such as
which an entity can perceive. Hence, an entity the number of avatars in a region, or the compu-
A can perceive another entity B only if A’s nimbus tational power of the node managing a region
intersects with B’s aura. However, in this case, (Deng and Lau 2014). An evolution of the
B is not necessarily aware of A (i.e., the relation is dynamic partitioning approach is the hierarchical
not mutual). In practical implementations, how- partitioning, which is usually implemented by
ever, aura and nimbus coincide in a circle (or a considering tree-like structures, such as
sphere) with a predefined radius and centered on QuadTrees (Backhaus and Krause 2010). An
the avatar. In such case, the circle is simple advantage of this method is that the size of the
Artificial Intelligence 125

regions can be abstracted at the correct level by (ICCCN), 2012 21st International Conference on,
simply navigating the tree structure. A special pp. 1–5. IEEE, (2012)
Deng, Y., Lau, R.W.: Dynamic load balancing in distributed
instance of geographical AOIM is the one based virtual environments using heat diffusion. ACM Trans. A
on Voronoi tessellation (Hu et al. 2006). Multimedia Comput. Commun. Appl. 10(2), 16 (2014)
A Voronoi tessellation is a decomposition of a Hosseini, M., Pettifer, S., Georganas, N. D.: Visibility-
metric space determined by the distances from a based interest management in collaborative virtual
environments. In Proceedings of the 4th international
set of sites (usually determined by the position of conference on Collaborative virtual environments,
the Avatar) of a discrete set of objects in the space. pp. 143–144. ACM, (2002)
Given a set of N sites on a two-dimensional Hu, S.-Y., Chen, J.-F., Chen, T.-H.: Von: A scalable peer-
euclidean plane, the plane is partitioned into to-peer network for virtual environments. IEEE Net-
work. 20(4), 22–31 (2006)
N nonoverlapping regions, each one containing Hu, S.-Y., Chen, K.-T.: Vso: Self-organizing spatial pub-
all the points closer to that region site than to any lish subscribe. In: Self-Adaptive and Self-Organizing
other one. Voronoi-based AOIM approaches, and Systems (SASO). Fifth IEEE International Conference
the corresponding Delaunay triangulation (Ricci on, pp. 21–30. IEEE, (2011)
Prodan, R., Nae, V.: Prediction-based real-time resource
et al. 2015), are especially used in decentralized provisioning for massively multiplayer online
MMOG architectures as they facilitate the task of games. Future Generat Comput Syst. 25(7),
connecting neighbors with P2P mechanisms. In 785–793 (2009)
general, the Voronoi tessellation is dynamically Ricci, L., Carlini, E.: Distributed virtual environments: From
client server to cloud and p2p architectures. In: High
computed by considering the positions of the ava- Performance Computing and Simulation (HPCS), 2012
tars, then a peer-to-peer communication overlay is International Conference on, pp. 8–17. IEEE, (2012)
defined by connecting two peers managing ava- Ricci, L., Genovali, L., Carlini, E., Coppola, M.: Aoi-cast
tars whose Voronoi regions share at least an edge in distributed virtual environments: An approach based
on delay tolerant reverse com-pass routing. Concur.
(i.e., are neighbors) (Ricci et al. 2013). Comput. Pract. Exper. 27(9), 2329–2350 (2015)
Ricci, L., Genovali, L., Guidi, B.: Managing virtual entities
in mmogs: A voronoi-based approach. In: International
Cross-References Conference on E-Business and Telecommunications,
pp. 58–73. Springer, (2013)

▶ Interaction

References Area of Interest, AOI

Ahmed, D. T., Shirmohammadi, S.: A dynamic area of ▶ Peer-to-Peer Gaming

interest management and collaboration model for p2p
mmogs. In: Proceedings of the 2008 12th IEEE/ACM
International Symposium on Distributed Simulation
and Real-Time Applications, pp. 27–34. IEEE Com-
puter Society, (2008) Artificial Cognitive
Backhaus, H., Krause, S.: Quon: A quad-tree-based over- Intelligence
lay protocol for distributed virtual worlds. Int. J. Adv.
Media Commun. 4(2), 126–139 (2010)
Boulanger, J.-S., Kienzle, J., Verbrugge, C.: Comparing
▶ Computer Games and Artificial Intelligence
interest man-agement algorithms for massively multi-
player games. In: Proceedings of 5th ACM SIGCOMM
Workshop on Network and System Support for Games,
p. 6. ACM, (2006).
Carlini, E., Ricci, L., Coppola, M.: Reducing server load in
Artificial Intelligence
mmog via p2p gossip. In: Network and Systems Sup-
port for Games (NetGames), 2012 11th Annual Work- ▶ Genetic Algorithm (GA)-Based NPC Making
shop on, pp 1–2. IEEE, (2012) ▶ Foundations of Interaction in the Virtual Real-
Carter, C., El Rhalibi, A., Merabti, M.: A survey of AOIM,
distribution and communication in peer-to-peer online
ity Medium
games. In: Computer Communications and Networks ▶ Human Interaction in Machine Learning (ML)
126 Artificial Intelligence Agent

for Healthcare Introduction

▶ RTS AI Problems and Techniques
▶ StarCraft Bots and Competitions The ARC is always tied to a human user Scholz
and Smith (2016); Dirin and Laine (2018). It
cannot exist without a human being because it is
just an experience. It is possible for many humans
Artificial Intelligence Agent to view and interact with the same artificial reality
(Kaufmann (2003); Pidel and Ackermann
▶ Assassin’s Creed, an Analysis (2020)). Recently, collaborative artificial reality
applications are commonly referred to as the
Metaverse Mystakidis (2022). Generally, the arti-
ficial reality continuum is a virtual concept and is
Artificial Reality experienced through the visual sense of a human.
Besides vision, the hearing sense also plays a
▶ Origin of Virtual Reality secondary important role. Through sound, the
level of immersiveness of the artificial reality
experience is increased. In some cases, the visual
artificial reality gets extended and stimulates the
Artificial Reality Continuum other human senses touch, taste, and smell.
The disciples in the ARC are Augmented Real-
Manuel Rebol1,2 and Krzysztof Pietroszek3 ity, Mixed Reality, and Virtual Reality. It is also
1
American University, Washington, DC, USA important to define the boundaries of the ARC. An
2
Graz University of Technology, Graz, Austria important factor is the degree of visual
3
Immesive Designs, Experiences, Applications immersiveness and interaction with the environ-
and Stories (IDEAS) Lab, School of ment. A traditional theater experience can be
Communication, American University, physiologically immersive. People’s emotion
Washington, DC, USA change because they feel with the characters.
Yet, the acting takes place in reality and is there-
fore not considered artificial reality. Sitting and
Synonms watching in a movie theater can be immersive,
and the reality that is shown on the screen is
Augmented reality; Extended reality spectrum artificial, but still this is not considered artificial
reality because there is no active interaction
between the movie and the environment the
Definition viewer is in.
The ARC was defined from a device perspec-
The Artificial Reality Continuum (ARC) tive by Milgram et al. (1994). They create a tax-
describes the spectrum of technologies that aug- onomy that divides the ARC in three dimensions:
ment or replace the natural sensory stimuli: reality, immersion, and directness. The three axes
vision, hearing, touch, smell, or taste. On one refer to the extent of world knowledge, reproduc-
side of the spectrum is the reality itself, as per- tion fidelity, and extent of presence metaphor.
ceived with our own senses without artificial Skarbez et al. (2021) adapted the taxonomy by
augmentation. On the other side of the spectrum Milgram et al. (1994) and introduced the three
is the virtual reality, where sensory stimuli are dimensions’ extent of world knowledge, immer-
entirely artificially generated. In between the sion, and coherence. This work analyses the ARC
reality and virtual reality, the degree of augmen- from a user experience and application perspec-
tation, extension, or replacement of natural sen- tive focusing on current technological capabilities
sory stimuli varies. with current devices. Moreover, we define Mixed
Artificial Reality Continuum 127

Reality (MR) as a stage between Augmented the virtual environment can influence the real
Reality (AR) and Virtual Reality (VR) rather environment. For example, the virtual environ-
than an umbrella term describing the whole spec- ment can tell the user how to perform a certain A
trum. Instead, Extended Reality (XR) includes operation in the real environment. Examples
AR, MR, and VR. An overview of how the ARC include remote maintenance (Mourtzis et al.
is presented in this work is shown in Fig. 1. 2020), and health-care assistance (Rebol et al.
2022). Once the user performed the instructions
visualized in the virtual environment, they indi-
Disciplines Within the Artificial Reality rectly changed the real environment.
Continuum
Augmented Reality
People start to talk about the artificial reality con- In augmented reality (AR), the information flow
tinuum as soon as the information that is visually between the virtual and the real environment goes
displayed depends on the environment the user is only from the real to the virtual environment. The
in. In all forms of artificial reality, the awareness real environment gets extended with visual and
between virtual and reality goes only in one direc- auditory information (Reipschlager et al. 2021).
tion. From the real environment to the virtual In some AR applications, the physical environ-
environment. The interaction can never happen ment influences how the virtual world is rendered.
in the opposite direction because this would However, this is not a requirement for AR. Some
mean that reality changes directly from the influ- AR applications scan the physical environment to
ence of the virtual reality (VR) which is physically provide anchored augmentations depending on
not possible. However, the user can act as a bridge which real environment the user sees. This allows
between virtual and real environment. Indirectly for predefined visualizations to get displayed on

Artificial Reality Continuum

Extended Reality
XR

"XR combines AR, HTC Vive XR Elite

MR, and VR."

Augmented
Mixed Reality Virtual Reality
Reality
MR VR
AR
Virtual World
Real World "Visuals are "Augmented visuals
"The user is visually
overlayed onto the are anchored and
transfered into a
user's real-world interact with the
virtual world"
view" real world"

Sample Head-
mounted display
Microsoft Meta Oculus
Google Glass
Hololens Quest

Sample
Application
Remote Assistance Medical Procedure
Star Identification
for Repair Simulation

Artificial Reality Continuum, Fig. 1 An overview of the Artificial Reality Continuum (ARC)
128 Artificial Reality Continuum

top of reality. Augmented reality is not designed collaboration. Virtual objects are used to explain
to change the real environment. The boundary concepts in the real environment. One person can
between augmented reality and mixed reality is guide another person if the mixed reality views are
at the point where the augmented information tries transferred over the Internet (Rebol et al. (2022);
to influence the real environment. The sole pur- Mourtzis et al. (2020)). One person can give
pose of augmented reality is to enrich reality by direction using virtual objects on the other side.
providing additional visualizations. The person that receives the direction through
In museums, AR is used to provide text mixed reality can work based on the directions
descriptions and animations for exhibits (Ding received. In industries, mixed reality is often used
et al. 2017). In books, AR is used to extend tradi- in cases where a remote expert guides a local
tional 2D illustrations with 3D AR scenes (Dünser operator (Rebol et al. 2022). Traditional 2D
et al. 2012). Interactivity does not take place with video guidance is sometimes not practical because
the AR objects itself. Only small changes in the the complexity of the problem requires spatial
virtual environment are possible such as changing information. In the logistics industry, special
the position where the virtual elements appear. machines are deployed in warehouses (Mourtzis
Pokemon Go pok (2022) is the most famous AR et al. 2020). Only few experts are able to fix the
game. Besides the entertainment applications, AR machines whenever they stop working. In order to
is also used in heads-up displays in air crafts and repair machines efficiently, a remote expert guides
vehicles (Jose et al. 2016). The virtual elements in a local mechanic through the procedure with
the game (Pokemon) appear augmented through a MR. The remote expert would otherwise need to
screen on the real world. Augmented reality travel long distances to fix the problem which
devices are displays of phones and tables, pro- would take more time and cost would be high.
jectors, and glasses with integrated see-through
displays. Virtual Reality
Virtual reality (VR) tries to completely detach the
Mixed Reality user from reality and take the user to an all virtual
In contrast to augmented reality, mixed reality environment (Auda et al. 2020). Although the
(MR) not only enriches the real environment by user keeps physically in the real world, virtual
displaying visualizations for the user but also reality creates the impression that the user enters
makes the user interact with the environment. a new virtual world. Compared to augmented and
The virtual environment changes the real environ- mixed reality, the user does not see anything from
ment with the help of the user. The interaction the real environment, except for themselves in
must include the physical environment and can cave automatic virtual environment (CAVE)
also take place in the virtual world. MR headsets experiences (Muhanna 2015). Most of the time,
that support hand tracking allow the user to intu- the visual experience is also paired with audio and
itively alter the virtual world. Moreover, in every the user wears headphones which makes the user
mixed reality application the virtual environment hear sounds from the virtual environment. Mod-
is fully aware of the real environment. Mixed ern virtual reality devices are displays mounted
reality is the most complex technology because very close to the user’s eyes to cover the real
it tries to create a new seamlessly combined real- environment. They are head-mounted displays
ity. In AR, the attention is on the real environment (HMDs) consisting of the display, headphones,
and the augmentations just enrich the reality. In microphone, and sensors to track the user’s
virtual reality, the attention is on the virtual envi- motion. In order to make the virtual experience
ronment and the user is not visually aware of the of Artificial Reality Continuum 5 as realistic as
real environment. In MR, the attention switches possible, the motion of the user is tracked and
between the real and the virtual environment. transferred to motion in the virtual environment.
Mixed reality creates a cooperative scene VR device tracking ranges from head tracking
which can be used for virtual meetings and only to full body tracking.
Artificial Reality Continuum 129

Similar to mixed reality, virtual reality is used currently in. Those devices only need a projecting
for virtual meetings and collaboration. Distant display. They do not need sensor to scan the
people can meet in the virtual world to discuss environment. A
work in the real world or work together on virtual The more sophisticated version of AR is par-
tasks. People see each other as represented as tially aware of its environment. This version of AR
avatars in the VR. The degree of avatar detail scans the environment and searches for predefined
and realism changes. The degree of avatar realism elements on which information is augmented. For
has an influence on the perception in VR example, in an AR book (Dünser et al. 2012), the
Latoschik et al. (2017). More abstract avatar rep- AR device is searched for illustrations or markers.
resentations require less computing power. Real- Whenever something is detected, the
istic representations can be retrieved by a corresponding visual information is overlaid for
volumetric capture of a person (Yu et al. 2021). the user. This detection requires additional sensors
However, in combination with realistic capture, for the AR device. Mostly, these types of AR
precise motion tracking and mesh deformations devices contain a display and a color camera.
are required to reach a high degree of realism. Devices are mainly phones, tablets, and glasses.
One difference between virtual reality and aug- The color camera captures the real scene and
mented and mixed reality is the fact that it is applies image recognition algorithms to detect
completely detached from the real environment. given objects in the scene. Consequently, these
Therefore, the user can be taken into a completely types of AR devices require more computations
new world which the user would not be able to resources compared to the most simple form of
experience in reality. It can be a place on real AR. Despite recognition, tracking is performed to
world that is somehow unreachable for the user stabilize augmentations when the user moves the
such as the cockpit of a rocket or distant vacation AR device. The tracking makes this type of AR
destination. It can also be a completely fantasy more enjoyable than simple AR displays. It makes
place. This allows for the application to simulate the virtual and the real environment merge and the
situations and train people. Examples include fire experience for the user is better because the user
rescue, medical training, and pilot training. By does not need to switch visually between the
combining a visual VR experience with additional decoupled virtual and real environment. Accurate
realism such as sound, haptic feedback, environ- tracking is an important factor for a smooth AR
ment motion, and smell, a highly realistic scene is experience. Slow or inaccurate tracking results in
created. The user’s body is physically moved high cognitive load for the user and can result in
according to the visual VR experience. motion sickness (Kaufeld et al. 2022).
Mixed reality is very closely related to the
The Continuum more sophisticated version of AR described
The transition between AR and MR is seamless. above. The main difference is that in mixed reality
MR applications include AR. AR and MR can be the awareness of the real environment is even
distinguished by the sensors the display device is higher when creating the virtual environment.
using. In AR, there exist two types of augmenta- Therefore, MR devices have more sensors than
tion. The most simple version of AR augments AR devices. They spatially capture the surround-
information onto a display without taking the real ings the user is in. A depth sensor, a stereo camera
environment into account. One example for this is setup, or laser-based systems are used to capture
a heads-up display in a car that projects cruising the 3D scene. In order to process this information,
information such as speed and direction onto the more computations resources are required com-
windshield (Jose et al. 2016). Another example pared to AR devices. Typical MR devices include
would be wearable see-through glasses that pro- phones which 3D capture capabilities and head-
ject the current time and weather information into mounted displays.
the field of sight of the user. This information is In MR, device tracking becomes an important
projected regardless of the environment the user is part because all the virtual objects are linked to the
130 Artificial Reality Continuum

real objects and need to stay at the same position. (2017); Solbiati et al. (2020)). The most famous
On head-mounted displays, the motion of the AR Game is Pokemon Go. The most popular AR
user’s head needs to be tracked (Sathyanarayana application outside gaming is the heads-up dis-
et al. 2020). On phones, the position of the device play in cars. Mixed reality applications are not
needs to be tracked. Especially with head- widely spread. They are mainly used for commer-
mounted displays presenting a high portion of cial and research purposes (Rebol et al. 2022). MR
virtual environment, motion sickness (Kaufeld is used for remote assistance, for example, special
et al. 2022) becomes a concern, and therefore machine maintenance (Mourtzis et al. 2020). It is
accurate head-tracking becomes essential. Even also used in medicine for 3D visualization of
with a low degree of virtual environment, head- recorded volumetric data on patients. Virtual real-
tracking needs to be accurate in order to decrease ity is much more popular than MR. Especially the
cognitive load and to support seamless integration gaming industry uses VR head-mounted displays
of the virtual and the real environment. To allow to create immersive gaming experiences. The
for accurate tracking of user motion, the data from movie industry has produced documentaries
multiple sensors inside the MR device are utilized, in VR.
including depth sensors, multiple RGB cameras, One obstacle toward the broader adoption of
accelerometer, and gyroscope. the ARC is the lack of standardization of devices.
Mixed reality applications typically do contain AR, MR, and VR devices are designed with dif-
AR elements that are not aware of the real envi- ferent sensors and capabilities which makes cross-
ronment and are visualized on a fixed screen posi- platform development difficult. Standardized
tion. These elements are typically application application development between different
menus that are used frequently by the user and mobile devices, head-mounted displays, and
therefore need to be close. Mixed reality applica- other ARC devices needs to be introduced. The
tions have the option to guide the user to engage type of sensors on devices needs to provide stan-
not only with the virtual environment, but also dardized capabilities such as device tracking and
with the real environment. Thus, MR is the pre- remote connection.
ferred discipline within the artificial reality con-
tinuum for effective remote collaboration.
Although MR applications with a high portion Interaction in the Artificial Reality
of virtual elements come close the virtual reality Continuum
experience, the concept of VR is substantially
different from that of AR and MR. The user drifts The interaction with the ARC depends on the
completely into the virtual world. VR devices are device. Modern mobile devices including phones
designed such that they block the sight of the real and tables offer interaction with AR capabilities.
environment. VR is predominantly consumed They 3D scan the environment with a multi-
with a head-mounted display which blocks any camera setup and support AR and MR applica-
light coming from the real environment to the tions. Phone displays offer low-entry VR support
user’s eyes. A very high immersiveness factor with additional head-mounted hardware. AR
within the artificial reality continuum is reached applications for mobile phones are popular
when VR does not only use the visual sense to because of the wide availability of the hardware.
bring the user into the virtual world. The real The fact that the display needs to be carried makes
environment might react corresponding to the vir- mobile phones a less suitable device for ARC
tual environment to simulate motion and touch. applications. Moreover, state-of-the-art phones
do not offer stereoscopic display capabilities.
Applications of Artificial Reality Consequently, only 2D views are supported.
The artificial reality continuum is present in video In contrast to phones, glasses and lenses offer
games as well as other applications (Das et al. 3D visualization and are easier to carry. The
Artificial Reality Continuum 131

display and the computing unit need to be sepa- have to become handier and part of people’s
rated on these devices for comfortable usage. everyday life.
Glasses and lenses offer a much more intuitive A
way of consuming ARC content. When using
mobile device, the interaction can happen through Cross-References
traditional touchscreen controls but can also hap-
pen through more intuitive in-the-air gestures. ▶ Tabletop Storytelling
Gestures are detected using integrated hand pose
estimation algorithms which are less accurate than
touchscreen controls. On head-mounted displays, References
interaction is possible with gestures. Integrated
depth cameras can ensure high precision such Auda, J., Gruenefeld, U., Mayer, S.: It takes two to tango:
Conflicts between users on the reality-virtuality contin-
that the interaction with virtual objects feels sim-
uum and their bystanders. In XR@ ISS (2020)
ilar to interaction with physical objects. Das, P., Zhu, M.’o., McLaughlin, L., Bilgrami, Z.,
Within the ARC, augmented reality does not Milanaik, R.L.: Augmented reality video games: New
support interaction. Mixed reality applications possibilities and implications for children and adoles-
cents. Multimodal Technol. Interact. 1(2) (2017).
require spatial scene understanding to allow for
https://doi.org/10.3390/mti1020008. ISSN 2414-4088.
interaction with both physical and virtual objects https://www.mdpi.com/2414-4088/1/2/8
in a natural way. Natural interaction happens Ding, M. et al.: Augmented reality in museums. Museums
through user gesture control. Hence, gesture & augmented reality–A collection of essays from the
arts management and technology laboratory,
detection is built in. In VR, gestures are detected
pp. 1–15 (2017)
with controllers or stereo cameras. Besides ges- Dirin, A., Laine, T.H.: User experience in mobile aug-
tures, voice recognition is used for communica- mented reality: Emotions, challenges, opportunities
tion in a virtual environment. The disadvantage and best practices. Computers. 7(2) (2018). https://
doi.org/10.3390/computers7020033. ISSN 2073-
of gesture recognition with controllers is that the
431X. https://www.mdpi.com/2073-431X/7/2/33
user has to hold a physical object which is less Dünser, A., Walker, L., Horner, H., Bentall, D.: Creating
natural than gesture recognition with cameras. interactive physics education books with augmented
Gesture recognition with separate static hand- reality. In: Proceedings of the 24th Australian
Computer-Human Interaction Conference, OzCHI’12,
tracking devices is feasible put limits the area of
pp. 107–114. Association for Computing Machinery,
interaction and is therefore mainly suitable New York (2012). https://doi.org/10.1145/2414536.
for VR. 2414554. ISBN 9781450314381
Jose, R., Lee, G.A., Billinghurst, M.: A comparative study
of simulated augmented reality displays for vehicle
navigation. In: Proceedings of the 28th Australian Con-
Conclusion ference on Computer-Human Interaction, OzCHI’16,
pp. 40–48. Association for Computing Machinery,
The artificial reality continuum offers different New York (2016). https://doi.org/10.1145/3010915.
3010918. ISBN 9781450346184
ways for the user to interact with virtual content: Kaufeld, M., Mundt, M., Forst, S., Hecht, H.: Optical see-
augmented reality, mixed reality, and virtual real- through augmented reality can induce severe motion
ity. Current consumer products come from both sickness. Displays. 74, 102283 (2022). https://doi.org/
ends of the spectrum which are augmented reality 10.1016/j.displa.2022.102283. ISSN 0141-9382.
https://www.sciencedirect.com/science/article/pii/
and virtual reality. Especially for productivity and S0141938222001032
collaboration, mixed reality plays an important Kaufmann, H.: Collaborative augmented reality in education.
role. It is widely adopted in industry and is on Institute of software technology and interactive systems,
the way to enter consumer markets. Vienna University of Technology, pp. 2–4 (2003)
Latoschik, M.E., Roth, D., Gall, D., Achenbach, J.,
Head-mounted displays combined with Waltemate, T., Botsch, M.: The effect of avatar realism
human user tracking offer the most intuitive tech- in immersive social virtual realities. In: Proceedings of
nology to experience the artificial reality contin- the 23rd ACM Symposium on Virtual Reality Software
uum. In the future head-mounted devices will and Technology, VRST’17. Association for Computing
132 Artistic Data Visualization in the Making

Machinery, New York (2017). https://doi.org/10.1145/ Solbiati, L., Gennaro, N., Muglia, R.: Augmented reality:
3139131.3139156. ISBN 9781450355483 From video games to medical clinical practice (2020)
Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Aug- Yu, T., Zheng, Z., Guo, K., Liu, P, Dai, Q., Liu, Y.:
mented reality: A class of displays on the Function4d: Real-time human volumetric capture from
reality-virtuality continuum. Telemanipul. Telepres. very sparse consumer rgbd sensors. In Proceedings of
Technol. 2351, 01 (1994). https://doi.org/10.1117/12. the IEEE/CVF Conference on Computer Vision and
197321 Pattern Recognition (CVPR), pp. 5746–5756 (2021)
Mourtzis, D., Siatras, V., Angelopoulos, J.: Real-time
remote maintenance support based on augmented real-
ity (ar). Appl. Sci. 10(5) (2020). https://doi.org/10.
3390/app10051855. ISSN 2076-3417. https://www.
mdpi.com/2076-3417/10/5/1855 Artistic Data Visualization
Muhanna, M.A.: Virtual reality and the cave: Taxonomy, in the Making
interaction challenges and research directions. J. King
Saud Univ. Comp. Inf. Sci. 27(3), 344–361 (2015)
Mystakidis, S.: Metaverse. Encyclopedia. 2(1), Rebecca Ruige Xu
486–497 (2022). https://doi.org/10.3390/encyclope- Syracuse University, Syracuse, NY, USA
dia2010031. ISSN 2673-8392. https://www.mdpi.
com/2673-8392/2/1/31
Pidel, C., Ackermann, P.: Collaboration in virtual and
augmented reality: A systematic overview. In: De Synonyms
Paolis, L. T., Bourdot, P. (eds.) Augmented Reality,
Virtual Reality, and Computer Graphics, pp. 141–156 Artistic visualization; Data visualization
(2020). Springer International Publishing, Cham. ISBN
978-3-030-58465-8
Pokémon go, Dec 2022. https://en.wikipedia.org/wiki/Pok Definition
%C3%A9mon_Go
Rebol, M., Pietroszek, K., Ranniger, C., Rutenberg, A.,
Hood, C., Sikka, N., Li, D., Gütl, C.: Mixed reality Artistic data visualization is visualization of data
communication for medical procedures: Teaching the done by artists with the intent of expressing a
placement of a central venous catheter. In 2022 IEEE point of view.
International Symposium on Mixed and Augmented
Reality (ISMAR) (2022)
Reipschlager, P., Flemisch, T., Dachselt, R.: Personal aug-
mented reality for information visualization on large
Introduction
interactive displays. IEEE Trans. Vis. Comput. Graph.
27(2), 1182–1192 (2021). https://doi.org/10.1109/ In recent years, we have seen an increasing of
TVCG.2020.3030460 interest in data visualization in the artistic com-
Sathyanarayana, S., Leuze, C., Hargreaves, B., Daniel, B.,
Wetzstein, G., Etkin, A., Bhati, M.T., McNab, J.A.:
munity. Many data-oriented artworks use sophis-
Comparison of head pose tracking methods for ticated visualization techniques to express point
mixed-reality neuronavigation for transcranial mag- of views or achieve persuasive goals. Meanwhile
netic stimulation. In: Fei, B., Linte, C.A. (eds.) Medical the attitude that visualizations can be used to
Imaging 2020: Image-Guided Procedures, Robotic
Interventions, and Modeling, vol. 11315, p. 113150L.
persuade as well as analyze has been embraced
International Society for Optics and Photonics, SPIE by more people in the information visualization
(2020). https://doi.org/10.1117/12.2547917 community.
Scholz, J., Smith, A.N.: Augmented reality: Designing Here I will share my experience and reflection
immersive experiences that maximize consumer engage-
ment. Bus. Horiz. 59(2), 149–161 (2016). https://doi.org/
in creating data visualization as artwork via case
10.1016/j.bushor.2015.10.003. https://www. study of two recent projects. It presents a
sciencedirect.com/science/article/pii/S0007681315- workflow from conceptual development, data
001421. ISSN 0007-6813 analysis, to algorithm development, procedural
Skarbez, R., Smith, M., Whitton, M.C.: Revisiting
milgram and kishino’s reality-virtuality continuum.
modeling, and then final image production. It
Front. Virt. Reality. 2 (2021). https://doi.org/10.3389/ hopes to offer insight into the artist’s effort of
frvir.2021.647997. ISSN 2673-4192 finding balance between persuasive goals and
Artistic Data Visualization in the Making 133

analytic tasks. Furthermore, it raises the question Case Study: Perpetual Flow
of the roles of artistic data visualization played in
assisting people to comprehend data and the influ- This project explores an aesthetic-oriented A
ence of this artistic exploration in visualization approach to visualizing federal spending in the
might have injected in shifting public opinions. United States as 3D compositions in a photo-
realistic style. Using procedural modeling with
Python programming and Maya API, an organic
Case Study: Out of Statistics: Beyond
flow of intermingled geometrical units is formed
Legal
to represent the profile of federal spending for
each state, loosely resembling the idea of money
This project produces a series of 52 abstract draw-
flow. The total amount of spending is scaled to a
ings based on US crime statistics as digital prints
per capita basis to make different states compara-
on rice paper and silk panels. Each image repre-
ble, while the overall surface area or volume
sents the crime status in one of the states, with the
occupied by each type of geometrical pattern rep-
seven most significant crime-conviction statistics
resents its associated spending data (Xu and Zhai
of each state embedded.
2013).

Biography

Rebecca Ruige Xu currently teaches computer

art and animation as an Associate Professor in
College of Visual and Performing Arts at Syra-
cuse University. Her artwork and research inter-
ests include experimental animation, visual
music, artistic data visualization, interactive
installations, digital performance, and virtual real-
ity. Her recent work has been appeared at Ars
Electronica; SIGGRAPH Art Gallery; Museum
134 Artistic Visualization

of Contemporary Art, Italy; Aesthetica Short Film Action A game that requires physical
Festival, UK; CYNETart, Germany; International game skills and challenges such as
Digital Art Exhibition, China; Los Angeles Center hand–eye coordination and
for Digital Art; Boston Cyberarts Festival. She has reaction time
also been a research fellow at Transactional Adventure An interactive story driven by
Records Access Clearinghouse, Syracuse game exploration, dialogues, and
University since 2011. puzzle-solving
Artificial An automated agent that performs
intelligence tasks by mimicking some levels of
References agent human intelligence
AnvilNext A game engine created in 2007 by
Xu, R.R., Zhai, H.S.: Visualizing federal spending. Ubisoft Montreal video game
Leonardo J. Int. Soc. Arts Sci. Technol. 46(4),
414–415 (2013)
developers for use on Microsoft
Windows, Nintendo Switch,
PlayStation 3, PlayStation 4,
PlayStation Vita, Wii U, Xbox
Artistic Visualization 360, and Xbox One

▶ Artistic Data Visualization in the Making Abstract

Assassin’s Creed is an action-adventure video

Assassin’s Creed game franchise created by Patrice Désilets, Jade
Raymond, and Corey May, developed and
▶ Assassin’s Creed, an Analysis published by Ubisoft using the game engine
AnvilNext. Coded mainly in C++ and C#, Anvil
is a game engine created in 2007 by Ubisoft
Montreal for use on Microsoft Windows,
Assassin’s Creed, an Analysis Nintendo Switch, PlayStation 3, PlayStation 4,
PlayStation Vita, Wii U, Xbox 360, and
Michael McMillan2 and Newton Lee1,2 Xbox One.
1
Institute for Education, Research, and
Scholarships, Los Angeles, CA, USA
2
Vincennes University, Vincennes, IN, USA Assassin’s Creed IV: Black Flag

Have you even wondered what it would be like to

Synonyms be a pirate in the middle of the Caribbean during
the height of pirating? Well, look no further than
Action-adventure game; AnvilNext; Artificial the awesome sequel to the Assassin’s Creed Series:
intelligence agent; Assassin’s Creed Assassin’s Creed IV – Black Flag. The Ubisoft
company, specifically the Ubisoft Montreal office,
made every effort to make sure this game was a
Definition blockbuster on its release back on the 29 October of
2013, to include utilizing the advanced AnvilNext
Action- A game with a mix of elements Engine for the Xbox 360 release. The game was
adventure from an action game and an during its time one of the most advanced action-
game adventure game adventure/stealth games of that year with an initial
Assassin’s Creed, an Analysis 135

release on Xbox 360, PlayStation 3, and Wii U; it largest map ranging everywhere south of the
was later patched over to the PlayStation 4, Xbox tip of Florida to Jamaica as shown in the
One, and Microsoft Windows after the release of following picture: A
the next-gen consoles on 22 November 2013. This
gory single-player/multiplayer will have you sit-
ting on the edge of your seat, taking full advantage
of the ESRB rating of M for Mature.
The game’s main target audience was young
adults between the ages of 18 and 25. The story
was an easy hook for people who are fans of the
series or not. Being a ship hand named Edward
Kenway during the 1700s in the Caribbean who
turns pirate to earn fortune has a brief run in with
an assassin captain of an enemy vessel, whom
Edward kills and steals his gear to impersonate a
captain of a ship without realizing the uniform has
other ties besides just the title of “Captain.” With
the gear he is consistently being pursued by an
organization called the Templars who are seeking
a device in the Caribbean and the Assassin’s
Brotherhood who seek him out for assistance in With this large map, most players would be
stopping the Templars. This game is so appealing daunted to think about the number of load screens
not just due to the awesome aspects of pirating but that would have to go through every time they made
also due to doing it during the greatest age of a move toward one place or the other. The load time
pirating! Plus, this series has a long working is a little above average, with approximately a
story arc over many games when it comes to the 1–2 min load screen just to get into the game, but
“present-day” story portion. once you are in, you are in a completely open world
with only two exceptions. The first being story cut
scenes which is a given, but they flow well and do
Games Mechanics not have a load screen to prompt them. The other
being the quintessential “Fast Travel” locations in
The games mechanics were unique even for this game, which do have a load screen, but it is dra-
series. Each game in the Assassin’s Creed fran- matically reduced compared to the opening screen,
chise has brought a little something new to with only about a 30 s load time.
improve on the previous game, but they are If you are thinking a map this large, surely, you
usually minor like a slight adjustment to the would speed all your time just using the fast travel
combat mechanics or the ability to parkour points to get around. Wrong! You have a ship that
(wall climb) up and down with a simplified but- can travel around this area at a reasonably quick
ton combo than in the previous game. In Black pace. You can go from “driving” to “walking
Flag however, they broke the mold of Assassin’s around” in character at a push of a button. The
Creed. fluid ship to person back to ship interaction makes
Assassin’s Creed has been well known it really easy to travel because all you have to do is
since the first game for its large landscapes, pull up next to an island you wish to visit, stop
big maps, and the use of the parkour system driving the ship, and just jump off the side to swim
to navigate the map in literally three dimen- ashore (again no loading screen for visiting a new
sions. With that said, Black Flag had the location in the map).
136 Assassin’s Creed, an Analysis

Artificial Intelligence Agent Controversy

Ubisoft did this mechanic due to another feature The only controversy that popped up for this
that they added into the game: a literal never- game is that PETA, the People for the Ethical
ending number of AI NPC ships that you can Treatment of Animals, criticized the game for its
attack and loot or even take over by engaging in usage of harpoons and the glorification of whal-
hand-to-hand combat while boarding the ship. ing. PETA also came out and said that it was
These AI ships range from smaller ships which disgraceful that the video game industry did not
are easy to take out for the early game play to condemn it. Ubisoft’s response was a statement
four “God” ships in hidden locations spread saying that Assassin’s Creed is based on history
around the map which are near impossible to and it is a “work of fiction which depicts real
defeat unless you have your ship fully rigged events during the Golden Era of Pirates.” Ubisoft
with the best gear. also claimed they do not condone whaling, along
with other mechanics and features in the fourth
game in the series as shown in the following
Storyline pictures:

The story is extremely in depth, with plenty of

actual real-world locations to explore and
dozens of historical figures from the Golden
Age of Piracy to interact with. An example is
Edward Teach who is better known as “Black
Beard,” the notorious English pirate who was a
scourge of the American coast for many years
before his death at the hands of sailors sponsored
by the Governor of Virginia to capture or kill the
pirate. Other big names from the day are Anne
Bonny, Charles Vane, John Rackham better
known as Calico Jack, and Benjamin
Hornigold – all of whom are notorious pirates
from that time.
Assassin’s Creed IV: Black Flag’s open-world
concept with a free-flowing story and the option
to ignore the story and just do pirating stuff has
received great public reception and was gener-
ally held as vastly superior to its predecessor
Assassin’s Creed III. Two days after the game’s
release, IGN ranked Assassin’s Creed IV: Black
Flag as the second best game in the Assassin’s
Creed series, only behind Assassin’s Creed II Criticisms
(Sliva 2013).
This game is perhaps the best game in the The game was released only a month prior to a
series by far. It has all the best features of new generation of consoles coming out, and there-
the series with double story, open range fore it was built using an engine for the older
action/adventure gameplay, and the awesome generation of consoles (Xbox 360, PlayStation 3).
aspect of interacting with some of history’s When it was ported over to the new consoles
biggest names during the Golden Era of (Xbox One, PlayStation 4), Ubisoft did not opti-
Pirates. mize the game for the better hardware in the newer
Assassin’s Creed, an Analysis 137

Assassin’s Creed, an Analysis, Fig. 1 In the image, the in the water, but it is invisible to the player due to a
large “hole” in the water is actually the player’s ship resting software bug caused by the newer hardware

Assassin’s Creed, an Analysis, Fig. 2 Assassin’s Creed the Third Crusade in multiple locations in and around
was released in 2007. You play as Desmond Miles in the Jerusalem
modern story and as Altaïr Ibn-La’Ahad who lives during

consoles. As a result, many glitches happened Another major complaint from the gaming
when attempting to play the game on the newer community is about the annual release of a
consoles. In the image above, the large “hole” in new game. Ubisoft has been criticized for
the water is actually the player’s ship resting in the causing an oversaturation of the genre and a
water, but it is invisible to the player due to a loss of innovation for the story arcs in
software bug caused by the newer hardware. each game.
138 Assassin’s Creed, an Analysis

Assassin’s Creed, an Analysis, Fig. 3 Assassin’s Creed and as Ezio Auditore da Firenze during 1476–1499 cen-
II was released in 2009. You play as Desmond Miles again tered around Florence, Italy

Assassin’s Creed, an Analysis, Fig. 4 Assassin’s Creed Miles again and as Ezio Auditore da Firenze during 1499–
Brotherhood was released in 2010. You play as Desmond 1507 centered around Rome, Italy

Evolution nine sequels to Assassin’s Creed offer

only minor improvements in graphics and
The changes in each game in the series over audio.
the years have been minor. Comparing to the The following series of pictures showcases the
first Assassin’s Creed on Xbox One S, the evolution from 2007 to 2018:
Assassin’s Creed, an Analysis 139

Assassin’s Creed, an Analysis, Fig. 5 Assassin’s Creed Miles again and as Ezio Auditore da Firenze during 1511–
Revelations was released in 2011. You play as Desmond 1512 Constantinople, Istanbul

Assassin’s Creed, an Analysis, Fig. 6 Assassin’s Creed who is a Native American turned assassin during the
III was released in 2012. You play as Desmond Miles again American Revolutionary War between the years 1765 and
and as Ratonhnhakén:ton also known as Connor Kenway, 1777
140 Assassin’s Creed, an Analysis

Assassin’s Creed, an Analysis, Fig. 7 Assassin’s Creed modern story and as Edward Kenway during the Golden
IV: Black Flag was released in 2013. You play as an Era of Piracy between 1715 and 1722 in the Caribbean
unnamed employee of the company Abstergo during the

Assassin’s Creed, an Analysis, Fig. 8 Assassin’s Creed of consoles. You play as an unnamed Abstergo employee
Rouge released in 2014 was the final game designed for the and as Shay Patrick Cormac who is an Assassin turned into
Xbox 360 and PlayStation 3 generation of consoles but a Templar (the assassin’s rivals) during the 7 Years’ War
was later given a remastered edition for the new generation from 1756 to 1776 in the British colonies of North America
Assassin’s Creed, an Analysis 141

Assassin’s Creed, an Analysis, Fig. 9 Assassin’s Creed a player of the Abstergo game called “Helix” and as Arno
Unity released in 2014 was the first Assassin’s Creed game Dorian who is an assassin during the French 252 Revolu-
created and released only for the new generation of con- tion in 1789 and who goes to the Thermidorian Reaction in
soles. It was also the first to incorporate a cooperative 1794 in Paris, France
multiplayer in the story. You play as the “Initiate” who is

Assassin’s Creed, an Analysis, Fig. 10 Assassin’s play as the “Initiate” who has fully bought into the Assas-
Creed Syndicate released in 2015 is the first Assassin’s sin Brotherhood and is working for them in the present
Creed to include two main characters that are played time, and during the past, you play as twins Jacob and Evie
simultaneously in the memory segments of the gameplay Frye during 1868 Victorian Era London, England
to include the series’ first playable female character. You
142 Assassin’s Creed, an Analysis

Assassin’s Creed, an Analysis, Fig. 11 Assassin’s Division during the present-day story, and as a Medjay
Creed Origins was released in 2017. You play as Layla named Bayek during 49–43 BC Ptolemaic, Egypt
Hassan, a researcher at Abstergo’s Historical Research

Assassin’s Creed, an Analysis, Fig. 12 Assassin’s fic tional Peloponnesian War between Athens and Sparta.
Creed Odyssey was released in 2018. The game takes You play as a male or female mercenary who fights for both
place in Ancient Greece in the year 431 BCE during the sides

1. Assassin’s Creed was released in 2007. multiple locations in and around

You play as Desmond Miles in the Jerusalem.
modern story and as Altaïr Ibn-La’Ahad 2. Assassin’s Creed II was released in 2009. You
who lives during the Third Crusade in play as Desmond Miles again and as Ezio
Asset Creation 143

Auditore da Firenze during 1476–1499 cen- the Assassin Brotherhood and is working for
tered around Florence, Italy. them in the present time, and during the past,
3. Assassin’s Creed Brotherhood was released you play as twins Jacob and Evie Frye during A
in 2010. You play as Desmond Miles again 1868 Victorian Era London, England.
and as Ezio Auditore da Firenze during 10. Assassin’s Creed Origins was released in
1499–1507 centered around Rome, Italy. 2017. You play as Layla Hassan, a researcher
4. Assassin’s Creed Revelations was released in at Abstergo’s Historical Research Division
2011. You play as Desmond Miles again and during the present-day story, and as a Medjay
as Ezio Auditore da Firenze during named Bayek during 49–43 BC Ptolemaic,
1511–1512 Constantinople, Istanbul. Egypt.
5. Assassin’s Creed III was released in 2012. 11. Assassin’s Creed Odyssey was released in
You play as Desmond Miles again and as 2018. The game takes place in Ancient Greece
Ratonhnhakén:ton also known as Connor in the year 431 BCE during the fictional Pelo-
Kenway, who is a Native American turned ponnesian War between Athens and Sparta.
assassin during the American Revolutionary You play as a male or female mercenary who
War between the years 1765 and 1777. fights for both sides (Juba 2018).
6. Assassin’s Creed IV: Black Flag was released
in 2013. You play as an unnamed employee of
the company Abstergo during the modern Cross-References
story and as Edward Kenway during the
Golden Era of Piracy between 1715 and ▶ Multiplayer Games
1722 in the Caribbean. ▶ Video games
7. Assassin’s Creed Rouge released in 2014 was
the final game designed for the Xbox 360 and
PlayStation 3 generation of consoles but was References
later given a remastered edition for the new
generation of consoles. You play as an For a game series that spans 11 games between 2007 and
2018 for various gaming platforms, there are countless
unnamed Abstergo employee and as Shay
articles, magazines, blog posts, and online articles that
Patrick Cormac who is an Assassin turned talk excessively about this series: from IGN to The
into a Templar (the assassin’s rivals) during Game Informer Magazine published by Game Stop.
the 7 Years’ War from 1756 to 1776 in the This series has been in the spotlight and is still going
strong in spite of the “dark times” of the original Assas-
British colonies of North America.
sin’s Creed and the failure of Assassin’s Creed Unity
8. Assassin’s Creed Unity released in 2014 was due to rushed game development. This game series
the first Assassin’s Creed game created and continues to be at the forefront of the RPG arena due
released only for the new generation of con- to extraordinary writing and storylines along with
above average graphics and annual title releases from
soles. It was also the first to incorporate a
Ubisoft. Some of the outstanding references are:
cooperative multiplayer in the story. You Juba, J.: Assassin’s creed odyssey. https://www.gamei
play as the “Initiate” who is a player of the nformer.com/review/assassins-creed-odyssey/fighting-
Abstergo game called “Helix” and as Arno for-glory (2018)
Sliva, M.: Assassin’s creed 4: black flag review. https://
Dorian who is an assassin during the French
www.ign.com/articles/2013/10/29/assassins-creed-4-
Revolution in 1789 and who goes to the black-flag-review (2013)
Thermidorian Reaction in 1794 in Paris,
France.
9. Assassin’s Creed Syndicate released in 2015 is
the first Assassin’s Creed to include two main Asset Creation
characters that are played simultaneously in the
memory segments of the gameplay to include ▶ 3D Game Asset Generation of Historical Archi-
the series’ first playable female character. You tecture Through Photogrammetry
play as the “Initiate” who has fully bought into ▶ Planetary Generation in Games
144 Assisted Healthcare

that prevent their equitable use of sites and site

Assisted Healthcare information.

▶ Virtual Human for Assisted Healthcare: Appli-

cation and Technology Introduction

Because bots exist, there will always be a need for

processes to protect sites and users from their
Audio and Facial Recognition interference (Fanelle et al. 2020). Even so, visual
CAPTCHAs for Visually impairments require the installation of alternatives
Impaired Users to standard visual CAPTCHAs, which include
character-based and audio-based options. At the
Mansour Alqarni1, Fangyi Yu1, Rupendra Raavi2 same time, these alterative CAPTCHA designs
and Mahadeo Sukhai3,4 may not be as effective, easy to use, or timely as
1
Ontario Tech University, Oshawa, ON, Canada standard visual CAPTCHAs, based on these via-
2
Faculty of Business and IT, Ontario Tech bility requirements, and the fact that audio
University, Oshawa, ON, Canada CAPTCHAs in particular are easily hacked
3
ARIA Team, Canadian National Institute for the (Meutzner et al. 2015). This means that
Blind, Toronto, ON, Canada CAPTCHAs for visually impaired users can lack
4
CNIB Foundation, Kingston, ON, Canada both accessibility and security.
Our contributions in this paper are as the
following:
Synonyms
• We have used advanced biometrics for facial
Accessibility; Biometric and facial recognition; recognition model in a real-time face detection.
CAPTCHA; Cyber security; Machine learning; • We proposed to use our impeded model for
Visual impairment Webcam to improve the Audio CAPTCHAs.
• Finally, we are presenting the state-of-the-art
accuracy for both facial recognition detection
Definitions time and CAPTCHAs improvements for Visu-
ally Impaired Users.
CAPTCHA (Completely Automated Public
Turing test to tell Computers and Humans Apart)
is a security check software placed on websites as Using Biometric for CAPTCHAS
a means to allow to differentiate users from bots
upon login (Jiang and Dogan 2015). Security is The availability of alternatives to standard visual
needed to ensure that bots cannot enter protected CAPTCHAs is limited. Biometric designs may be
environments, such as those associated with seen to be an easy solution to the issue (Noorjahan
games or purchases, in order to protect human 2019), but the reality is that most visually
users and companies alike from the impact of impaired users engage in web-based sites through
unwanted interference of all kinds, including specialized equipment which may or may not be
financial, social, or otherwise (Fanelle et al. able to accommodate Fig. 1 provide some of the
2020). Accessibility is a measure of equity in common equipment used for fingerprint, and they
access to sites and site information for people compared to the proposed method of using the
with a range of disabilities (Berton et al. 2020). actual key on the keyboard as an authenticated
The article refers to visually impaired users as key. That would be as an option in the near term
inclusive of both legally blind users as well as for using speech recognition instated of Audio
those who have sight differences or deficiencies CAPTCHAs (Fanelle et al. 2020). This means
Audio and Facial Recognition CAPTCHAs for Visually Impaired Users 145

10

8
A
6 Usibility
Repets
4
Time
2

Audio and Facial Recognition CAPTCHAs for Visually 0

Impaired Users, Fig. 1 Type of biometrics is available RAC ABAC
for fingerprint
Audio and Facial Recognition CAPTCHAs for Visually
Impaired Users, Fig. 2 Percentage of users for in the
that options that can take advantage of standard survey for each category of disability
hardware, i.e., hardware that is used most fre-
quently by visually impaired users, need to be Audio and Facial Recognition CAPTCHAs for Visually
prioritized (Alnfiai 2020). Whether standard hard- Impaired Users, Table 1 The characterized data of the
ware changes in the future or not, CAPTCHA participants were collected during our evaluation
options need to reflect what users prefer in order Age Number of participants Visually impaired%
to meet accessibility and equity expectations 6–15 10 50
(Berton et al. 2020). 16–35 42 40
35+ 11 80

Alternative CAPTCHAS for Visually

Impaired Users Figure 2 shows the results of our comparison
between Regular Audio CAPTCHAs (RAC) and
Given these needs, a number of alternative our proposal Advanced Biometrics Audio
CAPTCHAs for visually impaired users have CAPTCHA (ABAC). Table 1 presents overall
been tested in scholarly research in recent years data of the participants during the research.
and a number of factors that ought to be considered
have been prioritized. The most ideal CAPTCHA
is one that works for both sighted and visually Our Proposal and Implementation
impaired user with little to no differentiation, so
that potential programming gaps are addressed We currently use the deep learning-based facial rec-
(Alnfiai 2020; Shirali-Shahreza et al. 2013). This ognition model called tiny face detection, built on the
means that solutions need to be equally seamless top of the single-shot shot detector (SSD) (Shafiee
for all users, and decrease extant time gaps for et al. 2017). Our deep learning model is made with
people with visual challenges (Alnfiai 2020; the help of JavaScript programming, which helps us
Fanelle et al. 2020). However, this is not always deploy the model over the web. After the model was
feasible, and most solutions need to work with built, we deployed the model over the web.
existing web audio accessibility inputs such as The main motive of our model is to detect if
Google reCAPTCHAs (Berton et al. 2020). there is a human in front of the computer or not.
CAPTCHAs that allow a user to swipe on a So once the visually impaired person clicks the
phone or handset when they hear a sound are likely identify button, camera will open to determine if
to be successful (Alnfiai 2020). Having users reply the person is present or not based on the facial
using their own voices can be successful if security recognition model. If a person is identified, then
gaps are addressed by recombining CAPTCHA the captcha is a success and will go further if the
words in a control sequence that is connected camera fails to recognize the person. Then it says
with reCAPTCHA technologies (Jain et al. 2019). captcha is failed and ask’s the user to try again. In
146 Audio and Facial Recognition CAPTCHAs for Visually Impaired Users

Audio and Facial Recognition CAPTCHAs for Visually Impaired Users, Fig. 3 Webcam model face recognition
CAPTACHA

Audio and Facial Recognition CAPTCHAs for Visually Impaired Users, Fig. 4 Testing the accuracy of our model

Fig. 3 the architecture design for our proposed Limitations

Webcam CAPTCHA.
This is the architecture of our deep learning There are certain limitations with our face detec-
model where initially, a person initiates the web- tion model, such as our model is currently not able
cam, and then the feature extraction is done for to distinguish between a photo and a normal
every frame. Then it classifies whether based on person. In Fig. 5 the model will detect a face
the features extracted whether to allow or deny the from picture where we intended to enhance the
access. The model will detect the face in a box for security to provide a real-time face detection with
higher accuracy. Figure 4 shows defrent types of audio. The user will be able to hear the instruc-
experiments we ran for testing our CAPTCHA. tions for head adjustments, head rotation right,
left, up, and down will eliminate any security
issues we are facing from robots.
Proposed Audio CAPTCHA Assistant

In our CAPTCHA, we have designed an audio Conclusion and Future Work

assistant not only to provide speech to users but
also provide guidance for visually impaired users. For visually impaired users, alterative CAPTCHA
Instructions will be provided on how to best place designs may not be as effective, easy to use, or
the face for any visually impaired users on front of timely as standard visual CAPTCHAs. They may
the webcam as shown in Fig. 4. also lack security, especially if they are audio-based
Audio Description 147

References

Alnfiai, M.: A novel design of audio CAPTCHA for visu-

ally impaired users. Int. J. Commun. Netw. Inf. Secur. A
12(2), 168–179 (2020)
Berton, R., Gaggi, O., Kolasinska, A., Palazzi, C.E.,
Quadrio, G.: Are captchas preventing robotic intrusion
or accessibility for impaired users? In: 2020 IEEE 17th
Annual Consumer Communications & Networking
Conference (CCNC), pp. 1–6. IEEE (2020). https://
www.math.unipd.it/~gaggi/doc/ads20.pdf
Fanelle, V., Karimi, S., Shah, A., Subramanian, B., Das, S.:
Blind and human: exploring more usable audio
{CAPTCHA} designs. In: Sixteenth Symposium on
Usable Privacy and Security (SOUPS 2020),
pp. 111–125 (2020)
Jain, M., Tripathi, R., Bhansali, I., Kumar, P.: Automatic
generation and evaluation of usable and secure audio
ReCAPTCHA. In: The 21st International ACM
SIGACCESS Conference on Computers and Accessi-
bility, pp. 355–366 (2019, October)
Jiang, N., Dogan, H.: A gesture-based captcha design
supporting mobile devices. In: Proceedings of the 2015
British HCI Conference, pp. 202–207 (2015, July)
Audio and Facial Recognition CAPTCHAs for Visually Meutzner, H., Gupta, S., Kolossa, D.: Constructing
Impaired Users, Fig. 5 Our model detects the photo as a secure audio captchas by exploiting differences
person and allows the captcha between humans and machines. In: Proceedings of
the 33rd Annual ACM Conference on Human Fac-
tors in Computing Systems, pp. 2335–2338 (2015,
April)
alternatives. Nonetheless, new CAPTCHA options Noorjahan, M.: A bio metric based approach for using
need to align to standard web protocols as well as captcha-to enhance accessibility for the visually
standard hardware for visually impaired users and impaired. Disabil. Rehabil. Assist. Technol. 15(2),
153–156 (2019)
web browser accessibility requirements. This sug-
Shafiee, M.J., Chywl, B., Li, F., Wong, A.: Fast YOLO: a
gests that extra time is often the compromise that is fast you only look once system for real-time embedded
made in designing CAPTCHA options that meet object detection in video. arXiv preprint
security and accessibility needs. Visually impaired arXiv:1709.05943 (2017)
Shirali-Shahreza, S., Penn, G., Balakrishnan, R., Ganjali,
users must spend extra time to get through a Y.: Seesay and hearsay captcha for mobile interaction.
CAPTCHA sequence in comparison with standard In: Proceedings of the SIGCHI Conference on Human
users, which means that a truly equitable solution is Factors in Computing Systems, pp. 2147–2156 (2013,
lacking. Improvements to alternative CAPTCHA April)
designs need to be created in order to ensure equity
in the future, to avoid current pitfalls that create
barriers to true security, accessibility, effectively
and usability by introducing a real-time face detec- Audio Collage
tion CAPTCHAs.
▶ Dynamic Music Generation: Audio Analysis-
Synthesis Methods
Cross-References

▶ Accessibility of Virtual Reality for Persons

with Disabilities Audio Description
▶ Secure Gaming: Cheat-Resistant Protocols and
Game History Validation ▶ Visual Accessibility in Computer Games
148 Audio Design

invert this logic by either making graphics sec-

Audio Design ondary to the audio or not using graphics at all,
focusing on the sound instead. These are known
▶ Sonic Interactions in Virtual Environments as audiogames: games “fully playable by reliance
on their audio output” (Karhulahti 2015).
For some game designers, stripping away the
visual component of the game can be a way to
Audio Game experiment with the medium. Dark Room Sex
Game (2008), a party rhythm game playable
▶ Audiogame with the keyboard or Nintendo Wii’s Wiimote
controller, is one example, focusing on audio
and haptic feedback to simulate sexual intercourse
in a dark room (Oldenburg 2013).
Audio Mosaicing More significant is audiogames’ accessibility
to visually impaired players, extending the reach
▶ Dynamic Music Generation: Audio Analysis- of digital games beyond their traditionally
Synthesis Methods assumed audience of sighted people (Friberg
and Gärdenfors 2004). Online communities
such as Audiogames.net have become hubs for
visually impaired gamers, providing game
Audiogame reviews and publicizing events and competi-
tions. The Game List section on Audiogames.
Mikhail Fiadotau net currently lists over 600 titles (most of them
Centre of Excellence in Media Innovation and developed by hobbyists), though the list is not
Digital Culture (MEDIT), Tallinn University, extensive and primarily focuses on English-
Tallinn, Estonia language games.

Synonyms History

Audio game; Audio-only game Atari’s Touch Me (1974), an arcade and handheld
sequence memorization game, is considered to be
a precursor to modern audiogames (Karhulahti
Definitions 2015). While it did not exclusively rely on audio
output (the sequence was duplicated as audio
An audiogame is a digital game that relies solely tones and light-emitted diodes lighting up),
or primarily on audio output. Most audiogames Touch Me could be played without using visual
can be played by both visually impaired and cues, although that would require the player to
sighted players. memorize the correspondence between the tones
and the buttons first.
Audiogames for the personal computer became
Introduction a possibility following the 1984 release of the
original Apple Macintosh, which bundled with
While most digital games today heavily rely on speech synthesis software MacInTalk. This
sound in order to communicate information to the enabled the Macintosh port of popular text adven-
player, the role of audio in games is generally less ture Colossal Cave Adventure to support voice
significant than that of graphics (Collins and output, making the game accessible to visually
Kapralos 2012). Some digital games, however, impaired users. Other text adventure games
Audiogame 149

followed, making the genre popular with the visu- construct sequences for the player to
ally impaired gamer community. memorize.
The first notable commercial audiogame was 2. Rhythm and duration: rhythm games such as A
Real Sound: Kaze no Riguretto (1997) developed PaRappa the Rapper (1996) and Beatmania
by WARP Corp. for the Sega Saturn (Matsuo et al. (1997) emphasize the timing of pressing the
2016). It was conceived by game designer Kenji controls, and some subsequent games such as
Eno after he learned of people with a visual dis- 2010’s Rock Band 3 can process actual music
ability who played his previous, visually intensive parts performed on a MIDI-enabled musical
games by the ear. In terms of gameplay, Real instrument.
Sound was an interactive radio drama where the 3. Spatial cues: such games as Papa Sangre and
player influenced the outcome of the story by Blindside focus on exploring invisible environ-
choosing between several dialogue options at cer- ments using sonic feedback such as
tain points within the game. While not a commer- reverberation.
cial success, Real Sound set an important 4. Narrative: Real Sound: Kaze no Riguretto and
precedent by being an audio-only game released the BBC’s The Inspection Chamber
by a major game publisher. (2017) rely on voice narration to communicate
The 2010s have seen a rising number of the story and use player choices as their core
audiogames appear: Papa Sangre (2010), Sound mechanic. Notably, The Inspection Chamber is
Swallower (2011), The Nightjar (2011), Blindside controlled by voice input from the player
(2012), Audio Defence: Zombie Arena (2014), instead of conventional controllers such as the
and many others (Beksa et al. 2015). The best- keyboard.
known of them, Papa Sangre (2010), is a survival
horror game which takes place in a pitch-black It should be noted that many of the games
dungeon, where sound positioning and reverbera- mentioned above are not audiogames themselves;
tions provide the sole cues for navigation. Many they just contain elements which can be used to
contemporary audiogames, including Papa construct one. These cues can also serve different
Sangre, are designed for tablets and smartphones; functions: since audio output becomes the main
they also rely on touch controls, raising the ques- vehicle for communicating information to the
tion of their accessibility to players with a visual player, game sounds in audiogames need to fulfill
disability. a variety of roles, many of which are usually
Some mobile audiogames, on the other hand, associated with game graphics. Friberg and
combine sound output with Braille writing to cre- Gärdenfors (2004), for example, divide sounds
ate a more accessible for visually impaired players in audiogames into avatar, object, character, orna-
while also helping them master Braille (Araújo mental, and instruction sounds.
et al. 2016). These games, however, may not be Various combinations of these types of cues
easily playable by sighted players who lack and sound functions can be used to create a wide
Braille literacy. range of game dynamics, from role-playing games
(Matsuo et al. 2016) to shooting games (Beksa
et al. 2015) to platformers (Oren 2007) to puzzle
Design games such as Preludeamals (2016).

On a basic level, there are several types of audio

cues audiogames can rely on: Conclusion

1. Tone/pitch: for example, Atari’s Touch Me While for most of their history, audiogames
(1974) and Simon (1978), another memory enjoyed little popular recognition, their recent
game inspired by it, used sounds of a different resurgence, embodied by such titles as Papa
pitch to identify the four key elements used to Sangre and The Inspection Chamber, suggests
150 Audio-Only Game

that they do yet have a potential for mainstream

acceptance. This leaves hope for the reimagining Augmented and Gamified
of digital games as a medium that is accessible to Lives
sighted and non-sighted players alike.
Róbert Tóth and Marianna Zichar
Faculty of Informatics, University of Debrecen,
References Debrecen, Hungary

Araújo, M.C., Silva, A.R., Darin, T.G., de Castro, E.L.,

Andrade, R., de Lima, E.T., Sánchez, J., de C Filho, J.
A., Viana, W.: Design and usability of a braille-based
Synonyms
mobile audiogame environment. In: Proceedings of the
31st Annual ACM Symposium on Applied Computing, Augmented reality; Gamification; Serious games
pp. 232–238. ACM, New York (2016)
Beksa, J., Fizek, S., Carter, P.: Audio games: investigation of
the potential through prototype development. In:
Biswas, P., Duarte, C., Langdon, P., Almeida, L. (eds.) Definition
A Multimodal End-2-End Approach to Accessible Com-
puting, pp. 211–224. Springer, London (2015) Gamification uses game-based mechanics, aes-
Collins, K., Kapralos, B.: Beyond the screen: what we can
learn about game design from audio-based games. In:
thetics, and game thinking to engage people, moti-
5th International Conference on Computer Games, Mul- vate action, promote learning, and solve problems
timedia and Allied Technology, Bali, Indonesia (2012) (Kapp 2012). In this context, augmented reality
Friberg, J., Gärdenfors, D.: Audio games: new perspectives (AR) as a popular technology can extend its func-
on game audio. In: Proceedings of the 2004 ACM
SIGCHI International Conference on Advances in
tionality with the use of a camera and various
Computer Entertainment Technology, pp. 148–154. sensors of a device to provide information with
ACM, New York (2004) computer-generated methods.
Karhulahti, V.-M.: Defining the videogame. Game Stud.
15, (2015)
Matsuo, M., Sakajiri, M., Miura, T., Onishi, J., Ono, T.:
Accessible action RPG for visually impaired gamers: Introduction
development of a game software and a development
environment for the visually impaired. Trans. Virtual Nowadays, it is easier than ever to reach people in
Real. Soc. Jpn. 21, 303–312 (2016). https://doi.org/10.
18974/tvrsj.21.2_303
digital platforms, while the spread of COVID-19 is
Oldenburg, A.: Sonic mechanics: audio as gameplay. also forced the experts to find new methods to engage
Game Stud. 13, (2013) and support their students, clients, or participants in
Oren, M.A.: Speed sonic across the span: building a plat- any field of life, but in a virtual format. This entry
form audio game. In: CHI’07 Extended Abstracts on
Human Factors in Computing Systems,
aims to provide a short conclusion about the key
pp. 2231–2236. ACM, New York (2007) concepts and the actual trends in the field of
gamification and augmented reality. After giving the
definitions, examples of already existing, well-
known, or innovative applications and research pro-
jects will be described to illustrate the most important
Audio-Only Game takeaways about these fields. Gamification can be
also applied to offline activities (Kermek 2019); how-
▶ Audiogame ever, the entry primarily focuses on a digital context.

Gamification and Serious Games

Auditory Impairment
Gamification is a popular and widely used method
▶ Computer Games for People with Disability to engage people and motivate the participants of
Augmented and Gamified Lives 151

an in-person or digital activity. As its name sug- a game, but with a well-defined purpose and meth-
gests, the method was derived from the world of odology. Thus, the development of a serious game
games, and it can be efficiently used in many differs from the core definition of gamification A
different areas of our life, especially in the busi- (see Fig. 1).
ness sector and in the education. Of course, the There are many examples of successfully
design and the implementation depend on the gamified applications or successfully developed
actual use-case; thus, gamification has many sim- serious games, but there are many anti-patterns, of
ilar, but different definitions. One of the most course. The most relevant keynote is that the
acceptable and abstract – thus use-case indepen- elements of games should be used as motivation
dent – definitions is Karl M. Kapp’s one: and to provide positive feedback; thus, the goals
Gamification is using game-based mechanics, should be precisely designed, and the activity of
aesthetics, and game thinking to engage people, the user should be controlled, while negative
motivate action, promote learning, and solve rewards and other feedback elements should be
problems (Kapp 2012). Gamification can be omitted. Thus, the success massively depends on
applied to any existing process, and the elements the relationships between the different elements;
can engage the participants with their effect. How- thus, great gamified processes and serious games
ever, the original process and the functionality implement a bunch of elements. The set of
must be retained; gamification can only be used implemented elements can depend on a specified
to extend and improve them. On the other hand, user type for which an application is being devel-
the design and implementation of a serious game oped (Ponick and Stuckenholz 2019). As in appli-
can be used for the same purposes, but the method cation development, various patterns can be
clearly differs from the definition of gamification. discovered in gamified applications and the most
In the context of a serious game, a brand-new common elements can be easily collected and
activity is being designed with the experience of divided into groups:

Augmented and Gamified Lives, Fig. 1 A serious game which aim is to provide an environment for practicing the
built-in functions of spreadsheet software products
152 Augmented and Gamified Lives

• Activities – Elements that offer activities and surfaces. More interaction (such as scaling,
control the user such as missions, challenges, moving the objects) can be achieved easily.
and goals. • Cloud anchors – This function is the extended
• Feedback – Elements that give positive feed- and improved version of the instant placement
back to users such as scoreboards, progress in which multiple users can interact in the
bars, charts, experimental points (XPs), and extended reality.
experience levels. • Augmented images – This feature depends on a
• Rewards – Completions of activities should be computer vision algorithm which simulta-
rewarded with badges, experimental points neously processes the picture of the camera
(XPs), lifelines, privileges, and domain- and searches for features of specified images.
specific items. With the use of this feature, users can instanti-
• Sociality and interaction – Users have profiles ate 3D objects by scanning images.
and can see each other’s results in scoreboards, • Augmented faces – This function is widely
charts. They can compete in duels or special used in mobile operation systems, messaging
interactive activities. applications and other gadgets. The face of the
• Time – This element always appears in a com- user is being detected with a computer vision
bination with another elements and is an algorithm and various objects can be put on
important factor in the control of the user. them. After that, each movement of the user
• Design and modality – Both in digital and will be tracked by the sensors and the 3D
in-person contexts, all the use of other ele- objects will move in a synchronized way.
ments depends on the user experience
(UX) which is based on the design and the
Concurrent Applications
modality.
It is clear that motivation is one of the most
Augmented Reality important factors in the success of a learning
process. In the last decades, many platforms
Augmented reality (AR) applications use the and educators have introduced gamification to
camera and various sensors of a device engage their students. The most well-known lan-
(typically cell phones, tablets) to enhance the guage learning environment is one of the best
visual experience with computer-generated per- examples for a carefully designed and precisely
ceptual information. In the last decade, a great implemented gamified process (Huynh et al.
number of AR applications have been developed 2018). It offers the basic mechanism of an online
in many different areas. AR is a preferred alter- learning process – topics, exercises, notes, tests –
native to virtual reality (VR) because no addi- but also engages its users with the friendly inter-
tional devices (only the built-in ones) are face, social items, rewards, and time elements.
required. Also, the latest mobile devices are Gamified elements can be discovered in loyalty
designed with several additional cameras and programs of various companies, such as airlines
sensors to improve the quality of their AR func- (https://www.miles-and-more.com) and restau-
tions. The first well-known applications with AR rants (Tyson 2019). The goal in the first approach
functions were games, but the AR can be used in is quite simple however, complex systems can be
education, tourism, military, or for social pur- designed on the collected data as well. Users
poses. According to a widely used library, AR create their profiles, so their transactions can be
functions can be divided into four main groups: associated and tracked. Users collect points,
achieve milestones or levels, and get privileges
• Instant placement – This function scans for flat or discounts, while companies can execute data
surfaces, then the users can instantiate 3D mining algorithms to analyze their behavior. The
objects and place them on the detected well-designed loyalty programs contain the most
Augmented and Gamified Lives 153

common elements of gamification. Fitness appli- sociality to improve the user experience of the
cations also use gamified elements to motivate the game. However, complex games and game stores
users in the hard process of diets. In the most also offer gamified elements to promote their A
popular applications, users can choose their products, engage and reward their users. On the
goals, while the applications can automatically other hand, well-known applications are starting
offer goals, rewards, reminders, and give feed- to motivate their users to contribute by applying a
back in the form of various statistics. However, few elements such as points, levels, and badges
in the last years, gamification elements are being (see Fig. 2).
applied to games that are distributed in various The first well-known AR application was
formats. Games with very simple logic and user released in 2016 in the form of a game which
experience implement the most common elements was based on a Japanese anime and its fictive
such as leaderboards, XPs, missions, time, and characters (Rapp et al. 2018). The characters

Augmented and Gamified Lives, Fig. 2 Features of an application (Tóth et al. 2021) which offers Mental Cutting Test
exercises to its users. Gamified elements can be implemented without changing the basic logic of the application

Augmented and Gamified Lives, Fig. 3 The visual cookbook uses an AR function to support its users who live with
disabilities
154 Augmented and Gamified Lives

Augmented and Gamified Lives, Fig. 4 The AR function of the previously mentioned learning application offers an
opportunity to instantiate the model of the actual exercise and interact with it

were placed on selected geographical coordinates built on augmented functions. Thus, children can
and users had to find and catch them with their interact with their sets gaining more experience.
mobile phones. Many other games are still be The largest tech-companies are also engaged in
developed based on AR functions. People living the mission to develop and distribute AR features
with disabilities can be supported with AR appli- to enhance the use of their products and provide
cations designed to give some help for them in an tools for the developer community. The navigation
activity that is trivial for others. Six European of a well-known software (Ranieri 2020) now
universities and a nongovernment organization offers an AR function to provide useful informa-
run a project that focuses on how emerging tech- tion in an enhanced format, with the use of instant
nologies (including AR) can be used in develop- placement function. On the other hand, a new
ing such applications (Barbas et al. 2020). search function was also introduced with the use
Figure 3 shows a visual cookbook application of which users can directly instantiate 3D models
which is already published, where augmented and interact with them (https://support.google.
image function is used to instantiate the ingredi- com/websearch/answer/9817187). Also, they pro-
ents of dishes (Pevec 2020). vide a well-detailed and flexible used API for
Of course, AR is a perfect tool to be applied in developers who want to develop their AR applica-
education as well. Teaching and learning of courses tions, based on the same core. Multiple platforms
where demonstrations are of high importance, can and devices are supported, that were produced in
be supported by AR applications (Nørgård et al. the last few years with the latest versions of oper-
2018; Salinas et al. 2015; Grunewald Nichele and ation systems. On the other hand, sample projects
do Nascimento 2017). Thus, more students can and the source codes of the development kits are
access the extra information and online activities published in public repositories (https://github.
can also be supported (see Fig. 4). A well-known com/google-ar/arcore-android-sdk).
toy manufacturer company has also started to
develop innovations with the use of augmented
functions. Their applications are based on the aug- Cross-References
mented image function, offering the opportunity to
instantiate the models of their current brochures ▶ Augmented Reality
(https://play.google.com/store/apps/details? ▶ Challenge-Based Learning in a Serious Global
id¼com.lego.catalogue.global), while brand new Game
product themes were also introduced massively ▶ Gamification and Serious Games
Augmented Learning Experience for School Education 155

▶ History of Augmented Reality

▶ Mixed Reality Augmented Learning
Experience for School A
Education
References
Satyaki Roy, Pratiti Sarkar and Surojit Dey
Barbas, M.P., Babic, J., Car, Z., Gace, I., Lovrek, I., Design Programme, Indian Institute of
Topolovac, I., Vdovic, H., Zilak, M., Gourves, A.:
Technology, Kanpur, Uttar Pradesh, India
Analysis of emerging technologies for improving
social inclusion of people with disabilities. University
of Zagreb Faculty of Electrical Engineering and Com-
puting (2020) Synonyms
Grunewald Nichele, A., do Nascimento, G.: Augmented
reality in teaching chemistry. In: INTED2017 Proceed-
ings. IATED (2017) Augmented reality; Gamification; K-12 education
Huynh, D., Zuo, L., Iida, H.: An assessment of game
elements in language-learning platform duolingo.
In: 2018 4th International Conference on Com- Definition
puter and Information Sciences (ICCOINS). IEEE
(2018)
Kapp, K.M.: The Gamification of Learning and Instruc- Augmented reality is one of the technologies that
tion: Game-Based Methods and Strategies for Training works in real time to overlay virtual computer-
and Education. Wiley, Nashville (2012) generated graphics on to the real-world environ-
Kermek, D.: Gamification in web programming course –
ment so as to provide the users with open-ended
are three years enough to see positive effects and justify
efforts to implement it. In: INTED2019 Proceedings. experience.
IATED (2019) Gamification involves the use of several ele-
Nørgård, C., O’Neill, L., Nielsen, K.G., Juul, S.H., Chem- ments of game design in a context that does not
nitz, J.: Learning anatomy with augmented reality. In: actually belong to a game. This provides a
EDULEARN18 Proceedings. IATED (2018)
Pevec, D.: LeARn to Cook – AR-based application for playful experience to users and keeps them
people with disabilities, http://sociallab.fer.hr/ engaged and encouraged to attain the defined
archives/learn-to-cook-ar-based-application-for- goals.
people-with-disabilities/ (2020) The K-12 education is a combination of pri-
Ponick, E., Stuckenholz, A.: Impact of user types in
gamified learning environments. In: INTED2019 Pro- mary and secondary education in schools which
ceedings. IATED (2019) starts from Kindergarten, till 12th standard.
Ranieri, M.: A new sense of direction with Live View,
https://blog.google/products/maps/new-sense-
direction-live-view/ (2020)
Rapp, D., Niebling, F., Latoschik, M.E.: The impact of Introduction
pokémon go and why it’s not about augmented reality –
results from a qualitative survey. In: 2018 10th Interna- The years are progressing with the development in
tional Conference on Virtual Worlds and Games for technology. Augmented reality (AR), being one
Serious Applications (VS-Games). pp. 1–2. IEEE
(2018) such technology, has been a key research area
Salinas, P., Quintero, E., Ramirez, P., Mendívil, E.: Foster- from quite a long time and has been applied in
ing spatial visualization through augmented reality in various domains. In the field of education and
calculus learning. In: 2015 ASEE Annual Conference learning, several works have been done to
and Exposition Proceedings. ASEE Conferences (2015)
Tóth, R., Zichar, M., Hoffmann, M.: Improving and mea- improve the understanding and knowledge using
suring spatial skills with augmented reality and AR. Gamification has been another means by
gamification. In: Advances in Intelligent Systems and which the students are made to be engaged with
Computing, pp. 755–764. Springer International Pub- the experiential learning and stay motivated. With
lishing, Cham (2021)
Tyson, L.: Whopper detour: App gamification wins with the advancements in the Internet usage and devel-
location, https://bluedot.io/blog/whopper-detour-app- opment of portable electronic devices, these two
gamification/ (2019) technologies are now reachable to everyone.
156 Augmented Learning Experience for School Education

These means are being introduced in various sec- Augmented Reality in School Education
tors including education and learning to enhance
the existing experiences. In the current education The technology of augmented reality (AR) helps
system, a lot of data is supposed to be memorized in interactively combining the virtual and real
for writing the examination. This requires a need world in real time. AR systems are considered to
to reduce the cognitive load of the students with work with the combination of various factors. It
the developing technologies to make it easy for overlays the virtual information layer on top of the
them to memorize and at the same time enjoy the real surroundings instantly. The contents of the
learning experience. computer-generated virtual overlay have to be
In this entry, a case study on geography subject interactive and meaningful in the real environ-
has been discussed to show a gamified way of ment (Azuma et al. 2001). In the course of many
learning maps using augmented reality for K-12 years, the importance and application of AR has
education system in schools to deliver better been realized in several domains.
understanding, easy memorizing, and incremental AR helps to visualize the virtual objects as real.
learning. One can see the live 3D representation of the
computer-generated virtual imagery data. It can
thus help students to interact with the 3D spatial
K-12 Education System in Schools structures and enhance the learning skills. In the
field of school education, the applications of AR
With the advent of technology, K-12 education was realized in some of the subject areas that
system in numerous schools is gaining importance included astronomy, chemistry, biology, mathe-
and various methods of imparting knowledge matics and geometry education, and physics
have been adopted to make it more engaging for (Lee 2012). Through the evolution of gadgets
the students and enjoy the learning experience. In like personal computers, smartphones, tablets,
this form of education system in schools, a lot of and other electronic innovations, AR has become
attention is given to individual learning so that the more powerful in becoming a prominent
students can on their own explore and develop technology.
their learning and understanding skills. Teachers The AR technology in school education can be
are introducing various interactive activities as the observed through mediums like AR books,
mode of learning for the students to provide them objects modeling, AR games, discovery-based
an easier way of better understanding about what training, and skills training (Yuen et al. 2011).
they are studying. AR can potentially increase the collaboration
In many schools, blackboards are now getting and understanding between teachers and students
replaced by projector screens to provide digital in classroom. While teaching in classes, many a
learning experience. Numerous schools are times the teachers and students are not on the same
adopting such smart-class solutions where mental platform. Teachers try to make the students
teachers project the digital content related to text- visualize some concepts in 3D but are unaware if
book on the screens for the students to better the students are able to do it or not. With the
understand the concepts. These digital contents application of AR, now the 3D representation of
include 2D or 3D animations, audios, videos, the concepts is shown to the students in class
etc. Students are also able to practice and learn at which highly aids their understanding. Classroom
home on their own using various online modules. learning using AR gives a very engaging experi-
With the introduction of more evolving technolo- ence where students can look at things from every
gies and digital platforms in the K-12 education angle and this turns to be very useful in explaining
system in schools, it is believed that there will be many things from textbook like the solar system,
growth and development in the learning capabil- atomic structures in chemistry, etc. AR books are
ities of students. also an important innovation which bridges the
Augmented Learning Experience for School Education 157

gap between the digital and physical world. The traditional method of classroom teaching
Because books are printed media, they have in schools can be improved by providing the stu-
some limitations like poor printing quality, dull- dents with interactive visualization of 3D objects A
ness, etc. But the conversion of 2D contents of the to give them the real-life interactive experience to
textbook into 3D objects in real environment understand better and enable easy retention of the
helps to enhance the learning experience. Such shown content.
tools have also been developed which can convert
2D sketches into 3D objects, using which students
can develop 3D models in space. A lot of appli- Gamification in Education
cations have been developed around games in AR
as it is believed that games can play a key role in Gamification helps a student to stay engaged and
keeping students motivated and can also help motivated with the learning methods. It is now
them in grasping the concepts easily. There are emerging as a useful tool in various domains
some AR applications which during the field trips including school education. Viriyapong et al.
to museums, historical spots, caves, etc. display developed a game that helps in escalating the
the overlaid information on the real environment, learning experience of mathematics for high
making it an interactive learning experience. school students using game methods. With the
Thus, the means of providing the AR experience focus on plotting linear and polynomial functions
may vary with the mode of teaching. on graph using coefficients, the game motivates
the user by providing points for each correct
answer based on the time taken to answer. The
Memorization Using Augmented Reality difficulty level increases at each level of the game
(Viriyapong et al. 2014). Another game is being
In the school education system, there are multiple designed that uses spaced repetition with a
subjects in each standard. Further, each subject gamified experience to enhance STEM (science,
has multiple chapters which the students have to technology, engineering, and mathematics) edu-
understand, learn and recall at the time of exami- cation for students of K-12 education system (Yeh
nation. Thus, the students find it a challenging et al. 2016). Many such gamified experiences
task as there is a lot of data that they have to have been provided across various subjects for
memorize. the students to stay motivated toward learning
Several researches have been done to find the new concepts.
relationship between AR and memorization for For the implementation of gaming in AR, sev-
providing a better scope in school education. eral elements can be used such as points, levels,
Among the recent works, the use of an AR appli- badges, experience points, leaderboards, chal-
cation has been observed to display visual direc- lenges (Nah et al. 2014). The introduction of
tives at the location of some drawers for easy these elements brings up engagement, participa-
memorization of the visually displayed objects tion, motivation, enjoyment, performance, recog-
(Fujimoto et al. 2012). In some cases, marker- nition, status, sense of accomplishment, and sense
based AR has been used to make users learn of achievement in any game. But it becomes dif-
foreign language by displaying 3D objects and ficult to make the whole game, especially an edu-
their corresponding spelling and pronunciation cational game interesting and motivating by using
(Beder 2012). Among the various techniques to only one of these elements. Best results are
memorize, memory palace memorization method obtained when these elements are used in combi-
where a user makes use of the familiar spaces to nations where they are interlinked with each other.
construct their respective memory palace to mem- These elements are to be incorporated in the game
orize has been applied and observed using AR in such a way that it connects the student with the
(Rosello et al. 2016). gameplay and their fellow players.
158 Augmented Learning Experience for School Education

Combining Augmented Reality and students to perform well in their examinations.

Gamification in Education The system focuses on dividing the whole content
in appropriate modules with required amount of
Various attempts have been made for combining data and takes few examinations based on that.
AR and gamification in the field of school educa- However, this kind of school education lacks
tion. AR system can be either marker-based or enough motivation for the students to stay
marker-less (Johnson et al. 2010). In marker- engaged with the subjects. In many subjects that
based AR system, the camera of the device first are related to real life, students find it difficult to
identifies the marker related to any image or asso- correlate because the pictorial contents they have
ciated data, then matches with the related content in their books are in two dimensions whereas in
in the database, and finally superimposes the real environment they are in three dimensions.
related 3D visualization and/or audio onto the Among the various subjects that the students
real world. Marker-less AR on the other hand have to learn, geography is one such course that
does not require any former information about involves a lot of information to understand as well
the real world environment. It instead involves as memorize for writing examinations. In this
the use of GPS and compass of the AR device subject, it becomes difficult to visualize things in
connected through Internet and applies image rec- three dimensions, and it turns out to be even more
ognition methods to track certain defined features difficult for the teachers to teach and make the
in the live environment and overlay the virtual students understand. In every standard, starting
content onto it. from class 5, the students are made familiar with
Virtual laboratories in the real world are being different type of maps, the way to read them and
introduced using “markers” to provide interactive the way to find latitude and longitude of a location
hands-on learning and experimental experience in the map using atlas or vice-versa. Since, there is
for learning and education (Eleftheria et al. a lot of textbook data and numerous locations that
2013). In this, a science AR book was created the students have to keep in mind, this results in
along with gamification. The knowledge and students losing the interest in that subject and
learning was tested through several challenges as studying it just for the sake of examinations.
part of game. Thus, the design of an AR-based game for learn-
In another AR game, the use of GPS, compass, ing the maps easily and with interest has been
and Internet in a hand-held device, enables stu- further discussed.
dents to discover why the aliens entered the earth The two methods, marker-based and marker-
(Dunleavy et al. 2009). In the AR environment less AR are being extensively used and explored
created, the students interact with the computer- in various works on school education. However,
generated characters, digital objects and solve in this case study the use of marker-based
related puzzles based on math, science, and approach has been emphasized upon for the
language arts. existing geography textbooks.
Thus, the feature of memorization using aug- In the design, augmented reality has been used
mented reality and motivation using gamification to make the contents more interactive. Additional
can be combined to further enhance the learning printed AR markers are provided corresponding
capabilities of students and make it an enjoyable to each map in the textbook. On scanning the
experience for them. markers with the AR device, users can see the
maps in 3D. In the maps which show terrains,
wind movements, etc., users are able to see those
Case Study: Applying AR and visual contents in three dimension. This applica-
Gamification in Geography tion is based on two modules. First is to help the
users memorize the locations as well as under-
In the existing K-12 education system in schools, stand the related concepts and the second one is
there is a lot of data that needs to be memorized by
Augmented Learning Experience for School Education 159

to help them practice to mark the locations on content. The short time limit challenge in the
the map. game helps in getting motivated to challenge
In order to memorize the locations on the their own retention capability. The user can get A
map, a 3D visualization of the map is aug- to see their performance on the progress meter for
mented. If it is the map of wildlife reserves, for every time they play which ultimately help to
example, when the AR device is brought on the develop confidence (Fig. 3).
2D map, the 3D visualization of the The progress meter of every player is reflected
corresponding wildlife reserve on the map gets on their profile. When they feel they are confident
projected. This is followed by popping of names in a topic they can raise challenges with their
and related brief information in a sequential friends. The application persuades the user to
manner. The users have the option to revise the take up challenges with the friends of their class
locations as many times as needed. Once the or locality as on winning a challenge the user
names of the wildlife reserves with their loca- earns twice the number of points as compared to
tions are memorized, one can proceed to play normal practice mode. This encourages the user to
and score (Figs. 1, 2, and 3). take more number of challenges.
In the game, within 20 s, one has to mark a list The game consists of scores, levels, and badges
of 12 locations. The user has the option to choose for the users to stay motivated in learning the
the type of map to practice with the help of the maps quickly on their own with enjoyment. For
marker. The maximum number of locations every 1000 points a level is attained and the user
marked within the time limit provides better gets a badge for that which shows their profi-
scores. The property of spaced repetition has ciency. The user also gets badges for certain
been used in this game for easy retention of the defined tasks. For example, on scoring 100 points
on 3 different maps the user gets a new badge.
Similarly, the user can get 25 different badges on
performing such defined tasks.

Limitations of Augmented Reality

AR has significantly helped to improve the learn-

ing experience in classrooms but some issues of
using AR have also been observed, most of which
are the hardware and software problems
(Dunleavy et al. 2009). Cases have been seen
where students find it challenging to learn the
new technology and respond to the corresponding
activities in timely manner, thus developing some
cognitive stress. In the classroom experience, at
times it becomes difficult for teachers to manage
the student groups involved in doing AR activi-
ties. The students also tend to lose track of their
surroundings while performing the AR-based
activities as they get totally involved in that. On
the development end, at times it becomes difficult
to create and deploy highly rendered 3D objects.
This thus leads to making low-poly objects
Augmented Learning Experience for School Educa- instead, which might provide less clarity in visu-
tion, Fig. 1 Demonstration of prototype alization. However, these challenges may be
160 Augmented Learning Experience for School Education

Augmented Learning Experience for School Education, Fig. 2 3D visualization of topographical map in AR

worked upon with the passage of time to provide a visualization in the real world. There are various
better AR experience. means like AR books, AR games, object model-
ing, and many more, by which this augmented
reality–based teaching and learning is done in
Conclusion school education. To keep the students motivated
with their learning skills, AR-based games are
This entry discussed the scope of augmented real- also introduced. These games have combination
ity in school education to enhance the learning of game elements like points, levels, badges, lead-
experience of students. Several researches are erboard, etc. to encourage their participation in the
being done to improvise the traditional form of game-based learning (Deterding et al. 2011). AR
textbook-based teaching so as to reduce rote and gamification together makes it interactive and
learning effort made by the students. Augmented motivating for the students to learn easily and can
reality instead provides an engaging interactive help to reduce the memorizing efforts being put by
experience to share knowledge through the students. In the entry, one such case study on
Augmented Learning Experience for School Education 161

References

Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S.,
MacIntyre, B. Recent advances in augmented reality. A
IEEE Comput. Graph. Appl. 21(6), 34–47 (2001).
IEEE Computer Society
Beder, P.: Language learning via an android augmented
reality system [Internet] [Dissertation]. 2012. Available
from: http://urn.kb.se/resolve?urn=urn:nbn:se:bth-5982
Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From
game design elements to gamefulness: defining
gamification. In: 15th international academic MindTrek
conference: Envisioning future media environments
2011, pp. 9–15. ACM (2011)
Dunleavy, M., Dede, C., Mitchell, R.: Affordances and
limitations of immersive participatory augmented real-
ity simulations for teaching and learning. J. Sci.
Educ. Technol. 18(1), 7–22 (2009)
Eleftheria, C.A., Charikleia, P., Iason, C.G., Athanasios, T.,
Dimitrios, T.: An innovative augmented reality educa-
tional platform using Gamification to enhance lifelong
learning and cultural education. In: Information, Intel-
ligence, Systems and Applications (IISA), 2013 Fourth
International Conference, pp. 1–5. IEEE (2013)
Fujimoto, Y., Yamamoto, G., Taketomi, T., Miyazaki, J.,
Kato, H.: Relationship between features of augmented
reality and user memorization. In: Mixed and Aug-
mented Reality (ISMAR), 2012 I.E. International Sym-
posium, pp. 279–280. IEEE (2012)
Johnson, L., Levine, A., Smith, R., Stone, S.: Simple
Augmented Reality. The 2010 Horizon Report,
pp. 21–24. The New Media Consortium, Austin (2010)
Lee, K.: Augmented reality in education and training.
TechTrends. 56(2), 13–21 (2012)
Nah, F.F.H., Zeng, Q., Telaprolu, V.R., Ayyappa, A.P.,
Eschenbrenner, B.: Gamification of education: a review
Augmented Learning Experience for School Educa-
of literature. In: International Conference on HCI in
tion, Fig. 3 Opponents competing in a stipulated time
Business, pp. 401–409. Springer International Publish-
limit
ing 2014
Rosello, O., Exposito, M. , Maes, P.: NeverMind: using
augmented reality for memorization. In: 29th Annual
Symposium on User Interface Software and Technol-
learning the maps in geography using AR-based ogy, pp. 215–216, ACM, 2016
quiz game is discussed. However, there are certain Viriyapong, R., Yosyingyong, P., Nakrang, J. , Harfield, A.:
A case study in applying gamification techniques on
limitations while using AR in school education.
mobile technology for mathematics high school stu-
These include software and hardware limitations, dents to learn polynomial functions (2014). The Elev-
deployment issues with highly rendered 3D enth International Conference on eLearning for
objects, skills of the teachers and students to Knowledge-Based Society, 12–13 December 2014,
Thailand
adapt to the new technology, managing the stu-
Yeh, M.K.C., Toshtzar, A., Guertin, L., Yan, Y.: Using
dent groups, intensive involvement and avoiding spaced repetition and gamification to enhance K-12
the current surrounding happenings, etc. These student science literacy with on-demand mobile short
challenges may be overcome with the evolution reads. In: Frontiers in Education Conference (FIE),
pp. 1–4. IEEE 2016
of the technology reachability over time. Thus, the
Yuen S, Yaoyuneyong G and Johnson E.: Augmented
learning can be made to be a more enjoyable reality: an overview and five directions for AR in edu-
experience with augmented reality getting intro- cation. J. Educ. Technol. Dev. Exch. 4(1), 119–140
duced in the school education. (2011)
162 Augmented Reality

▶ Tracking Techniques in Augmented Reality for

Augmented Reality Handheld Interfaces
▶ Virtual Reality and Robotics
▶ 3D Puzzle Games in Extended Reality Environ-
ments
▶ 3D Selection Techniques for Distant Object
Interaction in Augmented Reality Augmented Reality Displays
▶ Artificial Reality Continuum
▶ Augmented and Gamified Lives ▶ Enhanced Visualization by Augmented Reality
▶ Augmented Learning Experience for School
Education
▶ Augmented Reality Ludo Board Game with
Q-Learning on Handheld Augmented Reality
▶ Conceptual Model of Mobile Augmented Real- Entertainment: Taking
ity for Cultural Heritage Gaming Out of the Box
▶ History of Augmented Reality
▶ Interacting with a Fully Simulated Self- G. Stewart Von Itzstein, Mark Billinghurst,
Balancing Bipedal Character in Augmented and Ross T. Smith and Bruce H. Thomas
Virtual Reality School of Information Technology and
▶ Interaction with Mobile Augmented Reality Mathematical Sciences, University of South
Environments Australia, Mawson Lakes, SA, Australia
▶ Interactive Augmented Reality Pop-Up Book
with Natural Gesture Interaction for Handheld
▶ Interactive Augmented Reality to Support Edu- Synonyms
cation
▶ Immersive Visualizations Using Augmented Augmented reality gaming; Spatial augmented
Reality and Virtual Reality reality gaming; Virtual reality entertainment; Vir-
▶ Integrating Virtual Reality and Augmented tual reality gaming
Reality into Advertising Campaigns: History,
Technology, and Future Trends
▶ Key Early Verticals: Challenges and Limita- Definition
tions in Implementation of Augmented Reality
▶ Live Texture Mapping in Handheld Augmented Augmented reality (AR) is technology that seam-
Reality Coloring Book lessly adds virtual imagery over a view of the real
▶ Making Virtual Reality (VR) Accessible for world, so that it can be seen and interacted with in
People with Disabilities real time. Azuma says that an AR system is one
▶ Mixed Reality that has three key defining characteristics
▶ Mixed Reality and Immersive Data Visualiza- (Sutherland 1968): (1) It combines real and virtual
tion content, (2) It is interactive in real time, and (3) It
▶ Mixed Reality, Gamified Presence, and Story- is registered in 3D.
telling for Virtual Museums
▶ Object Manipulation Using Real Hand Gesture
for Augmented Reality Interior Design Introduction
▶ Potential of Augmented Reality for Intelligent
Transportation Systems AR can be used in many possible application
▶ Shadow Shooter: All-Around Game with domains, such as in medicine to show virtual
e-Yumi 3D anatomical structures in a real patient’s body
Augmented Reality Entertainment: Taking Gaming Out of the Box 163

Augmented Reality Entertainment: Taking Gaming Out of the Box, Fig. 1 Examples of AR experiences (a)
medical, (b) marketing, and (c) architectural

(Fig. 1a), marketing where computer graphics on Milgram’s virtuality continuum (Milgram and
appear to pop out of a magazine (Fig. 1b), and Kishino 1994) that arranges computer interfaces
architecture where unbuilt virtual buildings can according to how much of the user experience is
appear in the real world (Fig. 1c). replaced by computer graphics (Fig. 2). At the left
In this chapter, an overview of using AR for end of this continuum is the real world with no
gaming and entertainment is provided, one of the virtual information, while VR is at the right end,
most popular application areas. There are many where the user’s entire experience is computer
possible AR entertainment applications. For generated. Mixed reality is everything in between,
example, the Pokémon Go mobile phone game including the overlay of virtual content in the real
has an AR element that allows people to see world (AR) and adding elements of the real world
virtual Pokémon to appear in the live camera into VR (virtual reality).
view, seemingly inhabiting the real world. In this
case, Pokémon Go satisfies Azuma’s three AR
criteria: the virtual Pokémon appears in the real Augmented Reality Technology
world, the user can interact with them, and they
appear fixed in space. Azuma’s definition of AR provides guidance as to
AR is complimentary to virtual reality (VR), the technology required for AR entertainment
technology that tries to fully immerse a person in a systems. To combine real and virtual content,
computer-generated environment. While AR uses there needs to be some display technology where
virtual information to enhance a user’s interaction both can be seen at the same time. To allow the
in the real world, VR separates people from the user to interact with the virtual content, there
real world entirely. AR and VR can both be placed needs to input technology. Finally, to create the
164 Augmented Reality Entertainment: Taking Gaming Out of the Box

Augmented Reality Mixed Reality (MR)

Entertainment: Taking
Gaming Out of the Box,
Fig. 2 Milgram’s virtuality Real Augmented Augmented Virtual
continuum and mixed Environment Reality (AR) Virtuality (AV) Environment
reality technologies

Augmented Reality Entertainment: Taking Gaming Out of the Box, Fig. 3 (a) Head-mounted AR, (b) Handheld
AR, (c) Spatial AR

illusion that the virtual content is fixed in space usually touch-enabled and so many hand-held AR
tracking technology needs to be used to track the systems use touch input. Some HMD-based AR
user’s viewpoint. systems have additional sensors that track the
There are a wide variety of AR displays, users’ hands and so enable gesture interaction. In
which can be categorized into three types this case, it can be very natural for users to reach
depending where the display is placed: head- out and grab the virtual content that appears in
attached, hand-held, and spatial (Davis et al. space directly in front of them. Finally, for spatial
2007) (Fig. 3). Head-attached displays include AR the experience is often at a larger scale and so
head-mounted displays (HMD), head-mounted cameras can track a user’s full body motion to
projectors, and retinal projection systems which enable them to interact with the AR content.
shine images directly into the eyes. AR HMDs The final requirement for an AR system is to
can be optical see-through (OST) which use have some form of user viewpoint tracking so that
transparent optics to enable the user to see virtual the AR content can appear fixed in space while the
images directly overlaid on the real world, or user moves around. There are many different
video see-through (VST) which displays com- tracking approaches that can be used; however,
puter graphics overlaid on a video of the real for indoor-based systems, computer vision
world. Hand-held AR displays are the most com- methods are the most popular. Marker-based com-
mon and include devices such as mobile phones, puter vision tracking enables a user to point a
tablets, and small projectors. Finally, spatial dis- hand-held or head-worn camera at a known
plays are those that include using a fixed projec- printed image, have the camera pose calculated
tor to shine virtual content onto a real object, relative to the image, and then draw graphics
such as a car, and are often used as public dis- overlaid on the marker (Fig. 4a). More recently,
plays for multiple people. software has been developed that supports simul-
In terms of interactivity, there are a wide vari- taneous localization and mapping (SLAM) and
ety of different input devices that can be used, can calculate the camera’s position without know-
often depending on the type of AR display. For ing anything about the users’ environment
example, for hand-held AR the display screen is (Fig. 4b). Outdoors, it is common to use GPS-
Augmented Reality Entertainment: Taking Gaming Out of the Box 165

Augmented Reality Entertainment: Taking Gaming Out of the Box, Fig. 4 (a) Marker tracking, (b) SLAM
Tracking Points (Courtesy of Huidong Bai), (c) SLAM Axis Overlay (Courtesy of Huidong Bai)

based systems combined with computer vision or a pilot with an AR view of instruments while
inertial compass input to find the user’s location. flying.
Most mobile phones have cameras, GPS, and In the late 1980s, several universities began
inertial sensors in them and so have all the tech- conducting research in the AR space, developing
nology needed for AR tracking. various fundamental technologies. For example,
at the University of North Carolina researchers
created novel AR displays and tracking technol-
History of Augmented Reality Gaming ogy. The University of Toronto had scientists
exploring input devices and the human factors of
The history of AR gaming applications began AR, while people at the University of Washington
nearly fifty years ago. In 1968, Ivan Sutherland and Columbia University explored collaborative
created the first complete AR system with a very and mobile AR systems, among other topics.
bulky see-through head-mounted display (HMD) By the mid-1990s, enough basic research had
connected to a mechanical head tracker, showing been completed that people could begin to explore
very simple graphics (Sutherland 1968). Follow- the application space. The first complete mobile
ing on from this the US military experimented AR systems were developed using backpack com-
with HMDs and heads up displays (HUDs) that puters, and people began to explore how AR
were designed to overlay vehicle instrument infor- could be applied in medicine, education, engi-
mation over the real world, for example providing neering, and other areas. For example, engineers
166 Augmented Reality Entertainment: Taking Gaming Out of the Box

Augmented Reality Entertainment: Taking Gaming Out of the Box, Fig. 5 (a) ARQuake. Player view, (b)
ARQuake. Backpack system

at Boeing were using AR to help with creating the objects, and to the game players. This created a
wire bundles for aircraft. Boeing engineer Tom new type of location-based gaming experiences.
Caudell published the first research paper with the The employment of a table top as an AR gam-
term “augmented reality” in it (Caudell and Mizell ing surface provides for several interesting user
1992), and this led to many other companies interface opportunities for HMD, hand-held or
developing industrial applications. table top-projected AR entertainment applica-
By the early 2000s, the underlying technology tions. The games may range from extensions to
was reaching maturity, and so the first wide- traditional non-computer-based games, such as
spread entertainment applications of AR began AR Chinese checkers (Cooper et al. 2004) to
to appear. In 2002, ARQuake (Thomas et al. new robotic interaction games such as Augmented
2002) was an early port of a popular commercial coliseum (Kojima et al. 2006). There are many
game to a backpack AR system. The Quake game advantages to playing games on a computer rather
was modified to use a real university campus as than on a physical board, such as the ability to
the setting for an invasion of monster’s players introduce animation and other multimedia presen-
run around the real campus seeing virtual mon- tations. The animation can do more than add
sters moving around in the campus setting excitement to the gameplay, it can also help the
(Fig. 5). This became a forerunner of many players learn the game and understand invalid
mobile AR games that were released a decade moves.
later on mobile phones. In 2006, Fox Sports began to implement AR
Around the same time, there were several into their sports broadcasts. Initially, limited to
examples of indoor AR gaming experiences NFL football, it later moved to other sporting
developed. This includes the indoor mobile AR codes. AR was used to show the player stats and
game, MIND-WARPING (Starner et al. 2000). scores as a virtual billboard that occupied an
This employed an HMD and allowed users to unused section of the field. More recently, BT
physically walk/run throughout a floor of a build- Sports in Europe has introduced AR technology
ing fighting with virtual monsters that were con- for describing soccer plays (Fig. 6), and AR
trolled by a second remote player on an interactive enhancements were shown in the most recent
table top display. The mobile nature of the equip- Olympics.
ment allowed for AR information to be attached to In 2007, Sony released the Eye of Judgement
physical locations in the space, to tracked physical game. This used a camera with the PlayStation
Augmented Reality Entertainment: Taking Gaming Out of the Box 167

Augmented Reality
Entertainment: Taking
Gaming Out of the Box,
Fig. 6 AR soccer strategy A
(BT Sport)

console to show virtual monsters on real cards. It current AR entertainment applications that show
was a face to face multiplayer game where players how AR can be used in many different entertain-
looked at a TV screen to see the AR view. The ment domains. Initially, AR mobile apps were
game used the camera to show a live view of the simple and used for information browsing appli-
cards overlaid with virtual game characters. As cations such as a bank app that places virtual tags
players put the cards next to each other, the ani- in the real world showing were ATM machines
mated characters would battle each other. It sold are. Recently, these AR apps have branched into
over 300,000 copies, making it the most success- location-based gaming with the most well-
ful AR entertainment experience at the time. known being Pokémon Go (2017) (and its pre-
Around the same time processing and graphics decessor Ingress). These games allow players to
power on mobile phones had grown to the point travel to a location and do battle with virtual
where they could run mobile AR applications. enemies which are overlayed on the real world.
The first AR computer vision application for a Being the fastest mobile app ever to achieve
mobile phone appeared in 2004 (Mohring et al. more than $1 Billion USD in revenue, Pokémon
2004), followed soon after by AR Tennis, the first Go shows the huge potential that combining
collaborative AR game on a mobile phone location-based gaming, AR and a well-known
(Henrysson et al. 2005). In AR, Tennis players brand can have.
sat across from each other playing an AR tennis Another popular category for mobile AR is
game using their phones as real rackets, viewing interactive books and coloring experiences.
virtual content attached to a real marker between Beginning with the MagicBook (Billinghurst
them. By 2007, phones had integrated GPS and et al. 2001), there are many AR book experiences.
compass sensors in them enabling the first mobile Typically, these allow users to view a normal book
phone outdoor AR experiences, such as viewing through their mobile phone and see the pages
virtual buildings in the real world. This opened come to life with interactive AR content (Fig. 7).
AR to the masses and allowed developers to build These are particularly popular for children’s
a whole new class of entertainment applications. stories or educational books. A variation of this
is the AR coloring application which allows chil-
dren to color pages and then see AR content
Current Augmented Reality appear out of the page with their colors on the
Entertainment Applications content. This was popularized by the company
Quiver (Website 2017), but has since been devel-
Following from the early beginnings reviewed in oped by dozens of other companies around the
the previous section, there are a wide range of world. Children really enjoy being able to see
168 Augmented Reality Entertainment: Taking Gaming Out of the Box

Augmented Reality Entertainment: Taking Gaming Out of the Box, Fig. 7 (a) AR book, (b) Quiver AR
coloring book

Augmented Reality
Entertainment: Taking
Gaming Out of the Box,
Fig. 8 Microsoft
RoomAlive (Courtesy
Microsoft Research)

virtual characters that they have colored come to space. In this way, graphics can be seamlessly
life. projected on the walls and the user can move
Spatial AR technology has been used to create freely around the space and interact with the vir-
room scale AR entertainment applications. For tual content (Fig. 8). For example, a hand-held
example, Microsoft’s Room Alive spatial AR sys- gun prop can be used to shoot the virtual creatures
tem turns living rooms into interactive AR play in the living room.
spaces (Jones et al. 2014). Depth-based cameras There are few widely available AR HMDs, but
are employed to map the real environment and to the Microsoft HoloLens (Website 2017) shows
capture gestures and movements of people in the the type of AR HMD-based entertainment
Augmented Reality Entertainment: Taking Gaming Out of the Box 169

Augmented Reality
Entertainment: Taking
Gaming Out of the Box,
Fig. 9 Augmented reality A
gaming. Microsoft
HoloLens RoboRaid

experiences that are currently possible. The more by turning their books into an AR play
HoloLens contains depth sensors that can map space.
the user’s surroundings, allowing an application Another area of endeavor is allowing users to
to make use of the surrounding environment and grasp and feel the texture of virtual objects in
annotate 3D objects over that environment. AR. Devices such as Massie’s Phantom (Massie
A good example of this is RoboRaid (Fig. 9:) and Salisbury 1994) provide a mechanical arm held
where the user’s room is attacked by aliens that by a user’s fingers allowing computer-controlled
break through the real walls and try to defeat the haptic feedback to be programmed. Simon et al.
user. This is achieved by the HoloLens creating a (2014) have been investigating the use of a tech-
3D map of the room and interpreting where the nology called layer jamming to provide haptic
physical walls are in the room them allowing the sensations with the low-profile actuator that is
game to apply game oriented textures to the walls. encapsulated in a mitten. This technology has
both benefits in that it is low profile but disadvan-
tages as it requires a vacuum source to activate the
Conclusion actuators providing the stimulus.
Spatial augmented reality (SAR) (Fig. 3c) has a
As can be seen from the previous sections, AR has lot of potential for gaming and entertainment as it
a long history and entertainment is one of the is not a solitary experience. Because the projec-
widespread uses today. AR is pervading industry tions are independent of the user, many users can
with new start-ups appearing almost weekly, participate in a shared experience; most other AR
while older established corporations, such as approaches do not directly support shared
Apple, have been buying up these start-ups almost gameplay due to the nature of the projection,
as fast as they appear. It’s clear that they see a e.g., helmet or phone. SAR games could include
future in the technology. shared gaming environment (AKA Star Treks’
One of the reasons for this growing popularity Holodeck) where people share the experience.
of AR is that it offers a fundamentally different However, due to the added a complexity, cost
entertainment experience. Non-AR experiences and calibration challenges it has not been adopted
typically focus the user on a screen (mobile at the same rate as other forms of augmented
games) or completely immerses them into a dig- reality. Cooperative and competitive games will
ital space (VR). On the other hand, AR expands become very playable once the problems of cost
the users’ interaction with the real world, and calibration are solved satisfactorily.
whether encouraging people to walk outside to Overall, AR provides unique entertainment
find Pokémon’s, or motivating children to read options not available with other types of
170 Augmented Reality for Human-Robot Interaction in Industry

digital media. However, in many ways devel- Quiver Website: 12/3/2017. Available from: http://www.
opers are still just beginning to explore the quivervision.com/ (2017)
Simon, T.M., Smith, R.T., Thomas, B.H.: Wearable
full potential of AR for entertainment. With Jamming Mitten for Virtual Environment Haptics, in
new devices and research outputs becoming ISWC’14. Seattle. ACM New York, NY, USA (2014)
available, future AR systems will be a signif- Starner, T., et al.: MIND-WARPING: towards creating a
icant advance over those available now. There compelling collaborative augmented reality game. In:
Proceedings of the 5th International Conference on
is a bright future ahead for AR entertainment Intelligent User Interfaces. ACM Press, New Orleans
experiences. (2000)
Sutherland, I.E.: A head-mounted three dimensional dis-
play. Proc. AFIPS. 68, 757–764 (1968)
Thomas, B., et al.: First person indoor/outdoor augmented
References reality application: ARQuake. In: Personal and Ubiq-
uitous Computing. Springer-Verlag London Ltd, UK
Billinghurst, M., Kato, H., Poupyrev, I.: The magicbook- (2002)
moving seamlessly between reality and virtuality. IEEE
Comput. Graph. Appl. 21(3), 6–8 (2001)
Caudell, T.P., Mizell, D.W.: Augmented reality: an appli-
cation of heads-up display technology to manual
manufacturing processes. In: System Sciences, 1992. Augmented Reality for
Proceedings of the Twenty-Fifth Hawaii International Human-Robot Interaction in
Conference (1992)
Cooper, N., et al.: Augmented reality chinese checkers. In: Industry
Proceedings of the 2004 ACM SIGCHI International
Conference on Advances in Computer Entertainment Federico Manuri, Francesco De Pace and
Technology, pp. 117–126. ACM Press, Singapore Andrea Sanna
(2004)
Davis, S.B., et al.: Smell Me: Engaging with an Interactive Dipartimento di Automatica e Informatica,
Olfactory Game. In: Bryan-Kinns N., Blanford A., Politecnico di Torino, Turin, Italy
Curzon P., Nigay L. (eds) People and Computers XX
— Engage. Springer, London (2007)
Henrysson, A., Billinghurst, M., Ollila, M.: Face to face
collaborative AR on mobile phones. In mixed and Synonyms
augmented reality. In: Fourth IEEE and ACM Interna-
tional Symposium, IEEE (2005) Collaborative robot augmented reality; HRI AR;
Hololens Website: 12/3/2017. Available from: https:// Industrial robot augmented reality
www.microsoft.com/microsoft-hololens/en-us (2017)
Jones, B., et al.: RoomAlive: magical experiences enabled
by scalable, adaptive projector-camera units. In: Pro-
ceedings of the 27th Annual ACM Symposium on User Definitions
Interface Software and Technology, pp. 637–644.
ACM, Honolulu (2014)
Kojima, M., et al.: Augmented coliseum: an augmented Augmented reality for human-robot interaction in
game environment with small vehicles. In: First IEEE industry is the usage of augmented reality tech-
International Workshop on Horizontal Interactive nologies for displaying computer-generated digi-
Human-Computer Systems. 2006, IEEE: TableTop tal contents, correctly aligned to real objects, in
2006, Adelaide (2006)
Massie, T.H., Salisbury, J.K.: The PHANToM Haptic inter- order to enhance and enrich the communication
face: a device for probing virtual objects. Dyn. Sys. interface of users, which operates robots in an
Control. 1(55), 295–301 (1994) industrial environment.
Milgram, P., Kishino, F.: A taxonomy of mixed reality
visual displays. IECE Trans. Inf. Syst. E77-D(12),
1321–1329 (1994)
Mohring, M., Lessig, C., Bimber, O.: Video see-through Introduction
AR on consumer cell-phones. In: 3rd IEEE/ACM Inter-
national Symposium on Mixed and Augmented Real- The industry domain has taken advantage of
ity. IEEE Computer Society (2004)
Pokemon Go website: 12/03/2017. Available from: http:// augmented reality (AR) since its origin
www.pokemongo.com/ (2017) (Sutherland 1968): technicians are often
Augmented Reality for Human-Robot Interaction in Industry 171

involved in complex assembly, repair, or main- Tracking System

tenance procedures, and they need to refer to The tracking system has the task to establish an
instruction manuals to complete their tasks absolute reference system, based on some features A
(Caudell and Mizell 1992). However, these related to the environment: this reference system
activities often require a high cognitive load is fundamental to properly display the augmented
due to a continuous switch of the attention reality content with respect to the user’s view
between the physical device and the paper man- and the real-world objects. The tracking system
ual (Henderson and Feiner 2007). AR can effi- is called object dependent if the reference system
ciently overcome this issue providing the same depends on the position of a real object: this kind
content in a digital form and properly displayed of systems adopts computer graphics algorithms
on the physical object involved in the task. to identify the object from frames of the environ-
In this entry, AR technologies adopted to visu- ment provided by a digital camera. Other, less
alize digital contents in industry, the different common solutions consist of inertial, mechanical,
kinds of robots involved in, and the most common and magnetic tracking systems or involve the
tasks which benefit of AR contents are discussed. usage of absolute references such as GPS coordi-
nates (Foxlin 2002).

Technologies Content Generator

The content generator has the task to compute the
Among human sensory inputs, sight, hearing, and graphical content to be displayed depending on
touch are currently the senses enhanced by an AR the coordinates provided by the tracking system
system through digital contents. However, since and the frames provided by the digital camera.
industrial environments may provide different
kinds of limitations depending on the task, such Combiner
as anti-noise headphones or gloves, the AR con- The combiner has the task to overlap the assets
tent is usually provided visually. Different devices to the user view. It acts in different ways according
can be adopted to display the AR content to the used AR paradigm.
depending on the task and the environment. Over- Optical see-through devices blend physical
all, an AR system is characterized by three blocks: and virtual objects holographically using transpar-
a tracking system, a content generator, and a ent mirrors and lenses (Fig. 1). These devices are
combiner. wearable and most of the time also mobile.

Augmented Reality for

Human-Robot
Interaction in Industry,
Fig. 1 Optical see-through
AR system
172 Augmented Reality for Human-Robot Interaction in Industry

Augmented Reality for

Human-Robot
Interaction in Industry,
Fig. 2 Handheld AR
system

(e.g., physical trackpad, joypad, or keyboard) are

required to be added to the system to provide a
proper interaction interface.
Handheld technologies include all the devices
that allow to display both the physical world,
recorded through a camera, and the AR content
on a display (Fig. 2). These devices are usually
mobile, e.g., smartphones and tablets, since the
user needs to freely move them in order to frame
the point of interest in the environment and expe-
rience the AR content. The most common disad-
vantages of handheld devices are two: firstly, the
need to use one or both hands to handle the
devices, thus making them unavailable to perform
tasks; secondly, the user may experience disorien-
tation for parallax effect due to the camera posi-
Augmented Reality for Human-Robot Interaction in tion with respect to the viewer’s true eye location.
Industry, Fig. 3 Projective AR system
Projective devices allow to directly display AR
contents over physical objects (Fig. 3). They do
not require special eyewear and accommodate the
Moreover, since it is possible to wear them like user’s eyes during focusing. Moreover, they can
glasses, the user’s hands are free, which is usually cover large surfaces for a wide field of view. In the
mandatory in industrial tasks. One disadvantage industrial domain, these devices are usually
of this kind of devices is the limited field of view adopted to display AR content on a big industrial
that may cause clipping of virtual images at the robot, such as robotic arms. Commonly, projec-
edges of the mirrors or lenses. Moreover, it is tion surfaces may vary from flat, plain colored
difficult to occlude a real object because their walls to complex scale models. The main limita-
light is always combined with the virtual image tion of this technology is that the AR content is
due to the lenses properties. Finally, other devices perceived as bidimensional instead of three-
Augmented Reality for Human-Robot Interaction in Industry 173

dimensional. Other common disadvantages com- and different AR contents are provided to the user.
prehend occlusion and the need for other input When used with robotic arm manipulators, tech-
devices for interaction. Moreover, projectors need nicians can benefit from AR systems to visualize A
to be calibrated each time the environment or the one or more of the following features: end-
distance to the projection surface changes. effector path, end-effector direction, object
(s) manipulated or involved in the task, workspace
involved in the robot task, forces applied by the
Robots end-effector, and faults that occur on the industrial
manipulator. When used with AGVs, the features
AR technologies are usually applied to human- visualized through AR usually consist of the path
robot interaction (HRI) with two categories of and workspace of the robot. In the following, each
industrial robots: robotic arm manipulators and feature will be introduced and explained.
automated guided vehicles (AGVs).
A robotic arm manipulator is defined as an Path
n-degree-of-freedom (nDoF) arm robot. It is Depending on the types of paths and trajectories,
composed of links connected by joints that are different AR systems can be adopted. Trajectories
controlled by using either DC or brushless electric can be divided in 2D paths and 3D paths. The first
motors. Their positions are sampled by means ones are usually visualized on 2D areas using
of encoders, and the joints’ velocities are projectors mounted directly on the industrial
measured with tachometers. Joints are divided manipulator or on appropriate supports placed
into two different categories: revolute and nearby the robotic arm. They normally consist of
prismatic. Revolute joints allow rotations along one or more connected lines of the same color.
one local axis (usually the z-axis), whereas pris- The 3D paths are visualized in the real environ-
matic joints allow translations along one local axis ment by means of wearable or handheld devices.
(usually the z-axis). Robotic arms are also It is possible to interact with both types of path
equipped by different custom tools depending on using some specific tracked devices to modify
the task to accomplish: these tools are positioned them: this allows users to change the trajectories
at the end of the kinematic chain and are called of the industrial manipulator through the interac-
end-effectors. tion with the augmented reality paths.
AGVs are vehicles that can move along When displaying the path of an AGV, since
predefined path and predetermined directions these robots freely move around all the environ-
automatically and autonomously, without human ment, the AR projection system is commonly
interference. They are usually equipped by placed directly on the mobile robots. Thus,
sensors that allow them to identify and eventually workers can work without wearing ad hoc AR
avoid obstacles along their path. Their primary devices. AGVs are equipped with projectors that
task is to transport equipment around a allow to visualize the AGV’s intentions directly
manufacturing facility. AGVs are equipped with on the floor of the facilities. Projectors are
automatic guiding, either electromagnetic or opti- mounted on the AGVs at different heights; there-
cal, they can follow a predefined path through fore the projected area varies. Data projected can
visual analysis of a familiar environment, or they represent the future path that the AGVs are going
can use vision system to understand their location to follow by means of arrows (Fig. 4) and lines or
in an unknown environment. only the occupied space (Matsumaru 2006,
Coovert et al. 2014, Chadalavada et al. 2015).

Tasks Direction
Direction features are useful to understand in real
Depending on the type of robot considered and its time the direction of the end-effector. These fea-
task, various types of AR systems are adopted, tures are commonly represented by 3D virtual
174 Augmented Reality for Human-Robot Interaction in Industry

Augmented Reality for Human-Robot Interaction in Industry, Fig. 4 Example of AR contents used to display the
AGV’s path (yellow arrows) and the area that it will occupy (in blue). (Reproduced from RoboCup 2013 (CC BY 2.0))

arrows placed in the position of the end-effector

(Fig. 5), and they are visualized in the real envi-
ronment using wearable or handheld devices
(Michalos et al. 2016).

Object Manipulated
Objects that are going to be manipulated by the
robotic arms can be highlighted using both 2D and
3D features. The 2D features are commonly
represented by means of icons or geometric planar
shapes projected directly on the object (Fig. 5). The
projectors are mounted directly on the industrial
manipulator or on appropriate supports placed
near the robotic arm. The 3D features are commonly
represented by 3D virtual replicas of the real objects
that are superimposed on the object manipulated by
the robotic arm (Akan and Çürüklü 2010).

Workspace
It is possible to identify two different workspaces: Augmented Reality for Human-Robot Interaction in
the first one is the workspace of a robot arm Industry, Fig. 5 Example of AR contents used to display
the object to be manipulated (in green), the direction of the
manipulator, defined as the set that comprehend end-effector (in blue), and the forces on it (three axes on the
all the positions it can reach; the second one is the end-effector). (Reproduced from RoboCup 2013 (CC BY
“collaborative workspace,” defined as the work- 2.0))
ing area in which both the human operator and the
industrial manipulator work together. base of the industrial manipulator. The diameter
The robot workspace is commonly visualized of the sphere can also vary, depending on the
using handheld or wearable devices. The operat- movements of the end-effector.
ing area of the industrial manipulator can be The second workspace can be visualized using
represented using a 3D sphere, centered in the optical see-trough (Makris et al. 2017) or
Augmented Reality for Human-Robot Interaction in Industry 175

projected AR systems (Vogel et al. 2011). Pro-

jectors are placed near the robotic arm, in an
elevated position, and the “collaborative A
workspace” is projected directly on the floor.
Optical see-trough systems are usually composed
by cameras, placed in the corners of the environ-
ment. Depending on the distance of the user from
the robot, at least two different areas are projected:
the furthest area that is considered the safest oper-
ating area for the human worker, and it is com-
monly colored green and the closest area that is
considered the most dangerous operating area for
the human worker, and it is commonly colored
red. When the human worker operates in the fur-
thest area, the robotic arm works normally. On the
other hand, when the human worker is in the
closest area, the robotic arm stops or slows down
its motion in order to avoid any possible damage
(Fig. 6).

Forces
Forces applied by the end-effector can be moni-
tored and visualized using wearable or handheld
devices. Forces can be represented using 3D vir-
tual vectors, applied on the tool center (Fig. 4). Augmented Reality for Human-Robot Interaction in
The force components in X, Y, and Z are Industry, Fig. 6 Example of AR contents used to display
workspace of the robotic arm, with different colors
displayed as well as the resulting vector. Further- depending on the safety for the user based on distance
more, the components are colored with different and arrows displaying the direction of the movement
colors, depending on the intensity of the force (in green) and an error state for one joint (in red).
(Mateo et al. 2014). (Reproduced from Humanrobo 2009 (CC BY-SA 3.0))

Faults using a specific 3D asset, superimposed on

When a fault occurs on an industrial manipulator the error’s location. Faults on the velocity sensors
that is working side by side with a human opera- are represented using 3D circular arrows placed
tor, stress and anxiety may increase in the worker near the robotic arm’s joints. These arrows
because he is not able to realize in real time which rotate as long as the velocity sensor is
is the cause of the error. Moreover, since it is not acquiring correct data from the motor encoder
possible to realize the cause of the fault immedi- (Fig. 6, green arrow). When a fault occurs on the
ately, the time and resources required to solve it velocity sensor, the arrows stop moving, and they
strongly increase. change their color to red to highlight the problem
There are at least four different categories (Fig. 6, red arrow). Faults on the actuation system
of faults that can be visualized using AR technol- can affect the joint’s motor. If an error occurs on it,
ogies: faults on the velocity sensor, faults on the a 3D model representing the motor is super-
actuation system, faults due to overloading imposed on the real motor. Moreover, it starts
problems, and faults caused by collision blinking to emphasize the location of the fault.
(De Pace et al. 2018). These typologies of errors Faults due to collision can be represented using a
can be visualized using both hand-handled and 3D sphere center at the base of the industrial
wearable devices. Each fault is represented manipulator. A collaborative manipulator is able
176 Augmented Reality for Human-Robot Interaction in Industry

to foresee collisions, and when it detects an unex- SIGRAD, vol. 52, pp. 55–58. Link¨oping University
pected object, it stops its movements to avoid the Electronic Press, Vsters (2010)
Caudell, T.P., Mizell, D.W.: Augmented reality: an appli-
collision. Human operators may not understand cation of heads-up display technology to manual
why the manipulator has stopped its movements, manufacturing processes. In: Proceedings of HICSS,
misunderstanding its actions. To avoid these mis- vol. 2, pp. 659–669. IEEE, Kauai (1992)
judgments, when the manipulator foresees the Chadalavada, R.T., Andreasson, H., Krug, R., Lilienthal,
A.J.: That’s on my mind! robot to human intention
collision, the 3D sphere starts blinking to high- communication through onboard projection on shared
light the intentions of the industrial robotic arm. floor space. In: Proceedings of ECMR, pp. 1–6. IEEE,
Finally, errors due to overloading problems sud- Lincolnm (2015)
denly stop the movements of the manipulator. Coovert, M.D., Lee, T., Shindev, I., Sun, Y.: Spatial aug-
mented reality as a method for a mobile robot to com-
When this type of fault occurs, a 3D anvil, along municate intended movement. Computers in Human
with a warning signal, is superimposed on the Behavior. 34, 241–248 (2014)
payload. De Pace, F., Manuri, F., Sanna, A., Zappia, D.: An aug-
mented interface to display industrial robot faults.
In: International Conference on Augmented Reality,
Virtual Reality and Computer Graphics, vol. 2,
pp. 403–421. Springer (2018)
Cross-References
Foxlin, E.: Motion Tracking Requirements and Technolo-
gies. Handbook of Virtual Environment Technology,
▶ Augmented Reality Entertainment: Taking vol. 8, pp. 163–210. Mahwah, N.J.; London: Lawrence
Gaming Out of the Box Erlbaum Associates (2002)
Henderson, S.J., Feiner, S.K.: Augmented reality for main-
▶ Augmented Reality for Maintenance
tenance and repair (armar). Technical report, DTIC
▶ Augmented Reality in Image-Guided Surgery document (2007)
▶ Conceptual Model of Mobile Augmented Real- Humanrobo.: TOSY Industrial Robot: Arm Robot
ity for Cultural Heritage by Humanrobo. https://commons.wikimedia.org/wiki/
File:TI_A620-30.JPG CC BY-SA 3.0 https://creativec
▶ Integrating Virtual Reality and Augmented
ommons.org/licenses/by-sa/3.0/deed.en (2009)
Reality into Advertising Campaigns: History, Makris, S., Tsarouchi, P., Matthaiakis, A. S.,
Technology, and Future Trends Athanasatos, A., Chatzigeorgiou, X., Stefos, M., . . ..,
▶ Interacting with a Fully Simulated Self- Aivaliotis, S.: Dual arm robot in cooperation with
humans for flexible assembly. CIRP Ann. 66(1),
Balancing Bipedal Character in Augmented
13–16 (2017)
and Virtual Reality Mateo, C., Brunete, A., Gambao, E., Hernando, M.:
▶ Interaction with Mobile Augmented Reality Hammer: an android based application for end-user
Environments industrial robot programming. In: Proceedings of
MESA, pp. 1–6. IEEE, Senigallia (2014).
▶ Interactive Augmented Reality to Support
Matsumaru, T.: Mobile robot with preliminary-
Education announcement and display function of forthcoming
▶ Mixed Reality motion using projection equipment. In: Proceedings
▶ Mixed Reality and Immersive Data of ROMAN, pp. 443–450. IEEE, Hatfield (2006)
Michalos, G., Karagiannis, P., Makris, S., Tokçalar, Ö.,
Visualization
Chryssolouris, G.: Augmented reality (AR) applica-
▶ Mixed Reality, Gamified Presence, and Story- tions for supporting human robot interactive coopera-
telling for Virtual Museums tion. Procedia CIRP. 41, 370–375 (2016)
▶ Potential of Augmented Reality for Intelligent RoboCup.: BvOF RoboCup2013 – Junior Rescue
by RoboCup2013, https://www.flickr.com/photos/
Transportation Systems
robocup2013/9154255582/in/photostream/ CC BY 2.0
▶ Training Spatial Skills with Virtual Reality and https://creativecommons.org/licenses/by/2.0/ (2013)
Augmented Reality Sutherland, I.E.: A head-mounted three dimensional dis-
play. In: Proceedings of AFIPS, pp. 757–764. ACM,
San Francisco (1968)
Vogel, C., Poggendorf, M., Walter, C., Elkmann, N.:
References Towards safe physical human-robot collaboration: a
projection-based safety system. In: Intelligent Robots
Akan, B., Çürüklü, B.: Augmented reality meets industry: and Systems (IROS), 2011 IEEE/RSJ International
interactive robot programming. In: Proceedings of Conference on, pp. 3355–3360. IEEE (2011)
Augmented Reality for Maintenance 177

gained more interest also in industrial and

Augmented Reality for manufacturing contexts due to the recent
Maintenance advances in the field. Indeed, even though main- A
tenance, repair, and assembly are considered stra-
Gianluca Paravati tegic application fields for augmented reality
Dipartimento di Automatica e Informatica, technologies from a couple of decades, only in
Politecnico di Torino, Torino, Italy recent times it has been possible to widen the
development of related applications by focusing
the efforts in the exploitation of consumer elec-
Synonyms tronics devices. Previously, ad hoc hardware only
used by specialists was developed and involved in
Mediated reality; Mixed reality; Real-time limited experimental tests. Figure 1 shows an
augmentation applicative use case which makes use of a con-
sumer device (a smartphone or a tablet) to provide
the user step-by-step instructions about the proce-
Definition dure for manual replacement of exhausted toner
for a commercial printer. The application running
Augmented reality for maintenance is a techno- on the handheld device recognizes the item sub-
logic solution encompassing both software and ject to maintenance and provides contextualized
hardware resources with the aim of providing hints by superimposing relevant animations to
additional virtual information as a support for perform the replacement procedure.
assisting technicians in performing complex The development of augmented reality appli-
tasks. cations in manufacturing processes represents the
evolution of the industry in the digital era. Starting
from seventeenth century, during the first indus-
Introduction trial revolution, mechanical production systems as
a substitute of hand production methods appeared
Augmented reality (AR) is considered a computer for the first time in manufacturing processes. Dur-
vision solution aimed at “augmenting” the real ing the second industrial revolution, large-scale
world with additional information in the reality- manufacturing of machine tools together with
virtuality continuum by Milgram and Kishino improvements in transports lead the industry to
(1994). Differently from virtual reality (VR), mass production methods. More recently, the
where entire virtual reproductions of the real introduction of information technology in the
world are generated, in augmented reality the industry revolutionized again the manufacturing
central idea is to add virtual objects (or overlay processes by pervading them with the digital
visual aids) into a real scene. Augmented scenes transformation. For instance, programmable con-
are conveyed to the user either in see-through trollers and computer-based software applications
head-mounted displays or in images captured by added another dimension to the control capabili-
a camera. Computer vision algorithms have the ties of machine tools. The last epochal evolution
key role of analyzing the images provided by the in manufacturing concerns the switch from the
camera and solve the positioning and orientation introduction of information technology to the
problem of virtual objects within the real scene. integration of cyber-physical systems, which is
Augmented reality technology has been and is characterized by a higher degree of design
currently used in a wide range of application sce- complexity.
narios encompassing those concerning gaming, In this context, characterized by a trend in
entertainment, cultural heritage, tourism, con- developing ever more complex manufacturing
struction, education, health care, navigation, mil- systems, all maintenance processes become ever
itary, to name a few. Recently, this technology more challenging, also for experienced operators.
178 Augmented Reality for Maintenance

Augmented Reality for Maintenance, device to provide step-by-step instructions to replace inter-
Fig. 1 Augmented reality use case. Virtual animations nal components of a consumer device
are superimposed to the user’s view through a mobile

The variety of adopted industrial equipment and maintenance procedure, i.e., the replacement of
machinery are becoming more functionally rich. bearings in stone cutting machines in Fig. 2 and
Indeed, new industrial technologies are character- cleaning of lens in a laser-based precision mea-
ized often by a combination of advanced electron- surement tool in Fig. 3. The illustrated use cases
ics and software, thereby requiring operators to are representative of the main features belonging
possess an in-depth knowledge which can be built to augmented reality-based applications:
only after years of work in their field. Addition-
ally, it should be also taken into account that the
– The superimposition of virtual objects and
fast pace of change in technology-related areas
hints onto the real world in real time
requires constant updates for the technical person-
– The possibility to take into account of the
nel. Often, even the more experienced operator
user’s viewpoint by means of a tracking system
has to deal with new situations to solve technical
– The possibility to insert virtual objects in a real
issues in complex cyber-physical systems. As a
world scene by automatically calculating all
result, operational and functional failures are more
their 6 degrees of freedom (DOFs) which con-
difficult and expensive to detect, troubleshoot,
trol the objects’ location, orientation, and scale
and repair.
In this increasingly complex industrial
machine scenario, augmented reality-based solu- Benefits of Using Augmented Reality in
tions aim to improve technician’s performance by the Industry
enabling intuitive representations and real-time
visualization of relevant information regarding Augmented reality gives the opportunity to make
both corrective and predictive maintenance easier for industry workers to deal with specific
(Antonijević et al. 2016). Figures 2 and 3 show industrial processes. Typical applications range
two applicative use cases of augmented reality in from maintenance tasks, where specific and punc-
the field of maintenance in an industrial scenario, tual information concerning cyber-physical systems
conceived in the EASE-R12 European Project are contextualized in real settings, to virtual engi-
(http://www.easer3.eu/). In both cases, a techni- neering, where virtual prototypes constitute the basis
cian is engaged in carrying out a specific for engineering tasks (Vilacoba et al. 2016).
Augmented Reality for Maintenance 179

Augmented Reality for Maintenance, Fig. 2 Augmented reality industrial use case concerning the replacement of
bearings in stone cutting machines

Augmented Reality for Maintenance, Fig. 3 Augmented reality industrial use case concerning the cleaning of lens in
a laser-based precision measurement tool

The most common application concern the demand and representing an effective tool for
transformation of conventional manuals into dig- guided troubleshooting (Ghimire et al. 2016).
ital instructions – showing operations to be Based on the situation faced by an operator, a
performed at the right time in the right place – visualization system – either in the form of hand-
which may have the characteristics of being held devices (HHD) or head-mounted displays
always updated thanks to connectivity, thereby (HMD) – retrieve all necessary information and
providing additional documentation available on provides appropriate instructions back to the user
180 Augmented Reality for Maintenance

through the augmented reality display and other seamlessly integrate computer graphics with real
components (e.g., audio). Instructions can be of environments by avoiding the use of encumbering
arbitrary choice, from simple reproduction of technologies. Today, the majority of mobile aug-
technical manuals, design diagrams, schematics mented reality systems are based on head-
to contextualized 3D static models, virtual anima- mounted displays (HMDs) and see-through
tions, short movies, images, and audio instruc- devices. HMDs involve the use of complex equip-
tions. Instructions are superimposed to the field ment to be worn by the user (e.g., the Oculus Rift).
of view of maintenance operators, which can See-through systems became popular due to the
safely work on faulty equipment. rapid prototyping possibilities offered by several
Augmented reality can be seen as a method- software libraries (e.g., ARToolkit). However, a
ology to provide just-in-time training, by deliv- current bottleneck is constituted by the use of
ering only the information needed in a particular transparent display technology, which is not
context while performing a specific task. Tech- mature yet.
nicians exploit this information to operate on the There exist several options for overlapping
task and are able to increment their prior personal digital contents to the surrounding world (Navab
knowledge in a lifelong learning process. This 2004):
way, they are also allowed to deal with an
increased range of maintenance tasks than before – Projecting virtual models onto objects in the
with less training, which translates in evident real world
advantages from the point of view of a more – Integrating virtual models into camera views
cost effective allocation of the workforce in the obtained from video cameras
industry. In fact, in this scenario, a single opera- – Projecting virtual components onto the user’s
tor is able to handle several equipment and cus- retina
tomers’ machineries, thus improving – Visualizing the virtual components through a
responsiveness and customer satisfaction. semitransparent display while observing the
It has been proven that augmented reality real world
allows users to perform maintenance tasks with a
lower rate of errors and faster than using paper- In industrial scenarios, the last two options
based instruction manuals (Lamberti et al. 2014, should be preferred since they provide unhindered
2017; Henderson and Feiner 2011). This is due to views while remaining safe and secure. In fact,
an improved user experience driven by the fact portable and unobtrusive devices like see-through
that virtual tools can be directly shown in real glasses guarantee hands-free operations by repair
working environments and real-time information technicians.
is always available in the scene for the technician. Virtual retinal displays (VRDs) have been also
It is worth recalling that faster intervention times recently introduced to overlay diagnostics and
at the factory floor reduces machinery downtime, repair instructions directly onto the view of main-
thereby reducing loss of profits for an industry. tenance operators.
Moreover, other key drivers for the adoption of Whatever the technology used to visualize the
these systems concern the reduction of techni- augmented components, one of the key problems
cians’ mental workload during operations, the in augmented reality is the determination of the
possibility to count on technical assistance during position and orientation of a camera in 3D space,
maintenance, and the reduction in training time known as camera pose estimation problem. It
for new technicians. should be considered that the level of accuracy
needed in pose estimation in augmented reality
applications for maintenance is, in general, more
Trends and Challenges demanding than in other fields. Indeed, small
changes in a pose can be the cause of huge devi-
To maximize the effectiveness and usability of ations in the visualization of the virtual content
augmented reality interfaces, their design should (Lakshmprabha et al. 2015).
Augmented Reality for Maintenance 181

The majority of augmented reality applica- into a problem in precise positioning of virtual
tions are based on the use of markers (like QR components.
codes) to solve the real-time camera pose esti- Additional challenges are raised by the inter- A
mation problem (Antonijević et al. 2016). action paradigm needed for using the application
Marker-based techniques are particularly suit- effectively and in natural way. Though many aug-
able for stationary objects. They require to mod- mented reality applications are developed just as
ify the aspect of the real world manufacturing an informative means (e.g., furniture placement in
scenario by introducing specific markers on the interior design), thus letting the interaction
top of machines or within the real environment through touch-based screen devices be suitable
to correctly locate virtual assets. Therefore, for the specific task, the use cases which can be
often the use of markers is perceived as a limi- faced in manufacturing and industrial scenarios
tation of augmented reality (De Crescenzio et al. have different requirements in terms of interac-
2011). Marker-based techniques require that tion. In fact, technicians should carry out hands-
illumination is sufficient and markers are always free tasks. This requirement makes troublesome
visible. the use of input devices based on hand interaction,
On the other hand, marker-less techniques per- e.g., touch- and multitouch-based. A possible
mit both to track moving parts and avoid the use of solution is related to the use of speech recognition
external markers to be placed in the environment technology, today enough mature to ensure an
(Lamberti et al. 2017). Marker-less techniques are intuitive and natural user experience. Recent pro-
based on image processing algorithms aimed to posals in the field encompass also the possibility
detect image features and perform 3D reconstruc- of automatically generating icon-based interfaces
tion. The process involves different phases, to associate unconstrained voice commands
namely feature extraction, features detection, fea- issued by the user to application functionalities
tures matching, and image registration. The most based on the semantic similarity with the evoked
promising algorithms used for feature extraction command (Lamberti et al. 2017). However, it
are Scale Invariant Feature Transform (SIFT) and should be considered that speech recognition
Speed-Up Robust Features (SURF). Features technology is robust in controlled situations, i.e.,
detection algorithms apply classification mecha- non-noisy environments. Thus, the noise at the
nism like Support Vector Machine (SVM), factory floor should be taken into account in the
K-Nearest Neighbor (KNN), and Sparse design of an augmented reality solution. Gaze-
Representation-based Classification (SRC). Fea- based communication represents a possible alter-
tures matching, which consists in finding the cor- native solution to the problem of hands-free inter-
respondence between two set of features, can be action. Gaze recognition provides a thoughtful
achieved by applying the well-known k-d Tree and natural interaction mode. On the other hand,
algorithm. Finally, Image Registration concerns special purpose hardware (i.e., infrared cameras)
the calculation of the geometrical transform that is needed to realize a robust gaze-based interface,
aligns the featured points of two images of the which can be in competition with the hardware
same object. A suitable methodology is provided used for augmented reality (e.g., HMD and see-
by random sample consensus (RANSAC) through glasses). Other possible solutions concern
(Dandachi et al. 2015). the use of gesture/pose recognition technologies,
Given the overview of the algorithms used in where the user could interact with the application
augmented reality, a possible bottleneck concerns interface both by using hand gestures/static poses
the need of sophisticated processing on devices (e.g., recognized through consumer devices such
that should be portable. The challenge is to as the Leap Motion and Microsoft Kinect) and
develop solutions capable to take into account wearable solutions (e.g., inertial-based trackers)
the currently achievable speed and accuracy. In Henderson and Feiner (2010). However, in all
fact, one of the issues with today’s augmented these latter cases, the technician should suspend
reality applications is the lack of accuracy in 3D the maintenance procedure to interact with the
resolution (Vilacoba et al. 2016), which translates application.
182 Augmented Reality Gaming

Conclusion and Discussion Henderson, S., Feiner, S.: Opportunistic tangible user inter-
faces for augmented reality. IEEE Trans. Vis. Comput.
Graph. 16(1), 4–16 (2010)
Augmented reality domain can be applied in dif- Henderson, S., Feiner, S.: Exploring the benefits of aug-
ferent application scenarios, encompassing gam- mented reality documentation for maintenance and
ing, entertainment, cultural heritage, tourism, repair. IEEE Trans. Vis. Comput. Graph. 17(10),
construction, maintenance, education, health 1355–1368 (2011).
Lakshmprabha, N.S., Kasderidis, S., Mousouliotis, P.,
care, navigation, and military. Among them, Petrou, L., Beltramello, O.: Augmented reality for
industrial maintenance and repair represents a maintenance application on a mobile platform. In:
strategic thread for the application of augmented IEEE Virtual Reality (VR), Arles, pp. 355–356 (2015)
reality-related technologies, due to the economic Lamberti, F., Manuri, F., Sanna, A., Paravati, G., Pezzolla,
P., Montuschi, P.: Challenges, opportunities, and future
revenues resulting from the possible reduction in trends of emerging techniques for augmented reality-
time required for maintenance of complex sys- based maintenance. IEEE Trans. Emerg. Topics
tems and the automation of training processes. Comput. 2(4), 411–421 (2014).
This entry described the benefits of using aug- Lamberti, F., Manuri, F., Paravati, G., Piumatti, G., Sanna,
A.: Using semantics to automatically generate speech
mented reality technologies in the maintenance interfaces for wearable virtual and augmented reality
and repair field. Open issues are also discussed, applications. IEEE Trans. Hum. Mach. Sys. 47(1),
mainly focusing on technological problems that 152–164 (2017).
currently limit a further spread of augmented real- Milgram, P., Kishino, F.: A taxonomy of mixed reality
visual displays. IEICE Trans. Inf. Syst. E77-D(12),
ity solutions. 1321–1329 (1994)
Navab, N.: Developing killer apps for industrial aug-
mented reality. IEEE Comput. Graph. Appl. 24(3),
16–20 (2004).
Cross-References Vilacoba, D., Trujillo, M.Á., Viñuales, A., Weber, P.: Press
dedicated machine show-room, a direct application of
augmented reality in industry. An industrial Aug-
▶ Interaction with Mobile Augmented Reality
mented reality experience. In: IEEE International Con-
Environments ference on Industrial Technology (ICIT), Taipei,
▶ Interactive Virtual Reality Navigation Using pp. 1990–1995 (2016)
Cave Automatic Virtual Environment
Technology

Augmented Reality Gaming

References
▶ Augmented Reality Entertainment: Taking
Antonijević, M., Sučić, S., Keserica, H.: Augmented real- Gaming Out of the Box
ity for substation automation by utilizing IEC 61850
communication. In: 39th International Convention on
Information and Communication Technology, Elec-
tronics and Microelectronics (MIPRO), Opatija,
pp. 316–320 (2016) Augmented Reality in
Dandachi, G., Assoum, A., Elhassan, B., Dornaika, F.:
Machine learning schemes in augmented reality for
Image-Guided Surgery
features detection. In: Fifth International Conference
on Digital Information and Communication Technol- Fabrizio Cutolo
ogy and its Applications (DICTAP), Beirut, Department of Information Engineering,
pp. 101–105 (2015)
De Crescenzio, F., Fantini, M., Persiani, F., Di Stefano, L.,
University of Pisa, Pisa, Italy
Azzari, P., Salti, S.: Augmented reality for aircraft
maintenance training and operations support. IEEE
Comput. Graph. Appl. 31(1), 96–101 (2011) Synonyms
Ghimire, R., Pattipati, K.R., Luh, P.B.: Fault diagnosis and
augmented reality-based troubleshooting of HVAC
systems. In: IEEE AUTOTESTCON, Anaheim, Augmented reality in surgery; Image-guided sur-
pp. 1–10 (2016) gery; Surgical navigation
Augmented Reality in Image-Guided Surgery 183

Definition reconstructions of the anatomy. Those visualiza-

tion modalities have been designed to act as sur-
Augmented reality visualization in image-guided gical guidance and as tools for surgical planning A
surgery provides the surgeon with the ability to or for diagnosis (Peters 2000, 2006).
access the radiological images and surgical plan- The idea of integrating in situ the surgeon’s
ning contextually to the anatomy of the real perceptive efficiency with the aid of new aug-
patient. It aims to integrate surgical navigation mented reality (AR) visualization modalities has
with virtual planning. become a dominant topic of academic and indus-
trial research in the medical domain since the
1990s. The high expectations accrued among
Introduction researchers, technicians, and physicians regarding
this fascinating new technology were predomi-
The general ability to see into a living human nantly related to the improvements potentially
system and to transfer the three-dimensional com- brought by AR-based devices to surgical naviga-
plexity of the human body into a comprehensive tion and planning. In this context, AR visualiza-
and useful visual representation has historically tion is indeed regarded as capable of providing the
been considered of utmost importance by physicians surgeon with the ability to access the radiological
in their will to pass on the acquired knowledge and images and surgical planning contextually to the
experience to future generations (Fig. 1). In more real patient anatomy.
recent times, the growing availability of new med- Consequently, in image-guided surgery (IGS)
ical imaging modalities together with the need to systems, AR technology appears as a significant
reduce the invasiveness of the surgical procedures development, because it aims to profitably inte-
has encouraged the research for new 3D visuali- grate surgical navigation with virtual planning
zation modalities of patient-specific virtual (Kersten-Oertel et al. 2013).

Augmented Reality in Image-Guided Surgery, Fig. 1 Pictorial representation of the th7ree-dimensional complexity
of the human anatomy (Frans Denys, The Anatomy Lesson of Dr. Joannes van Buyten, 1648, oil on canvas)
184 Augmented Reality in Image-Guided Surgery

State-of-the-Art Work translation into immediate and reliable

applications.
Background Many of those systems did not take into
In 1986 (Roberts et al. 1986) a first implementa- account the operational constraints imposed by
tion of a medical AR neurosurgical guidance the surgical context and did not satisfy the sur-
system was proposed: the system was intended geons’ practical needs and requirements. Gener-
to allow the proper projection of computed ally, many AR systems have been regarded as
tomography (CT) image data into the optics of disrupting the surgeon’s act primarily because
an operating microscope. After an interactive most of them have lacked of a systematic evalua-
patient-to-CT image registration, an ultrasonic tion within a clinical context (Kersten-Oertel et al.
non-real-time tracking of the microscope allo- 2012; Meola et al. 2016). In fact, the basic condi-
wed proper reformatting of the CT data before tion for the acceptance of a new technology
being projected into the eyepieces of the micro- (as AR) in the operating room (OR) is related to
scope. Since then, tools (or defined applications) its capacity of being smoothly integrated into the
employing AR have been designed and tested in workflow of the intervention, without affecting
the context of several surgical and medical dis- and disturbing the user during the rest of the
ciplines (Rankin et al. 2015). These applications procedure (Navab et al. 2007; Sielhorst
comprise video see-through systems et al. 2008).
(in conjunction with standard workstations, lap- Despite all of this, with the advances in tech-
tops, and tablet PC), modified endoscopes, and nology, AR will surely soon represent a leading-
modified operating microscopes. AR has been edge solution in the context of IGS, especially
proposed in many surgical contexts, including once the balance between technological expecta-
maxillofacial surgery (Marmulla et al. 2005; tions and realistic and application-driven out-
Zinser et al. 2013b; Badiali et al. 2014; Suenaga comes will be achieved.
et al. 2015; Wang et al. 2017; Zhu et al. 2017),
neurosurgery (Edwards et al. 2004; Lovo et al. Taxonomy for Classification
2007; Low et al. 2010; Inoue et al. 2013; Cabrilo In 2012, a taxonomy of AR visualization systems
et al. 2014, 2015; Kersten-Oertel et al. 2015; in IGS was proposed (Kersten-Oertel et al. 2012),
Meola et al. 2016; Cutolo et al. 2017), ENT and a systematic overview of the trends and solu-
(ear, nose, and throat) surgery (Caversaccio tions adopted in the field to that day presented
et al. 2008), orthopedic surgery (Navab et al. (Kersten-Oertel et al. 2013). The acronym for
2010; Abe et al. 2013; Wu et al. 2014; Fritz the taxonomy (DVV) is derived from its three
et al. 2014; Cutolo et al. 2016a; Elmi-Terander key components: data type, visualization pro-
et al. 2016), robotic surgery (Falk et al. 2005; Su cessing, and view. According to the taxonomy,
et al. 2009; Zeng et al. 2017), and general/lapa- for classifying and assessing the efficacy of a
roscopic surgery (Feuerstein et al. 2008; new AR system for IGS, the attention ought to
Baumhauer et al. 2007; Nicolau et al. 2011; be focused on the particular surgical scenario in
Muller et al. 2013). which the visualization system aims to be
integrated.
Open Issues The surgical scenario affects each of the three
Despite such widespread diffusion, there are DVV factors, namely, the type of data that
still major reasons why AR IGS systems are should be displayed at a specific surgical step,
not yet routinely used in the medical the visualization processing technique
workflow. Among them, there is the fact implemented to provide the best pictorial repre-
that most of the systems were historically, sentation of the augmented scene, and how and
and still are, developed as proof-of-concept where the output of the visualization processing
devices that were/are mostly conceived for should be presented to the end user (i.e.,
research users more than for their immediate the view).
Augmented Reality in Image-Guided Surgery 185

Different AR Display Units in IGS between the 3D representation of the real world
and the nature of the virtual content, rendered as a
In the realm of AR-based IGS systems, different 2D image. This causes few perceptual conflicts in A
technological approaches and specifically differ- depth perception. To cope with these issues, alter-
ent embodiments of the display units have been native and promising approaches, based on the
proposed, each with its own advantages and draw- integral imaging technology, were proposed
backs. In Table 1 advantages and drawbacks of (Liao et al. 2001, 2004). Integral imaging displays
different AR-based solutions are reported. In the use a set of 2D elemental images from different
light of avoiding abrupt changes to the surgical perspectives to generate a full-parallax 3D visual-
setup and workflow, historically the first ization. Therefore, with integral imaging-based
AR-based systems in surgical navigation have displays, a proper 3D overlay between virtual
been implemented starting from commonly used content and real scene can be obtained. Some
devices. Augmented operating microscopes were embodiments of such technology have been spe-
proposed in neurosurgery and in maxillofacial cifically designed and tested for maxillofacial sur-
surgery (Birkfellner et al. 2002). In these systems, gery and neurosurgery (Iseki et al. 1997; Narita
generally the augmentation happens by et al. 2014; Liao et al. 2006, 2010; Suenaga et al.
“injecting” the virtual content directly into the 2013). The integral imaging paradigm thus is able
optical path of the real image, hence by inserting to provide the user with an egocentric viewpoint
a beam splitter into the microscope optics. The and a full-parallax augmented view in a limited
fixed configuration of the eyepieces with respect viewing zone (imposed by the integral imaging
to the surgical scene makes them not usable with display).
different viewpoints. In laparoscopy, and generally in endoscopic
Other solutions featured the use of spatial mon- surgery, the part of the environment where the
itors and video-based tracking modalities and attention of the surgeon is focused during the
were used in neurosurgery (Grimson et al. 1996; surgical task (DVV’s perception location) is a
Deng et al. 2014), maxillofacial surgery (Zinser stand-up monitor. Indeed, in such procedures,
et al. 2013a), and general surgery (Muller et al. the surgeon operates watching endoscopic video
2013). Such systems, as the ones based on exter- images reproduced on the spatial display unit
nal monitors, have a reduced logistic impact (Caversaccio et al. 2008; Freysinger et al. 1997).
within the operating room, but they do not provide Therefore, the virtual information is usually
an egocentric viewpoint of the surgical scene. merged with the real-time video frames grabbed
Another category of AR systems was based on by the endoscope and presented on a stand-up
the use of half-transparent screens in conjunction monitor. These systems were also tested in robotic
with display technologies providing monoscopic, surgery (Su et al. 2009; Mourgues and Coste-
stereoscopic, or autostereoscopic parallax. Black- Maniere 2002; Devernay et al. 2001). Particularly
well (Blackwelll et al. 1998) and Wesarg (Wesarg in IGS, the quality of an augmented reality
et al. 2004) introduced two different prototypes of (AR) experience depends on how well the virtual
AR windows. The first system by Blackwell pro- content is integrated into the real world spatially,
vided a stereoscopic vision of the virtual content photometrically, and temporally (Sielhorst et al.
by means of a pair of shutter glasses, whereas the 2008). For this reason, wearable AR systems par-
second one by Wesarg was monoscopic. Stetten ticularly offer the most ergonomic solution in
et al. (2001) have proposed a simple and interest- those medical tasks manually performed under
ing optical see-through handheld half-silvered user’s direct vision (open surgery, introduction
mirror that overlays ultrasound scans aligned of biopsy needle, palpation, etc.) since they min-
with the scanned area. imize the extra mental effort normally required for
The major shortcomings of the optical see- switching the attention between the surgical area
through paradigm implemented in standard AR and the augmented view presented on the external
windows are due to the intrinsic incompatibility display.
186 Augmented Reality in Image-Guided Surgery

Augmented Reality in Image-Guided Surgery, Table 1 Different embodiments of AR displays for IGS
AR display type Advantages Drawbacks
Surgical microscope Surgical microscopes are already Bulkiness
routinely used in several
microsurgical procedures
Fixed configuration: degradation of Fixed configuration: not adaptable to comply
the calibration is decreased with different viewpoints
Video see-through handheld Reduced logistic impact Parallax problem (non-egocentric viewpoint)
displays (tablet-smartphones) No stereopsis. Other depth cues are needed
for depth perception
Accurate image registration – no Difficult to use in those surgical tasks
eye-to-display calibration is needed manually performed under surgeon’s direct
vision
Peripheral view not registered with AR view
Camera-mediated view of the surgical area
Optical see-through external Direct view of the surgical area AR image registration is not so accurate:
screens (stereoscopic and robust and reliable eye-to-display calibration
monoscopic) would be needed
Display brightness may not be sufficient
under scialytic lamps
Perceptual conflicts due to mismatched
accommodations between real scene and
virtual content
Parallax problem (non-egocentric viewpoint)
Difficult to use in those surgical tasks
manually performed under surgeon’s direct
vision
Video see-through external External monitors are already Parallax problem (non-egocentric viewpoint)
displays present in the surgical rooms Mostly 2D displays: no stereopsis
Accurate image registration – no Difficult to use in those surgical tasks
eye-to-display calibration is needed manually performed under surgeon’s direct
vision
Camera-mediated view of the surgical area
Full-parallax 3D displays Full-parallax view of the scene. Bulkiness
Natural depth perception is
recovered
Accurate image registration – no Limited depth range
eye-to-display calibration is needed Limited resolution and limited viewing zone
Optical see-through wearable Egocentric viewpoint Perceptual conflicts due to mismatched
displays (monoscopic or accommodations between real scene and
stereoscopic) virtual content
Reduced logistic impact AR image registration is not so accurate:
robust and reliable eye-to-display calibration
would be needed
Display brightness may not be sufficient
under scialytic lamps
Video see-through wearable Egocentric viewpoint Perceptual vergence-accommodation conflict
displays (monoscopic or Restricted peripheral view
stereoscopic) Accurate image registration – no Surgeon’s field of view is restricted by the
eye-to-display calibration is needed display
Reduced logistic impact Camera-mediated view of the surgical area
Augmented Reality in Image-Guided Surgery 187

Wearable AR Displays in IGS the reality by means of simple virtual elements

(models, icons, or text labels), but shortcomings
Wearable AR systems based on HMDs intrinsi- remain both from a technical and a perceptual A
cally provide the user with an egocentric view- standpoint, especially in case of virtual contents
point, and by generating both binocular parallax of greater complexity.
and motion parallax, they smoothly augment the Although the optical see-through HMDs were
user’s perception of the surgical scene (Cutolo once at the leading edge of the AR research, their
et al. 2016b). In these HMDs, the see-through degree of adoption and diffusion did slow down
capability is accomplished either through the over the years due to technological and human-
aforementioned video see-through paradigm or factor limitations. Just to mention a few of them:
through the optical see-through paradigm. the presence of a small augmentable field of view,
Typically, in optical see-through systems, the the reduced brightness offered by standard LCOS
user’s direct view of the real world is augmented microdisplays, the perceptual conflicts between
with the projection of virtual information on a the 3D real-world and the 2D virtual image on
beam combiner and then into the user’s line of the SSP, and the need for frequent recalibrations of
sight (Rolland et al. 1994). Differently, in video the HMD for yielding accurate spatial registra-
see-through systems, the virtual content is merged tion. Some of the technological limitations, like
with camera images captured by two external the small field, are being and will be likely solved
cameras rigidly fixed on the visor. along with the technological progress. The
The industrial pioneers, as well as the early remaining two limitations are harder to cope with.
adopters of AR technology, properly considered The difference in the user’s perception of the
the camera-mediated view typical of the video real 3D world and of the 2D projection of the
see-through paradigm, as drastically affecting the virtual content on the SSP creates perceptual con-
quality of the visual perception and experience of flicts due to mismatched accommodations. These
the real world. By contrast, optical see-through perceptual conflicts often are reflected in reducing
systems provide the user with a natural view of the the benefits brought by the optical paradigm of
real world with full resolution. The fundamental leaving unaltered the user’s view of the real world.
optical see-through paradigm of HMDs is still the In optical see-through HMDs, the user is indeed
same as described by Benton (Benton 2001). forced to accommodate his/her eye for focusing
A straightforward implementation of the optical all the virtual objects on the SSP placed at a fixed
see-through paradigm comprises the employment distance. On the other hand, the focus distance of
of a half-silvered mirror or beam combiner to each physical object in the 3D world depends on
merge real view and virtual content. The user’s its relative distance from the observer and may
own view is herein augmented by rendering the dynamically vary over time. This means that even
virtual content on a two-dimensional (2D) micro- if an accurate geometric registration of virtual
display and by sending it to the beam combiner. objects to the real scene is attained on the 2D
Lenses can be placed between the beam combiner SSP plane, the user may not be able to view both
and the display to focus the virtual 2D image so the virtual and real content in focus at the same
that it appears at a comfortable viewing distance time. This aspect is particularly relevant in appli-
on a semitransparent surface of projection (SSP) cations devoted to surgical navigation, since it
(Rolland and Cakmakci 2005; Holliman et al. reduces the user’s capacity to interact with the
2011). As an alternative, the use of high-precision real surgical field while maintaining the virtual
waveguide technologies allows the removal of the aid in focus. The second major shortcoming of
bulky optical engine placed in front of the eyes the standard optical see-through HMDs is related
(Mukawa et al. 2008). The optical see-through with the geometric registration required to obtain
paradigm is particularly suitable for augmenting a geometrically consistent augmentation of the
188 Augmented Reality in Image-Guided Surgery

reality, which is an essential prerequisite for being needed to estimate user’s eye pose in relation to
considered as reliable surgical guidance system. the SSP reference system (b) often entails a
As a general rule, for obtaining accurate geo- tedious and error-prone method (Tuceryan et al.
metric registration in AR applications, the process 2002; Genc et al. 2002; Gilson et al. 2008; Kellner
of image formation generated by the virtual view- et al. 2012). Further, this process should be
point must perfectly mimic, both intrinsically and repeated any time the HMD moves with respect
extrinsically, the one of the real viewpoint (Cutolo to the user’s eyes causing a change in the relative
et al. 2014). In optical see-through HMDs, the position between DRS and user’s eye reference
spatial alignment of the virtual content with the system, and it should be autonomous and real
real 3D world needs for: time. Current and more advanced calibration
methods (Plopski et al. 2015, 2016), even if they
(a) The tracking of the HMD SSP in relation to work in real time, do not incorporate the user-
the reference system of the real world (SRS) specific and real-time estimation of the eye pro-
(b) A user-specific calibration for estimating the jective model (c), which can change over time
pose between HMD SSP and user’s eye (i.e., with the focus distance due to the accommodation
extrinsic calibration) process.
(c) The definition of a projective model of the Differently, the pixel-wise video-mixing tech-
virtual viewpoint that is consistent to the nology that underpins the video see-through para-
human eye projective model (i.e., intrinsic digm, once integrated with monocular or binocular
calibration) HMDs, can offer high geometric coherence
between virtual and real content. In these systems,
State-of-the-art methods for tracking the a user-specific calibration routine is not necessary,
HMDs (a) yield accurate results in terms of and this is the major advantage of the video versus
HMD pose estimation, whether they exploit exter- the optical see-through approach. In video see-
nal trackers or not. Differently, the calibration step through systems, real scene and virtual information

Augmented Reality in Image-Guided Surgery, Fig. 2 Video see-through paradigm of the stereoscopic HMD used to
aid maxillary repositioning (Cutolo et al. 2015)
Augmented Reality in Image-Guided Surgery 189

can be synchronized, whereas in optical see- In the ideal AR system, especially if designed
through devices, there is an intrinsic lag between for aiding complex operations that demand great
the immediate perception of the real scene and the dexterity as the ones pertaining to surgical pro- A
inclusion of the virtual elements. Further, from a cedures, there should not be any perceivable dif-
perceptual viewpoint, in video see-through sys- ference between the user’s natural view of the
tems, the visual experience of both the real and world and his/her augmented view through the
virtual content is unambiguously controllable by device display. For this goal, the conditions to be
computer graphics, with everything on focus at the satisfied are twofold: accurate registration and
same apparent distance from the user. Additionally, ergonomic interaction.
video see-through systems are much more suited In practical terms, if a new AR system for IGS
than optical see-through systems, to rendering were to comprise tedious calibration routines, the
occlusions between real and virtual elements or to introduction of obtrusive instrumentation into the
implementing complex visualization processing OR, and the presentation of complex and
modalities. Unfortunately, the loss of the confounding computer-generated content and fur-
unobstructed real-world view and the limited field thermore it were to bring questionable benefits to
of view of the displays embedded in commercial the surgical procedure, then it would be likely
3D visors are still the major drawbacks of the video refused. Hence, to facilitate the accomplishment
see-through HMDs (Fig. 2). This aspect is of par- of such ambitious goals in the near future,
ticular importance in IGS applications, wherever researchers and early adopters ought to actively
our goal is of trying to mimic the perceptive effi- collaborate with physicians so that AR-based IGS
ciency of the human visual system to allow a systems could smoothly and profitably get into the
smoother interaction with the augmented visual surgical workflow.
information (Cutolo et al. 2015).

Cross-References
Conclusions and Future Goals
▶ Augmented Learning Experience for School
In recent times, the emerging of modern medical Education
imaging technologies together with the need to ▶ Augmented Reality Entertainment: Taking
reduce the invasiveness of the surgical procedures Gaming Out of the Box
has encouraged the research for new 3D visuali- ▶ Augmented Reality for Maintenance
zation modalities that could act as tools for surgi-
cal guidance. Current limits of standard IGS
systems are mainly due to the increase of the References
surgical time needed for system setup and man-
agement and to the increase of technical complex- Abe, Y., Sato, S., Kato, K., Hyakumachi, T., Yanagibashi,
Y., Ito, M., Abumi, K.: A novel 3D guidance system
ity in the surgical workflow.
using augmented reality for percutaneous
It was thus clear from the outset that, in this vertebroplasty. J. Neurosurg. Spine. 19(4), 492–501
context, AR technology could have represented a (2013). https://doi.org/10.3171/2013.7.Spine12917
groundbreaking development, because its primary Badiali, G., Ferrari, V., Cutolo, F., Freschi, C., Caramella,
D., Bianchi, A., Marchetti, C.: Augmented reality as an
goal is to integrate surgical navigation with virtual
aid in maxillofacial surgery: validation of a wearable
planning contextually to the real patient’s system allowing maxillary repositioning. J. Cranio-
anatomy. Maxillofac. Surg. 42(8), 1970–1976 (2014). https://
AR technology can in fact provide the surgeon doi.org/10.1016/j.jcms.2014.09.001
Baumhauer, M., Simpfendoerfer, T., Schwarz, R., Seitel,
with a direct perception of where the virtual con- M., Müller-Stich, B., Gutt, C., Rassweiler, J., Meinzer,
tent is located within the surgical field, wherever H.-P., Wolf, I.: Soft tissue navigation for laparoscopic
the virtual content is generally obtained during prostatectomy: evaluation of camera pose estimation
preoperative planning. for enhanced visualization. In: Medical Imaging 2007,
190 Augmented Reality in Image-Guided Surgery

pp. 650911-650911-650912. Society for Optics and PC. Stereotact. Funct. Neurosurg. 92(1), 17–24 (2014).
Photonics (2007) https://doi.org/10.1159/000354816
Benton, S.A.: Selected Papers on Three-dimensional Dis- Devernay, F., Mourgues, F., Coste-Maniere, E.: Towards
plays. SPIE Optical Engineering Press, Bellingham endoscopic augmented reality for robotically assisted
(2001) minimally invasive cardiac surgery. In: International
Birkfellner, W., Figl, M., Huber, K., Watzinger, F., Workshop on Medical Imaging and Augmented Real-
Wanschitz, F., Hummel, J., Hanel, R., Greimel, W., ity, Proceedings, pp. 16–20. (2001)
Homolka, P., Ewers, R., Bergmann, H.: A head- Edwards, P.J., Johnson, L.G., Hawkes, D.J., Fenlon, M.R.,
mounted operating binocular for augmented reality Strong, A., Gleeson, M.: Clinical experience and per-
visualization in medicine – design and initial evalua- ception in stereo augmented reality surgical navigation.
tion. IEEE Trans. Med. Imaging. 21(8), 991–997 In: Yang, G.Z., Jiang, T. (eds.) MIAR, pp. 369–376.
(2002). https://doi.org/10.1109/Tmi.2002.803099 Springer, Berlin/Heidelberg (2004). https://doi.org/10.
Blackwelll, M., Nikou, C., DiGioia, A.M., Kanade, T.: An 1007/978-3-540-28626-4_45
image overlay system for medical data visualization. Elmi-Terander, A., Skulason, H., Soderman, M., Racadio,
Med. Image Comput. Comput. Assist. Interv. J., Homan, R., Babic, D., van der Vaart, N., Nachabe,
Miccai’98. 1496, 232–240 (1998) R.: Surgical navigation technology based on aug-
Cabrilo, I., Bijlenga, P., Schaller, K.: Augmented reality in mented reality and integrated 3D intraoperative imag-
the surgery of cerebral arteriovenous malformations: ing: a spine cadaveric feasibility and accuracy study.
technique assessment and considerations. Acta Spine. 41(21), E1303–E1311 (2016). https://doi.org/
Neurochir. 156(9), 1769–1774 (2014). https://doi.org/ 10.1097/BRS.0000000000001830
10.1007/s00701-014-2183-9 Falk, V., Mourgues, F., Adhami, L., Jacobs, S., Thiele, H.,
Cabrilo, I., Schaller, K., Bijlenga, P.: Augmented reality- Nitzsche, S., Mohr, F.W., Coste-Maniere, T.: Cardio
assisted bypass surgery: embracing minimal invasive- navigation: planning, simulation, and augmented real-
ness. World Neurosurg. 83(4), 596–602 (2015). https:// ity in robotic assisted endoscopic bypass grafting. Ann.
doi.org/10.1016/j.wneu.2014.12.020 Thorac. Surg. 79(6), 2040–2048 (2005). https://doi.
Caversaccio, M., Giraldez, J.G., Thoranaghatte, R., Zheng, org/10.1016/j/athorascur.2004.11.060
G., Eggli, P., Nolte, L.P., Ballester, M.A.G.: Aug- Feuerstein, M., Mussack, T., Heining, S.M., Navab, N.:
mented reality endoscopic system (ARES): preliminary Intraoperative laparoscope augmentation for port
results. Rhinology. 46(2), 156–158 (2008) placement and resection planning in minimally inva-
Cutolo, F., Parchi, P.D., Ferrari, V.: Video see through AR sive liver resection. IEEE Trans. Med. Imaging. 27(3),
head-mounted display for medical procedures. Int. 355–369 (2008). https://doi.org/10.1109/Tmi.2007.
Sym. Mix Augment, 393–396 (2014) 907327
Cutolo, F., Badiali, G., Ferrari, V.: Human-PnP: ergonomic Freysinger, W., Gunkel, A.R., Thumfart, W.F.: Image-
AR interaction paradigm for manual placement of rigid guided endoscopic ENT surgery. Eur. Arch. Otorhino-
bodies. Augmented Environ. Comput. Assist. Interv. laryngol. 254(7), 343–346 (1997). https://doi.org/10.
Ae-Cai. 9365, 50–60 (2015). https://doi.org/10.1007/ 1007/Bf02630726
978-3-319-24601-7_6 Fritz, J., U-Thainual, P., Ungi, T., Flammang, A.J.,
Cutolo, F., Carbone, M., Parchi, P.D., Ferrari, V., Lisanti, Kathuria, S., Fichtinger, G., Iordachita, I.I., Carrino, J.
M., Ferrari, M.: Application of a new wearable aug- A.: MR-guided vertebroplasty with augmented reality
mented reality video see-through display to aid percu- image overlay navigation. Cardiovasc. Intervent.
taneous procedures in spine surgery. In: De Paolis, L.T., Radiol. 37(6), 1589–1596 (2014). https://doi.org/10.
Mongelli, A. (eds.) Augmented Reality, Virtual Reality, 1007/s00270-014-0885-2
and Computer Graphics: Third International Confer- Genc, Y., Tuceryan, M., Navab, N.: Practical solutions for
ence, AVR 2016, Lecce, Italy, June 15–18, 2016. Pro- calibration of optical see-through devices. In: Interna-
ceedings, Part II, pp. 43–54. Springer International tional Symposium on Mixed and Augmented Reality,
Publishing, Cham (2016a) Proceedings, pp. 169–175 (2002)
Cutolo, F., Freschi, C., Mascioli, S., Parchi, P., Ferrari, Gilson, S.J., Fitzgibbon, A.W., Glennerster, A.: Spatial
M., Ferrari, V.: Robust and accurate algorithm for calibration of an optical see-through head-mounted
wearable stereoscopic augmented reality with three display. J. Neurosci. Methods. 173(1), 140–146
indistinguishable markers. Electronics. 5(3), (2008). https://doi.org/10.1016/j.jneumeth.2008.05.
59 (2016b) 015
Cutolo, F., Meola, A., Carbone, M., Sinceri, S., Cagnazzo, Grimson, W.E.L., Ettinger, G.J., White, S.J., LozanoPerez,
F., Denaro, E., Esposito, N., Ferrari, M., Ferrari, V.: T., Wells, W.M., Kikinis, R.: An automatic registration
A new head-mounted display-based augmented reality method for frameless stereotaxy, image guided surgery,
system in neurosurgical oncology: a study on phantom. and enhanced reality visualization. IEEE Trans. Med.
Comput. Assist. Surg. 22(1), 39–53 (2017). https://doi. Imaging. 15(2), 129–140 (1996). https://doi.org/10.
org/10.1080/24699322.2017.1358400 1109/42.491415
Deng, W., Li, F., Wang, M., Song, Z.: Easy-to-use aug- Holliman, N.S., Dodgson, N.A., Favalora, G.E., Pockett,
mented reality neuronavigation using a wireless tablet L.: Three-dimensional displays: a review and
Augmented Reality in Image-Guided Surgery 191

applications analysis. IEEE Trans. Broadcast. 57(2), Lovo, E.E., Quintana, J.C., Puebla, M.C., Torrealba, G.,
362–371 (2011). https://doi.org/10.1109/Tbc.2011. Santos, J.L., Lira, I.H., Tagle, P.: A novel, inexpensive
2130930 method of image coregistration for applications in
Inoue, D., Cho, B., Mori, M., Kikkawa, Y., Amano, T., image-guided surgery using augmented reality. Neuro- A
Nakamizo, A., Yoshimoto, K., Mizoguchi, M., surgery. 60(4 Suppl 2), 366–371.; discussion 371–362
Tomikawa, M., Hong, J., Hashizume, M., Sasaki, T.: (2007). https://doi.org/10.1227/01.NEU.0000255360.
Preliminary study on the clinical application of aug- 32689.FA
mented reality neuronavigation. J. Neurol. Surg. Low, D., Lee, C.K., Dip, L.L., Ng, W.H., Ang, B.T., Ng, I.:
A. Cent. Eur. Neurosurg. 74(2), 71–76 (2013). https:// Augmented reality neurosurgical planning and naviga-
doi.org/10.1055/s-0032-1333415 tion for surgical excision of parasagittal, falcine and
Iseki, H., Masutani, Y., Iwahara, M., Tanikawa, T., convexity meningiomas. Br. J. Neurosurg. 24(1),
Muragaki, Y., Taira, T., Dohi, T., Takakura, K.: 69–74 (2010). https://doi.org/10.3109/
Volumegraph (overlaid three-dimensional image- 02688690903506093
guided navigation). Clinical application of augmented Marmulla, R., Hoppe, H., Muhling, J., Eggers, G.: An
reality in neurosurgery. Stereotact. Funct. Neurosurg. augmented reality system for image-guided surgery.
68(1–4 Pt 1), 18–24 (1997) Int. J. Oral Maxillofac. Surg. 34(6), 594–596 (2005).
Kellner, F., Bolte, B., Bruder, G., Rautenberg, U., https://doi.org/10.1016/j.ijom.2005.05.004
Steinicke, F., Lappe, M., Koch, R.: Geometric calibra- Meola, A., Cutolo, F., Carbone, M., Cagnazzo, F., Ferrari,
tion of head-mounted displays and its effects on dis- M., Ferrari, V.: Augmented reality in neurosurgery: a
tance estimation. IEEE Trans. Vis. Comput. Graph. systematic review. Neurosurg. Rev. (2016). https://doi.
18(4), 589–596 (2012) org/10.1007/s10143-016-0732-9
Kersten-Oertel, M., Jannin, P., Collins, D.L.: DVV: a Mourgues, F., Coste-Maniere, P.: Flexible calibration of
taxonomy for mixed reality visualization in image actuated stereoscopic endoscope for overlay in robot
guided surgery. IEEE Trans. Vis. Comput. Graph. assisted surgery. Med. Image Comput. Comput-Assist.
18(2), 332–352 (2012). https://doi.org/10.1109/ Interv. Miccai. 2488(Pt 1), 25–34 (2002.) 2002
TVCG.2011.50 Mukawa, H., Akutsu, K., Matsumura, I., Nakano, S.,
Kersten-Oertel, M., Jannin, P., Collins, D.L.: The state of Yoshida, T., Kuwahara, M., Aiki, K., Ogawa, M.: Dis-
the art of visualization in mixed reality image guided tinguished paper: a full color eyewear display using
surgery. Comput. Med. Imaging Graph. 37(2), 98–112 holographic planar waveguides. In: 2008 Sid Interna-
(2013). https://doi.org/10.1016/j.compmedimag.2013. tional Symposium, Digest of Technical Papers,
01.009 vol Xxxix, Books I–Iii 39, pp. 89–92 (2008)
Kersten-Oertel, M., Gerard, I., Drouin, S., Mok, K., Sirhan, Muller, M., Rassweiler, M.C., Klein, J., Seitel, A., Gondan,
D., Sinclair, D.S., Collins, D.L.: Augmented reality in M., Baumhauer, M., Teber, D., Rassweiler, J.J.,
neurovascular surgery: feasibility and first uses in the Meinzer, H.P., Maier-Hein, L.: Mobile augmented real-
operating room. Int. J. Comput. Assist. Radiol. Surg. ity for computer-assisted percutaneous
10(11), 1823–1836 (2015). https://doi.org/10.1007/ nephrolithotomy. Int. J. Comput. Assist. Radiol. Surg.
s11548-015-1163-8 8(4), 663–675 (2013). https://doi.org/10.1007/s11548-
Liao, H., Nakajima, S., Iwahara, M., Kobayashi, E., 013-0828-4
Sakuma, I., Yahagi, N., Dohi, T.: Intra-operative real- Narita, Y., Tsukagoshi, S., Suzuki, M., Miyakita, Y., Ohno,
time 3-D information display system based on integral M., Arita, H., Saito, Y., Kokojima, Y., Watanabe, N.,
videography. In: Niessen, W., Viergever, M. (eds.) Moriyama, N., Shibui, S.: Usefulness of a glass-free
Medical Image Computing and Computer-Assisted medical three-dimensional autostereoscopic display in
Intervention – MICCAI 2001. Lecture Notes in Com- neurosurgery. Int. J. Comput. Assist. Radiol. Surg. 9(5),
puter Science, vol. 2208, pp. 392–400. Springer Berlin, 905–911 (2014). https://doi.org/10.1007/s11548-014-
(2001) 0984-1
Liao, H., Hata, N., Nakajima, S., Iwahara, M., Sakuma, I., Navab, N., Traub, J., Sielhorst, T., Feuerstein, M.,
Dohi, T.: Surgical navigation by autostereoscopic Bichlmeier, C.: Action- and workflow-driven aug-
image overlay of integral videography. IEEE Trans. mented reality for computer-aided medical procedures.
Inf. Technol. Biomed. 8(2), 114–121 (2004) IEEE Comput. Graph. Appl. 27(5), 10–14 (2007).
Liao, H., Inomata, T., Sakuma, I., Dohi, T.: Surgical nav- https://doi.org/10.1109/Mcg.2007.117
igation of integral videography image overlay for open Navab, N., Heining, S.M., Traub, J.: Camera Augmented
MRI-guided glioma surgery. Med. Imaging Aug- Mobile C-Arm (CAMC): calibration, accuracy study,
mented Real. 4091, 187–194 (2006) and clinical applications. IEEE Trans. Med. Imaging.
Liao, H.E., Inomata, T., Sakuma, I., Dohi, T.: 3-D aug- 29(7), 1412–1423 (2010). https://doi.org/10.1109/Tmi.
mented reality for MRI-guided surgery using inte- 2009.2021947
gral videography autostereoscopic image overlay. Nicolau, S., Soler, L., Mutter, D., Marescaux, J.: Aug-
IEEE Trans. Biomed. Eng. 57(6), 1476–1486 mented reality in laparoscopic surgical oncology.
(201 0). https://doi.org /10.1109/Tbme.2010. Surg. Oncol. Oxf. 20(3), 189–201 (2011). https://doi.
2040278 org/10.1016/j.suronc.2011.07.002
192 Augmented Reality in Surgery

Peters, T.M.: Image-guided surgery: from X-rays to virtual Imaging. 15, 51 (2015). https://doi.org/10.1186/
reality. Comput. Methods. Biomech. Biomed. Engin. s12880-015-0089-5. ARTN 51
4(1), 27–57 (2000) Tuceryan, M., Genc, Y., Navab, N.: Single-point active
Peters, T.M.: Image-guidance for surgical procedures. alignment method (SPAAM) for optical see-through
Phys. Med. Biol. 51(14), R505–R540 (2006). https:// HMD calibration for augmented reality. Presence-
doi.org/10.1088/0031-9155/51/14/R01 Teleop Virt Environ. 11(3), 259–276 (2002). https://
Plopski, A., Itoh, Y., Nitschke, C., Kiyokawa, K., Klinker, doi.org/10.1162/105474602317473213
G., Takemura, H.: Corneal-imaging calibration for opti- Wang, J., Suenaga, H., Yang, L., Kobayashi, E., Sakuma,
cal see-through head-mounted displays. IEEE Trans. I.: Video see-through augmented reality for oral and
Vis. Comput. Graph. 21(4), 481–490 (2015). https:// maxillofacial surgery. The international journal of
doi.org/10.1109/Tvcg.2015.2391857 medical robotics + computer assisted surgery:
Plopski, A., Orlosky, J., Itoh, Y., Nitschke, C., Kiyokawa, MRCAS 13(2) (2017). https://doi.org/10.1002/rcs.
K., Klinker, G.: Automated spatial calibration of 1754
HMD systems with unconstrained eye-cameras. In: Wesarg, S., Firle, E.A., Schwald, B., Seibert, H., Zogal, P.,
Proceedings of the 2016 15th IEEE International Sym- Roeddiger, S.: Accuracy of needle implantation in
posium on Mixed and Augmented Reality (Ismar), brachytherapy using a medical AR system – a phantom
pp. 94–99 (2016). https://doi.org/10.1109/Ismar. study. Med. Imaging 2004: Visual. Image-Guided Pro-
2016.16 ced. Display. 5367, 341–352 (2004). https://doi.org/10.
Rankin, T.M., Slepian, M.J., Armstrong, D.G.: Augmented 1117/12.535415
reality in surgery. In: Latifi, R., Rhee, P., Gruessner, Wu, J.R., Wang, M.L., Liu, K.C., Hu, M.H., Lee, P.Y.:
R.W.G. (eds.) Technological Advances in Surgery, Real-time advanced spinal surgery via visible patient
Trauma and Critical Care, pp. 59–71. Springer, model and augmented reality system. Comput.
New York (2015) Methods Prog. Biomed. 113(3), 869–881 (2014).
Roberts, D.W., Strohbehn, J.W., Hatch, J.F., Murray, W., https://doi.org/10.1016/j.cmpb.2013.12.071
Kettenberger, H.: A frameless stereotaxic integration of Zeng, B., Meng, F., Ding, H., Wang, G.: A surgical
computerized tomographic imaging and the operating robot with augmented reality visualization for
microscope. J. Neurosurg. 65(4), 545–549 (1986). stereoelectroencephalography electrode implanta-
https://doi.org/10.3171/jns.1986.65.4.0545 tion. Int. J. Comput. Assist. Radiol. Surg. 12(8),
Rolland, J.P., Cakmakci, O.: The past, present, and future 1355–1368 (2017). https://doi.org/10.1007/s11548-
of head-mounted display designs. In: Photonics Asia 017-1634-1
2005, p. 10. SPIE Zhu, M., Liu, F., Chai, G., Pan, J.J., Jiang, T., Lin, L., Xin,
Rolland, J.P., Holloway, R.L., Fuchs, H.: A comparison of Y., Zhang, Y., Li, Q.: A novel augmented reality system
optical and video see-through head-mounted displays. for displaying inferior alveolar nerve bundles in max-
Telemanipulator Telepresence Technol. 2351, 293–307 illofacial surgery. Sci Rep. 7, 42365 (2017). https://doi.
(1994) org/10.1038/srep42365
Sielhorst, T., Feuerstein, M., Navab, N.: Advanced medical Zinser, M.J., Mischkowski, R.A., Dreiseidler, T., Thamm,
displays: a literature review of augmented reality. O.C., Rothamel, D., Zoller, J.E.: Computer-assisted
J. Disp. Technol. 4(4), 451–467 (2008). https://doi. orthognathic surgery: waferless maxillary positioning,
org/10.1109/JDT.2008.2001575 versatility, and accuracy of an image-guided visuali-
Stetten, G., Chib, V., Hildebrand, D., Bursee, J.: Real time sation display. Brit. J. Oral. Max. Surg. 51(8),
tomographic reflection: Phantoms for calibration and 827–833 (2013a). https://doi.org/10.1016/j.bjoms.
biopsy. In: IEEE and ACM International Symposium 2013.06.014
on Augmented Reality, Proceedings, Los Alamitos, Zinser, M.J., Sailer, H.F., Ritter, L., Braumann, B.,
IEEE Press, 11–19 (2001) Maegele, M., Zoller, J.E.: A paradigm shift in ortho-
Su, L.M., Vagvoigyi, B.P., Agarwal, R., Reiley, C.E., Tay- gnathic surgery? A comparison of navigation,
lor, R.H., Hager, G.D.: Augmented reality during computer-aided designed/computer-aided
robot-assisted laparoscopic partial nephrectomy: manufactured splints, and “classic” intermaxillary
toward real-time 3D-CT to stereoscopic video registra- splints to surgical transfer of virtual orthognathic plan-
tion. Urology. 73(4), 896–900 (2009). https://doi.org/ ning. J. Oral. Maxillofac. Surg. 71(12.), 2151), e2151–
10.1016/j.urology.2008.11.040 e2121 (2013b). https://doi.org/10.1016/j.joms.2013.
Suenaga, H., Tran, H.H., Liao, H.G., Masamune, K., Dohi, 07.007
T., Hoshi, K., Mori, Y., Takato, T.: Real-time in situ
three-dimensional integral videography and surgical
navigation using augmented reality: a pilot study. Int.
J. Oral Sci. 5(2), 98–102 (2013). https://doi.org/10.
1038/ijos.2013.26
Suenaga, H., Tran, H.H., Liao, H., Masamune, K., Dohi,
T., Hoshi, K., Takato, T.: Vision-based markerless reg-
Augmented Reality in Surgery
istration using stereo vision and an augmented reality
surgical navigation system: a pilot study. BMC Med. ▶ Augmented Reality in Image-Guided Surgery
Augmented Reality Ludo Board Game with Q-Learning on Handheld 193

2014). An example of a popular traditional board

Augmented Reality Ludo game is Ludo (Singh et al. 2019). However, the
Board Game with Q-Learning number of traditional board game players has A
on Handheld declined since the appearance of digital games
(Rizov et al. 2019). Hence, the Ludo board game
Mohamad Yahya Fekri Aladin1,2 and can be digitized to retain the existing players
Ajune Wanis Ismail3 while attracting new players.
1
Mixed and Virtual Reality Research Lab, Augmented reality (AR) is a technology that
Vicubelab, Universiti Teknologi Malaysia, combines the real environment with the virtual
Johor Bahru, Malaysia environment or objects created by a computer
2
School of Computing, Faculty of Engineering, (Tan and Soh 2011). This technology is added to
Universiti Teknologi Malaysia, Johor Bahru, a board game to further enhance the gaming expe-
Malaysia rience (Rizov et al. 2019). Nowadays, AR tech-
3
Mixed and Virtual Reality Research Lab, nology is widely used in handheld devices such as
Vicubelab, School of Computing, Faculty of smartphones that are equipped with advanced fea-
Engineering, Universiti Teknologi Malaysia, tures (Zsila et al. 2018). Sensors that can be used
Johor Bahru, Malaysia to capture image, movement, and touch are
equipped by the handheld devices (Grandi et al.
2018) as the rise of the devices’ computational
Synonyms power (Sanches et al. 2019).
An autonomous agent is an agent that can
3D board game; Augmented reality; Ludo game decide to perform an action by itself in a given
environment (Coutinho et al. 2015). The develop-
ment of an autonomous agent is often tightly linked
Definition
to artificial intelligence (AI), especially on
decision-making (Moharir et al. 2019). In games,
An autonomous agent works with artificial intelli-
a non-player character (NPC) can act as an auton-
gence (AI) to decide its actions to adapt and respond
omous agent (Feng and Tan 2016). Common NPCs
to the changes in a dynamic environment. The
found in games are usually scripted with specific
autonomous agent can be developed in games as a
actions and are unable to learn from previous
non-player character to interact with the changes of
actions and adapt to a dynamic environment (Lim
state in the game environment. Traditional board
et al. 2012). Hence, the autonomous agent can be
games such as Ludo have had many players since
developed in a game through a machine learning
olden days but have slowly lost attraction to the
algorithm to adapt to the changes in the environ-
public, especially the younger generations, as digi-
ment, besides increasing the difficulty of the game
tal games become more popular. Although the Ludo
and making it enjoyable (Feng and Tan 2016).
board game can be digitized to fascinate the players
Q-learning is an example of a reinforcement learn-
through implementing augmented reality
ing algorithm under machine learning that allows
(AR) technology on handheld devices, common
the agent to explore the environment and learn the
NPCs found in games have predetermined actions
best action in a specific state (Lee et al. 2017). The
and are unable to learn from experience and adapt to
Q-learning algorithm is commonly used in a
the changes in the game environment. AR Ludo
dynamic environment that involves one or multiple
board game with Q-learning applied in handheld.
players [13]. Hence, the autonomous agent that
works with the Q-learning algorithm can be devel-
Introduction oped for the AR Ludo board game.
AR is defined as a technology that combines real
Board games have appeared since ancient times and virtual contents in the physical environment
with a lot of experienced players (ChePa et al. (Tan and Soh 2011). This technology overlays
194 Augmented Reality Ludo Board Game with Q-Learning on Handheld

digital data on the real environment to present var- • If a player moves a token to the place that is
ious information such as text or images [14]. An AR occupied by the opponent’s token, the oppo-
system focuses on four domains: sensing, tracking, nent’s token is kicked and returned to
interaction, and display [15]. An example of a track- his home.
ing technique is feature-based tracking provided by • A token is safe from being kicked back to
Vuforia Engine to detect, identify, and track targets home once it lands on the safe square. Each
such as images or three-dimensional (3D) objects player’s start node is also considered as a safe
[16]. According to [17], Vuforia analyzes and square.
detects the features of an image uploaded by a • The game ends once all the tokens of a player
user. The more features detected in an image, the have landed on the goal field.
more suitable it is to be used as the image target. The
analyzed result is represented through a star rating The development of autonomous Agent
range from 0 to 5. The highest rating indicates that it A starts with designing an algorithm based on
is the easiest for Vuforia to track the image target. Q-learning as in Fig. 2. Based on Fig. 2, the
algorithm starts from initialization of the values
in the Q-table to zero. After Agent A takes a turn
Ludo Board Game to roll the dice, Agent A can select a random
action through exploration or choose a greedy
A board game is defined as a game that involves a action through exploitation based on the epsilon-
board that allows pieces to move on top of it [18]. greedy (ε-greedy) policy. A greedy action is an
According to (Singh et al. 2019), the Ludo board action with the highest value in the Q-table at the
game involves 2–4 players taking turns to roll a given state. The epsilon value, ε, is declared as
dice and move the tokens on the Ludo board. 1.0 at the beginning and continuously decreases
Figure 1 shows the Ludo board game. for each action taken by Agent A. Random action
The rules for the Ludo board game, as in Fig. 1, is taken if the randomly generated value ranges
are based on [19]: from 0 to 1 is smaller than ε; otherwise, the greedy
action is taken. The reward, r, is then measured
• At the beginning of the game, the players take based on the state-action pair taken. Bellman
turns to roll the dice and a player that rolls a six equation, Q(s,a) ¼ Q(s,a) þ α [r þ γ 〖max〗_
can move a token to the start node. Each player (a^0 ∈ A) Q(s^0 ,a^0 )-Q(s,a)], is used to update the
has four tokens at the beginning of the game. value of the state-action pair taken in the Q-table.
• The players can only move a token according The learning rate, α, and discount factor, γ, are set
to the number rolled if the token is not at as 0.7 and 0.1, respectively. These steps are
the home. repeated until the end of an episode. The training
• Once a player rolls a six, that player can take session is set up for Agent A with 20 episodes. An
another turn. episode is ended once all the four tokens of a
player have successfully moved into the goal
field.

AR Ludo: Augmented Reality Ludo

Board Game

The Ludo board game consists of a dice and four

sets of tokens to match the four different colors of
the home at each corner on the board. However,
Augmented Reality Ludo Board Game with only two sets of tokens are used for the AR Ludo
Q-Learning on Handheld, Fig. 1 Ludo board game [19] board game, as the game is limited to two players.
Augmented Reality Ludo Board Game with Q-Learning on Handheld 195

Augmented Reality Ludo

Board Game with
Q-Learning on
Handheld, A
Fig. 2 Q-learning
algorithm in Ludo
board game

Augmented Reality Ludo

Board Game with
Q-Learning on
Handheld, Fig. 3 AR
Ludo game design

At the beginning of the game, one player is are placed in the home. Next, each player takes a
assigned with the four blue color tokens; mean- turn to roll the dice to move a token from the start
while, the opponent player is assigned with the node into the goal field. The goal of the game is to
four green color tokens. Each player’s four tokens move all the four tokens into the goal field located
196 Augmented Reality Ludo Board Game with Q-Learning on Handheld

Augmented Reality Ludo Board Game with Q-Learning on Handheld, Fig. 4 Virtual Ludo game with Q-learning
algorithm

Augmented Reality Ludo Board Game with Q-Learning on Handheld, Fig. 5 Data stores for Q-learning

at the center of the board to win the game. Based interaction on the mobile screen using a one-
on Fig. 3, there are four purple color nodes on the finger gesture.
board called safe nodes and the dotted line shows The designed Q-learning algorithm has been
an example of a blue tokens’ path from the start implemented in order to generate the optimal
node towards the goal field. values in the Q-table, a training with 20 episodes
In order to implement the AR tracking in a is conducted between Q-learning agent (Agent A)
handheld device, the Vuforia AR software and AR random players, and Agent A is assigned
development toolkit (SDK) is integrated with with the green color tokens. The other three ran-
Unity3D software to track and compare the fea- dom players take random actions to move in the
tures of the marker with the target resource data- game while training. Figure 4 shows the Ludo
base. A PNG image based on the design of the board game training environment performed on
Ludo board in RGB is uploaded to Vuforia Tar- the Windows platform using Unity3D. The game
get Manager. This image is transformed into environment is reset before starting a new episode
grayscale and the features extracted are col- after each episode ends.
lected and stored in the database. The virtual At the end of the training, an output file that
AR Ludo board game is displayed once the includes the finalized values of the Q-table is
marker is detected and the marker’s features generated as in Fig. 5. Based on Fig. 5, the rows
match with the data stored in the database. The represent the states; meanwhile, the columns rep-
AR Ludo includes badges as a gamification ele- resent the actions.
ment to reward the players. Feature-based track- An example is shown in Fig. 6 where the
ing is chosen for AR Ludo in handheld focuses tokens inside the home have the state “In Home”
on Android mobile phones. The users can inter- and the action of moving a token to the start node
act with the application through touch-based from the home is “Out Home.” A token placed on
Augmented Reality Ludo Board Game with Q-Learning on Handheld 197

Augmented Reality Ludo Board Game with Q-Learning on Handheld, Fig. 6 Tokens with the state “In Home” and
action “Out Home” or state “On Free Space” and action “Just Move”

Augmented Reality Ludo Board Game with Q-Learning on Handheld, Fig. 7 User plays AR Ludo to against the
Q-learning agent

a white node that surrounds the board has the state “On Free Space” with a value of approximately
“On Free Space” and the action of moving a token 1.483 based on Fig. 5.
from a white node to another without any oppo-
nent’s token landed on it is “Just Move.” Based on
Fig. 6, Agent A can choose to move a token six AR Ludo on Handheld
steps forward or move a token out from home
while a six is rolled. Agent A decides to move a The setup as in Fig. 7 is that the AR marker is
token out of home as the value in the Q-table for color printed on an A4 size paper and placed on
this state-action is approximately 0.563 that is the table with a height of approximately 75 cm.
higher than the action “Just Move” at the state The user sits approximately 30 cm in front of the
198 Augmented Reality Ludo Board Game with Q-Learning on Handheld

Augmented Reality Ludo Board Game with Q-Learning on Handheld, Fig. 8 AR Ludo board game on handheld

table with one hand holding the handheld device References

and the other hand interacting with the AR Ludo
board game through touchscreen input. The hand- ChePa, N., Alwi, A., Din, A.M., Mohammad, S.: Digitiz-
ing Malaysian traditional game: e-Congkak. In:
held device with a camera used to track and dis-
Knowledge Management International Conference
play the AR Ludo board game is approximately (KMICe), Malaysia 2014
45 cm away from the marker. Coutinho, L.R., Galvão, V.M., Batista adA, Moraes, B.R.
The 3D Ludo board is developed with numer- S., Fraga, M.R.M.: Organizational gameplay: the
player as designer of character organizations. Int.
ous cubes scaled in different sizes and colors of
J. Comp. Games Technol. 2015, 11 pages (2015).
material. Each corner of the Ludo board has dif- https://doi.org/10.1155/2015/731031
ferent colors, either yellow, blue, green, or red to Feng, S., Tan, A.-H.: Towards autonomous behavior learn-
represent each player’s home. There are four pur- ing of non-player characters in games. Expert Syst.
Appl. 56, 89–99 (2016)
ple color nodes that represent the safe nodes on
Grandi, J.G., Debarba, H.G., Bemdt, I., Nedel, L., Maciel,
the Ludo board, and each player’s start node is set A.: Design and assessment of a collaborative 3D inter-
as the node that is nearest to the home with the action technique for handheld augmented reality. In:
same color. A token is created using a capsule, a 2018 IEEE Conference on Virtual Reality and 3D
User Interfaces (VR), pp. 49–56 (2018)
3D game object provided in Unity3D. In order to
Lee, J., Kim, T., Kim, H.J.: Autonomous lane keeping
show a token can interact, an arrow model is based on approximate Q-learning. In: 2017 14th Inter-
imported into Unity3D to attach with the capsule. national Conference on Ubiquitous Robots and Ambi-
The dice model is also imported into the game. ent Intelligence (URAI), pp 402–5 (2017)
Lim, M.Y., Dias, J., Aylett, R., Paiva, A.: Creating adaptive
The UI of the AR Ludo board game is developed affective autonomous NPCs. Auton. Agent. Multi-
with buttons, checkboxes, and panels for interac- Agent Syst. 24, 287–311 (2012)
tion. The AR tracking is also enabled in handheld. Moharir, M., Mahalakshmi, A.S., Kumar, G.P.: In: Kumar,
The AR Ludo board game allows the human R., Wiil, U.K. (eds.) Recent Advances in Computationa
Intelligence, pp. 255–261. Springer International Pub-
player to play with the opponent player, Agent lishing, Cham (2019)
A. The human player is represented as the blue Rizov, T., Đokić, J., Tasevski, M.: Design of a board game
color tokens; meanwhile, the Agent A is with augmented reality FME. Transactions. 47,
represented as the green color tokens. On a 253–257 (2019)
Sanches, S.R., Oizumi, M.A., Oliveira, C., Sementille,
human player’s turn, the player can select the A.C., Corrêa, C.G.: The influence of the device on
dice to roll or select a token to move as in Fig. 8 user performance in handheld augmented reality. SBC
by touchscreen input after rolling a six. J. Interact. Syst. 10, 43–51 (2019)
Automated Game Design Testing Using Machine Learning 199

Singh, P.R., Elaziz, M.A., Xiong, S.: Ludo game-based Definitions

metaheuristics for global and engineering optimization.
Appl. Soft Comput. 84, 105723 (2019)
Tan, C.T., Soh, D.: Augmented reality games: a review EA Electronic Arts A
(2011). https://www.researchgate.net/profile/Chek- NFT Non-Fungible Tokens
Tien-Tan/publication/260480270_Augmented_Real CEO Chief Executive Officer
ity_Games_A_Review/links/56fb0b0708ae3c0f264c0 MMO Massively Multiplayer Online Game
8b3/Augmented-Reality-Games-A-Review.pdf
Zsila, Á., Orosz, G., Bőthe, B., Tóth-Király, I., Király, Game
O., Griffiths, M., Demetrovics, Z.: An empirical study
on the motivations underlying augmented reality Introduction
games: the case of Pokémon Go during and after
Pokémon fever. Personal. Individ. Differ. 133,
56–66 (2018) Games design is the field of study and practice of
improving and measuring the structure and sys-
tems found within an analogue or digital game.
The process of improving a game throughout the
development and lifecycle of a game is key to the
Augmented Virtuality survival of the game. Notable examples include
changes found within the rules of chess through-
▶ Mixed Reality out its long lifespan and different iterations and
▶ Virtual Reality and Robotics versions of the game (Murray 2015). Game
designers incrementally change the game to pro-
vide better entertainment to its participating
players. Game designer and developer Robert
Zubek defines game design by breaking it down
Auralization into its elements, which he says are the following
(Zubek 2020):
▶ Sound Spatialization
• Gameplay, which is the interaction between the
player and the mechanics and systems
• Mechanics and systems, which are the rules
Autism and objects in the game
• Player experience, which is how users feel
▶ Computer Games for People with Disability when they are playing the game

Improving a game design can be achieved

through the use of analytical or experimental
Automated Game Design processes.
Testing Using Machine
Learning Paper Overview
This paper focusses on the different strategies
Conor Stephens and Chris Exton and examples of the use of deep learning tech-
University of Limerick, Limerick, Ireland niques when testing game design. This focusses
on two aspects. A suitable use case of player
replacements is to simulate experimental game
Synonyms design approaches such as automated play test-
ing and content evaluation. Furthermore, this
Game Design Evaluating Using Machine paper concludes with a brief summary of
Learning research measuring sentiment toward automated
200 Automated Game Design Testing Using Machine Learning

game design testing processes by industry changes are moved to the live game environment.
stakeholders. Through experiments, designers can collect qual-
itative and quantitative data, which they can use to
Types of Game Design Techniques make informed choices about the trajectory and
Both analytical and experimental game design changes that will be made to the game.
processes are resource intensive through the
required analysis and participation of the player
within the design process. Game publishers and Analytical Game Design Improvements
academics have numerous testing approaches to
accelerate the creation of games and improve the Analytical design improvement is through the col-
quality of their design in each part of the game lection and comparison of numbers from game
lifecycle. Early attempts showed how games are elements directly from the game content, such as
controlled by procedural code such as behavior weapon parameters or gameplay metrics generated
trees (Sekhavat 2017). from players playing the game. Using techniques
Large companies such as Ubisoft (Roy et al. such as cost curve analysis, game designers can
2022) and EA (Bergdahl et al. 2021; Sestini et al. accurately measure the fairness and effectiveness
2022) have pioneered several approaches toward of all the content available from the game. Tools
intelligent systems within the games development and techniques to evaluate the validity of game
process allowing the assessment of games design parameters have evolved over the years. A new
such as automated playtesting using deep rein- addition to the game designer tool belt are machi-
forcement learning agents, as well as the creation nation diagrams, a visual-graph-based language
of artistic content using neural networks capable allowing designers to test the design of different
of creating state-of-the-art animation controllers game systems (Adams and Dormans 2012). Ana-
that were trained using unsupervised learning lytical methods follow a reductive process that
techniques applied to motion capture data, prov- allows designers to see the impact of changes
ing the real-world interest in the applications of immediately within the games’ internal systems.
intelligent tools in games development. Cost curve analysis lists every mechanic and item
within a modern game as an item with a benefit and
a cost. As the benefit of an item increases, so does
Experimental Game Design the cost Carpenter (2003) value. Designers place
the cost increases of an item in line with the benefit
Experimental game design is immensely popular of a game, mechanic, or resource, and each item
in both early development and later content within the world has been referred to as Jedi Curve
updates (Waern and Back 2015). Toward the end within Magic The Gathering Credits (2012). The
of the development of a new game or feature, cost–benefit relationship between each item forms
designers with programming skills focus on the shape of the curve. During the balancing pro-
rapid prototyping to see how the newly created cess, a designer can use it to see where a specific
content fits within the existing content and how component of the game fits within the system.
the implementation achieves the esthetic goals the Small imbalances are allowed, as choosing the
designers have for the new feature during most efficient or effective item creates interesting
gameplay. Later within the development cycle of decisions for the players to choose from.
the game, publishers can create experimental ver-
sions of the game that players can opt-in to try and
explore. This allows designers to test new features Player Replacements
on a live player base that is aware that they may
experience uncompetitive and less polished expe- This section outlines the different approaches to
riences, with the benefit being that they are early implementing player replacements, which can
adopters and will be more prepared when the generate gameplay data for use in automated
Automated Game Design Testing Using Machine Learning 201

Automated Game Design Testing Using Machine Learning, Fig. 1 Crush Saga: Deep Learning AI (Gudmundsson
et al. 2018)

playtesting and experimental game design behavior as a problem domain for reinforcement
workflows. Player replacements refer to the use learning, such as having learning agents design
of artificial intelligence agents that simulate the levels (Zhang et al. 2017; Khalifa et al. 2020) or
player playing the game, making conventional using cutting-edge reinforcement learning algo-
and advanced game testing easier and faster. rithms designed to emulate player behavior to
New approaches have been proposed to test the generate gameplay data by simulating the game
design and balance of video games to improve being played (Holmgård et al. 2014; Holmgård
iteration time and precision using automation, et al. 2015). Previous research has shown real
and replacements for players have been proposed. progress in testing single-player games. Minimal
Notable examples have been proposed by the research has been conducted on the use of
AAA games studio King Entertainment publisher machine learning techniques to accelerate the
of the popular “Crush Saga” games. King design process of multiplayer games.
researched the evaluation of match-3 level design Researchers have shown how mathematical rep-
through predicted play test results using machine resentations of simple games can be used to assess
learning techniques (Gudmundsson et al. (2018) the difficulty of the game for different demo-
shown in Fig. 1). Electronic Arts (EAs) have graphics of players (Lee et al. 2021). Research
recently proposed a framework for testing their was carried out on target–movement games,
games using learning agents to accurately reflect which are a popular genre for mobile games that
the goals of designers and the demographics of the include Flappy Bird and Temple Run. Evaluating
players found in each game (Zhao et al. 2019). the difficulty is possible by collecting cognitive data
Academic organizations and researchers have from specific player bases and using response times
proposed a broad variety of ideas for how artificial as input into the mathematical representation of the
intelligence (AI) can help game designers create game. Answers to player demographics, response
games by treating game balance as an optimiza- time, and representation of the game allow games to
tion problem and using AI to reduce the possible be tuned for different skilled players.
search space, an example of this is by parameter
tuning a game “Shoot ‘Em Up” using active learn-
ing techniques (Zook et al. 2019). Another para- Automated Playtesting
digm to accelerate the creation and testing of
games that has received attention in academic Playtesting a common practice of Analytical
research is to consider game design or player Game Design is an integral component of user
202 Automated Game Design Testing Using Machine Learning

testing, a fun and engaging game, designed to extensions have incorporated different types of
review design decisions within interactive player motivation into reinforcement learning
media, such as games, to measure the esthetic algorithms by including an additional reward sig-
experience of the game compared to the nal that emulates the type of motivation for each
designer’s intentions (Fullerton 2008). Playtesting type of player (Ariyurek et al. 2021).
is a lengthy process in terms of both complexity,
cost, and time. Several approaches have been
presented over the past few years to automate Player Replacements Deep
individual sections of this process (Stahlke and Reinforcement Learning
Mirza-Babaei 2018). These solutions aim to allow
faster, more accurate iterations of design and less Work by Google highlights the strength of using
waste during development within a professional machine learning to evaluate game design, and
game development cycle. Google researchers developed a prototype game
Automated playtesting approaches have been called Chimera (Hun Kim and Wu 2021). Google’s
pioneered by researchers and game studios to research focusses on balancing the game by simu-
provide automated analytical testing of modern lating millions of games using a deep reinforcement
video games. Computer-controlled agents gener- learning agent. One key issue with this research is
ate data for analysis by designers and stakeholders the iterative approach and the analysis required
by playing the game at accelerated speeds and in before making changes. This tool shows the
parallel. Intelligent agents provide greater cover- power of machine learning that allows designers
age of available game states, especially in open- to test an iteration, evaluate the gameplay metrics,
ended game worlds, and are implemented using and then re-test the game with appropriately
deep neural networks (Bedder 2019). An excel- designed changes, but does not help designers
lent example of automated playtesting using rein- move the game easily and quickly to a more optimal
forcement learning was developed by Prowler.io composition. We believe that this work is a good
to balance two-player games using soft Q-learning example of how to automate testing in multiplayer
(Grau-Moya et al. 2018) that allowed automated games. The game is not team but features asymmet-
playtests to occur in multiplayer environments. ric gameplay, an example of such a project that was
not achieved both at the start of this research.
Approximately, a hundred repetitions are
Player Replacements & Personas required during the game testing phase of the
development cycle to ensure that user interactions
Personas have a long history within game design; produce the expected result (TestDel 2019). A key
separating what motivates players into categories component of this difficulty is the reliance on
allows the game to be assessed for different con- human observations to find bugs (Politowski
sumer groups that purchase games for different et al. 2021), and excellent research by Ubisoft
reasons; each internal driver influences how uses computer vision combined with player
players interact with the game and can be incor- replacements (Paduraru et al. 2021) to detect
porated into automated game design testing prac- issues without human intervention. Intelligent
tices. Personas allow the inclusion of different player replacements are able to explore the
motivations players have when playing games, breadth and depth of the different available game
and personas have been shown to extensively states, and a trained computer vision model per-
test various single-player games and allow testing forms anomaly detection on the screen output that
in asymmetric multiplayer games with different would be presented to the player. A benefit of this
playable characters (Holmg et al. 2018). Early approach is that issues that are flagged to the
examples of persona-based player replacements developers during the testing process are immedi-
were powered by Monte Carlo tree search and ately noticeable to the player and allow the prior-
genetic programming techniques. Modern ity of the issue; similar techniques that observe the
Automated Game Design Testing Using Machine Learning 203

state of the game world may identify issues that adversarial network (GAN) that is popular for
are not noticeable to players. Issues identified content generation due to the high levels of fidel-
using computer vision are highlighted with ity that can be generated from them. This is con- A
bounding boxes, allowing the developer tasked tributed due to the adversarial nature of the
with investigating the issue to easily see the anom- network architecture; GANs (Goodfellow et al.
aly alongside a recording of the required input to 2014) have two neural networks, one to generate
recreate the issue, saving hours of debugging and the content and a second to evaluate how accurate
investigation. the output is; GANs are often taught to students
such as counterfeiters and police agents. As the
counterfeiter improves creating forgeries, the
police network becomes better at discovering
Assessing Content with Deep Learning
fakes or outputs that are not good enough for use
case within the game.
Approaches to assess generated content that can be
The most recent work in this area and the most
achieved using statistical measures have received
relevant to this research was conducted by Liapis
continued support; work by Lucas and Volz (2019)
et al. (2019) and explored using deep learning
has shown how computer vision models can be
techniques to evaluate procedural generated con-
used to generate procedural levels for Super
tent. This work is spread over four key papers.
Mario. The work uses adversarial neural networks
This work uses deep learning to predict the out-
to assess and generate new levels using two sepa-
come of games from generated data sets. The first
rate neural networks trained simultaneously.
paper titled Using a Surrogate Model of
Which can give insight into the generative space of Gameplay for Automated Level Design refers to
a content generator and its biases within that space.
this neural network as a “Surrogate Model of
(Liu et al. 2020.
Gameplay” (Karavolos et al. 2018), and this ter-
Generated content, such as the level’s geome- minology is carried throughout the 4 papers.
try, can be assessed through artificial play of the Liapis et al. create a data set to solve several
generated content. This pursuit has resulted in the supervised learning problems by simulating
generation of different reinforcement learning games within a first-person shooter by having
agents that can simulate the playtesting of PCG agents play against each other using behavior
(Procedural Content Generation) games. This trees. Behavior trees define each agent’s policy
approach has been shown to evaluate Mario levels as a branching tree that takes its state from the
generated through an adversarial PCG pipeline environment. This decision results in each agent’s
(Volz et al. 2018). Automating the testing process policy being constant throughout the data
can give PCG techniques quick insights into the generation.
quality of the generated content. Liapis et al.
(2019) approach for generating levels for certain Metanomic: Game Economy Manager
game outcomes using PCG uses AI agents to With the rise of NFTs and blockchain games,
assess the quality of the generated levels. The managing economies have grown in importance
work of Baron analyzes different PCG techniques as well as intercompatibility. “Metanomic’s game
to generate dungeons using analytical methods economy infrastructure allows developers to
and allows the adaptation of different algorithms build, simulate, and run balanced economies and
to adjust the content created for different esthetic core gameplay loops in a live, realtime environ-
experiences (Baron 2017). ment” – Theo Priestley, CEO of Metanomic.
The first type of network composition explored The dashboard and analytics tools allow finer-
is LSTMs, a popular type of RNN (Hochreiter and grained controls of the MMO world economies to
Schmidhuber 1997), which are used to generate understand the current supply and interactions
time sequence data such as text, music, and sound. players are having with the world’s resources.
The second type of network is a generative An image of the dashboard can be seen in Fig. 2,
204 Automated Game Design Testing Using Machine Learning

Automated Game Design Testing Using Machine Learning, Fig. 2 Metanomic Dashboard

which shows the current inventory of a user and Toward Automated Video Game Testing
their character’s history in the game. Metanomic
integrates with machination diagrams and loot Exploratory work was carried out to explore the
tables to provide a comprehensive economy man- current sentiment toward automated game design
ager for designers; integration with existing game techniques within the broader game development
engines is not currently integrated, preventing industry (Politowski et al. 2022). The research
game mechanics from altering the economic sys- concluded that the desire for automated testing
tems, a key source of exploits. Metanomic has tools and approaches for game development was
acquired Intoolab (announced on the 18th of gaining traction among the survey participants;
May 2022), an artificial intelligence company however, the difficulties of incorporating suitable
that specializes in providing intelligent data anal- techniques into existing workflows remain pro-
ysis on game economies. hibitive toward wider adoption (Politowski et al.
Automated Game Design Testing Using Machine Learning 205

Automated Game Design Testing Using Machine Learning, Table 1 Selected papers for survey, improving game
testing with reinforcement learning (Politowski et al. 2022)
Author Test Obj. Game tested
A
Gudmundsson et al. (2018) Balancing Match 3
Roohi et al. (2020) Balancing Puzzle
Gordillo et al. (2021) Exploration 3D third person
Holmgård et al. (2014) Exploration Doom
Zheng et al. (2019) Exploration MMO
Pfau et al. (2017) Finding bugs Adventure
Ariyurek et al. (2020) Finding bugs 3D adventure

Automated Game Design Testing Using Machine Learning, Fig. 3 Histogram of all the 166 papers studying
automated game testing over time, improving game testing with reinforcement learning (Politowski et al. 2022)

2022). Politowski et al. (2022) surveyed the sen- This survey has clean insights into perceptions
timent of developers toward a variety of different of automated testing in the game’s industry
automated testing approaches from academic despite the small sample size. The technical
research. This was achieved by summarizing the requirements and processes for each paper were
research and asking developers how desirable, captured in the survey questions, and participants
viable, and feasible the research would be were asked their interest in incorporating or using
included in their workflow. The final research each technique within their workflow. Some key
papers that were included in the questionnaire quotations are given from the survey responses;
are included in Table 1. The included research an example of this is shown below.
are the filtered results of 166 papers, with
80 papers applying automated solutions to test The biggest bang for the buck would be as a build
acceptance test on a CI/CD pipeline, making sure
video games. Each article was separated into the catch obvious blocking bugs. Otherwise, it
semi-automatic and automatic methods; a histo- decreases significantly in usefulness. (Politowski
gram of the articles over time can be seen in Fig. 3. et al. 2022.
206 Automated Game Design Testing Using Machine Learning

This comment is a good assessment of the either class imbalance issues during the game
current consensus on industry sentiment. The big- balancing process or a lack of data during the
gest obstacle to automated testing is the technical assessment process.
burden that incorporating these features into the
game code carries when simply managing a build
server brings about significantly more benefits.
Cross-References
Other groups have seen this issue around incorpo-
rating automated testing into game development
▶ Area of Interest Management in Massively
by lightening the burden on the developers by
Multiplayer Online Games
packaging the automation as a game engine tool,
▶ Computer Games and Artificial Intelligence
which has no technical burden on the game source
▶ Game Player Modeling
code. An example is modl.ai’s glitch finder, which
▶ Machine Learning for Computer Games
plays the game through Unreal Engine’s input
▶ Overview of Artificial Intelligence
API and does not add code to the game objects;
this framework-styled automation system is
wonderful for brute-force tests but falls short
References
of the requirements that would be needed
when introducing reinforcement learning player Adams, E., Dormans, J.: Game Mechanics: Advanced
replacements. Game Design, 1st edn. New Riders Publishing, Berke-
ley (2012)
Aponte, M.V., Levieux, G., Natkin, S.: Measuring the level
of difficulty in single player video games. Entertain.
Conclusion Comput. 2, 205 (2009). https://doi.org/10.1016/j.
entcom.2011.04.001
A variety of different aspects of game design have Ariyurek, S., Betin-Can, A., Surer, E.: Enhancing the
been explored by automated testing approaches Monte Carlo tree search algorithm for video game
testing. In: 2020 IEEE Conference on Games (CoG),
that range from difficulty prediction (Aponte et al. pp. 25–32 (2020). https://doi.org/10.1109/CoG47356.
2009; Gudmundsson et al. 2018; Lee et al. 2021) 2020.9231670
that assesses how many lives it would take a Ariyurek, S., Sürer, E., Betin-Can, A.: Playtesting: what is
player to complete new content and tailor games beyond personas. CoRR abs/2107.11965. https://arxiv.
org/abs/2107.11965 (2021)
to different demographics or intended experi- Baron, J.R.: Procedural dungeon generation analysis and
ences, to higher-level game design that assesses adaptation. In: Proceedings of the SouthEast Conference,
the different options available to players and the Association for Computing Machinery, New York, NY,
viability of the various intended strategies that can USA, ACM SE ‘17, pp. 168–171 (2017). https://doi.org/
10.1145/3077286.3077566
be played (Pfau et al. 2020). Automating higher- Bedder, M.: AI tools for automated game testing. https://web.
level game design metrics provides greater value archive.org/web/20200805074140/ https://www.prowler.
for the production team; however, it comes with io/blog/ai-tools-for-automated-game-testing (2019)
greater complexity to implement and compute to Bergdahl, J., Gordillo, C., Tollmar, K., Gisslén, L.:
Augmenting automated game testing with deep rein-
assess. Testing of mechanics and systems, such as forcement learning. CoRR abs/2103.15819. https://
balancing intransitive relationships, is important; arxiv.org/abs/2103.15819 (2021)
areas of game design that have not been explored Carpenter, A.: Applying risk analysis to play-balance
focus on both the dynamics, such as the pacing of RPGs Credits E (2012) Perfect imbalance – why unbal-
anced design creates balanced play. https://www.
the game session alongside the frequency and youtube.com/watch?v¼e31OSVZF77w, in comment
location of key gameplay moments, which are section (2003). Accessed 26 Oct 2021
influenced by player cooperation and strategy Fullerton, T.: Game Design Workshop. A Playcentric
between different playable characters within Approach to Creating Innovative Games,
pp. 248–276 (2008). https://doi.org/10.1201/b22309
multiplayer games. Persistent game worlds have Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B.,
little exploration due to the difficulty of exploring Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.:
the search space by learning agents, which causes Generative adversarial nets. In: Proceedings of the 27th
Automated Game Design Testing Using Machine Learning 207

International Conference on Neural Information Pro- Murray, H.: A History of Chess: The Original 1913 Edi-
cessing Systems – volume 2, MIT Press, Cambridge, tion. Skyhorse. https://books.google.ie/books?
MA, USA, NIPS’14, pp. 2672–2680 (2014) id¼3GzujwEACAAJ (2015)
Gordillo, C., Bergdahl, J., Tollmar, K., Gisslén, L.: Improv- Paduraru, C., Paduraru, M., Stefanescu, A.: Automated A
ing playtesting coverage via curiosity driven reinforce- game testing using computer vision methods. In: 2021
ment learning agents. CoRR abs/2103.13798. https:// 36th IEEE/ACM International Conference on Auto-
arxiv.org/abs/2103.13798 (2021) mated Software Engineering Workshops (ASEW),
Grau-Moya, J., Leibfried, F., Bou-Ammar, H.: Balancing pp. 65–72 (2021). https://doi.org/10.1109/
two-player stochastic games with soft Q-learning. ASEW52652.2021.00024
CoRR abs/1802.03216 (2018) Pfau, J., Smeddinck, J.D., Malaka, R.: Automated game
Gudmundsson, S.F., Eisen, P., Poromaa, E., Nodet, A., testing with ICARUS: intelligent completion of adven-
Purmonen, S., Kozakowski, B., Meurling, R., Cao, L.: ture riddles via unsupervised solving. In: Extended
Human-like playtesting with deep learning. In: 2018 Abstracts Publication of the Annual Symposium on
IEEE Conference on Computational Intelligence and Computer-Human Interaction in Play, Association for
Games (CIG), pp. 1–8 (2018). https://doi.org/10.1109/ Computing Machinery, New York, NY, USA, CHI
CIG.2018.8490442 PLAY ‘17 Extended Abstracts, pp. 153–164 (2017).
Hochreiter, S., Schmidhuber, J.: Long short-term memory. https://doi.org/10.1145/3130859.3131439
Neural Comput. 9(8), 1735–1780 (1997) Pfau, J., Liapis, A., Volkmar, G., Yannakakis, G.N.,
Holmg, C., Green, M.C., Liapis, A., Togelius, J.: Auto- Malaka, R.: Dungeons & replicants: automated game
mated playtesting with procedural personas through balancing via deep player behavior modeling. In: 2020
MCTS with evolved heuristics. CoRR IEEE Conference on Games (CoG),
abs/1802.06881 (2018) pp. 431–438 (2020). https://doi.org/10.1109/
Holmgård, C., Liapis, A., Togelius, J., Yannakakis, G.N.: CoG47356.2020.9231958
Evolving personas for player decision modeling. In: Politowski, C., Petrillo, F., Guéhéneuc, Y.: A survey of
2014 IEEE Conference on Computational Intelligence video game testing. CoRR abs/2103.06431. https://
and Games, pp. 1–8 (2014) arxiv.org/abs/2103.06431 (2021)
Holmgård, C., Liapis, A., Togelius, J., Yannakakis, G.: Politowski, C., Guéhéneuc, Y.G., Petrillo, F.: Towards
Evolving models of player decision making: personas automated video game testing: still a long way to
versus clones. Entertain Comput. 16 (2015). https://doi. go. arXiv e-prints arXiv:2202.12777 (2022)
org/10.1016/j.entcom.2015.09.002 Roohi, S., Relas, A., Takatalo, J., Heiskanen, H.,
Hun Kim, J., Wu, R.: Leveraging machine learning for Hämäläinen, P.: Predicting game difficulty and churn
game development. https://ai.googleblog.com/ without players. CoRR abs/2008.12937. https://arxiv.
2021/03/leveraging-machine-learning-for-game. org/abs/2008.12937 (2020)
html (2021) Roy, J., Girgis, R., Romoff, J., Bacon, P.L., Pal, C.J.: Direct
Karavolos, D., Liapis, A., Yannakakis, G.N.: Using a sur- behavior specification via constrained reinforcement
rogate model of gameplay for automated level design. learning. In: Chaudhuri, K., Jegelka, S., Song, L.,
In: 2018 IEEE Conference on Computational Intelli- Szepesvari, C., Niu, G., Sabato, S. (eds.) Proceedings
gence and Games (CIG), pp. 1–8 (2018) of the 39th International Conference on Machine
Khalifa, A., Bontrager, P., Earle, S., Togelius, J.: PCGRL: Learning, PMLR, Proceedings of Machine Learning
procedural content generation via reinforcement learn- Research, vol. 162, pp. 18828–18843 (2022) https://
ing. CoRR abs/2001.09212. https://arxiv.org/abs/2001. proceedings.mlr.press/v162/roy22a.html
09212 (2020) Sekhavat, Y.: Behavior trees for computer games. Int.
Lee, I., Kim, H., Lee, B.: Automated Playtesting with a J. Artif. Intell. Tools. 26 (2017). https://doi.org/10.
Cognitive Model of Sensorimotor Coordination, 1142/S0218213017300010
pp. 4920–4929. Association for Computing Machin- Sestini, A., Gisslén, L., Bergdahl, J., Tollmar, K.,
ery, New York (2021). https://doi.org/10.1145/ Bagdanov, A.D.: CCPT: automatic gameplay testing
3474085.3475429 and validation with curiosity-conditioned proximal tra-
Liapis, A., Karavolos, D., Makantasis, K., Sfikas, K., jectories. https://doi.org/10.48550/ARXIV.2202.
Yannakakis, G.: Fusing level and ruleset features for 10057. https://arxiv.org/abs/2202.10057 (2022)
multimodal learning of gameplay outcomes, Stahlke, S.N., Mirza-Babaei, P.: Usertesting without the
pp. 1–8 (2019). https://doi.org/10.1109/CIG.2019. user: opportunities and challenges of an AI-driven
8848015 approach in games user research. Comput. Entertain.
Liu, J., Snodgrass, S., Khalifa, A., Risi, S., Yannakakis, 16(2), 1 (2018). https://doi.org/10.1145/3183568
G.N., Togelius, J.: Deep learning for procedural content TestDel: Why is it so difficult to test games? https://testdel.
generation. Neural Comput. Appl. 33, 19 (2020). medium.com/why-is-it-so-difficult-to-test-games-
https://doi.org/10.1007/s00521-020-05383-8 955d643ba55e (2019)
Lucas, S.M., Volz, V.: Tile pattern KL-divergence for Volz, V., Schrum, J., Liu, J., Lucas, S.M., Smith, A.M., Risi,
analysing and evolving game levels. CoRR S.: Evolving Mario levels in the latent space of a deep
abs/1905.05077. http://arxiv.org/abs/1905.05077 convolutional generative adversarial network. CoRR
(2019) abs/1805.00728. http://arxiv.org/abs/1805.00728 (2018)
208 Automated Image Captioning for the Visually Impaired

Waern, A., Back, J.: Experimental Game Design, Variational A machine learning
pp. 341–353. ETC Press, Pittsburgh (2015) Auto-Encoder architecture that encodes input
Zhang, H., Wang, J., Zhou, Z., Zhang, W., Wen, Y., Yu, Y.,
Li, W.: Learning to design games: strategic environ- (VAE) data into a representative,
ments in deep reinforcement learning. CoRR lower-dimensional feature
abs/1707.01310. http://arxiv.org/abs/1707.01310 (2017) space, then decodes it into the
Zhao Y, Borovikov I, Beirami A, Rupert J, Somers C, desired output format.
Harder J, De Mesentier Silva F, Kolen J, Pinto J,
Pourabolghasem R, Chaput H, Pestrak J, Sardari M, Convolutional A neural network that uses an
Lin L, Aghdaie N, Zaman K.: Winning isn’t every- Neural image’s convolution pyramid
thing: Enhancing game development with intelligent Network to extract its features into a
agents. (2019) (CNN) vector representation.
Zheng, Y., Xie, X., Su, T., Ma, L., Hao, J., Meng, Z., Liu,
Y., Shen, R., Chen, Y., Fan, C.: Wuji: automatic online Recurrent A neural network that is fed
combat game testing using evolutionary deep rein- Neural data in sequence and retains
forcement learning. In: 2019 34th IEEE/ACM Interna- Network information from previous
tional Conference on Automated Software Engineering (RNN) iterations when calculating the
(ASE), pp. 772–784 (2019). https://doi.org/10.1109/
ASE.2019.00077 output of the current one.
Zook, A., Fruchter, E., Riedl, M.O.: Automatic playtesting
for game parameter tuning via active learning. Introduction
1908.01417 (2019)
Zubek, R.: Elements of Game Design. MIT Press (2020)
https://books.google.ie/books?id¼0s_tDwAAQBAJ Image content in documents can provide impor-
tant context about the topic discussed within.
Though the visually impaired can’t access visual
content directly, the information images contain
can be summarized through captioning. Captions
Automated Image Captioning can be conveyed through any conventional
for the Visually Impaired method by which the visually impaired access
text; these primarily include braille readers and
Nicholas Bode1 and Mahadeo Sukhai2,3 text-to-speech systems. Many documents come
1
Ontario Tech University, Oshawa, ON, Canada pre-captioned however, the captioning tends to
2
ARIA Team, Canadian National Institute for the be geared towards clarifying image relevance for
Blind, Toronto, ON, Canada sighted individuals rather than describing the con-
3
CNIB Foundation, Kingston, ON, Canada tents themselves. To remedy this, automated
captioning systems have been and are being
developed to give the visually impaired some
Synonyms access to this type of content.

Caption generation; Image captioning; Visual

impairment Designs of Image Captioning Systems

Most modern image captioning systems rely on a

Definition VAE architecture. Many alternatives existed
before the popularization of neural networks
Visually Used in this context to refer to (Kulkarni et al. 2011) but they have since been
impaired any significant reduction in either replaced by or enhanced with VAEs.
vision that cannot be The first neural image captioning system
compensated for using (Karpathy and Fei-Fei 2015) used a CNN to
conventional corrective lenses. encode image features and an RNN to generate
Includes total blindness. text captions word by word. While the basic
Automated Image Captioning for the Visually Impaired 209

architecture remains the same, many tweaks to detailed information about individual actors. They
this system have been developed since. also need to be able to handle more complex
A common extension is to include an attention interactions between them. A
mechanism (Xu et al. 2015) or even replace the The addition of object detection before convo-
RNN and/or the CNN with a Transformer (Liu lution as in Neural Baby Talk (Lu et al. 2018)
et al. 2021). allows for more detailed information about indi-
Encoding is the step that has undergone the vidual actors in the scene. This preprocessing step
most drastic changes. These changes primarily significantly increases performance on more
take the form of various pre-processing steps. sophisticated image captioning datasets.
Neural Baby Talk (Lu et al. 2018) first applies an Another implementation (Makav and Kılıç
object detection algorithm to the source image 2019) uses a significantly more complicated
before feeding it into the CNN. architecture for the image convolution phase.
The architecture in question is VGG16
(Simonyan and Zisserman 2014) which combines
Adaptation for the Visually Impaired image convolution layers with fully connected
deep learning layers to allow for higher-level fea-
While automated image captioning may not have ture extraction.
been designed with the visually impaired in mind,
its connection is obvious. Several applications Combining Approaches
have been specifically designed to aid the visually Many applications geared towards the visually
impaired in navigating visual media online and in impaired lean heavily on a data-centric approach
the real world. These rely on two types of methods to improvement. While this is an important step,
which are not mutually exclusive. the models being used are not sophisticated
enough to capture the high-level language used
Data-Oriented Methods in the training data. As a result, these projects
A key factor for the successful adoption of these perform only as well as the associated model
models is training on image captioning datasets would on a normal dataset.
with labels better suited towards aiding the visu- Occasionally performance is even reduced in
ally impaired. These captions should contain not comparison to those on standardized datasets
just a basic scene description but also include because the model develops a tendency for mem-
detailed information about the scene and its orization due to its inability to learn the complex-
actors. ities present. A common outcome of this is object
One model (Elamri and de Planque 2016) uses hallucination (Rohrbach et al. 2018) where
a very similar architecture to the basic one objects that often appear together in the same
described here, with a CNN leading into an image both appear in the caption, even when one
LSTM but is trained on the Microsoft COCO is not in the image being captioned.
Caption dataset (Chen et al. 2015) which uses
strong descriptive language in its labels.
Another (Ahsan et al. 2021) uses the Vizwiz Limitations of Current Systems
dataset which contains images labeled in concert
with visually impaired individuals. This training Image captioning systems used today have a few
data includes all-important image features that important limitations that need to be addressed.
would assist the visually impaired in understand- The most prominent of these is that captions are
ing a scene. generally simple and cannot explain interactions
in detail. This issue is in large part a matter of
Model-Centric Methods limited compute resources and model complexity.
Better data is insufficient for solving this problem. The solution of applying an object detection
Models also need to be extended to include more algorithm before convolution remedies this
210 Automated Image Captioning for the Visually Impaired

somewhat but in turn limits the amount of infor- Unfortunately, this would likely result in the text
mation carried about the scene as a whole. This itself being lost in the process.
type of general information would be important to An alternative solution would be to use object
convey to a visually impaired individual. detection and then explicitly connect the observed
Another issue arises in situations where there is text to its corresponding bounding box. This
text contained within images. While text extrac- would certainly better capture the contents of the
tion from images is very nearly a solved problem, text but may result in poorly phrased captions in
no existing solutions take that data into account unusual circumstances.
when generating captions. Without information
about the text contained in the scene, the vast Document-Aware Captions
majority of an image’s meaning could be lost. While images often exist independently online,
These are the primary issues with modern the visually impaired will rarely interact with
image captioning systems designed for the visu- them. In general, most images would be encoun-
ally impaired. tered within another document or web page.
Leveraging this fact could allow the detection
of named entities within the text and connecting
Next Steps them to objects detected within the image. This
could allow the caption to contain relevant con-
Addressing these limitations is by no means a textual information such as the actual name of a
simple task. As one tries to train a model to learn given person or object rather than just the type of
more complex representations, the required model object. The corresponding text data could also be
complexity and size goes up drastically. To used as a basis for generating captions, using a
achieve real progress in these areas it may be similar methodology to question answering sys-
necessary to focus on improving model efficiency tems trained on large documents.
and downscaling before moving to bigger chal-
lenges. In the meantime, it does not hurt to con-
sider potential methods for resolving these issues. Conclusion and Discussion

Holistic Scene Understanding In conclusion, automated image captioning is a

Concerning complete scene understanding out- complex and interesting field at the intersection of
side of object interaction, increasing the CNNs natural language processing and computer vision.
depth, with a VGGNet-based model, for example, The visually impaired would benefit from some
may improve overall scene comprehension. access to the information contained within
Employing a visual self-attention mechanism images and captioning is a good starting point to
within the CNN could provide similar results by achieve this.
emphasizing important connections between While some work has already been done, there
image regions (Xu et al. 2015). Taking this a is plenty of room for improvement. Current sys-
step further and replacing the CNN entirely with tems don’t capture enough information to be
a Transformer has been explored as well (Liu much more useful than baseline models. This is
et al. 2021). unfortunate, but it means that, as the baseline
models improve, so too will their applicability to
In-image Text Data the situation of the visually impaired.
As far as including text recognition in image
captioning systems, the obvious solution would
be to simply run text detection first and incorpo- Cross-References
rate an embedded vector representation into the
feature vector from the image processing network. ▶ Visual Accessibility in Computer Games
Automatic Sound Synthesis 211

References Rohrbach, A., Hendricks, L.A., Burns, K., Darrell, T.,

Saenko, K.: Object hallucination in image captioning.
Ahsan, H., Bhalla, N., Bhatt, D., Shah, K.: Multi-modal In: EMNLP (2018)
image captioning for the visually impaired. In: CoRR Simonyan K., Zisserman, A.: Very deep convolu-tional A
(2021) networks for large-scale image recognition. In: ArXiv
Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., (2014)
Dollár, P., Zitnick, C.L.:Microsoft coco captions: data Xu, L., Ba, J., Kiros, R., Cho, K., Courville, A.,
collection and evaluation server. In: arXiv preprint Salakhutdinov, R., Zemel, R., Bengio, Y.: Show, attend
arXiv:1504.00325 (2015) and tell: neural image caption generation with visual
Elamri, C., de Planque, T.: Automated neural image cap- attention. In: ICML (2015)
tion generator for visually impaired people. (2016)
Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments
for generating image descriptions. In: CVPR,
pp. 3128–3137 (2015)
Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S.,
Choi, Y., Berg, A.C., Berg, T.L.: Babytalk: understand-
Automated Testing
ing and generating simple image descriptions. In:
CVPR, pp. 1601–1608 (2011) ▶ Quality Assurance-Artificial Intelligence
Liu, W., Chen, S., Guo, L., Zhu X., Liu J.: CPTR: full trans-
former network for image captioning. In: ArXiv (2021)
Lu J., Yang, J., Batra, D., Parikh, D.: Neural baby talk. In:
CVPR, pp. 7219–7228 (2018)
Makav, B., Kılıç, V.: A new image captioning approach for
visually impaired oeople. In: 11th International Con-
Automatic Sound Synthesis
ference on Electrical and Electronics Engineering
(ELECO), pp. 945–949 (2019) ▶ Procedural Audio in Video Games
B

Basketball Game powerful player character. They typically ask a

player to complete a level filled with minor ene-
▶ NBA 2K, a Brief History mies with a more powerful enemy at the end.

Battle

▶ Fortnite: A Brief History

Bayonetta 2, an Analysis
Basic Information
Isaac Wake2 and Newton Lee1,2
1
Institute for Education, Research, and
• Game Title: Bayonetta 2
Scholarships, Los Angeles, CA, USA • Developer/Publisher: PlatinumGames/Nintendo
2
Vincennes University, Vincennes, IN, USA • Series: Bayonetta
• Game Engine: Criware
• Platforms: Wii U, Switch
Synonyms • Release Date: October 24, 2014 on Wii U,
February 16, 2014 on the Switch
Hack-and-Slash • Genre(s): Action, Hack “n” slash
• Mode(s): Single player, multiplayer
• ESRB Rating: M
Definitions

Action – a genre of games defined by quick Game Overview and Target Audience
gameplay and fighting several enemies in real time.
Hack-and-Slash – a subgenre of action games Bayonetta 2 is an action and hack-and-slash game.
that challenges the player to fight hordes of ene- The target audience for this game are the people who
mies that seem weak individually compared to the are looking for a fast-paced, thrilling action game.

© Springer Nature Switzerland AG 2024

N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
214 Bayonetta 2, an Analysis

Gameplay and Mechanics World, Story, and Characters

The gameplay of Bayonetta is a hack-and-slash Bayonetta 2 takes place on the mountain of

game that keeps gameplay fresh with a variety of Fimbulventr and the depths of Hell. While travel-
unlockable weapons that can be equipped to ing through and between these realms, the player
Bayonetta’s hands and legs giving the player a will encounter enemies from Heaven and Hell.
variety of customization while also changing the Bayonetta 2 revolves around a female witch,
way Bayonetta controls and/or interacts with the named Bayonetta, and her journey to save her
environment. For example, Bayonetta can equip friend Jeanne’s soul from the literal depths of
weapons resembling a chainsaw to her feet and Hell. Bayonetta travels up the mountain of
katanas to her legs allowing her to kick with Fimbulventr with an unexpected partner named
sword attacks and swing with chainsaw attacks, Loki, a boy who deals with his forgotten past and
but if a player equips the katanas to her hands she powers he did not know he could wield. A weapon
has a larger variety of attack and equipping the supplier and infernal bar owner named Rodin
chainsaws to her feet allows her to skate with helps Bayonetta by creating weapons from the
chainsaws on her feet, making her faster and souls of the dead. Bayonetta’s somewhat love
doing chip damage as she whirrs by enemies interest, called Luka, follows Bayonetta trying to
while also being able to still do kick attacks. solve several mysteries surrounding the events of
Holding the slide button while attacking slices Loki, Heaven, and Hell. There are only two real
through enemies like a chainsaw would do to antagonists, Loptr and the Masked Lumen, both
a tree. of which are shrouded in mystery.

Levels

Most of the levels in this game take place in the

destroyed town of Noatun, temples, and the
depths of Hell.

One of the biggest features of the Bayonetta

series is Witch Time. Witch Time is activated
when Bayonetta perfectly dodges an attack. It
gives the player the upper hand by slowing
down time for a couple seconds, allowing for a
flurry of attacks to be performed while the ene-
mies are slowed. Witch Time earns Bayonetta
more climax toward the Climax Gauge.
When the Climax Gauge is filled, Bayonetta
can activate this gauge for an assortment of
options. The most destructive option is the
Umbran Climax. This allows Bayonetta to per-
form insanely powerful and flashy attacks, like
literally smashing enemies across levels. The
gauge can also be used to charge certain weapons
and torture attacks.
Bayonetta 2, an Analysis 215

Interface The bottom right is home to a health bar for

larger enemies or bosses.
The game contains a very solid interface which
contains everything a player needs to know while
defeating angels and demons. This is Bayonetta’s B
health bar and her climax meter.

Public Reception

The public reception of Bayonetta 2 on Switch was

received well with 9/10’s all down the board from a
majority of most major gaming sites. GameSpot
reviewer Mark Walton said the game “will be
remembered as an absolute classic” and awarded
On the right side of the screen, the amount of it a perfect 10/10, the seventh game on the site to
halos/money the player has and their combo get a perfect score. This game is praised for being a
points counter is shown. sequel that feels exactly like the original while also
fixing and improving upon the game’s formula.

Controversies

Most of the gripes from this game come from it

being a Nintendo console exclusive, originally
releasing on the Wii U. Although this is true,
Nintendo helped revive the franchise as Sega
(the previous publisher of Bayonetta 1) wanted
to pursue other interests. Although the Wii
U version did not sell well due to the console not
selling that well, the Nintendo Switch has sold
considerably better and the rerelease of Bayonetta
1 and 2, with a third game being teased, has
appeased some who had availability complaints.
There is also a strong controversy around the
visual representation of Bayonetta as a character
in the game. Anita Sarkeesian uses Bayonetta as
an example in her “Lingerie is not armor” video
on YouTube. However, some other people opine
that Bayonetta is a feminist icon.

Similar Games

God of War and Devil May Cry are very similar

games (Bayonetta Director Hideki Kamiya
worked on the Devil May Cry games), but what
216 BCI

sets Bayonetta apart from these games are the

customization options, game mechanics, and the Binuaral Headphone
protagonist being a female. Reproduction

▶ Sound Spatialization
References

GameSpot.: “Bayonetta 2 Review” by Mark Walton.

(2018). https://www.gamespot.com/reviews/
bayonetta-2-review/1900-6415911/
Biofeedback
Eurogamer.: “Switch Bayonetta 2 is a turbo-charged Wii
U port” by Richard Leadbetter. (2018). https://www. ▶ Computer Graphics, Video Games, and
eurogamer.net/articles/digitalfoundry-2018-bayonetta- Gamification Impacting (Re)Habilitation,
2-switch-is-a-turbocharged-wii-u-port
Internet Research 16: The 16th Annual Conference of the
Healthcare, and Inclusive Well-Being
Association of Internet Researchers. “BEYOND
BAYONETTA'S BARBIE BODY” by Todd Harper.
(2015). https://journals.uic.edu/ojs/index.php/spir/arti
cle/view/8464
Medium.: “A F**king Celebrity: Bayonetta and Feminist
Biometric and Facial
Perspectives” by Gamer_152. (2019). https://gamer- Recognition
152.medium.com/a-fucking-celebrity-bayonetta-and-
feminist-perspectives-bfd36858fd36 ▶ Audio and Facial Recognition CAPTCHAs for
YouTube.: “Lingerie is not Armor - Tropes vs Women in
Video Games” by Anita Sarkeesian. (2016). https://
Visually Impaired Users
www.youtube.com/watch?v¼jko06dA_x88

Biometric Authentication
BCI
▶ Brain Signals as a New Biometric Authentica-
▶ Color Detection Using Brain Computer Interface tion Method Using Brain-Computer Interface

BCI, Brain–Computer Interface

Biosensing in Interactive Art:
▶ EEG as an Input for Virtual Reality A User-Centered Taxonomy

Luís Aly1, Rui Penha2 and Gilberto Bernardes3

1
FEUP, University of Porto, Porto, Portugal
2
Binaural Hearing INESC-TEC and FEUP, University of Porto,
Porto, Portugal
3
▶ User Acoustics with Head-Related Transfer INESC TEC and University of Porto, Faculty of
Functions Engineering, Porto, Portugal

Synonyms
Binaural Sound
Biosensor technology; Biosignal; Human–com-
▶ User Acoustics with Head-Related Transfer puter interaction; Interactive art; Interactive game
Functions design; Interactive sound design
Biosensing in Interactive Art: A User-Centered Taxonomy 217

Definition further detail their user-centered taxonomy.

Examples of interactive application scenarios of
In an interactive artistic context, biosensing stud- a user-centered taxonomy are presented in section
ies the detection, measurement, variation, and “Application Scenarios of a User-Centered Tax-
translation of electrical potentials in the human onomy,” which discussed typical mappings B
nervous and motor system as a strategy for con- between biosensor technology and interactive
trolling parameters in the virtual domain of a content creation using the proposed taxonomy.
digital interactive system. Finally, on section “Conclusions,” the authors
outline the conclusions of their study.

Introduction
Human–Computer Interaction in
The unprecedented technological advances in Interactive Art
terms of computational power, software integra-
tion, and miniaturized sensor technologies have Digital art is increasingly interactive. Some of it is
fostered new artistic content creation methods in built on interactions that evolved from computer
the domains of interactive music, art installations, games and device usage. Much of the interaction
and digital game design, to name but a few. In this is intended to engage the audience in some form
context, biosensing is becoming a pervasive of interactive experience that is a key element in
modality of control in interactive systems. In the aesthetics of the art. The interactive artist is
other words, artistic-related practices have been often concerned with how the artwork behaves,
adopting psychophysiological electrical potentials how the audience interacts with it and, ultimately,
of human subjects, i.e., biosignals, as a strategy to in participant experience and degree of engage-
control the creation of interactive content towards ment with the art object.
adaptive, symbiotic, and immersive experiences. In interactive art, the art object has an internal
In this entry, the authors present a user- mechanism that enables it to change or be modi-
centered taxonomy of biosensing technology, fied by an environmental factor, or human, which
which aims to guide interactive artists in selecting has an active role in influencing the degree of
appropriate biosensors for a particular interactive changes (Edmonds et al. 2004). In today’s inter-
application design. Particular emphasis is given to active art, where the artist and the audience play
the mappings between biosignals’ level of control integral participant roles, the computer’s role has
and temporal response and the nature of the sys- immense potential in defining the degree of inter-
tem output. In pursuing such a user-centered per- action, and also managing the real-time result of
spective over biosensing technology, the authors that interaction. Issues relating to human–com-
seek to extend existing taxonomies beyond the puter interaction could be considered as important
technical specifications of the sensors, thus, pro- to interactive art creation as colors are to painting
moting a fluid use of such technology by interac- (Candy and Ferguson 2016).
tive artists. Figure 1 shows the information flow of a
The remainder of this entry is organized as human–computer interaction Bongers (2000) as
follows. Section “Human–Computer Interaction a two-way control and feedback process. When
in Interactive Art” defines concepts such as inter- interacting with a computer, humans take action in
active art, human–computer interaction. Then, in response to a sensed environment. In turn, com-
section “Interaction Modalities,” is presented puters capture the transformed environment and
artistic-related interaction modalities including act accordingly using transducers – sensor
biosensing. In section “Towards an User-Centered devices that translate real-world signals into
Taxonomy of Biosensing,” the authors review a machine-world signals – and actuators that trans-
range of taxonomic perspectives of biosensing late machine-world signals into real-world signals
technology proposed in related literature and that can be perceived by humans.
218 Biosensing in Interactive Art: A User-Centered Taxonomy

Biosensing in Interactive Art: A User-Centered Tax- which sense the external world using natural and artificial
onomy, Fig. 1 Redraw of Bongers (2000) interaction sensors. Both human effectors and computers actuators
scheme, which includes two agents: a human and a com- disturb the environment and act upon it
puter. A feedback loop is created between both agents,

An example of this interaction loop can be mutually influenced by the intervention of both
demonstrated with a simple user-control case. humans’ and machines’ memory and cognition
When a user presses a computer keyboard key, permitting the interaction with information, the
the system (i) senses the mechanical force applied changing of the environment, and thereby altering
onto the key, and assigns that action to a specific the subsequent information that is received back
programmed instruction – a real-world signal is by the system.
translated into machine-world signal – and, in
turn, the system (ii) maps that instruction to a
specific symbol and translates a machine-world Interaction Modalities
signal into real-world signal. The result being the
visual feedback of the assigned character on the In Bongers (2002), human–computer interaction
computer screen which can guide the user for makes use of what the author refers as interaction
future actions. modalities – as communication channels between
Thus, to foster human–computer interaction, a a human and a system – and those interaction
system should be equipped with (i) the ability to modalities involve (i) input modalities which
sense the external environment, i.e., a system imply senses such as seeing, hearing, smelling,
capable of converting some kind of physical tasting, and touching which are used to explore
energy, e.g., kinetic or biosignal into electricity the surrounding environment and (ii) output
and then to codify that input into digital data in modalities, mainly involving the motor system,
order for it to be recorded, analyzed, and manip- e.g., handwriting, speaking, or moving things
ulated and (ii) the ability to actuate on the external around, that are used use to act on the
environment, i.e., being capable of converting environment.
digital data into some form of energy that can be Biosensing, as a measurement of the human
perceived by a human being, e.g., visual/sound or psychophysiological activity and its use as a strat-
mechanic cues. Sensing and actuating are specifi- egy for controlling parameters in the domain of a
cations that allow a system to be controlled, report digital interactive system, is an interaction modal-
its current state, and guide the user towards the ity of special interest for the present study.
next possible actions. In detail, human psychophysiological activity
In Bongers (2000), both the human and relates to brain, skeleton, and cardiac muscles, but
machine’s memory and cognition are essential also skin functions, which all generate electrical
components in building the interaction loop. In potentials. These signals can be measured by elec-
an interactive system, ideally the “conversation” trodes and used to control a digital interactive
between the human and the system should be system. Biosensing captures biosignals (Arslan
Biosensing in Interactive Art: A User-Centered Taxonomy 219

et al. 2005; Ortiz-Perez et al. 2011; Ortiz et al. energy measured by a sensor, i.e., mechanical,
2015) by detecting, measuring, and translating the gravitational, electrical, thermal, or magnetic. In
electrical potentials in the human nervous and Bongers (2000), and referring to the design of
motor system functions, e.g., electromyographic interactive musical systems, sensor technologies
signals, measured on the skin which are related to are categorized based on the ways humans can B
muscle activity, and electroencephalographic sig- change the state of the surrounding environment
nals, measured on the scalp which is related to pointing output modalities mainly related to mus-
brain activity. Recent sensor technology has been cle actions, which result in mechanical move-
developed to detect and measure these functions, ment, air flow, or sound production.
notably to support medical care (Stern et al. 2001; In game research, Kivikangas et al. (2011)
Cacioppo et al. 2007; Webster and Eren 2017). review the biosense method by presenting a taxo-
Beyond medical applications, biosensor tech- nomically review of the application scenarios of
nology has been attracting the attention of inter- biosignal as a way to assess game experience
active artists who have been increasingly adopting arising from emotional reactions, mainly related
this technology to control parameters of interac- to valence and arousal dimensions, and in Leite
tive digital systems. et al. (2000), Kleinsmith et al. (2003), and
Bernhaupt et al. (2007), a taxonomy is presented
that takes into account factors such as affective
Towards an User-Centered Taxonomy of responses by the player to game playing.
Biosensing Taking into account the use of biofeedback to
control game mechanics in Nacke et al. (2011),
Existing Taxonomic Perspectives Pedersen et al. (2010), and Figueiredo and Paiva
A wide range of taxonomic perspectives of (2010) Nogueira et al. 2016) formalize biofeed-
biosensing technology, rooted in different disci- back game mechanics with respect to players’
plines and applications, have been proposed in emotional states – modeling player experience
related literature. In Horowitz and Hill (1989), for driving interactive content creation.
sensor technologies are organized technically Table 1 lists biosensor technology commonly
according to their electronic circuit design and in adopted in interactive art domain and details the
(Sinclair 2000) according to the kind of physical nature of the psychophysiological electrical

Biosensing in Interactive Art: A User-Centered Taxonomy, Table 1 Commonly used sensor technologies in the
field of interactive arts and their respective measurable phenomena expressed in hertz (Hz)
Sensor technology Abbreviations Measurable phenomena Frequency (Hz)
Gaze interaction GAZE Position, movement, and pupil dilation of gaze with 30
a sensor located on the screen
Electromyography EMG Activation of facial or body muscle tissues 20–2000
Respiration RESP Chest the breathing rate and volume Measured in
extension capacity
Temperature TEMP Thermal feedback Up to 5000
Electrocardiogram ECG Electrical activity of the heart 0.05–100
Heart rate variability HRV Time difference between two sequential heartbeats HF (0.15–0.40)
LF (0.04–0.15)
Electrooculgram EOG Eye motion analysis with a body-worn sensor DC to 10
Electrodermal activity EDA Tonic level of electrical conductivity of skin 0–2.8
Electroencephalogram EEG Electrical changes on the scalp 0.05–100
Sensors such as ECG, EEG, EMG, TEMP, and EOG measures in Aller et al. (2000), EDA (da Silva et al. 2014), and HRV
(Bakhtiyari et al. 2017). GAZE and RESP sensors have different responses; the latter is measured in extension capacity
range, e.g., ranging from 35% to 65%, and the former’s accuracy depends on the angular average distance from the actual
gaze point
220 Biosensing in Interactive Art: A User-Centered Taxonomy

potentials it measures. The sensors were selected mechanics such as daytime or the weather condi-
based on their common application in interactive tions by pointing to a low temporal variability
arts, availability, low cost, miniaturization, and response sensor with an indirect type of control.
quick integration with software applications. Figure 2 shows the user-centered taxonomy of
biosensing technology for interactive art contexts.
A User-Centered Taxonomy of Biosensing It includes the biosensors technology listed in
The authors propose a user-centered taxonomy of Table 1. The two dimensions of the taxonomy,
biosensor technology to assess the broader picture i.e., the temporal variability of the psychophysio-
on the use of biosensing technologies as a strategy logical function and the type of control over par-
for controlling parameters in the virtual domain of ticular function, are assigned to the horizontal and
a digital interactive system. In pursuing such a vertical axes, respectively.
user-centered perspective over biosensing tech- The temporal variability in the horizontal axis
nology, the aim is to extend existing taxonomies reports the degree of temporal variability from
beyond the technical specifications of the sensors, low to high. For example, a GAZE sensor has
thus, promoting a fluid use of such technology, high variability as it measures eye movement,
and its intuitive use, by interactive artists. which can naturally be very fast. On the other
In greater detail, the proposed user-centered side of the spectrum, the EEG sensor has a much
taxonomy guides the process of selecting the lower variability as it measures brainwaves can
most suitable sensor technology for a specific have a slower rate of change. The temporal vari-
task based on two dimensions: (i) the degree of ability is related to the measurable phenomena
control over psycho-physiological human func- expressed in hertz presented in Table 1.
tions, i.e., the ability the subject has to manipulate The degree of control over the psycho-
her own psychophysiological activity and conse- physiological functions by subjects is denoted in
quently alter the sensor response and (ii) the tem- the vertical axis and is expressed in a scale from
poral variability, i.e., the rate of temporal direct to indirect control. In greater detail, the
noticeable change in the captured electrical poten- scale reports the degree of control humans have
tials. For example, the author’s taxonomy pro- over their psychopsychological functions and the
vides an answer to the artist, which aims to use ability to deliberately alter the response of the
biosignals to control the (long-term) digital game captured data. For example, humans have a

Biosensing in Interactive Cluster

Control Type
Art: A User-Centered B
Taxonomy, Direct
Fig. 2 Physiological Cluster
GAZE
measures according to a A RESP
distribution on two axis: on
the horizontal axis, the EMG
temporal variability
response stimuli/sensor, and
on the vertical axis, the level TEMP
of direct control of the EOG
electrical potentials
generated. The authors ECG
denote two clusters,
A and B, grouped according HRV
to sensor’s type of control
and response. The EDA
measurable phenomena
captured by the different EEG
sensors is explained in
Table 1 Indirect Temporal
Low High Variability
Biosensing in Interactive Art: A User-Centered Taxonomy 221

direct-explicit control over a muscle impulse, cap- measure can be used to define artificial intelli-
tured by an EMG and a more indirect-implicit gence parameters of nonplayer characters’ level
control over skin conductivity using an EDA. of reaction to the player presence. For creating
In Fig. 2, a diagonal disposition of the biosen- interactive game audio, the sensors from cluster
sors can be identified, showing that the horizontal A allow the control of higher level musical fea- B
and vertical axes are intertwined in such a way tures such as tempo, i.e., a faster or slower tempo
that the faster the responses obtained from the in music expressed in bpm (beats per minute), or
sensors, the more direct control users have over the soundtrack general mood, i.e., a more tense or
their measures and evolution. Moreover, from this relaxed type of music can be mapped to an HRV
tendency the authors highlight two overlapping sensor.
clusters. This overlap is due to the fact that there Biosensors from cluster B are better adapted to
are sensor technologies which have slow chang- control explicit interactions or foreground actions.
ing responses that can be altered by a sudden For example, an EMG sensor, which measures
change in the environment that they are measur- fast-changing facial or body muscle tissues, is
ing. An example of this overlap is the TEMP well-adapted to control player rapid actions, e.g.,
sensor which typically has slow response but the define the impulse of a character’s jump. The fast
user can induce a more immediate response by response and highly controllable RESP sensor can
blowing air into it. be used to define the number of enemies when the
One remaining dimension is of consideration player is trying to accomplish an undercover mis-
here: body intrusion. Despite its relevancy in the sion. For interactive game audio cluster B is better
choice of biosensors for a particular task, the authors adapted to control low-level sound features such
believe that the miniaturization of sensor- as the audio level of determined sound effect, e.g.,
technology will eventually make it ubiquitous and a GAZE sensor can be used to raise the audio level
pervasive in all artistic applications scenarios. Even of an observed game object to focus the player’s
so, interactive artists must be aware of the pertinence attention or to define the location of a sound event
of this dimension when building interactive content. in the scope of the stereo image.

Application Scenarios of a User- Conclusions

Centered Taxonomy
The authors presented a user-centered taxonomy
Based on the two identified clusters in Fig. 2, the of biosense for interactive arts which aim is to
authors now discuss typical mappings between provide artists with a framework to assess the
biosensor technology and interactive content in broader picture on its use as a strategy for control-
two main domains: game design and game ling parameters in a digital interactive system. In
audio. The authors believe these same principles pursuing such a user-centered perspective over
can be applied to other forms of interactive biosensing technology, the authors sought to
biosensing driven art contexts, such as interactive extend existing taxonomies beyond the technical
music or performance art. specifications of the sensors in order to assess a
In game design, biosensors from cluster A are broader picture of biosensing technologies, thus,
typically applied to control implicit, slow- promoting a fluid and intuitive use of such tech-
adapting, or background aspects of a game, such nology by interactive artists. By providing use
as level generation conditions. For example, the cases examples which discussed typical mappings
slow-changing HRV function can be mapped to between biosensor technology and interactive
evolve long-term game settings such as the content creation in domains such as game design
weather conditions or to control artificial intelli- and game audio, the authors intended to validate
gence aspects of game that are not so noticeable. their user-centered taxonomy of biosense in inter-
On the other hand, the slow pace of an EEG active art.
222 Biosensor Technology

Cross-References use of psychophysiological methods in game research.

J Gaming Virtual Worlds. 3(3), 181–199 (2011)
Kleinsmith, A., Fushimi, T., Takenaka, H., Bianchi-
▶ Stress Reduction, Relaxation, and Meditative Berthouze, N.: Towards bidirectional affective
States Using Psychophysiological Measure- human-machine interaction. J. Three Dimens. Images.
ments Based on Biofeedback Systems via 17, 61–66 (2003)
HRV and EEG Leite, I., Pereira, A., Mascarenhas, S, Castellano, G.,
Martinho, C., Prada, R., Paiva, A.: Closing the loop:
from affect recognition to empathic interaction. In: Pro-
ceedings of the 3rd International Workshop on Affec-
References tive Interaction in Natural Environments, pp. 43–48.
ACM (2000)
Aller, M., Fox, S., Pennell, J.: Measurement, Instrumenta- Nacke, L.E., Kalyn, M., Lough, C., Mandryk, R.L.: Bio-
tion and Sensors Handbook, CRC Press LLC, New feedback game design: using direct and indirect phys-
York (2000) iological control to enhance game interaction. In:
Arslan, B., Brouse, A., Castet, J., Filatriau, J.-J., Léhembre, Proceedings of the SIGCHI Conference on Human
R., Noirhomme, Q., Simon, C.: From biological signals Factors in Computing Systems, pp. 103–112. ACM
to music. In: 2nd International Conference on Enactive (2011)
Interfaces, Genoa (2005) Nogueira, P.A., Torres, V., Rodrigues, R., Oliveira, E.,
Bakhtiyari, K., Beckmann, N., Ziegler, J.: Contactless Nacke, L.E.: Vanishing scares: biofeedback modula-
heart rate variability measurement by IR and 3D tion of affective player experiences in a procedural
depth sensors with respiratory sinus arrhythmia. Pro- horror game. J. Multimodal User Interfaces. 10(1),
cedia Comput. Sci. 109, 498–505 (2017) 31–62 (2016)
Bernhaupt, R., Boldt, A., Mirlacher, T., Wilfinger, D., Ortiz, M., Grierson, M., Tanaka, A.: Brain musics: history,
Tscheligi, M.: Using emotion in games: emotional precedents, and commentary on whalley, mavros and
flowers. In: Proceedings of the International Confer- furniss. Empir. Musicol. Rev. 9(3–4), 277–281 (2015)
ence on Advances in Computer Entertainment Tech- Ortiz-Perez, M., Coghlan, N., Jaimovich, J., Knapp, R.B.:
nology, Salzburg, pp. 41–48. ACM (2007) Biosignal-driven art: beyond biofeedback. Ideas
Bongers, B.: Physical interfaces in the electronic arts. Sonica/Sonic Ideas. 3(2), (2011)
Trends in Gestural Control of Music, IRCAM-Centre Pedersen, C., Togelius, J., Yannakakis, G.N.: Modeling
Pompidou, Paris, pp. 41–70 (2000) player experience for content creation. IEEE Trans.
Bongers, B.: Interactivating spaces. In: Proceedings of the Comput. Intell. AI Games. 2(1), 54–67 (2010)
Symposium on Systems Research in the Arts, Infor- Sinclair, I.: Sensors and Transducers. Elsevier (2000)
matics and Cybernetics, Barcelona (2002) Stern, R.M., Ray, W.J., Quigley, K.S.: Psychophysiolog-
Cacioppo, J.T., Tassinary, L.G., Berntson, G.: Handbook of ical recording. Oxford University Press, Oxford
psychophysiology. Cambridge University Press, Cam- (2001)
bridge (2007) Webster, J.G., Eren, H.: Measurement, instrumentation,
Candy L., Ferguson S.: Interactive Experience, Art and and sensors handbook: Spatial, mechanical, thermal,
Evaluation. In: Candy L., Ferguson S. (eds) Interactive and radiation measurement. CRC Press, Boca Raton
Experience in the Digital Age. Springer Series on Cul- (2017)
tural Computing. Springer, Cham (2014)
da Silva, H.P., Guerreiro, J., Lourenço, A., Fred, A.L.N.,
Martins, R.: Bitalino: A novel hardware framework for
physiological computing, pp 246–253. PhyCS (2014)
Edmonds, E., Turner, G., Candy, L.:. Approaches to inter- Biosensor Technology
active art systems. In: Proceedings of the 2nd Interna-
tional Conference on Computer Graphics and ▶ Biosensing in Interactive Art: A User-Centered
Interactive Techniques in Australasia and South East Taxonomy
Asia, pp. 113–117 ACM (2004)
Figueiredo, R., Paiva, A.: “I want to slay that dragon!”-
influencing choice in interactive storytelling. In: Joint
International Conference on Interactive Digital Story-
telling, pp. 26–37. Springer (2010) Biosignal
Horowitz, P., Hill, W.: The Art of Electronics. Cambridge
University Press, Cambridge (1989)
Kivikangas, J.M., Chanel, G., Cowley, B., Ekman, I., ▶ Biosensing in Interactive Art: A User-Centered
Salminen, M., Järvelä, S., Ravaja, N.: A review of the Taxonomy
Bounding Sphere Hierarchy 223

Bishōjo Game Blockchain Games

▶ Visual Novel ▶ NFT Games

B

Bit Blue Noise Sampling

▶ Color: Pixels, Bits, and Bytes ▶ Poisson-Disk Sampling: Theory and

Applications

Blendshape
Board Game
▶ Position-Aware 3D Facial Expression Mapping
Using Ray Casting and Blendshape
▶ Protection Korona: A Game Design on Covid-
19

Blind Bags
Bodily Presence in Digital
▶ Loot Boxes: Gambling-Like Mechanics in Games
Video Games
▶ Player-Avatar Link: Interdisciplinary Embodi-
ment Perspectives

Blind Boxes
Booster Packs
▶ Loot Boxes: Gambling-Like Mechanics in
Video Games
▶ Loot Boxes: Gambling-Like Mechanics in
Video Games

Blindness
Bounding Box Hierarchy
▶ Making Virtual Reality (VR) Accessible for
People with Disabilities ▶ Bounding Volume Hierarchies for Rigid Bodies

Block-Based Programming Bounding Sphere Hierarchy

▶ Unified Modeling Language (UML) for Sight ▶ Bounding Volume Hierarchies for Rigid Bodies
Loss
224 Bounding Volume Hierarchies for Rigid Bodies

then a bounding volume hierarchy (BVH) is

Bounding Volume Hierarchies used to represent each part of the object in such
for Rigid Bodies a way that the complete object is represented in
each level of the BVH. A BVH is a tree structure
Simena Dinas in which the problem of detecting a collision
Departamento de Electrónica y Ciencias de la between two or more objects can be reduced to
Computación, Pontificia Universidad Javeriana detect collisions between their BVHs representa-
Cali – Colombia, Cali, Valle, Colombia tion. On the one hand, if bounding volumes of two
nodes do not overlap, there is no collision for the
parts bounded by these bounding volumes. On the
Synonyms other hand, if bounding volumes of two leaf nodes
overlap, the corresponding triangles must be
Bounding box hierarchy; Bounding sphere tested for intersection. In the next section, it is
hierarchy presented a brief explanation of spheres, axis-
aligned bounding boxes (AABB), and oriented
bounding boxes (OBB).
Definition

A bounding volume is a geometric primitive that Spheres

encloses one or more objects and leads to cheaper
overlapping tests (Ericson 2005). The bounding It encloses an object with the minimal sphere; it is
volumes most frequently used are spheres, boxes, usually called the bounding circumference in a two-
and ellipsoids. The bounding volume hierarchy is dimensional space and bounding sphere in a three-
a tree structure used to represent the set of geo- dimensional space (Arcila 2011; Weller 2013;
metric objects, which are enclose in bounding Dinas and Bañón 2015). One advantage of sphere
volumes. A rigid body is a solid body without as bounding volumes is its efficiency to calculate
deformation or whose deformation is so insignif- the intersections and distance between a pair of
icant that it can be neglected. sphere. Although spheres are invariant to translation
and rotations, they are not good bounding volume
for elongated objects. To calculate the bounding
Introduction sphere, it is found the largest distance between a
pair of vertices of the polygon (in two dimensions,
Bounding volumes and binary space partitioning see Fig. 1) or polyhedron (in three dimensions).
belong to the acceleration structures used to The intersection between spheres is straight-
increase the speed for applications that require to forward; it is just enough to calculate the distance
determine intersection between two or more between their centers and the sum of their radius,
objects. Bounding volume is a technique for and to compare these quantities. For instance, the
object partitioning, which is based on object sub- pair of spheres depicted in Eqs. 1 and 2 have
division and objects level of details, whereas centers (x1, y1, z1) and (x2, y2, z2) and radius r1
binary space partitioning is a technique for space and r2, respectively.
partitioning, which is based on the space subdivi-
sion and space level of details (Luque et al. 2005; ðx  x1 Þ2 þ ðy  y1 Þ2 þ ðz  z1 Þ2 ¼ r1 2 (1)
Weller 2013). Techniques based on hierarchical
bounding volumes are commonly used for colli- ðx  x2 Þ2 þ ðy  y2 Þ2 þ ðz  z2 Þ2 ¼ r2 2 (2)
sion detection. An object is a triangular mesh,
which is bounded by a hierarchy of bounding There is not intersection between the circum-
volumes for collision detection. It is divided hier- ferences if the condition 3 holds; otherwise they
archically, usually by a recursive method, and intersect.
Bounding Volume Hierarchies for Rigid Bodies 225

Bounding Volume Bounding Volume

Hierarchies for Rigid Hierarchies for Rigid
Bodies, Fig. 1 Bounding Bodies, Fig. 3 OBR
circumference

Bounding Volume
Hierarchies for Rigid rectangle (OBR) and oriented bounding box in
Bodies, Fig. 2 AABR
a two-dimensional and in a three-dimensional
space, respectively (Zhigang et al. 2010; Arcila
2011; Weller 2013). The most significant
advantage of this volume is the invariance to
translation and rotations; however, the collision
ðx2  x1 Þ2 þ ðy2  y1 Þ2 þ ðz2  z1 Þ2 test for OBBs (or OBRs) is computationally
more expensive than the AABBs (or AABRs)
> r 1 þ r2 (3)
test (Fig. 3).
The same procedure used to determine if two
Axis-Aligned Bounding Box (AABB) boxes overlap can be used to determine the inter-
section between OBBs.
An AABB is a minimal enclosing box that con-
tains the object; it is usually called the axis-
aligned bounding rectangle (AABR) in a two-
dimensional space and axis-aligned bounding An Example of Bounding Volumes
box in a three-dimensional space (Zhigang Hierarchies for Spheres
et al. 2010; Arcila 2011; Weller 2013). The
main benefits are that it is simple to find the Figure 4 depicts an example of bounding vol-
box, the box is invariant to translations, and umes of a lamp using spheres. Figure 4a shows a
the test between a pair of boxes is straightfor- nonconvex polyhedron with lamp shape, and it
ward; nevertheless, this volume is not invariant was decomposed into a set of ten parts, which
to rotation, as a result, changes in the objects are convex polyhedra. The lamp is compounded
direction require updates in the bounding boxes by one (1) base, one (1) lamp shade, four
(Fig. 2). (4) pipes or tubes, and four (4) junctions or
The procedure of testing whether two boxes connectors. In Fig. 4b, there are depicted the
overlap is simple. Check if one box lies three levels of the hierarchy, which has 23 ¼ 8
completely on the half-space (not containing bounding volumes in each part (80 spheres in
the object) of one face of another box, then the total). Figure 4c, d depict the levels 6 and 12 of
two boxes do not overlap. That is, if there exists the hierarchy, respectively; the level 6 has
a separating plane so that two boxes lay on 26 ¼ 64 bounding volumes in each part
different half-spaces of the plane, they do not (640 spheres in total) and the level 12 has
overlap. 212 ¼ 4.096 bounding volumes per part
(40.960 spheres in total).

Orientation Bounding Box (OBB)

Some Bounding Volumes
It is the minimal box that encloses the object
and takes into account the object orientation; it Following, a short list of common bounding
is usually called the oriented bounding volumes.
226 Bounding Volume Hierarchies for Rigid Bodies

(a) Non-Convex
Object: Lamp (b) Level 3 (c) Level 6 (d) Level 12

Bounding Volume Hierarchies for Rigid Bodies, Fig. 4 Representation of an object Lamp by spheres. (Dinas
et al. 2009)

• Ellipsoid: Ellipsoids are tighter fitting reconstruction; they are important for animated
than spheres for elongated objects (Rubino three-dimensional graphics.
et al. 2015).
• Cylinder: It uses the radius of bounding cir- The combination of two or more BVHs was
cumference of the shape and a swept line proposed by (Arcila 2011) as a double BVH. The
(Chan and Tan 2004). outer (minimal) bounding volume has been
• K-Discrete Orientation Polytopes (K-DOPs) widely used; it determines the collision and accel-
or Fixed-Direction Hull (FDH): A DOP is a erates the process in broad-phase, whereas the
generalized AABB. It is constructed by taking inner (maximal) bounding volume accelerates
a number k of appropriately oriented planes at the collision acceptance process. Additionally, a
infinity and bringing them closer to the object combination of AABB and ellipsoids for three-
until they collide (Weller 2013; Dinas and dimensional for a sequence of images was
Bañón 2015). recently proposed by Rubino et al. (2015), they
• Oriented Discrete Orientation Polytopes worked on objects reconstruction from a sequence
(Or-DOPs): An Or-DOP is similar to a of images in open and close scenarios.
K-DOP, but it is a generalization of an OBB
rather than an AABB (Suaib et al. 2013).
• Swept Sphere Volume: The representation is Bounding Volumes Requirements
straightforward: a radius and a swept volume.
The most important swept volumes are point Several literature about the bounding volumes
swept spheres (PSS), line swept spheres (LSS), includes the importance, minimal requirements,
and rectangle swept spheres (RSS). The advan- construction cost, test cost, evaluation tests, tech-
tages of these volumes correspond to the niques, applications, among others (Zhigang et al.
sphere advantages (Tang et al. 2014). 2010). Because of their importance, several
• Cloud of Points and Convex Hull (CPCH): It authors have reported their impact and cost of
constructs a convex hull for a cloud of points; it computing, and have implemented strategies to
is the smallest convex volume containing the decrease the cost (Yoon and Manocha 2006),
object and, hence, a hull is a tight bounding whereas other authors have reported works on
volume (Figueiredo et al. 2010). Research on requirements for optimal bounding volumes
clouds of points includes mostly surface (Weller 2013):
Bounding Volume Hierarchies for Rigid Bodies 227

• Tight fitting to approximated objects: Tighter Bounding Volumes Performance

fitting bounding volumes are computationally
expensive; it is hard to calculate a unique Performance of Bounding Volumes can be evalu-
bounding volume that adjusts different objects. ated using Eq. 4 (Bradshaw 2002; Weller 2013;
Hierarchies are one solution to approximate the Dinas and Bañón 2015). B
object (Bradshaw 2002).
• Efficient creation: The complexity of the T ¼ N uCu þ N v Cv , (4)
bounding volume is mainly the complexity
of its creation; however, more complex volumes where T is the total cost function for detecting
should be tighter fitting (Lauterbach et al. 2009). interference between a pair of objects represented
• Efficient updating: If a bounding volume is by a bounding volume. Nu is the number of
invariant to rotation and translation, it does bounding volumes updates during the traversal
not require to be updated; however, not always of the hierarchies, and Cu is the average cost of
they are invariant to both geometrical transfor- updating a bounding volume due to the motion. Nv
mations (Spillmann et al. 2007). is the number of overlapping tests performed over
• Efficient overlap tests: Low computational the bounding volumes, and Cv is the average cost
costs are associated with simpler bounding of performing an overlapping test between a pair
volumes (as spheres and boxes); however, of bounding volumes.
they are not as tighter fitting as others However, Eq. 4 was extended to Eq. 5 in order
(Zhigang et al. 2010). to separate bounding volumes and primitives
• Low memory usage: The simpler bounding (Weller 2013).
volumes use lower memory and have low com-
putational cost; nevertheless, they are not as T ¼ N uCu þ N v Cv þ N p Cp, (5)
tighter fitting as others (Yoon and Manocha
2006; Weller 2013). where Np is the number of overlapping tests
• Suitable for hierarchy construction: performed over primitives, and Cp is the aver-
A bounding volume tree helps to decrease the age cost of performing an overlapping test
number of overlapping tests, and they can be between primitives. (Suaib et al. 2013)
used to approximate the objects shape included Co in Eq. 6. Co is used to include any
(Spillmann et al. 2007). additional time caused by transformation
updates or coordinates updates for the objects.
All requirements are essential; however, there Similarly, Ericson (2005) define Co as a one-
is not a bounding volume that fulfills them time processing.
completely; the selection depends on the applica-
tion. Nevertheless, efficiency in bounding vol- T ¼ N u Cu þ N vCv þ N pCp þ Co (6)
umes is equally important to the following
requirements of the hierarchy (Bradshaw 2002): For instance, due to spheres and OBBs are
invariant to rotation, they eliminate the values Nu
• Volume Approximation: Children nodes repre- and Cu from Eqs. 4 and 5 but the value Nv
sent tighter fit of the object than its parent. increases. In contrast, the reduction of over-
• Covered Area: Children nodes must cover lapping test over the bounding volumes (Nv)
the same parts of the object covered by increases the number of overlapping test over
their parent. primitives (Np).
• Automatic Creation: Hierarchy construction
does not use human interaction.
• High Degree of Accuracy: Bounding volumes Cross-References
should fit the original model as tightly as
possible. ▶ Collision Detection
228 Braille

References Zhigang, F., Jianxun, J., Jie, X., Xiaochi, W.: Efficient
collision detection using bounding volume hierarchies
Arcila, O.: Nuevas Representaciones Dobles: Estudio e of obb-aabbs and its application. In: 2010 International
Implementación. Linköping studies in science and Conference on Computer Design and Applications
technology: Thesis. Editorial Academica Espanola (ICCDA), vol. 5, pp. V5:242–V5:246 (2010)
(2011)
Bradshaw, G.: Bounding Volume Hierarchies for Level-of-
Detail Collision Handling. PhD thesis, Trinity College
Dublin, Dublin (2002)
Chan, C., Tan, S.: Putting objects into a cylindrical-
Braille
rectangular bounded volume. Comput. Aided Des.
36(12), 1189–1204 (2004) ▶ Unified Modeling Language (UML) for Sight
Dinas, S., Bañón, J.M.: A literature review of bounding Loss
volumes hierarchy focused on collision detection –
revisin de literatura de jerarqua volmenes acotantes
enfocados en deteccin de colisiones. Ingeniera y
Competitividad. 17, 63–76 (2015)
Dinas, S., Arcila, O., Bañón, J.M.: An’alisis de la para- Brain Computer Interface
lelizaci’on de un esferizador geom’etrico. Cuarto
Congreso Colombiano de Computación – 4CCC
(2009) ▶ Color Detection Using Brain Computer
Ericson, C.: Real-Time Collision Detection (The Morgan Interface
Kaufmann Series in Interactive 3-D Technology). Mor-
gan Kaufmann, Amsterdam (2005)
Figueiredo, M., Oliveira, J., de Arajo, B.R., Pereira, J.A.
M.: An efficient collision detection algorithm for point
models. Graphicon 2010 – International Conference on Brain Control Interface
Computer Graphics and Vision (2010)
Lauterbach, C., Garland, M., Sengupta, S., Luebke, D., ▶ Gaming Control Using BCI
Manocha, D.: Fast bvh construction on gpus. Comput.
Graphics Forum. 28(2), 375–384 (2009)
Luque, R.G., Comba, J.L.D., Freitas, C.M.D.S.: Broad
phase collision detection using semi-adjusting
bsp-trees. In: Proceedings of the 2005 Symposium on Brain Signals as a New
Interactive 3D graphics and Games, I3D’05, Biometric Authentication
pp. 179–186. ACM, New York (2005)
Rubino, C., Crocco, M., Perina, A., Murino, V., and Bue,
Method Using
A. D. (2015). 3d structure from detections. Submitted Brain-Computer Interface
to Computer Vision and Pattern Recognition 2015
Spillmann, J., Becker, M., Teschner, M.: Efficient updates Fares Yousefi and Hoshang Kolivand
of bounding sphere hierarchies for geometrically
deformable models. J. Vis. Commun. Image Represent.
Department of Computer Science, Faculty of
18(2), 101–108 (2007) Engineering and Technology, Liverpool John
Suaib, N., Bade, A., Mohamad, D.: Hybrid collision Moores University (LJMU), Liverpool, UK
culling by bounding volumes manipulation in massive
rigid body simulation. TELKOMNIKA Indonesian.
J. Electr. Eng. 11(6), 3115–3122 (2013)
Tang, M., Manocha, D., Kim, Y.J.: Hierarchical and con- Synonyms
trolled advancement for continuous collision
detectionof rigid and articulated models. IEEE Trans. Biometric authentication; Brain-computer inter-
Vis. Comput. Graph. 20(5), 755–766 (2014)
Weller, R.: A brief overview of collision detection. In: New
face; EEG signal
Geometric Data Structures for Collision Detection and
Haptics, Springer Series on Touch and Haptic Systems,
pp. 9–46. Springer International Publishing, Heidel- Definitions
berg (2013)
Yoon, S.-E., Manocha, D.: Cache- efficient layouts of
bounding volume hierarchies. Comput. Graphics Human biometric techniques are presented as
Forum. 25(3), 507–516 (2006) another type of security authentication to cover
Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface 229

the problems of password authentication. systems that provide communications between

Brainwave is another human biometric, which human beings and machines.
recently is one of the popular subjects for scientists In the past, people always liked to read each
and researchers. Brain-computer interface (BCI) is other’s mind or it was a wish for them to control
a method of communication based on neural their environments or replace objects with their B
activity’s communication created by the brain. brainpower. Today technology made those dreams
happen. For instance, transfusing signals straight to
someone else’s brain to allow them to experience
Introduction new sensory inputs like sight, hearing or feel. One
of the potential outcomes of the future could be the
In the past, people used to have a suitcase to keep manipulation of computers and associated devices
their important documents like keys, money, bank with the simple transferring of a thought. Consid-
account booklets, letters, photos, etc. which they ering this potential BCI could be a very significant
could lock the suitcase to keep them secure. Today, breakthrough within technology. BCIs are
people can keep all of that information in their becoming increasingly popular in medical and non-
personal computers, mobile devices, social net- medical areas as a way of communication to be
works, and the cloud storages, which in this case, conducted between humans and machines. Nowa-
information security and data protection play a days brain signal authentication using BCI devices
crucial role in them. Security and accurate authen- is one of the popular subjects for researchers.
tication methods have become a top priority within This article is a survey about biometric authenti-
information security, which is necessary as it cation techniques and using brain signals via BCI
allows companies and people to keep their systems technologies as a new biometric authentication tech-
and devices protected by authorizing only authen- nique of a human body, which could be the most
ticated users to use important resources (Margaret). secure technology in the future.
There are several methods of authentication
(Darril) such as something you know (password
or PIN), something you have (smart card, com- Overview on EEG Signals
mon access card (CAC) (DeBow and Syed 2016),
personal identity verification (PIV) (Kittler et al. A review about biometric authentication in a cou-
2002), or RSA token (Vangie)), and something ple of research papers and a literature review
you are (using biometrics). Password or PIN about BCI and brain signal authentication from
authentication and different kinds of smart cards the previous experiences of the other researchers,
and tokens are easy to implement but because of which are categorized to different subjects and
this ease can be very easily stolen or lost. Bio- goals, are as follows:
metrics is a new technological alternative to solve
this problem (Jain et al. 1999). These typical bio- Biometrics
metric authentication technologies have some dis- The word biometrics is a combination of two Greek
advantages (Erden). Therefore, a new biometric words, “bios” (life) and “metrikos” (measure). This
method needs to be produced to reduce the num- technology is mostly used for access control and
ber of disadvantages that are within current sys- identification or for identifying people who are
tems. Brain signal is a human’s characteristic, under investigation (Faundez-Zanuy 2006). There
which does not have the problem of visibility to are two different concepts in biometrics, which the
copy it, and it does not have the disadvantages of authors should concentrate on, behavioral/physical
other biometric systems if the authors could biometrics and authentication/identification. Behav-
employ it as a biometric authentication. Human ioral biometrics concentrates on analyzing the non-
brain signals are one of the characteristics that, physiological or non-biological structures of any
nowadays, scientist and researchers are working human. It studies the unique psychological charac-
on using brain-computer interface (BCI). BCIs are teristics of humans like signature, voice, gait, and
230 Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface

Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface,

Fig. 1 Physiological and behavioral biometric types

keystrokes. Physical biometrics is doing the oppo- most flexible, working even when the subject is
site, which is focusing on analyzing the physiolog- unaware of being scanned” (Jafri and Arabnia
ical and biological structures of the human 2009). It works by methodically analyzing partic-
(Agrafioti et al. 2009). As you can see in Fig. 1, ular characteristics that are common to human’s
there are a couple of different behavioral and phys- face such as the size of the nose, the space
iological human biometric types. between the eyes, position of cheekbones, jaw-
line, and so forth (Margaret, Facial recognition).
Biometric Authentication Types, Advantages, Table 1 shows some specific advantages and dis-
and Disadvantages advantages of this biometric technique (Masupha
A biometric technique can work in two modes, et al. 2015).
authentication and identification, which are the
heart of the biometric science (Prasanna et al. Fingerprint Identification
2012). Biometric authentication is one of the most Fingerprints are the most famous biometric
popular ways to provide personal identification which remain constant throughout life. It is
because these characteristics of a human are spe- more than 100 years in worldwide fingerprint
cific and unique. Most of these specific features are comparison that no two same fingerprints were
so hard to duplicate and accurately produce found. “Fingerprint identification involves com-
(Kodituwakku 2015). In terms of information secu- paring the pattern of ridges and furrows on the
rity, physiological biometric traits appear more fingertips, as well as the minutiae points (ridge
practical. The most popular physiological biomet- characteristics that occur when a ridge splits into
ric techniques are as follows: two, or ends) of a specimen print with a database
of prints on file” (Kute and Kumar 2014).
Face Recognition Table 2 shows some specific advantages and
In comparison with the different biometric identi- disadvantages of this biometric technique
fication techniques, “face recognition is one of the (Tarun).
Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface 231

Brain Signals as a New Biometric Authentication Brain Signals as a New Biometric Authentication
Method Using Brain-Computer Interface, Method Using Brain-Computer Interface,
Table 1 Face recognition biometric advantages and Table 3 Retina biometric advantages and disadvantages
disadvantages
Retina scan
Face recognition
Advantages Disadvantages
Advantages Disadvantages
B
Like fingerprints, retina Retina scan enrolments
Prevent card counters, Isn’t accurate at all times traits remain stable take longer than both iris
etc. from entering Hindered by masks, glasses, throughout life scan and fingerprinting
casinos long hair, etc. Its resistance to false Users claim discomfort
Identify criminals, Pictures must be taken when matching or false with eye-related
terrorists, etc. the users have a neutral face positives, regarding to technology in general and
Find missing people “Considered an invasion of pupil movements the fact that retina scan
privacy to be watched” The eye from a dead technology has limited
Prevents elector frauds
person would deteriorate uses
Targets shoppers Easy to abuse
too fast to be useful, so no Users commonly fear that
extra precautions have to the device itself or the light
be taken with retinal scans inside the device can harm
to be sure the user is a their eyes in some way
Brain Signals as a New Biometric Authentication living human being Users claim discomfort
Method Using Brain-Computer Interface, The retina is located deep with the fact that they must
Table 2 Fingerprint biometric advantages and within one’s eyes and is position their eye very
disadvantages highly unlikely to be close to the device
altered by any Many also feel that these
Fingerprint identification
environmental or temporal retina scans can be linked
Advantages Disadvantages condition to eye disease
It is one of the most It needs more computer
developed biometrics memory to store scanned
Very high accuracy data
Iris Scan
Easy to use
Is the most economical Using the fingerprint
Iris is another part of the eyes, which has complex
biometric PC user scanner does not take into patterns that are stable, unique, and, in compare to
authentication technique consideration when a the retina, can be observable from a long distance.
It is standardized person physically changes The pattern-recognition method in Iris Scan process
Small storage space It can make mistakes with is using video images from a person’s iris. In iris
required for the biometric the dryness or dirtiness of identification, the probability of error is the lowest
template, reducing the size the finger’s skin, as well as
of the database memory with the age (is not of all biometrics. (Shekar and Bhat 2015). Table 4
required appropriate with children, shows some specific advantages and disadvantages
because the size of their of this biometric technique (Biometrictoday).
fingerprint changes
quickly)
Current Biometric Technique Advantages and
Disadvantages
Biometrics have too many different techniques
and methods. After investigating some of the
Retina Scan physiological biometric methods and the specific
In the backside of the eyeball, there is a layer of advantages and disadvantages of them, the
cells, which is the retina. This part of the eye authors are going to peruse the current biometric
converts light into nerve signals. To replicate a technique’s advantages and disadvantages.
retina, there is no known way discovered. The Table 5 shows advantages and disadvantages of
pattern of the blood vessels at the back of the recent biometric methods (Le and Jain 2009).
eye is unique. It stays the same for the whole
lifetime (Choraś 2012). Table 3 shows some spe- Brainwaves as a New Biometric Authentication
cific advantages and disadvantages of this biomet- The potential for using brainwaves as human bio-
ric technique (Jatin). metric identification has risen to the surface once
232 Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface

Brain Signals as a New Biometric Authentication Brain Signals as a New Biometric Authentication
Method Using Brain-Computer Interface, Method Using Brain-Computer Interface,
Table 4 Iris biometric advantages and disadvantages Table 5 Biometric methods’ advantages and
disadvantages
Iris scan
Advantages Disadvantages Biometric advantages and disadvantages
Scalability: This Distance: Iris is small and Advantages Disadvantages
technology is highly cannot be located from a Convenient: The Physical traits are not
scalable and can be used in few meters distance credentials are with human changeable: Users can
both large- and small-scale forever, so it does not reset a password, but they
programs require you to memorize never can change their
Accuracy: Iris Expensive: Iris scanners or note down anything fingerprints or retina; these
recognition is one of the are relatively higher in are fixed
best biometric modalities cost compared to other Security: Biometric Unhygienic: In contact-
in terms of accuracy biometric modalities technology brings based biometric
Stable: Iris patterns Infrared light: The different types of techniques, a biometric
remain stable throughout constant use of this system solutions, which are nearly device is used a lot of
an individual’s life. It is may cause harm to the iris impossible to hack unlike times by enormous
protected by the body’s because it is constantly passwords amount of people.
own mechanism being scanned with Everyone is actually
infrared light sharing his or her germs
Noninvasive: Iris Movement: A person has with each other via the
recognition can be done to be steady in front of the device
with simple video device to be enrolled by Scalability: Unlike other Error rate: Usually,
technology. No use of iris scanners. It means this solutions, biometrics are biometric devices make
laser technology is device cannot be used like highly scalable solutions two types of errors, false
necessary to scan the iris face recognition devices to for all types of projects. It acceptance rate (FAR) and
making it a noninvasive scan anybody, regardless is possible for any kinds of false rejection rate (FRR)
technology altogether of their movements projects because of the (Wayman et al. 2005).
Easy to use: Iris Obscure: Eyelashes, scalability of its solutions When the device accepts
recognition system is plug lenses, and reflections, an unauthorized person, it
and play compared to which create a problem, is known as FAR, and
other modalities of more often than not, when it rejects an
biometric recognition. obscure it authorized person, it is
A person needs to stand known as FRR
still in front of the camera, Accuracy: Biometric Delay: Some biometric
and the job is done works with individual’s devices take more than the
instantly. It is a physical traits such as accepted time and a long
comfortable process for fingerprints, face, and queue of workers form
everyone retina among others that waiting to be enrolled in
Fast: With iris recognition Reflection: In some cases, will always serve you large companies
system, a person can it is hard to perform an iris accurately anywhere,
complete the process scan due to the presence of anytime
within just a few seconds reflections. It could Flexibility: People have Environment and usage
happen in case of their own security matters: Environment
eyelashes, lenses, and credentials with you, so and usage can affect the
anything in general that you do not need to bother overall measurements
would cause a reflection memorizing awkward taken
Traceable: The encoding Memory space: A lot of alphabets, numbers, and
and decision making of memory is required for the symbols required for
iris pattern is traceable. It data to be stored and later creating a complex
takes only 30 milliseconds accessed password
for the image analysis and Transformation: Iris may Save money: With a little Physical disability: Some
the subsequent encoding deform nonelastically as money, any company can people are not fortunate
the pupil may change its track their employees and enough to be able to
size due to medical or reduce the extra costs they participate in the
other conditions are paying for years enrolment process. They
might have lost or
(continued)
Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface 233

Brain Signals as a New Biometric Authentication Electroencephalography (encephalon ¼ brain)

Method Using Brain-Computer Interface, Table 5 or EEG is an electrophysiological observing tech-
(continued)
nique to capture electrical activity generated by
Biometric advantages and disadvantages the brain from electrodes placed on the scalp
Advantages Disadvantages surface (Niedermeyer and da Silva, 2005). “EEG B
damaged body parts such refers to the recording of the brain’s spontaneous
as fingers or eyes
electrical activity over a period, as recorded from
Save time: Biometric
solutions are highly time
multiple electrodes placed on the scalp”
conserving (Niedermeyer and da Silva 2005). Many of the
Trustable: Reports claim brain’s emotion recognition techniques have been
that the young generations implemented and proposed for the last few years
trust biometric solutions which most of them involved the extraction of
more than other solutions
EEG signals (Abuhashish et al. 2015). In compar-
ison to other imaging methods, EEG has some
benefits. It is an excellent tool for studying the
again, an idea presented as a way to distinguish processes of neurocognitive underlying person
humans with thoughts. Before becoming a behavior because of some reasons such as the
method of security, it could be the measuring following: (1) EEG has very high time resolution
standard for biometric identification in the near and captures cognitive processes in the time frame
future, but it needs more time and work on in which cognition occurs. (2) EEG directly mea-
it. Brain signal can be one of the most practical sures neural activity. (3) EEG is inexpensive, light-
biometric authentication methods (Fig. 2). It is weight, and portable. (4) EEG monitors cognitive-
obviously a possible technique of identification, affective processing in the absence of behavioral
and in terms of security, it could have a big role to responses (Catarino et al. 2011). In terms of fre-
play in the feature. quency, there are five types of brainwave (gamma,
beta, alpha, theta, delta), and “each frequency is
Neural Oscillations (Brainwaves) measured in cycles per second (Hz) and has its own
Neural oscillations or brainwaves are an essential set of characteristics representing a specific level of
mechanism to enable the synchronization of neu- brain activity and a unique state of consciousness”
ral activity inside and around brain areas and help (Korde and Paikrao 2018). This is represented in
the accurate temporal organization of neural pro- Table 7. According to the types of brainwaves, four
cesses underlying memory, cognition, behavior, different kinds of mental activities such as move-
and perception (Neustadter et al. 2016). “The ment, emotions, talking, and motor imagery have
interaction between neurons can give rise to oscil- been measured by investigators (Abuhashish et al.
lations at a different frequency than the firing 2014).
frequency of individual neurons” (Roux and
Uhlhaas 2014). Brain-Computer Interface (BCI)
“Brain-computer interface is a method of commu-
Electroencephalography (EEG) nication based on neural activity generated by the
Hans Berger was a German psychiatrist who brain and is independent of its normal output path-
observed the first human neural oscillations as ways of peripheral nerves and muscles”
early as 1924. He invented electroencephalogra- (Vallabhaneni et al. 2005). BCIs are developed by
phy (EEG) for the recording of “brainwaves” by the research community with some applications in
measuring electrical activity in the patient’s brains mind to the generation of new assistive devices
in the hospital which had a skull damage (Millet (Rao and Scherer 2010). Brain-computer interface
2002). EEG is one of the techniques for brain technology is a powerful communication tool for
imaging. Table 6 shows the different brain imag- both users and systems. “The BCI has the capacity
ing techniques (Imotion EEG packet guide). to access brain activities and impart relevant
234 Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface

Brain Signals as a New

Biometric
Authentication Method
Using Brain-Computer
Interface, Fig. 2 Brain
signals beside the most
practical biometric
authentication techniques

Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface, Table 6 The
methodology of brain imaging techniques and the ways that they work
Brain imaging techniques
Methodology What is imaged? How?
Electroencephalography Changes in electrical brain Electrodes placed on scalp measure electrical
(EEG) current brainwaves
Positron emission Emissions from radioactive Radioactive isotopes injected into the blood are
tomography (PET) chemicals in the blood detected like X-ray
Computed (axial) X-ray images of the brain Multiple images (tomograms) are taken by
tomography scan (CT or rotating X-ray tubes. Doesn’t image function
CAT)
Functional magnetic Blood flow; oxyhemoglobin-to- Relies on the magnetic properties of blood.
resonance imaging (fMRI) deoxyhemoglobin ratio Shows brain function spatially and temporally
Magnetoencephalography Changes in electrical brain Similar to EEG but magnetic brainwaves are
(MEG) current measured instead of electrical brainwaves

information on the emotional status of the user” BCI Types

(AbuHashish 2015). There is no need for any exter- The BCI can be separated into invasive, partially
nal devices or muscle involvement to issue instruc- invasive, and noninvasive types (Abhang et al.
tions and complete the communication (Bi et al. 2016). In invasive BCI, recording the signals
2013). Normal individuals have been targeted in occurs when electrodes enter the brain tissue.
most recent studies by exploring the use of BCIs as This is a permanent basis method, which buries
an input device and exploring the generation of electrodes within the brain. Partially invasive BCI
hands-free applications (van Erp et al. 2012). is a process in which electrodes are placed inside
Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface 235

of the skull but rest outside the brain rather than in the BCI area that can read the electrical brain
within the gray matter above the brain’s surface. functions (Abuhashish et al. 2015b). The follow-
A good example for partially invasive BCI is ing image (Fig. 3) shows the different layers of the
electrocorticography (ECoG). ECoG is a type of brain and where the signal is taken from by three
monitoring that uses electrodes placed directly on different methods such as EEG, ECoG, and B
the bare surface of the brain to record brainwaves implant.
from the cerebral cortex (Palmini 2006). In non-
invasive type of BCI, no surgery is needed. BCI System Process
Instead, the sensors or electrodes are placed over A BCI is a system that can distinguish a definite
the head (via a hat, belt, patch, or a headset) set of forms in brain signals following five
to measure electroencephalography (EEG), sequential stages (Fig. 4): signal acquisition, pre-
which reads the rhythm of brain activities processing or signal enhancement, feature extrac-
(Mayoclinicstaff Electromyography (EMG)). tion, classification, and the application interface
There are many brain devices that are used to (Khalid et al. 2009). You can see in Fig. 4 that the
capture brain activities. These devices are brain first part of BCI process starts with acquiring the
controllers, which are very common technologies signals from the brain and goes to the next section,

Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface, Table 7 Types of
brainwaves and their frequency rates and mental state situation
Wave Frequency Mental state
Gamma Above 40 Hz Thinking, integrated thought

Beta 13–40 Hz Alertness, focused, integrated, thinking, agitation, aware of self and
surroundings

Alpha 8–12 Hz Relaxed, non-agitated, conscious state of mind

Theta 4–7 Hz Intuitive, creative, recall, fantasy, dreamlike, drowsy, and knowing

Delta 0.1– 4 Hz Deep, dreamless sleep, trance, and unconscious

Brain Signals as a New Non-Invasive BCI (EEG)

Biometric EEG
Authentication Method ECoG
Using Brain-Computer Implant
Interface, Fig. 3 The way
Partially-Invasive BCI (ECoG)
that BCI captures the
signals from the human
brain (Mayoclinicstaff
Electromyography (EMG))
Invasive BCI (implant)

= The length of time from the

1s recorded signal
236 Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface

which has three subsections for signal processing called signal enhancement (Norani et al. 2010).
to make the signal ready to use in different appli- In general, the acquired brain signals are unclear
cations and for different purposes. by noise and artifacts. The artifacts are eye
Signal acquisition is a considerable challenge blinks, eye movements, and heartbeat. In addi-
in the field of BCI. Some methods focus on EEG tion to these, muscular movements and power
signals; however, other methods exist that can line intrusions are also mixed with brain signals
capture neurological activity. End use is a factor (Bin et al. 2009). A couple of different methods
that is intended by the designer which filters out are used for artifact removal which “the most
which method you should use for capturing spe- frequently used methods are Common Average
cific signals (Major and Conrad 2014). Different Referencing (CAR), Surface Laplacian (SL),
methods for signal acquisition have been studied. Common Spatial Patterns (CSP), Independent
There are two general classes of brain acquisition Component Analysis (ICA), Principal Compo-
methods, which are invasive and non-invasive nent Analysis (PCA) and Adaptive Filtering”
(Fig. 5). Each method is using different types of (Lee et al. 2010). Overall, these techniques
BCI devices. have specific purposes that could match each
After signal acquisition part, signals are going objective of experiments conducted (Mallick &
to be preprocessed. Signal preprocessing is also Kapgate, 2015).

Signal Processing

Signal Preprocessing
Feature Classification Application
Extraction (Detection)
Acquisition Interface

Applications
Feed Back
Spelling Device
Neuroprosthesis
VR
etc.

Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface, Fig. 4 Brain-
computer interface process

Signal Acquisition Methods

Invasive Non-Invasive

Cortical Surface EEG MEG fMRI fNIRS

Intracortical
(ECoG)

Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface, Fig. 5 Brain signal
acquisition’s methods
Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface 237

After preprocessing and filtering, the EEG sig- scientists and researchers have been doing many
nals will pass through feature extraction process attempts to observing the pattern uniqueness of
and select particular features by some feature the brain signal. Several different methods have
selection methods. Some researcher used a hybrid been used to analyze EEG signals. In regard to the
BSS-SVM system to extract the movement- recent progression of EEG signal acquisition B
related features from the EEGs (Peterson et al. devices, the capability of providing better results
2005). In most existing BCI, this identification is going higher, and these processes are getting
relies on a classification algorithm. Using classi- simpler. The authors are going to review a couple
fication algorithms is the most popular tactic for of different tactics of EEG capturing methods to
this purpose. These procedures are used to iden- acquire better accuracy and check the applicability
tify “patterns” of brain activity (McFarland of using signal authentication purposes. There are
et al. 2006). four different studies in this research area that
Classification algorithms divided into five dif- were more successful and reported better accuracy
ferent categories: linear classifiers, neural net- in their experiences (Jayarathne et al. 2017) which
works, nonlinear Bayesian classifiers, nearest are linear discriminant analysis (LDA), cosine
neighbor classifier, and combinations of classi- similarity ! LDA, power spectral density (PSD)
fiers (Lotte et al. 2007). BCI has many applica- and spectral coherence (COH) ! Mahalanobis
tions, especially for disabled persons. It reads the distance and match-score fusion, and event-
signals generated by the brain and translates them related potentials (ERP). These studies used dif-
into activities and commands that can control the ferent tasks, extracted features, and classifiers for
computers (Lotte 2006). Figure 6 shows the dif- doing their experiments to get higher accuracy
ferent types of BCI applications. rates of brainwaves to use them for authentication
purposes. Besides the studies mentioned above,
BCI Security Authentication Using EEG Signals Table 8 summarizes some other studies with
As the authors mentioned before, a couple of achieved accuracy and other characteristics. Gen-
different ways are there which are designed for erally, accuracy of each system depends highly on
acquiring the brain activities noninvasively, these aspects.
including magnetoencephalography (MEG),
functional magnetic resonance imaging (fMRI),
positron emission tomography (PET), near- Discussion
infrared spectroscopy (NIRS), and electroenceph-
alography (EEG). In comparison to other Recent biometric user authentication techniques
methods, EEG is a noninvasive method, which is have some problems and limitations. To cover
not very expensive and allows recording the sig- the recent biometric limitations, we need a
nals passively. EEG-based user authentication new biometric brainwave-based authentication,
systems are currently popular in BCI security which is another technique in the extensive range
and authentication applications. Recently, of authentication systems. There are many

BCI
Applications

Neuroergonomic
Neuromarketing & Educational & Games Security &
Medical & Smart
Advertisement Self-Regulation & Entertainment Authentication
Environment

Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface, Fig. 6 BCI
application fields
238 Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface

Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface, Table 8 Summary
of various studies (in decreasing order of accuracy) (Jayarathne et al. 2017)
No. of Derived or extracted Avg.
Author(s) Channels subjects Task feature Classifier accuracy
Rui- 3 50 Visual simulation of Event-related Normalized 100%
Blondet 400 images potentials (ERP) cross-correlation
et al.
La Rocca 64 108 Relaxation with Power spectral density Mahalanobis 100%
et al. opened eyes and (PSD), spectral distance-based
closed eyes coherence (COH) classifier and
match-score
fusion
Chen et al. 16 29 Rapid serial visual Point-biserial Linear 100%
presentation (RSVP) correlation discriminant
coefficients, Fisher’s analysis (LDA)
transformation
Palaniappan 6 6 5 tasks: relaxation, Auto-regressive LDA 100%
math activity, coefficients (AR),
geometric figure spectral power (SP),
rotation, mental inter-hemispheric
letter composition, power differences
visual counting (IHPD), inter-
hemispheric linear
complexity (IHLC)
Ashby et al. 14 5 4 tasks: relaxation, AR, SP, IHPD, IHLC, Support vector 100%
limb movement, PSD machine (SVM)
geometric figure
rotation, visual
counting
Chuang 1 15 7 tasks: breathing, Cosine similarity of k-Nearest 99%
et al. simulated finger the vector neighbor (k-NN)
movement, sport representation
activity, singing/
passage recitation,
audio listening, color
identification, and
pass-thought
Palaniappan 61 20 Drawing of common Multiple signal k-NN, Elman 98%
objects as visual classification neural network
simulation (MUSIC) (ENN)
Riera et al. 4 – Relaxation AR, fast Fourier Fisher 98%
transform (FFT), discriminant
mutual information, analysis (FDA)
coherence, cross-
correlation (EEG and
ECG data)
Jayarathne 14 12 Imagining four-digit Common spatial LDA 97%
et al. number as cognitive patterns (CSP)
task
Liew et al. 8 10 Apprehension of Coherence, cross- Fuzzy-rough 92%
images as visual correlation, mean nearest neighbor
simulation amplitude (FRNN)
Yeom et al. 18 10 Apprehension of Difference of average Nonlinear SVM 86.1%
images of faces signals, positive/ classifier
including self-face negative peaks at
as visual simulation specific latencies
Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface 239

researches about brain signal patterns and using not functioning properly if this classification was
them as a person authentication. Electroenceph- used for a larger group of users on traditional
alogram (EEG) signals are the most popular hardware and it is less than 100%.
method in this process. A couple of different Ruiz-Blondet et al. (2016) presented a protocol
approaches are presented in this way to capture known as CEREBRE with a band-pass filtering B
EEG signals and classify them with different between 1 and 55 Hz, and based on normalized
classification methods to find the unique signals cross-correlation, a simple discriminant function
and use them as an authentication method with was used for classification. The nominal (four cate-
more accuracy. gories, three channels) classifier showed the highest
Chen et al. (2016) proposed a system within accuracy when all the patterns were used, but both
authentication, which is centered on rapid serial maximum and minimum classifiers showed 100%
visual presentation (RSVP) stimulus. A brain accuracy. The results presented that the most accu-
amplifier was used to obtain EEG signals and rate was for the stimulus oddball and food. The
linear discriminant analysis (LDA) to classify resting pattern had a reduced performance in terms
them. A specific association constant calculated of classification. Authentication centered on a
the important features. According to the author’s memory-evoking task (also known as “pass-
notation, a password can be hidden effectively in thoughts” in other studies) (Thorpe et al. 2005)
certain compulsive situations. also showed weak results; this is due to the incon-
Chuang et al. (2013) presented a new approach stant time that was consumed to allow thinking.
which used the MindWave to obtain data. Seven According to the limitations of the methods
tasks were executed, including sports activity, which have been presented in the published
breathing, audio listing, simulation of finger papers and researches, the authors need better
movement, color, reciting and identifying techniques like using different tasks and user strat-
music with singing, and pass-thoughts. The clas- egies to acquire brain signals, better methods for
sification process is done with the k-nearest preprocessing and feature extraction, and better
neighbor (k- NN) algorithm. The most accurate classifiers to find the unique brain signal and use
strategies were for color, audio, and sport. The it as a new biometric authentication.
most difficult one was for the pass-thought task
according to the results of the questionnaire that
determined user-friendliness with different Conclusion
tasks. Breathing, audio, and color were the
straightforward tasks. In the near future, biometric authentication
La Rocca et al. (2014) presented an approach methods will be the most useful methods for
centered around connectivity within EEG spectral devices and applications for security because of
coherence. In this method, data samples were gath- the usability and security level and they are easier
ered from 108 participants during open resting and to use. However, there are some disadvantages for
closed eyes positions. EEG data was captured some methods. Brainwave is another human bio-
using a system consisting of 64 different channels metrics. There were some experiments using brain
with a rate of 160 Hz. Data was filtered to 50 Hz via signals as an authentication method, which in
a low-pass anti-aliasing filter. Spectral coherence some methods high accuracy is acquired. How-
(COH) and power spectral density (PSD) analysis ever, they had some limitations, which can be
techniques were used to extract mental features. To improved in the future.
calculate uniqueness, two different algorithms Brainwaves are another human biometric that
were used separately in this process which were could be the most secure biometric technique. In
Mahalanobis classifiers that were based on distance comparison with other biometric techniques in
and match-score fusion system. This technique is terms of security, the human brain signal has a
strong and very accurate for user identification. The couple of important advantages. It is the only
performance of classification has the possibility of biometric that is changeable, it is not visible to
240 Brain Signals as a New Biometric Authentication Method Using Brain-Computer Interface

duplicate and does not have shoulder surfing Chuang, J., Nguyen, H., Wang, C., Johnson, B.: I think,
problem, and it is more useful for disabled people therefore I am: usability and security of authentication
using brainwaves. In: International Conference on
in which a good example would be the famous Financial Cryptography and Data Security, pp. 1–16.
scientist Stephen Hawking who was using a kind Springer, Berlin (2013)
of brain-computer interface technology. Darril.: https://blogs.getcertifiedgetahead.com/identification-
authentication-authorization. Last accessed 24 June 2018
DeBow, B., Syed, K.: 802.11 wireless network end-user
authentication using common access cards. In: Military
References Communications Conference 2006, MILCOM2006,
pp. 1–5. IEEE Washington DC, USA (2016)
Abhang, P.A., Gawali, B.W., Mehrotra, S.C.: Introduction Erden, M.: Advantages and Disadvantages of Biometric
to EEG and Speech-Based Emotion Recognition. Aca- Authentication. http://www.sestek.com/2016/11/advanta
demic, Amsterdam (2016) ges-disadvantages-biometric-authentication. Last acce-
AbuHashish, F.A.M.: Classification technique for human ssed 15 June 2018
emotion in virtual reality using game-based brain com- Faundez-Zanuy, M.: Biometric security technology. IEEE
puter interface. Doctoral dissertation, Universiti Aerosp. Electron. Syst. Mag. 21(6), 15–26 (2006)
Teknologi Malaysia (2015) Imotion EEG packet guide.: https://imotions.com/wp-
Abuhashish, F.A., Sunar, M.S., Kolivand, H., Mohamed, content/uploads/Guides/iMotions_Guide_EEG_Pocket
F., Mohamad, D.B.: Feature extracted classifiers based Guide_2016.pdf. Last accessed 3 July 2018
on eeg signals: a survey. Life Sci. J. 11(4), 364, Malay- Jafri, R., Arabnia, H.R.: A survey of face recognition
sia (2014) techniques. Jips. 5(2), 41–68 (2009)
Abuhashish, F.A.M., Kolivand, H., Shahrizal, M.: Frame- Jain, A., Bolle, R., Pankanti, S.: Biometrics: Personal Iden-
work of controlling 3D virtual human emotional walk- tification in Networked Society. Kluwer Academic,
ing using BCI. Jurnal Teknologi, 75(4), Malaysia Norwell (1999)
(2015) Jatin, C.: Retina scan: eye on security. https://www.
Abuhashish, F.A., Zraqou, J., Alkhodour, W., Sunar, M.S., inteligentcomp.com/2012/10/all-about-retina-scan.html.
Kolivand, H.: Emotion interaction with virtual reality Last accessed 29 June 2018
using hybrid emotion classification technique toward Jayarathne, I., Cohen, M., Amarakeerthi, S.: Survey of
brain signals. Int. J. Comput. Sci. Inf. Technol. 7(2), EEG-based biometric authentication. In: Awareness
159 (2015b) Science and Technology, iCAST, 2017 IEEE 8th Inter-
Agrafioti, F., Bui, F.M., Hatzinakos, D.: Medical biomet- national Conference, pp. 324–329. IEEE, Taichung,
rics: the perils of ignoring time dependency. In: Taiwan (2017)
Proc. IEEE 3rd Int. Conf. Biometrics, Theory, Appl, Khalid, M.B., Rao, N.I., Rizwan-i-Haque, I., Munir, S.,
Syst, pp. 1–6. Washington DC, USA (2009) Tahir, F.: Towards a brain computer interface using
Bi, L., Fan, X.A., Liu, Y.: Eeg-based brain-controlled wavelet transform with averaged and time segmented
mobile robots: a survey. IEEE Trans. Hum-Mach adapted wavelets. In: Computer, Control and Commu-
Syst. 43(2), 161–176 (2013) nication, 2009. ic4 2009. 2nd International Conference,
Bin, G., Gao, X., Yan, Z., Hong, B., Gao, S.: An online pp. 1–4. IEEE Karachi, Pakistan (2009)
multi-channel SSVEP-based brain–computer interface Kittler, J., Ballette, M., Czyz, J., Roli, F., Vandendorpe, L.:
using a canonical correlation analysis method. J. Neural Enhancing the performance of personal identity
Eng. 6(4), 046002 (2009) authentication systems by fusion of face verification
Biometrictoday: 25 advantages and disadvantages of iris experts. In: Multimedia and Expo, 2002. ICME’2. Pro-
recognition. Available at: https://biometrictoday.com/25- ceedings. 2002 IEEE International Conference, vol. 2,
advantages-disadvantages-iris-recognition. Last accessed pp. 581–584. IEEE, Lausanne, Switzerland (2002)
29 June 2018 Kodituwakku, S.R.: Biometric authentication: a review.
Catarino, A., Churches, O., Baron-Cohen, S., Andrade, A., Int. J. Trend Res. Dev. 2(4), 2394–9333 (2015)
Ring, H.: Atypical eeg complexity in autism spectrum Korde, K.S., Paikrao, P.L.: Analysis of EEG signals and
conditions: a multiscale entropy analysis. Clin. biomedical changes due to meditation on brain: a
Neurophysiol. 122(12), 2375–2383 (2011) review. Analysis 5(01), 603–606 (2018)
Chen, Y., Atnafu, A.D., Schlattner, I., Weldtsadik, W.T., Kute, A., Kumar, V.: Fingerprint recognition system: A
Roh, M.C., Kim, H.J., Lee, S.W., Blankertz, B., Fazli, review. International Journal of Electrical, Electronics
S.: A high-security eeg-based login system with rsvp and Computer Engineering, 3(2), 61. (2014)
stimuli and dry electrodes. IEEE Trans. Inf. Forensic La Rocca, D., Campisi, P., Vegso, B., Cserti, P., Kozmann,
Secur. 11(12), 2635–2647 (2016) G., Babiloni, F., Fallani, F.D.V.: Human brain distinc-
Choraś, R.S.: Retina recognition for biometrics. In: Digital tiveness based on EEG spectral coherence connectivity.
Information Management, ICDIM, 2012 Seventh Inter- IEEE Trans. Biomed. Eng. 61(9), 2406–2412 (2014)
national Conference, pp. 177–180. IEEE, Macau, Le, C., Jain, R.: A Survey of Biometrics Security Systems.
China (2012) EEUU. Washington University in St. Louis (2009)
B-Spline 3D Curves 241

Lee, P.L., Sie, J.J., Liu, Y.J., Wu, C.H., Lee, M.H., Shu, C. Rao, R.P., Scherer, R.: Brain-computer interfacing [in the
H., Li, P.H., Sun, C.W., Shyu, K.K.: An ssvep-actuated spotlight]. IEEE Signal Process. Mag. 27(4), 152–150
brain computer interface using phase-tagged flickering (2010)
sequences: a cursor system. Ann. Biomed. Eng. 38(7), Roux, F., Uhlhaas, P.J.: Working memory and neural oscil-
2383–2397 (2010) lations: alpha–gamma versus theta–gamma codes for
Lotte, F.: The use of fuzzy inference systems for classifi- distinct wm information. Trends Cogn. Sci. 18(1), B
cation in EEG-based brain-computer interfaces. In: 3rd 16–25 (2014)
International Brain-Computer Interfaces Workshop and Ruiz-Blondet, M.V., Zhanpeng, J., Sarah, L.: Cerebre: a
Training Course (2006) novel method for very high accuracy event-related
Lotte, F., Congedo, M., Lécuyer, A., Lamarche, F., potential biometric identification. IEEE Trans. Inf.
Arnaldi, B.: A review of classification algorithms for Forensic Secur. 11(7), 1618–1629 (2016)
EEG-based brain–computer interfaces. J. Neural Eng. Shekar, B.H., Bhat, S.S.: Steerable riesz wavelet based
4(2), R1 (2007) approach for iris recognition. In: Pattern Recogni-
Major, T.C., Conrad, J.M.: A survey of brain computer tion, ACPR, 2015 3rd IAPR Asian Conference,
interfaces and their applications. In: SOUTHEASTCON pp. 431–436. IEEE, Kuala Lumpur, Malaysia
2014, IEEE, pp. 1–8. IEEE (2014) (2015)
Mallick, A., Kapgate, D.: A review on signal pre- Tarun, A.: Fingerprint Identification. https://www.
processing techniques in brain computer interface. Int. elprocus.com/fingerprint-identification. Last accessed
J. Comput. Technol. 2(4), 130–134 (2015) 29 June 2018
Margaret, R.. https://searchsecurity.techtarget.com/defini Thorpe, J., van Oorschot, P.C., Somayaji, A.: Pass-
tion/authentication. Last accessed 24 June 2018 thoughts: authenticating with our minds. In: Proceed-
Margaret, R.: Facial recognition. https://searchenterpriseai. ings of the 2005 Workshop on New Security Paradigms
techtarget.com/definition/facial-recognition. Last accessed pp. 45–56. ACM (2005)
29 June 2018 Vallabhaneni, A., Wang, T., He, B.: Brain-computer
Masupha, L., Zuva, T., Ngwira, S., Esan, O.: Face recog- interface. In: He B. (eds) Neural Engineering. Bio-
nition techniques, their advantages, disadvantages and electric Engineering pp. 85–121. Springer, Boston
performance evaluation. In: Computing, Communica- (2005)
tion and Security ICCCS, 2015 International Confer- van Erp, J., Lotte, F., Tangermann, M.: Brain-computer
ence, pp. 1–5. IEEE, Pamplemousses, Mauritius (2015) interfaces: beyond medical applications. Computer.
Mayoclinicstaff Electromyography (EMG). https://www. 45(4), 26–34 (2012)
mayoclinic.org/tests-procedures/emg/about/pac-203939 Vangie, B.: (n.d.). https://www.webopdia.com/TERM/R/
13. Last accessed 5 July 2018 rsa_secure_id.html. Last accessed 24 June 2018
McFarland, D.J., Anderson, C.W., Muller, K.R., Schlogl, A., Wayman, J., Jain, A., Maltoni, D., Maio, D.: An introduc-
Krusienski, D.J.: Bci meeting 2005-workshop on bci sig- tion to biometric authentication systems. In: Biometric
nal processing: feature extraction and translation. IEEE Systems, pp. 1–20. Springer, London (2005)
Trans. Neural Syst. Rehabil. Eng. 14(2), 135–138 (2006)
Millet, D.: The origins of EEG. In: 7th Annual Meeting of
the International Society for the History of the Neuro-
sciences, ISHN (2002)
Neustadter, E., Mathiak, K., Turetsky, BI.: EEG and MEG
probes of schizophrenia pathophysiology. In: The Neu- Brain-Computer Interface
robiology of Schizophrenia, pp. 213–236 (2016)
Niedermeyer, E., da Silva, F.L.: Electroencephalography:
Basic Principles, Clinical Applications, and Related ▶ Brain Signals as a New Biometric Authentica-
Fields. Lippincott Williams & Wilkins (2005) tion Method Using Brain-Computer Interface
Norani, N.M., Mansor, W., Khuan, L.Y.: A review of signal
processing in brain computer interface system. In: Bio-
medical Engineering and Sciences, IECBES, 2010
IEEE EMBS Conference, pp. 443–449. IEEE, Kuala
Lumpur, Malaysia (2010) Brawl
Palmini, A.: The concept of the epileptogenic zone: a mod-
ern look at Penfield and Jasper’s views on the role of
▶ Super Smash Bros.: A Brief History
interictal spikes. Epileptic Disord. 8(2), 10–15 (2006)
Peterson, D.A., Knight, J.N., Kirby, M.J., Anderson, C.W.,
Thaut, M.H.: Feature selection and blind source separa-
tion in an eeg-based brain-computer interface. EURASIP
J. Adv. Sig. Process. 2005(19), 218613 (2005)
Prasanna, S.R.M., Sahoo, S.K., Choubisa, T.: Multimodal
B-Spline 3D Curves
biometric person authentication: a review. IETE Tech.
Rev. 29, 54–75 (2012) ▶ B-Splines
242 B-Spline Computer-Aided Design

Introduction
B-Spline Computer-Aided
Design Before the evolution of computer graphics, the
aircraft wings and automobile parts were designed
▶ B-Splines through splines. A spline constitutes long wood or
plastic pieces of flexible nature where rectangular
section is put in place at several positions using
heavy lead weights commonly known as ducks.
B-Spline Polygons The duck places the spline at fixed positions with
respect to the drawing board (Beach 1991). This
▶ B-Splines helps spline to take the natural shape considering
ducks. The spline’s shape can be changed through
ducks’ movement. This has several drawbacks
such as duck positions recording, drafting equip-
B-Spline Surfaces ment required towards complex parts, consumer
costs, absence of closed form solutions, etc. (Buss
▶ B-Splines 2003).
As such, polygons give good rendering. But a
better way is required towards generating the curved
surfaces. For a designer, it is difficult to manipulate
B-Splines directly billions of polygons which make up the
rendered model. A general way is required to spec-
Arindam Chaudhuri ify arbitrary curved surfaces that can be converted to
Samsung R & D Institute Delhi, Noida, India rendering polygons. For this, a mechanism is
required which allows to specify any smooth desired
curved surface. The solutions are generally provided
Synonyms by three categories of surfaces, viz., Bézier surfaces,
B-Spline surfaces, and subdivision surfaces. In this
B-Spline 3D curves; B-Spline computer-aided direction, the computer-aided design industry uses
design; B-Spline polygons; B-Spline surfaces NURBS surfaces as its standard definition mecha-
nism. The visual effects industry uses both NURBS
and subdivision surfaces. With the introduction of
Definition UNISURF which is a CAGD software tool by Pierre
Bezier (https://en.wikipedia.org/wiki/Pierre_
B-Splines are one of the most promising curves in Bézier), the smooth curves can be easily projected
computer graphics. They are blessed with some on screens and monitors at low physical storage
superior geometric properties which make them an space. This resulted in the evolution of various
ideal candidate for several applications in computer- CAD based software such as Maya, Blender,
aided design industry. In this entry, some basic 3DMax, etc. This followed the development of a
properties of B-Spline curves are presented. Two new mathematical structure called spline which is a
significant B-Spline properties, viz., convex hull smooth curve represented through few points.
property and repeated points’ effects are discussed. With this motivation, in this entry, we present
The B-Splines’ computation in computational the basic properties which define the B-Spline
devices is also illustrated. An industry application curves. Some of the significant B-Spline proper-
based on image processing where B-Spline curve ties discussed here are convex hull property and
reconstructs the 3D surfaces for CT image datasets repeated points’ effects. The computation task for
of inner organs further highlights the strength of B-Splines in computational devices is also
these curves. highlighted. An industry application revolving
B-Splines 243

around image processing with B-Spline applica- The knots should have parameter values ts
tions on 3D surface reconstruction towards inner along with them. The ts values at knots are
organs taken from CT images further highlights represented as tsj+n considering knot joint of
the significance of these curves. This entry is jthand (j + 1)th polynomial segments. This is in
organized as follows. Section “B-Splines: Over- addition to the parameter values tsn and tsN which B
view” gives an overview of the B-Spline curves correspond towards start and end for complete
with computation task for these curves. This is curve. There are also 2n parameter values
followed by an industry application involving ts0, . . . . . . , tsn1 and tsN+1, . . . . . . , tsN+n values
image processing with B-Splines. Finally, the which are associated blending polynomials. The
conclusions are given. values tsj are monotonically increasing values
which may be either equally spaced, integers or
positive.
B-Splines: Overview The functions BS nk ðtsÞ are recursively defined as:

The B-Splines are highly capable for describing ts  tsk

BS nk ðtsÞ ¼ BS n1 ðtsÞ
various forms of curves (Chaudhuri 2018). They tskþn  tsk k
form a special case for splines, to be more specific tskþnþ1  ts
þ BS n1 ðtsÞ (2)
the Bezier curves generalization. They are tskþnþ1  tskþ1 kþ1
constructed with orthonormal basis of recursive
functions. They comprise of curves which are of With unit step function being defined as:
polynomial nature at points which are basically
knots. The polynomial’s degree is identical to that 1 ts > 0
of B-Spline. The cubic segments form an impor- usðtsÞ ¼ (3)
0 ow
tant part of B-Splines. Such curves are known as
cubic B-Splines. Now we present some B-Splines The 0th order polynomial BS is:
along with certain examples highlighting the spe-
cial scenarios. BS 0k ðtsÞ ¼ usðts  tsk Þusðtskþ1  tsÞ (4)
Considering a, b, c coordinates with respect
to parameter t, B-Spline can be represented It is to be noted that BS nk 6¼ 0 considering
as: a ¼ a(ts), b ¼ b(ts), c ¼ c(ts). For B-Spline range tsk < ts < tsk + n + 1. Figure 1 illustrates
parametric curve, there exists certain discontinu-
ities for parametric functions at parameter values
which depict the knots. For B-Spline, there are
several polynomial curve sections as control
points number minus polynomial’s degree. Con-
sidering N control points and nth degree polyno-
mial, there exists (N  n) sections. The joining of
these sections happens at (N  n  1) knots. For
control points cpi ; i ¼ 1, . . . . . . ,N with cpi ¼
hxi , yi , zi i. For polynomial curve’s dimensionality
connecting knots being n, then B-Spline’s para-
metric equation is:

N
V ðtsÞ ¼ BS nðk1Þ ðtsÞcpk tsn  ts  tsN ,
k¼1
N nþ1
(1) B-Splines, Fig. 1 Some representative B-Spline curves
244 B-Splines

some B-Spline curves. The B-Splines are basi-

cally characterized by the following properties:

(a) They can easily be represented through piece-

wise polynomial. This allows a B-Spline
curve to be represented as a linear combina-
tion of number of B-Splines. Any B-Spline
curve can be refined as linear combination of
piecewise segments as shown in Fig. 2. The
curves can also be refined through linear oper-
ation on control points.
(b) The unit integral can effectively represent a
þ1
B-Spline as: BS k ðtsÞd ðtsÞ ¼ 1.
1
(c) They are nonnegative in nature such that:
BSk(ts)  0.
(d) B-Splines always partition of unity: BS k
j
ðts  jÞ ¼ 1.
(e) They are highlighted through the support fac-
tor: BSk(ts) 6¼ 0, ts ∈ [0, k].

B-Splines, Fig. 2 B-Spline curve refined as linear com-

bination of piecewise segments
Now let us discuss some significant prop-
erties of B-Splines. The two important prop-
erties of B-Splines worth mentioning are the ts  k n1
BS nk ðtsÞ ¼ BS k ðtsÞ
convex hull property and the effect of n
repeated points. The justifications are trivial k þ n þ 1  ts n1
þ BS kþ1 ðtsÞ (7)
in nature. n
It is known that by recursion BS nk > 0, and it
can also be shown for one n at a time. As a result It is to be noted that BS 0k ðtsÞ ¼ BS 00 ðts  k Þ, such
of this, using recursion leads to: that by recursion BS nk ðtsÞ ¼ BS n0 ðts  k Þ: Hence,
for knots’ integer spacing which results in
N B-Splines of uniform nature, only single blended
BS nðk1Þ ðtsÞ  1,tsn < ts < tsN (5)
k¼1 polynomial is required towards each spline’s degree.
Now some examples are highlighted for cubic
With this, B-Splines satisfy the convex hull B-Splines. The higher degree polynomials become:
requirements.
The B-Spline curve moves near the coordi-
BS 10 ðtsÞ ¼ tsfusðtsÞusð1  tsÞg
nate, when adjacent points of control have þ ð2  tsÞfusðts  1Þusð2  tsÞg
identical coordinates. Considering n identical
(8)
adjacent control points, the spline interpolates
the point. 1
Next, we consider the case where integer knots BS 20 ðtsÞ ¼ ts2 ½fusðt Þusð1  tsÞg þ ftsð2  tsÞ
2
are equally spaced. Let us consider tsk ¼ k, then þ ð3  tsÞðts  1Þgfusðts  1Þusð2  tsÞg
from the above equations we have:
þ ð3  tsÞ2 fusðts  2Þusð3  tsÞg
BS 0k ðtsÞ ¼ usðts  k Þusðk þ 1  tsÞ (6) (9)
B-Splines 245

1 3 The above equations are further generalized as

BS 30 ðtsÞ ¼ ts fusðtsÞusð1  tsÞ
6 follows:
þ ts2 ð2  tsÞ þ tsðts  1Þð3  tsÞ
1
þ ðts  1Þ2 ð4  tsÞgfusðts  1Þusð2  tsÞg BS k ðtsÞ ¼
2k B
2
þ tsð3  tsÞ þ ðts  1Þð3  tsÞð4  tsÞ nþ1 nþ1
 BS k ð2ts  k Þ
k¼0 k
2
þ ðts  2Þð4  tsÞ gfusðts  2Þusð3  tsÞg (13)
3
þ ð4  tsÞ usðts  3Þusð4  tsÞ 
The subdivision operators can readily be applied
(10) on B-Splines. The subdivision works through bases
and control points. Considering a B-Spline, subdi-
The graphical plots corresponding to the four vision operator S can be enforced such that:
lowest order blending polynomials are shown in
Fig. 3. BS ðtsÞ ¼ BS ð2tsÞS (14)
With respect to the above equations, B-Spline
can also be refined as linear combination of dilates The term S in above equation represents the
and translates of the curve itself. This can be subdivision matrix. Figure 6 represents a typical
written recursively for k ¼ 0, 1 as: subdivision matrix. The subdivision operation is
stationary in nature.
BS 0 ðtsÞ ¼ BS 0 ð2tsÞ þ BS 0 ð2ts  1Þ (11) The subdivision can be successively applied on
control polygon using control points’ sequence.
BS 1 ðtsÞ The subdivision in B-Splines leads the resulting
1
¼ ½BS 1 ð2tsÞ þ 2BS 1 ð2ts  1Þ þ BS 1 ð2ts  2Þ curve to convergence, smoothness, and approxi-
2 mation. The subdivision matrix converges the
(12) curve towards cubic B-spline representation.
This allows the control polygon to be drawn
The above equations are represented through
instead of the curve. Using an iterative refinement
the curves shown in Fig. 4. Some of the refine-
process, the curve approximation is achieved
ment masks are also shown in Fig. 5.
through several refinements as shown in Fig. 7.

B-Splines, Fig. 3 The four

lowest order blending
polynomials corresponding
to uniform B-Splines
246 B-Splines

The surface approximation is done using tensor As the initial step, the uniform knot vector of
product which is defined over regular quadrilat- length n + k is considered. The calculation is
eral meshes. Sometimes semi-regular quadrilat- performed as follows. Each knot is initialized as
eral meshes are used which have few sufficiently zero. Then considering each 1 < i  n + k, it is
separated vertices. The semi-regular meshes rep- required to be verified whether conditions i > n
resent the boundary surface of an object. In geo- and i < n + 2 are satisfied. If the conditions hold,
metric modeling, patches of regular meshes are then current knot takes the value of previous knot
traditionally used and stuck together. But it with an increment of 1 otherwise, the current knot
becomes difficult towards handling patch density. considers the previous knot value. Now consider-
ing k ¼ n ¼ 4, the knot vector takes the
value {0, 0, 0, 0, 1, 1, 1, 1}. Before calculating
Computation Task for B-Splines the curve set points, it is required to fix a correct
step value for parameter ts. The step is reached by
The B-Splines are generally computed through dividing knot’s value considering one less than
fast computing devices (Chaudhuri 2018) because number of points on curve set.
small time steps are required by a smooth curve. Now we initiate the computing task towards
To make the computations simple, the calcula- curve set points for B-spline. Care should be taken
tions are specified with respect to regular uniform such that the number of basis functions which are
B-Splines. Let us start considering the spline calculated for each time step is equal to that of
order. Considering the order as k, a set of n control number of points in control set. Thus considering
points are specified. The number of points on the the entire spline, number of basis computations is
curve may vary depending on the smoothness same as the product of magnitudes of control and
desired by the user. These set of points are denoted curve sets. Hence for 20 points on curve set with
as the curve set. 4 control points, 80 basis functions are required to
be calculated.
At each step, Cox-de Boor algorithm is used in
order to calculate the basis function’s value. During
each iteration through the steps, the entire knot
vector is taken as with step value ts. It is observed
that Cox-de Boor algorithm is recursive. As a result
of this, care needs to be taken such that the basis is
1 calculated correctly. When the set of n basis func-
1/2 1/2 tions for control points at specified step ts is avail-
able, the curve points’ coordinates are calculated
by multiplying the ith basis function to ith control
point. The values obtained are then embedded into
B-Splines, Fig. 4 The curves on left and right are curve set. This helps towards generation for collec-
represented through above equations tion of points which when plotted together as well

B-Splines, Fig. 5 The

curves on left and right
represent the refinement
masks for 12 ð1,2,1Þ and
8 ð1,4,6,4,1Þ
1

1/2 (1, 2, 1) 1/8 (1, 4, 6, 4, 1)

B-Splines 247

as connected with line segments resembles a curve. considering the tomography frames. The least
The number of calculations required to make squares B-Spline fitting approximates filtered
B-spline are appreciably large. contours. The organ surface is approximated
through B-Spline surface. The technique requires
no human intervention. As such, high-quality sur- B
Industry Application faces in versatile format are created.
The contour selection process is a significant
In this section, a novel industry application step in surface reconstruction. Each contour is
involving B-Spline surfaces with image pro- selected with utmost care. Then noise is mini-
cessing is presented which is adopted from Ref. mized such that resulting surface is not affected.
(Partyka 2014). The experimental dataset contains Also, a high-fidelity product needs to be
about 1900 images from the CT scan of the entire maintained. Here enhancement of each CT slice
body. Only 500 images considering the chest and done and canny edge detector detects the con-
upper abdomen have been used for analysis. Here tours. An empirical threshold for edge detector is
3D surface is reconstructed for organs developed set. The morphological operations and filtering for
through tomography. It uses k-means clustering small contours is done to eliminate unwanted
and Hu moments in order to filter contours noise. The effects of filtering are presented in the
second and third images for the Figs. 8, 9, and 10.
The unwanted contours are removed through
B-Splines, Fig. 6 automatic noise filtering where larger datasets are
A subdivision matrix created using contour detection. The right lung is
taken as region of interest (RoI). About 1996,
contours covering chest, left lung, stomach,
aorta, etc. are created through contour detection
and filtering step. The right lung is selected man-
ually from 450 images. Because of shape and
contours location diversity, these contours are fil-
tered through k-means clustering. To address the
contour’s shape, an n-dimensional vector is cre-
ated which is composed of (X, Y) coordinates for
contour’s centroid and varying Hu moments.
After detecting and prefiltering contours, cluster-
ing is done. Considering each contour, Hu

B-Splines, Fig. 7 The iterative refinement process on curves

248 B-Splines

B-Splines, Fig. 8 Images from first dataset (original image, detected edges, detected contours)

B-Splines, Fig. 9 Images from second dataset (original image, detected edges, detected contours)

B-Splines, Fig. 10 Images from third dataset (original image, detected edges, detected contours)

moments and centroid are calculated which are testing is done for all combinations of feature
used after normalization towards k-means cluster- vectors based on contour centroid as well as its
ing. These clusters are used towards examined Hu moments. The best results are achieved
contours classification considering Euclidean dis- through second Hu moment with respect to cen-
tance from cluster which represents RoI. The troid coordinates.
B-Splines 249

B-Splines, Fig. 11 The correctly classified contours

B-Splines, Fig. 12 The false +ves classified as contours

B-Splines, Fig. 13 The

surface approximations
through cross sections
250 B-Splines

The n-dimensional cluster centroids experiments were done using 1996 labeled
achieved during k-means are utilized towards contours from which 450 are taken as right
segmenting the desired contours. Actually, lung and others are ignored. Figures 11 and
good responses are received with 3D vectors 12 show the correctly classified contours and
where second Hu moment and (X, Y) coordi- false positives classified as contours, respec-
nates for contour’s gravity center are used. The tively. It is to be noted that second Hu moment

B-Splines, Fig. 14 The

lung surface reconstructed
through cross-section
fragments

B-Splines, Fig. 15 The

lung’s twisted surface
B-Splines 251

alone with (X, Y) coordinates for contour’s

gravity center produces appreciable results.
The 3D vector substantially reduces the calcu-
lations and yields faster and more responsive
application. B
After the contour segmentation, the algorithm 470

is approximated and least-squares B-Spline fitting

460
for every contour is done. The contour points in
preceding algorithm steps are used towards fitting 450
cubic B-Spline for every segmented contour cross
section. 440
The curve points form the organ contour are in
place and the control points are required to be 430

computed. This can be represented as the follow-

420
ing equation where CV, NB, and N denote the
curve, orthonormal basis, and control points, 410
respectively:
400
CV ¼ NB ∙ N (15)
390
220
The above equation can be solved towards
380 200
N considering least squares such that:
180
370 160
1
N ¼ NBT ∙ NB ∙ NBT ∙ CV (16) 140
230 240 250
The resulting N matrix comprises of control
B-Splines, Fig. 16 The lung surface being rendered con-
point vectors towards every cross section. To sidering cross sections
achieve a smooth surface, all approximated cross
section splines are made compatible. They are control points with third-degree B-Spline. The
required to have similar degree which is 3 here higher order curves look towards greater number
and need to be specified through identical knot of control points and produces curves which are
vector. The common knot vector can be achieved prone with loops and wiggles.
by taking the average of the consequent knot Here there is no need for curve alignment.
vectors considering all the cross sections. Figure 15 represents twisted surface. This does
The control points for all cross-section curves not imply any other problems, except slightly
are taken column by column and a second curves’ unpleasant visual effects. Figure 16 shows the
family is fitted towards approximating these con- lung surface being rendered considering cross
trol points columns. This leads to second B-Spline sections.
curves family. The two control point vectors set
forms control net for tensor surface which can be
plotted. The control net and two knot vectors Conclusion
taken from each curve family completely define
the surface. Figure 13 presents a fragment of right In this entry, we have presented some of the
lung surface which is constructed using 3D points. important geometric properties of B-Spline
Figure 14 presents a detailed view of lung frag- curves. The properties worth mentioning are con-
ment with ribs impression in lung surface. The vex hull property and repeated points’ effects.
appreciable results are obtained when cross- These curves find superior applications in
section contours are approximated through computer-aided design industry. The computation
252 Bug Detection

of B-Splines’ in computational devices is also

highlighted. An industry application on image Build Toolchain
processing with B-Spline curve for reconstructing
the 3D surface for CT image datasets of inner ▶ Plug-in-Based Asset Compiler Architecture
organs further justifies the importance of these
curves.

Building Information
References Modelling of Construction
Beach, R.C.: An Introduction to Curves and Surfaces of
▶ Construction Management Processes in a Dig-
Computer Aided Design. Van Nostrand Reinhold,
New York (1991) ital Built Environment, Modelling
Buss, S.R.: 3D Computer Graphics – A Mathematical
Introduction with OpenGL. Cambridge University
Press, New York (2003)
Chaudhuri, A.: Some Investigations with B-Spline Curves.
Technical Report, TR–8918. Samsung R&D Institute, Building Product Modelling
New Delhi (2018)
Partyka, A.W.: Organ surface reconstruction using
B-Splines and Hu moments. Acta Polytechnica
▶ Construction Management Processes in a Dig-
Hungarica. 10(11), 151–161 (2014) ital Built Environment, Modelling

Bug Detection Byte

▶ Quality Assurance-Artificial Intelligence ▶ Color: Pixels, Bits, and Bytes

C

Calibration Thermal Imaging History of Call of Duty

▶ Smart Calibration Between RGB-D and Ther- As of 2016, the Call of Duty series as a whole has
mal Cameras for ROI Detection and Tracking in sold 250 million copies and five of the games have
Physiological Monitoring made the top 50 best-selling video games of all-
time list (McWhertor 2016). The first title released
in the first-person shooter Call of Duty series was
Call of Duty: Finest Hour. Finest Hour was
released on October 29, 2003. The series started
Call of Duty Franchise, an as a Microsoft exclusive and it ran on the IW game
Analysis engine. Today, one can find Call of Duty on any
console or handheld device. Activision-Blizzard
Dylan Hires2, Sam Romershausen2 and currently owns the series. From 2006 until 2014,
Newton Lee1,2 there was a rotation every year between two dif-
1
Institute for Education, Research, and ferent developers, Infinity Ward and Treyarch. In
Scholarships, Los Angeles, CA, USA that time span, Infinity Ward developed the titles
2
Vincennes University, Vincennes, IN, USA such as The Big Red One, Modern Warfare 1–3,
and Ghosts. Treyarch, on the other hand, devel-
oped World at War, Finest Hour, and Black Ops
Synonyms 1 & 2. In 2014, a new company took on the mantle
and developed the Call of Duty title for that year
Esports; First-person shooter as well as being added to the rotation. Sledgeham-
mer developed the game Advanced Warfare. The
games developed since then have been Treyarch
Definitions with Black Ops 3, Infinity Ward with Infinite
Warfare, and Sledgehammer with World War 2.
First-person shooter (FPS) ¼ a genre of games All titles in the series have an ESRB rating of
where the camera is in a first-person viewpoint. M (ESRB 2021).
Players make use of guns to eliminate targets Call of Duty, being a franchise about war, is
and/or secure other objectives. usually set in a war zone. Call of Duty has been set
Esports ¼ A sports competition with video in the past as well as the future. Call of Duty has
games as the focus rather than traditional also brought some real-life events to the game. In
sports. the most recent game, the developers brought the
© Springer Nature Switzerland AG 2024
N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
254 Campaign

invasion of Normandy into the game with a new Cross-References

game mode. Call of Duty is adapting to the current
game market by creating their own battle royale ▶ First-Person Shooter Games, a Brief History
mode called “Blackout.” Call of Duty is also
known for its Zombies game mode, where players
are tasked with surviving against increasingly dif- References
ficult waves of zombies with up to three other
players. Delaney, M.: The call of duty movie is delayed indefinitely,
says the director https://www.gamesradar.com/the-call-
Some similar FPS games to Call of Duty are
of-duty-movie-is-delayed-indefinitely-says-the-
Halo, Battlefield, CSGO, and Rainbow Six Siege. director/ (2020)
These games are played by people of all ages, but ESRB: Call of duty ®: Modern warfare ®. https://www.esrb.
according to surveys 31% of Call of Duty players org/ratings/36491/Call+of+Duty%C2%AE%3A
+Modern+Warfare%C2%AE/ (2021)
are between the ages of 25 and 34 (Knezovic
Knezovic, A.: Call of duty analysis: How it shot to the top
2021). and stayed there. https://www.blog.udonis.co/
Call of Duty also has its own line of comic mobile-marketing/mobile-games/call-of-duty-analy
adaptations. The “Official Comic of Call of sis (2021)
Lionade Games: Who won call of duty championship
Duty ®: Black Ops 4” is a ten-issue series intro-
2022?. https://lionadegames.com/who-won-call-of-
ducing the iconic Specialists of the Black Ops duty-championship-2022/ (2021)
Universe. The game also inspires a number of McWhertor, M.: Call of duty surpasses 250 million games
short films. In 2015, Activision announced the sold worldwide. https://www.polygon.com/2016/1/14/
10773162/call-of-duty-surpasses-250-million-games-
creation of a cinematic universe based on the sold-worldwide. (2016)
Call of Duty franchise with a first film projected
to be released in 2018 or 2019, and directed by
Stefano Sollima. However, in an interview in
2020, Stefano Sollima stated that films were
“no longer a priority for Activision” (Delaney
Campaign
2020).
More importantly, however, Call of Duty burst
▶ Integrating Virtual Reality and Augmented
into the esports scene in 2006 following the
Reality into Advertising Campaigns: History,
release of Modern Warfare. Activision-Blizzard
Technology, and Future Trends
hosts their own World Championship where the
top teams compete each year. The Call of Duty
League was launched in 2020, featuring the
world’s greatest professional players. As of
2021, the lineup of COD 2022 teams includes
Atlanta FaZe, Florida Mutineers, London Royal Canal Surface
Ravens, LA Guerrillas, LA Thieves, Minnesota
Rokkr, New York Subliners, OpTic Texas, Paris ▶ Theory of Minkowski-Lorentz Spaces
Legion, Seattle Surge, and Toronto Ultra. The
original 12 teams became 11 teams due to the
merger between Dallas Empire and OpTic Chi-
cago to form OpTic Texas. The most famous COD
professional player is Chris “Simp” Lehr who has CAPTCHA
earned over $546,000 in Call of Duty tournaments
despite being only 19 years old. He plays for the ▶ Audio and Facial Recognition CAPTCHAs for
Atlanta FaZe team (Lionade Games 2021). Visually Impaired Users
Cellular Automata Methods 255

Caption Generation Cellular Automata Methods

▶ Automated Image Captioning for the Visually Sicilia Ferreira Judice

Impaired Faculty of Technical Education State of Rio de
Janeiro, FAETERJ Petropolis, Petropolis, Brazil
C

Synonyms
Casual Game
Numerical methods; Physics simulation
▶ Gardenscapes and Homescapes, Casual Mobile
Games
Definitions

Cellular Automata
Causal Game Mathematical models based on simple and local
rules capable of generating complex behaviors.
▶ Animal Crossing: A Causal Game

Introduction

In his articles collection about Cellular Automata,

CAVE
specifically in the article entitled Cellular Autom-
ata from 1983, Wolfram (1994) introduces the
▶ Interactive Virtual Reality Navigation Using
subject with a reflection on the basic laws of
Cave Automatic Virtual Environment Technology
physics relevant to everyday phenomena. Many
of these laws involve complex systems, composed
of numerous components, each obeying simple
rules, so that the complex behavior of the studied
Cave Automatic Virtual phenomenon can be obtained. To discover and
Environment analyze the mathematical basis of complex sys-
tems, it is necessary to identify simplified mathe-
▶ Interactive Virtual Reality Navigation Using matical systems that are capable of capturing the
Cave Automatic Virtual Environment Technology essence of the process, and the Cellular Automata
are strong candidates in this matter.
Cellular Automata are mathematical models
based on simple and local rules capable of gener-
Cellular Automata ating complex behaviors. They were originally
introduced by John von Neumann, under the
▶ Lattice Boltzmann Method for Diffusion-Reac- name of cellular spaces, as an idealization of bio-
tion Problems logical systems, with the particular aim of model-
▶ Lattice Boltzmann Method for Fluid Simula- ing systems capable of self-reproduction (Sarkar
tion 2000).
▶ Lattice Gas Cellular Automata for Fluid In 1940, John von Neumann was involved in
Simulation the planning of the first digital computers. His
256 Cellular Automata Methods

goal was to obtain complex behaviors through Many papers had their interests focused on
simple rules for spatial interactions and temporal applications of automata to computing and biol-
evolution. It would be a machine analogous to ogy. However, in the last decades, there is also
the human brain whose processing and memory considerable interest in the field of physics
units were not separated from each other, mas- (Daniel 1994). In particular, models based on
sively parallel and capable of self-reproduction cellular automata for fluids have been developed,
(Chopard and Droz 1998). such as the Lattice Gas Cellular Automata
Following suggestions by S. Ulam, John (LGCA) (Frisch et al. 1986), electromagnetic
von Neumann proposed a discrete universe models, such as Ising Spin Models, and models
based on a two-dimensional lattice, whose cells for diffusion phenomena (Weimar et al. 1991).
are interconnected locally to one another. Each According to Wolfram (1994), physical systems
cell is characterized by an internal state, that contain discrete elements and whose interac-
representing a finite state machine. Through a tions occur locally can be modeled through the use
local rule of evolution, each cell updates its state of Cellular Automata. Although they are defined by
in function of its own and the states of some simple local rules, automata can display complex
neighboring cells. All cells of the lattice evolve dynamic behaviors on a wide scale. Such rules can
according to the same local rule, which makes be seen as a simplification of the microscopic
the system homogeneous, like many physical dynamics, which reproduce the expected macro-
and biological systems. These cellular universes scopic behavior. For example, on a microscopic
proposed by John von Neumann are now known scale each cell can represent a particle of the system.
as Cellular Automata (Chopard and Droz 1998; Several authors have been directing their lines
Kari 2005). of research for the understanding and application of
In the cellular automata proposed by Neumann Cellular Automatics. According to (Chopard and
(1966), a neighborhood formed by the four Droz 1998), some of these lines involve the use of
nearest neighbors (north, south, east, west) and automaton models for the simulation of physical
29 possible states for each cell was defined for and biological systems. The success obtained in
each two-dimensional cell of the lattice. The the- these applications lies in the fact that Cellular
ory of Cellular Automata was consolidated with Automata possess several fundamental properties
the work of Burks (1970) and underwent a con- of the physical world: they are massively parallel,
siderable simplification in Codd’s work (1968). homogeneous, and all interactions occur locally.
Wolfram’s researches (1994) became the pio- Other physical properties such as reversibility and
neer author in the work with Cellular Automata as conservation laws can be programmed through an
mathematical models for the study of the phenom- appropriate choice of local evolution rule.
enon of self- organization. He suggested the use of
one-dimensional automata, each cell having as its
neighbor the left and right cells and a set of two Formal Definition
possible states.
Martin used polynomial algebra tools to In a formal way, a Cellular Automata is a
deduce the characterization of uniform cellular quadruple:
automata (identical rules applied to each cell of
the automaton) with a periodic boundary (Martin A ¼ ðL; S; N; f Þ,
et al. 1984). A new era of research began in 1992
with the work on analytical characterization of the where L is a set of indices or cells; S is the finite set
behavior of automata based on matrix algebra of states, that is, of the possible values that each
tools. The technique proposed in (Das et al. cell can assume; N : L ! Lk is the mapping that
1992) is able to characterize hybrid cellular defines the neighborhood of each cell i as a col-
automata (different rules applied to different lection of k cells; and f : Sk ! S is the evolution
cells) with periodic or zero frontiers. function or the update rule of the automaton
Cellular Automata Methods 257

(Wolfram 1994). The f rule is responsible for the Figure 1 shows a von Neumann neighborhood
overall dynamics of the Cellular Automaton and is for two-dimensional automata.
applied at every instant of time in all cells. The In addition to the four geographic neighbors
collection of cell states at any time step is called cited in the von Neumann neighborhood (north,
the configuration or global state of the automaton south, east, west), Moore’s neighborhood also
(Chopard and Droz 1998). contains the four diagonal neighbors (north-east,
north-west, south-east, south-west) (Chopard and C
Neighborhoods Droz 1998). Such neighborhood of radius r for
An f update rule for an automaton is a local rule, cell x0 is defined by (Wolfram 1994) as:
since it only depends on the state of neighboring
cells. At first there is no restriction on the size of N u x0 ¼ x :jxi  x0i j r, i ¼ 1, . . . , d , ð3Þ
a neighborhood. In practice, however, it is com-
mon to use only adjacent cells (Wolfram 1994). where d is the space dimension and xi is the i-th
For Cellular Automata, two neighborhoods are space component.
considered: a von Neumann Neighborhood and For two-dimensional automatas (d ¼ 2), the
Moore’s Neighborhood (Chopard and Droz definition (3) is given by:
1998).
The simplest von Neumann neighborhood con- N u x01 , x02
sists of a central cell, which is exactly the cell to be
¼ ðx1 , x2 Þ :jx1  x01 j  r, jx2  x02 j  r :
updated by rule f, and its four geographic neigh-
bors north, south, east, and west (Chopard ð4Þ
and Droz 1998). Generalizing, (Wolfram 1994)
defines the von Neumann neighborhood of radius Figure 2 shows Moore’s neighborhood for
r of cell x0 as: two-dimensional automatas.

Boundary Condition
N u x0 ¼ x : x  x0  r , ð1Þ
In practice it becomes impracticable to work with
an infinite lattice. Once the boundaries of the
where k  k is the norm of the sum.
lattice are defined, it is clear that the cells present
For two-dimensional Cellular Automata, the
at the border will not have the same neighborhood
definition (1) is given by:
as the inner cells. It is therefore necessary to
identify which cells belong to the border and
N u ðx0 , x0 Þ
ð2Þ apply different rules to them. Following this meth-
¼ ðx1 , x2 Þ :jx1  x01 j þ jx2  x02 j  r : odology, it is possible to define several types of

Cellular Automata
Methods, Fig. 1 A von
Neumann neighborhood for
two-dimensional automata:
(a) r ¼ 1, (b) r ¼ 2
258 Cellular Automata Methods

Cellular Automata
Methods, Fig. 2 Moore’s
neighborhood for two-
dimensional automata:
(a) r ¼ 1, (b) r ¼ 2

Cellular Automata
Methods, Fig. 3 Periodic
boundary condition for
unidimensional automata

borders, each with a different behavior (Wolfram can use horizontal periodic boundary and the ver-
1994). However, instead of defining distinct rules tical reflective one (Chopard and Droz 1998).
for frontier cells, another possibility is to extend
the neighborhood to these cells. A widely used
solution is the periodic boundary condition, which Classification
assumes that the lattice is embedded in a torus-
type topology. Such a condition is obtained by A unidimensional Cellular Automata with the set
extending the lattice periodically, as shown in of possible states defined by S ¼ {0, 1} and ray
Fig. 3. In the case of a two-dimensional automata, neighborhood r ¼ 1 is called elementary (Kari
the periodic boundary condition assumes that the 2005). In this configuration, there are 23 ¼ 8 pos-
right and left sides are connected, as well as the sible combinations in the vicinity of a given cell,
upper and lower part of the lattice (Chopard and where each of these combinations can be mapped
Droz 1998). in 0 and 1. Thus, there are 28 ¼ 256 different
Other types of boundary conditions add a set of Elementary Cellular Automata (Kari 2005).
cells along the boundary, as shown in Fig. 4a–c These elementary rules were studied and classi-
(Chopard and Droz 1998). The fixed boundary fied by Wolfram (1994), who introduced a meth-
condition defines predefined values for these odology to name them: each of these elementary
new cells. The adiabatic frontier condition dou- rules is specified by a sequence of eight bits,
bles the value of the cells that are at the border for
the additional cells. The reflective boundary con- f ð111Þf ð110Þf ð101Þf ð100Þf ð011Þf ð010Þ
dition copies the value of the cell next to the f ð001Þf ð000Þ,
border cell to the additional cell.
Choosing which type of boundary condition to where f is the automaton update rule. The bits
use will depend on the nature of the problem being sequence is the binary expansion of an integer in
modeled. In some cases, a combination of more the range of 0. . .255, called Wolfram Nomencla-
than one type of condition can be used, for exam- ture of the Cellular Automata (Wolfram 1994;
ple, in the simulation of a long channel, where one Kari 2005).
Cellular Automata Methods 259

Cellular Automata
Methods,
Fig. 4 Boundaries
condition: (a) Fixed,
(b) Adiabatic, and
(c) Reflective

Cellular Automata
Methods, Fig. 5 Wolfram
Nomenclature

The numerical value that will identify the rule • Class 1: The evolution of the automaton leads
is obtained by combining the values resulting to a homogeneous state in which, for example,
from applying the update rule f to each of the all cells have 0 value (Fig. 7a);
eight possible neighborhoods, thus forming a • Class 2: The evolution of the automaton leads
binary number. The representation of this number to a set of stable or periodic structures, which
in the decimal base gives the numerical value for are simple and separate (Fig. 7b);
the rule (Kari 2005). For example, Fig. 5 shows • Class 3: The evolution of the automaton leads
the process of identifying the cellular automaton to a chaotic pattern (Fig. 7c);
with r ¼ 1, known as Rule 90. • Class 4: The evolution of the automaton leads
Examples of dynamics of one-dimensional to complex structures (Fig. 7d).
Cellular Automata are often represented in a spa-
tiotemporal diagram. Horizontal lines are consec- The existence of only four classes implies a
utive configurations, and the top line is the initial considerable universality in the behavior of cellu-
configuration. For example, Fig. 6 shows dia- lar automata. Many characteristics of a given
grams of Rule 110 on two different scales, where automaton depend only on the class in which it
black represents the state 1 and white the state belongs and not on the details of its evolution.
0 (Kari 2005).
Wolfram (1994) did numerous experiments
with rules of Elemental Cellular Automata using Examples of Cellular Automata
random initial configurations. Based on the types
of spatial-temporal diagrams observed, he classi- Using the traditional nomenclature (Sarkar 2000),
fied the rules into four categories (Chopard and at each time step, the collection of cell states
Droz 1998; Kari 2005), namely: is called the configuration or global state of
260 Cellular Automata Methods

Cellular Automata
Methods, Fig. 6 Space-
time diagrams of Rule 110

Cellular Automata
Methods,
Fig. 7 Spatiotemporal
diagrams: some examples
of Wolfram’s classification
Cellular Automata Methods 261

Cellular Automata
Methods,
Fig. 8 Evolution of one-
dimensional Cellular
Automaton

Cellular Automata
Methods, Fig. 9 Cellular
Autamata Rule 90.
Evolution after 15 time
steps

the automaton. In the initial step t ¼ 0, we have the effected. Below we can see a spatiotemporal dia-
initial configuration, and as time increases, the gram after 15 steps of evolution.
local rule is applied on all cells, as (generically) It is interesting to note that, starting from a
represented in Fig. 8 for one-dimensional case. simple initial configuration, where only one initial
Similar definitions are made for two-dimensional cell has state 1 and the others have state 0, after
cases. 15 steps already it is possible to observe the for-
mation of a nontrivial pattern.
Rule 90
The Cellular Automata Rule 90 consists of an one- Game of Life
dimensional vector, whose set of possible states is In 1970, the mathematician John Conway pro-
formed by S ¼ {0, 1} and the update rule f is the posed the famous Game of Life (Gardner 1970).
binary sum of neighboring cell states (Wolfram His motivation was to find a 2D automaton that
1994). Thus, given a cell xi at instant t, its state ati would generate complex behaviors (Chopard and
is defined by: Droz 1998). The Game of Life consists of a two-
dimensional lattice, where each cell can contain
ati ¼ ai1
t1 t1
aiþ1 , ð5Þ the 0 or 1 values. For this set of states, Conway
classified the cell in vivo, if it contains the value 1,
where is the binary sum. or dead, to the value 0. This automaton uses the
At the top of Fig. 9 we have the eight possible Moore’s Neighborhood (Kari 2005).
configurations for the neighborhood of the cell to The update function f seeks to simulate an
be updated and the result of the binary sum artificial life, based on the following rules:
262 Cellular Automata Methods

Cellular Automata
Methods, Fig. 10 Game
of life: (a) initial step,
(b) step 1

Cellular Automata
Methods, Fig. 11 Gliders
at Game of Life: (a) initial
step, (b) step 1, (c) step 2,
(d) step 3, (e) step 4,
(f) step 5

• A cell whose state is 0 and which has exactly automaton, it is possible to notice the emer-
three living cells in its neighborhood passes to gence of complex structures, such as the
the state 1. so-called gliders, shown in Fig. 11. Gliders
• A cell whose state is 1 and which contains less are specific cell configurations which have
than two living cells in its neighborhood passes the property of moving around in space. We
to the state 0. can observe in Fig. 11 that the configurations
• A cell whose state is 1 and which has in its (e) and (f) are translations of configurations
vicinity more than three living cells passes to (a) and (b), respectively.
the state 0.
Figure 10 shows an example of Game of HPP Model
Life in a 20  20 lattice. The HPP model is part of a group of specific
Cellular Automata, called Lattice Gas. It was
The Game of Life has a very rich behavior. developed by Jean Hardy, Olivier de Pazzis, and
During the evolution process of the Yves Pomeau in the 1970s. His proposal is to
Cellular Automata Methods 263

model fluids via cellular automata, using simple cell. Therefore, in order for the molecular
and local rules that mimic a particle dynamics. dynamics of the HPP to be compatible with the
The essential characteristics of the actual micro- automata’s dynamics, there must be a restriction
scopic interactions that are taken into account are of the quantity of particles present simulta-
the laws of conservation of the linear momentum neously in a given cell (Chopard and Droz
and conservation of the number of particles 1998). Such a constraint is called Principle of
(Chopard and Droz 1998). Exclusion which guarantees that, given a lattice C
The HPP is defined in a two-dimensional lat- cell, there can only be at most one particle inci-
tice, and the basic idea of its dynamics is to work dent in a given direction of motion, at a given
with particles that move along the directions of the instant of time. Thus, four bits of information for
lattice, following appropriate rules. The set of each cell are sufficient to describe the system
possible states is composed of S ¼ {0, 1} and during its evolution.
represents the absence or presence of particles. The HPP update rule consists of two steps:
Figure 12 shows an example of a configuration collision and propagation (Chopard and Droz
of the HPP particles. 1998). The collision phase specifies the interac-
By definition, Cellular Automata work with a tion between particles entering the same lattice
finite number of bits to represent the state of each node. It is at this stage that the particles are
“rearranged” in different directions, in order to
ensure that the exclusion principle is satisfied, as
shown in Fig. 13b–c. In propagation phase, each
particle moves to the neighboring cell, in the
direction corresponding to its direction of motion,
as shown in Fig. 13a.
The purpose of the HPP rule is to repro-
duce certain aspects of actual interactions
between particles, such as the conservation
of momentum and the amount of particles.
Looking for Fig. 13b–c we can observe that
during the collision process a pair of particles
with null moment along a given direction is
transformed into another pair of particles, also
with the null moment, moving on the
perpendicular axis.
Another important feature in the HPP model is
invariance under time reversal (reversibility). At
Cellular Automata Methods, Fig. 12 Particles config- any given moment, if the direction of movement
uration example in HPP model of all particles is reversed, the system will recover

Cellular Automata
Methods, Fig. 13 HPP
Rules: (a) Propagation,
(b, c) Collision
264 Centralized Architectures

its entire history. This property becomes a limiter

when the intention is to use the model to describe Centralized Architectures
a real fluid, since in this case there is the presence
of viscosity. Other limitations are related to alge- ▶ Client/Server Gaming Architectures
braic invariants of the adopted lattice (Wolfram
1994).

Cerebral Palsy
Cross-References
▶ Computer Games for People with Disability
▶ Fluid Simulation
▶ Lattice Gas Cellular Automata for Fluid
Simulation
CG
References ▶ Planetary Generation in Games
▶ Postproduction in Game Cinematics
Burks, A.W.: Essays on Cellular Automata. University of
Illinois Press, Urbana (1970). A collection of several
papers on cellular automata.
Chopard, B., Droz, M.: Cellular Automata Modeling of
Physical Systems. Cambridge University Press, CG, Computer Graphics
Cambridge (1998)
Codd, E.F.: Cellular Automata. Academic, New York
(1968) ▶ The New Age of Procedural Texturing
Rothman, D.H., Zaleski, S.: Lattice-gas models of phase
separation: interface, phase transition and multiphase
flows. Rev. Mod. Phys. 66, 1417–1479 (1994)
Das, A.K., Sanyal, A., Palchaudhuri, P.: On characteriza-
tion of cellular automata with matrix algebra. Inf. Sci. Challenge-Based Learning in a
61(3), 251–277 (1992) Serious Global Game
Frisch, U., Hasslacher, B., Pomeau, Y.: Lattice-gas autom-
ata for the navier-stokes equation. Phys. Rev. 56,
1505–1508 (1986)
David Gibson1, Leah Irving2 and Katy Scott1
1
Gardner, M.: The fantastic combinations of john Curtin University, Perth, WA, Australia
2
conway’s new solitaire game life. Sci. Am. 223, Charles Sturt University, Wagga Wagga, NSW,
120–123 (1970) Australia
Kari, J.: Theory of cellular automata: a survey. Theor.
Comput. Sci. 334(1–3), 3–33 (2005)
Martin, O., Odlyzko, A., Wolfram, S.: Algebraic properties
of cellular automata. Commun. Math. Phys. 93, Synonyms
219–258 (1984)
Sarkar, P.: A brief history of cellular automata. Technical
report, Indian Statistical Institute (2000)
Collaborative problem solving; Contextual learn-
von Neumann, J.: Theory of Self-Reproducing Automata. ing and teaching; Learning challenges; Open-
University of Illinois Press, Urbana (1966) ended problem-based learning; Project-based
Weimar, J.R., Watson, L.T., Tyson, J.J.: Cellular automaton learning
models for reaction diffusion equations. In: Sixth Dis-
tributed Memory Computing Conference, pp. 431–434.
IEEE Computer Society (1991)
Wolfram, S.: Cellular Automata and Complexity: Col- Definition
lected Papers, 1st edn. Addison-Wesley Pub.
Co. Portland, OR, USA, USA http://ieeexplore.ieee.
org/document/633207/?reload=true http://www.
Challenge-based learning offers a call to act-
stephenwolfram.com/publications/books/ca-reprint/ ion that inherently requires learners to make
(1994) something happen. In a serious game context
Challenge-Based Learning in a Serious Global Game 265

learners often work in teams in a digital game- challenges engage students’ curiosity and desire
based platform to research their topic, brainstorm to learn by making central the solving of open-
strategies and solutions that are both credible and ended problems as a member of a self-organizing
realistic in light of time and resources, and then and self-directing international team (Harris and
develop and execute a solution that addresses the Nolte 2007). When delivered as a mobile learning
challenge in ways both they themselves and others experience, challenges can integrate twenty-first-
can see and measure. Teams often compete with century tools, require collaboration, and assist C
each other for high scoring solutions, recognition, students in managing their time and work sched-
and rewards. For example, a business might offer ules, while effectively scaling to large numbers of
an “X-Prize” for a solution needed for driverless students. Set in the environment of a friendly
cars. Challenge-based learning can thus be seen as serious game competition where people experi-
a way to incentivize crowd-sourced ideation and ence elements such as automated feedback,
solutions. points, leader boards, badges, and leveling up for
rewards, challenge-based learning increases moti-
vation toward high performance (Gibson and
Introduction Grasso 2007). Research on challenge-based learn-
ing is beginning to show impacts such as
The term challenge-based learning arose in the increased engagement, increased time working
USA early in the twenty-first century with the on tasks, creative application of technology, and
support of innovative technology groups such as increased satisfaction with learning (Johnson et al.
Apple Education, the New Media Consortium, forthcoming, 2018; Roselli and Brophy 2006).
The Society for Information Technology and Similar to problem-based and project-based
Teacher Education, and the US Department of learning, and borrowing liberally from those
Education Office of Educational Technology. well-established approaches (see Gibson et al.
Unique applications supporting challenge- 2011) and the Buck Institute for Education
based learning in higher education are now aris- (www.bie.org), the additional structure of global
ing, supported by a cloud-based mobile technol- relevance, international collaboration, and team-
ogy platform that can be used for bridging based competition leads to a unique objective,
informal to formal learning, recruiting students expressed well in a recent report by the New
into university, reaching larger numbers of peo- Media Consortium (Johnson et al. 2009).
ple with game-based approaches, envisioning
student engagement in work-integrated learning,
and assisting students to acquire evidence of Design and Implementation Roles
attainment of graduate capabilities such as lead-
ership, critical thinking, creativity, communica- The design method for the future of challenge-
tion skills, and experience in international team based digital learning experiences is a team-based
collaboration (Gibson et al. forthcoming, 2018a, effort of people knowledgeable in subject matter,
b). dramatic narrative, mechanics of game-like inter-
actions and rewards, digital-media artists and
communicators, and computational science tools
State of the Art Work for algorithms and visualizations (Gibson et al.
2007). The mission of such interdisciplinary
Challenge-based learning is a teaching model teams when creating challenge-based learning
that incorporates aspects of collaborative experiences is to create a symbolic space for
problem-based learning, project-based learning, transmedia narrative (Passalacqua and Pianzola
and contextual teaching and learning while focus- 2011) to be introduced as well as to evolve
ing on current real-world problems (Johnson et al. through the participatory culture (Jenkins et al.
2009). In particular, online global learning 2006) shared by those who take up the challenge.
266 Challenge-Based Learning in a Serious Global Game

Challenge-Based Learning in a Serious Global Game, Fig. 1 Curtin Challenge platform as a solution to personal-
izing learning at scale

Instructors have a special role in constructing and learning in response to the myriad changes
a problem space with key ideas, essential ques- taking place in higher education today
tions, resources, and evaluation criteria. In the (Grummon 2010).
challenge-based learning framework, instructors
are designers of the digital-learning experience
who put in most of their time in up front and then Show Case
take a backseat during the implementation while
individuals and teams are learning, working, At Curtin University, in Perth Australia, a “Chal-
communicating, creating, and submitting arti- lenge platform” online toolkit has been devel-
facts. Subject matter experts (SMEs) in a disci- oped to assist with authoring and delivering
pline work with a digital-media team and game- challenge-based individual and team learning
creation team to engineer the learning experi- at scale (Fig. 1). The platform can be compared
ence. Most of the input from subject-matter- to a game-engine marketplace with a common
expert authors is gathered during the design API for learning outcomes across the various
phase and is embedded into the digital experi- learning experiences. Individual challenges
ence through public scoring rubrics, artifact include learning about oneself as a leader or
descriptions for final submission, and scaffold- exploring a career and team challenges available
ing activities, which the team members can in “Balance of the Planet” including solving one
choose to experience or ignore. A new structure or more of the UN Sustainable Development
of teaching, created via collaborations among Goals (Fig. 2).
SMEs working with learning-experience The platform roadmap includes features that
designers and technical teams, is arising as an support individual- and team-based learning in a
innovation to create new structures of teaching serious game context:
Challenge-Based Learning in a Serious Global Game 267

Challenge-Based Learning in a Serious Global Game, Fig. 2 Individual and team-based challenges provide a
serious game context for learning

• Self-organizing teams References

• Self-determined paths of action
• Transparent milestones for seeing progress Gibson, D., Grasso, S.: The global challenge: save the
world on your way to college. Learn. Lead. Technol.
• Self-scoring and peer scoring of artifacts
5191, 12–16 (2007)
• Expert scoring for awards and recognition Gibson, D., Aldrich, C., Prensky, M.: In: Gibson, D.,
• Openness to external mentors and advisors Aldrich, C., Prensky, M. (eds.) Games and Simulations
helping solve problems in Online Learning: Research and Development Frame-
works. Information Science Publishing, Hershey
• Automated feedback on progress
(2007)
• 24-7 access Gibson, D., Knezek, G., Mergendoller, J., Garcia, P., Red-
• Support tools for chatting, coproduction, and mond, P., Spector, J.M., Tillman, D.: Performance
automated messaging assessment of 21st century teaching and learning:
insights into the future. In: Koehler, M.,
• Administrative dashboard for research data Mishra, P. (eds.) Proceedings of Society for Informa-
and monitoring tion Technology & Teacher Education International
Conference 2011, pp. 1839–1843. AACE, Cheaspeake
(2011)
Gibson, D., Irving, L., Scott, K.: Technology-enabled
Cross-References challenge-based learning in a global context. In: Col-
laborative Learning in a Global World, p. 450 (forth-
▶ Game-Based Learning (GBL) coming, 2018a)
268 Challenges Facing the Arab Animation Cinema

Gibson, D., Irving, L., Seifert, T.: Assessing personal other global cultures. They seek to build a bridge
learning in online collaborative problem solving. between the Arab world and the West through
In: Collaborative Learning in a Global World, p. 450
(forthcoming, 2018b) animated films which have been adapted from
Grummon, P.T.H.: Trends in higher education. Plan. High. Arab and Islamic sources, but speak to the univer-
Educ. 12, 122 (2010). https://doi.org/10.2307/1974977 sal human condition. The relationship between
Harris, D., Nolte, P.: Global Challenge Award: External Islam and the West, though, remains very compli-
Evaluation Year 1 2006–2007. Vermont Institutes Eval-
uation Center, Montpelier (2007) cated; the West looks at these projects and already
Jenkins, H., Purushotma, R., Clinton, K., Weigel, M., has a perspective about them as religious and
Robison, A.: Confronting the challenges of participa- ideological propaganda, especially after 9/11,
tory culture: media education for the 21st century. In: 2001. Thus, the majority of these Arabic animated
New Media Literacies Project. MIT, Cambridge, MA
(2006). Retrieved from http://mitpress.mit.edu/sites/ films are rejected by the West because of concerns
default/files/titles/free_download/9780262513623_ that these films represent the unwelcome princi-
Confronting_the_Challenges.pdf ples of foreign cultures. Inherently, there is an
Johnson, L., Smith, R., Smythe, J., Varon, R.: Challenge- Islamophobia about Islamic cultural products as
Based Learning: An Approach for our Time. The New
Media Consortium, Austin (2009). Retrieved from soon as they come to the West; there is suspicion
http://search.ebscohost.com/login.aspx?direct¼true& of them and extensive interrogation of them. Iron-
db¼eric&AN¼ED505102&site¼ehost-live ically, when Western artifacts are exported to Arab
Johnson, L., Adams, S., Apple: Challenge based learning: countries, though almost inherently at odds with
the report from the implementation project. Media
(forthcoming, 2018) Muslim ideology and Muslim politics, they some-
Passalacqua, F., Pianzola, F.: Defining transmedia narra- times find distribution and audiences. The conse-
tive: problems and questions. Dialogue with Mary- quences of this relationship between Arab
Laure Ryan. Enthymema. (2011). https://doi.org/10. countries and the West is not only ideological,
13130/2037-2426/1188
Roselli, R., Brophy, S.: Effectiveness of challenge-based however, and also concerned with the fact that
instruction in biomechanics. J. Eng. Educ. 95(4), Arab filmmakers and producers face economic
311 (2006). Retrieved from http://findarticles.com/p/ challenges, and a number of Arab animation stu-
articles/mi_qa3886/is_200610/ai_n16810356 dios went out of business or stopped making more
feature animated films due to the difficulties of
reaching international marketplaces. Thus, the
focus of contemporary Arab animation is mostly
Challenges Facing the Arab low-budget projects distributed through YouTube
Animation Cinema and social media, which became the main plat-
form for Arab animation artists to distribute their
Tariq Alrimawi political works during the “Arab Spring” in Tuni-
Graphic Design Department, University of Petra, sia, Egypt, Libya, Yemen, Syria, and elsewhere in
Amman, Jordan the Middle East since 2011.

Synonyms Introduction

Arab animation; Arab cinema; Arab filmmakers; After 9/11, Arab animation producers struggle to
Muslim filmmakers; Political communication; screen their films at cinemas in Europe and the
Visual culture USA. The irrational fear of Arabs and the Islamic
religion [has] increased in the West, and Muslims
have become targets of increased hostility, creat-
Definition ing the now so-called Islamophobia. (Kalin 2011).
The first use in print of the term Islamophobia was
Arab filmmakers attempt to export their animated in the report of the Commission on British Mus-
films to an international market and try to speak to lims and Islamophobia in 1997 (Iqbal 2010). This
Challenges Facing the Arab Animation Cinema 269

commission was established in 1996 by the Run-

nymede Trust, an independent research and social
policy agency. The report was called
Islamophobia: A Challenge for Us All and
describes the main features of Islamophobia and
the challenge it poses to Muslim communities. It
covers central topics about Muslim communities C
and concerns in Britain, media coverage, vio-
lence, and building bridges by intercommunity
projects and dialogue. The report also contains
many subtopics separately from the main text,
such as diversity and difference within Islam,
perception of Islam as a threat, opposition to
immigration, and other topics. Moreover, the
report shows statistical tables of Muslim issues
such as residential patterns, employment and
unemployment, population in some cities in Brit-
ain, and incidents seen as racially motivated. The
commission distributed more than 3,500 copies to
government departments, organizations, social
communities, universities, and the media.
The report defined the term Islamophobia as
“the practical consequences to such hostility in
unfair discrimination against Muslims individual Challenges Facing the Arab Animation Cinema,
and communities, and to the exclusion of Muslims Fig. 1 Muhammad: The Last Prophet (2002) (Directed
by Richard Rich)
from mainstream political and social affairs”
(Runneymede Trust 1997). Islamophobia has
affected Muslim film producers’ capacity to Muhammad and the journey with his followers
show their Arab/Islamic animated films to a West- from Mecca to Madina, set around 1400 years
ern audience. For example, a Saudi film produc- ago during the early years of Islam (Fig. 1).
tion company called Badr International invested There were two versions released of the feature
around 12 million US dollars, which is the highest and the short films, one in the Arabic language
budget for any Arabic animated film so far, in their and one dubbed in English hoping to gain the
first feature animated film, Muhammad the Last attention of non–Arabic speaking audience. Badr
Prophet (2002) (Jammal 2012). These movies made an agreement with many companies and
were made by a team of artists from Hollywood cinemas to distribute and screen the film Muham-
who combined traditional hand drawing with mad the Last Prophet in the USA. However, the
computer graphics and effects. The director who film’s production finished at around the same time
had the unique experience and challenge of mak- of 9/11 in 2001. The consequence was that most of
ing this movie was Richard Rich, who worked for the agreements were cancelled by US cinemas and
Disney for many years. The soundtrack was com- distributors due to Islamophobia. Badr held the
posed by Emmy-award winning composer Wil- film for 2 years without screening it in cinemas.
liam Kidd. In consequence, the movie was They did not want to sell the film’s DVD to the
described as being an “(old-fashion) Western- market before the theatrical release. Later, a new
style entertainment about a distinctly Arabic distribution company based in the USA
non-Western subject” (Stevens 2004). However, called FineMedia arranged a theatrical release in
this movie was the first feature-length animated 37 US cities with Eid al-Fitr in 2004. The venture
film that focused on the biography of the prophet was not successful and revenues were very small.
270 Challenges Facing the Arab Animation Cinema

Therefore, Badr International stopped making any From the inspiration of the Arabian Nights stories
more animated films and went out of business in Western film producers have developed animated
animation field. films such as The Adventures of Princes Achmed
(1925), Aladdin (1992), and Azur and Asmar
(2006). Arabs, however, have not used their own
Limited Resources and Release past Arabian stories to reach either the domestic
and international animation marketplaces.
The bibliography related to animation in the Arab Bendazzi recalls:
world is very limited, and it was hard to find Arab animators should participate to the great inter-
published materials related directly to the subject; national festivals; watch the masterpieces of ten,
only two published references relating to Arab twenty, seventy years ago, and read translated
animation were found: the first one being Car- books. They first must be great animators with a
distinctive style, and only then adapt any text from
toons: One Hundred Years of Cinema Animation any literature. (Bendazzi 2012)
by Giannalberto Bendazzi (1994) and the second,
the booklet Animation in the Arab World, The Arab animation industry needs people
A glance on the Arabian animated films since with strong skills in animation techniques and
1936 by Mohamed Ghazala (2011). process such as character design, animation,
Bendazzi’s book covers the history of ani- editing, lighting, compositing, sound, music, and
mated films assessing over 3,000 films in more marketing, and then start thinking about making
than 70 countries worldwide. Nevertheless, the successful animation feature films to screen to the
book covers only 3 of 22 Arab countries and Arab audience and then export these films to the
contains only small animated productions from international audience. However, one of most
Egypt, Tunisia, and Algeria. Most of those Arab important parts of any successful film in the con-
animation productions were TV series and TV temporary era is the story; the film would be good
commercials, and a few made as short films. An as soon as the story is good. Also, quality could
electronic communication was arranged with come in different method and ways; it does not
Bendazzi (2012) to ask what the reasons were have to imitate Disney and Pixar styles. The Arab
for having such a small selection of Arab anima- filmmakers should think of using contemporary
tion in his book (Bendazzi 2012). Bendazzi’s first tools and creating fresh and unique styles such
sentence was “I think you will be disappointed by as the Iranian animated films Persepolis (2007)
my answers”; this sentence immediately gives a and the documentary The Green Wave (2011). Thus,
negative impression about Arab animation cin- the Arab filmmaker should focus more on making
ema and the challenges it faces. universal stories with different styles in order to
Bendazzi points out that when he was writing show them to audiences all around the world.
the book, from 1983 to 1988, it was hard to find In March 2012, an invitation has been received
Arab animated films due to the lack of connections from the Cairo International Film Festival for
at that time such as internet, e-mails, and social Children (CIFFC) to present my short animated
network websites. In addition, Bendazzi faced lan- film Missing. The CIFFC, organized by the Min-
guage difficulties communicating with Arab film istry of Culture in Egypt, is one of the biggest
historians and filmmakers. Moreover, Arab critics children’s film festivals in the Arab region. There
did not pay attention to animation films. were more than 100 short and long feature live-
In contrast to all the challenges that face Arab actions, documentaries, and animated films at the
animation, and the small number of animation official competition. Most of them were interna-
productions compared with the Western anima- tional productions and few were from Arab film-
tion productions, the Arab world is abundant makers, and there were no Arabic feature length
with magnificent folktales such as the One Thou- animated films. This shows the limited amount of
sand and One Nights stories which are suitable for animated short and feature film productions in
adaptation to make into many animated films. Arab countries.
Challenges Facing the Arab Animation Cinema 271

During the festival, an interview was arranged the Arab region, most of the experiments that hap-
with one of the festival’s jury committee, pened by the Arab animation filmmakers to show
their Arabic identity and culture through animation
Dr. Mohamed Ghazala (2012), the author of Ani- are independent and without serious support from
mation in the Arab World: A Glance on the Ara- the Arab governments. Most of the high quality
bian Animated Films Since 1936, the sole booklet animation productions in Arab countries such as
on the market about Arab animation history. Egypt, Jordan, Tunisia and Morocco focus on TV
Ghazala is also the founder and director of the
commercials and TV series especially for Ramadan,
and don’t have interest in producing artistic films
C
regional African and Arabian chapter of the Inter- for cinema and festivals. You can only see the
national Association of Animation Filmmakers graduation projects by the Arab art students, who
(ASIFA). The aim of this organization is to make artistic animation films, then after the gradu-
ation, they work in the industry to produce com-
involve Arabic and African animation filmmakers mercial works and the artistic work is disappearing
in creating, developing, and promoting their own quickly. (Ghazala 2012)
regional identity and having an impact in the
international animation market by participating Arab film productions focus more on making
in festivals and setting up some animation live-action films. For example, in Egypt, “The
workshops. Arab Hollywood,” there had been produced thou-
Ghazala notes that the booklet is a collection of sands of live-action films (Shafik 2007). Surpris-
five articles about Arab animation published in a ingly, Al Sahar Studio, one of the biggest
South Korean animation magazine called animation studios in Egypt, has had financial dif-
Animatoon in the Korean language. Every two ficulties since 1998 in attempting to complete
months, he covered the animation in different their first Egyptian feature length animated film
areas of the Arab world such as Egypt, North The Knight and the Princess (Ghazala 2011).
Africa, the Ash-sham area, and the Gulf area. Therefore, it is appropriate to question the reasons
Subsequently, he collected the five articles with why there have been so few animated feature films
some editing then presented them as a conference successfully produced in the Arab world; Is it
paper at Athens Animfest Animation Festival in because the Arab film producers think that anima-
2011 in Greece. The booklet contains only tion is only for television and children? Ghazala
56 pages and includes a small filmography of points out that for a long time, Arab national
Arab animation with some valuable data that is television has imported all the animation films
important to any researcher interested in Arab and TV series that suited children:
animation. When I was a kid I thought that the animation is
Ghazala explains the reasons behind the small only Walt Disney’s productions. When I grew up, a
friend gave me a video tape of animated films
selection of Arab animation in his booklet; he produced in the Czech Republic, which has
collected those Arab animated films individually completely different style than the films we used
by himself. The main problem was that he could to watch in our national televisions. These Czech
not find any official archive or library for Arab animated films were abstract, artistic and nice
stories. In that time I realized that animation
animation. It was hard to find the original copy of could be for kids and adults. The Arab did not
the animated films, and few Arab animated films screen the East European animated films that
are available on the internet, and then, only with were produced in the 60s and the 70s in the cin-
low resolution. Ghazala points out the problems emas and television; these films could inspire the
Arab of making such artistic animation films.
of Arab animation films in terms of quantity and (Ghazala 2012)
quality compared with the Western animation
productions: Another challenge facing Arab animation is the
lack of animation academies in the Arab region;
I have attended many international animation festi- many universities have Arts schools that focus on
vals as a filmmaker or jury member; unfortunately,
graphic design and the fine arts, but only a very
there were hardly any Arab animation in those
international festivals. There is no systematic few schools teach animation. In addition, there are
approach to producing and educating animation in a few animation festivals in some Arab counties
272 Challenges Facing the Arab Animation Cinema

such as the Cairo Children’s Film Festival in

Egypt, the Meknes International Festival for Ani-
mation Films in Morocco, and the newly
established festival JoAnimate in Jordan. In con-
trast, the governments of Europe, Japan, and
North America acknowledge the importance of
the animation industry by giving financial support
and arranging many animation festivals which
develop the filmmakers and the animation indus-
try in their countries.
Making animated feature films in the region is
a massive risk due to the unstable market and the
high expenses of making them. On the other hand,
the Arab countries include more than 300 million
people who speak the same language and share
the same culture, and this would clearly be a
promising marketplace if there were appropriate
methods for media marketing to reach it. The Arab Challenges Facing the Arab Animation Cinema,
producers should take the Western animation mar- Fig. 2 The Conqueror (1993) (Produced by Osama
Khalifa)
kets as a model, and see how animation could
have huge success and profits at the box office.
In 2009, Aljazeera News, one the biggest news feature animated film The Lion of Ain Jalout
broadcasting channels in the Middle East, had an (Asad Ain Jalout, 1998) which tells a story about
interview with the animation producer and the the Egyptian Muslim leader Prince Saif Al-din
cofounder of Ella Cartoon Studios, Mr. Osama Qutuz when he led the Muslims to achieve victo-
Khalifa from Saudi Arabia. The title of this inter- ries against the Crusaders in Mansoura in Egypt
view was The reasons for the weaknesses of the and Mangouls in Ain Jalout in Palestine in the
Arab animation industry (Without borders [Bila thirteenth century.
Hodoud] 2009). Khalifa produced more than However, Khalifa also produced some reli-
14 Arabic animated feature-length films; started gious animated feature films such as The Immortal
with the feature animated film The Island of the Journey (Rehlat Al-kholoud, 1996) directed by
light (1988) which was based on the Arabian Darwish Yasin (Fig. 3). The story is adapted
novel Hay Bin Yakzan (Living son of Awake) from the Holy Quran, Surat Al-Buruj (The Man-
which was written by the Andalusian philosopher sions of the Stars) Chapter 85 verses 4–9. All of
and novelist Ibn Tufail in the early twelfth the animation and illustrations were made in
century. cooperation with a studio in Turkey. However,
Most of Ella Cartoon Studios films are histor- the style of the illustrations and animation is sim-
ical, such as The Conqueror (Al Fatih, 1995), a ilar to the Japanese cartoon visual style used in
feature animated film that tells a story about the Anime and Manga, using common visual ele-
conquer of Constantinople in the fifteenth century ments such as big eyes, minimum facial details,
by the Turkish leader Sultan Mehmed Al Fatih a lot of light and shade, and numerous camera
(Fig. 2). Also, the animated film Conquered rotations. Khalifa also produced a number of
Andalucia (Fateh Al Andalous, 1998), which political animated films especially about the Pal-
tells a story about the Muslim hero Tariq Ben estinian Israeli conflicts, such as The Olive’s
Ziad when he conquered Andalucia in Spain in Dream (Holm Al Zaytoun, 2009) and a 3D ani-
the early eighth century to spread the religion of mated film Martyr of World (Shaheed Al-alam,
Islam in the West. The studio also made the 2003).
Challenges Facing the Arab Animation Cinema 273

his production company stopped making more

films. The evidence suggests that there are a num-
ber of reasons for this struggle domestically and
internationally, such as cultural challenges; the
majority of the Arab people think that animation
films are only for children and for that reason there
is no success for theatrical releases of any Arabic C
animated films so far. However, Arab animation
filmmakers are trying to convince investors and
Arab audiences that the target audience of their
animated films is general and refer to the huge
success in animation in the West. Another reason
could be political challenges; some of the films
were made about the Palestinian Israeli conflict
and those films could be difficult to screen in
Europe and USA, because they might be consid-
ered as anti-Semitic. Moreover, most of Arab
Challenges Facing the Arab Animation Cinema, films have Muslim heroes who achieved victories
Fig. 3 The Immortal Journey (1994) (directed by:
Darwish Yasin, produced by Osama Khalifa) against the Crusaders and Byzantine empires. The
Arab film producers want to demonstrate the
importance of making animated films appropriate
None of the above films were supported by the to the Islamic religion and Muslim civilization.
Arab governments; they were made using their However, there is view that by making such his-
own money only. Khalifa indicates that the Arabic torical stories, especially the conflicts between
media market was empty of animation film pro- Muslims and Crusaders, would remind both Mus-
ductions and he decided to take a risk and make lims and non-Muslims about the past, and it would
Arabic animated films for the Arab and interna- “illustrate feelings, fears and animosities in the
tional audiences. However, most of his animated present” (Runneymede Trust 1997). Therefore,
films have been made using high quality overseas the target audience for any historical and religious
production in Turkey, Ukraine, and China due to films that show conflict with others might be
the lack of Arab animation artists. Khalifa noted limited to Muslim audiences only, and these
that the Arab animation industry produces less types of animated films would be difficult to
than 0.5 % of Western animation production. screen for an international audience due to subject
One year after the Without Borders interview, matter that might offend the Western audience.
Ella Cartoon Studio went out of business, after This is the same as when Arabs and Muslims are
30 years of animation production. offended by Western animated films that stereo-
type the image of Arabs and Islam in a negative
manner.
Conclusion Most of the Arab animated films were
discussed so far are political, historical, and reli-
Arab film producers made number of animated gious which could be one of the main reasons why
films by using a variety of stories such as reli- those films are not reaching the international film
gious, historical, political, and folk tales. In addi- marketplace. The previous examples show that
tion, the target audiences of most of his films are the Arab animation industry is struggling in
the general public and families. Nevertheless, they terms of budget, marketing, broadcasting, distri-
did not make enough profit for the studio to keep bution, and government support. However,
producing Arabic animated films. For this reason, reaching the international market could be
274 Character - Avatar

achievable if Arab animation filmmakers and pro- Shafik, V.: Arab Cinema History and Cultural Identity,
ducers make universal stories that are suitable to 2nd edn. The American University in Cairo Press,
Cairo (2007)
everyone in the world. It is worth mentioning that Stevens, D.: Animated Retelling of the Birth of Islam. New
a number of animated films from Arab countries York Times [online], 13 Nov. Available at: http://
found that the best way of making low-budget movies.nytimes.com/2004/11/13/movies/13prop.html?
animated films/clips was to distribute them for _r¼0 (2004). Accessed 26 Jan 2012
Without borders [Bila Hodoud]: [TV program] Aljazeera
free via the Internet and social media such as Channel, 10 Apr 2009. 10:30 (in Arabic) (2009)
YouTube, Vimeo, Facebook, and other social
media networks. The number of Arabic online
videos increased greatly during and after the
recent Arab revolution, the “Arab Spring” in Character - Avatar
2011. This recent revolution gave Arab artists
the freedom of expression to discuss whatever ▶ Interacting with a Fully Simulated Self-
subjects they wanted to, including the political Balancing Bipedal Character in Augmented and
issues which attract a large number of audiences Virtual Reality
and received international channels’ attention
such as CNN, France 24, ABC News, Aljazeera,
and other international channels, during the Arab
Spring. Character AI

▶ Character Artificial Intelligence

References

Bendazzi, G.: Cartoon: One Hundred Years of Cinema

Animation. John Libbey & Company Ltd., London
(1994) Character Animation Scripting
Bendazzi, G.: Talk about the reason of having a small Environment
selection of Arab animation in the book ‘Cartoons:
One Hundred Years of Cinema Animation’ Interviewed Christos Mousas1 and Christos-Nikolaos
by . . .Tariq Alrimawi [email], 13 July 2012 (2012)
Ghazala, M.: Animation in The Arab World: A Glance on Anagnostopoulos2
1
the Arabian Animated Films Since 1936. LAP LAM- Visual Computing Lab, Department of Computer
BERT Academy Publishing, Saabrucken (2011) Science, Dartmouth College, Hanover, NH, USA
Ghazala, M.: Talk about his booklet: Animation in the Arab 2
Intelligent Multimedia and Virtual Environments
World, A glance on the Arabian animated films since
1936. Interviewed by . . .Tariq Alrimawi [Personal] Lab, Department of Cultural Technology and
Cairo, 29 Mar 2012 (2012) Communication, University of the Aegean,
Iqbal, Z.: Understanding islamophobia: conceptualizing Mytilene, Greece
and measuring the construct. Eur. J. Soc. Sci. 13(4),
574–590 (2010) [e-journal]
Jammal, O.: Talk about the feature animated film ‘Muham-
mad: The Last Prophet’. Interviewed by . . .Tariq Synonyms
Alrimawi [Personal] Chicago, 30 Nov 2012, 16:00
(2012) Animation scripting; Scripting environment;
Kalin, I.: Islamophobia and the limits of multiculturalism,
Chapter 1. In: Esposito, J., Kalin, I. (eds.) Scripting language
Islamophobia: The Challenges of Pluralism in the 21st
Century, p. 8. Oxford University Press, Oxford (2011)
Runneymede Trust: Islamophobia: A Challenge for All of Definition
Us [pdf]. Runnymede Trust Commission on British
Muslims and Islamophobia, London. Available at:
http://www.runnymedetrust.org/publications/17/32.html Scripting languages for character animation can
(1997). Accessed 12 Dec 2012 be characterized as the earliest type of motion
Character Animation Scripting Environment 275

control systems. In scripting environments, the virtual characters evolve can be a very complex
required animations are generated by writing a process.
script in the animation language. This means that Based on the aforementioned difficulties that
a user must learn the animation scripting lan- inexperienced programmers can face, this paper
guage. Such systems typically allow scenes and introduces a simple, easy-to-use, scripting envi-
objects to be described, along with their relation- ronment for animating virtual characters, which is
ship with each other. Among other advantages, based on a small number of scripting commands. C
they require no user interface and can therefore The scripting environment presented (see Fig. 1),
be implemented as cross-platform solutions. which is called CHASE, provides a user with the
ability to script the action of a character as well as
to script possible interaction between a character
Introduction and objects that are located within the virtual
environment.
Character animation can be characterized as a In order to implement CHASE, the following
complex and time-consuming process. This is parts were developed. Firstly, identify the basic
especially true when animating virtual characters actions that a character should be able to perform
based on key-frame techniques, as this requires and also generate the basic scripting commands.
prior knowledge of software solutions. Moreover, Secondly, a number of parameters that should
artistic skills are also required since the virtual allow the user not only to synthesize the required
character should animate as naturally as possible. motion of a character but also to gain a higher
In order to avoid time-consuming processes in level of control of each action of the character
animating virtual characters, motion capture tech- were defined. By using a reach number of motions
nologies now provide high-quality and realistic that a character can perform, as well as by associ-
animated sequences. This is possible because the ating these actions with specified keywords, a
ability to capture real humans in the act of motion dataset is created. The input commands
performing is achieved through the provided are handled by a number of developed back-
required motions. The advantages of motion cap- ground algorithms, which are responsible for
ture techniques are numerous, especially in the retrieving the desired motions and synthesizing
entertainment industry. However, the captured the requested actions of the character. During the
motion data, itself, is not always usable, since application’s runtime, CHASE synthesizes the
virtual characters should be able to perform tasks requested motion of the character and displays
in which the required constraints are not always the final animated sequence.
fulfilled. Thus, methodologies that retarget The remainder of this paper is organized as
(Gleicher et al. 1998), wrap (Witkin et al. 1995), follows. The section “Related Work” covers related
blend (Kovar et al. 2003; Park et al. 2002), splice work in character animation by presenting previous
(Van Basten and Egges 2012), interpolate (Kovar solutions for animating virtual characters that are
et al. 2002; Mukai and Kuriyama 2005), etc., the based on interactive or automatic techniques. Pre-
motion data have become available to help the viously developed scripting environments for the
animators to create the required motion animation of virtual characters are also presented
sequences. In addition to the motion synthesis and discussed. A system overview of CHASE is
techniques that are based on software solutions, presented in section “System Overview.” The
animating a virtual character through program- script commands, possible parameters, and addi-
ming is also difficult. This is especially true in tional functionalities that have been developed for
cases where animators, artists, and students do not CHASE are presented in section “Scripting Char-
have the required programming skills. Hence, ani- acter Animation.” Finally, conclusions are drawn
mating virtual characters in order to visualize and potential future work is discussed in section
ideas and generate simple scenarios in which “Conclusions and Future Work.”
276 Character Animation Scripting Environment

Character Animation Scripting Environment, Fig. 1 The interface of CHASE

Related Work Hodgins 2005), while the motion is captured by

motion capture technologies. Each of the previ-
This section presents work that is related to the ously mentioned methodologies has advantages
solution presented. Specifically, the following and disadvantages. The choice of the most appro-
paragraphs present methodologies that use differ- priate input device depends on the actual control
ent input devices or easily specified constraints for of the character’s motion that the user requires.
animating virtual characters, systems that pro- A variety of methodologies for the animation
vides to a user the ability to synthesize task- of a virtual character based on easily specified
based or scenario-related animated sequences, constraints have also been examined. These solu-
and previously proposed scripting environments tions are based on motion graphs (Kovar
for character animation. Finally, the advantages et al. 2002); literature such as Safonova and
provided by CHASE comparing by previous solu- Hodgins (2007); simple footprints (Van De
tions are presented. Panne 1997) that a character should follow, on
Interactive character control can be classified space-time constraints as proposed in Cohen
according to the input device that is used for the (1992); or statistical models (Min et al. 2012)
character animation process (Sarris and Strintzis that are responsible for retrieving and synthesiz-
2003). In general, the character controller can be a ing a character’s motion. However, even if easily
standard input device, such as a keyboard and a specified constraints enable a user to animate a
joystick (McCann et al. 2007). Alternatively, it character, different frameworks that permit either
can be more specialized, such as text input the interactive or automatic animation of a char-
(Oshita 2010), prosodic features of speech acter have been developed. In Feng et al. (2012),
(Levine et al. 2009), drag and drop systems which is a task-based character animation system,
where the motion sequences are placed into a by using a number of screen buttons, the user is
time-line (Oshita 2008), sketch-based interfaces able to animate a character and make it interact
(Davis et al. 2003), or the body of a user (Chai and with objects that are located within the virtual
Character Animation Scripting Environment 277

environment. Other methods (Thiebaux to script such actions as gestures and postures.
et al. 2008; Kapadia et al. 2011; Shoulson This methodology, which is based on the formal
et al. 2013), which can be characterized as semantics of dynamic logic, provides a solid
scenario-based character animation systems, pro- semantic foundation that enriches the number of
vide automatic synthesizing of a character’s actions that a character can perform.
motion based on AI techniques. The majority of previously developed scripting
In the past, researchers developed scripting environments and markup languages provide only C
languages and systems in the field of embodied specific actions that a character can perform. An
conversational agents. The XSAMPL3D additional limitation is the inability of such systems
(Vitzthum et al. 2012), AniLan (Formella et al. to enhance a character’s synthesized motion. There-
1996), AnimalScript (Rößling and Freisleben fore, a user always receives a lower level of control
2001), SMIL-Agent (Balci et al. 2007), and of the synthesized motion of a character. Moreover,
many others enable a user to script a character’s in cases in which a user must generate an animated
actions based only on predefined command. sequence where many characters will take part, a
Among the best known markup languages for great deal of effort will be required due to the
scripting the animation of virtual characters are difficulty of scripting multiple actions for multiple
the Multimodal Presentation Markup Language characters. This is especially true for users who wish
(Prendinger et al. 2004), the Character Markup to generate a sequence with animated characters, but
Language (Arafa et al. 2003), the Multimodal are inexperienced in programming.
Utterance Representation Markup Language These difficulties are overcome in the pre-
(Kranstedt et al. 2002), the Avatar Markup Lan- sented scripting environment. Firstly, instead of
guage (Kshirsagar et al. 2002), the Rich Repre- enabling a user to script an animated character
sentation Language (Piwek et al. 2002), the based on XML-related formats, a simplified
Behavior Markup Language (Vilhjalmsson et al. scripting environment with its associated scripted
2007), and the Player Markup Language (Jung language, which is based only on three com-
et al. 2008), which were developed for controlling mands, is introduced. Secondly, since a character
the behavior of virtual characters. should be able to perform concurrent actions, a
The representation of all previously mentioned simple extension of the basic command handles
languages is based to an XML-style format that this. Therefore, the user achieves a higher level of
allows users to script tasks featuring virtual char- control of a character’s action. Moreover, in cases
acters. However, these languages focus more on where a user must animate more than one charac-
communicative behavior such as gestures, facial ter simultaneously, one can specify the character
expression, gaze, and speech of virtual reality that should perform the requested action by
characters, instead of providing functional char- adding an additional method to the existing com-
acters that can generate scenario-related animated mand for a character. Finally, in cases where a user
sequences. must generate an animated character in a multitask
Various solutions that are similar to the pre- scenario, by simply specifying the row in which
sented methodology were proposed previously for the task should appear, the system will synthesize
the animation of virtual characters based on the tasks requested automatically.
scripting commands. StoryBoard (Gervautz et al. We assume that the described unique function-
1994) provides the ability to integrate a scripting alities that are implemented in CHASE will enable
language into an interactive character animation a user to synthesize compelling animated
framework. Improv (Perlin et al. 1996), another sequences in which a variety of virtual characters
framework with which to create real-time are involved. Hence, in view of the simplicity of
behavior-based animated actors, enables a user the developed commands, in conjunction with the
to script the specific action of a character based associated parameters, the proposed methodology
on simple behavior commands. STEP (Huang is quite powerful in comparison with the previous
et al. 2002) framework provides a user the ability solution. In addition, the easy-to-use and easy-to-
278 Character Animation Scripting Environment

remember commands make the presented capabilities, and examples of scenes can be
scripting environment effective, especially for found on the CHASE project page.
users who are inexperienced in programming.
Third-Party Implementations
A number of techniques and libraries are used to
System Overview construct CHASE. CHASE uses the Recast/
Detour library (Mononen 2014) for the path-
This section briefly describes the proposed sys- finding process and collision avoidance with the
tem. Specifically, a variety of background algo- environment. Concurrent actions are generated
rithms are responsible for recognizing the input based on a simple layering methodology similar
commands and synthesizing the motion of a char- to the one proposed in Oshita (2008). Finally, a
acter. The developed background algorithms com- similar (Lang 2014) full-body inverse kinematics
municate with the animation system, which is solver was implemented to handle the postures of
responsible for generating a character’s motion, a character while interacting with objects located
as well as with a path-finding methodology to within the virtual environment.
retrieve the path that the character should follow
when a locomotion sequence is required. Finally,
CHASE synthesizes and displays the requested Scripting Character Animation
motion sequence. Figure 2 represents the
procedure. Developing scripting commands for animating a
virtual character can be characterized as a com-
Interface plex process since a virtual character should be
The interface of CHASE (see Fig. 1) is character- able to perform variety of actions. In this section,
ized by its simplicity. In its current implementa- the identifications of the basic scripting com-
tion, it consists of a scene panel that displays the mands that are necessary to enable the virtual
resulting animations, an edit mode panel to edit character to navigate and interact within a virtual
the input objects, a progress bar that shows the environment are presented. Moreover, by intro-
progress of the displayed animation, a scripting ducing additional methods called by the main
box, and a few buttons for use in building, scripts, the system generates concurrent actions
playing, and clearing the written scripts. Finally, of a character, as well as animates multiple char-
buttons that save the scripted code and export the acters simultaneously. Finally, an additional func-
generated animated sequences are also provided. tionality of CHASE for scripting multitask
A downloadable version of the presented sys- animated sequences for the generation of
tem, documentation specifying all of its scenario-related animated characters is presented.

CHASE

Path Finding
Background Final Motion
Input Command
Algorithms
Animation
System

Character Animation Scripting Environment, Fig. 2 The architecture of CHASE

Character Animation Scripting Environment 279

Identifying Scripting Commands be performed by walking, running, etc. motions.

The application that is presented has been devel- Hence, in cases where a user needs a higher level
oped for users who are inexperienced in program- of control of the synthesized motions of a charac-
ming. Thus, simple, easily memorized scripting ter, parameters that enhance these actual actions
commands are necessary. To generate the required generated by the previously mentioned scripting
scripting commands, one must begin by identify- commands should be defined.
ing the possible actions or type of actions that a The first command that implemented the do C
character should perform. Generally, a character (parameters) command enables a user to script
should be able to perform simple actions such as simple actions of a character. This command has
waving its hand, tasks related to locomotion such a single mandatory parameter, which indicates
as moving to a target position, and interaction the action that the character should perform.
tasks such as grasping with its hand an object However, optional parameters to specify the
that is located in the three-dimensional environ- body part or the duration of the task can also
ment. It is apparent that these are the three basic be used. Specifically, the user can request a
types of actions that a virtual character should be single action by calling do(action), as well as
able to perform. Based on this general description, specify the target where the action should be
three basic scripting commands were developed: performed, the duration of the action, and the
the do(parameters), the goTo(parameters), and the body part that should perform the requested
interactWith(parameters). action. This command initially permitted a char-
The do(parameters) command provides a char- acter to perform the requested action without the
acter with the ability to perform a single action. need to perform a locomotion sequence (i.e., to
The goTo(parameters) forces a character to move wave its hand while staying in its position).
within the given virtual environment. The final However, the do(parameters) command can
command is responsible for making the virtual also be used to permit the character to perform
character capable of interacting with a variety of locomotion tasks, since one can request that a
tasks. Hence, the third command, the interactWith character performs a walking motion. Based on
(parameters), is responsible for providing the abil- these parameters that can be inserted into the do
ity to control a variety of the character’s actions. (parameters) command, a user has the means not
For these commands, the parameters within the only to generate the requested action but also to
parentheses indicate the possible parameters that generate an action that should fulfill user-
each of the scripting commands could receive (see specified constraints.
section Command Parameters). Due to the various The goTo(parameters) command enables the
parameters that each command receives, a user is character to perform locomotion tasks. The user
provided with the means to develop both abstract identifies a mandatory parameter, which is the
and specified action of a character. For example, target position that the character should reach.
with the goTo(parameters) command, it is possi- However, the user is also able to use an additional
ble not only to generate the required locomotion optional parameter that specifies the motion style
of a character but also to enable a user to gain that will animate the character. Therefore, a char-
better control of the synthesized motion of a char- acter’s locomotion to a target position can be
acter, since the user can specify how the locomo- scripted either by (i) inserting the target position
tion of a character should be generated. The such as goTo(target) when a simple walking
following section presents the basic parameters motion of the character is desired or (ii) inserting
that each command receives. goTo(target, motion style) when both target posi-
tion and motion style are specified.
Command Parameters The final command that is implemented in
A task assigned to a character can be performed in CHASE, the interactWith(parameters), can be
a variety of different ways. For example, a characterized as more complex than the two pre-
sequence of locomotion to a target position can viously mentioned commands. The reason is that
280 Character Animation Scripting Environment

there are numerous possible interactions between A simple example of this capability of the
a character and an object. If a character is asked to commands illustrates this. While using the do
interact with an object, various actions can be (parameters) command, the user may request that
generated. Even if it is possible to associate only either the body part or the duration parameter
actions with specific body parts of a character in or both of these be filled. In any case, the system’s
a preprocessing stage, there are also possible var- decision in generating the requested motion is not
iations of the required actions. These variations influenced by other factors since it is capable of
may be related to the character’s body or to the recognizing the correct form of the scripted com-
duration of the display of the action. For example, mand in all of the aforementioned cases.
scripting a character to kick a ball may also The three commands that are examined in this
require specifying the foot that should perform paper in conjunction with the associated parame-
this action. Moreover, asking a character to ters that can be used to animate a virtual character
knock a door may also require specifying the are summarized in Table 1. In addition, a small set
duration in the knocking. For that reason, four of possible keywords that the user could employ
different parameters have been defined. The first in order to animate virtual characters is presented.
two parameters (object name and interaction mod- It is assumed that an additional control parameter
ule) are mandatory. They indicate the object that for the synthesized motion could be quite benefi-
the character should interact with and the interac- cial, since it enables the user not only to animate a
tion module that should be generated. However, character but also to force the system to synthesize
depending on the user’s requirements for generat- the user’s actual wish. Complete documentation
ing a specific action, two more optional parame- of all possible actions that can be synthesized by
ters could also be inserted. The first one (body the character can be found in the CHASE
part) enables the user to choose which of the webpage (URL omitted for review purposes).
character’s body parts should perform the
requested action. In the current implementation, Scripting Concurrent Actions
the user is permitted to choose the hand or foot Virtual characters, such as humans, should be able
that will perform the action. The second parameter to perform more than one action simultaneously.
(duration) enables the user to choose the time This section presents the scripting process for
(in seconds) required for the requested action. concurrent actions that a character can perform.
Based on the possible parameters that each The concurrent action functionality is based upon
command could receive, the following should the ability to specify the body part that should
be noted. Firstly, while the user did not specify perform the action in conjunction with the base
any optional parameter for a scripted command, action that has been requested. The concurrent
the system generates the required action taking action lies between the do(parameters) and either
into account a predefined set of parameters that the goTo(parameters) or the interactWith(param-
are associated with each action of the character. eters) commands. Specifically, to have a character
For example, if a user requests that a character perform concurrent actions, the do(parameters)
kick a ball, the system will display only a single command is attached to either the goTo(parame-
kick by the character. The reason is that a ball ters) or the interactWith (parameters). A simple
kicking action is defined as to be performed example follows. To cause a character to perform
only once to avoid synthesizing meaningless a motion, such as waving its hand while walking
and repeated motions. Secondly, it should be to a target position, the system permits the user to
noted that each optional parameter is indepen- script the desired walking motion of a character
dent. This means that the user is not required to and to request the additional motion that the sys-
specify all of the optional parameters provided tem should generate. Hence, the previous example
by each command. Therefore, the user may can be requested simply by scripting goTo(target,
control specific components of the requested walk).do(wave hand, handR). Thus, by permitting
action. the user to generate additional actions of a
Character Animation Scripting Environment 281

Character Animation Scripting Environment, Table 1 Commands and associated parameters that can be used in
CHASE to request an action by an animated virtual character
Commands Parameters Parameter examples
do(parameters);
do(action); Action Wave hand
dofaction, target); Jump
dofaction, duration); Walk C
dofaction, body part, target); Kick
dofaction, body part, duration); Etc.
Target Vector3 (x,y,z) object name
Duration Time in seconds
Body part HandR
HandL
FootR
FootL
UpperB
LowerB
goTo(parameters);
goTo(target); Target Vector3 (x,y,z)
goTo(target, motion style); Object name
Motion style Walk
Run
Jump
Walk back
Etc.
interactWith(parameters);
interactWith(object name, interaction module); Object name Any object’s name contained in the
interactWith(object name, interaction module, body part); scene
interactWith(object name, interaction module, duration);
interactWith(object name, interaction module, body part,
duration);
Interaction Kick
module Punch
Grasp
Sit
Open
Close
Etc.
Body part HandR
HandL
FootR
FootL
Duration Time in seconds

character, while another action is in progress can, Scripting Multiple Characters

be quite beneficial when more complex animated In animated sequences, it is quite common for
sequences are required. Therefore, this additional more than one character to participate in a single
functionality provides a higher level of control scenario. Hence, by extending the three scripting
over a requested action of a virtual character. commands, CHASE also enables a user to script
282 Character Animation Scripting Environment

more than one character simultaneously. This is assigned a different index value of the task array.
achieved by attaching an additional command to A simple example of generating the actions of two
one of the three basic commands, called different characters appears in Algorithm 2. How-
characterName(parameter). This command spec- ever, a user should be able to animate virtual
ifies the character that should perform an action, characters simultaneously in multitask scenarios.
permitting the user to control multiple characters, This is achieved in CHASE by using a
in cases where more than one character partici- two-dimensional array named tasks[index]
pates in the animation process. A simple example [index]. In this array the first index value repre-
of forcing a specific character to perform an action sents the row in which each action in generated,
follows. Consider a character named Rudy who is whereas the second index value represents the
required to walk to target. This procedure could be number of the character. It should be noted that
called by simply scripting goTo(target). each character should be represented by the same
characterName(Rudy). index value while developing a multitask sce-
nario. Hence, the background algorithms that are
Scripting Multiple Tasks implemented recognize and generate the
In scenario-related sequences that involve virtual requested tasks as separate entries. This enables
characters, the latter should be able to perform a the user to animate a number of characters simul-
variety of tasks one after the other. Thus, this paper taneously. A simple example in which there are
presents a method to script multiple tasks, such as two characters in a multitask scenario appears in
enabling a user to synthesize long animated Algorithm 3. It should be noted that a multitask
sequences. Generally, the tasks that a character scenario where multiple characters evolve in a
can perform are characterized by their linearity. general form can be represented as an array that
Specifically, a task begins while a previous task is has a dimensionality equal to M  N, where
completed, and the procedure continues until there M denotes the total number of characters evolving
are no other tasks for a character to perform. in the multitask scenario and N denotes the total
Based on the foregoing, a multitask scenario in a number of tasks that a character should perform.
general form can be represented as components of
an array that has a dimensionality equal to N  1,
where N denotes the total number of tasks that a Conclusions and Future Work
character should perform. By assigning each of the
actions an array called task[index], a user can gen- In this paper, a novel scripting environment,
erate long animated sequences. This is achieved by called CHASE, for use in animating virtual char-
allowing the user to assign singe tasks at each index acters was presented. CHASE enables a user to
value of the task array. A simple example of a request a variety of actions that a character can
multitask scenario appears in Fig. 3, as well as in perform by simply using three commands. Each
the accompanying video. Its scripting implementa- command, which receives a variety of parameters,
tion is represented in Algorithm 1. is associated with specific actions that the charac-
It is quite common in multitask scenarios to ter is able to perform. Moreover, the commands
involve multiple characters. Two different communicate with a variety of background algo-
approaches can be used in CHASE to script rithms that are responsible for generating the
more than one character simultaneously in a mul- actions requested of the character. In addition to
titask scenario. The first approach animates each the scripting commands, by introducing three
character one after the other. This means that the additional functionalities, the user is able to script
action required of a characterB is generated after concurrent actions of a character, multiple charac-
the action of a characterA has been completed. ters at the same time, and multitask scenarios in
The reason is that each task of the characters order to generate scenario-related sequences that
taking part in the multitask scenario have been involve animated characters.
Character Animation Scripting Environment 283

Character Animation Scripting Environment, Fig. 3 A multitask scenario generated by using Algorithm 1
284 Character Animation Scripting Environment

Character Animation Scripting Environment, Algo- Balci, K., Not, E., Zancanaro, M., Pianesi, F.: Xface open
rithm 1 A simple example for generating a multitask source project and smil-agent scripting language for
scenario creating and animating embodied conversational
agents. In: International Conference on Multimedia,
Data: input commands of a user
pp. 1013–1016. ACM Press, New York (2007)
Result: the result animated sequence Chai, J., Hodgins, J.K.: Performance animation from
task[1] ¼ do(wave hand, handR, 3); low-dimensional control signals. ACM Trans. Graph.
task[2] ¼ goTo(ball, walk).do(wave hand, handL); 24(3), 686–696 (2005)
task[3] ¼ interactWith(ball, punch, handR); Cohen, M.F.: Interactive spacetime control for animation.
ACM SIGGRAPH Comput. Graph. 26, 293–302
task[4] ¼ do(jump);
(1992)
task[5] ¼ do(wave hand, handR, 2); Davis, J., Agrawala, M., Chuang, E., Popović, Z., Salesin,
D.: A sketching interface for articulated figure anima-
tion. In: ACM SIGGRAPH/Eurographics Symposium
on Computer Animation, pp. 320–328. Eurographics
Character Animation Scripting Environment, Algo-
Association, UK (2003)
rithm 2 By placing the actions of two different characters
Feng, A.W., Xu, Y., Shapiro, A.: An example-based
at different index values of the task array, the system
motion synthesis technique for locomotion and object
generates each character action one after the other
manipulation. In: ACM SIGGRAPH Symposium on
Data: input commands of a user Interactive 3D Graphics and Games, pp. 95–102.
Result: the result animated sequence ACM Press, New York (2012)
task[1] ¼ goTo(ball, walk).characterName(characterA); Formella, A., Kiefer, P.P.: Anilan – an animation language.
In: Computer Animation, pp. 184–189. IEEE, New
task[2] ¼ goTo(ball, walk).characterName(characterB);
York (1996)
Gervautz, M., Schmalstieg, D.: Integrating a scripting lan-
guage into an interactive animation system. In: Com-
puter Animation, pp. 156–166. IEEE Press, New York
Character Animation Scripting Environment, Algo- (1994)
rithm 3 A multitask scenario in which there are two Gleicher, M.: Retargetting motion to new characters. In:
characters. In this scenario, characterA moves to its target 25th Annual Conference on Computer Graphics and
position while walking, and characterB moves to its target Interactive Techniques, pp. 33–42. ACM Press, New
position while running. Finally, characterA punches York (1998)
characterB with his right hand Huang, Z., Eliëns, A., Visser, C.: Step: A scripting lan-
guage for embodied agents. In: Workshop of Lifelike
Data: input commands of a user
Animated Agents, pp. 87–109. Springer, Berlin (2002)
Result: the result animated sequence Jung, Y.A.: Animating and rendering virtual humans:
tasks[1][1] ¼ goTo(target, walk).characterName Extending x3d for real time rendering and animation
(characterA); of virtual characters. In: International Conference on
tasks[1][2] ¼ goTo(target, run). characterName Computer Graphics Theory and Applications,
(characterB); pp. 387–394. SCITEPRESS, UK (2008)
tasks[2][1] ¼ interactWith(characterB, punch, handR). Kapadia, M., Singh, S., Reinman, G., Faloutsos, P.:
characterName(characterA); A behavior-authoring framework for multiactor simu-
lations. Comp. Graph. Appl. 31(6), 45–55 (2011)
Kovar, L., Gleicher, M.: Flexible automatic motion blend-
ing with registration curves. In: ACM SIGGRAPH/
Eurographics Symposium on Computer Animation,
Cross-References pp. 214–224. Eurographics Association, UK (2003)
Kovar, L., Gleicher, M., Pighin, F.: Motion graphs. ACM
▶ Teaching Computer Graphics by Application Trans. Graph. 21(3), 473–482 (2002)
Kranstedt, A., Kopp, S., Wachsmuth, I.: Murml:
A multimodal utterance representation markup lan-
guage for conversational agents. In: AAMAS Work-
References and Further Reading shop Embodied Conversational Agents – Let’s Specify
and Evaluate Them! (2002)
Arafa, Y., Mamdani, A.: Scripting embodied agents behav- Kshirsagar, S., Magnenat-Thalmann, N., Guye-Vuillème,
iour with cml: character markup language. In: Interna- A., Thalmann, D., Kamyab, K., Mamdani, E.: Avatar
tional Conference on Intelligent User Interfaces, markup language. In: Workshop on Virtual Environ-
pp. 313–316. ACM Press, New York (2003) ments, pp. 169–177. Eurographics Association (2002)
Character Artificial Intelligence 285

Lang, P.: Root-motion. http://www.root-motion.com/. Autonomous Agents and Multiagent Systems, ACM
Accessed 29 Nov 2014 Press, New York (2008)
Levine, S., Theobalt, C., Koltun, V.: Real-time prosody- Van Basten, B., Egges, A.: Motion transplantation tech-
driven synthesis of body language. ACM Trans. Graph. niques: a survey. Comput. Graph. Appl. 32(3), 16–23
28(5), 1–10 (2009). Article No. 28 (2012)
McCann, J., Pollard, N.: Responsive characters from Van De Panne, M.: From footprints to animation. Comput.
motion fragments. ACM Trans. Graph. 26, p. Article Graph. Forum 16, 211–223 (1997)
No. 6 (2007)
Min, J., Chai, J.: Motion graphs++: A compact generative
Vilhjalmsson, H., Cantelmo, N., Cassell, J., Chafai, N.E.,
Kipp, M., Kopp, S., Mancini, M., Marsella, S., Mar-
C
model for semantic motion analysis and synthesis. shall, A.N., Pelachaud, C., Ruttkay, Z., Thorisson, K.
ACM Trans. Graph. 31(6), Article No. 153 (2012) R., Welbergen, H.V., Werf, R.J.V.D.: The behavior
Mononen, M.: Recast/detour navigation library. https:// markup language: Recent developments and chal-
github.com/memononen/recastnavigation. Accessed lenges. In: Intelligent Virtual Agents, pp. 99–111.
29 Nov 2014 Springer, Berlin (2007)
Mukai, T., Kuriyama, S.: Geostatistical motion interpola- Vitzthum, A., Amor, H.B., Heumer, G., Jung, B.:
tion. Trans. Graph. 24, 1062–1070 (2005) Xsamp13d: An action description language for the
Oshita, M.: Smart motion synthesis. Comp. Graph. Forum animation of virtual characters. J. Virtual Reality
27, 1909–1918 (2008) Broadcast. 9, Article No. 1 (2012)
Oshita, M.: Generating animation from natural language Witkin, A., Popovic, Z.: Motion warping. In: 22nd Annual
texts and semantic analysis for motion search and Conference on Computer Graphics and Interactive Tech-
scheduling. Vis. Comput. 26(5), 339–352 (2010) niques, pp. 105–108. ACM Press, New York (1995)
Park, S.I., Shin, H.J., Shin, S.Y.: On-line locomotion gen-
eration based on motion blending. In: ACM
SIGGRAPH/Eurographics Symposium on Computer
Animation, pp. 105–111. Eurographics Association,
UK (2002)
Perlin, K., Goldberg, A.: Improv: A system for scripting Character Artificial
interactive actors in virtual worlds. In: 23rd Annual Intelligence
Conference on Computer Graphics and Interactive
Techniques, pp. 205–216. ACM Press, New York
(1996)
Youichiro Miyake
Piwek, P., Grice, M., Krenn, B., Baumann, S., Schroder, Square Enix Co., Ltd., Tokyo, Japan
M., Pirker, H.: Rrl: A rich representation language for
the description of agent behaviour in neca. In: AAMAS
Workshop on Embodied Conversational Agents (2002)
Prendinger, H., Descamps, S., Ishizuka, M.: Mpml:
Synonyms
A markup language for controlling the behavior of
life-like characters. J. Vis. Lang. Comput. 15(2), Character AI; Decision-making; Perception AI
183–203 (2004)
Rößling, G., Freisleben, B.: Animalscript: an extensible
scripting language for algorithm animation. ACM
SIGCSE Bull. 33, 70–74 (2001) Definition
Safonova, A., Hodgins, J.K.: Construction and optimal
search of interpolated motion graphs. ACM Trans. A character AI is a system that makes decisions
Graph. 26, 106 (2007). Article No. 106
Sarris, N., Strintzis, M.G.: 3D Modeling and Animation:
for what a character will do. Here, a decision
Synthesis and Analysis Techniques for the Human means a strategy, tactics, or a certain behavior.
Body. IGI Global, Hershey (2003) Character AI does not only refer to the
Shoulson, A., Marshak, N., Kapadia, M., Badler, N.I.: decision-making process, but also to the
Adapt: The agent development and prototyping
testbed. In: ACM SIGGRAPH Symposium on Interac-
whole system used for dynamically connecting
tive 3D Graphics and Games, pp. 9–18. ACM Press, the character’s AI and the game world. Such
New York (2013) systems implement what is called as an “agent
Thiebaux, M., Marsella, S., Marshall, A.N., Kallmann, M.: architecture” (Russell and Norvig 2016a). An
Smartbody: Behavior realization for embodied conver-
sational agents. In: International Joint Conference on
agent architecture of character AI has three
Autonomous Agents and Multiagent Systems, vol. modules: perception, decision-making, and
1, pp. 151–158. International Foundation for memory modules.
286 Character Artificial Intelligence

Introduction several software modules, such as a sensor mod-

ule, a recognition module, a decision-making
In this entry, an overview on the history of the module, a motion-making module, and an effector
development of character AI is presented. Char- module. A system that employs a sensor module
acters’ AI techniques differ depending on the and a recognition module is called a “perception
scale of the game. For small-size games, “scripted AI” system, which is used for recognizing the
AI” is the most used method for controlling a environment in the game (see section “Perception
character, whereby a scripted AI system uses AI”). The decision-making module is the largest
script languages to issue commands that directly module and is used for making decisions using
determine characters behaviors. For middle-size certain decision-making algorithms, such as rule-
games, decision-making algorithms, such as rule- based, state-based, behavior-based, utility-based,
based systems, state machines, or behavior trees, goal-based, task-based, and simulation-based
are used. For large-size games, character AI is decision-making algorithms (Miyake 2017) (see
controlled by a large system based on an agent sections “Decision Making for Game Characters”
architecture with a perception module, a decision- and “Memory and Knowledge Representation”).
making module, a motion-synthesis module, a
memory module, and so on.
Drastic changes in character AI technologies Perception AI
came from the first 3D games made in the 1990s.
In 3D games, characters must have their own A character in a digital game must recognize the
artificial intelligence. Each character must have environment around itself in real time to respond
sensors and a body of its own and perform to changes in the world. However, it is impossible
decision-making thinking. Characters with these for a character AI to recognize the game world
traits are called “autonomous agents.” From the directly, because it would be an excessively heavy
late 1990s to 2010, the focus of character agent load for an AI agent to analyze and understand the
research shifted from scripted agents to autono- environment around itself from its point of view in
mous agents (Miyake et al. 2017). real time. Perception AI systems are used to help
In the second half of the 1990s, research for character AIs to recognize the environment in
character AI began at the MIT (Massachusetts the game.
Institute of Technology), and many robotics tech- The role of perception AI is to acquire infor-
nologies have been applied for character AI in the mation of the environment via physical simula-
game industry ever since. The MIT synthetic char- tions, such as ray-casting and sound propagation,
acter group (MIT Synthetic Character Group and to abstract an essential whole-world image
2003) has studied AI for virtual creatures. They from this information. Perception AI essentially
introduced AI techniques from the field of robot- means how the game world is seen from the point
ics to autonomous agents in digital games. One of view of a character.
agent architecture that they developed using a Perception AI uses efficient methods to acquire
blackboard architecture is called the “C4 architec- information of objectives and terrain from a game
ture” and is shown in Fig. 1 (Isla et al. 2001; Burke stage. All objectives contain information that they
et al. 2001). The main feature of this model is to use to represent themselves. For example, a rock
separate intelligence modules and memory mod- in a game would have properties such as “rock,”
ules. Their research results were presented in “breakable,” and “movable,” and a door in a game
GDC 2001, and it had a profound impact in the would have properties such as “door,” “open,”
game industry. For example, Halo 2 and F. and “can go through,” and so on. In Halo 2, the
E.A.R. (2005, Monolith Productions) used this battlefield of the game is divided into three parts,
architecture (Orkin 2005, 2006). namely the front area, the middle area, and the
Similar to a C4 agent architecture, generally an back area. A representation of objectives is called
agent architecture for a game character consists of a knowledge representation (KR), and a
Character Artificial Intelligence 287

Character Artificial Intelligence, Fig. 1 C4 architecture (Isla et al. 2001; Burke et al. 2001)

representation especially for terrain and space is through and reach the object without any collision
called a world representation (WR). By using KR with other objects, it means that the character can
and WR information, a character can recognize its see the object. This is the principle behind visual
present situation. sensors.
Perception AI involves sensor modules for In stealth-based games in particular, perception
performing specific simulations and may include AI is very important. Stealth-based games are
five modes for representing the five human senses. games in which the player generally breaks into
Visual sensors are the most important in digital a building while hiding from enemies. All ene-
games because the human player’s actions also mies have vivid perception AI. For example, in
depend on the player’s sight. In general, the visual Splinter Cell: Conviction (Ubisoft 2010), each
sensors of a character have visual regions called enemy has a field of view in the shape of a coffin
“field of vision,” as shown in Fig. 2. When an because this shape is very useful for being able to
object enters the field of vision, one or multiple see the deepest and farthest region of narrow
rays are casted onto the object. If the ray can go rooms. In Final Fantasy XV (SQUARE ENIX
288 Character Artificial Intelligence

Character Artificial
Intelligence,
Fig. 2 Vision sensor with a
field of view

Character Artificial
Intelligence,
Fig. 3 Sound propagation

2016), each character has two visual regions coming from the origin. Furthermore, in some
consisting of a forward circular sector and a back- stealth-based games, as shown in Fig. 3, sound-
ward circular sector. The forward circular sector propagation simulations are performed to confirm
can recognize objects clearly, whereas the back- the existence of a sound path through space from
ward circular sector has ambiguous recognition the origin to a character by connecting the edges
functions for vaguely noticing objects. of walls. Moreover, in Sprinter Cell: Blacklist,
To implement hearing, the easiest way sound propagation was implemented. The
involves the use of a spherical sound source with shortest path for sounds is calculated by
a certain radius around a character. This sphere is connecting the edges of walls (Walsh 2014). In
used to determine if a character can hear the sound Hitman (Io-Interactive 2016), sound propagation
Character Artificial Intelligence 289

is calculated via diffraction and transparency machine methods are suitable for describing a
(Rørbech 2015). character’s behavior and its transitions in 3D
And typical collision or overlapping sensors space. Many game developers made original
are used for character’s touching to objects. state-machine GUI tools, and some middlewares
were supported by AI middleware companies. For
example, character AI in Uncharted 2: Among
Decision-Making for Game Characters Thieves (Naughty Dog, LLC, 2009) is based on C
a state-machine system (Gregory 2009) (Fig. 5).
The decision-making algorithms used in digital Utility-based algorithms represent a method
games can be classified in seven types, namely for selecting one option from many by calculating
rule-based, state-based, behavior-based, task- their value. For example, a character may have
based, goal-based, utility-based, and simulation- many actions that they could execute in a given
based algorithms (Fig. 4). situation. The character’s AI selects one action out
In the 1980s, almost all games used rule-based of all possible actions by calculating their
algorithms to control characters, as in, if the corresponding utility value. Utility can alternately
player’s position is to the right, then go left. mean “effectiveness” or “evaluation value.”
Some rules were prepared, and one rule was Behavior-based algorithms began to be used
selected to be executed after evaluating all rules. for character AI in Halo 2 in 2004 (Isla 2005a,
Even now, rule-based algorithms are very popular b). In Halo 2, a new method called “behavior tree”
for small- and middle-size (sometimes even big- was developed for character AI by Damian Isla,
size) games. For example, The Sims 3 (Maxis the AI lead of the game. A behavior tree is a
2009) uses rule-based methods to control NPCs hierarchical acyclic directed tree that originates
(nonplayer characters) by preparing and executing from a root and reaches behaviors, as shown in
multiple action rules (Evans 2010). Fig. 6 An advantage of behavior trees is their
In the 1990s, especially in the first 3D games, scalability, by which users can modify a behavior
state-machine methods became popular. State- tree without having to perform complex

Character Artificial Intelligence, Fig. 4 Comparison of seven decision-making algorithms

290 Character Artificial Intelligence

Character Artificial Intelligence, Fig. 5 Time table of decision-making algorithms

operations. When one state is added to a state employ a “random rule” execute one node
machine, all the connections between the new selected at random.
state and old states must be considered. However, Goal-based algorithms began to be used in
when one node is added to a behavior tree, only 2004. For example, F.E.A.R. uses goal-oriented
the local structure around the new node has to be action planning (GOAP). GOAP is a kind of
considered. Behavior trees have become the most STRIPS (Stanford Research Institute Problem
popular algorithms in the game industry. In more Solver) developed in the 1970s. Usually, this
than 70% of big-size and mobile games, at least solver is used for making long static plans. How-
one behavior tree is used. ever, Jeff Orkin applied STRIPS to the real-time
Behavior trees have a layered architecture. planning of characters’ actions by chaining many
Each layer of a behavior tree has several child actions, which are described by a precondition, an
nodes and one selection rule. Each layer action, and a postcondition, to produce a long
selects nodes to be executed via the selection actions plan (Orkin 2005, 2006).
rule. First, for each behavior, the tree judges Task-based algorithms are the most promising
whether it is possible to execute that behavior for the future of game AI because they can be used
or not under the current situation, and the to develop the most complex thinking patterns for
nodes that are deemed unfeasible to become character AIs. Hierarchical task networks (HTNs)
active are removed from the selection process. are decision-making algorithms for making plans
Then, nodes are selected based on rules similarly to node networks. They were developed
applied to the remaining ones. Based on the in academic research in the 1990s (Russell and
selection rules, the selection process unfolds. Norvig 2016b). HTNs have domains and
For example, a layer with a “sequence rule” methods. A domain means a collection of tasks,
executes its nodes in a certain order according which represents a region of a problem. Methods
to priority. On the other hand, layers with a divide one task into smaller tasks in a domain.
“priority rule” attempt to execute only one HTNs have hierarchical methods and domains.
node with maximum priority out of all the A task which can be resolved into subtasks further
nodes that could become active. Layers that by its method is called a composite task, and a task
Character Artificial Intelligence 291

Character Artificial Intelligence, Fig. 6 Behavior tree

which cannot be resolved further is called a prim- tree tools made by different game companies, and
itive task. Killzone 2 and 3 (Guerrilla games, many game engines contain proper decision-
2009, 2011) used HTNs (Straatman et al. 2009, making tools.
2013; Champandard 2008), and a few other com-
panies used them.
Simulation-based algorithms are used for solv- Memory and Knowledge Representation
ing complex problems that cannot be transformed
into more easily solvable problem. In such cases, A character’s AI stores information on its mem-
simulation-based methods are very effective. For ory. The information stored in this memory can
example, when an AI car goes through a narrow have many knowledge types, which are referred to
cave, it tests some combinations of turns and as knowledge representations (KRs), as explained
accelerations to find the best combination for in the section “Perception AI.” Any knowledge
moving smoothly. has a form of KR.
Although there are seven types of decision- For game characters, it is difficult to recog-
making algorithms, only one or two algorithms nize terrains, objects, or facts in the game by
are used in each game. The design of decision- themselves. Knowledge representations of all
making systems and tools is done by engineers, the things that a character must recognize are
whereas implementing decision-making thinking prepared during game development so that
using those tools is done by the game designer. characters can recognize the game world
For example, there are many different behavior via KRs.
292 Character Artificial Intelligence

A character’s memory is basically a collection manager, and each witness of that fact has a ref-
of variables. However, in highly developed AIs, erence to the event stored in the event manager.
memory must be formatted via KRs. Each event has a reference counter that indicates
For example, in Halo 2, some objects have the number of witnesses who remember the fact.
“affordance” information (Isla 2005b). When a witness is killed, the reference counter is
Affordance is a concept that indicates possible reduced.
activity. It is a term used in cognitive psychology
for concepts such as eatable, movable, breakable,
and so on. If the KR of an object is “movable,” the Conclusion
character recognizes that the object can be moved
by referring the object’s KR. Character AI represents one of the cores of a
In F.E.A.R., the KR of characters is more digital game’s AI. Many AI and robotics technol-
sophisticated and is called a World Memory Fact ogies are integrated in character AI technology.
(WMF). A WMF has a unified data structure During the early stages of character AI develop-
(Orkin 2005): ment, many robotics technologies were
employed. However, the speed of the technical
WorkingMemoryFact progress of character AI has been very fast, such
{
that AI technologies originally taken from robot-
Attribute<Vector3D> Position
Attribute<Vector3D> Direction ics have been developed in the game industry
Attribute<StimulusType> Stimulus more than in the robotics industry. In the near
Attribute<Handle> Object future, character AI techniques developed in the
Attribute<float> Desire
game industry will be used in fields involving
...
float fUpdateTime digital characters other than gaming. Further into
} the future, more learning or evolution systems will
be introduced and partially replace these decision-
Attribute<Type> making systems.
{
The reason for this is that character AI can be
Type Value
float fConfidence simulated in a game world without involving real
} bodies and avoiding physical troubles. Thus, a
game world can be regarded as a cradle where
(This is C++ code to define the WMF data characters can be developed at a fast speed.
type. Each piece of information has a confidence
value.)
Another notable example is Gunslinger (2002, References
not released, Surreal). The title was not released,
but it exerted much influence in the KR Alt, G., King, K.: A Dynamic Reputation System Based on
Event Knowledge, AI Game Programming Wisdom,
implementations of subsequent games. This title vol. 1, 8.6, pp. 426–435 (2002)
also had KR for facts in the game. The KR of an Burke, R., Isla, D., Downie, M., Ivanov, Y., Blumberg, B.:
event had components, such as Subject Group, Creature Smarts: The Art and Architecture of a Virtual
Verb, Object Group, Object Individual, Magni- Brain. In: Proceedings of the Game Developers Con-
ference, pp. 147–166. (2001)
tude, Where, When, and Template (Alt and King Champandard, A.: On the AI Strategy for KILLZONE 2’s
2002). First, any event that was seen by a charac- Multiplayer Bots, AiGameDev (2008). http://
ter was registered in the global memory with an aigamedev.com/open/coverage/killzone2/
ID. The characters who saw the event had a refer- Evans, R.: Modeling Individual Personalities in The Sims
3, GDC 2010 (2010). http://www.gdcvault.com/play/
ence to the event’s ID. For example, assume that 1012450/Modeling-Individual-Personalities-in-The
Joe was killed by the lawmen in the square at FINAL FANTASY XV is a trademark or registered trade-
noon. This fact is remembered by an event mark of SQUARE ENIX CO., LTD. (2016).
Children Privacy Protection 293

Gregory, J.: State-Based Scripting in Uncharted 2: Among

Thieves, GDC 2009 (2009). http://www.gdcvault.com/ Cheating
play/1730/State-Based-Scripting-in-UNCHARTED
Isla, D.: Managing Complexity in the Halo2 AI, Game
Developers Conference Proceedings 2005 (2005a). ▶ Detecting and Preventing Online Game Bots in
http://www.gamasutra.com/view/feature/130663/gdc_ MMORPGs
2005_proceeding_handling_.php
Isla, D.: Dude, where’s my Warthog? From Pathfinding to C
General Spatial Competence, AIIDE (2005b). http://
naimadgames.com/publications.html
Isla, D., Burke, R., Downie, M., Blumberg, B.: A Layered Cheat-Resistant Gaming
Brain Architecture for Synthetic Creatures. In: Pro-
ceedings of IJCAI (2001). ▶ Secure Gaming: Cheat-Resistant Protocols and
MIT Synthetic Character Group.: (2003). http://characters.
media.mit.edu/
Game History Validation
Miyake, Y.: Current Status of Applying Artificial Intelli-
gence in Digital Games, Handbook of Digital Games
and Entertainment Technologies, pp. 253–292.
Springer (2017)
Chess Variant
Miyake, Y., Shirakami, Y., Shimokawa, K., Namiki K.,
Komatsu, T., Joudan, T., Prasertvithyakarn, P.,
Yokoyama, T.: A Character Decision-Making System ▶ Contemporary Computer Shogi
for FINAL FANTASY XV by Combining Behavior
Trees and State Machines, chapter 11, GAME AI
PRO 3, AK Peters/CRC Press (2017)
Orkin, J.: Agent Architecture Considerations for Real-
Time Planning in Games, AIIDE 2005 (2005). http:// Children Privacy Protection
web.media.mit.edu/~jorkin/
Orkin, J.: Three States and a Plan: The AI of F.E.A.R., Patrick C. K. Hung1, Marcelo Fantinato2 and
Game Developers Conference Proceedings (2006).
http://web.media.mit.edu/~jorkin/
Jorge Roa3
1
Rørbech, M.: Modular Sandbox Design: Tools and Faculty of Business and IT, Ontario Tech
Workflows for Hitman, GDC Europe (2015). https:// University, Oshawa, ON, Canada
gdcvault.com/play/1022824/Sound-Propagation-in 2
School of Arts, Sciences and Humanities,
Russell, S., Norvig, P.: Intelligent Agents, chapter 2, Arti-
ficial Intelligence: A Modern Approach, Global Edi-
University of Sao Paulo, Sao Paulo, Sao Paulo,
tion. Pearson Education Limited (2016a) Brazil
3
Russell, S., Norvig, P.: Planning and Acting in the Real Research and Development Center of
World, chapter 11, Artificial Intelligence: A Modern Information Systems Engineering (CIDISI),
Approach, Global Edition. Pearson Education Limited,
(2016b)
Universidad Tecnológica Nacional – Facultad
Straatman, R., Verweij, T., Champandard, A.: Killzone Regional Santa Fe (UTN-FRSF),
2 Multiplayer Bots, Paris Game/AI Conference Santa FeSanta Fe, Argentina
(2009). http://www.guerrilla-games.com/publications.
html
Straatman, R., Verweij, T., Champandard, A., Morcus R.,
Kleve, H.: Hierarchical AI for Multiplayer Bots in Synonyms
Killzone 3, GAME AI PRO (2013). Chapter 29,
pp. 377–390. Children Protection Engine; Toy Privacy
The Sims 3 is a trademark or registered trademark of
Electronic Arts Inc.,and developed by Maxis
(2009).
Splinter Cell: Conviction is a trademark or registered trade- Definition
mark of Ubisoft (2010).
Walsh, M.: Modeling AI Perception and Awareness in
Splinter Cell: Blacklist, GDC (2014). http://www.
The children privacy protection aims to enable
gdcvault.com/play/1020195/Modeling-AI-Perception- parents or guardians to be in control of their chil-
and-Awareness dren’s privacy by specifying their privacy
294 Children Privacy Protection

preferences for a toy, under the assumption that daughter’s conversations without parental con-
the toy has published an accurate privacy policy sent, in violation of the COPPA in the United
and complies with it in a privacy protection engine States. Serious concerns about privacy and data
attached to the toy. security are raised by the personally identifiable
information (PII) that is collected by these smart
toys. These concerns are crucial since collected
Privacy Protection Laws data involves sensitive children’s information
(Tang and Hung 2017).
Referring to the direction of the United States
Federal Trade Commission (FTC), Children’s
Online Privacy Protection Act (COPPA 1998), Privacy Consequences and Precautions
and the European Union Data Protection Direc-
tive (EUDPD), a child is an individual under While smart toys provide new educational and
the age of 13 years old (Hung 2015). Children entertaining values, experts have warned con-
privacy protection aims to protect children’s per- sumers of the data security and privacy issues of
sonal information, which is particularly sensitive, these toys. A recent United States Senate report
especially when associated with their real identity. (Nelson 2016) states that these toys may gather a
Children provide a unique user base which child’s personal information, which may poten-
requires special attention in several key areas tially cause serious consequences such as identity
related to their privacy (Hung et al. 2016). In theft. Parents should be aware of the information a
general, children do not understand the concept toy is collecting about them and their child. For
of privacy, and hence, they tend to disclose as most parents, it is difficult to evaluate policies
much information to people they can trust regarding data collection and use since they are
(Rafferty et al. 2017). In this context, the not usually experts with respect to data privacy
COPPA indicates that to protect children privacy, and security. However, parents must find a way to
a child’s personal information cannot be collected reject smart toys that do not provide information
without parental consent. about how a toymaker collects, uses, and secures
their children data.

Smart Toys Privacy

Toy Industry and Privacy
Nowadays, many toys on the market are becom-
ing integrated with the sensory and networking The usage behavior of children indicates that they
capabilities of mobile technology, introducing are more open to giving out personal information,
new threats to privacy for children who are the which makes issues of sensitive data sharing of
primary users of these devices. Online privacy for great concern. The North American Toy Industry
children has been a great concern in this environ- Association (TIA) released a white paper regard-
ment, particularly when the child’s private infor- ing the changing privacy and data security land-
mation is involved and can be potentially shared scape the toy industry is facing with the emerging
with other parties. For example, an invention popularity of child-directed mobile apps. The TIA
called “Google Toy” has caused many criticisms iterates the issues of children’s marketing and
from the media as people express concern about privacy in this context, indicating that privacy
possible privacy breaching by Google, especially and data security issues affect day-to-day opera-
with their children at home. Mattel’s “Hello Bar- tions of toy companies. The toy industry has also
bie” has been criticized for the negative effects on issued regulations for toy safety; however, these
children along with privacy concerns since its regulations have no mention of privacy (Hung
introduction in February 2015. In December 2015). A report from Pew Research Center and
2015, Mattel was sued in California by two Berkman Center for Internet & Society at Harvard
mothers to allege that Hello Barbie records their University indicates that most parents in the
Children’s Games, from Turtle to Squirtle 295

United States are concerned about their children’s Hung, P.C.K., Fantinato, M., Rafferty, L.: A study of
online privacy, some of the main concerns being privacy requirements for smart toys. The 20th Pacific
Asia Conference on Information Systems (PACIS
related to strangers online, and the data advertisers 2016), Chiayi, 27 June–1 July 2016
are collecting about their children’s online Nelson, B.: Children’s Connected Toys: Data Security and
behavior. Privacy Concerns. Office of Oversight and Investiga-
tions Minority Staff Report, US Senate Committee on
Commerce, Science, and Transportation (2016)
Rafferty, L., Hung, P. C. K., Fantinato, M., Peres, S. M.,
C
Iqbal, F., Kuo, S. Y., Huang, S.C: Towards a privacy
Children Protection Engine rule model for smart toys. The IEEE 50th Hawaii
International Conference on System Sciences
Privacy can result in physical safety of child user, (HICSS-50), Big Island, Hawaii, 4–7 January 2017
e.g., child predators. While parents strive to Salgado, A.d.L., Agostini do Amaral, L., Castro, P.C.,
Pontin de Mattos Fortes, R.: Designing for Parental
ensure their child’s physical and online safety
Control: Enriching Usability and Accessibility in the
and privacy, they may wish to be in control of Context of Smart Toys. Computing in Smart Toys. The
how their personal data is shared through the Springer International Series on Computer Entertain-
devices they are using. However, there is no stan- ment and Media Technology, pp. 103–127. Springer,
Cham (2017)
dardized child protection model for parental con-
Tang, J., Hung, P.C.K.: Computing in Smart Toys. The
trol in this paradigm (Fantinato et al. 2018). Springer International Series on Computer Entertain-
Parental control is a feature in a smart toy for the ment and Media Technology. Springer, Cham (2017)
parents to restrict the content the children can
provide to the toy. The children privacy protection
aims to enable the parents or guardians to be in
control of their child’s privacy by specifying their
privacy preferences, under the assumption that the
Children Protection Engine
toy has published an accurate privacy policy and
▶ Children Privacy Protection
complies with it in a privacy protection engine
attached with the toy (Salgado et al. 2017).

Cross-References
Children’s Games, from Turtle
to Squirtle
▶ Children Privacy Protection
Krystina S. Madej
▶ Smart Toys
Georgia Tech, Atlanta, Georgia, USA
▶ Toy Computing

MIT 1967, Pokémon GO 2016

References

COPPA: Children’s Online Privacy Protection Act of 1998, We are acutely aware that children use media all
United States Federal Trade Commission. [Online]. day, every day and that a part of that use is playing
Available http://www.coppa.org/coppa.htm (1998) video games. Today four out of five US house-
Fantinato, M., Hung, P.C.K., Jiang, Y., Roa, J., Villarreal, holds own game consoles (Lofgren 2016) and in
P., Melaisi, M., Amancio, F.: Perceived innovativeness
and privacy risk of smart toys in Brazil and Argentina. 2017 sales of children’s mobile games alone are
The IEEE 51st Hawaii International Conference on expected to reach $2.2 billion (Children’s Mobile
System Sciences (HICSS-51), Big Island, Hawaii, 3– Game n.d.). In 2013, 38% of children who were
6 January 2018 2 years old used mobile devices; in the two previ-
Hung, P.C.K.: Mobile Services for Toy Computing. The
Springer International Series on Applications and ous years, the rate of use for two to four-year-olds
Trends in Computer Science. Springer, Switzerland climbed from 39% to 80% (Rideout 2013).
(2015) Now a 91 billion dollar business worldwide
296 Children’s Games, from Turtle to Squirtle

(2016) (Fuscaldo 2016), video games were first of playing games, whether with older siblings,
developed barely 60 years ago. Adults have long parents, or other children, and the nature of some
used children’s games to entertain as well games that can be played across age groups. Too,
as educate. Plato’s comments in The Republic children are precocious – their nature is to chal-
(380 bce) on the value of structured play for devel- lenge themselves with new things; they seldom
oping young children into socially responsible and stay within their age category for long for any
well-adjusted adults, John Locke’s entreaty in Some activity. As an important market, children have
Thoughts Concerning Education (1693) to make been a motivating force in the evolution of game
education entertaining because children “love to genres. While not all genres are covered in
be busy, change and variety are what they delight the following history, those discussed are repre-
in; curiosity is but an appetite for knowledge,” and sentative of innovations in technology and shifts
John Dewey’s proponing in Democracy and Edu- in interest that have encouraged change. These
cation (1922) that children learn best through expe- include early educational and arcade style
rience, demonstrate to us society’s ongoing desire to games, handheld games, massive multiplayer
take advantage of children’s proclivity to learn as online games, active games, augmented, virtual,
they play (Madej 2016). It’s not surprising then that and mixed reality games. Our trajectory takes us
children’s video games had their start with a desire from the first LOGO Turtle to today’s augmented
to use technology to benefit learning. It is also not reality Pokémon Go Squirtle.
surprising that, on their own, children prefer to
accentuate entertainment over education.
Since the late 1960s, when the first program- 1960s: The Very Beginnings – LOGO and
ming language for children was developed at MIT, Education Through Constructive Play
until today’s mixed reality applications, games for
children have travelled a continuum committed to Computer play environments developed specifi-
both learning and entertainment objectives. The cally for children date back to Seymour Papert and
brief history that follows shows children’s games LOGO, the programming language released in
as evolving from two different beginnings: com- 1967 that he and colleagues developed at MIT.
puter applications with educational objectives for Papert had studied with the child psychologist
specific age groups and computer and console Jean Piaget in Switzerland and considered the
games designed to entertain a wide audience. computer an ideal tool for learning by doing.
The differentiation between children’s and adult’s Using LOGO, children as young as three con-
games is difficult to establish because of chil- trolled a “Turtle” to create graphics (Fig. 1)
dren’s facility with technology, the social nature (Blikstein n.d.). The program proved to help
Children’s Games, from Turtle to Squirtle 297

Children’s Games, from Turtle to Squirtle, Fig. 1 (a) First Turtle. (b) Buttonbox for preschoolers. (c) LOGO graphic

children learn complex notions qualitatively, catalog. One of its popular school offerings, Ore-
more deeply, and with less effort (Papert et al. gon Trail (1978), a game about settlers who made
1979). Papert’s work was highly influential and the difficult pioneer journey from Independence,
encouraged the use of computers and software Kansas to the Willamette Valley in Oregon,
programs as a supplement to work in schools. became a notable success when it was released
LOGO spawned a number of research innova- to the general public as a game title in 1985
tions that moved out of education and into enter- (Wong 2017). Other programs that became
tainment including LEGO Mindstorms, a robotic popular included DinoPark Tycoon (1993) and
system of LEGO bricks that had children building StoryWeaver (1994).
robots with motors, lights, and sensors. In 1972, the first home console system, Odys-
sey Home Entertainment System, offered enter-
tainment games based on the simple mechanics
Early Interest in Education and of moving light blips across the screen with a
Entertainment controller. Plug-in programs changed how these
blips reacted to the control units and created dif-
Educators found that children were drawn to ferent games. Players attached gameboard over-
the active engagement computers offered and lays to the television screen and could play,
became interested in providing their students among other games, Tennis, Hockey, Football,
with subject-based programs to take advantage Cat and Mouse, and Haunted House (Fig. 2).
of this increased eagerness to learn. They were Educational topics were also included: players
supported by educational organizations such as could, for instance, learn the names of the US
the Minnesota Educational Computing Consor- states. Odyssey games included poker chips, play
tium (MECC), which developed statewide sys- money, and score cards that completed the game
tems for learning about how to use new play experience. The system inspired Atari’s
technologies and software applications for differ- Nolan Bushnell to create Pong, the popular
ent age groups, grade levels, and subject catego- game in which players use paddles to hit a ball
ries (For more information on MECC see Jancer back and forth (Winter 2013).
(2016). For a list of programs available through
MECC in 1983 see Atari Program Exchange
Catalog (1983)). To reinforce learning at home, Edutainment
MECC sold its school software to parents and by
1983 offered more than 150 subject-related appli- By the early 1980s, PCs began selling in the
cations for children age three-and-up in its annual millions and were being advertised as family
298 Children’s Games, from Turtle to Squirtle

Children’s Games, from Turtle to Squirtle, Fig. 2 The

Odyssey Home Entertainment System (1972) Children’s Games, from Turtle to Squirtle,
Fig. 4 Reader Rabbit: Menu Screen (1984)

by cofounder Dr. Leslie Grimm, set the pattern for

educational series. Using stills, simple animation,
and music, the click and point games engaged
children in learning about language. Children
picked individual letters, or combinations or
sequences of letters, to complete activities and
were rewarded with a jumping/tumbling bunny
and congratulatory sounds (Fig. 4) (Reader Rabbit
and the Fabulous Word Factory n.d.). Successful
Learning Company titles included Clue-Finders,
Children’s Games, from Turtle to Squirtle, Mind Power, Zoombini’s, and Super Solvers. Each
Fig. 3 Family computer systems were advertised for targeted a specific age-group and a subject area
both education and entertainment (The Learning Company n.d.). The company
aimed to interest parents in improving their chil-
computer systems that could be used both for dren’s skills and knowledge.
education and for entertainment. Ads often Broderbund (1980), on the other hand, started
showed parents playing together with their chil- as a commercial game developer (Prince of Per-
dren as a family. Video games were on their way sia, Myst). It moved into edutainment titles in
to becoming a fixture in children’s daily lives 1985 with the detective game Where in the
(Fig. 3). World is Carmen Sandiego. The company’s
Publishers and game developers noted the approach to edutainment was story rather than
success of education-related titles and began to arcade-skill based: in Carmen Sandiego children
intentionally combine entertainment games and were set the task of finding Carmen and travelled
education software to create edutainment, soft- around the world asking questions that would lead
ware that could be marketed as both entertainment them to the thief. This narrative approach had
and education. The Learning Company and more in common with text adventures than with
Broderbund were two companies that began to arcade action. After the initial success with
create and promote entertainment-based educa- Carmen, Broderbund founded Living Books, a
tional games from different perspectives that still series of interactive CD-ROM storybooks that
prevail. The Learning Company was started as an engaged children through click and point anima-
educational software developer. Its first commer- tions. It brought well-loved print stories – Mercer
cially developed edutainment title, Reader Rabbit Mayer’s Just Grandma and Me (1992), Marc
(1983), designed for elementary school children Brown’s Arthur’s Teacher Trouble (1992), Stan
Children’s Games, from Turtle to Squirtle 299

Children’s Games, from Turtle to Squirtle, Fig. 5 Lil Critter helps his sister with her math, Arthur worries about his
spelling, Dr. Seuss’s ABC’s

and Jan Berenstain’s The Berenstain Bears Get in questions of game characters, unlocking secrets,
a Fight (1995), Dr. Seuss’s ABC (1995) – to an and overcoming obstacles to reach a goal.
audience who enthused about the new game-like As Spacewar! (1962) gave rise to all later action
interactivity in these stories. Game publishers games beginning with Pong, Adventure
soon adopted popular story characters to bring (1975) became the first text adventure game that
their appeal to education-related topics such as spawned all other text adventures. Popular in uni-
reading, writing, math, art, puzzles, and thinking versity and corporate intranets, Adventure was
(Fig. 5). initially created to be shared with children. Will
Crowther, the designer, writes “I decided I would
fool around and write a program that was a
Evolving Entertainment and re-creation in fantasy of my caving, and also
Edutainment for Computers and would be a game for the kids [his two daughters],
Consoles in the 1980s and perhaps some aspects of the Dungeons and
Dragons that I had been playing. . .. The kids
Entertainment games evolved along two main thought it was fun” (The origins of “Adventure,”
paths in the late 1970s – as arcade-action games Crowther n.d.).
and as text adventure games. In an arcade-action As the game market grew and as technology
game, although the game might consist of cause evolved, exploration in different game-like
and effect events, it did not require a story to be experiences for children increased. When Hyper-
successful. By the early 1980s, original Pong Card, an easy to use software programming tool
action had evolved and included: maze games – was introduced by Apple in 1987, Amanda
Pac-Man (1980), players needed to avoid four Goodenough used it to create Inigo Gets Out for
ghosts while eating up all the dots; platform younger children. This first graphical hypertext
games, called climbing games at the time – Don- lets children explore a space the cat Inigo inhabits
key Kong (1981), players jumped obstacles and and move the story forward by clicking not on the
enemies through four different levels; and simu- object of action (i.e., the cat), but on where the
lation games – Star Trek: Strategic Operations object needs to move. For instance, clicking at two
Simulator (1982), players had to defend the Star- birds will make Inigo jump at them. Published by
ship Enterprise from Klingon invaders. Voyager, the simple narrative opened the eyes of
Text-adventure games, on the other hand, were artists and designers, as well as writers, to the
based in story. Players achieved their goals not by possibilities of making their children’s stories
repeating arcade-type actions but by asking interactive (Fig. 6) (Madej 2007).
300 Children’s Games, from Turtle to Squirtle

While Goodenough, a writer, was creating with of their predicament, i.e., they could help Mary by
hypertext for young children, Roberta Williams, a bringing back her lost lamb (Fig. 7). When com-
game designer, was developing the first edutain- pleted successfully, the task was rewarded by a
ment graphical adventure game, Mixed-up Mother congratulatory animation/sound. In contrast to
Goose (1987), to entertain and educate her two Reader Rabbit published four years earlier, chil-
preschoolers. Williams had set up Sierra Online dren could visualize themselves in the story
with her husband Ken and they had created the through the avatar, could explore the space, and
10+ adventure series King’s Quest (1984). Based could interact with story characters. Other
in stories, many adapted from traditional fairy mechanics such as picking up the clue, dropping
tales or adventure tales, King’s Quest was the it off, and being rewarded by an animation were
first game to introduce third-person play; until based on existing types of actions. Together these
that time players played in the first person. The sets of click and point actions became character-
third-person position allowed the player an impor- istic of children’s games, and indeed of many
tant choice – to think about the character either as adult games.
her/himself, or as a separate character altogether In 1986, Walt Disney released The Black Caul-
(DeMaria and Wilson 2002). In Mixed-up Mother dron based on its animated film of the same name.
Goose young children play as one of eight char- The company made the game to be more child-
acters that searches for lost items throughout the friendly than other games of the time; designer
land and returns them to help story characters out Al Lowe simplified the game commands by
replacing the text parser with function keys to
make it easier for children to play the game, an
innovation not used again for a number of years.
Lowe followed the original story and provided
more decision-making choices by adding new
plot branches and six endings. Sierra’s expertise
in graphics made the game visuals more realistic
and appealing than the linear and stark graphics of
other games (Fig. 8) (Lee and Madej 2012).
Animated movies were a natural source for
Children’s Games, from Turtle to Squirtle,
children’s action-based games as their storylines
Fig. 6 Inigo Gets Out title screen already consisted of action that designers

Children’s Games, from Turtle to Squirtle, Fig. 7 Mixed-Up Mother Goose of the rhyme, Mary Had a Little Lamb
Children’s Games, from Turtle to Squirtle 301

Children’s Games, from Turtle to Squirtle, Fig. 8 Black Cauldron graphics (left) versus Dark Crystal graphics (right)

could mine. The puzzle game All Dogs Go to (1978) that had been designed to simulate real
Heaven (1989), the side-scrolling platform game world activities and provide players with a greater
Aladdin (1993), and the action-adventure Casper appreciation of the real life events. The success of
(1996) featured only action sequences rather than The Sims in 1990 and of Railroad Tycoon in 1993
the entire story. Live films were also used to encouraged the development of simulation games
inspire action adventure games suitable for chil- for children such as Harvest Moon (1997). In this
dren. As is the case with animated films, most game, children were responsible for allocating
games use the story plot only loosely to take their time to best maintain a farm that had fallen
players through action sequences. In The Karate in disrepair. They were provided daily tasks to
Kid (1987), for instance, the player uses karate complete, different environments to explore,
moves on four levels and ends with Master and, among other things, weather to contend
Miyagi congratulating him/her on learning the with. Simulation games such as this one were
moves needed to help Daniel successfully “learning in context,” and fit the genre, although
meet all the challenges (See a partial list of movies were not always labeled, edutainment.
made into video games at https://en.wikipedia. Such a learning simulation game that leads us
org/wiki/Category:Video_games_based_on_films, into the next topic, handhelds, is Nintendogs
Great video games based on movies n.d.). (2005). Handhelds are particularly suited to pro-
The first video game that featured animation as viding children a means to carry out set tasks in
polished as in animated movies was Disney Inter- real time. In Nintendogs, children take care of
active’s The Lion King Animated Story Book their pet dog on the Nintendo DS. Using the
(1995). Media Station, the developer, used break- touchscreen, they can train their dog, take it to
through technology and skillful editing to ensure the park, and wash and brush it. They can also
the original quality and continuity while affording record commands that the puppy should obey if it
interaction. Designed for children 3–8, the game has been trained correctly. A real-time simulation,
became popular with children as well as adults the action is based in the DS’s calendar and clock
who had enjoyed the movie. Other Interactive so, for instance, a puppy would grow hungry if not
Storybooks based on Disney animated films fed on time; the game provides feedback for act-
followed: Winnie the Pooh and the Honey Tree, ing responsibly toward a pet.
Pocahontas, The Hunchback of Notre Dame, and
Mulan, among others.
A game phenomenon that began in the 1980s Handhelds Prepare the Way
and grew to great popularity in the 1990s was the
simulation game. Simulation games had grown During the 1980s, handheld games gained tre-
out of educational games such as Oregon Trail mendous popularity. Small and easily
302 Children’s Games, from Turtle to Squirtle

transportable, they could be used anywhere from the popularity of its games rather than in any
the sofa to the car seat; playing video games no sophisticated technology. In comparison to its
longer kept children tied to computers at a desk or competitors, its green screen was blurry and
console – a large part of handhelds’ charm for graphics were unimpressive, yet it became the
both children and parents. Introduced at the begin- most successful video game system ever – hand-
ning of a new era, when parents trekked their kids held or otherwise – because of its strong stable of
from activity to activity in that other phenomenon notable games. It was released with what
of the time, the mini-van, handhelds became a part became the most popular game of all time, Tetris
of children’s everyday life and probably more (Melanson 2006), and then featured a wide spec-
than any other game technology were the deter- trum of game genres: puzzle games – Boxxle
minant of today’s generation of digital natives. (1990), adventure games – Who Framed Roger
Handhelds had their origin in the single game Rabbit (1991), historic action adventure games –
electronic devices of the 1970s, such as Auto Race Prince of Thieves (1991), sports games – Ultra
(1976) and Football (1977) made by the toy man- Golf (1992), fighting games –Street Fighter II
ufacturer Mattel. When Nintendo introduced (1995), pinball games – The Getaway: High
Game and Watch in 1988, a single game format Speed II (1995), racing games – Street Racer
with one small screen and two or four buttons, it (1996), and side-scrolling platform games – Toy
had similar simple game action that also chal- Story (1996).
lenged skills. The games were both educational Children enjoyed playing these games – par-
and entertaining: Flagman (1980) was a memory ents saw their children were not only occupied but
game in which the character on screen showed a were also learning new skills: a win-win situation
random number the player had to memorize and that helped establish playing handheld games as a
input into a series of squares. The format’s popu- go-to activity, one that translated readily to
larity was increased through games that featured playing games on mobile phones, when that tech-
well-known cartoon and video game characters nology became available.
such as Mickey Mouse (1981), shown in Fig. 9, The most recent iteration of mobile gaming
Popeye (1981), Snoopy (1982), Donkey Kong that continues children’s (and adults’) enthrall-
(1982), and Mario Bros (1982), who brought ment with this genre of game play is the Nintendo
their own background stories as context and back- Switch (2017), a hybrid console that can be played
drop for the games (RolyRetro 2016). on-the-go as a handheld or on a TV at home. It
The success of Nintendo’s next handheld sys- consists of a main unit with Joy-Con controllers
tem, Game Boy, introduced in 1989, is based on attached to each side that can be slid off so the

Children’s Games, from

Turtle to Squirtle,
Fig. 9 In Game and Watch,
Mickey runs to catch eggs
falling from one of four
chutes
Children’s Games, from Turtle to Squirtle 303

screen can be propped up on a table for one- or (MMORPGs), which, at the time, were consid-
two-player games. The Switch system also ered an adult genre. In 1996, Starwave Corpo-
includes a charging cradle so games can be played ration brought the genre to children 8+ with the
on a big-screen TV. The design allows for players game Castle Infinity. Entertainment video
to get the same basic experience regardless of how games had over the years stirred up considerable
they play (Thang 2018). controversy with addictive and violent game
play dominating the discussion; diverse opin- C
ions were held about the value of games for
The Internet Introduces Community to children. Multiplayer online games were
Children’s Games equally suspect. Castle Infinity promoted the
use of nonviolent ways of problem solving:
At the same time as handhelds were keeping chil- how to get rid of monsters who were threatening
dren enthralled in the world of desktop computers, the last of the dinosaurs in their home, the Cas-
the internet was changing into a user-friendly tle. Children used a password and a unique name
place that was opening up doors for gaming of a to join players around the world in coopera-
different nature. In 1994, Netscape Navigator tively saving the dinosaurs.
brought the graphical user interface to the general Into the fray of concern about online enter-
public and changed the face of the World Wide tainment The Disney Company launched an
Web both for adults and children. At the time, ambitious site for children, Disney’s Daily
edutainment CD-ROMs were at their height. As Blast (1997). The site offered parents a gated
children’s authors took advantage of the new tech- environment in which their children could con-
nology to create edutainment websites such as nect with each other safely. Attractively
Banph, Chateau Meddybemps, Fablevision, and designed to be child-friendly, the site offered a
Little Critter World-Wide Network (Fig. 10), changing range of stories, comics, arcade games,
based in their own work, the industry faltered and educational games for online play, either
and would never again be so financially success- alone or with chosen friends (Fig. 11). Multi-
ful. All of the activities CD-ROMS provided were player games included the musical activity
now available online, if perhaps not quite the Music Room Composer, in which children
same quality, for the cost of a service provider could compose and record music, share music
(Madej 2007). with friends who could comment on the compo-
Online access also popularized Massive sition, and play in jam sessions (Lee and Madej
Multiplayer Online Role-Playing Games 2012).

Children’s Games, from Turtle to Squirtle, Fig. 10 Chateau Meddybemps and Little Critter World-Wide Network
with games, puzzles, and stories
304 Children’s Games, from Turtle to Squirtle

Children’s Games, from Turtle to Squirtle, Fig. 11 Disney’s Daily Blast presented a child-friendly portal page to a
gated environment

By the 2000s, the major children’s publishers Minecraft, children 6+ decide for themselves
and media companies, from PBS Kids to National what to do and how to do it. They collect
Geographic Kids, realized the promotional oppor- materials available in the space and build what-
tunities of engaging children online, and created ever their mind can imagine. As online friends
sites that were based in children’s favorite charac- they can cooperate to build worlds together.
ters and stories such as Arthur, Caillou, Clifford,
Dragontales, and Sesame Street. These sites
offered learning opportunities through entertain- Changing Handheld/Mobile Landscapes
ment, although initially they did not offer a multi-
player environment. Mobile technologies brought the connectivity of
Today, many games are available in versions the internet to the portable handheld. Tablets
that can be downloaded from the Internet. The replaced desktops and laptop computers for chil-
sandbox game Minecraft (2010), for instance, dren in many homes. Their small size, portability,
which is now the second most popular game relatively low cost, and Wi-Fi connectability
after Tetris, can be purchased for single play or made them a practical device for playing favorite
can be played online with friends. A sandbox games, including online games, anywhere in the
game is an open-world game akin to playing home, or indeed, even away from home. The
with LEGO blocks and building objects or scenes, smartphone has quickly replaced even the conve-
only the number of blocks is limitless. In nience of the tablet for Wi-Fi access (Fig. 12)
Children’s Games, from Turtle to Squirtle 305

Internet Usage Worldwide

October 2009 - October 2016
Desktop Mobile & Tablet

100%

90%

80%
Desktop C
48.7%
70%

60%

50%

40%

30%
Mobile & Tablet
20%
51.3%
10%

0%
Oct 2009 Oct 2010 Oct 2011 Oct 2012 Oct 2013 Oct 2014 Oct 2015 Oct 2016

Children’s Games, from Turtle to Squirtle, Fig. 12 Up from 2016, shown above, 2017 stats indicate mobile use
continues to increase. Mobile – 52.29%, Desktop – 43.29%, Tablet – 4.42%

(Mobile and Tablet Internet Usage 2016). It is learning to speak Spanish by saving trapped
useful to parents, engaging for their children, toys, they have access to hundreds of applications
and is ever present. Easy availability and mobility, through dozens of websites specifically for learn-
together with the fact that, like a handheld, a ing or identified as just-for-fun (during which
smartphone fits a child’s hands well and has a learning does go on). An example of a popular
responsive touch screen (tablets have this as (most downloaded game of all times) game is
well), makes it a most advantageous digital enter- Angry Birds (2009). A puzzle, turn-based game
tainment device for on-the-go parents who want to for ages 8+ in which cute birds aim to retrieve
occupy their children. In addition, game devel- their eggs from some greedy pigs, it has no new
opers have been assiduous in fulfilling parent’s types of interactivity, but does have engaging
requirements for educational games and chil- characters in fun and wacky side-scrolling
dren’s need for entertainment games both for gameplay (Cheshire 2011). Angry Birds has
IOS and android tablets. translated well into cross-media activities such
Mobile games for children, while limited by as children’s library programs and use in early
the speed of the technology and the size of a learning environments such as preschools. Cut
device’s memory, were initially not much differ- the Rope (2010), in which players must feed
ent from what children had available to them on a a little green creature, Om Nom, with candy they
desktop or laptop computer. A traditional edutain- retrieve from hanging ropes, requires more chal-
ment game for mobile devices, Brain PoP: lenging planning and dexterity in use of the touch
Today’s Featured Movies, for instance, uses inter- screen to cut the rope. It is also popular across
net connectivity to present a new set of movies media so that children have opportunities to trans-
each week about topics from math to social sci- fer their mobile knowledge to physical play
ence. Children choose their topic and accompany- (Fig. 13).
ing movie, watch, and then answer a pop quiz. Mobile devices have become very popular
Whether either at the behest of their parents or on with parents for very young children as well.
their own, children are learning the alphabet, Designers have put such technology as the touch
learning coding through puzzles and games, or screen to great advantage in games for them. In
306 Children’s Games, from Turtle to Squirtle

Children’s Games, from Turtle to Squirtle, this as for many other games, has provided players with
Fig. 13 Playing Cut the Rope on a smartphone requires real world artifacts for interacting with Om Nom
planning ahead and dexterity. Cross media marketing, for

Old Macdonald (2009), for instance, young chil-

dren can use their fingers to push a tractor, open
barn doors, jump with frogs, pop balloons, and
swish their way through the rhyme as the tune
plays to the animations. Together with parents or
alone, children can also read, listen, and record
themselves singing. This engages them not only
cognitively but also physically in their play activ-
ity, creating stronger connections in their learning
(Madej 2016).

Tangible Games

The most recent developments in tablet and

Children’s Games, from Turtle to Squirtle,
smartphone games, virtual reality (VR), aug- Fig. 14 The Coleco Telstar Arcade (1976) with its racing
mented reality (AR), and mixed reality (MR), wheel and pistol
are at the end of a trajectory of evolution in action
video games that, during the mid-1970s, took the reach new levels of ability. Coleco’s Telstar
form of accessories such as racing wheels, and Arcade (1976) (Fig. 14) exemplifies early interest
today, has players moving about the neighbor- in creating a simulated experience for home video
hood, the city, or even the world, searching and games. The three-sided Telstar featured two sides
catching virtual Pokémon. While manifested in that provided for a “real experience.” Players con-
different ways, the main purpose of action games trolled a racing wheel while playing racing car
is to provide tangible or embodied interaction that games on one side, or drew and fired a pistol for
engages the player physically and simulates real target shooting on another side. On the third side,
world activity. This could be by providing haptic players used typical game buttons to control pong
feedback, physically engaging interaction, an games such as tennis (Madej 2007). Racing
immersive environment, or by augmenting an wheels continue to be used and are still bundled
existing environment. with racing games for systems such as Mario Kart
Among the first live-action games were racing Wii (2008).
games and marksman games. Although entertain- New types of interactive artifacts that engaged
ment oriented, many of these games required chil- children physically were introduced in the late
dren to improve skills to acquire higher scores or 1990s and early 2000s. These included play
Children’s Games, from Turtle to Squirtle 307

mats on which children could enjoy dancing to the Wii games were promoted for children as
music video game Dance Dance Revolution young as toddlers and games like Kirby’s
(1998), konga drums they could beat in Donkey Return to Dreamland were positively viewed as
Konga (2003), and guitar controllers they could family friendly because of their multiage, cooper-
strum in Guitar Hero (2005). Each of these inter- ative play. But toddlers and preschoolers found
faces required improving skills to reach the next the controls difficult, both in this and other
level of proficiency and provided for learning games such as Reader Rabbit Kindergarten in C
while playing. which any precision was required (Healy 2011).
While mixed reality appears to be a very recent Issues also arose when Microsoft introduced
phenomenon in children’s video games, it was the next motion capture device to come on the
first seen in Sony’s EyeToy:Play in 2003. Sony market, the Kinect (2010). Older children could
used an inexpensive webcam to literally put the engage in simulated action in games but problems
player into the game through motion capture existed with recognition of younger children
video. Players could dance, Kung Fu, wash win- because of their height, as well as with light
dows, play soccer, or box against themselves. levels, the amount of space required to play,
While EyeToy caught the imagination, it was dif- and recognition of movements (Kinect Sucks for
ficult to get the actions right to appear on the Little Kids 2012).
screen. This difficulty caused frustration. It Mixed reality games entered a new era when
wasn’t until the Wii console system, which they began to include either or both VR and
detected the player’s movement in three dimen- AR in their mix. Virtual glasses such as Google
sions, was introduced by Nintendo in 2006 that Cardboard, a simple, inexpensive version of a
motion capture gameplay became seamless and VR headset, became an asset in the development
enjoyable rather than frustrating. Both Wii arcade of immersive adventures for children. Edutain-
games and its narrative games used the technology ment benefitted in particular in games such
to advantage. In the medical drama, Trauma Cen- as Jurassic Virtual Reality (2014), in which
ter: Second Opinion, the player, as Dr. Derek Stiles, children go back to the time of the dinosaurs
sets broken bones, cleans and stiches wounds, and and can observe the creatures in their natural
simulates the use of a defibrillator during an emer- habitat from any angle. The ability to engage in
gency situation: he has to “shove the two controllers a very personal way within an environment
forward to shock [the] patient[s]” (Fig. 15) (Trauma makes learning immediate and takes edutain-
Center 2006). ment to a new height.

Children’s Games, from Turtle to Squirtle, Fig. 15 Wii Trauma Center requires players to simulate real-world action
308 Children’s Games, from Turtle to Squirtle

Children’s Games, from Turtle to Squirtle, Fig. 16 (a) colAR/Quiver character, (b & c) Toy Car RC’s Western World
brought to life through AR

In September 2013, colAR Mix, now called learning for children who search for
Quiver, was announced as an AR coloring expe- Charmander, Bulbasaur, or Squirtle (Fig. 17).
rience for children. This application let children While Rolling Stone says Pokémon GO is “a
bring characters they colored on pages printed free-to-play, location-based, augmented reality,
from the app to life in 3d when they are viewed multiplayer online mobile game that also sup-
through a tablet or mobile phone and added an ports its own custom wearable tech” (Davison
engaging new experience to traditional coloring 2016), more humanistically speaking, and more
(Fig. 16a). Each new MR application seems to to the point, through its enthusiastic embrace of
raise the bar for engagement: Toy Car RC engagement with the real world Pokémon GO
(2014) for children 6–8, follows the adventures shows how technology, story, and environment
of a small car named Wheely that is always can be joined effortlessly to engage us commu-
searching for an adventure. Once children print nally in an entertaining game that engages chil-
out and position target pages in a space, they set dren in learning.
Wheely on his journey and connect the real world
with a virtual world, either Candy Land or West-
ern World (Fig. 16b, c). Finally
The AR game Pokémon GO (July 2016)
brings us to the end of the trajectory of action In 2017, 60 years passed since the LOGO Turtle
games, at least for our present time frame. It is was first introduced to the world. Seymour
also where, for the moment, this brief history of Papert’s interest during the 1960s in creating a
games ends. Pokémon GO has taken what is the playful environment with computer technology
most used platform for games, the smartphone to benefit children’s learning continues to be
(which we carry with us everywhere), paired it pursued with vigor by the games industry
with well-loved and storied characters (which today. Through an unflagging continuance of
few, if any, of us have not heard of), and simply effort, and whether developing games solely
asks of players that they look around their natural either for learning or play, or developing games
environment to find and collect Pokémon and intended to do both, the different streams that
then share these collections with others. have evolved are exploring technologies as
Pokémon GO is currently being used for class- quickly as they are emerging and taking advantage
room assignments: children “Keep a log of where of new types of engagement to offer a mix of
they go, what they see, and what they are learn- learning experiences. While some may not see
ing as they play, including historic places and searching for Squirtle as beneficial to learning,
points of interest” (Gracey 2017). But a class- others have taken the idea, and exploited its poten-
room situation isn’t needed to spark interest in tial to engage a generation of children, many of
Children’s Games, from Turtle to Squirtle 309

Children’s Games, from Turtle to Squirtle, Fig. 17 Pokémon GO: Searching for Squirtle

which, have never been without a digital gadget in Gracey, L.: Pokemon GO: what education should be.
their hands. Retrieved at https://www.tcea.org/blog/pokemon-go/
(2017)
Great video games based on movies. Retrieved at http://
www.retrojunk.com/article/show/4718/great-video-
References games-based-on-movies (n.d.)
Healy, C.: Reader Rabbit Kindergarten (Wii) Game review.
Atari Program Exchange Catalog (Fall 1983). Retrieved at Retrieved at www.commonsensemedia.org/game-
https://archive.org/stream/Atari_Program_Exchange_ reviews/reader-rabbit-kindergarten-wii (2011)
catalog_Fall_1983#page/n33/mode/2up (1983) Jancer, M.: How you wound up playing the Oregon Trail in
Blikstein, P.: Seymour Papert’s legacy: thinking about learn- computer class. Retrieved at. https://www.smithsonia
ing, and learning about thinking. Retrieved at https://tltl. nmag.com/innovation/how-you-wound-playing-em-ore
stanford.edu/content/seymour-papert-s-legacy-thinking- gon-trailem-computer-class-180959851/ (2016)
about-learning-and-learning-about-thinking (n.d.) Kinect Sucks for little kids. . .. Retrieved at http://forum.
Cheshire, T.: In depth: how Rovio made Angry Birds a notebookreview.com/threads/kinect-sucks-for-little-kids.
winner (and what’s next). Retrieved at http://www. 681434/ (2012)
wired.co.uk/article/how-rovio-made-angry-birds-a-win Lee, N., Madej, K.: Disney Stories. Springer, New York
ner (2011) (2012)
Children’s mobile game industry revenue worldwide from Lofgren, K.: 2016 Video game statistics & trends who’s
2015 to 2017. Retrieved at https://www.statista.com/ playing what & why. Retrieved at http://www.
statistics/506130/children-mobile-game-revenue-global/ bigfishgames.com/blog/2016-video-game-statistics-and-
(n.d.) trends/ (2016)
Crowther, W.: The Crowther and Woods ‘Colossal Cave Madej, K.: Characteristics of Early Narrative Experience.
Adventure’ game. Here’s where it all began. . .. Retrieved Ph.D. Thesis. SFU Institutional Repository. Summit.
at http://rickadams.org/adventure/a_history.html (n.d.) sfu.ca/system/files/iritems1/8448/etd3328.pdf (2007)
Davison, J.: WTF Is ‘Pokémon Go,’ explained. Retrieved Madej, K.: Physical Play and Children’s Digital Games.
at http://www.rollingstone.com/culture/features/wtf-is- Springer, New York (2016)
pokemon-go-explained-20160711 (2016) Melanson, D.: A brief history of handheld video games.
DeMaria, R., Wilson, J.: High Score: The Illustrated His- Retrieved at https://www.engadget.com/2006/03/03/a-
tory of Electronic Games. McGraw-Hill/Osborne, brief-history-of-handheld-video-games/ (2006)
Berkley (2002) Mobile and tablet internet usage exceeds desktop for first
Donkey Kong 3: classic arcade game video, history & time worldwide. Retrieved at http://gs.statcounter.com/
game play. Retrieved at https://arcadeclassics.net/80s- press/mobile-and-tablet-internet-usage-exceeds-desktop-
game-videos/donkey-kong-3/ (1981) for-first-time-worldwide (2016)
Fuscaldo, D.: Global video game sales to reach $91B in Pac-Man: classic arcade game video, history & game play
2016. Retrieved at http://www.investopedia.com/news/ overview. Retrieved at https://arcadeclassics.net/80s-
global-video-game-sales-reach-91b-2016/ (2016) game-videos/pac-man/ (1980)
310 Choices in Games

Papert, S.A., Watt, D., diSessa, A., Weir, S.: Final report of
the Brookline LOGO Project. Part II: project summary Classical Learning Method in
and data. Retrieved at https://dspace.mit.edu/handle/
1721.1/6323 (1979) Digital Games
Reader Rabbit and the Fabulous Word Factory. Retrieved
at https://archive.org/details/ReaderRabbit134amCrack Youichiro Miyake
(n.d.) Square Enix Co., Ltd., Tokyo, Japan
Rideout, V.: Zero to eight, children’s media use in America
2013. Retrieved at https://www.commonsensemedia.
org/research/zero-to-eight-childrens-media-use-in-
america (2013) Synonyms
RolyRetro.: A guide to the Nintendo Game & Watch hand-
held games – classic 80’s retro. Retrieved at https://
levelskip.com/classic/A-guide-to-the-Nintendo-Game- Genetic algorithm; Machine learning; Neural
Watch-handheld-games-of-the-80s (2016) networks
Star Trek: strategic operations simulator. Retrieved at
http://segaretro.org/Star_Trek:_Strategic_Operations_
Simulator (1982)
Thang, J.: Should you make the Switch? Retrieved at Definition
https://www.gamespot.com/articles/nintendo-switch-
review/1100-6448303/ (2018) Classical learning includes simple neural net-
The Learning Company: A list of series can be retrieved at works, genetic algorithms, and so on. Few game
http://gaming.wikia.com/wiki/The_Learning_Company
(n.d.) titles utilize classical learning because these must
Trauma Center: second opinion review. Retrieved at http:// be harmonized with game design. Thus, the main
ca.ign.com/articles/2006/11/14/trauma-center-second- problem of implementing the classical learning
opinion-review (2006) method in digital games is finding out how to
Winter, D.: Magnavox Odyssey, First home video game
console. Retrieved at http://www.pong-story.com/odys synergize a classical learning algorithm and
sey.htm (2013) game design.
Wong, K.: The forgotten history of ‘The Oregon Trail,’ as
told by its creators. Retrieved at https://motherboard.
vice.com/en_us/article/qkx8vw/the-forgotten-history-of-
the-oregon-trail-as-told-by-its-creators (2017) Introduction

Learning has not always been used in digital

games; rather, it is used only in specific and
Choices in Games limited contexts. There are two reasons for
why the application of learning is limited.
▶ Domain-Specific Choices Affecting Design First, the game is usually developed on a game
Effort in Gamification design, and it is adjusted to be strict good bal-
ance by a game designer’s sense. Learning AI
method introduces unpredictability and varia-
tions. It is considered difficult to combine learn-
Circles Space ing algorithms with game design. To address
this, advanced game design technologies that
▶ Theory of Minkowski-Lorentz Spaces absorb the fluctuations of learning algorithms
and engineering technology to safely control
learning are both needed. Second, learning
requires CPU and memory resources, which
Citizen Science sometimes puts heavy load on graphics and
other game characteristics. However, the latter
▶ Games in Science is gradually being addressed by improvements
Classical Learning Method in Digital Games 311

to the computing performance of the game and areas within a game. “Black & White” (Lionhead
development machines. Accordingly, learning Studios), which was released in 2001, is a game
algorithms are being incorporated gradually in which the user leads the inhabitants and make
into games and game design. them thrive. Simple neural networks (perceptron,
In the 1980s and mid-1990s, when artificial three for input, one for output) are set for the
intelligence (AI) technology was used, it almost creatures, which learn depending on the disci-
completely exhausted the memory and CPU pline enforced by the user (Evans 2002). In this C
resources available for the game. Thus, these manner, they learn what actions to take
were marketed as “AI Games.” Some examples depending on the motivation. In the “Forza
include enjoying a conversation with a virtual Motorsport” (Turn 10 Studios, 2005) series, a
woman in “Emmy II” (ASCII, 1984), training a mechanism called Drivatar was introduced to
puppy in “Puppy Love” (Addison Wesley Pub- learn the user’s driving technique (Microsoft
lishing), simulating the life of a family in “Little Research 2014). This was a type of “machine
Computer People” (Activision, 1985), and simu- learning” that automatically learned deviations
lating the life of a girl in “Appletown Story” from an ideal racing line set for each course; the
(Square, 1987). learnings were used as a parameter of “steering
Towards the latter half of the 1990s, the per- handling.” Moreover, as an example of related
formance of game machines improved reason- research, at Microsoft Research, (2004) the AI in
ably. Accordingly, AI algorithms such as the fighting game “Tao Feng” (Studio Gigante,
“Genetic Algorithms (GA),” and “Neural Net- 2003) was successfully improved by using the
works (NN)” were gradually incorporated into Q-Learning method, which is a type of reinforce-
games. Large-scale neural networks on the ment learning realized through interactions with
order of a few thousand nodes were utilized to players (Herbrich et al. 2008; Graepel
allow players to teach creatures “Creatures” et al. 2004).
(Millennium Interactive, 1996), where the char- With respect to targeting (selecting enemies),
acters in the game were taught to use objects among the AI techniques, one example of the use
within the game. Moreover, in the PlayStation, of neural networks in the 2010s, is “Supreme
AI game masterpieces by Mr. Yukihito Commander 2” (Gas powered Games, 2010)
Morikawa of muumuu continued to be released (Robbins 2013). It learns which enemy to target
(Morikawa 1999). “Ganbare Morikawakun around the character, from approximately 60 min
2gou” (English title: Pet in TV) (muumuu, of play by the developer, by using back-
1997) was a game for imparting intelligence to propagation. Moreover, in “Killer Instinct”
“Morikawakun 2gou,” where the intelligence in (Rare Ltd., Iron Galaxy Studio, 2014), the case-
Morikawakun 2gou was inculcated through the based reasoning method is adopted, where AI
learning of instructions from players utilizing a play is set up by using play data from users
back-propagation neural network. Moreover, in (Hayles 2015). Moreover, although its develop-
“Astronōka” (muumuu, 1998), the performance ment was cancelled, in “Fable Legends,” Monte
of the enemy characters was designed to evolve Carlo Tree Search (MTCS) was used to locate the
according to the way the players play by utilizing positions of enemy groups (Mountain 2015)
a GA. In the 1990s, the reverberations of the (Fig. 1).
second AI boom were late in infiltrating the Thus far, some examples of the application of
game industry. evolutionary learning algorithms have been
In the 2000s, the number of “AI games” grad- described; however, the reality is that, until
ually reduced with the rapid development of 3D 2018, the number of “cases of clear application
graphics. There was a gap of several years before to a game” was very small. Although the applica-
learning started to be incorporated into specific tion to games is described in the next section,
312 Classical Learning Method in Digital Games

Classical Learning Method in Digital Games, Fig. 1 Examples of the application of learning and evolutionary
algorithms to games (main unit)

there is a rapidly increasing trend. Even though

the performance of game machines continues to
increase, the problem regarding where this should
be incorporated in the game design still remains a
formidable challenge. We discuss each of the
cases.

Neural Network in “Creatures”

Because “Creatures” (Millennium Interactive Ltd,

1996) is a game for personal computers that was
released in the 1990s, it is not well known in
Japan; however, it is well known internationally,
and has been highly evaluated, winning many
awards. It is a game in which creatures called
“Norn” are raised, and in particular, they can
Classical Learning Method in Digital Games,
learn language as well as the names of the objects Fig. 2 Unit of Norn brain “Robe”
and actions inside the game. Once learning is
complete, simply by providing a word, they can
be made to perform the corresponding specific
action. Norn, upon which the attention lobe responds.
At that time, the brain of a Norn consisted of a The Norn then says the name of the target object
neural network comprising a few thousand nodes. (noun) or the action (verb) corresponding to the
A set of neurons is called a “lobe,” and neurons in behavior. The user by using a mouse gently
each individual lobe are interconnected with neu- caresses the Norn when a correct answer is
rons in other lobes (Grand and Cliff 1997) provided or slaps it when the answer is incor-
(Fig. 2). rect. In this manner, the neurons in the neural
The intelligence of Norn is based on the network learn the nouns and verbs used in the
agent architecture. The sensory lobe senses game. The users can also enter the words. The
external signals. For example, when a ball is neural network learns by matching the target
seen, the neurons responding to the ball become with the word, or the action with the word (see
active. The user shows various objects to the Fig. 3).
Classical Learning Method in Digital Games 313

Classical Learning Method in Digital Games, Fig. 3 Schematic of lobe (neural network) in “Creatures” (Grand et al.
1996)

Learning and Evolutionary Algorithms in Learning algorithms were incorporated into a

Japanese Game Scenes in the 1990s series of AI games created by Morikawa:
Back-propagation neural network in “Ganbare
The second AI boom is said to have occurred Morikawakun 2gou” (SIE, 1997)
during 1984–1994. Accordingly, its effect was Genetic algorithm in “Astronōka” (SQUARE
still observed in the latter half of the 1990s. In ENIX, 1998)
Back-propagation neural network in
the gaming industry, in the late 1990s, with a “Kokohore! Pukka” (English title: Dig a-Dig
slight delay, various games using AI were Pukka) (SIE, 2000)
released. “Creatures” was one such game. In the Autonomous lyric generation in “Kumauta”
Japanese gaming industry, two games using AI (SIE, 2003)
were released in the late 1990s, “Astronōka” and In “Ganbare Morikawakun 2gou,” specifying a
“Seaman” (SEGA, 1999). Yukihito Morikwa behavior (action) corresponding to an object dis-
developed the former, and Yutaka Saito developed covered by a character, enables it to undertake
the latter game. The characteristic of AI used in appropriate behavior corresponding to the sense
the games during this time was that AI attributed of perception of the character about the target
to a large proportion of the total content since a object. In “Kokohore! Bukka,” the player teaches
large portion of the resources needed to be allo- the agent the type of stones to be dug out through a
cated to AI. This implies that other elements such reward system, the agent gradually starts judging
as game design and CG were closely intertwined the stones by itself. In “Kumauta,” the agent
with AI. In such games, a game creator was autonomously creates lyrics corresponding to the
required to design the game consistently, from lyrics selected by the user, which the white bear
the game design to the technology adopted. Both then sings in the enka style.
Morikawa and Saito undertook such a design pro- A short introduction to “Astronōka,” where a
cess to integrate the game design with AI. genetic algorithm is used, is given below.
314 Classical Learning Method in Digital Games

Genetic Algorithm in “Astronōka” Each baboo challenges the traps, and the result
is given as a score; baboos with higher scores are
“Astronōka” uses a genetic algorithm to help selected so that they become parents with high
the enemy characters evolve. The players culti- probability based on their superior adaptability
vate vegetables and set “traps” in the field to and pass their genes to the next generation. Their
exterminate pest characters called “baboo” that mutation rate is set to 3% (Fig. 4).
harm the vegetables. The “traps” are regions While on the surface it appears that among the
divided into a grid-like mesh, where players 20 baboo characters in the game, each character is
can place fans, scarecrows, or pitfalls to trap challenging the trap only once, in the background,
the pests. simulations are carried out for all the 20 baboo
In the latter half of the 1990s, the media characters evolving over five generations; this
reported on the vicious cycle of “the spraying of mechanism can only be implemented in a game.
pesticides, followed by the increase in the pests’ This is done to make the users believe in the
resistance to the pesticides, and further spraying evolution of the enemy. If the rate of evolution is
of the pesticides” in Yumenoshima in the Tokyo low, more generations are added, and conversely,
Bay. This game was inspired by this incident. That if the rate of evolution is high, the number of
is, in the game, the stronger the “trap” set by the generational changes are reduced.
users, the more the “baboo” evolves.
Each character has a parameter column that
determines the performance of the character. The Study on Machine Learning at Microsoft
parameter column comprises 56 kinds of items, Research
such as physical fitness, endurance, arm strength,
leg strength, and resistance to various traps In the game industry, the third AI boom took place
(Morikawa 1999). after 2015, although research on the application of

Classical Learning Method in Digital Games, Fig. 4 Genetic algorithm mechanism in “Astronōka”
Classical Learning Method in Digital Games 315

ML in games has proliferated, Microsoft such engagements will become a “fleeing” char-
Research, from as early as 2004, was engaged in acter. This way, a character can be taught
research on ML in digital games. The success of through reinforcement learning, and the orienta-
this research is exemplified in “Forza tion of such learning is determined by the
Motorsports” and various papers. As mentioned rewards. In particular, Q-learning, which is an
earlier, before 2004, a group in Microsoft algorithm for reinforcement learning, was used
Research studied the fighting game “Tao Feng” in this study. Q-learning is suitable for continu- C
(Microsoft, Studio Gigante, 2003). They studied ous reinforcement learning with temporal tran-
the reinforcement learning of characters (Herbrich sitions, and it is frequently applied to learning in
et al. 2008; Graepel et al. 2004). Reinforcement digital games.
learning is an algorithm to learn from experience.
Instead of any clear teaching signals, rewards are Machine Learning in Forza Motorsports Series
set with respect to the orientation of learning that In Forza Motorsports, a player called “Drivatar”
enable the agent to learn actions from the sur- has the ability to learn driving skills and generate
rounding environment in accordance with the ghosts to drive. First, the course in Forza
rewards. Two types of rewards have been used Motorsports is divided into segments. The seg-
in fighting games. The first is rewarded when an ments can be straight or hairpin-shaped. An ideal
attack hits the opposing character (reducing the course is set for each such segment. Then, the
opponent’s physical strength). In such a reward deviations from these specified courses are mea-
system, reinforcement is realized through repeat- sured (Herbrich et al. 2008) (Fig. 5).
edly learning the skills to use in a particular situ- Next, to reproduce the deviations as a ghost,
ation. As a consequence, a character that has they are reverse transformed and reflected in the
undergone reinforced learning through tens of controller operation to reproduce the previously
such engagements will become a “strong” charac- measured deviations. Instead of forcing the car to
ter. Conversely, the reward is provided when the follow the externally specified course, it is
character is able to skillfully dodge an attack from reduced to a reverse problem, where the car
the enemy. Consequently, the character after attempts to reproduce the course “just as the user
undergoing reinforced learning through tens of had deviated.”

Classical Learning Method in Digital Games, Fig. 5 Reinforcement learning algorithm (Herbrich et al. 2008)
316 Classical Learning Method in Digital Games

Case-Based Reasoning in Killer Instinct Monte Carlo Tree Search in TOTAL WAR
“Case-based reasoning” (CBR) is an algorithm for
obtaining a solution to the problem under consid- Monte Carlo Tree Search (MCTS) is a revolu-
eration from the recollections of similar cases in tionary and powerful method that was invented
the past. Although a few studies on the application for the computer Go program. Simulation is the
of CBR to digital games have been reported, very primary approach in this method. Further, as it
few actually adopted them (Aha et al. 2005). requires no evaluation, it is very useful in game
“Killer Instinct” (Rare Ltd., Microsoft Corp., development. Hence, many related studies have
1994–2016), created “Shadow” that grows been conducted on its application to computer
through learning the fighting and style of players games (Fig. 7).
from past data (Hayles 2015). By accumulating When there are multiple candidate moves, the
game logs and using the concept of abstract dis- MCTS algorithm performs random simulation of
tance, multi-dimensional and large-dimensional all possible moves of the game beyond that point
data are vectorized into “Game State,” and the and selects one after evaluating the win–loss sta-
most effective action corresponding to each such tus of all such moves. Moreover, the number of
state is extracted. When the game is played, the simulations is increased for promising moves, to
Shadow searches such data for the action most generate an index called upper confidence bound
appropriate for a specific situation (Fig. 6). (UCB), which is the cornerstone of this algorithm.

Classical Learning Method in Digital Games, Fig. 6 Case-based learning principle

Classical Learning Method in Digital Games 317

Classical Learning Method in Digital Games, Fig. 7 Monte Carlo Tree Search Method

MCTS has superior applicability to simu- Resource assignment MCTS-based decision

lation games and has been used for adjust- planner determines the sequence of
ments in the campaign mode (single-user actions
mode including tutorials) in “TOTAL WAR:
ROME II” (Creative Assembly, 2013) and In this manner, once the AI playing “TOTAL
“Total War: Attila” (Creative Assembly, WAR” is developed, it can be made to play many
2015) (Champandard 2014). It enables the games very rapidly by issuing thousands of com-
measurement of the effectiveness of AI mands instantly. Accordingly, it enables the veri-
strategy by repeating the playouts until fication of game balance at a very high speed
the end. without human intervention.
There are three levels in the decision-making Moreover, in “Fable Legends” (Lionhead Stu-
by AI when playing “TOTAL WAR”: dios, was not released), MCTS was used to
decide the advance of multiple characters
Task creation Set a number of high-level tasks (Mountain 2015). Assuming each position as a
(these do not use MCTS) node, a tree structure that comprised the choices
Resource allocation Limited resources are allo- of actions at the respective nodes was considered
cated to tasks (use MCTS) (Fig. 8).
318 Clicker Game

Classical Learning
Method in Digital
Games, Fig. 8 Monte
Carlo Tree Search in action
games

Cross-References ac.kr/courses/CourseSyntheticCharacter/
grand96creatures.pdf
Hayles, B.: Case-based Reasoning for Player Behavior
▶ Navigation Artificial Intelligence Cloning in Killer Instinct, nucl.ai Conference 2015.
https://archives.nucl.ai/recording/case-based-
References reasoning-for-player-behavior-cloning-in-killer-
instinct/
Aha, D., Molineaux, M., Ponsen, M.: Learning to Win: Herbrich, R., Graepel, T., Quiñonero Candela, J., Halo,
Case-Based Plan Selection in a Real-Time Strategy Xbox Live: The Magic of Research in Microsoft Prod-
Game, ICCBR 2005: Case-Based Reasoning Research ucts, Microsoft Research (2008). http://research.micro
and Development pp 5–20 soft.com/en-us/projects/drivatar/ukstudentday.pptx
Astronoka, SQUARE ENIX, 1998 Kokohore! Pukka, Sony Interactive Entertainment, 2000
Champandard, A.J.: Monte-Carlo Tree Search in TOTAL Kumauta, Sony Interactive Entertainment, 2003
WAR: ROME II’s Campaign AI, AiGameDev. (2014). Microsoft Research:Drivatar™ in Forza Motorsport, 2014.
http://aigamedev.com/open/coverage/mcts-rome-ii/ http://research.microsoft.com/en-us/projects/drivatar/
Evans, R.: Varieties of Learning, AI Wisdom, 2002. Vol.1, forza.aspx
11.2, pp. 567–578. Jeffrey Schlimmer, Drivatar and Morikawa, Y.: Use of artificial intelligence for video game.
Machine Learning Racing Skills in the Forza Series, J. Jpn. Soc. Artif. Intell. 14(2), 214–218 (1999)
nucl.ai Conference 2015 https://archives.nucl.ai/record Mountain, G: Tactical Planning and Real-time MCTS in
ing/drivatar-and-machine-learning-racing-skills-in- Fable Legends, nucl.ai Conference 2015. https://
the-forza-series/ archives.nucl.ai/recording/tactical-planning-and-real-
Ganbare Morikawakun 2gou, Sony Interactive Entertain- time-mcts-in-fable-legends/
ment, 1997 Robbins, M.: Using Neural Networks to Control Agent
Graepel, T., Herbrich, R., Gold, J.: Learning to Fight, Proceed- Threat Response. In: Game AI Pro, p. 391–399. 2013.
ings of the International Conference on Computer Games: Chapter 30
Artificial Intelligence, Design and Education (2004) Seaman, SEGA, 1999
Grand, S., Cliff, D.: Creatures: entertainment software
agents with artificial life. Auton. Agent. Multi-Agent
Syst. 1, 39–57 (1997)
Grand, S., Cliff, D., Anil, M.: Creatures: Artificial life
autonomous software agents for home entertainment.
Clicker Game
Millennium technical report 9601; University of Sus-
sex Technical Report CSRP434, 1996. http://mrl.snu. ▶ Incremental Games
Client/Server Gaming Architectures 319

et al. 2014). In this case, a single copy of the

Client/Server Gaming game state is maintained at the server-side. Each
Architectures event generated by a CE is sent to the single GSSE
that processes the event computing a new game
Stefano Ferretti and Gabriele D’Angelo state; then, the GSSE forwards the newly com-
Department of Computer Science and puted game state to all other CEs. During the
Engineering, University of Bologna, game, the GSSE controls the generated events’ C
Bologna, Italy validity.
In this scenario, clients are very simple since
they have only to take inputs from the players
Synonyms and render the output state. Consistency is easy
to maintain as well as cheating avoidance,
Centralized architectures because only a single GSSE maintains the
game state, and thus a single node is in charge
of determining the game state advancements.
Definitions For this reason, provided that the GSSE is a
trusted entity, illegal manipulations of the game
Client/server gaming architecture refers to a typi- state are extremely difficult/easily detected
cal distributed architecture for the support of (Ferretti 2008; Mauve et al. 2002). Often, these
networked games. In this architecture, a single advantages push game middleware providers to
node plays the role of the server, i.e., it maintains adopt this architectural solution (Bauer
the game state and communicates with all other et al. 2004).
nodes (the clients). The server notifies game However, the major drawback of such an
moves generated by players and computes the approach is that a centralized server could be
game state updates. the bottleneck of the system (Briceño et al.
2014); moreover, the GSSE is a single point of
failure. Furthermore, this architecture is not
Online Gaming Architectures scalable with the number of players (Reis
Cecin et al. 2004). Finally, users that experience
An online gaming architecture is typically com- different network delays, due to their (possibly)
posed of two types of entities: client entities (CEs) different types of connections, are treated
and game state server entities (GSSEs). A CE is a unfairly.
client software application that performs input/ Despite these clear drawbacks, this solution
output with its player and receives/notifies events is the preferred one in commercial networked
to the GSSE to which it is connected. Stated games. This is due to several reasons. First is the
simply, a CE acts as a viewport to the game state already mentioned ease of management of the
and passes commands issued by its player to the game state and game administration. Second,
GSSE. The GSSE computes the advancements of when the server is maintained by the game dis-
the game state. tributors, the server acts as an authoritative con-
trol node. In the pay-to-play business model, the
server permits to control the access to the gam-
Client/Server Architecture ing servers and to reduce (or avoid) the game
piracy. Moreover, this allows to profile users
The client/server architecture is the classic solu- and offer additional services related to the
tion used in commercial game products, e.g., game. An example is in-game stores in which
Quake, Ultima Online, and Minecraft (Briceño gamers can purchase items and services. In the
320 Clinical Skills

last years, many online game distributors have References

switched to a free-to-play business model in
which players have access to a significant por- Bauer, D., Iliadis, I., Scotton, P.: Communication architec-
tures for massive multi-player games. Multimed. Tools
tion of their content without paying. In this case,
Appl. 23, 47–66 (2004). https://doi.org/10.1023/B:
the presence of in-game stores is a key part of MTAP.0000026841.97579.1f
the business model. Third, today’s games allow Briceño, L.D., Siegel, H.J., Maciejewski, A.A., Hong, Y.,
users with the possibility of connecting to the Lock, B., Panaccione, C., Wedyan, F., Teli, M.N.,
Zhang, C.: Resource allocation in a client/server system
game using very diverse terminals and modali-
for massive multi-player online games. IEEE Trans.
ties of interaction. In this case, a dedicated Comput. 63(12), 3127–3142 (2014). https://doi.org/
server can provide each specific user with dif- 10.1109/TC.2013.178
ferent output modalities, based on his/her spe- Ferretti, S.: Cheating detection through game time model-
ing: a better way to avoid time cheats in P2P MOGs?
cific needs. An emerging trend in online gaming
Multimed. Tools Appl. 37(3), 339–363 (2008). https://
architectures is multiplayer browser games in doi.org/10.1007/s11042-007-0163-2
which the game is played over the Internet Marzolla, M., Ferretti, S., D’Angelo, G.: Dynamic resource
using a web browser. In this case, the client/ provisioning for cloud-based gaming infrastructures.
Comput. Entertain. 10(1), 4 (2012). https://doi.org/10.
server approach is now the most appropriate
1145/2381876.2381880
solution even if new technical solutions (e.g., Mauve, M., Fischer, S., Widmer, J.: A generic proxy sys-
WebRTC) could permit the development of tem for networked computer games. In: Proceedings of
peer-to-peer browser games. the 1st Workshop on Network and System Support for
Games, NetGames’02, pp. 25–28. ACM, New York
The client/server gaming architecture can be (2002)
naturally deployed over cloud computing infra- Reis Cecin, F., de Oliveira Jannone, R., Resin Geyer, C.F.,
structures (Marzolla et al. 2012). Multiple servers Garcia Martins, M., Barbosa, J.L.V: Freemmg: a hybrid
can be hosted in the cloud, which are devoted to peer-to-peer and client-server model for massively
multiplayer games. In: Proceedings of ACM
handle different gaming sessions. Such an SIGCOMM 2004 Workshops on NetGames’04,
approach increases the scalability of the provided pp. 172–172. ACM Press (2004)
game service. Moreover, in massively multiplayer
online games, the game world can simulate a vast
area. In these cases, the game state can be
partitioned and distributed over multiple servers.
Each server handles a specific portion of the game Clinical Skills
world and interacts with those CEs that are within
that area. According to this approach, once a ▶ Nursing Education Through Virtual Reality:
player leaves the area managed by a given server Bridging the Gap
and enters another area (handled by a different
server), it must disconnect from the previous
server and open a novel connection with the
newer server, all that without introducing consis- Clipping
tency errors or affecting the gaming experience of
the players. ▶ Speedrunning in Video Games

Cross-References

▶ Cloud for Gaming Clothing Brand

▶ Online Gaming Architectures
▶ Online Gaming Scalability ▶ Professional Call of Duty Player Matthew
▶ Peer-to-Peer Gaming “Nadeshot” Haag: An e-Sports Case Study
Cloud for Gaming 321

Measured service: cloud resource and service

Cloud for Gaming usages are optimized through a pay-per-use
business model and are monitored, controlled,
Gabriele D’Angelo, Stefano Ferretti and and reported transparently to both their cus-
Moreno Marzolla tomer and provider.
Department of Computer Science and
Engineering, University of Bologna, Bologna, The typical interaction between cloud provider C
Italy and customer works as follows: the customer con-
nects to a “cloud marketplace” through a Web
interface and selects the type and amount of the
Synonyms resources she needs (e.g., some virtual servers
with given number of CPU cores, memory, and
Cloud gaming infrastructure; Gaming as a service disk space). The resources are allocated from a
(GaaS) large pool that is physically hosted on some big
datacenter managed by the cloud provider. Once
instantiated, the resources are accessed by the
Definition customer through the network. Additional
resources can be acquired at a later time, e.g., to
Cloud for Gaming refers to the use of cloud com- cope with an increase of the workload, and
puting technologies to build large-scale gaming released when no longer needed. The customer
infrastructures, with the goal of improving scal- pays a price that depends on the type and amount
ability and responsiveness, improve the user’s of resources requested (e.g., CPU core speed,
experience, and enable new business models. memory size, disk space) and on the duration of
their usage.
The service model defines the level of abstrac-
What Is Cloud Computing? tion at which the cloud infrastructure provides
service (Fig. 1). In a Software as a Service
Cloud computing is a service model where the (SaaS) cloud, the system provides application ser-
provider offers computation and storage resources vices running in the cloud. “Google Apps” is an
to customers on a “pay-as-you-go” basis (Mell example of a widely used SaaS cloud. In contrast,
and Grance 2011). The essential features of a the capabilities provided by a Platform as a Ser-
cloud computing environment are: vice (PaaS) cloud consist of programming lan-
guages, tools, and a hosting environment for
On-demand self-service: the ability to provide applications developed by the customer. The dif-
computing capabilities (e.g., CPU time, net- ference between the SaaS and PaaS models is that
work storage) dynamically, as needed, without while the user of a SaaS cloud simply utilizes an
human intervention. application that runs in the cloud, the user of a
Broad network access: resources can be accessed PaaS cloud develops an application that can be
through the network by client platforms using executed in the cloud and made available to ser-
standard mechanisms and protocols. vice customers; the application development is
Resource pooling: virtual and physical resources carried out using libraries, APIs, and tools possi-
can be pooled and assigned dynamically to bly offered by some other company. Examples of
consumers, according to their demand, using PaaS solutions are App Engine by Google, Force.
a multitenant model. com from Salesforce, Microsoft’s Azure, and
Elasticity: from the customers’ point of view, the Amazon’s Elastic Beanstalk. Finally, an Infra-
provider offers unlimited resources that can be structure as a Service (IaaS) cloud provides its
purchased in any quantity at any time. customers with fundamental computing
322 Cloud for Gaming

Cloud for Gaming,

Fig. 1 Cloud service
model

I I

capabilities such as processing, storage, and net- responsible for rendering every frame of the
works where the customer can run arbitrary soft- game scene and compressing the video stream so
ware, including operating systems and that it can be transmitted to the user’s terminal
applications. The number of companies offering where the stream is decoded and displayed. User
such kind of services is continually growing, one inputs are acquired from the terminal and sent
of the earliest being Amazon with its EC2 back to the game engine that takes care of
platform. updating the game state accordingly. The advan-
The deployment model defines the mode of tage of the RR-GaaS model is that the workload
operation of a cloud infrastructure; these are the on the terminal is greatly reduced, since the com-
private cloud, the community cloud, the public putationally demanding step of rendering the
cloud, and the hybrid cloud models. A private game scenes is entirely offloaded to the cloud.
cloud is operated exclusively for a customer orga- This allows complex games to be played on less
nization; it is not necessarily managed by that powerful devices, such as mobile phones or cheap
organization. In the community cloud model, the game consoles, that are only required to be capa-
infrastructure is shared by several organizations ble of decoding the video stream in real time.
and supports a specific community with common However, the RR-GaaS model consumes consid-
concerns (e.g., security requirements, policy erable bandwidth to transmit the compressed
enforcement). In the public cloud model, the video stream and may be particularly sensitive to
infrastructure is made available to the general network delays. Examples of RR-GaaS
public and is owned by an organization selling implementations are GamingAnywhere (Huang
cloud services. Finally, the hybrid cloud model et al. 2014) and Nvidia GRID™ (http://www.
refers to cloud infrastructures constructed out of nvidia.com/object/cloud-gaming.html, Accessed
two or more private, public, or community clouds. on 2015/4/4).
In the local rendering GaaS model, the video
stream is encoded on the cloud as a sequence of
Cloud Computing for Gaming high-level rendering instructions that are streamed
to the player terminal (Fig. 2b); the terminal
The gaming industry embraced the cloud comput- decodes and executes the instructions to draw
ing paradigm by implementing the Gaming as a each frame. Since encoding of each frame as a
Service (GaaS) model (Cai et al. 2014). Different sequence of drawing instructions is often more
instances of the GaaS paradigm have been pro- space-efficient than compressing the resulting bit-
posed: remote rendering GaaS, local rendering map, the LR-GaaS model may require less net-
GaaS, and cognitive resource allocation GaaS. work bandwidth than RR-GaaS and therefore
In the remote rendering GaaS (RR-GaaS) eliminate the need for real-time video transmis-
model, the cloud infrastructure hosts one instance sion capability. This comes at the cost of requiring
of the game engine for each player (Fig. 2a). An a more powerful terminal with an adequate
encoder module running on the cloud is graphics subsystem.
Cloud for Gaming 323

a b c

I
I I

C
I I

I I I I I I

Cloud for Gaming, Fig. 2 Gaming as a Service models

Finally, in the cognitive resource allocation pattern that originates from the typical daily
GaaS model, the game engine is logically human activity. As an example, Fig. 3 shows the
partitioned into a set of modules that can be number of online players of RuneScape (http://
uploaded and executed at the client side www.runescape.com) (Marzolla et al. 2012), a
(Fig. 2c). As the game evolves, the terminal fantasy game where players can travel across a
receives and executes the appropriate modules fictional medieval realm. During the observed
and may keep or discard the unused ones. The period, more than 200,000 players are connected
CRA-GaaS model shifts the computation back to to the system at peak hours; this number reduces
the client terminal, therefore reducing the load on to about 110,000 players during off-peak hours.
the cloud. However, the client resources are used Hence, the daily churn (number of players leav-
efficiently, since at any time only the needed com- ing/joining the system during the day) is about
ponents are stored locally. This is a significant 100,000 users. It is evident that static resource
advantage if we consider that the data of a com- provisioning based on the average load results in
plete modern game takes a lot of space for tex- system overload roughly half the time; provision-
tures, 3D models, sounds, and code modules. ing for the worst case results in a massive resource
GaaS provides advantages for both game underutilization.
developers and players. The ability to offload To effectively implement a cloud-based gam-
some computation on the cloud allows simple ing infrastructure, it is necessary to address non-
terminals such as mobile devices to play complex trivial issues related to game state partitioning,
games. Since the game engine is accessed on responsiveness, synchronization, and security.
demand, flexible business models such as Partitioning The key factor for achieving scal-
pay-per-play or monthly subscription can be eas- ability of a GaaS infrastructure is the ability to
ily implemented. Finally, game operators can partition the workload across the cloud resources.
scale up and down the amount of cloud resources This is relatively easy if the workload consists of
used by the gaming infrastructure. the execution of independent game instances that
The last point is particularly important, espe- can be executed on any available resource,
cially for the so-called Massively Multiplayer irrespective of where other instances are running.
Online Games (MMOGs). Modern MMOGs are This is the case when the game does not allow
large-scale distributed systems serving millions of different players to interact. Things become com-
concurrent users which interact in real time with a plex if the instances are not independent, as in the
large, dynamic virtual world. The number of users case of a MMOG system where all players interact
playing the game at any given time follows a with the same virtual world. In this case, the game
324 Cloud for Gaming

RuneScape users online, may 5-16, 2011

250000

200000

150000
Users

100000

50000

0
May 05 May 06 May 07 May 08 May 09 May 10 May 11 May 12 May 13 May 14 May 15 May 16

Cloud for Gaming, Fig. 3 Number of online players of the RuneScape MMOG; the data refers to the period from May
5 to May 16, 2011

engine must maintain a large shared state, allo- zone increases, in order to keep the response time
wing the players to “see” the effects of actions perceived by players below a predefined maxi-
performed by the other players operating in the mum threshold. When the workload decreases,
same virtual location. the game operator can release surplus resources
This is achieved by partitioning the virtual world in order to reduce costs.
across multiple zones, each handled by a separate set Synchronization The success of a gaming
of cloud resources. Given that communication system is based on having players perceiving the
between resource instances may incur significant game state as identical and simultaneously evolv-
delays, it is important that interaction across neigh- ing on every player participating to a gaming
boring zones is minimized. For example, each par- session. If the game state is replicated in different
tition may hold a collection of “islands” such that all cloud servers, a synchronization algorithm is
interactions happen within the collection, while needed to maintain the consistency of the redun-
players can jump from one “island” to another. dant game state. To this aim, different schemes
Depending on the (virtual) mobility pattern of have been proposed in the literature (Game event
each player, some areas of the game field may synchronization 2006). They mainly differ from
become crowded, while others may become less classic synchronization algorithms employed by
populated. In order to cope with this variability, distributed systems in their additional requirement
each zone controller is physically hosted on for keeping the computation quick and respon-
resources provided and operated by a cloud infra- sive. To this aim, some schemes relax the require-
structure. The cloud provider is in general a sep- ments for full consistency during the game state
arate entity providing computational and storage computation.
resources to the game operator on a pay-as-you-go A basic distinction is between conservative
model. This means that the game operator can and optimistic synchronization. Conservative
request additional servers and/or additional stor- synchronization approaches allow the processing
age space at any time and release them when no of game updates only when it is consistency-safe
longer needed. Thus, the game operator can to do so. Lockstep (Fujimoto 1999), time-bucket
request more resources when the workload on a synchronization (Fujimoto 1999), and
Cloud for Gaming 325

interactivity restoring (Ferretti 2014) are some guaranteed that all players perceive the same and
examples in the literature. simultaneous game evolution at the same time.
Optimistic synchronization mechanisms pro- GaaS infrastructures represent an effective
cess game updates as soon as they receive them, tool to provide responsive and fair gaming expe-
thus increasing the responsiveness of the system. riences. Cloud servers can manage the game state
Yet, it is assumed that most updates are received in evolution in a scalable manner. Multiple server
the correct order and that, in any case, it would be instances can be run in the same datacenter, when C
acceptable to recover later from possible needed. Moreover, if the game involves world-
inconsistencies. wide distributed players, one might think to
Examples of optimistic approaches available in introduce a federation of cloud servers, geo-
the scientific literature are the optimistic bucket graphically distributed, so that each client/player
synchronization (Diot and Gautier 1999), the might connect to its nearest server. This could
combination of local lag and time warp proposed balance the network delays between the player
in Mauve et al. (2002), the trailing state synchro- and its server, thus augmenting the fairness level
nization (Cronin et al. 2002), and the improved provided by the system. However, when multiple
time warp equipped with the dropping scheme servers are involved, each one with a redundant
and a correlation-based delivery control approach copy of the game state, synchronization algo-
(Ferretti 2014). rithm is needed to maintain game state
Responsiveness The task of providing a pleas- consistency.
ant experience to players becomes challenging Security and reliability The security issues of
when trying to deploy a large-scale and highly GaaS infrastructures have become mainstream
interactive online game. Responsiveness means after the PlayStation Network outage that, in
having small delays between the generation of a 2011, has halted the Sony online gaming network
game update at a given player and the time at for 23 days. The network was shut down after
which all other players perceive such update. detecting an external intrusion that led to a huge
How much such delays must be small depends number of accounts being compromised and the
on the type of online game. Obviously, the shorter exposure of the players’ personal information.
the delay, the better. But it is possible to identify a From the reliability point of view, large cloud
game-specific responsiveness threshold Tr that systems provide some level of redundancy to cope
represents the maximum delay allowable before with failures, including the use of geographically
providing a game update to players. The typical Tr distributed datacenters, so that catastrophic events
for fast-paced games (e.g., first-person shooter, do not cause a complete outage. Unfortunately,
racing vehicles) is 150–200 ms, but this value the GaaS infrastructure may still represent a single
can be increased to seconds in slow-paced games point of failure; the PlayStation Network outage is
(e.g., strategic, role-playing games) (Ferretti just one example: in that case a security incident
2014; Pantel and Wolf 2002). prompted the system administrators to temporar-
A key point is that each player is geographi- ily shut down the whole service. Other possibili-
cally distributed. Thus, his latency to reach the ties must be considered as well: for example, the
game server on the cloud is usually different from company operating the GaaS infrastructure may
other players. If a classic client-server approach is go bankrupt, depriving all players from the game
employed, it might thus happen that a responsive service they might already have paid for.
service is provided to some subset of users, while From the security point of view, GaaS infra-
the other players can perceive a nonresponsive structures are affected by the typical issues of
game evolution. This raises another main issue, cloud computing (e.g., insider attacks; Zissis and
i.e., fairness provision. This means guaranteeing Lekkas 2012) and online gaming (e.g., cheating;
that all players have the same chance of winning, Hu and Zambetta 2008). Online games are an
regardless of their subjective network conditions appealing target for hacks because players often
(Ferretti 2014). To this aim, it should be invest huge amount of time in their character
326 Cloud Gaming Infrastructure

development, and it is therefore quite easy to Mell, P.. Grance, T.: The NIST Definition of Cloud
monetize game items on the black market. Addi- Computing (Draft)–Recommendations of the
National Institute of Standards and Technology.
tionally, individual accounts on online gaming Special publication 800-145 (draft), Gaithersburg,
platforms often contain information, such as Jan (2011)
credit card numbers, that are the typical target of Pantel, L., Wolf, L.C.: On the impact of delay on real-time
cybercriminals. Details of the avatar of each multiplayer games. In: Proceedings of the 12th Interna-
tional Workshop on Network and Operating Systems
player can provide information such as sexual Support for Digital Audio and Video, NOSS-DAV’02,
preferences (Huh and Williams 2010) that could pp. 23–29. ACM, New York (2002)
cause considerable embarrassment if made public. Zissis, D., Lekkas, D.: Addressing cloud computing secu-
rity issues. Futur. Gener. Comput. Syst. 28(3), 583–592
(2012)

Cross-References

▶ Virtual World, a Definition Incorporating Dis-

tributed Computing and Instances Cloud Gaming Infrastructure

▶ Cloud for Gaming

References

Cai, W., Chen, M., Leung, V.C.M.: Toward gaming as a

service. IEEE. Internet. Comput. 18(3), 12–18 (2014) Clustering
Cronin, E., Filstrup, B., Kurc, A.R., Jamin, S.: An efficient
synchronization mechanism for mirrored game archi-
tectures. In: Proceedings of the 1st Workshop on Net- ▶ Machine Learning for Computer Games
work and System Support for Games, NetGames’02,
pp. 67–73. ACM, New York (2002)
Diot, C., Gautier, L.: A distributed architecture for multi-
player interactive applications on the internet. Netw.
IEEE. 13(4), 6–15 (1999) Co-creation
Ferretti, S.: Synchronization in Multiplayer Online Games,
pp. 175–196. Wiley, New York (2014) ▶ Game Prosumption
Fujimoto, R.M.: Parallel and Distribution Simulation Sys-
tems, 1st edn. Wiley, New York (1999)
Ferretti, S., Roccetti, M., Salomoni, P. Game event syn-
chronization. In: Borko, F. (ed.) Encyclopedia of Mul-
timedia, pp. 256–257. Springer, US, New York (2006) Codec
Hu, J., Zambetta, F.: Security issues in massive online
games. Secur. Commun. Netw. 1(1), 83–92 (2008)
Huang, C.-Y., Chen, K.-T., Chen, D.-Y., Hsu, H.-J., Hsu, ▶ Postproduction in Game Cinematics
C.-H.: Gaminganywhere: the first open source cloud
gaming system. ACM Trans. Multimed. Comput.
Commun. 10(1s), 10:1–10:25 (2014)
Huh, S., Williams, D.: Dude looks like a lady: gender
swapping in an online game. In: Bainbridge, Cognitive Disabilities
W.S. (ed.) Online Worlds: Convergence of the Real
and the Virtual. Human-Computer Interaction Series, ▶ Making Virtual Reality (VR) Accessible for
pp. 161–174. Springer, London (2010)
Marzolla, M., Ferretti, S., D’Angelo, G.: Dynamic resource
People with Disabilities
provisioning for cloud-based gaming infrastructures.
Comput. Entertain. 10(1), 4:1–4:20 (2012)
Mauve, M., Fischer, S., Widmer, J.: A generic proxy sys-
tem for networked computer games. In: Proceedings of
the 1st Workshop on Network and System Support for
Cognitive Games
Games, NetGames’02, pp. 25–28. ACM, New York
(2002) ▶ Games and Active Aging
Cognitive Processing of Information Visualization 327

cognitive load, and sensemaking. The hope is that

Cognitive Graphics Tool this will provide readers enough of an understand-
ing of visualization through visual perception and
▶ 2-Simplex Prism as a Cognitive Graphics Tool cognition theories and methods. We presented two
for Decision-Making case studies that illustrated how cognitive theories
inform and impact our research.
C

Cognitive Processing State of the Art Work

▶ Cognitive Processing of Information Vision’s Constructive Power

Visualization Gardner (1983) advocated for a multiple intelli-
gence theory in 1983 that has drawn the attention
of researchers due to its premise that each individ-
ual difference’s intelligence is composed of mul-
Cognitive Processing of tiple intelligences. The intelligences are
Information Visualization independent and have their own operating sys-
tems within the brain. People with higher visual
Chen Guo1, Shuang Wei2 and Yingjie Chen2 and spatial intelligences respond better to visual
1
School of Media Arts & Design, James Madison cues by storing, manipulating, and recreating
University, Harrisonburg, VA, USA visual information. The constructive power of
2
Department of Computer Graphics Technology, human vision has drawn the attention of
Purdue University, West Lafayette, IN, USA researchers for many years. This section reviews
the work of Hoffman, Gombrich, and Arnheim in
the domain of visual intelligence and discusses
Synonyms how they inform, impact, and relate the creation
and viewer interpretation of information
Information visualization visualizations.

Hoffman and Vision’s Constructive Power

Definition Hoffman (2000) attempted to explain the complex
mental construction process of all sighted individ-
Information visualization is the use of computer uals. In Visual Intelligence (Hoffman 2000), he
supported, interactive, visual representations of introduced sets of universal and specialized rules
abstract data to amplify cognition (Card that govern our perception of line, color, form,
et al. 1999). depth, and motion. Hoffman’s rule of generic
views and 10 rules of visual intelligence clearly
explain how individuals interpret and construct
Introduction visual objects. These rules indicate how our
mind organizes data and turn data into knowledge.
Information visualizations turn raw data into They can be widely applied to visualization
information and enable researchers to gain insight design. People have countless interpretations of
from the data. Understanding how viewers inter- what they see, but humans prefer to perceive
pret different types of visual information contrib- things quickly and efficiently. The generic views
utes to the creation of effective and intuitive rule is that designers should construct a stable
visualizations. This paper introduces the cognitive view and a constant image. If an object has more
processing of visualizations from the angles of salient-part boundaries, humans will see it as a
pre-attentive processing, visual working memory, figure because it is more efficient for our
328 Cognitive Processing of Information Visualization

perception to process clearer evidence and a stron- focus), involvement (meaning attached to the
ger boundary. Havre et al. (2000) were inspired by awareness), and attitude (feeling resulting from
the perceptual processes of identifying curves and the meaning) (Gombrich 1977). The first step is
silhouettes, recognizing parts, and grouping them to be attracted to parts of the image. Then a viewer
together into objects. They created a novel visu- attaches some meaning to the parts. Thereafter,
alization tool called ThemeRiver that employs the viewers will normally generate some feeling or
river metaphor to depict thematic variations over attitude towards the image. At this point, looking
time. Temporal thematic changes can be easily changes into seeing with a statement of the image.
recognized because the system uses smooth and The attitudes, in turn, affect the way viewers per-
continuous curves to bound a theme and distinct ceive the image. The engagement process clearly
colors to differentiate themes. presents how viewers interact with pictures, and
Moreover, Hoffman (2000) found that if two therefore enlightens the visualization design.
visual structures have a non-accidental relation, a Shneiderman (1996) was inspired by
designer should group and assign them to a com- Gombrich’s schemata and proposed a famous
mon origin. He also stated that if three or more visual information-seeking mantra: Overview
curves intersected at a common point in an first, zoom and filter, then details-on-demand.
image, they should be interpreted as intersecting The mantra has served as a golden rule in visual
at a common point in space. These rules can analytics because it takes human perceptual abil-
guide the design of movement data through ities in current design into consideration. It is very
grouping and stack trajectories based on time easy for audiences to scan, recognize, and recall
proximity and visual similarity. Crnovrsanin images rapidly. Audiences detect changes in size,
et al. (2009) plotted trace as distance to the color, shape, movement, or texture. It is intuitive
explosion (y-axis) vs. time (x-axis). By applying for audiences to perform tasks like dragging one
proximity, audiences can easily depict the entire object to another. Almost all successful visualiza-
event a glance and identify different patterns, tion design supports the overview, zoom and filter,
such as spatial concentration, co-incidence, then details-on-demand. Guo et al. (2014) used
trends, and divergence. In order to make a pow- dodecagons to visualize GPS data. The system
erful design and compelling product, visualiza- provides an effective overview to show the com-
tion researchers need to integrate these rules and mon patterns. It also allows analysts to filter and
construct what human beings desire to see with examine individual patterns in detail through var-
little effort. ious interactions.

Gombrich and Constructivist Perception Arnheim and Gestalt Principles

Gombrich (1977) proposed in Art and Illusion that Arnheim (1969) defined picture perception as the
visual perception is always functioning as a pro- composition of circles, lines, squares, and other
jection of prior experience and imagination, or the forms of graphs into shapes and patterns. The
so-called constructive. As a constructivist, innate laws of the structure are called Gestalt
Gombrich pointed out that artists manipulate the theory. In Art and Visual Perception:
inherited pictorial schemata to directly observe A Psychology of the Creative Eye (Arnheim
the world, and in turn correct the schemata based 1974), Arnheim detailed picture-making based
on their interaction experience. Gombrich (1977) on balance, shape, form, growth, space, light,
also pointed out that the ability to recognize color, movement, dynamic, and expression.
objects was the result of perceptual tuning and Gestalt laws, such as figure/ground, simplicity,
selection attention. He differentiated looking completeness, good continuation, and the like
from seeking, and stated that viewers experience were named as fundamental to human perception
four-step processes of image engagement while and visual design.
looking at images. The four steps include atten- Visual balance is an innate concept as well as a
tion (focus), interest (cognitive awareness of the key principle with which designers convey
Cognitive Processing of Information Visualization 329

messages (Pornstein and Krinsky 1985). Learning grouping. This approach facilitated abstraction by
visual balance enables designers to create visual- identifying the most relevant information items
izations that “look visually right” (Carpenter and after assigning an importance value to each item.
Graham 1971) with respect to color and layout. GVA could be applied to geographic map-based
Arnheim (1974) illustrated the balance concept interfaces to support incident management and
with a structural net that determined balance. He decision-making. Visual abstraction helps to
described that every visual work had nine hotspots transfer the meaning of the original data into a C
and visual elements on the main axes or at the slightly different but clearer form. Additionally,
centers that should be in visual balance. Weight visual abstraction also supports the cluttered
and direction led to visual balance. More specifi- graphical representation of numerous big data
cally, the characteristics of visual objects, such as sets by replacing the data with new visual ele-
location, size, color, shape, and subject matter, ments corresponding to higher levels of abstrac-
influenced visual balance. In spatiotemporal visu- tion (Novotny 2004).
alization, it is not an easy task to arrange map
elements – legends, scales, borders, areas, place Preattentive Processing
names, and glyphs – into an aesthetically pleasing Human brains can rapidly and automatically
design. Dent (1999) employed the structural net as direct attention to information that has the highest
a guide for thematic map creation. The research salience as well as suppress irrelevant information
effectively used all spaces and retained a harmo- based on simple computations of an image
nious balance among visual elements. (Healey and Enns 2012). This is often called pre-
Arnheim proposed that visual thinking attentive processing and provides the informa-
occurred primarily through abstract imagery tional basis of attentional selection (Logan
(Arnheim 1969). Arnheim stressed the impor- 1992). Detection precedes conscious attention.
tance of reasoning with shapes and identified the Selection cannot occur until the ensemble coding
nature of abstraction in visual representation. and feature hierarchy of pre-attentive process is
Designers always use visual abstraction to clean complete.
up the display and impress observers. When using Many research efforts have tried to address the
visual abstraction in information visualization, following central question: which properties of
researchers should keep in mind that the meaning visualizations rapidly attract people? Selective
of the raw data sets should be preserved true to attention usually binds features, such as color,
their original form. Numerous visual abstraction shape, location, and texture, into a perceptual
approaches have emerged in the visualization object representation (Wheeler and Treisman
field. Agrawala and Stolte (2001) presented 2002). Ware (2012) identified visual properties
route maps to depict a path from one location to that are pre-attentively processed in visualization.
another. Through specific generalization tech- These are also referred to as pre-attentive attri-
niques involving distortion and abstraction, route butes that can be perceived in less than 10 ms
maps were able to present trajectory in a clear, without conscious effort, and require
concise, and convenient form. Lamping et al. 200–250 ms for large, multi-element displays.
(1995) created a novel, hyperbolic geometry These attributes are grouped into four categories:
approach to visualize large hierarchies. Interac- color (hue and intensity), form (line orientation,
tion techniques, such as manipulating focus, line length, line width, line collinearity, size, cur-
using pointer clicks, and interactive dragging, vature, spatial grouping, blur, added marks, and
emphasized important actions that viewers tend numerosity), motion (flicker and direction of
to focus at the expense of distorting less important motion), and spatial position (2D position, stereo-
information. Humphrey and Adams (2010) scopic depth, and convex/concave shape based on
employed the General Visualization Abstraction shading) (Ware 2012). The result, which makes
(GVA) algorithm in providing a novel technique symbols pop out, can be applied to information
for information abstraction, such as selection and visualization design.
330 Cognitive Processing of Information Visualization

Five notable theories explain how pre-attentive elements. This is referred to as a Boolean map.
processing occurs. The feature integration theory Viewers were able to generate Boolean maps in
proposed a model of how low-level human vision two ways: by specifying a single value of a feature
is composed of a set of feature maps that can or applying union and intersection onto two
detect features either in parallel or in serial existing maps (Huang and Pashler 2007).
(Treisman 1991), and a master map of locations
that is required to combine featured activities at a Visual Working Memory
common spatial location (Treisman and Gelade Baddeley (1992) stated that the working memory
1980). Texton Theory focused on statistical anal- model consists of a phonological loop that main-
ysis of texture patterns. A group of texture pat- tains verbal–linguistic information, a visuospatial
terns consists of three categories: elongated blobs sketchpad that maintains visual and spatial infor-
(e.g., rectangles, ellipses, line segments) with spe- mation, a central executive to control and coordi-
cific properties, such as hue, orientation, and nate the operation of the systems, and an episodic
width; terminators (ends of line segments); and buffer to communicate with long-term memory.
crossing line segments (Julesz 1981, 1984). Working memory decides which activities to per-
Researchers stated that only a difference in form, inhibits distracting information, and stores
textons or in their density can be detected (Julesz information while accomplishing a complex task
1981). Instead of supporting the dichotomy of (Miyake and Shah 1999). Luck and Vogel (2013)
serial and parallel search modes, Duncan and defined visual working memory as the active
Humphreys (1989) explored two factors that maintenance of visual information to serve the
may influence search time in conjunction needs of ongoing tasks. The last 15 years have
searches: the number of information items seen a surge in research on visual working mem-
required to identify the target and how easily a ory that aims to understand its structure, capacity,
target can be distinguished from its distractors. and the individual variability present in its cogni-
Duncan and Humphreys (1989) assumed that the tive functions. There are three essential theoretical
search ability depends on the type of task and the issues related to visual working memory: discrete-
display conditions. Search time is related to two slot versus shared-resource, visual representation,
factors: T-N similarity and N-N similarity and visual context.
(Duncan 1989). T-N similarity refers to the Visual working memory research has largely
amount of similarity between targets and nontar- focused on identifying the limited capacity of the
gets that have a positive relationship with search working memory system and exploring the nature
time and a negative relationship with search effi- of stored memory representations. The field has
ciency. N-N similarity represents the amount of recently debated whether the capacity of visual
similarity within the nontargets themselves that working memory is constrained by a small set of
have a negative relationship with search time “discrete fixed-precision representations,” the
and a negative relationship with search efficiency. discrete-slot model, or by a pool of divisible
Guided search theory was proposed by Wolfe resources in parallel, the shared-resource model
(1994). He constructed an activation map for the (Luck and Vogel 2013; Huang 2010; Zhang and
visual search based on bottom-up and top-down Luck 2008). Visual working memory allows peo-
visual information (Wolfe 1994). Users’ attention ple to temporarily maintain visual information in
is drawn to the highest hills in the activation map, their minds for a few seconds after its disappear-
which generates the largest combination of ance (Luck and Vogel 1997). Some researchers
bottom-up and top-down influences (Healey and proposed that working memory stores a fixed
Enns 2012). More recently, Boolean Map Theory number of high-precision representations when
was presented (Huang and Pashler 2007). people are faced with a large number of items,
Researchers divided the visual search into the and no information is retained about the
two stages of selection and access, and divided remaining objects (Luck and Vogel 1997; Pashler
the scene into selected elements and excluded 1988; Zhang and Luck 2008). Luck and Vogel
Cognitive Processing of Information Visualization 331

(1997) also stated that it is possible to store both guided by what we store in our memory. Attention
the color and orientation of four items in visual and visual working memory are closely
working memory. Some researchers found that connected. The central executive of working
both the number of visual objects and visual infor- memory manages the direction of attention and
mation load imposed capacity limits on visual the supervision of information integration. More-
working memory up to approximately four or over, both attention and visual working memory
five objects (Alvarez and Cavanagh 2004; Luck have a limited capacity of visual features that can C
and Vogel 1997; Pashler 1988), and six spatial be detected or maintained. Preattentive processing
locations represented allocentrically in a spatial plays a critical role in which visual properties
configuration (Bor et al. 2001). Thus, the tempo- our eyes are drawn to, and therefore helps people
rary storage of visual information is more related deal with visual and spatial information in work-
to integrated objects rather than individual fea- ing memory.
tures. This statement is also consistent with the
selective attention metaphor that visuospatial Cognitive Load
attention is like the beam of a flashlight. People Cognitive load refers to “the total amount of men-
are unable to split their attention to several loca- tal activity that the working memory imposes on
tions and are, instead, always paying attention to working memory at an instance in time” (Cooper
the most important events while simultaneously 1998). Cooper (1998) also stated, “the most
filtering out all distractions. important factor that contributes to cognitive
However, other researchers claimed that the load is the number of elements that need to be
visual working memory is able to store imprecise attended to.” Sweller and his colleagues
representations of all items, including low- (Chandler and Sweller 1991, 1992; Sweller et al.
resolution representations of the remaining 1998) identified three sources of cognitive load:
objects (Bays et al. 2009, 2011; Bays and Husain intrinsic, extraneous, and germane. The intrinsic
2008; Frick 1988; Wilken and Ma 2004). They cognitive load is determined by the basic charac-
thought of visual working memory as many low- teristics of the information (Sweller 1993). The
resolution digital photographs and challenged the extraneous cognitive load is imposed by the
concept of working memory by examining the designer as they organize and display information
distribution of recall errors across the visual (Chandler and Sweller 1991, 1992). Designers are
scene. Based on a Bayesian decision model, always striving to reduce cognitive load and help
more visual objects are held in visual working viewers grasp the underlying information more
memory and fewer resources are allocated to effectively and efficiently. Finally, the germane
each object. Thus, in contrast to the discrete slots cognitive load is the roaming free capacity that
model, the continuous resource model emphasizes uses the extraneous load to build a new, complex
that the storage capacity of the visual working schema (Sweller et al. 1998).
memory is not limited to the number of visual Miller (1956) developed our understanding of
objects. Recent empirical evidence on recurrent working memory by using information chunks
neural networks suggests that a discrete item limit that could be strung together. He and his followers
is more favorable (Luck and Vogel 2013). also believed that working memory had a capacity
Although there is still much ongoing debate of between seven and ten chunks at any given time
regarding the models for resource allocation, (Merriënboer and Sweller 2005; Miller 1956). In
there is general agreement that visual working terms of visual working memory, the capacity is
memory has an important object/resolution limited to approximately four or five visual ele-
trade-off: as more items are stored in visual work- ments (Alvarez and Cavanagh 2004; Luck and
ing memory, less fidelity per visual item can be Vogel 1997; Pashler 1988) and six spatial loca-
maintained (Brady et al. 2011). tions where conscious thought transpires (Bor
Additionally, what we see depends on where et al. 2001). Generally speaking, visual elements
our attention is focused, and our attention is are schemas that can be understood as models that
332 Cognitive Processing of Information Visualization

organize our knowledge. Different people men- Sensemaking

tally store different numbers of visual objects. Sensemaking is the process through which people
Due to the nature of the material, it is almost make trade-offs and construct new knowledge of
impossible to change intrinsic cognitive load the world (Weick 1995). These processes include
with design. However, people can control their the encoding, retention, and retrieval of informa-
extraneous cognitive load using design tech- tion in a temporary workspace or working mem-
niques. The level of the extraneous cognitive ory. Sensemaking is constrained by the structure
load may be modified based on how designers and capacity of working memory; the operation of
present information to users. The germane cogni- the knowledge base is constrained by its own
tive load is more so focused on individual self- nature and architecture (Gilhooly and Logie
regulation and concerns schema automation (Paas 2004). Visual analytics is defined as the science
et al. 2004). Some research posited that the third of analytical reasoning facilitated by interactive
resource germane cognitive load may have a pos- visual interfaces (Thomas and Cook 2005). The
itive impact on working memory, whereas the field aims to support sensemaking processes and
intrinsic and extraneous cognitive loads are con- gain insight using an interactive, visual explora-
sidered as negative in impact (Sweller et al. 1998). tion of the data set. However, due to the limited
One metric for the evaluation of visualization is to capacity of visual working memory, it is very
compare the sum of the intrinsic cognitive load difficult for people to discover and keep track of
and extraneous cognitive load with the working all patterns while looking at visualization graphs.
memory capacity. If the additive effect produced Also, inconsistencies between mental models and
by the three resources is less than the working external representations increase the cognitive
memory capacity, the visualization system overload and thereby further hinder sensemaking
involves lower cognitive overload and good outcomes.
usability, which is more likely to be successful. There are three phases and loops in the
On the contrary, if the sum exceeds the user’s sensemaking process: information foraging, infor-
working memory capacity, the visualization sys- mation schematization and problem-solving, and
tem has higher cognitive overload and poor decision-making and action (Ntuen et al. 2010).
usability, which is far less likely to be successful. Information foraging is a cost and benefit assess-
Visual working memory has a limited amount ment of maximizing the rate of gaining valuable
of processing power and capacity. Users will get information and minimizing the number of con-
overwhelmed and abandon a visualization task sumed resources. Based on the metaphor of an
when the amount of information exceeds their animal foraging for food, information foraging
visual working memory capacity. The designer theory helps visualization researchers discover
must use various design strategies to keep the effective ways to represent massive amounts of
cognitive load imposed by a user interface to a data and provide effective mechanisms for navi-
minimum; therefore, more visual working mem- gation. Challenges include formalizing contribu-
ory resources are available for activities. Also, tions, such as identifying trends or outliers of
cognitive load varies by user. Examples include interest, posting explanatory hypotheses, and pro-
the involvement required for interaction. For an viding retrieval mechanisms. Information sche-
experienced user, adding interaction may assist matization is regarded as an information fusion
them to gain insight from the data. Such interac- tool or thinking process that uses new information
tivity might increase cognitive overload and make to explain surprise and update prospective, pre-
the visualization more difficult for the novice user. dictive states of a situation. Decision-making sup-
By reducing extraneous cognitive load with visual ports situational understanding in which a
analytics techniques, we can minimize the total stimulus is placed into a framework to understand,
cognitive load imposed by the visualization inter- explain, attribute, extrapolate, and predict a pro-
face, which will increase the portion of available cess that leads to situational understanding.
working memory to attend to information. Sensemaking deals with seeking, collating, and
Cognitive Processing of Information Visualization 333

interpreting information to support decision- According to Arnheim (1969), people will inter-
making (Ntuen et al. 2010). Information content pret the composition of different forms of graphs
is held in active working memory. Sensemaking into shapes and patterns. The user will automati-
can be applied to information visualization with cally categorize the same color dots into a group.
the intent to reduce complexity and simplify the The good continuation between the same colored
volume of data collected in order to create under- dots will make them into a line from Gestalt
standing. Visualization can serve to amplify or principles of perception (Arnheim 1974). The C
weaken cognitive ability. line depicts the waiting people and the waiting
The following section presents two spatial- time’s change through time, and the distinct colors
temporal data visual analytics systems to illustrate differentiate the data on different days. In Fig. 1,
how cognitive theories could be applied to the people will know that the waiting people and time
design and development of visualization system. on Friday is much lesser than the weekend.
To reduce users’ visual working memory, in
CrowdAnalyzer, every visualization has a legend
Case Study 1: Vast Challenge 2015 on the corner. Windows are arranged side by side
CrowdAnalyzer to convenient users’ exploration and comparison.
Hundreds of thousands of tourists visit the park in
CrowdAnalyzer is a system developed from the 3 days. To reduce users’ workload from both
IEEE Visual Analytics Science and Technology perception and memory aspects, it is important
(Vast) challenge 2015 (Wei et al. 2015). With a to categorize visitors into typical groups.
background that DinoFun World is a typical Researchers applied statistical analysis (K-means
amusement park, sitting on hundreds of hectares and EM algorithms) on data criteria (such as vis-
and hosting thousands of visitors every day, a iting date, entering location, and population) to
dataset including 3 days’ movement tracking cluster individual tourists into hundreds of small
information of visitors is provided for researchers groups. Then visualize these groups using parallel
to identify the patterns and outliers of attendances coordinates. The parallel coordinates have eight
and park’s activities during those 3 days. vertical axes (group ID, walking steps, the number
CrowdAnalyzer is designed to visualize visi- of the entrance, group population, the number of
tors’ movements, analyze parks’ facility status, thrill rides, the number of kiddie rides, the number
and identify visitors’ group and movement pat- of everyone rides, and the number of shows). The
terns. The system has four parts: facility manager, line connecting axes represents a typical visitor
group hierarchical inspector, enter-leave time group. Users can filter on the parallel coordinates
group viewer, and map (Fig. 1). Due to the huge to figure out patterns and outliers. For example, in
amount of data, it is impossible to present the data Fig. 2, the groups that haven’t taken any rides but
at once. CrowdAnalyzer allows the viewer to get entered the park many times are picked out. The
an overview first and then decide which area of groups’ moving trajectories and the key time
the map they did like to further investigate points are presented on the map. There are two
through zooming in and filtering the details of groups of concentric arcs on the map. Each arc
the data. represents one group. The arcs with the same
The tab “Facility Manager” presents the parks’ radius represent the same group, so we know
entertainment facilities’ status: the number of peo- there are six groups of visitors. As the parallel
ple waiting and the estimated waiting time. Mul- coordinates show there are only one people in
tiple line charts are used to present the data. The each group, there are six people in total. The
positive y-axis represents the number of people, start point of the arc represents the visitor’s enter-
the negative y-axis is the waiting time, and the ing time, and the end of the arc represents the
x-axis is the timeline. The colored points on the visitor’s leaving time. We can tell that six people
coordinate show the number of people waiting to entered and left the park at the same time. The
take the ride at the corresponding time points. yellow color on the arc represents the duration that
334 Cognitive Processing of Information Visualization

Cognitive Processing of Information Visualization, Fig. 1 Facility manager in CrowdAnalyzer

Cognitive Processing of Information Visualization, Fig. 2 Enter-leave time group view and parallel coordinates in
CrowdAnalyzer
Cognitive Processing of Information Visualization 335

the visitor stayed at the place on the map. In between arcs strengthens the boundary of arcs.
conclusion, from the trajectories and arcs on the The radius of each arc represents the number of
map, the six people arrived and left the park at the outliers of employees’ movements. Although in
same time, shared exactly the same path, took a the weekend, employees took breaks and the arcs
rest at the entrance before entering the park, then are not shown, people still have the visual ability
took a rest again at the other entrance of the park. to make up the absent part, recognize the pie
Then, we can reasonably consider these people chart, and further understand the information C
are not visitors but the park staffs who are respon- that pie chart is designed to express. The seven
sible for float performances. They performed on a pie charts showing movement anomalies of
fixed route, repeated the performance five times seven types of employees present that the move-
each day, and took a rest after the performance ments of employees from the administration,
was done. Through filtering out abnormal data on engineering, facilities, and IT departments are
parallel coordinates and presenting detail infor- more flexible than the employees of the execu-
mation on the map, users are able to observe and tive, HR, and security department.
understand the characters of different types of By clicking on each pie chart, a detailed visu-
visitors. alization combining coordinate, color, shade, and
shapes are provided (Fig. 4). Human is sensitive to
the difference in color, shape, and shade. Different
Case Study 2: Vast Challenge 2016 visual factors are adopted to take advantage of
Metacurve human visual perception. The x-axis of the coor-
dinate in Fig. 4 is the timeline. The y-axis is
The second system MetaCurve aims to help corresponding to the building floors and zones.
analysts interpret and analyze periodical time From the bottom of the y-axis (first-floor zone 1)
series data using IEEE VAST 2016 mini Chal- to the top (third floor zone 7), each proximity zone
lenge 2 dataset. The dataset registered an office is presented. As employee’s movement data is
building’s temperature and HAVC (heating, ven- periodical data, their proximity records are
tilation, and air conditioning) system status in stacked together in the chart. If an employee
2 weeks, and the employees’ movements in the appeared at a zone less than four times in
building. The first section of MetaCurve is the 2 weeks, a light shade would be marked at the
movement patterns and outliers of employees. time. If the appearance time is only once, a dot
Summarized pie charts (Fig. 3) are designed to will be marked on the shade. The orange line
attract users’ attention and guide users to click marked on the zone shows the location of the
the pie charts to explore more. In Fig. 3, fourteen employee’ office. Figure 4 clearly depicts the
arcs make up a pie chart. Color is used to differ- movement differences between a security people
entiate different arcs, and the black stroke and an administer.

Cognitive Processing of Information Visualization, Fig. 3 Pie charts in MetaCurve

336 Cognitive Processing of Information Visualization

Cognitive Processing of Information Visualization, Fig. 4 The timeline chart showing the movement differences
between a security people and an administer

Cognitive Processing of Information Visualization, Fig. 5 Statistical measurements of CO2 concentration in

MetaCurve

The building temperature and HVAC data are Conclusion

also periodical data, which is overlaid onto the
timeline in MetaCurve. For continuous data, The information visualization field is a multi-
such as CO2 concentration in Fig. 5, after using disciplinary field and this field still lacks sufficient
statistical measurements to compute the median theoretical foundations that inform design and
value, variation, and interquartile, the range evaluation. This paper provided an overview of
between the first quartile and the third quartile the literature related to the cognition theories of
is plotted using color and considered as a normal information visualization. It introduced the under-
data range. The data points located out of the lying cognitive knowledge and the basic percep-
range are considered as major outliers and tion theories central to the development of the
marked with dots. The color of dots is ideas in visualization. The two case studies of
corresponding to the date. information visualization show how the basic
By using colorful pie charts, MetaCurve cognition theories inform and impact the design
firstly attracts users’ attention and then pro- and development of information visualization.
vides detailed spatial-temporal information Investigating the cognitive processing of informa-
through combining coordinate, color, curve, tion visualization can provide a useful way for
and shapes. researchers to describe, validate, and understand
Cognitive Processing of Information Visualization 337

the design work. The value of theories in cogni- Chandler, P., Sweller, J.: The split-attention effect as a
tion and perception can further defend and secure factor in the design of instruction. Br. J. Educ. Psychol.
62, 233–246 (1992)
the information visualization discipline. Cooper, G.: Research into cognitive load theory and
instructional design at UNSW. (1998). 347–362.
http://dwb4.unl.edu/Diss/Copper/UNSW.htm
Cross-References Crnovrsanin, T., Muelder, C., Correa, C., Ma, K.L.:
Proximity-based visualization of movement trace
data. In: 2009 I.E. Symposium on Visual Analytics
C
▶ Multivariate Visualization Using Scatterplots Science and Technology. pp. 11–18 (2009)
▶ Teaching Computer Graphics by Application Dent, B.D.: Cartography: Thematic Map Design. WCB/
McGraw-Hill, Boston (1999)
Duncan, J.: Boundary conditions on parallel processing in
human vision. Perception. 18, 457–469 (1989)
References Duncan, J., Humphreys, G.W.: Visual search and stimulus
similarity. Psychol. Rev. 96, 433–458 (1989)
Agrawala, M., Stolte, C.: Rendering effective route maps: Frick, R.W.: Issues of representation and limited capacity
improving usability through generalization. In: Pro- in the visuospatial sketchpad. Br. J. Psychol. Lond.
ceedings of the 28th Annual Conference on Computer Engl. 1953. 79(Pt 3), 289–308 (1988)
Graphics and Interactive Techniques, pp. 241–249. Gardner, H.: Frames of Mind: The Theory of Multiple
ACM, New York (2001) Intelligences. Basic Book, New York (1983)
Alvarez, G.A., Cavanagh, P.: The capacity of visual short- Gilhooly, K., Logie, R.H.: Working Memory and Think-
term memory is set both by visual information load and ing: Current Issues In Thinking And Reasoning. Psy-
by number of objects. Psychol. Sci. 15, 106–111 (2004) chology Press, Hove (2004)
Arnheim, R.: Visual Thinking. University of California Gombrich, E.: Art and Illusion: A Study in the Psychology
Press, Berkeley (1969) of Pictorial Representation. Phaidon Press,
Arnheim, R.: Art and Visual Perception. University of London/New York (1977)
California Press, Los Angeles (1974) Guo, C., Xu, S., Yu, J., Zhang, H., Wang, Q., Xia, J.,
Baddeley, A.: Working memory. Science. 255, 556–559 Zhang, J., Chen, Y.V., Qian, Z.C., Wang, C.,
(1992) Ebert, D.: Dodeca-rings map: Interactively finding pat-
Bays, P.M., Husain, M.: Dynamic shifts of limited terns and events in large geo-temporal data. In: 2014 I.-
working memory resources in human vision. Science. E. Conference on Visual Analytics Science and
321, 851–854 (2008). https://doi.org/10.1126/scien Technology (VAST). pp. 353–354 (2014)
ce.1158023 Havre, S., Hetzler, B., Nowell, L.: Theme River: visualiz-
Bays, P.M., Catalao, R.F.G., Husain, M.: The precision of ing theme changes over time. In: IEEE Symposium
visual working memory is set by allocation of a shared on Information Visualization 2000. INFOVIS 2000.
resource. J. Vis. 9, 7 (2009). https://doi.org/10.1167/9. Proceedings. pp. 115–123 (2000)
10.7 Healey, C.G., Enns, J.T.: Attention and visual memory
Bays, P.M., Wu, E.Y., Husain, M.: Storage and binding of in visualization and computer graphics. IEEE
object features in visual working memory. Neuropsy- Trans. Vis. Comput. Graph. 18, 1170–1188 (2012).
chologia. 49, 1622–1631 (2011). https://doi.org/10.10 https://doi.org/10.1109/TVCG.2011.127
16/j.neuropsychologia.2010.12.023 Hoffman, D.D.: Visual Intelligence: How We Create what
Bor, D., Duncan, J., Owen, A.M.: The role of spatial We See. W. W. Norton, New York (2000)
configuration in tests of working memory explored Huang, L.: Visual working memory is better characterized
with functional neuroimaging. Scand. J. Psychol. 42, as a distributed resource rather than discrete slots.
217–224 (2001) J. Vis. 10, 8 (2010). https://doi.org/10.1167/10.14.8
Brady, T.F., Konkle, T., Alvarez, G.A.: A review of Huang, L., Pashler, H.: A Boolean map theory of
visual memory capacity: Beyond individual items and visual attention. Psychol. Rev. 114, 599–631 (2007).
toward structured representations. J. Vis. 11, 4 (2011). https://doi.org/10.1037/0033-295X.114.3.599
https://doi.org/10.1167/11.5.4 Humphrey, C.M., Adams, J.A.: General visualization
Card, S.K., Mackinlay, J.D., Shneiderman, B. (eds.): abstraction algorithm for directable interfaces:
Readings in Information Visualization: Using Vision component performance and learning effects. IEEE
to Think. Morgan Kaufmann Publishers Inc., Trans. Syst. Man Cybern. Part Syst. Hum. 40,
San Francisco (1999) 1156–1167 (2010). https://doi.org/10.1109/TSMCA.2
Carpenter, P., Graham, W.: Art and Ideas: An Approach to 010.2052604
Art Appreciation. Mills and Boon, London (1971) Julesz, B.: A theory of preattentive texture discrimination
Chandler, P., Sweller, J.: Cognitive load theory and the based on first-order statistics of textons. Biol. Cybern.
format of instruction. Cogn. Instr. 8, 293–332 (1991) 41, 131–138 (1981)
338 Cognitive Psychology Applied to User Experience in Video Games

Julesz, B.: A brief outline of the texton theory of Thomas, J.J., Cook, K.A.: Illuminating the Path: The
human vision. Trends Neurosci. 7, 41–45 (1984). Research and Development Agenda for Visual Analyt-
https://doi.org/10.1016/S0166-2236(84)80275-1 ics. IEEE Computer Society/Pacific Northwest
Lamping, J., Rao, R., Pirolli, P.: A Focus+Context tech- National Laboratory (PNNL), Los Alamitos/Richland
nique based on hyperbolic geometry for visualizing (2005)
large hierarchies. In: Proceedings of the SIGCHI Treisman, A.: Search, similarity, and integration of features
Conference on Human Factors in Computing Systems, between and within dimensions. J. Exp. Psychol. Hum.
pp. 401–408. ACM Press/Addison-Wesley, New York Percept. Perform. 17, 652–676 (1991)
(1995) Treisman, A.M., Gelade, G.: A feature-integration theory
Logan, G.D.: Attention and preattention in theories of of attention. Cogn. Psychol. 12, 97–136 (1980). https://
automaticity. Am. J. Psychol. 105, 317–339 (1992) doi.org/10.1016/0010-0285(80)90005-5
Luck, S.J., Vogel, E.K.: The capacity of visual working van Merriënboer, J.J.G., Sweller, J.: Cognitive load
memory for features and conjunctions. Nature. 390, theory and complex learning: recent developments
279–281 (1997). https://doi.org/10.1038/36846 and future directions. Educ. Psychol. Rev. 17,
Luck, S.J., Vogel, E.K.: Visual working memory capacity: 147–177 (2005). https://doi.org/10.1007/s10648-
from psychophysics and neurobiology to individual 005-3951-0
differences. Trends Cogn. Sci. 17, 391–400 (2013). Ware, C.: Information Visualization, Third Edition: Per-
https://doi.org/10.1016/j.tics.2013.06.006 ception for Design. Morgan Kaufmann, Burlington
Miller, G.A.: The magical number seven, plus or (2012)
minus two: some limits on our capacity for Wei, S., Hu, K., Cheng, L., Tang, H., Du, W., Guo, C.,
processing information. Psychol. Rev. 63, 81–97 Pan, C., Li, M., Yu, B., Li, X., Chen, Y.V., Qian, Z.C.,
(1956). https://doi.org/10.1037/h0043158 Zhu, Y.M.: Crowd Analyzer: a collaborative visual
Miyake, A., Shah, P.: Models of Working Memory: analytic system. In: 2015 I.E. Conference on Visual
Mechanisms of Active Maintenance and Executive Analytics Science and Technology (VAST).
Control. Cambridge University Press, Cambridge pp. 177–178 (2015)
(1999) Weick, K.E.: Sensemaking in Organizations. SAGE,
Novotny M.: Visually effective information visualization Thousand Oaks (1995)
of large data. In: Proceedings of the 8th Central Euro- Wheeler, M.E., Treisman, A.M.: Binding in short-term
pean Seminar on Computer Graphics (CESCG 2004), visual memory. J. Exp. Psychol. Gen. 131, 48–64
pp. 41–48. CRC Press, Boca Raton (2004) (2002)
Ntuen, C.A., Park, E.H., Gwang-Myung, K.: Designing Wilken, P., Ma, W.J.: A detection theory account of change
an information visualization tool for sensemaking. detection. J. Vis. 4, 1120–1135 (2004). https://doi.org/
Int. J. Hum. Comput. Interact. 26, 189–205 (2010). 10.1167/4.12.11
https://doi.org/10.1080/10447310903498825 Wolfe, J.M.: Guided Search 2.0 A revised model of
Paas, F., Renkl, A., Sweller, J.: Cognitive load theory: visual search. Psychon. Bull. Rev. 1, 202–238 (1994).
instructional implications of the interaction between https://doi.org/10.3758/BF03200774
information structures and cognitive architecture. Zhang, W., Luck, S.J.: Discrete fixed-resolution repre-
Instr. Sci. 32, 1–8 (2004). https://doi.org/10.1023/B: sentations in visual working memory. Nature. 453,
TRUC.0000021806.17516.d0 233–235 (2008). https://doi.org/10.1038/
Pashler, H.: Familiarity and visual change detection. Per- nature06860
cept. Psychophys. 44, 369–378 (1988). https://doi.org/
10.3758/BF03210419
Pornstein, M.H., Krinsky, S.J.: Perception of symmetry
in infancy: the salience of vertical symmetry and
the perception of pattern wholes. J. Exp. Child Psychol. Cognitive Psychology Applied
39, 1–19 (1985). https://doi.org/10.1016/0022-0965 to User Experience in Video
(85)90026-8
Shneiderman, B.: The eyes have it: a task by data type
Games
taxonomy for information visualizations. In: IEEE
Symposium on Visual Languages, 1996. Proceedings. Celia Hodent
pp. 336–343 (1996) Epic Games, Cary, NC, USA
Sweller, J.: Some cognitive processes and their
consequences for the organisation and presentation
of information. Aust. J. Psychol. 45, 1–8 (1993).
https://doi.org/10.1080/00049539308259112 Synonyms
Sweller, J., van Merrienboer, J.J.G., Paas, F.G.W.C.:
Cognitive Architecture and Instructional Design.
Educ. Psychol. Rev. 10, 251–296 (1998). https://doi.
HCI; Human Factors; Human-Computer Interac-
org/10.1023/A:1022193728205 tion; UX; UX Design
Cognitive Psychology Applied to User Experience in Video Games 339

Definition Considering the Player’s Mind

The user experience (UX) entails a person’s per- The user experience of video games happens in
ceptions and interactions with a product or soft- the player’s mind (see Schell 2008). An important
ware (such as a video game) and the satisfaction perspective when considering video games’ UX is
and emotions elicited via this interaction. UX that the game designers and end players may
overall refers to an overarching discipline focused invoke different mental models. Norman C
on evaluation and improvement of users’ experi- described mental models in his seminal book
ence of a given product or software in The Design of Everyday Things (Norman 1988).
development. According to Norman, a system (such as a video
Cognitive psychology is a discipline dedicated game) is designed and implemented based on the
to understanding the human mind via mental pro- designer’s mental model of what the system
cesses such as problem solving, language, percep- should entail and how it should function. Players
tion, attention, and memory. then develop their own mental model of how they
think the game works through their interactions
with it, given their prior knowledge and expecta-
Introduction tions. The main objective of UX is to ensure that
users experience the game (the system image) the
The designer Donald Norman popularized the way the game developers intended, through
notion of user experience (UX) in the 1990s players’ perception of the game and their interac-
(Norman et al. 1995). Originating in the fields of tion with it. The developers have to adjust the
human factors and human-computer interaction, vision of the game in development to comply
UX as a discipline incorporates knowledge and with the limitations of the system (e.g., platform,
methodologies from behavioral sciences – includ- performance) and the limitations of game produc-
ing cognitive psychology – to evaluate the ease of tion resources (e.g., timeline, workforce). Simi-
use and emotions elicited from a product or sys- larly, the developers must comply with the
tem. Video game studios have increasingly turned capabilities and limitations of the human mind to
to this relatively new discipline to ensure that the offer a compelling experience to the players.
games they develop offer a compelling experience Playing a video game is a learning experience,
to the targeted players. The inclusion of UX con- from discovering the game to mastering its sub-
siderations in the design process saves rather than tleties. Information that the brain processes origi-
costs a studio money as it allows for more suc- nates from perceived input that then impacts the
cessful game development, contrary to some mis- memory of a subject. The quality of the pro-
conceptions (see Hodent 2015). According to cessing – and ultimately the quality of the reten-
game designer Tracy Fullerton, to design a game tion – depends highly on the attentional resources
is to create an “elusive combination of challenge, applied, which are also dependent on the emotions
competition, and interaction that players just call and motivation felt by the players. Thus, to
‘fun’” (Fullerton 2014, p. XIX). However, no improve the experience of the players, video
objective definition of fun has emerged, nor any game developers must take into account the per-
detailed parameters to attain it. UX offers a frame- ception, memory, and attention limitations of the
work to ensure that the experience intended is the brain, as well as the emotions and motivation felt
one ultimately felt by the target audience. UX by the players.
representatives use guidelines (heuristics) and
methodologies (user research) to anticipate and Perception
evaluate how end users interact with a specific Perception involves all the mental processes that
game, software, or service and the emotions allow us to sense our environment and construct
elicited via this interaction. our own mental representations of it. Thus, these
340 Cognitive Psychology Applied to User Experience in Video Games

processes are bottom-up proceeding from sensa- human mind perceives and organizes the environ-
tion to cognition (access to semantics) and also ment (Wertheimer 1923). For example, the Gestalt
top-down whereby cognition (i.e., previous law of proximity describes how elements that are
knowledge, expectation) impacts one’s sensa- close to one another are interpreted as belonging
tions. For example, the save icon (usually sym- to the same group. When considering the heads-
bolized by a floppy disk) is likely meaningless to up display (HUD) of a game, displaying the icons
young children who do not have a mental repre- and symbols representing features that are related
sentation for this object, until they learn what it next to each other enacts this law. Thus, it is what
symbolizes when using a computer. This example the end players subjectively perceive and under-
illustrates that perception is subjective. It varies stand about the game interface that matters, not
depending on the context in which the input is the reality of what the developers and designers
presented and on one’s previous knowledge or have implemented.
expectations. Therefore, game players or technol-
ogy users may understand a specific element dif- Memory
ferently than what the designer had intended. To Memory allows us to encode, store, and retrieve
ensure that the game menus and signs and feed- information and has been seen as comprised of
back will be understood as designed, it is impor- sensory memory, working memory, and long-term
tant to assess them with the end users during memory (Atkinson and Shiffrin 1971; Baddeley
usability tests whereby, for example, a sample of 1986). Sensory memory is part of perception and
target users are presented with icons and they have retains sensory information for a very short period
to explain what the icons look like and denote. of time (such as a fraction of a second) without it
Ideally, the form (shape) of an icon should cor- being consciously processed. For example, the
rectly inform the players about its function (what persistence of vision (e.g., fleeting images)
it does or how to interact with it). reflects sensory memory, which allows us to per-
The signs in a video game refer to all the ceive a 24-image-per-second display as an
perceptible cues that either urge the player to uninterrupted animation. Working memory is a
execute a specific action or inform the player of short-term component that allows for temporary
a system status. For example, a yellow exclama- storage (e.g., a few minutes) and manipulation of
tion mark above a nonplayer character (NPC) a very limited amount of new or already stored
encourages the player to interact with that information. This system maintains active mental
NPC. Other signs, such as a green bar or red representation necessary to perform a task. For
hearts, may inform the player of a system status, example, performing a mental calculation entails
such as the avatar’s health. Game feedback is the keeping numbers active in the working memory
reaction of the system to the player’s action. For while manipulating them. Working memory
example, an avatar may animate when the player requires substantial attentional resources (see the
uses the thumbstick or WASD keys. Another description of “attention” below) and therefore is
example is the ammunition count depleting very limited in duration and capacity. In fact,
when the player is shooting. Overall, all possible learning can be hampered and result in cognitive
interactions should have signs and feedback asso- load when work-memory limits are exceeded
ciated with them. These signs and feedback, and (Sweller 1994). Long-term memory is a
the user interface overall, should be perceptible multiple-system component that allows us to
and provide enough clarity to help the player store knowledge of events and skills (know-
understand the game mechanics. The Gestalt prin- how). Long-term memory has no known limits
ciples provide useful guidelines that should help and is seen as potentially storing information
designers organize the game interface in a way indefinitely although forgetting is possible.
that will be correctly understood by the players In 1885, the psychologist Hermann
(see Johnson 2010, for examples in software Ebbinghaus illustrated with the forgetting curve
design). Gestalt principles account for how the how memory retention declines exponentially
Cognitive Psychology Applied to User Experience in Video Games 341

with time (Ebbinghaus 1885). Retention of infor- conscious attention. Only an attention-grabbing
mation, especially if not engaging emotionally or event – such as a sudden loud sound or light
meaningful, can be very fragile. Some variables flash – can then draw attention away from the
have an impact on the strength and quality of the current information attended. When attention is
encoding and storage of information, such as the divided, for example, when driving while having
level of processing (the deeper the process the a conversion over the phone, it requires more
better the retention) and the amount of repetition cognitive load to process the different informa- C
over time. Not only the brain is prone to memory tion, therefore leading to more fatigue and mis-
lapses, but it can also distort memories. Because takes. In fact, the brain cannot usually multitask
of these limitations, developers cannot rely too efficiently; either one task or both are performed
heavily on players’ memories. Even if some infor- less efficiently, unless at least one of the tasks is
mation has been encoded via tutorials during the very simple or automatic (such as chewing gum
onboarding part of the game, it is likely going to while reading). Similarly, the more demanding a
fade with time unless used regularly. This is why it specific task is in terms of cognitive load (e.g.,
is generally a good practice to reduce to a mini- complex mental calculation), the less a subject can
mum the information that the players have to allocate mental effort to accomplish another task,
remember in order to enjoy the game (i.e., even though simple (such as pressing a button
mechanics, controls, objectives) and to give fre- when a red light goes off; cf. Kahneman 1973).
quent reminders, especially since a long time can Subsequently, the more attention is allocated to a
elapse between two gaming sessions. It is also task or information, the better it will be retained,
important to prioritize the information players therefore learned, as seen in the “Memory” sec-
have to learn and to distribute learning over tion above. Thus, it is critical to draw the players’
time. Lastly, the strength of retention can be attention to the elements that they need to learn.
increased if the players can learn by doing (see Given that all of our mental processes are using
Lesgold 2001) in a meaningful context – instead the same limited attentional resources, the devel-
of first reading tutorial texts and then doing. opers must mind the cognitive load the game
Therefore, it is a better practice to place the demands from the player, especially during the
players in a situation when they have to execute onboarding of a video game, when the players
a new action to accomplish an immediate goal. have a lot of new information to process.
For example, placing a chest beyond a hole is a When elements are unattended, they are likely
meaningful and active way to teach players about not perceived at all, in a phenomenon called
jumping and looting mechanics. inattentional blindness (Mack and Rock 1998).
This phenomenon was best illustrated in the
Attention well-known “gorilla” experiment (Simons and
Our senses are continuously assailed by multiple Chabris 1999) whereby subjects had to watch a
inputs from our environment. Attention entails video in which two teams of people were moving
allocating more cognitive resources to process around and passing basketballs. One team was
selected inputs while the others will be ignored wearing black shirts and the other team white
(selective attention). The brain’s attentional shirts. The subjects were asked to count basketball
resources being very limited, we do not method- passes made by players of the white team only. In
ically process all the available information from the middle of the video, a person in a black gorilla
the environment. Instead, attention works like a suit walked into the scene, pauses, and then
spotlight, focusing resources to process and retain walked off the scene. The results showed that
particular elements and neglecting the other most subjects, directing their attention into
inputs. For example, when in a loud and crowded counting the basketball passes from the white
cocktail party, one can pay attention to a specific team, missed the gorilla although quite prominent
conversation but cannot process all the other con- in the scene. This study explains why players,
versations at earreach; these are suppressed from when focused on a task, can stay blind to any
342 Cognitive Psychology Applied to User Experience in Video Games

other information conveyed at the same time. For player to experiment with the game environment
instance, if tutorial text information about the in a creative way). Relatedness involves primar-
health mechanic is displayed while the players ily the need to feel connected to others. Related-
are experiencing their first combat, they will likely ness in games is often addressed through
not process or even perceive that information as multiplayer features allowing players to interact
all their attention is focused on surviving their first with each other in real time or asynchronously,
enemy encounter. Therefore, it is preferable to via cooperative or competitive goals. Sustained
avoid displaying important information when the motivation and emotional connection both have
players are directing their attention to an impact on the enjoyment of a game. These
another task. components also have an impact on the learning
experience and the quality of information
Emotion and Motivation retention.
According to Norman (2005), “the emotional
side of design may be more critical to a product’s
success than its practical elements” (p. 5). The Usability and Gameflow, the Two
emotional aspect in video games is frequently Components of User Experience in Video
addressed through aesthetics, music, or narra- Games
tive. However, an important aspect of emotional
game design has to be considered as well: the To ensure a good video game user experience, it is
“game feel.” Game designer Steve Swink (2009) important to consider its usability and gameflow.
describes game feel as including “feelings of Making software – such as a video game – usable
mastery and clumsiness, and the tactile sensation means “paying attention to human limits in mem-
of interacting with virtual objects” (p. 10). ory, perception, and attention; it also means antic-
Accounting for the game feel involves carefully ipating likely errors that can be made and being
designing the camera, controls, and characters. ready for them, and working with the expectations
For example, if the camera of the game has a very and abilities of those who will use the software”
narrow field of view (FOV) it may give players a (Isbister and Schaffer 2008, p. 4). Usability is
feeling of claustrophobia, which would be inap- about removing or at least alleviating all the frus-
propriate for a peaceful exploration game. It trations and confusion the player could experience
could however be appropriate for a horror sur- while playing the game, if they are not intended by
vival game, depending on the game design design. Broad guidelines – heuristics – can be
intentions. used to attain usability. Many usability heuristics
Players’ motivation is another important var- have been developed, both in web (e.g., Nielsen
iable to consider when developing a game. 1994) and game design (e.g., Desurvire
According to Przybylski et al. (2010) “both the et al. 2004; Laitinen 2008). These heuristics take
appeal and well-being effects of video games are into account the human brain capabilities and
based in their potential to satisfy basic psycho- limitations in perception, attention, and memory
logical needs” (p. 154). Therefore, a game that described earlier. The gameflow component refers
satisfies basic psychological needs for compe- to how enjoyable and engaging the video game
tence, autonomy, and relatedness (c.f. Deci and is. It takes its roots from the notion of flow,
Ryan 1985) will more likely be engaging. Com- described by psychologist Mihaly
petence entails the players’ sense of mastery and Csikszentmihalyi as the optimal experience
feeling of progression towards clear goals (i.e., whereby “a person’s body or mind is stretched to
Nintendo’s Legend of Zelda series require its limits in a voluntary effort to accomplish some-
increasing mastery to progress in the game). thing difficult and worthwhile” (Csikszentmihalyi
Autonomy encompasses offering meaningful 1990, p. 3). The gameflow component offers a set
choices to the players and opportunities for self- of criteria, or heuristics, to improve the emotion
expression (i.e., Mojang’s Minecraft allows the response and motivation felt by the players, in an
Cognitive Psychology Applied to User Experience in Video Games 343

adaptation of the concept of flow into games Deci, E.L., Ryan, R.M.: Intrinsic Motivation and Self-
(Chen 2007; Sweetser and Wyeth 2005). By con- Determination in Human Behavior. Plenum, New
York (1985)
sidering both usability and gameflow heuristics, a Desurvire, H., Caplan, M., Toth, J.A.: Using heuristics to
UX framework can be developed to provide a evaluate the playability of games. In: Extended Abstracts
useful checklist for game developers (see Hodent CHI, pp. 1509–1512. ACM, New York (2004)
2014a, for an example of a UX framework applied Ebbinghaus, H.: Über das Gedchtnis. Untersuchungen zur
to game design).
experimentellen Psychologie. Duncker & Humblot,
Leipzig (1885)
C
Fullerton, T.: Game Design Workshop: A Playcentric
Approach to Creating Innovative Games. CRC Press,
Conclusion Boca Raton (2014)
Hodent, C.: Developing UX practices at Epic games. Pre-
sented at the 2014 game developers conference Europe,
To warrant an engaging and enjoyable user expe- Cologne. http://www.gdcvault.com/play/1020934/
rience, game developers need to consider human Developing-UX-Practices-at-Epic (2014a)
capabilities and limitations by adopting a UX Hodent, C.: Toward a playful and usable education. In:
Blumberg, F.C. (ed.) Learning by Playing: Video Gam-
framework (Hodent 2014b). Such framework is ing in Education. Oxford University Press, New York
taking into account the limitations of the human (2014b)
brain in perception, attention, and memory. It also Hodent, C.: 5 misconceptions about UX (User Experience)
considers the emotional response and motivation in video games. Gamasutra. http://gamasutra.com/blogs/
CeliaHodent/20150406/240476/5_Misconceptions_abo
felt by the players. It can be used during the ut_UX_User_Experience_in_Video_Games.php (2015)
development of a video game as a checklist to Isbister, K., Schaffer, N. (eds.): Game Usability. Elsevier,
ensure that the usability and gameflow guidelines Burlington (2008)
are respected, therefore increasing the chances of Johnson, J.: Designing with the Mind in Mind: Simple
Guide to Understanding User Interface Design Guide-
offering a compelling user experience to the lines. Elsevier, Burlington (2010)
targeted audience. A UX framework provides Kahneman, D.: Attention and Effort. Prentice Hall, Engle-
game developers with useful guidance to improve wood Cliffs (1973)
the quality of their game and ensure that their Laitinen, S.: Usability and playability expert evaluation.
In: Ibister, K., Schaffer, N. (eds.) Game Usability.
intended design is the one experienced by the Elsevier, Burlington (2008)
target audience. Lesgold, A.M.: The nature and methods of learning by
doing. Am. Psychol. 56, 964–973 (2001)
Mack, A., Rock, I.: Inattentional Blindness. MIT Press,
Cambridge, MA (1998)
Cross-References Nielsen, J.: Heuristic evaluation. In: Nielsen, J., Molich,
R.L. (eds.) Usability Inspection Methods. Wiley, New
York (1994)
▶ Accessibility of Virtual Reality for Persons Norman, D.A.: The Design of Everyday Things. Double-
with Disabilities day, New York (1988)
▶ Games and the Magic Circle Norman, D.A.: Emotional Design: Why We Love (or Hate)
Everyday Things. Basic Books, New York (2005)
▶ Game-Based Interventions in Public Health: Norman, D.A., Miller, J., Henderson, A.: What you see,
Exploiting the Engaging Factor of Gameplay some of what’s in the future, and how we go about
doing it: HI at Apple computer. In: Proceedings of CHI,
Denver (1995)
Przybylski, A.K., Rigby, C.S., Ryan, R.M.: A motivational
References model of video game engagement. Rev. Gen. Psychol.
14, 154–166 (2010)
Atkinson, R.C., Shiffrin, R.M.: The control of short-term Schell, J.: The Art of Game Design. Elsevier/Morgan
memory. Sci. Am. 225, 82–90 (1971) Kaufmann, Amsterdam (2008)
Baddeley, A.D.: Working Memory. Oxford University Simons, D.J., Chabris, C.F.: Gorillas in our midst:
Press, New York (1986) sustained inattentional blindness for dynamic events.
Chen, J.: Flow in games (and everything else). Commun. Perception 28, 1059–1074 (1999)
ACM 50, 31–34 (2007) Sweetser, P., Wyeth, P.: GameFlow: a model for evaluating
Csikszentmihalyi, M.: Flow: The Psychology of Optimal player enjoyment in games. ACM Comput. Entertain.
Experience. Harper Perennial, New York (1990) 3, 1–24 (2005)
344 Collaborative Engineering

Sweller, J.: Cognitive load theory, learning difficulty, Introduction

and instructional design. Learn. Instr. 4, 295–312
(1994)
Swink, S.: Game Feel: A Game Designer’s Guide to Vir- Virtual prototyping is a method in the process of
tual Sensation. Morgan Kaufmann Publishers, Amster- product development. It uses outputs of Computer
dam (2009) Aided Design (CAD) and Computer Aided Engi-
Wertheimer, M.: Untersuchungen zur Lehre der Gestalt II, neering (CAE) software to validate the design
Psychol. Forsch. 4, 301–350 (1923) Translation
published as Laws of organization in perceptual before making a physical prototype. Traditionally,
forms. In: Ellis, W.A. (ed.) Source Book of Gestalt it is done by using one or combination of more
Psychology, pp. 71–88. Routledge/Kegan Paul, than one 3D computer software to generate geo-
London (1938) metrical shapes and combining them in order to
test mechanical motions, functions, and fit. These
geometrical shapes represent parts of the project
and they could be manipulated by CAE software
to simulate the behavior of the part in the real
Collaborative Engineering world.
With the introduction of affordable consumer
▶ Collaborative Engineering and Virtual Pro- and enterprise head-mounted displays (HMD),
totyping Within Virtual Reality engineering groups started to study about imple-
mentation of virtual reality technology in order to
provide new pathways for engineers, designers,
and their customers to experience new products
Collaborative Engineering and development processes, without the need of
and Virtual Prototyping making physical prototypes and mock-ups.
Within Virtual Reality Virtual reality can provide any HMD equipped
engineer, designer, or customer to have realistic
Ozan Özkan1,2 and Özhan Tıngöy2 digital prototypes accessible in any place with
1
Augmea Simulation Technologies A.S., instant manipulation and assembly/disassembly
Istanbul, Turkey capabilities.
2
Department of Information Technologies,
Marmara University, Istanbul, Turkey
Background

Synonyms The product design and development process was

used to rely primarily on engineers’ experience
Collaborative engineering; Computer aided and judgement in producing an initial concept
design; Computer aided engineering; Pro- design. Engineers were using CAD software for
totyping; Virtual reality designing parts in basic geometrical shapes in a
3D environment; then designed parts were
converted to 2D engineering drawings in order
Definitions to produce them in Computer Numerical Control
(CNC) machines without any way to evaluate its
Computer Aided Design (CAD) and Computer performance in advance. The initial prototype was
Aided Engineering (CAE) are now benefiting highly unlikely to meet expectations. Engineers
from advantages of virtual reality systems in usually had to redesign the part multiple times
terms of visual collaboration between engineers, and weaknesses were revealed only in physical
designers, and customers, reducing prototyping testing.
costs and minimizing required design and Nowadays, customers are requesting more
development time. customized products in high-quality standards
Collaborative Engineering and Virtual Prototyping Within Virtual Reality 345

with affordable costs and shorter lead time. This

understanding causes a significant increase in
product variety and competition in the market.
Manufacturers are under pressure to reduce the
production time, cost and optimize products to
higher standards of performance and reliability.
Manufacturers are changing their design and C
manufacturing processes in order to keep up
with the customer demands. Latest progresses
revealed that having feedbacks from customers
in every step of design and manufacturing pro-
cess is highly important, even if the product is in
the idea phase (Tseng et al. 1998). This approach
enabled the utilization of concurrent engineer-
ing in design and manufacturing processes.
Today, with the help of immersive technologies
such as virtual reality and augmented reality,
engineers, designers, and customers are con-
stantly able to evaluate the product in function-
ality, reliability, and aesthetic dimensions and
give their feedback in order to change the prod-
uct design in the design phase without the need
of creating physical prototypes (Vosniakos
1998).

Collaborative Engineering and Virtual Prototyping

Utilizing Virtual Reality Within Virtual Reality, Fig. 1 HTC Vive usage with a
backpack PC
Virtual reality gives an opportunity to experience
any product in any phase with the support of register an arm, hand, and finger movements,
photorealistic visuals, dynamic lights and which gives the user an opportunity for manipu-
shadows, simulated physics and dynamics with- lating, assembling, or disassembling 3D objects in
out creating a physical prototype. Also, it is pos- the virtual world with collaboration.
sible to assemble, disassemble, and test Using an HMD-based virtual reality device is
mechanical motions of 3D CAD models in virtual also cost-effective over 3D projection systems
environment with the collaboration of multiple such as spherical or CAVE-type virtual environ-
participants such as designers, engineers, and cus- ments. These environments require special 3D
tomers. This can be done using only an HMD and projectors, projection screens, and a significant
a computer in any physical space. In order to use physical space. Devices such as 3D projectors
it, users need to put the HMD onto their heads and are very expensive for utilizing, operating, and
use controllers for manipulating objects as shown maintaining. Also, projection environments need
in Fig. 1. They also have an opportunity to use real a dedicated space, and it is not possible to use it as
life navigation such as walking and crouching, a mobile system. Today, HMDs are in the con-
which give an opportunity to users for walking sumer market with affordable prices and they only
around the object, looking closer to it from differ- require a gaming PC and no dedicated space at all.
ent angles in real time, as if looking to a physical Manufacturers can use HMD-based virtual reality
prototype. With the help of built-in or external systems for off-site demonstrations or collabo-
tracking systems, virtual reality systems can rated projects.
346 Collaborative Environments for Augmented and Virtual Reality Applications

In order to use a virtual reality system for or customer to have realistic digital prototypes
collaboration and virtual prototyping, required and collaboration space.
virtual world must be created with the help of
special software. Most of the features for a
development of such a virtual prototyping plat- Cross-References
form are available in game engines supported by
the game development industry, which has a ▶ Augmented Reality for Maintenance
value of 25.1 billion USD according to 2010
ESA reports. However, since game engines are
developed for making games, some of the spe- References
cific features are not available and some features
do not meet requirements of virtual prototyping Altundag, A.M., Edwards, L., Demirkan, C., Aksoy, O.:
“Oyun Motoru Kullanilarak Sanal Prototipleme
standards. As a solution to that problem, some
Altyapisi: Augmea Immersive Platform TM.”
virtual reality and simulation companies started EEMKON (2017)
to develop virtual prototyping platforms. One of Tseng, M.M., Jiao, J., Chuan-Jun, S.: Virtual prototyping
them is Augmea Immersive Platform (AIP), for customized product development. Integr. Manuf.
Syst. 9(6), 334–343 (1998)
which is developed by Augmea Simulation
Vosniakos, G.C.: Feature-based product engineering: a
Technologies A.S. and it uses the power of critique. Int. J. Adv. Manuf. Technol. 14(7), 474–480
state-of-the-art game engine, CryEngine (1998)
(Altundag et al. 2017). Software products like
AIP gives an opportunity to manufacturers for
creating virtual worlds in order to use the best
features of virtual prototyping technologies in Collaborative Environments
industry standards with cost- and time-effective for Augmented and Virtual
approach. Reality Applications

Federico Manuri1, Francesco De Pace2 and

Conclusion Andrea Sanna1
1
Dipartimento di Automatica e Informatica,
Utilization of the virtual reality technology for Politecnico di Torino, Turin, Italy
2
virtual prototyping gives an important opportu- Institute of Visual Computing and Human-
nity to manufacturers for experiencing the Centered Technology, TU Wien, Wien, Austria
design, assembling, and disassembling them in
order to evaluate the performance of parts or the
whole product without the need of creating Synonyms
physical prototypes. Designers and engineers
can work together for evaluation and collabora- Shared digital workspaces; Shared environments
tive design, even in different physical locations.
They are able to use full body movement, con-
trollers, and sensors in the virtual environment Definition
with the photorealistic visuals, dynamic lights,
shadows, and simulated physics. HMD-based Collaborative environments for augmented and
virtual reality devices are cost-effective over virtual reality applications are virtual environ-
3D projection systems, and they do not need ments which can be experienced by more than
dedicated installation spaces. With the introduc- one user to cooperate in a collaborative task.
tion of affordable HMD devices and special Collaborative environments can be designed to be
virtual prototyping software, virtual reality can compatible with augmented reality (AR) interfaces,
provide any HMD equipped engineer, designer, virtual reality (VR) interfaces or both.
Collaborative Environments for Augmented and Virtual Reality Applications 347

Introduction implementing virtual environments, whereas

many other 3D engines exist, either licensed
Human communication and collaboration through (e.g., CryEngine) or open source (e.g., Blender).
digital systems have been the focus of research for A 3D engine allows designers to create a virtual
more than 30 years. This research field is known environment, furnish it with virtual objects, script
as Computer Supported Cooperative Work the objects’ behavior, and design and develop the
(CSCW). Though CSCW can greatly benefit user interaction layer. C
from AR and VR, only recently AR and VR Once the virtual environment is available, the
have reached technological progress that enables second relevant technology necessary to deploy a
researchers to focus on exploiting these technolo- CE is the physical interface adopted to experience
gies to support collaboration through collabora- it. Milgram’s Reality-Virtuality Continuum
tive environments (CEs). Tech companies such as (Milgram and Kishino 1994), depicted in Fig. 2,
Microsoft, Vive and Apple are continuously is widely adopted to explain the design space of
developing new hardware and software to support mixed reality interfaces. Among the physical
AR and VR, and among the various application environment and the virtual one there is a contin-
fields, the creation of collaborative systems is uum of systems, which merge the physical and
widely accepted as one of the most valuable ones. virtual world, known as Mixed Reality:
The design of CEs, the technologies adopted to depending on the balance of physical and virtual
create and experience CEs, and the most common elements, it is possible to further distinguish
applications’ fields are discussed in this chapter. between augmented reality (AR) and augmented
virtuality (AV). An AR system adds virtual ele-
ments to a direct or indirect representation of the
Technologies real world, whereas an AV system adds real ele-
ments to a virtual environment.
Computer-generated CEs are based on 3D real- AR environments are experienced through either
time engines, which enable designers to represent video see-through or optical see-through devices
the digital contents to be displayed to the users. (Fig. 3). Video see-through devices record the phys-
Unity 3D (Fig. 1) and Unreal Engine are the most ical world through a camera and display it combined
known software adopted for designing and with the digital contents. These devices can be

Collaborative Environments for Augmented and Virtual Reality Applications, Fig. 1 Unity 3D user interface
348 Collaborative Environments for Augmented and Virtual Reality Applications

Collaborative
Environments for
Augmented and Virtual
Reality Applications,
Fig. 2 Milgram’s Reality-
Virtuality Continuum

Collaborative Environments for Augmented and Virtual Reality Applications, Fig. 3 An example of an AR
optical see-through device, the Magic Leap 1 headset. Photo by Bram Van Oost on Unsplash

Collaborative
Environments for
Augmented and Virtual
Reality Applications,
Fig. 4 An example of a VR
headset, the Oculus Quest
2 produced by Vive. Photo
by Remy Gieling on
Unsplash

further classified as desktop (e.g., a computer and a Overall, a CE can be designed to be compat-
webcam) or mobile (e.g., smartphones and tablets). ible with both VR and AR devices: whereas VR
VR environments are experienced through users will experience a complete virtual world,
either traditional displays, which can be classified AR users will see only a subset of virtual ele-
as mobile or desktop, or immersive virtual reality ments, which do not inhibit the perception of the
headsets, which consist of closed helmets with reality. This can be obtained by scripting accord-
two displays in front of the user’s eyes, thus ingly the behavior of the elements of the virtual
detaching people from the real environment environment, depending on what the AR users
(Fig. 4). should see.
Collaborative Environments for Augmented and Virtual Reality Applications 349

Collaborative
Environments for
Augmented and Virtual
Reality Applications,
Fig. 5 CSCW Time-space
matrix

Design

The first step in the design of a CE is to define the

level of collaboration required or expected for a
given task. Since CEs for AR and VR are a spe-
cific type of CSCW, the time-space matrix pro-
posed by Johansen (1988) depicted in Fig. 5
provides a clear representation to understand pos-
sible typologies and requirements for collabora-
tive activities. The collaboration can be
synchronous or asynchronous, whereas the users
may interact in the same physical space or they
may be located in different places.
Benford et al. (1998) proposed a more complex
taxonomy, which combines together the time-
space matrix and the reality-virtuality continuum Collaborative Environments for Augmented and Vir-
based on three dimensions: artificiality, transpor- tual Reality Applications, Fig. 6 Artificiality and trans-
portation dimensions of Benford’s taxonomy
tation and spatiality (Fig. 6). Artificiality repre-
sents the ratio of virtual content and real content in
the collaborative environment, e.g., a completely environment involves a 3D Cartesian shared spa-
synthetic world for a VR system or a physical tial frame.
world enriched with digital assets for an AR The time-space matrix and Benford’s taxon-
system. omy are valuable tools to understand the different
Transportation defines how much the user is types of collaborative environments. Moreover, it
detached from its environment and moved to a is possible to talk about symmetric collaboration
remote environment. This feature should not be if all the users experience the shared environment
confused with Artificiality, since the collabora- by the same level of transportation, spatiality, and
tion can either take place in the user environment artificiality. Examples of the symmetric collabo-
or in a remote environment through telepresence, rative environment are two users located in the
and in both cases, AR and VR would not be same space and collaborating through AR, or two
involved since the Artificiality would be at the users experiencing the same virtual environment
bottom level. Spatiality defines how much the through immersive VR. Asymmetric collabora-
collaboration involves a common spatial frame. tion instead takes place when at least one of the
For example, a collaboration through a chat box three dimensions of Benford’s taxonomy differs
would have no spatial reference, the usage of a from one user to another. An example could be a
2D dashboard would consist of a limited shared user who interacts in his/her physical space
spatial frame, whereas a shared virtual through an AR device and a remote user
350 Collaborative Environments for Augmented and Virtual Reality Applications

collaborating into a digital twin of the real envi- several actors that may be positioned in different
ronment through immersive VR. physical locations. Thanks to technological
advancements, CEs can effectively exploit VR
and AR technologies, thus providing physical
Application Fields embodiment, immersive experiences, and high
levels of interaction. Users can obtain an imme-
CEs are broadly used in several domains and the diate feedback on the design process regardless
most important ones are the following: collabora- of the physical location of the involved actors.
tive design, education and training, industry, med- Blueprints, construction plans, and design
ical scenarios, tourism, and cultural heritage. sketches can be displayed in the collaborative
Motivation for adopting CEs and examples are environments allowing users to provide inputs
introduced and discussed for each one of these and changes that are applied on the fly to the
research fields in the following sections. virtual assets. Figure 7 shows an example of a
collaborative meeting done using AR interfaces.
• In the Architecture, Engineering, Construction,
and Operation (AECO) area, collaboration and Representative works are (Lin et al. 2015;
interaction are of primary importance, and archi- Ibayashi et al. 2015). Lin Liu Tsai et al. (2015)
tects, engineers, and subcontractors must collab- proposed a co-located, synchronous AR system to
orate closely during each design phase. The improve discussions related to construction pro-
project team should make high-quality decisions cesses. Public data can be visualized using sta-
and innovations by combining the experience tionary displays whereas private information is
and expertise of various professionals. As the directly conveyed to the mobile devices of the
complexity of the projects increases, the involved users. The stationary display consists of
corresponding amount of data and information a tabletop interface with a dedicated personal
increases too, thus potentially generating an computer whereas the mobile devices are
uncontrolled working environment. Further- represented by Android mobile tablets. Data
more, in AECO scenarios, operations and displayed on the stationary display can be modi-
resources are frequently distributed among fied via tangible interaction paradigms, that is, the

Collaborative Environments for Augmented and Virtual Reality Applications, Fig. 7 An example of a CEs for
AECO seen through AR. Photo by Patrick Schneider on Unsplash
Collaborative Environments for Augmented and Virtual Reality Applications 351

users can manipulate the virtual information by • Education and training can greatly benefit from
using paper markers tracked with an external cam- collaborative environments. By presenting
era. The authors compared their system with tra- educational content in the 3D virtual space,
ditional paper-based approaches, and the main learners can easily and effectively understand
outcomes show a great reduction in the task com- structured and complex subjects, especially the
pletion time. Regarding architecture, Ibayashi ones coming from the mathematical or engi-
et al. (2015) presented a synchronous VR system neering domains. Furthermore, by exploiting C
to improve the design phase of architecture participatory design and gamification tech-
sketches. Several users can interact with the niques, users can improve their knowledge by
same virtual sketches by using two different being directly involved in the learning process.
modalities: a first-person view (by an immersive In collaborative environments, two or more
VR device) to see the fine local details and a top- users collaborate and learn together to achieve
down view (by a tabletop interface) to have an a common goal (Laal 2013). Due to this strong
overview of the entire project. One interesting link among learners, users are responsible for
capability of the proposed system is the so-called their own learning process and that of others
“God-like” interaction technique. When the users (Doymus 2007; Gokhale 1995) (a representative
are touching a specific object of interest on the example is shown in Fig. 8).
tabletop interface, the VR users can clearly detect
the same object of interest by visualizing a 3D Some representative works can be found in
representation of a human hand that pinpoints the (Punjabi et al. 2013; Pareto 2012). Punjabi et al.
objects. Although the system has been described (2013) presented CrowdSMILE, a remote, asyn-
as a co-located scenario, it can be easily extended chronous system to effectively handle learning
to a remote one. data coming from different sources and tools.

Collaborative Environments for Augmented and Virtual Reality Applications, Fig. 8 Usage of VR headsets for
education. Photo by stem.T4L on Unsplash
352 Collaborative Environments for Augmented and Virtual Reality Applications

Contents can be explored using desktop and hand- graphical representation of numbers can greatly
held interfaces, which are used by three different assist the students to accomplish mathematical
components: (i) POI Explorer Application, problems they had never been able to solve.
(ii) Social Publisher Client Application, and
(iii) CrowdSMILE Server. The first component • Industrial scenarios have been also deeply con-
runs on Android mobile devices integrating AR sidered and evaluated. CEs have been mainly
technologies to display learning contents related used for Assembly-Repair-Maintenance (ARM)
to the physical environment. The second compo- tasks by leveraging the traditional local-
nent allows the users to publish new learning unskilled remote-skilled interaction paradigm.
material that will be then explored by the POI The remote expert tries to help the local
Explorer Application. The users can add new unskilled user through a series of procedures
location-based learning material by means of and both users interact in the collaborative envi-
handheld and desktop interfaces. The third com- ronment by using a combination of AR and VR
ponent provides the main functionalities of the devices. In such environments, conveying
system allowing the exchange of data between effective instructions becomes of primary
the remaining components. The system has been importance, and researchers have devoted
evaluated through a series of user studies showing great effort to understand which virtual meta-
high levels of usability. Pareto (2012) proposed a phors should be used to improve ARM tasks.
co-located, synchronous environment to improve Early researches have explored the use of
the learnability of mathematical concepts for chil- abstract metaphors, such as virtual arrows or
dren with intellectual disabilities. Users can play a generic shapes controlled by the remote user.
series of 2D games to foster reasoning skills and More recently, the visualization of human body
strategic thinking. The games can be experienced parts by the remote expert has been researched,
using desktop interfaces, interactive whiteboards, showing that displaying user’s hand gestures in
and interactive spatial AR systems. The system the local user real environment can improve task
has been assessed through a series of user studies awareness and decreases cognitive load (Fig. 9).
and the main results show that the gameplay can As an example, Huang et al. (2018) evaluated
improve the students’ “number-sense” whereas the the effectiveness of sharing hand gestures and

Collaborative Environments for Augmented and Virtual Reality Applications, Fig. 9 An example of a virtual
environment in an industrial context experienced through a VR headset. Photo by XR Expo on Unsplash
Collaborative Environments for Augmented and Virtual Reality Applications 353

sketches made by a remote user to help a local the visualization of a 3D environment, remote
unskilled operator. The two users interact by expert medical teams can provide instruction to
using an immersive VR device (remote opera- local unskilled doctors during medical proce-
tor) and a handheld interface (local operator) dures. As an example, Sirilak et al. (Sirilak and
connected on the same Local Area Network. Muneesawang 2018) proposed a remote, syn-
The main results show that by sharing both chronous AR system to support local doctors
hand gestures and sketches the local users can during medical operations. The local doctor can C
complete the tasks with less time with respect to visualize by a wearable AR device patients’ data
sharing only hand gestures. The independence captured through dedicated sensors. At the same
of the view has been also deeply investigated. time, remote specialists can visualize the local
The remote user could visualize the local scene scene captured by the cameras of the wearable
by seeing the environment from the local user devices allowing them to collaborate with the
perspective (dependent view) or from arbitrary local doctors. Finally, the emulation of surgery
views (independent view), and some operations executed in VR can be linked to the
researchers have demonstrated the advantages data visualization in AR (Fig. 10).
of the independent view with respect to the
dependent one (Kim et al. 2018).
• CEs for tourism and cultural heritage scenarios
have been also researched and analyzed. Usu-
• In the medical context, CEs can foster commu- ally, these systems are used to draw the atten-
nication between patients and medical special- tion of the users towards points or objects of
ists. It is expected that medical costs will be interest; therefore, it becomes of primary
increasingly reduced using effective CEs. importance providing effective spatial cues to
Nonetheless, several important features, such make sure the users are not disoriented or get
as latency and tracking accuracy, have not lost. A typical scenario involves a local AR
been properly evaluated in this context yet. user who is physically walking through an
Similar to the industrial scenarios, by exploiting archaeological site or museum and a remote

Collaborative Environments for Augmented and Vir- environment. Right image: an AR interface can be used
tual Reality Applications, Fig. 10 Left image: a surgi- to display in real-time patient’s data. Images from (Jo et al.
cal operation can be emulated in an immersive VR 2021)
354 Collaborative Environments for Augmented and Virtual Reality Applications

Collaborative Environments for Augmented and Virtual Reality Applications, Fig. 11 Usage of VR headsets in a
cultural heritage context. Photo by Lucrezia Carnelos on Unsplash

guide who can add spatial virtual cues and Cross-References

instructions in the real environment (Fig. 11).
Due to their wide adoption, handheld devices ▶ Augmented Learning Experience for School
are still much more used in these scenarios Education
although head-mounted displays are increas- ▶ Augmented Reality for Human-Robot Interac-
ingly researched and studied. Chen Lee Swift tion in Industry
et al. (2015) proposed a remote, synchronous ▶ Augmented Reality for Maintenance
system to support cave explorations. Local ▶ Augmented Reality in Image-Guided Surgery
users can map the real cave creating a virtual ▶ Collaborative Engineering and Virtual Pro-
reconstruction of the archeological site by totyping Within Virtual Reality
using AR wearable devices. Specifically, a ▶ Foundations of Interaction in the Virtual Real-
local user, wearing a Microsoft HoloLens ity Medium
device, produces a virtual representation of ▶ Immersive Visualizations Using Augmented
the cave whereas a remote user can either inter- Reality and Virtual Reality
act with a Skype call or he/she can add virtual ▶ Interaction with Mobile Augmented Reality
annotations by means of a tablet device, thus Environments
fostering the cultural experience. ▶ Interactive Augmented Reality to Support
Education
For further information, interested readers may ▶ Mixed Reality
refer to (Ens et al. 2019; de Belen et al. 2019). ▶ Spatial Perception in Virtual Environments
Collectable Card Game 355

▶ User-Centered Design and Evaluation Method- Lin, T.H., Liu, C.H., Tsai, M.H., Kang, S.C.: Using aug-
ology for Virtual Environments mented reality in a multiscreen environment for con-
struction discussion. J. Comput. Civ. Eng. 29(6),
▶ Virtual Reality Applications in Education 04014088 (2015)
▶ Virtual Reality Game Engines Milgram, P., Kishino, F.: A taxonomy of mixed reality
visual displays. IEICE Trans. Inf. Syst. 77(12),
1321–1329 (1994)
References
Pareto, L.: Mathematical literacy for everyone
using arithmetic games. In: Proceedings of the
C
9th International Conference on Disability, Vir-
Benford, S., Greenhalgh, C., Reynard, G., Brown, C., tual Reality and Associated Technologies, vol.
Koleva, B.: Understanding and constructing shared 9, pp. 87–96. University of Readings, Reading
spaces with mixed-reality boundaries. ACM Trans. (2012)
Comput.-Hum. Interact. (TOCHI). 5(3), Punjabi, D.M., Tung, L.P., Lin, B.S.P.: CrowdSMILE:
185–223 (1998) A crowdsourcing-based social and mobile inte-
Chen, H., Lee, A. S., Swift, M., Tang, J. C.: 3D collabora- grated system for learning by exploration. In:
tion method over HoloLens™ and Skype™ end points. 2013 IEEE 10th International Conference on Ubiq-
In: Proceedings of the 3rd International Workshop on uitous Intelligence and Computing and 2013 IEEE
Immersive Media Experiences, pp. 27–30 (October 10th International Conference on Autonomic and
2015) Trusted Computing, pp. 521–526. IEEE, New York
de Belen, R.A.J., Nguyen, H., Filonik, D., Del Favero, D., (2013)
Bednarz, T.: A systematic review of the current state of Sirilak, S., Muneesawang, P.: A new procedure for advanc-
collaborative mixed reality technologies: 2013–2018. ing telemedicine using the HoloLens. IEEE Access. 6,
AIMS Electron. Electr. Eng. 3(2), 181–223 (2019) 60224–60233 (2018)
Doymus, K.: Effects of a cooperative learning strategy on
teaching and learning phases of matter and one-
component phase diagrams. J. Chem. Educ. 84(11),
1857 (2007)
Ens, B., Lanir, J., Tang, A., Bateman, S., Lee, G.,
Piumsomboon, T., Billinghurst, M.: Revisiting collab- Collaborative Problem
oration through mixed reality: The evolution of group-
ware. Int. J. Hum.-Comput. Stud. 131, 81–98 (2019) Solving
Gokhale, A.A: Collaborative learning enhances critical
thinking. J. Technol. Educ. 26, 17–22 (1995). https:// ▶ Challenge-Based Learning in a Serious Global
doi.org/10.1300/J123v26n01_06 Game
Huang, W., Billinghurst, M., Alem, L., Kim, S.:
HandsInTouch: Sharing gestures in remote collabora-
tion. In: Proceedings of the 30th Australian Conference
on Computer-Human Interaction, pp. 396–400
(December 2018)
Ibayashi, H., Sugiura, Y., Sakamoto, D., Miyata, N., Tada,
M., Okuma, T., . . . Igarashi, T.: Dollhouse vr: A multi- Collaborative Robot
view, multi-user collaborative design workspace with Augmented Reality
vr technology. In: SIGGRAPH Asia 2015 emerging
technologies, pp. 1–2 (2015) ▶ Augmented Reality for Human-Robot Interac-
Jo, Y.J., Choi, J.S., Kim, J., Kim, H.J., Moon, S.Y.: Virtual
Reality (VR) simulation and Augmented Reality tion in Industry
(AR) navigation in orthognathic surgery: A case report.
Appl. Sci. 11(12), 5673 (2021)
Johansen, R.: Groupware: Computer support for business
teams. The Free Press, New York (1988)
Kim, S., Billinghurst, M., Lee, G.: The effect of collabo-
ration styles and view independence on video-mediated Collectable Card Game
remote collaboration. Comput. Supported Coop. Work
(CSCW). 27(3), 569–607 (2018)
Laal, M.: Positive interdependence in collaborative learn- ▶ Hearthstone: A Collectable Card Game
ing. Procedia Soc. Behav. Sci. 93, 1433–1437 (2013) Through the Lens of Problem Solving
356 Collective Intelligence

bottleneck for several applications (Moll et al.

Collective Intelligence 2014), especially, for real-time applications
(Pabst et al. 2010; Dinas and Bañón 2015).
▶ Gamification in Crowdsourcing Applications

Classification of Collision Detection

Approaches
Collision Detection
CD problem can be classified mainly into four
Simena Dinas approaches: static, pseudodynamic, dynamic
Departamento de Electrónica y Ciencias de la (Basch 1999), and kinetic. However, another divi-
Computación, Pontificia Universidad Javeriana sion found in the literature is only based on static
Cali – Colombia, Cali, Valle, Colombia and dynamic approaches (Ericson 2005). In the
static approach, the collision is queried in a par-
ticular object position; then, time and usually tra-
Definition jectories are not relevant; the object may change
the position randomly, and, consequently, queries
Collision detection is the process of determining are independent (Weller 2013). This approach is
computationally if two or more objects intersect, used to refine and evaluate the performance time
the time, and the parts of the objects that are for CD algorithms; besides, the static CD is the
involved in the intersection. first step of a collision detector (Arcila 2008).
Commonly, pseudo-dynamic CD approaches
check static interference tests in short intervals
Introduction of time; they extended the static CD by adding
trajectories and times and applying motion (Dinas
Collision detection (CD) concerns to determining et al. 2010). In the literature, temporal coherence
if the intersection between a pair of objects hap- is the use of preview steps information in a colli-
pens, and after a positive answer, to determining sion detector. It takes advantage of small changes
when, and where it happens (Ericson 2005). If a of the objects position to reduce calculations
pair of objects intersect must give information based on the similarity between consecutive
about the existence of the intersection, whereas motions.
when they intersect must give temporal informa- Dynamic CD approaches take advantage of
tion about the time of contact. Finally, where the swept volumes to check the collision, and it
objects intersect must give information about depends on initial values. There are two types of
the area of each object that is involved in the initial information: the “a posteriori” in which the
intersection. objects may have a trajectory to navigate through,
For applications on robotic and graphic com- then the time and the trajectory define the object
puter areas, it is important to model and simulate motion. As a result, the collision is detected after it
the physics of the objects, thus, these areas have happens. In contrast, the “a priori” dynamic
used highly CD approaches whereas, specific approach is able to predict the collision before it
areas using it include physical-based, surgical, happens.
dynamic and cloth simulations, motion planning, Pseudo-dynamic and dynamic approaches deal
molecular modeling, virtual environments, obsta- with oversampling and undersampling problems.
cle avoidance, virtual prototyping, virtual assem- Oversampling happens when there is a lot of
bly, computer animation, among others (Tang resource spent into collision evaluations; as a
et al. 2010; Wong 2011; Zhao et al. 2013; Moll result, there are unnecessary calculations. In con-
et al. 2014; Du et al. 2015; Wang et al. 2018). trast, undersampling tests use fewer resources
Because of the computational complexity, CD is a than required to detect collisions; then, some
Collision Detection 357

collisions get lost (Basch 1999). Finally, kinetic parent node, and (iii) the bounding volume should
approaches take advantage of predicates and cer- fit the original node as tightly as possible with a
tificates functions to determine the CD on time. high degree of accuracy for the original model.
A characterization introduced by Zhang et al. However, two opposite criteria that guide the
(2006) divides CD approaches into discrete, selection of bounding volumes are: they should
which checks statically collisions between mov- be as tight as possible, and the intersection test for
ing objects based on discrete parameter intervals two of them should be as efficient as possible C
(Pabst et al. 2010; Zhao et al. 2013), and contin- (Nguyen 2006).
uous which tries to determine the time of contact
(ToC) as exactly as possible (Tang et al. 2010;
Wong 2011; Du et al. 2015). Continuous CD is a Collision Detection Objects
problem highly explored; the most common tech-
niques are the algebraic solution, the swept vol- According to the changes on the objects after the
ume, the bisection, the Minkowsky sum, the collision, the methods can be classified into two
conservative advancement, and the kinetic data categories: rigid-body models, which do not
structure approaches (Zhang et al. 2006; Tang change after colliding (Tang et al. 2013), and
et al. 2010; Du et al. 2015). deformable models or soft-bodies, which change
after colliding (Tang et al. 2010; He et al. 2015).
By exploiting the temporal coherence, models can
Collision Detection Phases be deformable along the time, even though they
do not collide with others, for instance, animated
CD methods are commonly divided into two objects like animals and people, cloth, among
phases the broad-phase and the narrow-phase. others. The collision that involves soft bodies
(i) Broad-phase methods determines the pair of and rigid bodies usually calculates the penetration
objects involved in a potential collision and depth and the area of collision of the objects (Zhao
(ii) narrow-phase methods check exact collision et al. 2013).
for every pair identified as suspected to be in a
collision by the broad-phase (Avril et al. 2014;
Wang et al. 2018). Moreover, potential collisions Cross-References
identified in broad-phase are not always colli-
sions; narrow-phase checked the existence of ▶ Crowd Evacuation Using Simulation
collision. Techniques
Broad-phase avoids highly expensive compu-
tations for faraway bodies (Weller 2013). An all-
pair test or a brute force method calculates exhaus- References
tively the collision between each pair of objects.
Narrow-phase checks each pair for intersection Arcila, O.: Nuevas Representaciones Dobles (Externas e
Internas) en Detectores de Colisiones Jer´arquicos.
between pairs involved selected as potential col-
PhD thesis, Facultad de Ingeniera, Universidad del
liders in the broad-phase (Weller 2013; Avril et al. Valle, Cali (2008)
2014). Convex objects, spatial partitioning, and Avril, Q., Gouranton, V., Arnaldi, B.: Collision detection:
bounding volume hierarchies are the most impor- Broad phase adaptation from multi-core to multi-GPU
architecture. J. Virtual Real. Broadcast. 11(6), 1–13
tant narrow-phase techniques (Wang et al. 2018). (2014)
Bounding volume hierarchies are widely used Basch, J.: Kinetic Data Structures. PhD thesis, Stanford
in the literature. The three requirements for University, Stanford (1999)
bounding volume hierarchies are: (i) each level Dinas, S., Bañón, J.M.: A literature review of bounding
volumes hierarchy focused on collision detection –
in the hierarchy represents a tighter fit than its
revisin de literatura de jerarqua volmenes acotantes
parents, (ii) a set of child nodes in the hierarchy enfocados en deteccin de colisiones. Ing. Compet. 17,
depicts the same part of the object covered by its 63–76 (2015)
358 Color

Dinas, S., Arcila, O., Bañón, J.M.: Un detector de

colisiones dina´mico basado en esferas exteriores e Color Detection
interiores. In: Quinto Congreso Colombiano de
Computacio´n – 5CCC (2010)
Du, P., Zhao, J.-Y., Pan, W.-B., Wang, Y.-G.: GPU accel- ▶ Color Detection Using Brain Computer
erated real-time collision handling in virtual disassem- Interface
bly. J. Comput. Sci. Technol. 30(3), 511–518 (2015)
Ericson, C.: Real-Time Collision Detection. The Morgan
Kaufmann Series in Interactive 3-D Technology.
Morgan Kaufmann, Burlington (2005)
He, L., Ortiz, R., Enquobahrie, A., Manocha, D.: Interac-
tive continuous collision detection for topology chang- Color Detection Using Brain
ing models using dynamic clustering. In: Proceedings
of the 19th Symposium on Interactive 3D Graphics and
Computer Interface
Games, i3D ’15, pp. 47–54. ACM, New York (2015)
Moll, M., Sucan, I.A., Kavraki, L.E.: An extensible Yinsheng Chen
benchmarking infrastructure for motion planning algo- Liverpool John Moores University,
rithms. CoRR, abs/1412.6673 (2014)
Nguyen, A.: Implicit Bounding Volumes and Bounding
Liverpool, UK
Volume Hierarchies. PhD thesis, Stanford University,
Stanford (2006)
Pabst, S., Koch, A., Straer, W.: Fast and scalable CPU/GPU Synonyms
collision detection for rigid and deformable surfaces.
Comput. Graph. Forum. 29(5), 1605–1612 (2010)
Tang, M., Manocha, D., Tong, R.: Fast continuous colli- BCI; Brain Computer Interface; Color detection;
sion detection using deforming non-penetration filters. EEG signal
In: I3D ’10: Proceedings of the 2010 ACM
SIGGRAPH Symposium on Interactive 3D Graphics
and Games, pp. 7–13. ACM, New York (2010)
Tang, M., Manocha, D., Kim, Y.J.: Hierarchical and con- Definitions
trolled advancement for continuous collision detection
of rigid and articulated models. IEEE Trans. Vis. Color plays a critical role in information deliv-
Comput. Graph. 99(PrePrints), 1 (2013)
Wang, X., Tang, M., Manocha, D., Tong, R.: Efficient
ery in this vivid world. Capable of detecting
BVH-based collision detection scheme with ordering imaginary colors can extend and enhance the
and restructuring. Comput. Graph. Forum current business model as well as the entertain-
(Proc. Eurographics 2018). 37(2), 1–13 (2018) ment areas. In order to efficiently determine the
Weller, R.: A brief overview of collision detection. In: New
Geometric Data Structures for Collision Detection and
imaginary color, essential knowledge of EEG
Haptics. Springer Series on Touch and Haptic Systems, signals, classification of the signals, and how
pp. 9–46. Springer International Publishing, New York to handle these signals are briefly discussed in
(2013) this article. BCI applications are mainly divided
Wong, S.-K.: Adaptive continuous collision detection for
cloth models using a skipping frame session. J. Inf. Sci.
into three categories: active BCI, reactive BCI,
Eng. 27(5), 1545–1559 (2011) and passive BCI. In these days, most research
Zhang, X., Lee, M., Kim, Y.J.: Interactive continuous purpose devices are noninvasive BCIs. EEG
collision detection for non-convex polyhedra. Vis. signals are commonly considered by four pri-
Comput. 22, 749–760 (2006)
Zhao, J., Ye, J., Li, J.: Resolving cloth penetrations with
mary bands that carry different information. It is
discrete collision detection. In: 2013 International Con- necessary to denoise the captured brainwaves
ference on Computer-Aided Design and Computer prior to analysis and processing, Linear filter,
Graphics (CAD/Graphics), pp. 443–444 (2013) Bandpass filter, and Independent Component
Analysis, three common approaches of
denoising are reviewed. Feature extraction and
classification methods of brain signals are
Color briefly introduced with former experiment
examples. Finally, advantages and adoption
▶ Color: Pixels, Bits, and Bytes preferences of methods are discussed
Color Detection Using Brain Computer Interface 359

functionally. This article aims to provide a rel- Active BCI: An active BCI application pro-
atively comprehensive walkthrough of basic duces its final output from brain activity which is
knowledge of brain signal processing towards directly consciously controlled by the user such as
detection of imaginary colors. using mental command, thinking to control hands,
and execute actions. Such type of brain activity is
independent from external events to control a
Introduction BCI-application. C
Reactive BCI: A reactive BCI application pro-
BCI is generally referred to Brain Computer duces its final output from brain activity arising in
Interface or Brian Computer Inter-action, reaction to external stimuli such as viewing pic-
which enrich the communication method as tures or particular objects. Such type of brain
well as the information delivery between activity is indirectly modulated or reacted by non-
human-being and machine. Brain signals are subjective consciousness of user to control a
captured through BCI devices and then trans- BCI-application.
ferred to computer for analysis and processing. Passive BCI: A passive BCI application
BCI devices are mainly divided into two cate- produces its outputs by analyzing arbitrary
gories: invasive BCI devices are directly brain activity without specific events for
implanted into people’s brain and medically enriching a human-computer interaction
targeting restoration of fundamental function- with implicit information that user may not
ality; noninvasive BCI devices are easy to wear aware from the subjective consciousness such
and widely used in various applications. Non- as the degree of engagement and relaxing
invasive BCI devices avoid the surgery risk and status.
are relatively portable, headsets such as Emotiv Color has an important role in digital society
EPOC+ and Neurosky MindWave are popular to present additional information to contents that
in researchers. However, the draw-back of needs emphasizing and have specific intentions.
being external device is the relatively poor People coming from different backgrounds and
effectiveness of signal acquirement compared countries could have various understanding on
to invasive BCI devices. There are many par- particular colors but commons can be observed.
ticularly interesting, entertaining wearables Three primary colors, red, green, and blue are
such as the “necomimi” project (the Necomimi usually applied in color-related researches.
Emotion Con-trolled Brainwave Cat Ears These three colors are deemed as the ideal pri-
Headband), where the headset has two catlike maries for additive color mixing with the light-
ears that are programmed to wave based on the ing sources (Rossing and Chiaverina 1999). Red
wearer’s emotion that evaluated through the associates with energy, danger, strength, power,
brain signals captured from the brain wave sen- determination and passion, desire, and love.
sor integrated into the headset (Kulkarni and Green denotes the nature which usually symbol-
Bairagi 2018). izes growth, harmony, freshness, and passing.
BCI applications have been deployed into var- Blue is the color of the sky and sea and com-
ious aspects of world such as home-automation, monly associated with depth and stability as well
lie-detection, brain fingerprinting, trust assess- as safety. Color detection models are mostly
ment, sleep stage recognition, disease observa- categorized into reactive BCI applications as
tion, and neurorehabilitation. BCI applications most applications require participants to be stim-
could be macroscopically divided into three ulated by a number of colors in order to achieve
main types that differ in the primary characteris- initial calibrations before the actual usages.
tics of BCI-interactive models in various applica- Capability of recognizing colors through brain
tions. Interaction and design models are hence signals could enable researchers to understand
differentiated in numerous researches in the BCI deeper and further flexibilities available to
field. developers.
360 Color Detection Using Brain Computer Interface

Methods from various part of the brain. It defines the place-

ment of electrodes that should be adhered to the
EEG Overview calculated distance between nasion and inion for a
Noninvasive Brain Computer Interfaces are usu- particular person. The separation of electrode
ally based on the Electroencephalography (EEG), placement is normally defined by particular EEG
the electrophysiological monitoring approach to device depending on the number of electrodes
record electrical activity, the communications available. In general, a 10% or 20% of the refer-
happened between brain neurons inside human’s ence distance is utilized, and in circumstances that
brain. Extensive researches undergoing in medi- a greater granularity (resolution in other words) is
cation and science have the root from Liverpool in required, the placement distance of electrodes are
1875, a physician practicing, Richard Caton who further shrink to 10% (Chong et al. 2007).
demonstrated the presence of electrical currents in Usually a greater granularity demands more
the brain (Caton 1875); the first EEG recording of electrodes to be set up into the cluster that sepa-
human is achieved by the German physiologist rated on the scalp. A typical prototype of 10/20
and psychiatrist Hans Berger who afterwards placement system is shown in Fig. 1. The nomen-
extended the research and invented the electroen- clature of electrodes that placed by the 10/20
cephalogram which is described as one of the placement system has particular rules that corre-
most outstanding advancement in the history of sponds to identifications of area or regions of
clinical neurology (Haas 2003). brain that electrodes is reading from such as:
Unlike invasive EEG devices that directly Frontalpolar (Fp), Frontal (F), Temporal (T),
planted into the brain to monitor brain activities Occipital (O), and Central (C) (Chong et al.
along with a great resolution, noninvasive EEG 2007). Specific rules are applied on the numbering
devices need external auxiliary tools to monitor measure of electrode placements: even numbers
the brain. Electrodes are introduced in noninva- are assigned to the electrode places on the right
sive EEG devices to equip on the scalp: vertically side of the head and odd numbers are assigned to
beginning from the direction of nasion, sunken the left side. Each EEG channel presents different
part of the nose (between eyes, above the bridge type of information of brain activities; sophisti-
of the nose) to inion, the prominent bone at the cated analysis of multiple channels is always nec-
back of the head; and horizontally from the left ear essary in order to achieve higher accuracy of
lobe to the right (Oude 2007). classification of brain signals.
The operation that an electrode captures the
electrical activity in brain is called an EEG chan- EEG Bands
nel that typical EEG systems have only a few There are many bands available to carry different
channels available to use. More electrodes denote part of information of brain-waves and the analy-
a clear signal that can be observed from the brain sis of a particular band may differ depending on
activities. EEG devices are usually equipped up to the researcher’s final goal (Rasheed and Marini
256 channels and electrode placement on the head 2015). Conventional bands such as delta (< 4 Hz),
has a formal standard, namely, the International theta (4–7 Hz), alpha (8–12 Hz), and beta
10/20 system that usually shortened to 10/20 sys- (13–40 Hz) are usually considered as the primary
tem. The 10/20 system is the standard electrode frequency ranges. In the regards of the EEG cap-
location method to acquire EEG data and is uti- tured data, the EEG is usually discussed in terms
lized by most of current databases (Soutar 2014). of rhythmic activity (the rhythm) and transients
The standard placement system, the 10/20 sys- (the fluctuation of amplitude). Cerebral signals
tem, suggests the definition of exact position of observed in the noninvasive EEG devices always
electrode placement and assignment of symbols state in the frequency range of 1–20 Hz where out
that denotes particular electrodes to access partic- of range usually denotes a nonstandard recording
ular part of brainwaves that can be monitored technique or it is deemed as artifactual. General
Color Detection Using Brain Computer Interface 361

Color Detection Using

Brain Computer
Interface,
Fig. 1 Electrodes
placement (Sjölie 2011)

observations of brainwaves are categorized in Drowsiness arousal can produce theta waves that
below. are observable in older children and adults as well
Delta (<4 Hz): Delta waves were originally as during meditation. Excess theta waves always
introduced in 1936 by Gray Walter, an English de-note a number of clinical conditions, such as
physiologist who acquired very slow and high- epilepsy, or abnormal activity of adults such as
voltage potentials that were slower than formerly focal disturbance (Scher 2018). Additionally,
described wave patterns from the involved area of theta waves are usually associated with relaxation,
a patient who was undergoing neurosurgery for a meditation, and creation states.
malignant tumor (Marcuse et al. 2015). The char- Alpha (8–12 Hz): The first recording of human
acteristic of delta band aims at the highest in EEG waves by Hans Berger is in the alpha range.
amplitude and has the slowest waves. It is usually It is observed in the occipital cortex, the posterior
found frontally in adults in slow-wave sleep and areas of both side of the head (Baars and Gage
posteriorly in children. As the rule, delta waves 2013). Higher amplitude always shows on the
are absent in wakeful adults, and in contrary, dominant side. Alpha waves usually relate to
presence of delta waves during wakefulness motor activities such as eye movements. Particu-
implies cerebral dysfunction (Marcuse et al. larly, closing eyes with comfortableness raises the
2015). Furthermore, delta waves are usually asso- alpha waves and attenuates with eye opening and
ciated with the deepest levels of relaxation and mental concentration (Bazanova and Vernon
restorative, healing sleep, in other words, ade- 2014). The alpha waves are also subject to age
quate production of delta waves enhances the dynamics as the alpha rhythms in healthy subjects
immune system, natural healing and restorative reduces associated with age in the interval from
deep sleep (Abhang et al. 2016). 7 to 80 years in terms of power which reaches the
Theta (4–7 Hz): Theta rhythms are generally highest values around 20 years of age and slowly
observed in the frontocentral areas of the brain in declines (Kropotov 2016).
young children and are related to drowsiness or Beta (13–40 Hz): Beta waves are observable
heightened emotion states (Scher 2018). with increased possibility in frontal or central
362 Color Detection Using Brain Computer Interface

regions of brains compared to posterior areas of Rosen 2015). In actual, frontal lobes composite of
the cortex (Kropotov 2010). Beta waves are high- a number of regions that participate various brain
frequency and low-amplitude brain waves in processes such as attention, memory, executive
awaken sate and are produced associated with functions, insight and self-monitoring, language,
logical and conscious thinking, where production and social and emotional functions (Pressman and
of adequate beta waves enhances concentration Rosen 2015). Furthermore, common sense has
(Abhang et al. 2016). Active concentration, theoretically different perceptions to the executive
busy, active thinking leads to a lower amplitude functions, are fundamentally classified as the abil-
beta wave, the beta one waves, the low beta waves ities to retain and process objects not physically
(12–15 Hz); Mid-range beta waves are described placed in the surrounding environment
as beta two waves associated with improved per- (Goldman-Rakic 1996). However, the functional
formance and anxiety; High beta waves are organization of frontal cortex is still controversial
described as beta three waves associated with where the primary theory proposes that frontal
significant anxiety, paranoia, and high arousal lobes are organized correspondingly to
(Ab-hang et al. 2016). rostrocaudal axis to participate in the hierarchical
cognitive control (Badre and Nee 2017). In sum-
Brain Lobes and Functions mary, frontal lobe is functioning in cognition,
EEG signals are captured from various positions problem solving and reasoning; development of
of the brain. It is critical to understand the brain motor skills; participation in speech; Impulse con-
functions of different cerebral cortices that each of trol; spontaneity; regulation of emotions; regula-
which has important roles to composite the con- tion of sexual urges; and planning (Abhang
sciousness. The cerebrum dominates the most et al. 2016).
spaces of the human brain associated with com- Parietal Lobe: Parietal lobe is commonly
plex brain functions among cogitations and activ- described with correlation to the sensory pro-
ities. The cerebral cortex is divided into four lobes cessing in the brain. It occupies approximately
and two hemispheres which are shown in the 20% of the cerebral cortex and is categorized
Fig. 2. into two primary regions, the somatosensory cor-
Frontal Lobe: Frontal lobe is commonly con- tex and the posterior region of the parietal cortex
sidered as a synonym to the executive functions, which engages in various cognitive operations,
utilizing the capacity of the frontal lobes to admin- especially involved in representations of spatial
istrate and improve other cognitive processes subjects (Behrmann and Shomstein 2009). Gold-
more efficient through attention, to inhibit basic berg (2001) described five primary functions: the
response in approaching a complex objective, fundamental analysis of somatic sensation (such
abstraction and other mechanisms (Pressman and as touch, the limb positions, temperature percep-
tion); space measurements utilizing sensory
modalities; the specification of spatial objects for
Side view Top view the motor system; generation of attention; and the
Parietal analysis of visual motions. In summary, parietal
Frontal Occipital
lobe is functioning in sensing pain, pressure, and
touch; regulation and process of traditional five
senses: vision, audition, gustation, olfaction, and
somatosensation; movement and visual orienta-
Temporal tion; speech; visual perception and recognition;
and cognition and process of intelligence
Left Right (Abhang et al. 2016).
hemispheres hemispheres
Occipital Lobe: The occipital lobe holds
Color Detection Using Brain Computer Interface, responsibility for visions primarily and is situated
Fig. 2 Brain lobes and hemispheres (Bermúdez 2014) posteriorly to both the parietal and temporal lobes.
Color Detection Using Brain Computer Interface 363

It is the smallest of the four lobes and it occupies EEG equipment or the external interferences, such
approximately 12% of brain cortex; visual sensa- as radio waves and mobile phones. Pre-processing
tions are produced from the occipital lobe through methods are commonly required prior to pro-
direct electrical stimulations and damage to the cessing and analysis of the recordings. In the
occipital lobe can influence visions in different former experiment conducted with color detec-
degrees depending on the location damaged and tions, a number of methods were utilized to
severity of the damage (Galetta 2017). Most of the remove artifacts: visual inspection, linear filter, C
knowledge of structural organization and pro- bandpass filter, and independent component anal-
cessing mechanism of occipital lobe comes from ysis (ICA). For event-related potentials (ERP),
the studies of macaque’s occipital lobe which applying visual inspection along with linear filter
shares a certain degree of similarity with humans. or band-pass filter at specific frequency can
In addition to the spatial specification of visual remove skin potentials at low-frequency bands
objects that are maintained in the occipital lobe, and muscle and electrical noises at high-frequency
visual attributes such as local shape, motion, bands. Visual inspection is usually necessary to
color, and depth are also extracted and organized manually remove undesirable segments of brain
(Braddick 2001). In summary, occipital lobe is signals in a trial as the recordings may contain
functioning in visualspatial handling; movement, further instructions other than the valuable data,
and color recognition (Abhang et al. 2016). the potentials in recordings with color stimuli. In
Temporal Lobe: Temporal lobe associates with advanced environment, ICA is significant in
perception and recognition, is the region where removal of artifactual components such as muscle
sound is processed, which regulates the auditory activity, eye blinks, and electrical noises (Delorme
language and speech comprehensions (Baars and et al. 2007). Applying various methods to reduce
Gage 2013). Along with the frontal lobe, the tem- artifacts could improve the rate of successful clas-
poral lobe functions significantly in humans, indi- sifications, such as Phillips et al. (2015) applied
cating the anatomical relationship of two areas Joint Approximate Diagonalization of Eigen-
towards acoustic, intelligence, and memory matrices (JADE) algorithm as the ICA and the
(Bellack and Hersen 1998). In summary, temporal classification accuracy shows an approximate
lobe is functioning in management of long-term 5% improvement over the 20 trials conducted
memories and handles new information, forma- during the experiment.
tion of visual and verbal memories, and interpre-
tation of olfaction and audition (Abhang Feature Extraction
et al. 2016). Feature extraction is to extract the adequate fea-
ture that is valuable in each single trial. Extensive
Pre-processing of Brainwaves ways are available to select different features,
Captured brainwaves always contain further such as the simplest, calculating the peak in
noises than the useful information. Pre-processing power spectrum, and calculating the mean of a
is usually necessary for clearing the noise, reduce set of trials. Abuhashish et al. (2014) concluded a
the noise ratio of raw EEG signals as the captured number of common features are extracted: the
raw signals are generally contaminated with irrel- value of the EEG signals largeness, band powers,
evant activities such as eye blinking and physi- power spectral density ideals, autoregressive and
cally micro movements. Such activities are called adaptive autoregressive parameters, time-
artifacts, which can cause accuracy and efficiency frequency features, and inverse model-based fea-
issues in the system. During the recording and tures. Yu and Sim (2016) conducted an experi-
amplification period, main’s electrical impulse, ment that classifies the brainwaves captured
minor physical movements such as eye blinking during the simulation released by the specially
or muscle movements and artifacts could interfere made visual stimulator that manually created
the captured data (Rasheed and Marini 2015). using an Arduino board and a couple of LEDs.
Artifacts can also be yielded by the problematic They extracted N2 (N200, the negative-going
364 Color Detection Using Brain Computer Interface

wave that peaks 200–350 ms after stimulus types and has an overall good performance (Peter
(Folstein and Van Petten 2008)), P3 (P300, occurs et al. 2019).
when people react to stimuli and surfaces as a Artificial Neural Network (ANN): ANN is the
positive deflection in voltage with a delay computational technique, one of the machine-
between stimulus and reaction ranging from learning approaches trying to learn the behavior
250 ms to 500 ms (Polich 2007)), and N4 of the brain signals. In analogy, ANN utilizes a
(N400, the normal brain reactions to meaningful large set of computational units, the artificial neu-
stimuli such as visual and auditory stimuli (Kutas rons to simulate brain neurons.
and Federmeier (2000)) of ERP data and achieved Classification Performance: Classification rate
classification rates ranging from 35.76% to can also be influenced by different combination of
87.11%. Aclo et al. (2016) conducted similar selected channels. Selecting one or a few appro-
experiments utilizing an LED screen monitor priate channels can improve efficiency and per-
doing color stimuli, and Kurtosis was found to formance of the application. Such as Yu and Sim
be the most significant feature in alpha and beta (2016) found that channel T7 and F4 are robust
bands for classification of colors. electrodes to classify color imaginations and
Aprilianty and Purwanegara (2016) found that
Classification F3 and F4 channels are having a better classifica-
Numerous classification algorithms are available tion rate. Additionally, Yu and Sim (2016) found
to deploy in different scenarios, such as Support that applying ANN without LDA can bring an
Vector Machine (SVM), Artificial Neural Net- approximate 10% improvement in the
work (ANN), Multi-player Perceptron (MLP), classification rate.
Naïve Bayes (NB), k-Nearest Neighbor (k-NN),
and K-fold Cross Validation. SVM and ANN are A Typical BCI Research Model
the common approaches adopted in researches A simple, typical BCI application model is
related to classification of imaginary colors and described in the Fig. 3. Most of the stimuli
are briefed in below: researches share commons in basic modelling of
Support Vector Machine (SVM): SVM was the application and can differ based on the final
introduced in the late 1990s and has been adopted purpose. This model can be applied in the color
in many engineering-related applications stimuli experiments, and the devices utilized to
(Gholami and Fakhari 2017). SVM is one of the present colors can be selected from a large group
machine learning, non-probabilistic approach uti- of different categories, such as the simplest one,
lizing a linear classifier to categories data into two computer monitors or some specifically made

Color Detection Using Brain Computer Interface, Fig. 3 A simple, typical BCI application model
Color Detection Using Brain Computer Interface 365

LED devices to represent various colors, achieved features. Finally, both of SVM and ANN are com-
in Yu and Sim (2016)’s research. Researchers monly adopted techniques, have the same concept
have the initiative to select particular colors that using linear learning model for pattern recogni-
can be represented by different devices. tion, and the main difference is the way nonlinear
According to the specifications designed, the data is classified (Ren 2012). Even though some
final results of different researches could show a studies suggest SVM has a better performance
certain degree of deviation when the preciseness than ANN, the average accuracy achieved is C
is critical. around 80% and the accuracy decreases dramati-
In the color stimuli experiments, a number of cally when the training trials are imbalanced.
participants will be asked to re-main quiet and Color classification can be used in various
comfortable before starting the EEG recording scenarios of world. For example, in the enhance-
sessions. Generally, the device represents a par- ment of the existing selling models, customers
ticular color for a few seconds and the brainwaves could simply think to get the right color of a
of participant will be captured and recorded by the desired clothes or items. It can certainly improve
BCI devices during each session. The recording the efficiency of shopping as well as reduce the
sessions will then be pre-processed before apply pressure caused on shop assistants during the peak
further analysis and actions. After filtering, hours. However, BCI devices are still relatively
denoising, and feature extraction, all recording complicated to be equipped and prone to be
sessions will be converted into small pieces. influenced by the interferences such as radio
These small pieces will then be analyzed and waves. The cost of device maintenance will also
classified into a number of different classes. be increased and new technical positions are pos-
Based on the analysis and classification result, sibly critical to be fulfilled. Capability of deter-
further actions can be taken and different stimuli mining the colors imagined can be also used in
scenario can be used to expand the recording BCI games to map further actions that cannot be
library. Usually, a larger library of recording ses- performed by human in the real world such as
sions with various scenarios will increase the flying. In the case of mapping a single color to a
accuracy and value of the final results. single action in the game, the traditional gaming
can be flawlessly shifted over to the BCI gaming
Discussion where a higher level of concentration and reaction
There are numerous feasible approaches to imple- of gamer could lead to a better result. Further-
ment the theoretical prescription described in the more, the research conducted by Abuhashish
method section into working solutions, such as (2015) also proposed the same idea of eliminating
BCILab and EEGLab (Delorme and Makeig conventional models of interaction between
2004), two available Matlab applications that human and computer graphics by analyzing the
greatly simplified the data analysis of brain sig- brain signals to map emotions into the virtual
nals with graphical interfaces. Regarding pre- world. Particularly, Abuhashish et al. (2015)
processing of brainwaves, although ICA can constructed the models of mapping human emo-
achieve a better result of removing noises in the tions with virtual 3D characters by utilizing
trials comparing to applying linear filter or Higuchi algorithm to extract human emotions
bandpass filter, the draw-back is the performance and then to synchronize with in-screen characters.
issue as it takes more computational power to
produce the processed trial. In the applications
that require responsiveness and are designed for Conclusion
portable devices with limited processing power
will prone to utilize the other two methods to This article provided a relatively comprehensive
denoise the signals. In the feature selection part, overview on the basics of BCI, EEG, and brain
the experiments con-ducted by former researchers functions and a number of feasible approaches of
are limited and there are potentials to explore new pre-process, extract, and classification in order to
366 Color Detection Using Brain Computer Interface

detect imaginary colors. Various terms used in Behrmann, M., Shomstein, S.: Spatial cognition and exec-
neuroscience are explained and some fundamen- utive function. In: Dronkers, N.F., Baldo, J.V. (eds.)
Encyclopedia of Neuroscience. Elsevier, Berlin, Ger-
tal knowledge of dealing with brain is briefly many (2009)
introduced. A number of signal pre-processing Bellack, A.S., Hersen, M.: Comprehensive Clinical Psy-
methods: linear filter, bandpass filter, and ICA chology: Foundations, vol. 1. Pergamon,
and classification methods: SVM and ANN are Amsterdam/New York (1998)
Bermúdez, J.L.: Cognitive Science: An Introduction to the
briefly discussed. Examples of how selling and Science of the Mind. Cambridge University Press,
gaming models can be benefitted with capability Cambridge (2014)
of determining imagined colors are finally Braddick, O.: Occipital lobe (visual cortex): functional
introduced. aspects. In N. J. Smelser & P. B. Baltes (Eds.) (Vol.
16, pp. 10826–10828). Elsevier Science Ltd./Pergamon
Press (2001)
Caton, R.: Electrical currents of the brain. J. Nerv. Ment.
Cross-References Dis. 2(4), 610 (1875)
Chong, D.J., Sahlem, G.L., Bazil, C.W.: Introduction to
▶ Augmented Learning Experience for School electroencephalography. In: Review of Sleep Medicine,
Education 2nd edn, pp. 105–141. Butterworth-Heinemann, Phila-
delphia, USA (2007)
▶ Augmented Reality for Human-Robot Interac- Delorme, A., Makeig, S.: EEGLAB: an open source tool-
tion in Industry box for analysis of single-trial EEG dynamics including
▶ Augmented Reality for Maintenance independent component analysis. J. Neurosci.
Methods. 134(1), 9–21 (2004)
Delorme, A., Sejnowski, T., Makeig, S.: Enhanced detec-
tion of artifacts in EEG data using higher-order statis-
References tics and independent component analysis. Neuroimage.
34(4), 1443–1449 (2007)
Abhang, P.A., Gawali, B.W., Mehrotra, S.C.: Technical Folstein, J.R., Van Petten, C.: Influence of cognitive con-
aspects of brain rhythms and speech parameters. In: trol and mismatch on the N2 component of the ERP: a
Introduction to EEG-and Speech-Based Emotion Rec- review. Psychophysiology. 45(1), 152–170 (2008)
ognition, pp. 51–79. Academic, Amsterdam (2016) Galetta, S.L.: Occipital Lobe. In: Cortex, C. (ed.) Refer-
AbuHashish, F.A.M.: Classfication technique for human ence Module in Neuroscience and Biobehavioral Psy-
emotion in virtual reality using game-based brain com- chology. Elsevier (2017)
puter interface, pp 4–16. Doctoral dissertation, Gholami, R., Fakhari, N.: Support vector machine: princi-
Universiti Teknologi Malaysia (2015) ples, parameters, and applications. In: Handbook of
Abuhashish, F.A., Sunar, M.S., Kolivand, H., Mohamed, Neural Computation, pp. 515–535. Academic Press
F., Mohamad, D.B.: Feature extracted classifiers (2017)
based on EEG signals: a survey. Life Sci. J. 11(4), Goldberg, M.E.: Parietal lobe. In: Smelser, N.J., Baltes,
(2014) P.B. (eds.) International Encyclopedia of the Social &
Abuhashish, F.A.M., Kolivand, H., Shahrizal, M.: Frame- Behavioral Sciences, vol. 11, pp. 11051–11054.
work of Controlling 3d Virtual Human Emotional Elsevier, Amsterdam (2001)
Walking Using BCI. Environ. 2, 4 (2015) Goldman-Rakic, P.S.: The prefrontal landscape: implica-
Aclo, D., Barraquio, B.M., Navea, R.F., Palomares, X.: tions of functional architecture for understanding
EEG-based color classification system using artificial human mentation and the central executive. Phil.
neural networks. LPU-Laguna J. Eng. Comput. Stud. Trans. R. Soc. Lond. B. 351(1346), 1445–1453 (1996)
3(2), 21–29 (2016) Haas, L.F.: Hans berger (1873–1941), richard caton
Aprilianty, F., Purwanegara, M.S.: Effects of colour (1842–1926), and electroencephalography. J. Neurol.
towards underwear choice based on electroencephalog- Neurosurg. Psychiatry. 74(1), 9–9 (2003)
raphy (EEG). Australas. Mark. J. AMJ. 24(4), 331–336 Kropotov, J.D.: Beta Rhythms. In: Quantitative EEG,
(2016) Event-Related Potentials and Neurotherapy,
Baars, B.J., Gage, N.M.: The brain is conscious. In: Fun- pp. 59–76. Academic, USA (2010)
damentals of Cognitive Neuroscience: A Beginner’s Kropotov, J.D.: Alpha rhythms. In: Functional
Guide, pp. 211–252. Academic, USA (2013) Neuromarkers for Psychiatry: Applications for Diag-
Badre, D., Nee, D.E.: Frontal cortex and the hierarchical nosis and Treatment, pp. 89–105. Academic, USA
control of behavior. Trends Cogn. Sci. 22, 1–4 (2017) (2016)
Bazanova, O.M., Vernon, D.: Interpreting EEG alpha Kulkarni, N., Bairagi, V.: Chapter two – electroencephalo-
activity. Neurosci. Biobehav. Rev. 44, 94–110 (2014) gram and its use in clinical neuroscience. In: EEG-
Color: Pixels, Bits, and Bytes 367

Based Diagnosis of Alzheimer Disease, pp. 25–35.

Academic, USA (2018) Color: Pixels, Bits, and Bytes
Kutas, M., Federmeier, K.D.: Electrophysiology reveals
semantic memory use in language comprehension.
Trends Cogn. Sci. 4(12), 463–470 (2000) Vien Cheung
Marcuse, L.V., Fields, M.C., Yoo, J.J.: The normal adult Leeds Institute of Textiles and Colour, University
EEG. In: Rowan’s Primer of EEG E-Book, pp. 39–66. of Leeds, Leeds, UK
Elsevier Health Sciences, China (2015)
Oude, D.B.: EEG-based emotion recognition—the influ-
C
ence of visual and auditory stimuli. Emotion. 57,
1798–1806 (2007) Synonyms
Peter, S.C., Dhanja, J.K., Malik, V., Radhakrishnan, N.,
Jayakanthan, M., Sundar, D.: Quantitative Structure-
Activity Relationship (QSAR): modeling approaches Bit; Byte; Color; Color primaries; File format;
to biological applications. In Encyclopedia of Bioinfor- Pixel; RGB
matics and Computational Biology, pp. 661–676
(2019)
Phillips, K., Fosu, O., Jouny, I.: Separation and classifica-
tion of EEG responses to color stimuli. In Biomedical Definitions
Engineering Conference (NEBEC), 2015 41st annual
Northeast, pp. 1–2. IEEE. April 2015 Color (in computing) is represented by three
Polich, J.: Updating P300: an integrative theory of P3a and values: red, green, and blue which are often also
P3b. Clin. Neurophysiol. 118(10), 2128–2148 (2007)
Pressman, P., Rosen, H.J.: Disorders of frontal lobe func- referred to as RGB.
tion. In: Neurobiology of Brain Disorders, Pixel is the basic, smallest and square-shaped
pp. 542–557. Academic Press, China (2015) element of a digital image.
Rasheed, S., Marini, D.: Classification of EEG signals Bit is a binary value and the basic unit of
produced by RGB colour stimuli. J. Biomed. Eng.
Med. Imaging. 2(5), 56 (2015) computational data.
Ren, J.: ANN vs. SVM: which one performs better in Byte is the smallest unit of digital information
classification of MCCs in mammogram imaging. in computational processing and storage.
Knowl.-Based Syst. 26, 144–153 (2012) File format is the structure with specifications
Rossing, T., Chiaverina, C.J.: Light Science: Physics and
the Visual Arts. Springer Science & Business Media, on how the information to be organized and
New York (1999) displayed.
Scher, M.: Pediatric neurophysiologic evaluation. In: Pixel and Path
Swaiman’s Pediatric Neurology, 6th edn, pp. 87–96. The rapid development of computing and
Elsevier, China (2018)
Sjölie, D.: Reality-based brain-computer interaction. Doc- imaging encourages digital creations. Creative
toral dissertation, Department of Computing Science, software can be largely divided into two types:
Umeå University (2011) one for image manipulation and the other for
Soutar, R.: An introductory perspective on the emerging image fabrication.
application of EEG in neurofeedback. In: Clinical
Neurotherapy, pp. 19–54. Academic Press, India Image manipulation refers to the changing or
(2014) enhancing the appearance of existing images.
Yu, J.H., Sim, K.B.: Classification of color imagination Those images are often regarded as raster images
using Emotiv EPOC and event-related potential in with pixels as their basic composition. Image fab-
electroencephalogram. Optik Int. J. Light Electron
Opt. 127(20), 9711–9718 (2016) rication is oriented for generating original illustra-
tions. The approach is more often vector-based, in
which lines and shapes are represented by equa-
tions. The basic unit of vector images is
called path.
Color Primaries One significant distinction between raster and
vector images is the fineness or smoothness in
▶ Color: Pixels, Bits, and Bytes terms of image quality when the magnification
368 Color: Pixels, Bits, and Bytes

reaches a certain level (see Fig. 1). The quality of there is still a dominance on the use of raster
vector images is not constrained by scale. images. This can be explained by the imaging
Another distinction is the requirement of pro- resources, including software compatibility and
cessing and storage capacities. Raster images tend file formats. Imaging devices render raster
to take up much more space than vector images. images, which include also photographs (see
Although vector images are more efficient, Fig. 2), and, thus, correspondingly majority of
both in fidelity and space, than raster images, the file formats support raster images.

Pixel and RGB

Computer adopts the binary system, 2x, to process

information, and its basic unit is called bit. With
the binary system, a bit can be only one of the two
values: 0 or 1. That said, two bits (22) allow four
possible conditions: [00], [01], [10], and [11].
Since the first condition is always regarded as 0,
two bits represent numbers between 0 and 3.
As of today, majority of computers are 8-bit
(28) devices, and thus they can offer 256 condi-
tions ranging from 0 to 255. These conditions
allow fine grayscale tonal variations for images.
Color images are typically represented by 24-bit
with 8-bit per each of the red (R), green (G), and
blue (B) primaries. That said, a typical computer
Color: Pixels, Bits, and Bytes, Fig. 1 Demonstrating the
appearance of a raster image (top left) with a vector image is capable to display images with approximately
(top right) at certain magnifying level (bottom) 16.7 million unique colors (Fig. 3).

Color: Pixels, Bits, and Bytes, Fig. 2 The basic unit of a 741 (height) pixels. These numbers also represent the
raster image is pixel. A raster image consists of an array of image size
pixels. This example image has a total of 1,100 (width) by
Color: Pixels, Bits, and Bytes 369

Color: Pixels, Bits, and Bytes, Table 1 File sizes – their

units and conversions
1,024 bytes (B) ¼ 1 kilobyte (KB)
1,024 kilobytes (KB) ¼ 1 kilobyte (MB)
1,024 megabytes (MB) ¼ 1 gigabyte (GB)
1,024 gigabytes (GB) ¼ 1 terabyte (TB)
C

saturated colors and smooth gradations, such as

sceneries and portraits. For graphical images, GIF
file format is preferable since it stores only 8-bit
Color: Pixels, Bits, and Bytes, Fig. 3 Each pixel has a per pixel. That said, a maximum of 256 unique
color that is defined by the red (R), green (G), and blue colors which are seen to be sufficient for simple
(B) values
graphics, for instance, logos, graphs, and charts.
A note to the approach of JPEG file format on
Pixel and Byte
reducing file size by removing data is its under-
standing on the characteristics of pixels in images
Image file size determines how much computing
and the limitations of the human eye. In particular,
storage space a file is required. The basic unit of
some pixel values are more common than others,
any file size is byte (B). There are also units such
and neighboring pixels have similar colors. Also,
as kilobyte (KB), megabyte (MB), gigabyte (GB),
very importantly, the fact that small color changes
and terabyte (TB) to indicate the larger file sizes.
are perceived less accurately by the human eye than
See Table 1.
small changes in brightness. Thus, the selection of
Every 8-bit, which is 256 values, occupies one
data that to remain and to remove is prioritized by
byte of computing storage space. Therefore, sim-
minimum perceptual differences between the orig-
ply one RGB color pixel will require three bytes
inal and compressed images.
of computing storage space. Image file size varies,
but it is not uncommon to see color image files of
several megabytes.
Cross-References

▶ Teaching Computer Graphics by Application

Pixel and File Format

Practically, for websites and apps, several mega-

Further Readings
bytes for each color image could take up signifi-
cant time to load or download. To allow better Berns, R.S.: Billmeyer and Saltzman’s Principles of Color
efficiency, there are several file formats for users Technology, 3rd edn. Wiley, New York (2000)
to opt for depending on the intended applications. Cheung, V., Rhodes, P.A.: Uniform colour space. In:
Some commonly used file formats for raster Blankenbach, K., Yan, Q., O’Brien, R.J. (eds.) Hand-
book of Visual Display Technology, 3rd edn. Springer,
images are TIFF (.tiff or .tif), JPEG (.jpeg or . Berlin, Heidelberg. [in production for 2023]
jpg.), and GIF (.gif). TIFF is often recommended Fiete, R.D.: Formation of a Digital Image: The Imaging
for images that require high quality and raw data Chain Simplified. SPIE Press, Bellingham, Washington
being preserved. JPEG can compress images by (2012)
Galer, M., Horvat, L.: Digital Imaging, 3rd edn. Focal
removing certain selected data of the file. It is a Press, Amsterdam (2005)
popular alternative to TIFF files, especially since Westland, S., Cheung, V.: RGB systems. In: Blankenbach,
it has the flexibility to allow different levels (e.g., K., Yan, Q., O’Brien, R.J. (eds.) Handbook of Visual
5% or 20%) of data removal. This provides an Display Technology, 3rd edn, pp. 1–7. Springer, Berlin,
Heidelberg (2023). https://doi.org/10.1007/978-3-642-
advantage for images that contain significant 35947-7_12-3
370 Comic Arts

Definition
Comic Arts
Non-photorealistic rendering (NPR) is a topic
▶ Comic Arts in Games, Asset Production, and under computer graphics trying to achieve some
Rendering level of stylization in 3D with the use of geometry,
lighting, shading, and special effects. Authors are
specializing in recreating the aesthetics of comic
arts in video games via implementing the research
already done in the area and also adding new
research topics.
Comic Arts in Games, Asset
Production, and Rendering
Introduction
Metin Arıca and Burak Tezateşer
Nowhere Studios, Istanbul, Turkey Comic art stylization characteristics include toon
shading, outlines, line art, hatching, and inner
lines. The study of authors started with a literature
Synonyms review and implementing some of the real-time-
related works to unreal engine. Unreal engine has
Comic arts; Non-photorealistic rendering; NPR been chosen mainly for its rendering quality, and

Comic Arts in Games, Asset Production, and Rendering, Fig. 1 Organic objects samples from the original paper
(Praun et al. 2001) with tonal art maps
Comic Arts in Games, Asset Production, and Rendering 371

Comic Arts in Games, Asset Production, and Rendering, Fig. 2 Sample scene in unreal engine with flat surfaces

Comic Arts in Games,

Asset Production, and
Rendering,
Fig. 3 Character with
handmade line art texture

the ease of access to the source code if any change production pipeline. After trying different
in the renderer is required to create the desired methods, the tonal art map pattern introduced in
effects. real-time hatching (Praun et al. 2001) seemed like
the best option for creating seamless hatching
patterns. Unfortunately, it didn’t match the desired
Hatching Shading Method and Line Art quality in the case of a repeated use especially on
flat surfaces. We decided to limit the use of hatch-
Comic book art often uses different hatching tech- ing shading on background organic objects.
niques unique to its artist. Our goal was to mini- The desired effect is achieved not by
mize the texturing efforts in video game implementing a hatching shading in real time but
372 Comic Arts in Games, Asset Production, and Rendering

Comic Arts in Games,

Asset Production, and
Rendering,
Fig. 4 Distortion on the
UV outlines

Comic Arts in Games,

Asset Production, and
Rendering, Fig. 5 Final
Image with distorded UV
outlines
Comic Arts in Games, Asset Production, and Rendering 373

Comic Arts in Games,

Asset Production, and
Rendering, Fig. 6 Inlines
with lighter line color

Comic Arts in Games, Asset Production, and Rendering, Fig. 7 Outline reflecting the color of a light source

Comic Arts in Games,

Asset Production, and
Rendering,
Fig. 8 Screenshot from
Nowhere Studios upcoming
game Dan Black where line
art texture, distortion on
UV, smoothed toon
shading, and outlines have
been used
374 Comic Arts in Games, Asset Production, and Rendering

by using a handmade line art texture drawn espe- Toon Shader

cially on the parts suggested by the ambient occlu-
sion map by an artist. Most art directors use toon shader to create
Furthermore, the stylization of inner lines on paperlike characteristic. In video games, it is
hard surface models is achieved by adding a mostly difficult to achieve a good looking toon
distortion on the UV outlines. This is a very shader because of polycount limits. Hard transi-
similar technique to the one used in WYSIWYG tion between toon shading levels often gives away
NPR: Drawing Strokes Directly on 3D Models the low-poly geometry of objects. One solution to
(Kalnins et al. 2002). This can be done either real the problem is editing the normals of the model,
time because the additional cost to GPU is neg- but it requires time and effort. Our solution was to
ligible or while baking the texture so it can be use smooth transition between toon shading
modified manually if necessary. As UV outlines levels.
can be detected using an edge detection, the
distortion process doesn’t require any effort
from the artist. Conclusion and Possible Exploration
Topics

There are many further research topics including a

Outlines better hatching pattern and technique, possibly
using a deep learning methodology to analyze
Outlines are an important part of comic art. An different comic art styles; an outline method
edge detection algorithm is used to draw outlines. reflecting light intensity and geometry; and a solu-
Edge detection is a post process; therefore, all of tion for easily editing the normal maps for better
the methods create artifacts on their own. It is not looking toon shading.
possible to perfectly distinguish the object in the Comic art is one of the many possible tradi-
front from the background every time unless an tional art styles one might try to imitate in 3D.
inverted hull method introduced by “Guilty Gear Painting methods such as water painting, oil on
X’rd technical presentation” (Motomura 2015) is canvas and many more, and also all the art move-
used. The inverted hull requires the geometry to ments, these techniques have been used with pro-
be drawn a second time; therefore, it is more vide a very rich soil for non-photorealistic
costly. rendering research.
Our edge detection draws the line to the pixel
outside of the object, thus creating a proper sil-
houette of the object. The distance of the object Cross-References
from the background and the distance between the
object and the camera are reflected in thickness ▶ Rendering Equation
and color of the line drawn.
Also inlines are drawn with a lighter line
color similar to the technique used in suggestive References
contours for conveying shape (DeCarlo
et al. 2003). DeCarlo, D., Finkelstein, A., Rusinkiewicz, S., Santella,
A.: Suggestive Contours for Conveying Shape,
Comic book artists also consider lighting and SIGGRAPH. ISBN:1-58113-709-5, pp. 848–855.
geometry of the object while drawing the outline. (2003)
This is a further research area for NPR. Kalnins, R.D., Markosian, L., Meier, B.J., Kowalski,
The outline color can also reflect the color of M.A., Lee, J.C., Davidson, P.L., Webb, M., Hughes,
J.F., Finkelstein, A.: WYSIWYG NPR: Drawing
the light source enlightening the object or it can be Strokes Directly on 3D Models, SIGGRAPH.
used as a second outline layer. ISBN:1-58113-521-1, pp. 755–762. (2002)
Computational Steering for Computational Fluid Dynamics 375

Motomura, J.C.: GuiltyGearXrd’s art style: the X factor

between 2D and 3D, GDC 2015. (2015). https://www. Computational Steering for
gdcvault.com/play/1022031/GuiltyGearXrd-s-Art-Style-
The. Accessed 2 Jan 2018 Computational Fluid
Praun, E., Hoppe, H., Webb, M., Finkelstein, A.: Real Dynamics
Time Hatching. SIGGRAPH. ISBN:1-58113-374-X,
p. 581. (2001) Jan Linxweiler1, Adrian R. G. Harwood2 and
Christian F. Janßen3 C
1
Center for Mechanics, Uncertainty and
Simulation in Engineering, Technische Universität
Command System Braunschweig, Braunschweig, Germany
2
Research IT, IT Services, The University of
▶ Decoupling Game Tool GUIs from Core Manchester, Manchester, UK
Editing Operations 3
Institute for Fluid Dynamics and Ship Theory,
Hamburg University of Technology, Hamburg,
Germany

Commercial Rights
Definitions
▶ Computer Games and the Evolution of Digital
Rights Computational is the practice of manually
steering intervening with an otherwise
autonomous computational
process in order to change its
Comparative Studies outcome.
Computational is the prediction of fluid flow
▶ Cross-cultural Game Studies fluid dynamics by numerical methods.
(CFD)
In situ refers to the visualization of
visualization result data at runtime.
Competition
Introduction
▶ StarCraft Bots and Competitions
Most computational fluid dynamics (CFD) simu-
lations require massive computational power,
which is usually provided by traditional high-
Competitive Video Games performance computing (HPC) environments.
Often, simulations are executed on a massive
▶ Fortnite: A Brief History number of CPU cores (<check>O</check>
▶ Professional Call of Duty Player Matthew (1000)) that are hosted in a remote super-
“Nadeshot” Haag: An e-Sports Case Study computing center or an in-house supercomputing
▶ Super Smash Bros. Ultimate and E-sports facility. Due to several limitations of HPC envi-
ronments, the majority of present CFD simula-
tions are usually executed non-interactively,
although interactivity of the simulation process
Compressor/Decompressor is highly appreciated by scientists and engineers.
In this entry, different approaches for interac-
▶ Postproduction in Game Cinematics tive CFD simulations are briefly reviewed. As
376 Computational Steering for Computational Fluid Dynamics

Computational Steering for Computational Fluid Dynamics, Fig. 1 Traditional CFD process chain (Linxweiler
et al. 2011)

opposed to remote rendering solutions, where repeated. User interaction is obviously rather
data to render is pulled over a connection to limited and takes place exclusively during the
remote rendering stations, a recent trend is to preprocessing/modeling phase. Not surprisingly,
harness the parallel computational power of it is a long-term wish of scientists and engineers
locally available many-core CPUs, graphics pro- to closely interact with their simulations while
cessing units (GPUs), or mobile devices for they are running. Since the influential report of
general-purpose applications. Starting from the the US National Science Foundation (NSF) in
state of the art in conventional CFD simulations, 1987 (McCormick et al. 1987), new forms of
this entry focuses on hardware resources that are scientific visualization have evolved that are
available on the desktop and in the palm of your quite different from traditional post-processing.
hand as a basis for computational steering. The approach commonly referred to as computa-
tional steering has been the subject of wide-
spread interest in the scientific CFD
State of the Art in CFD Analysis community. The name goes back to one of the
first applications of interactivity demonstrated by
Traditionally, CFD analyses are done in a cyclic Marshall et al. (1990) who presented an interac-
sequence of independent steps (Fig. 1). The sim- tive simulation of a turbulence model using a
ulation is usually set up in a preprocessing phase simulation of Lake Erie. In the following years,
by defining the geometry of the flow domain as major contributions to the development of the
well as the initial and boundary conditions. After computational steering paradigm by Liere,
that, the calculation is carried out for a fixed Mulder, and van Wijk (Van Liere and Van Wijk
number of time steps. As CFD simulations are 1996; van Liere and Mulder 1997; van Liere et al.
computationally intensive, they are usually exe- 1997; Mulder et al. 1999; Mulder and van Wijk
cuted on high-performance systems. In such high- 1995; Van Wijk et al. 1997; Van Wijk and Van
performance computing environments, the job is Liere 1997) as well as Johnson and Parker
typically submitted to a queuing system and is not (Johnson et al. 1999; Johnson and Parker 1994;
executed until the needed resources become avail- Parker et al. 1997a; Parker and Johnson 1995;
able. The simulation may run for hours, days, or Parker et al. 1997b, c) are recognized.
even longer until in the end the results are stored A first step to more interaction with the simu-
on a file system. To conduct post-processing, this lation is the so-called tracking (Marshall et al.
data is often transferred to the users’ workstation 1990) approach, which is sometimes also called
to evaluate the simulation and identify errors or monitoring (Hart and Kraemer 1999). After each
possible modifications to model parameters that time step, intermediate results are sent to the
will be applied in a subsequent iteration. The visualization pipeline for analysis. This enables
whole process chain is illustrated in Fig. 1. the scientist to analyze the results in an early
For each adaptation that is to be made to the stage of the simulation to stop and restart the
computation, the whole process has to be simulation when required. The next step is
Computational Steering for Computational Fluid Dynamics 377

C
Computational Steering for Computational Fluid Dynamics, Fig. 2 Pipeline of computational steering (Linxweiler
et al. 2011)

computational steering which goes a lot further. Conventional CPU-Based Approaches

According to van Liere et al. (1997), it can be Although the advantages of computational
considered as the ultimate goal of interactive steering are obvious, its application in CFD, as
computing. Mulder and others define computa- well as other sciences, is still rare (Pickles et al.
tional steering as interactive control over a com- 2005; Wenisch et al. 2005; Wright et al. 2010;
putational process during execution (Mulder Janßen and Bindick 2019). There are various rea-
et al. 1999). As such, computational steering sons for this, more often related to integration with
closes the loop in that scientists and engineers traditional HPC systems. First and foremost, the
can respond to the results immediately by inter- development of a computational steering system
actively changing the setup of the simulation is a complex task, as it requires the integration of
without the need to restart the whole process. multiple disciplines such as physics, numerical
As changes to parameters become more instan- methods, user interface design, 3D graphics, and
taneous, the cause-effect relationship becomes HPC (Parker et al. 1997c; Van Wijk et al. 1997;
more obvious so that scientists can develop an Linxweiler et al. 2011). Furthermore, HPC envi-
intuition regarding the effect of a parameter, ronments are traditionally not set up for interac-
detect bugs, or deepen their understanding of tive usage. Instead, the resources are remote to the
the physics or the algorithms of the considered user and typically managed by a queuing system.
problem (Parker and Johnson 1995). With com- Computational steering violates many of the
putational steering, “What if?” questions can be assumptions on which resource management sys-
answered more efficiently and therefore guide tems rely. Therefore most site managers believe
the investigation at runtime (Geist II et al. that interactivity wastes CPU cycles (Johnson
1996). This leads to a significant increase in et al. 1999). As a consequence most research, as
productivity and a vast reduction of the time well as commercial CFD codes, do not support
between the generation of the results and explo- interaction. The exceptions are, e.g., Kühner
ration. Accordingly, the timeline of the compu- (2003), who demonstrated a 3D virtual wind tun-
tational steering pipeline can be summarized as nel, which combines an HPC system running the
depicted in Fig. 2. In addition to efficiency, com- CFD simulation with a virtual reality system for
putational steering also stimulates communica- facilitating exploration and handling user interac-
tion and discussion within engineering teams by tion. The application allows for dynamic adjust-
providing immediate visual representation of the ment of scalar parameters as well as manipulation
model and results (Van Wijk et al. 1997). These of primitive geometries during execution. Based
benefits make computational steering an impor- on this work, Wenisch (2008) presented a simula-
tant tool, especially for rapid prototyping and tion of a ventilation system for operating theaters
debugging, but also add value in other areas, which also works with complex geometries. In
e.g., for teaching (Linxweiler et al. 2007; cooperation with Borrmann (Borrmann 2007;
Glessmer and Janßen 2017). Borrmann et al. 2006), this application was
378 Computational Steering for Computational Fluid Dynamics

extended to allow for collaborative exploration of

the simulation results by multiple distributed
users.

y
ilit

Flo
tab

Scr
How to Measure Interactivity

/S
on

een
w R fresh
uti
acy
For interactive simulations, it is essential that

ate
Re
sol
cur
developers maintain awareness of the interplay

Re
Ac
between the perceived rate of flow to a user, the
accuracy/stability of a calculation, and the com-
putational throughput. These are the key qualities Parallel Capacity
of an interactive simulation as discussed in Computational Throughput
Harwood and Revell (2018) and Harwood
(2018, 2019) and are controlled therein through Computational Steering for Computational Fluid
Dynamics, Fig. 3 The interplay between the perceived
configurable parameters. The accuracy and stabil- rate of flow to a user, the accuracy and stability of a
ity of the numerical solver are controlled by the calculation and the throughput of the solver are important
simulation resolution, the perceived flow rate to for interactive applications
the user through the refresh frequency of the visu-
alization, and the computational throughput second. Each time step on our given compute
through the amount of parallel compute capacity device takes dtwall ¼ 0.036. Thus we have a
used (c.f. Fig. 3). Tr ¼ 0.01/0.036 ¼ 0.28. Therefore the simulation
As the accuracy increases, so does the resolu- is running at 3.6x slower than real time. Increasing
tion of the calculation. This puts increasing the resolution may increase the throughput
demands on computing resources, typically (assuming resources have not been saturated). So
shrinking the time step size. Thus, more iterations doubling the resolution, if the device scales per-
need to be completed for the same amount of time fectly, would double the throughput to 100,000
to be simulated. At a fixed computational through- cells every second. However, for some numerical
put, the simulation thus evolves slower, and the methods, such as the lattice-Boltzmann method,
refresh rate may need adjusting to compensate. to control errors and maintain stability, the time
The real-time ratio Tr (Harwood and Revell step would then need to be divided by four
2018) expresses the ratio of the time simulated (Krüger et al. 2009). Thus, dtwall ¼ 0.036 still
to the time taken to simulate that time interval but dtsim ¼ 0.0025 and hence Tr ¼ 0.0025/
with Tr ¼ 1 being the precise definition of a real- 0.036 ¼ 0.07. Hence, the ability to achieve true
time simulation. This provides an improved mea- real-time performance is dependent on both suit-
sure of the capability of an interactive calculation able numerical methods and suitable hardware
compared to just examining throughput alone. and cannot be achieved by one alone.
However, even if a calculation is running at a Real-time ratio just considers the performance
real-time rate, the refresh rate controls how the at a computational level. A user of an interactive
flow is perceived by the user. Therefore, for inter- simulation is probably less interested in whether
activity, we need to consider all these parameters the simulation is strictly real time but more inter-
simultaneously as we discuss below. ested in whether they see a smooth, yet accurate
Consider the following illustration: a simula- and expressive, visualization of the data in situ.
tion uses a dtsim ¼ 0.01 seconds. The throughput This forces us to consider other factors when
of the calculation is 50,000 grid cells every second defining performance. For simulations to be visu-
based on our chosen compute device. The resolu- alized smoothly in situ, it is widely accepted that
tion of the calculation is such that we have a 2D we need a refresh rate of our display of at least
domain 60  30 cells. This gives 1800 grid cells 24 frames per second (FPS). Returning to the
in the domain. Our compute device can then previous example, we would need to update our
update the whole grid around 28 times each display every iteration, giving us a frame rate of
Computational Steering for Computational Fluid Dynamics 379

28 > 24 FPS. Based on the throughput, we have a (2017, 2018); Harwood 2019) explores the feasi-
maximum perceived flow scale of up ¼ 28 cells bility of developing performant fluid dynamics
per second – i.e., the rate at which information is applications for mobile CPUs and GPUs. They
passed through our grid, one cell at a time. If we develop interactive CFD simulations suitable for
have 60 cells across a 30 cm width screen, then mobile devices and test their implementations for
this is a perceived physical flow rate of 0.14 m/s. a range of problem sizes. In Harwood and Revell
Of course, this is an upper limit achieved by (2018), the authors implement their simulation C
acoustics in most cases, and typically, flow fea- software using the CUDA API and use a combi-
tures of interest will propagate much slower than nation of software development kits to compile
this. We must be aware that structures of interest the application. The application successfully har-
may then propagate too slowly for interactive nesses the mass parallelism and power efficiency
applications. In these situations, our options are: of the GPU to be over 300x more efficient in terms
of combined throughput and power consumption
• Reduce refresh rate – encourage information to than the earlier CPU implementation (see
propagate more before updating the screen. Table 1). However, the work outlines the com-
• Reduce the time step (resolution) – often diffi- plexity at present in creating mobile applications
cult without introducing errors and instabilities which can efficiently leverage the raw power of
in the numerical method. mobile hardware.
• Increase the parallel capacity (throughput) – Recent work (Harwood (2019)) has explored
often limited by available computing the use of multiple mobile devices running a dis-
resources. tributed interactive simulation implemented using
OpenGL ES compute shaders and Java sockets.
As is discussed in Harwood (2019), as well Contained within an Android application, the
as in our example above, the adjustment of work examines the role of communication strat-
any one of these parameters in the pursuit of egy as well as refresh rate (termed render fre-
more speed, a higher accuracy, or a smoother quency fR in the entry) on the performance of the
frame rate may have an adverse effect on simulation as discussed in section. The render
another metric; thus, the pursuit of interactive frequency proved to be an influential factor in
simulations is almost always a delicate terms of overall computational throughput
balancing act. (Table) with the time to render the screen not
insignificant when performed frequently. Further-
more, the restriction of OpenGL render threads to
Using Mobile Devices for CFD 60 FPS by the Android OS artificially limited the
throughput. This is something that is universally
Mobile devices (phones and tablet computers) are applicable for general-purpose GPU computing
perhaps uniquely placed for computational
steering activities as user interaction with a mobile
form factor is more convenient and intuitive.
Computational Steering for Computational Fluid
However, the use of mobile hardware for Dynamics, Table 1 Comparison between CPU and
conducting CFD simulations of a degree of fidel- GPU implementations. (Data taken from Harwood and
ity appropriate for engineering applications has Revell 2018)
been largely unexplored in the literature to date. CPU GPU
Being able to perform interactive CFD simula- Measure (6-tasks) (3 warps block)
tions on mobile devices allows the development MLUPS 1.1 14.83
of portable, affordable simulation tools that can Memory usage (MB) 5.9 6.0
have a significant impact in engineering design as Energy/physical second 5.23 0.232
(kW h  103)
well as teaching and learning. The work of
Real-time ratio (Tr) 0.0395 0.832
Harwood and Revell (Harwood and Revell
380 Computational Steering for Computational Fluid Dynamics

which makes use of OpenGL for its implementa- involved and tool chains used to build the soft-
tion (Fig. 4). ware can vary dramatically from platform to plat-
The time taken to conduct the 2D simulations form and, thus, have a huge effect on the
over a wide range of resolutions, as expected, was performance of the final application. Applications
significantly lower than the time taken to pass can be developed as native applications, web
information between the networked devices. As applications, and hybrid applications.
such, the results of testing multiple devices Native applications use the native languages
(Fig. 5) illustrate two key findings: first, that and toolkits associated with a platform. On
with calculations at very high resolutions, it is Android, codes written in Java and C++ using
difficult to hide this communication latency by the Android SDK/NDK are forms of native devel-
adjusting the render frequency and perceptible opment. On iOS, code is written in Swift or
stuttering in the visualization is inevitable. Sec- Objective-C. Native applications are fast and can
ond, the weak scaling of the algorithm across leverage the full set of device capabilities through
multiple devices is relatively efficient; thus, platform-specific APIs. They give the best user
large-scale simulations across a cluster of mobile experience but at the expense of having to develop
devices is likely to be possible. a different code base for every target platform.
Cross-platform applications can be developed
Types of Mobile Application Development using a higher-level suite of tools and then built
Developing applications for mobile devices is a for multiple platforms from the single, common
similar process to developing a desktop applica- code base. Web applications are the simplest
tion with a user interface. However, the languages approach to cross-platform development. Apps

102
102
FPS > 24fps

101
Frames Per Second (FPS)

FPS > 24fps

MLUPS

101

100
32 32
64 64
128 128
256 256
512 512
1024 1024
24fps 24fps
100 10-1
100 101 102 103 100 101 102 103

(a) Performance in terms of MLUPS. (b) Performance in terms of FPS.

Computational Steering for Computational Fluid n and render frequencies fR. The target region for interac-
Dynamics, Fig. 4 Performance of the current GLSL- tive simulations is 24 FPS and is indicated by the shaded
based application for a combination of different resolutions regions. (Taken from Harwood 2019)
 8 500 25 1000 50 2000

400 20 800 40
6 1500
15
300 600 30
4 10 1000

MLUPS

MLUPS
MLUPS
200 400 20

Time (ms)
Time (ms)

Time (ms)
5
2 500
100 0 200 10

0 0 -5 0 0 0
100 101 102 103 100 101 102 103 100 101 102 103

70 4000 80 8000
Computational Steering for Computational Fluid Dynamics

60
3000 70 6000
50 2 Devices (MLUPS)
3 Devices (MLUPS)
40 2000 60 4000 2 Devices (Comm Time)

MLUPS
MLUPS
3 Devices (Comm Time)
Time (ms)

Time (ms)
30
42ms Reference
1000 50 2000
20

10 0 40 0
100 101 102 103 100 101 102 103

Computational Steering for Computational Fluid Dynamics, application run with the same configuration of resolution n and render frequency fR.
Fig. 5 Performance in million lattice updates per second (MLUPS) (left axis) and The green dotted line represents a time of 42 ms, the time of a single frame at 24 FPS.
time taken to conduct communication (right axis) of the multi-device application. (Taken from Harwood 2019)
Shaded gray rectangles indicate the range of performance in MLUPS of the serial
381

C
382 Computational Steering for Computational Fluid Dynamics

are quicker to develop and easier to maintain but is simply not on the same level as desktop sys-
lack access to all device capabilities and are tems. Researchers have recently explored the
restricted in many respects by the web browser potential of using local clusters of mobile devices
within which they run. Typically these applica- to share computing power through peer-to-peer
tions are written using a platform agnostic API networking (Harwood (2019)). Ultimately,
like the JavaScript, CSS, HTML, or combination Harwood and Revell (2018) conclude that the
of Cordova (PhoneGap). trade-off between accuracy, speed, and power
Hybrid applications are usually a combination consumption is explored with the choice of prob-
of the two approaches with some code written in a lem resolution ultimately being characterized by a
platform agnostic language leveraging a cross- desired accuracy, flow speed, and endurance of a
platform API for most device functionality. UI given simulation.
elements are translated to native UI components,
but some functionality may still need to be
implemented on a per-platform basis. GPU-Based Approaches
For maximum performance, such as would be
required by simulation applications, native In addition to mobile devices that allow to run
approaches are recommended. The just-in-the numerical simulations in the palm of your hand,
compilation of the Java Virtual Machine on another recent trend is to harness the parallel
Android can further be circumvented by writing computational power of graphics processing
performance-critical parts of code in C++, pre- units (GPUs) not only for rendering purposes but
compiling these components as libraries using also for general-purpose applications. Effectively,
the Android NDK, and then linking these compo- the availability of these compact, highly efficient
nents to the Java application through the Java accelerator boards introduced a new era of local
Native Interface (JNI). This approach is taken in supercomputing. NVIDIA introduced the first
Harwood and Revell (2018) to maximize perfor- GPU in 1999 (NVIDIA 2013a) as a dedicated
mance while enabling the use of CUDA for a rendering machine, handling all pipeline steps
supported mobile GPU. from vertex generation to pixel operations. Prior
to the release of the GeForce 256, the vertices of
Challenges of Using Mobile Devices for objects were generated and processed on the cen-
High-Performance CFD tral processing unit (CPU) and then sent to an add-
Mobile device hardware is designed to maintain a in graphics accelerator like the 3dfx Voodoo card,
delicate balance between performance and power which generated and processed primitives and
efficiency. In addition, mobile device software is fragments, and finally operated on the pixels
designed to enable an uninterrupted user experi- (Kirk and Hwu 2010; Patterson and Hennessy
ence ensuring (UIs) remain available at all times. 2011; Akenine-Moller et al. 2008). In late 2006
All applications written for mobile devices must NVIDIA then released the GeForce 8800 (G80),
necessarily be multi-threaded. The restriction on one of the first unified graphics and computing
the number of parallel threads and the reduction in GPUs (NVIDIA 2013a). Using the NVIDIA
priority of threads other than the UI thread are CUDA API and the built-in CUDA cores of the
compromises applied by the mobile operating GeForce graphics card, programmers without any
system (OS) in order to maintain this balance. background in graphics or familiarity with the
Developer design guidelines, such as those graphics pipeline were now able to implement
provided by Google for Android application their code on GPUs. This level of graphics card
design, are an essential source of advice when utilization for general-purpose applications is
writing native applications where delivering called GPGPU computing. Ever since the release
every last bit of performance possible is the pri- of NVIDIA’s G80, a continuous growth in com-
mary aim. However, when developing for mobile pute capability regarding floating-point opera-
devices, we must accept that the compute capacity tions per second (FLOPS), memory bandwidth,
Computational Steering for Computational Fluid Dynamics 383

Computational Steering for Computational Fluid Dynamics, Fig. 6 The VirtualFluids Interactive simulation
environment showing the airflow around cooling towers. (Taken from Linxweiler et al. 2011)

and number of transistors and streaming proces- example shows that convergence of massive par-
sors took place (NVIDIA 2013b). As an alterna- allel computational power and a steering environ-
tive to traditional massively parallel computing, ment into a single system significantly improves
GPU computing has also gained popularity in the the usability, the application quality, and the user-
CFD community as it allows for interactive 3D friendliness. Furthermore, using multiple GPUs,
simulations at reasonable problem sizes, as the approach allows for 3D simulations close to
discussed in the following. real time even for reasonable problem sizes. In
this case, responsiveness significantly benefits
Hybrid Visualization Approaches (GPU + CPU) from avoiding common bandwidth and latency
Linxweiler et al. (2007, 2011) adapted GPU com- bottlenecks inherent with traditional HPC clus-
puting for computational steering in CFD and ters. Those can be avoided as GPU computing
demonstrated the benefits from the use of GPUs. does not generally require network communica-
They developed a single desktop application tion, which also reduces the complexity of the
(VirtualFluids Interactive) (Fig. 6) integrating a application. Compared to traditional massive par-
complete pipeline for interactive CFD simulation allel environments, GPUs are affordable for small
including pre- and post-processing as well as the to medium enterprises and do not require addi-
simulation. In this approach the preprocessing and tional HPC knowledge from the end users. In
visualization are running on the CPU; likewise, addition, GPUs reduce power consumption with
the simulation is executed on the GPU. The a much better performance-per-watt ratio.
384 Computational Steering for Computational Fluid Dynamics

Computational Steering for Computational Fluid Germany, (b) in a typical classroom situation on a smart
Dynamics, Fig. 7 ELBEvis in action: (a) during the board. (Taken from Koliha et al. 2015)
poster session at the GACM 2013 conference in Hamburg,

Direct CUDA-OpenGL Interoperability and platform independence. OpenGL wraps

While the previous approach benefits from a fea- graphics driver calls into C commands and pro-
tured user interface with maximum usability, the vides additional functionality for vertex, primi-
method still relies on frequent data transfer from tive, and fragment generation and processing.
the GPU device back to the host and vice versa. Libraries such as the OpenGL extension wrangler
Such extensive communication via the PCIe bus (GLEW) or the OpenGL utility library (GLU)
hinders real-time simulations and interactivity, as were utilized to make use of OpenGL function
already discussed. For some applications, alterna- bundles.
tive concepts where visualization and data man- Due to the minimal-invasive concept and the
agement happen completely on the device have high performance of the numerical solver,
shown to be more feasible, without moving or ELBEvis allows for very responsive interactions
copying any data from or to the host during the between the user and the numerical simulation. In
simulation at all. Fig. 7, two examples for the use of ELBEvis are
Delbosc et al. (2014) present an implementa- shown, for a 2D application case with simple
tion of an optimized three-dimensional real-time inlet/outlet boundary conditions on the left/right
thermal and turbulent fluid flow solver with a end of the computational domain. The simulation
performance of half a billion lattice node updates is run on an NVIDIA GTX Titan board that is
per second. The method then has been applied to attached to an off-the-shelf laptop through an
study the time evolution of the turbulent airflow ExpressCard slot. The user can control the visual-
and temperature inside a test chamber and in a ization by selecting the desired visualizer features,
simple model of a four-bed hospital room (Khan color maps, and so on. On top, the computational
et al. 2015). domain can be modified by freely drawing shapes.
The authors of Koliha et al. (2015) present The effects on the flow field can immediately be
ELBEvis, a framework that is based on observed, which makes the coupled solver a great
OpenGL-CUDA interoperability (NVIDIA tool for understanding the basics of fluid mechan-
2013b) and maps OpenGL buffer objects, to the ics. Glessmer and Janßen (2017) discuss how to
CUDA context for use in its respective memory successfully use ELBE and ELBEvis in different
space. This allows CUDA routines to read and teaching scenarios for Bachelor, Master, and PhD
write from and to data arrays that are then used level work.
for rendering. The OpenGL graphics API has Apart from these interactive real-time applica-
been chosen by the authors to generate the images tions, ELBEvis can also be used for simulations
displayed on the screen due to its widespread use that run near real time, as the visualizer still
Computational Steering for Computational Fluid Dynamics 385

allows to control of the simulation progress and to Geist II, G.A., Kohl, J.A., Papadopoulos, P.M.:
analyze the flow field characteristics. For exam- CUMULVS: providing fault-tolerance, visualization
and steering of parallel applications. Int. J. High Per-
ple, Überrück and Janßen (2017) elaborately dis- form. Comput. Appl. 11, 224–236 (1996)
cuss the simulation of wave impact. The highly Glessmer, M., Janßen, C.: Using an interactive lattice
resolved numerical simulations were run approx- Boltzmann solver in fluid mechanics instruction. Com-
imately one order of magnitude slower than real putation. 5(4), 35 (2017). https://doi.org/10.3390/
time and could benefit from the innovative visu-
computation5030035. ISSN 2079-3197
Hart, D., Kraemer, E.: Consistency considerations in the
C
alizer concept. Thanks to the interactive monitor- interactive steering of computations. Int. J. Parallel
ing of the simulations, without time-consuming Distrib. Syst. Netw. 2, 171–179 (1999)
file-I/O and subsequent visualization with CPU- Harwood, A.R.G.: Interactive modelling and simulation
for engineering design and analysis. NAFEMS Bench-
based visualizer tools, information on the sloshing mark Mag. 2018, 20–24 (2018)
process and the model quality could be extracted Harwood, A.R.G.: GPU-powered, interactive flow simula-
easily as discussed in the entry. tion on a peer-to-peer group of mobile devices. Adv.
Eng. Softw. 133, 39–51 (2019). https://doi.org/10.
1016/j.advengsoft.2019.04.003
Harwood, A.R.G., Revell, A.J.: Parallelisation of an inter-
active lattice Boltzmann method on an android-powered
Summary mobile device. Adv. Eng. Softw. 104, 38–50 (2017).
https://doi.org/10.1016/j.advengsoft.2016.11.005. ISSN
Tremendous developments in recent years have 0965-9978. http://www.sciencedirect.com/science/arti
brought more and more computing power to the cle/pii/S0965997816301855
Harwood, A.R.G., Revell, A.J.: Interactive flow simulation
desktop and into the end user’s palm. Harnessing
using Tegra-powered mobile devices. Adv. Eng. Softw.
this power requires innovative concepts for the 115, 363–373 (2018). https://doi.org/10.1016/j.
visualization of the numerical data and, at the advengsoft.2017.10.005. ISSN 0965-9978. http://
same time, allows for innovative new concepts www.sciencedirect.com/science/article/pii/S09659978
17307135
of computational steering. Successful examples
Janßen, C.F., Bindick, S.: jFlowSim v0.8-beta. (2019).
include interactive steering in the context of con- https://doi.org/10.5281/zenodo.2653571
ventional CPU solvers, solvers that make use of Johnson, C.R., Parker, S.G.: A computational steering
GPU acceleration and, most recently, first model applied to problems in medicine. In: Super-
computing ‘94: Proceedings of the 1994 Conference
attempts to use the compute power of mobile
on Supercomputing, pp. 540–549. IEEE Computer
devices for numerical simulations and computa- Society Press, Los Alamitos (1994)
tional steering. Johnson, C., Parker, S.G., Hansen, C., Kindlmann, G.L.,
Livnat, Y.: Interactive simulation and visualization.
Computer. 32(12), 59–65 (1999)
Khan, M.A.I., Delbosc, N., Noakes, C.J., Summers, J.:
References Real-time flow simulation of indoor environments
using lattice Boltzmann method. Build. Simul. 8(4),
Akenine-Moller, T., Haines, E., Hoffman, N.: Real-Time 405–414 (2015)
Rendering. A K Peters/CRC Press (2008). ISBN: Kirk, D., Hwu, W.: Programming Massively Parallel Pro-
978-1-56-881424-7 cessors. Elsevier Inc, Amsterdam (2010). ISBN 978-0-
Borrmann, A.: Computerunterstützung verteiltkoo- 12-381472-2
perativer Bauplanung durch Integration interaktiver Koliha, N., Janßen, C., Rung, T.: Towards online visuali-
Simulationen und räumlicher Datenbanken. PhD the- zation and interactive monitoring of real-time cfd sim-
sis, Technische Universität München (2007) ulations on commodity hardware. Computation. 3(3),
Borrmann, A., Wenisch, P., van Treeck, C., Rank, E.: 444478 (2015) . ISSN 2079-3197). https://doi.org/10.
Collaborative computational steering: principles and 3390/computation3030444
application in HVAC layout. Integr. Comput. Aid. Krüger, T., Varnik, F., Raabe, D.: Shear stress in lattice
Eng. 13(4), 361–376 (2006) Boltzmann simulations. Phys. Rev. E. 79, 046704
Delbosc, N., Summers, J.L., Khan, A.I., Kapur, N., (2009)
Noakes, C.J.: Optimized implementation of the lattice Kühner, S.: Virtual Reality-basierte Analyse und
Boltzmann method on a graphics processing unit interaktive Steuerung von Strömungssimulationen im
towards real-time fluid simulation. Comput. Math. Bauwesen. PhD thesis, Technische Universität
Appl. 67, 462–475 (2014) München (2003)
386 Computed Fluid Flow

Linxweiler, J., Tölke, J., Krafczyk, M.: Applying modern Überrück, M., Janßen, C.: On the applicability of lattice
soft- and hardware technologies for computational Boltzmann single-phase models for the simulation of
steering approaches in computational fluid dynamics. wave impact in lng tanks. Int. J. Offshore Polar Eng.
In: Cyberworlds, 2007. CW ‘07. International Confer- 27(4), 390–396 (2017)
ence on, pp. 41–45 (2007) van Liere, R., Mulder, J.D.: Ubiquitous computational
Linxweiler, J., Krafczyk, M., Tölke, J.: Highly interactive steering. In: IEEE Visualization ‘97 (1997)
computational steering for coupled 3d flow problems Van Liere, R., Van Wijk, J.J.: Cse – a modular architecture
utilizing multiple gpus. Comput. Vis. Sci. 1–16 13(7), for computational steering. In: Proceedings of the Sev-
pp. 299–314 (2011). https://doi.org/10.1007/s00791- enth Eurographics Workshop on Visualization in Sci-
010-0151-3 entific Computing, pp. 257–266. Springer, New York
Marshall, R., Kempf, J., Dyer, S., Yen, C.-C.: Visualization (1996)
methods and simulation steering for a 3d turbulence van Liere, R., D. Mulder, J., van Wijk, J.J.: Computational
model of Lake Erie. SIGGRAPH Comput. Graph. steering. Futur. Gener. Comput. Syst. 12(5), 441–450
24(2), 89–97 (1990) (1997)
McCormick, B.H., De Fanti, T.A., Brown, M.D.: Visuali- Van Wijk, J., Van Liere, R.: An environment for computa-
zation in scientific computing: report of the nsf advi- tional steering. In: Centre for Mathematics and Com-
sory panel on graphics, image processing and puter Science (CWI), pp. 23–27. Computer Society
workstations. ACM Comput. Graph. 21(6), (1987) Press (1997)
Mulder, J.D., van Wijk, J.: 3D computational steering with Van Wijk, J.J., Van Liere, R., Mulder, J.D., Van Wijk, J.J.,
parametrized geometric objects. In: Visualization ‘95 Van Liere, R., Mulder, J.D.: Bringing computational
(Proceedings of the 1995 Visualization Conference), steering to the user. In: Presented at the Dagstuhl Sem-
pp. 304–311. IEEE Computer Society Press, Los inar on Scientific Visualization, pp. 304–313 (1997)
Alamitos (1995) Wenisch, P.: Computational Steering of CFD Simulations
Mulder, J.D., van Wijk, J.J., van Liere, R.: A survey of on Teraflop-Supercomputers. Ph.D. thesis, Technische
computational steering environments. Futur. Gener. Universität München (2008)
Comput. Syst. 15(1), 119–129 (1999) Wenisch, P., Wenisch, O., Rank, E.: Harnessing high-
NVIDIA.: NVIDIA’s Next Generation CUDA Compute performance computers for computational steering.
Architecture: Fermi. NVIDIA Whitepaper (2013a). In: Recent Advances in Parallel Virtual Machine and
http://www.nvidia.com/content/PDF/fermi_white_papers Message Passing Interface, pp. 536–543 (2005)
/NVIDIA_Fermi_Compute_Architecture_Whitepaper.pdf Wright, H., Crompton, R.H., Kharche, S., Wenisch, P.:
NVIDIA.: CUDA C Programming Guide. Technical Steering and visualization: enabling technologies for
report, NVIDIA (2013b) computational science. Futur. Gener. Comput. Syst.
Parker, S.G., Johnson, C.R.: Scirun: a scientific program- 26(3), 506–513 (2010)
ming environment for computational steering. In:
Supercomputing ‘95: Proceedings of the 1995
ACM/IEEE Conference on Supercomputing
(CDROM), p. 52. ACM, New York (1995)
Parker, S.G., Hansen, C.D., Johnson, C.R., Miller, M.: Computed Fluid Flow
Computational steering and the scirun integrated prob-
lem solving environment. In: DAGSTUHL ‘97: Pro- ▶ Fluid Simulation
ceedings of the Conference on Scientific Visualization,
p. 257. IEEE Computer Society, Washington, DC
(1997a)
Parker, S.G., Johnson, C.R., Beazley, D.: Computational
steering software systems and strategies. IEEE Computer Aided Design
Comput. Sci. Eng. 4(4), 50–59 (1997b)
Parker, S.G., Weinstein, D.W., Johnson, C.R.: The scirun
computational steering software system. In: Arge, E., ▶ Collaborative Engineering and Virtual Pro-
Bruaset, A.M., Langtangen, H.P. (eds.) Modern Soft- totyping Within Virtual Reality
ware Tools for Scientific Computing, pp. 5–44.
Birkhauser Boston Inc., Cambridge, MA (1997c)
Patterson, D., Hennessy, J.: Computer Organization and
Design. Elsevier Inc., Burlington (2011). ISBN 978-0-
12-374750-1 Computer Aided Engineering
Pickles, S.M., Haines, R., Pinning, R.L., Porter, A.R.:
A practical toolkit for computational steering. Philos.
Transact. A Math. Phys. Eng. Sci. 363(1833), ▶ Collaborative Engineering and Virtual Pro-
1843–1853 (2005) totyping Within Virtual Reality
Computer Games and Artificial Intelligence 387

Games and Game Theory

Computer Baduk
Games have traditionally existed in versions
▶ Computer Go involving more than one player (Books 2010)
and those that do constitute a formalized form of
interaction that is more than physical activity or
information exchange. The literature on games in C
Computer Games and general – as well as on specific games and their
Artificial Intelligence individual rules – is extensive (Tylor 1879). In the
field of game theory, which is concerned with the
Hanno Hildmann formal analysis of games and their properties, one
Departamento de Ingenieria de Sistemas y traditionally (Nash 2001; Osborne and Rubenstein
Automatica, Universidad Carlos III de Madrid, 1994; von Neumann and Morgenstern 1974) dis-
Leganes/Madrid, Spain tinguishes a variety of game types:

• Noncooperative vs. cooperative games: With

Synonyms regard to the actions available to a player dur-
ing a game, we call an action primitive if it
Computer Games: Video Games cannot be decomposed further into a series of
Artificial Intelligence: Machine Intelligence; Arti- other, smaller actions. The distinction is then
ficial Cognitive Intelligence made on the basis of whether such primitive
actions are performed by individual players
(noncooperative) or by groups of players
Definition (cooperative).
• Strategic vs. extensive games: A game is stra-
Computer Games are games that are realized in tegic if all decisions of all players are made a
one form or another using digital hardware. Arti- priori (i.e., before the start of the game) and
ficial Intelligence is commonly considered a independent of the opponent’s decision (e.g.,
branch of Computing Science which has matured Paper-Rock-Scissors). Extensive games allow
to become a field of its own right. It is concerned for decisions to be made during game play and
with the creation/simulation of intelligent behav- with respect to the opponents moves (e.g.,
iour in/by machines. Chess).
• Perfect information vs. imperfect information:
In, e.g., Chess both players have perfect insight
Background into the state of the game; thus Chess is a game
of perfect information. However, in most card
In the context of computer games and the com- games, there is an element of uncertainty as
puter games industry, games have become an one normally does not know the distribution of
essential part of the software and entertainment cards for the opponents. The latter games are
industry (Saarikoski and Suominen 2009). imperfect information games.
Recently there is increased interest in games
(Lowood 2009) from a sociological, cultural, Today, games are mainly of interest to the AI
and educational point of view (Hildmann and community if they are too complex to calculate all
Hildmann 2012a, b). For decades now, it has possible variations. For example, the game Nim,
been argued that games and the playing of though having a rich mathematical theory behind
games are culturally significant (Mechling it, is small enough to calculate a winning strategy
1987). (Conway 2001). Due to its mathematical
388 Computer Games and Artificial Intelligence

foundation, it has left its mark on combinatorial game of Chess – famously called the “Drosophila
game theory through terminology (e.g., nim- of AI” (McCarthy 1990). It was used by key
addition and nimsum) (Jørgensen 2009). How- figures in the field of artificial intelligence (such
ever, it is possible to precalculate the moves and as, e.g., Claude Shannon 1950; Allen Newell et al.
play a perfect game as there are known algorithms 1988; Herbert Simon 1970) in the context of their
for doing so (Spencer 1975); thus the theoretical research on machine and human intelligence. The
winner is already known, and there exist straight- interest in this game predates the computer age:
forward algorithms to determine which side will Charles Babbage already considered using his
have this advantage (Bouton 1902). Due to this, “Analytical engine” to play Chess in the 1840s,
the game is uninteresting today for both game and by 1949 researchers on both sides of the
theoreticians and researchers in AI. This was not Atlantic were advancing theoretical approaches
always the case though, and in the early years, a to automate the playing of this game (Hsu et al.
number of devices were built to play the game: 1995). Similarly, Checkers was used by, e.g.,
e.g., “Nimatron,” built by Westinghouse for the Arthur Samuel to study learning (Jørgensen
New York Worlds Fair in 1940 (http://www. 2009).
goodeveca.net/nimrod/nimatron.html) or “Nim- In the past, specifically the Atari 2600 game
rod,” developed for the Festival of Britain in console has been used extensively as a platform
London in 1951. These games were the highlights for developing and demonstrating AI algorithms.
where they were shown, with Nimatron playing The reasons for this are the that there are over
100.000 games (and winning 90.000) and Nimrod 900 game titles available for the Atari 2600 con-
“taking on all comers” at the Festival in London sole, the games are simple and concentrate on the
(Jørgensen 2009). problematic aspects of the game (while newer
games showcase the latest in video and graphic
performances), they have a small and discrete
Games and Artificial Intelligence action space, and many emulators are available
(Naddaf 2010). Games will continue to be a focal
Yan (2003) lists artificial intelligence as an inher- point of AI research. For example, the game Octi
ent design issue for online games, and indeed, (Schaeffer and van den Herik 2002) is a game
“appropriately challenging” AI is considered to specifically invented to be resistant to computer
be crucial to the commercial success of a game algorithms.
(Aiollil and Palazzi 2009). Generally it can be said
that “non-repetitive, adaptive, interesting, and in
summary intelligent behavior offers a competitive Ethical Considerations
edge for commercial games” (Naddaf 2010).
As the combinatorial challenge of complex Many things, including far-reaching business and
games such as Chess and Go (discussed below) political decisions, can be modeled as a game, and
is successively mastered, some emphasis (e.g., the application of artificial intelligence to the
Hildmann and Crowe 2011) is placed on design- therefore broad area of games has stirred many
ing AI players that instead of playing very well can controversies over the years. Among the relevant
play realistically in the context of playing non- discussion topics are whether a computer can be
repetitively and in an interesting way. Emphasis is said to play at all and whether computers should
placed on intelligent behavior (Hildmann 2013) in be allowed to play with (meddle in) some of the
the way that a human player would play, including more serious matters that can be cast into a game-
the handicaps, shortcomings and common mis- theoretic model.
takes made by humans (Hildmann 2011). For example, Weizenbaum (1972) and Wiener
Besides the commercial interest of the gaming (1960) wrote on the moral and ethical issues of AI
industry, the research field itself has always had a and allowing computers to make decisions, the
large interest in games. The prime example is the latter being criticized by Samuel (1960) and the
Computer Games and Artificial Intelligence 389

former by Coles (1972). Taube (1960) wrote in humans to exercise the brain (Smith 1956) and to
response to Wiener (1960) that machines cannot compete against each other on the basis of mastery
play at all. He supposes that the act of playing is in of skill and intelligence.
line with enjoyment, and the author makes the Therefore, games have been of large interest to
distinction between the notions of game as it is the AI community, and the design of programs
“used in the usual sense” contrasted with game that can match the performance of human top
after it is “redefined by computer enthusiasts with players has been considered the ultimate achieve- C
nothing more serious to do.” ment for a long time. Once a program can hold its
Johnson and Kobler (1962) agree with Wiener own against a human, the next step is to try and
(1960) that machines can be original but warn of outperform humans, and since the mid-1990s, one
allowing them to play war-games and simulations game after the other has been mastered by com-
of nuclear war, as their predictions might not be puters: in 1995 a program called “Chinook”
entirely accurate and they would lack the under- played the game of Checkers at world class level
standing of all the values that need to be consid- and succeeded in beating one of the world’s best
ered (i.e., to gain the insight ultimately exhibited players (Kroeker 2011). In 1996 “Deep Blue” beat
by the fictional computer system “Joshua”). While the reigning world champion in the game of Chess
the moral issues of artificial intelligence and polit- for the first time in history (Kasparov 1996), and
ical decisions have recently become a mainstream 1 year later, it won an entire chess tournament,
matter of discussion, this was already discussed in ending the supremacy of humans in this game
the context of game theory and its application (Schaeffer and van den Herik 2002). The TV
to the nuclear “first strike” doctrine in the cold show game Jeopardy! was conquered decisively
war. It was a well-known matter of disagreement by “Watson” in 2011. Poker, a game of bluffing
between Wiener and vonNeumann; the latter and intuition, was mastered in 2015 (Bowling
advocated the idea of a preemptive nuclear strike et al. 2015) and within months from each other
on Russia. His argument was that their reasoning (December 2016 and January 2017), two pro-
would lead them to the same conclusion, making grams, “Libratus” (Riley 2017) and “DeepStack”
it simply a matter of time before one side struck (Moravčík et al. 2017), significantly outper-
first. Since the mid-1990s, computers have suc- formed dozens of professional and top human
cessively dismantled the reign of human players players. Finally, the best human players of Go –
in the realms of perfect information games. More the board game considered by many to be the last
recently, this has also happened in games with great stronghold of human superiority in games –
imperfect information, where intuition or the recently succumbed to “AlphaGo.”
understanding of complex real-world relation- Since 2005 the Stanford Logic Group of
ships is required. Machines are increasingly mas- Stanford University, California is running the
tering the decision-making in situations that go General Game Playing Competition (http://
beyond those found in Chess and Go. Whether games.stanford.edu/) (GGPC) (Genesereth and
mastery of these situations includes the under- Love 2005) which is inviting programs to com-
standing that, e.g., “mutually ensured annihila- pete in playing games. The games to be played
tion” is undesirable even if it constituted a are, however, unknown in advance, and the pro-
victory by points is a question worth asking (but grams have to be able to compete in a number of
not the subject of this entry). different types of games, expressed in a Game
Description Language (Love et al. 2008).
“AlphaGo Zero” recently not only learned the
Landmark Artificial Intelligence games Shogi, Chess, and Go – all of which had
Victories defined computer programs for decades – from
scratch (at a significant computational cost, Silver
Games are culturally important (Sutton-Smith et al. (2017)), it also proceeded to beat the best
1986) and have been used for millennia to allow programs currently available for each of these
390 Computer Games and Artificial Intelligence

games. This might be an indication that the GGPC human versus machine intelligence (despite the
competition might also soon come to an end. game of Go being more complex). Far more
Progress is occurring at a nonlinear speed, and importantly, however, is that Kasparov later
certainly in the area of artificial intelligence, the claimed that during the game he “could feel” and
recent years have seen milestone events happen “smell” a “new kind of intelligence across the
with increasing frequency. table” (Kasparov 1996). What he meant was that
Before we briefly elaborate on some of the the moves of his opponent bore witness to an
most prominent examples of artificial intelli- intelligence that he, at the time, did not believe a
gence systems winning against top human computer could exhibit. It is not the superior play
players, it should be pointed out that there does and triumphant victory at its end but the outstand-
not seem to be a silver bullet, as the discussed ing demonstration of insight and intelligence that
games were conquered with different techniques makes Game 1 of the 1996 “Deep Blue versus
and approaches. The game of Chess was essen- Garry Kasparov” match a turning point in the
tially conquered by throwing sufficiently large history of artificial intelligence. While Kasparov
amounts of computational resources at it ended up winning the tournament (2–4), he had,
(as well as training the program to play a specific arguably, conceded a win of the Turing test to
human opponent) while the original program that Deep Blue. This test, famously proposed by
mastered Go used advanced machine learning Turing (1950), elegantly sidesteps the need to
techniques (combined with Monte Carlos Tree formally define the concept of intelligence before
Search (MCTS)) and had access to a large data- being able to assess it. It does so by suggesting
base of previously played games. Its revised ver- that if the behavior of an opponent (as opposed to
sion (which learned to play Go as well as Chess the physical appearance) could fool a human
from scratch) relied on learning from massive judge into believing that this opponent was
numbers of games it would play against itself. human, then that opponent should be considered
In contrast, the program that won Jeopardy! intelligent, independent of the embodiment of the
applied probabilistic methods before selecting intelligence. Arguably, in February of 1996, Deep
the most likely answer. A full technical discus- Blue did just that.
sion of these programs is beyond the scope of this In contrast, and despite being considered a
entry; the interested reader is referred to the pro- “watershed for AI” (Hassabis 2017), when Deep
vided references for in-depth discussion on the Blue finally defeated Kasparov 1 year later in
matter. 1997 3 12 to 2 12 , this constituted much less of a
victory in the sense of the Turing test. This date is
Chess: Deep Blue considered a “major success for artificial intelli-
In his recent book (Kasparov and Greengard gence and a milestone in computing history”
2017), Gary Kasparov writes that over his career, (Schaeffer and van den Herik 2002), but as
starting at the age of 12, he played about 2400 Wiener wrote 37 years earlier, pure calculation is
serious games of Chess and lost only around only a part of the process. One needs to consider
170 of these games. In 1989 he had played, and the playing history of the opponent as well and
beaten, the computer program “Deep Thought.” be able to adapt to it accordingly during the
On February 10, 1996, playing as the reigning game. In the case of Deep Blue, the machine
World Chess Champion, he lost a game of Chess knew every major game Kasparov had ever
to a computer, Deep Blue (Kasparov 1996), but played while Kasparov was completely in the
still won the tournament. This date is significant dark about Deep Blue’s capabilities.
for a number of reasons: firstly, and most widely Of course one can argue that in 1996 this
known, for the first time (under normal tourna- was true in reverse, as Kasparov (1996) himself
ment conditions), a computer program beat a top acknowledged. He admitted that, after losing the
human player in a game which – until that day – first game, his defense “in the last five games was
was considered the ultimate benchmark for to avoid giving the computer any concrete goal
Computer Games and Artificial Intelligence 391

to calculate toward.” He stated that he knew the one of 3 was a massive one, and it was taken by
machine’s priorities and that he played accord- AlphaGo in March 2016. Until then the reigning
ingly; he closes by conjecturing that he has top computer program, “Zen” was only able to
“a few years left.” In fact, he had little more than prevail against top professional players with a
a year before, on May 11th, 1997, Deep Blue won handicap of 4 (Lee et al. 2016b).
the deciding game of 6, thereby defeating the AlphaGo, created by DeepMind, entered
human world champion 3 12 to 2 12. The stronghold the circuit around October 2015. It did not just C
of human superiority and intelligence had finally capitalize on improved hardware and increased
fallen, and other landmark victories of AI were computational power; it was built differently and
soon to follow. combined at least two successful techniques:
Monte Carlo Tree Search (Cazenave 2017) with
Go: AlphaGo Deep Learning (DL) (LeCun et al. 2015). Techni-
Go has been described as the Mount Everest of AI cal details of DL are discussed in Clark and
(Lee et al. 2016b). This is fitting in the sense that it Storkey (2014), Gokmen et al. (2017), and Jiang
represents the highest peak we can climb but not et al. (2017). As the program played its top con-
necessarily the most difficult thing conceivable: temporary computer programs “Crazy Stone,”
Go is maybe the most complex game that is actu- “Zen,” “Pachi,” “Fuego,” and “GnuGo,” it pro-
ally played by humans, with the number of theo- ceeded to beat them all (winning 494 of
retically possible games being in the order of 495 games with the single-machine version and
10700 (Wang et al. 2016) – a number expressing triumphing in every single game with the distrib-
a quantity larger than the number of atoms in the uted version) (Silver et al. 2016).
universe (Lee et al. 2010). But if it was merely AlphaGo first beat the European champion
complexity we were after, harder challenges could Fan Hui 5 to 0 in September 2015 (becoming
easily be designed. However, it is also the fact the first program ever to defeat a professional
that humans have engaged in playing Go for player) and within half a year proceeded to defeat
millennia and through this have reached high the reigning human world champion Lee Sedol
levels of mastery that makes it of interest to the (who by some is “hailed as the greatest player
AI community. In the context of squaring off of the past decade” (Hassabis 2017)) 4 to 1 in
humans against computers, Go is the likely can- March 2016 (Fu 2016). It won the first three
didate for the ultimate turn-based board game of games, taking home a sweep victory (best of 5),
perfect information. but maybe more importantly it awed top human
From 1998 to 2016, competitions pitting com- players, not unlike Deep Blue had awed
puter programs against human players have been Kasparov in 1996: commenting on the legendary
held every year at major IEEE conferences, with move 37 in AlphaGo’s second game against
the handicap imposed on the human players Lee Sedol, Fan Hui is quoted to have said:
dropping from 29 in 1998 to 0 in 2016 (Lee “[i]t’s not a human move. I’ve never seen a
et al. 2016a). It is important to understand that human play this move, [. . .] So beautiful . . .
the advantage gained from the handicap imposed beautiful” (Metz 2016).
on the human player is not linear in the size of the Toward the end of 2017, AlphaGo Zero was
handicap. The handicap is implemented as stones introduced as the next incarnation of the system.
the computer may place before the game starts; It not only learned three games (Shogi, Chess, and
therefore, e.g., an advantage of 4 allows the com- Go) autonomously from scratch (Silver et al.
puter to claim or fortify all four corners, while just 2017) but then proceeded to beat the top programs
one handicap (stone) less allows the human player currently playing these games. In a way, AlphaGo
to do the same for one corner, arguably allowing Zero is not a program designed to play Go but a
for entirely different game play (Lee et al. 2012), program designed to play according to a set of
especially on a smaller (9  9) board. On a full rules. More specifically, AlphaGo Zero can at
(19  19) board, the step from a handicap of 4 to least learn and play turn-based games of perfect
392 Computer Games and Artificial Intelligence

information without chance and has demonstrated program to defeat top human players (Metz
its ability to play these games at the expert level. 2017). This victory has not come easy, as
As such, AlphaGo Zero might mark the end of “Claudiro,” an earlier program by the same
this sub-area in computational challenges. We can team, failed in 2015. But while Claudiro was
(and will) surely continue to build better programs defeated, it was so only by a margin: after a
and let them play each other, but as far as the combined $170 million had been bet, the humans
question of superior game play is concerned, were up by a mere 3/4 of a million (Tung 2015).
humans have met (and created) their masters. Claudiro’s performance, despite falling short of a
decisive victory, indicated that Poker was not
Poker: DeepStack and Libratus outside of what’s possible. Within 2 years, its
Both Go and Chess are games of perfect informa- creators returned with Libratus, and another
tion: all information about the game is known by long-standing challenge in the field of artificial
both players at all times. Zermelo’s Theorem intelligence was met (Moravčík et al. 2017).
(Zermelo 1913 – English version of the paper: At the same time as Libratus was developed
Schwalbe and Walker 2001) states that determin- at the Carnegie Mellon University, another group
istic (not based on chance) finite two-player at the University of Alberta designed DeepStack.
games with perfect information [. . .] have a non- By the time of Libratus’ victory in early 2017,
losing strategy for one player (i.e., either one DeepStack had already competed in a tournament
player can, theoretically, force a win, or both and won (in December 2016), with the resulting
players can force a draw). This means that – the- publication undergoing peer review. While
oretically – one could calculate all possible ver- DeepStack played 33 professional poker players
sions the games of Chess and Go can be played from 17 countries, Libratus competed against four
and then simply never choose a move that results of the best human players in the world. Both pro-
in a defeat. While this is practically impossible, grams showed a statistically significant superior
due to the exceedingly high number of possible performance over their human opponents (Riley
games, it means that for these two games – in 2017). DeepStack’s performance exceeded the
theory – a machine could be built that would threshold of what professional players consider a
never lose a single match. Therefore it is only seizable margin by a factor of 10 (Moravčík et al.
practical limitations that prevent computers from 2017). Both programs approached the problem dif-
outperforming humans and all that it takes is ferently, which goes to show that the field of AI has
clever algorithms that get around these limita- not just chipped away at the (next) “last” stronghold
tions. That being said, actually achieving this has of human AI, but it has done so in multiple ways,
been considered a major achievement in the field indicating that these victories were not achieved by
(Jørgensen 2009) and by no means should the machines capitalizing on human shortcomings but
triumphs of Deep Blue and AlphaGo be belittled. on progressively refining techniques which enable
As stated above, the defeating of the top human computers to improve their playing of the game.
players in these games are landmarks in the his- Libratus is using a supercomputer at the Pitts-
tory of artificial intelligence. burgh Supercomputing Center to build an exten-
Heads-up no-limit Texas Hold’em, a popular sive “game tree” to evaluate the expected outcome
version of the game Poker is considered the main of a particular play. DeepStack instead uses a
benchmark challenge for AI in imperfect- neural network to “guess” the outcome of a play,
information games (Brown and Sandholm 2017). not entirely unlike how humans use “intuition”
In 2017, a Poker playing AI designed at Carnegie (Metz 2017).
Mellon University (CMU), going by the name
Libratus, prevailed against four top human players Jeopardy!: Watson
in a tournament and over the course of an aggre- In early 2011, a natural language question
gated 120.000 hands. Libratus became the first and answering program called Watson (named
Computer Games and Artificial Intelligence 393

after IBM’s founder Thomas J. Watson (Brown process had to be painfully constructed on the
2013)) became world famous after winning the basis of the testimony of human experts. In con-
US TV show Jeopardy! (Kollia and Siolas 2016). trast, Watson uses probabilistic methods that
This game show has been on TV since 1984 and result in confidence scores for answers (Ahmed
revolves around contestants correctly identifying et al. 2017). Due to this, the program is able to
questions, on the basis of being given the resulting attempt answers to problems it has never seen
answers. The maximum time allowed is 5 seconds, before, i.e., it can operate under incomplete or C
and three contestants compete for being the first to missing information (Gantenbein 2014).
correctly identify the question (Ferrucci 2010). One more thing sets Watson apart from the
Winning the game requires the ability to identify computer programs mentioned before and their
clues involving subtleties, irony, and riddles, and landmark victories: while being built to compete
as such, the competition is firmly within areas and win in the TV show Jeopardy!, the 3-day
where humans excel (and machines traditionally competition that made it famous was only a first
fail) (Brown 2013). step in its (intended) career. IBM considered the
Over the course of 3 days, from February game show a real-world challenge in the area of
14 to February 16, 2011, IBM’s Watson pro- Open Domain Question Answering (Ahmed et al.
ceeded to beat the two highest performing humans 2017), but winning against a human was a means
in the history of the game: Brad Rutter, who had to an end, not the goal. The ability to perform on
been the show’s largest money winner ever, and human levels when subjected to open domain
Ken Jennings, the record holder for the longest questions demonstrated for the first time that a
winning streak. Not only did Watson beat program could engage in such an activity, inde-
both humans, but also it utterly defeated them: in pendent of the setting or context. From the start
the end Watson had won in excess of $77,000 Watson’s creators had more in mind than merely
while its opponents combined won less than winning a game show (Baker 2011). In what IBM
$46,000 (Ken Jennings $24,000 and Brad Rutter has called cognitive computing (Kelly and Hamm
$21,600). The match was watched by close 2013), the ability of programs to learn from expe-
to 35 million people on TV and an estimated rience and understand problems that were so far
70% of all Americans knew of the program firmly in the domain of humans (Asakiewicz et al.
(Baughman et al. 2014), making the program a 2017) has the potential to disrupt virtually all
bona fide celebrity. aspects of our lives, with all the commercial impli-
Watson was the result of a 7-year project cations and opportunities that come with it.
(Frenkel 2011), which resulted in a program that The nonlinear increase in data generation, stor-
could interpret and parse statements made in often age, and processing in recent years has arguably
messy and colloquial English and search through ushered in a new age, one where intelligent data
up to 200 million pages of text to identify and analytics and text analysis (Cvetković et al. 2017)
generate the appropriate question (Strickland and are rapidly increasing in relevance. The technol-
Guy 2013) for the answers provided. For years to ogy behind Watson has been applied to the
come, it was considered one of the leading intel- domains of, e.g., legal services, health care, bank-
ligent systems in existence (Abrar and Arumugam ing, and tourism (Gantenbein 2014; Murtaza et al.
2013). IBM itself has claimed it to be the first 2016). Of those, applications in the domain of
software capable of cognitive computing (Holtel health care constitute the largest benefit to society
2014) and considered its creation and victory at due to their societal importance (Kelly and Hamm
the game show the beginning of an “Era of Cog- 2013) and reliance on (massive amounts of) data
nitive Computing” (Kelly and Hamm 2013). (Ahmed et al. 2017). The exceptional perfor-
The expert systems of the early years of artifi- mance exhibited during the game show, especially
cial intelligence were systems that were designed with regard to natural language processing and
to reason using learned (hard coded) steps. This under the added challenge of colloquial speech,
394 Computer Games and Artificial Intelligence

ambiguity, and reading between the lines, has what humans can consciously grasp and explain;
made it clear that the true calling for systems many top players play games partly intuitively,
like Watson may be found outside the studio and that is, with a clear understanding of which
in our daily lives (Kelly and Hamm 2013). moves they prefer but without the ability to justify
this preference.
In the final years of the last millennia, this
Philosophical Thoughts stronghold of human intelligence came under
attack. While futile promises made by the most
Intelligence is a concept humans are very famil- prominent experts in the field of artificial intelli-
iar with yet to date have failed to define well. gence (e.g., Herbert Simon’s “a computer [will]
Part of the reason why the Turing test is still be the World Chess Champion before 1967” (Hsu
relevant is because it sidesteps this dilemma by et al. 1995)) have demonstrated time and time
using the one definition human can agree on: again that the advent of AI is not, by far, as
human intelligence. If a human can be fooled sweeping and complete as they hoped, it seems
to believe an opponent is human, then it must inevitable and unstoppable.
exhibit human-like behavior and, in this case, In this context, it is important to remember
intelligence. that these are just games. These are controlled
All games discussed above are landmark environments with often very clearly stated
games in the area of artificial intelligence, but rules (Naddaf 2010). In addition and maybe far
they are neither painting a complete picture nor more importantly, these are interactions in
are they the only ones that have been a key game which the evaluation of an outcome is clearly
for machines to compete against humans. One defined and often one-dimensional (i.e., win
obvious example is the game of Soccer, where vs. loss). Real life is of a different complexity,
entire leagues of various machines compete for and humans do not share the same views on how
titles. In this entry, we entirely ignore the physical outcomes are evaluated. Artificial intelligence
aspect of games and therefore all games that may be able to outperform humans in any one
require physical behavior. The motivation for subject, but the true benchmark for intelligence
this is twofold: on one hand, the progress made may be a matter of defining the meaning of the
in this area is equally stunning and pervasive, with concept intelligence for us. Machines are bound
new results and achievements being showcased in to be faster and more precise than humans; there
videos regularly. Providing a fair overview over should be no surprise about that. Whether they
these achievements is firmly outside the scope of can be better at something than humans is really
this entry (the interested reader is invited to search a question that cannot be answered until we
for YouTube videos of, e.g., Boston Dynamics). know what better means, in the respective
On the other hand, this entry considers the term context.
intelligence only in the intellectual sense and not One thing is becoming painfully obvious: we
in the context of mastery of the physical domain. are running out of games to have computers beat
For millennia the mastery of certain games, us at. Maybe the age of game-playing AI is
often based on exceedingly simple rules, was coming to an end. Or maybe the games need to
seen as the pinnacle of human intelligence. change, and winning by points is no longer the
Chess was considered an object of intellectual victory we prize the most. Cooperative game
skill (Jørgensen 2009) and diversion (Spencer play, with teams of humans and AI players work-
1975). Professional Go players are known ing together, might be a new challenge in the
to describe promising board configurations as years to come. In the end, when one has mastered
esthetically pleasing. In 200 BC, poetry and Go a game to perfection and is guaranteed to never
went hand in hand in Japan (Smith 1956). The lose a match, the only winning move is not to
complexity of games can extend far beyond play.
Computer Games and Artificial Intelligence 395

Cross-References Cazenave, T.: Residual networks for computer go. IEEE

Trans. Comput. Intell. AI Games. PP(99), 1–1 (2017)
Clark, C., Storkey, A. J.: Teaching deep convolutional
▶ Computer Games and the Evolution of Digital neural networks to play go. CoRR abs/1412.3409
Rights (2014)
▶ Computer Games in Education Coles, L.S.: Computers and society (reply to Weizenbaum
▶ Computer Go (1972)). Science. 178(4061), 561–562 (1972)
Conway, J.: On numbers and games. Ak Peters Series,
▶ Genetic Algorithm (GA)-Based NPC Making A.K. Peters (2001)
C
▶ History of Virtual Reality Cvetković, L., Milašinović, B., Fertalj, K.: A tool for
▶ Machine Learning for Computer Games simplifying automatic categorization of scientific
paper using Watson api. In: 2017 40th International
▶ Monte-Carlo Tree Search Convention on Information and Communication Tech-
▶ Overview of Artificial Intelligence nology, Electronics and Microelectronics (MIPRO),
pp. 1501–1505. (2017)
Ferrucci, D.: Build watson: an overview of DeepQA for
the Jeopardy! challenge. In: 2010 19th International
References Conference on Parallel Architectures and Compilation
Techniques (PACT), pp. 1–1. (2010)
Abrar, S. S., Arumugam, R. K.: Simulation model of IBM- Frenkel, K.A.: Schooling the jeopardy! champ: far from
Watson intelligent system for early software develop- elementary. Science. 331(6020), 999–999 (2011)
ment. In: 2013 UKSim 15th International Conference Fu, M. C.: AlphaGo and Monte Carlo Tree Search: The
on Computer Modelling and Simulation, pp. 325–329. simulation optimization perspective. In: 2016 Winter
(2013) Simulation Conference (WSC), pp. 659–670. (2016)
Ahmed, M.N., Toor, A.S., O’Neil, K., Friedland, D.: Cog- Gantenbein, R. E.: Watson, come here! The role of intelli-
nitive computing and the future of health care: the gent systems in health care. In: 2014 World Automation
cognitive power of ibm watson has the potential to Congress (WAC), pp. 165–168. (2014)
transform global personalized medicine. IEEE Pulse. Genesereth, M., Love, N.: General game playing: overview
8(3), 4–9 (2017) of the AAAI competition. AI Mag. 26, 62–72 (2005)
Aiolli, F., Palazzi, C.E.: Enhancing artificial intelligence on Gokmen, T., Onen, O. M., Haensch, W.: Training deep
a real mobile game. Int. J. Comput. Games Technol. convolutional neural networks with resistive cross-
2009, 1:1–1:9 (2009) point devices, CoRR abs/1705.08014 (2017)
Asakiewicz, C., Stohr, E.A., Mahajan, S., Pandey, L.: Hassabis, D.: Artificial intelligence: chess match of the
Building a cognitive application using Watson century. Nature. 544, 413 (2017)
DeepQA. IT Prof. 19(4), 36–44 (2017) Hildmann, H.: Behavioural game AI – a theoretical
Baker, S.: Final Jeopardy: The Story of Watson, the Com- approach. In: Internet Technology and Secured Trans-
puter That Will Transform Our World. Houghton Miff- actions (ICITST), 2011 International Conference for,
lin Harcourt, Boston (2011) Abu Dhabi, pp. 550–555. (2011)
Baughman, A.K., Chuang, W., Dixon, K.R., Benz, Z., Hildmann, H.: The players of games: Evaluating and
Basilico, J.: DeepQA Jeopardy! gamification: a recording human and AI game playing behaviour. In:
machine-learning perspective. IEEE Trans. Comput. 2013 International Conference on Advances in Com-
Intell. AI Games. 6(1), 55–66 (2014) puting, Communications and Informatics (ICACCI),
Books, L.: History of Games: History of Board Games, pp. 724–729. (2013)
History of Card Decks, History of Role-Playing Hildmann, H., Crowe, M.: Designing game playing behav-
Games, Playing Card, Alquerque, Senet. General iour for AI players. In: 2011 IEEE. Jordan Conference
Books LLC, Memphis (2010) on Applied Electrical Engineering and Computing
Bouton, C.L.: Nim, a game with a complete mathematical Technologies (AEECT 2011), Amman, pp. 1–7. (2011)
theory. Ann. Math. 3, 35–39 (1902) Hildmann, H., Hildmann, J.: Serious games and edutain-
Bowling, M., Burch, N., Johanson, M., Tammelin, O.: ment applications (Ma et al. (2012)). In: Ma et al.
Heads-up limit hold’em poker is solved. Science. (2012), Chapter 6: A Formalism to Define, Assess
347(6218), 145–149 (2015) and Evaluate Player Behaviour in Mobile Device
Brown, E.: Watson: the jeopardy! Challenge and beyond. Based Serious Games (2012a)
In: 2013 I.E. 12th International Conference on Cogni- Hildmann, J., Hildmann, H.: Serious games and edutain-
tive Informatics and Cognitive Computing, pp. 2–2. ment applications (Ma et al. (2012)). In: Ma et al.
(2013) (2012), Chapter 8: Augmenting Initiative Game Worlds
Brown, N., Sandholm, T.: Libratus: the superhuman AI for with Mobile Digital Devices (2012b)
no-limit Poker. In: Proceedings of the Twenty-Sixth Holtel, S.: Short paper: more than the end to information
International Joint Conference on Artificial Intelli- overflow: How IBM Watson will turn upside down our
gence, IJCAI-17, pp. 5226–5228. (2017) view on information appliances. In: 2014 IEEE. World
396 Computer Games and Artificial Intelligence

Forum on Internet of Things (WF-IoT), pp. 187–188. Mechling, J.: Toys as culture (brian sutton-smith). J. Am.
(2014) Folk. 100(397), 350–352 (1987)
Hsu, F.-H., Campbell, M.S., Hoane Jr., A.J.: Deep Metz, C.: The sadness and beauty of watching google’s ai
blue system overview. In: Proceedings of the 9th play go. WIRED Magazine (2016)
International Conference on Supercomputing, ICS 95, Metz, C.: A mystery AI just crushed the best human players
pp. 240–244. ACM, New York (1995) at Poker. WIRED Magazine (2017)
Jiang, Y., Dou, Z., Cao, J., Gao, K., Chen, X.: An effective Moravčík, M., Schmid, M., Burch, N., Lisý, V., Morrill, D.,
training method for deep convolutional neural network. Bard, N., Davis, T., Waugh, K., Johanson, M.,
CoRR abs/1708.01666 (2017) Bowling, M.: Deepstack: expert-level artificial intelli-
Johnson, D.L., Kobler, A.L.: The man-computer relation- gence in heads-up no-limit poker. Science. 356(6337),
ship. Science. 138(3543), 873–879 (1962) 508–513 (2017)
Jørgensen, A.: Context and driving forces in the develop- Murtaza, S. S., Lak, P., Bener, A., Pischdotchian, A.: How
ment of the early computer game nimbi. An. Hist. to effectively train IBM Watson: Classroom experi-
Comput. IEEE. 31(3), 44–53 (2009) ence. In: 2016 49th Hawaii International Conference
Kasparov, G.: The day that I sensed a new kind of intelli- on System Sciences (HICSS), pp. 1663–1670. (2016)
gence. Time Magazine (1996) Naddaf, Y.: Game-independent AI agents for playing Atari
Kasparov, G., Greengard, M.: Deep Thinking: Where 2600 console games, PhD thesis (2010)
Machine Intelligence Ends and Human Creativity Nash, J.: Non-cooperative games (PhD Thesis). In: Kuhn,
Begins. PublicAffairs, New York (2017) H., Nasar, S. (eds.) The Essential John Nash. Princeton
Kelly, J., Hamm, S.: Smart Machines: IBM’s Watson and University Press, Princetown (2001)
the Era of Cognitive Computing. Columbia Business Newell, A., Shaw, C., Simon, H.: Chess Playing Programs
School Publishing, New York (2013) and the Problem of Complexity, pp. 29–42. Springer,
Kollia, I., Siolas, G.: Using the IBM Watson cognitive New York (1988)
system in educational contexts. In: 2016 IEEE Sympo- Osborne, M.J., Rubenstein, A.: A Course in Game Theory.
sium Series on Computational Intelligence (SSCI), The MIT Press, Cambridge, MA (1994)
pp. 1–8. (2016) Riley, T.: Artificial intelligence goes deep to beat humans at
Kroeker, K.L.: Weighing Watson’s impact. Commun. poker. Science. 16, 37–43 (2017)
ACM. 54, 13–15 (2011) Saarikoski, P., Suominen, J.: Computer hobbyists and the
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature. gaming industry in Finland. An. Hist. Comput. IEEE.
521, 436 (2015) 31(3), 20–33 (2009)
Lee, C.S., Wang, M.H., Teytaud, O., Wang, Y.L.: The game Samuel, A.L.: Some moral and technical consequences of
of go at IEEE WCCI 2010. IEEE Comput. Intell. Mag. automation – a refutation [cf. Wiener (1960)]. Science.
5(4), 6–7 (2010) 132(3429), 741–742 (1960)
Lee, C.S., Teytaud, O., Wang, M.H., Yen, S.J.: Compu- Schaeffer, J., van den Herik, H.J.: Games, computers and
tational intelligence meets game of go at IEEE artificial intelligence. Artif. Intell. 134, 1–8 (2002)
WCCI 2012. IEEE Comput. Intell. Mag. 7(4), Schwalbe, U., Walker, P.: Zermelo and the early history of
10–12 (2012) game theory. Games Econ. Behav. 34(1), 123–137
Lee, C.S., Wang, M.H., Yen, S.J., Wei, T.H., Wu, I.C., (2001)
Chou, P.C., Chou, C.H., Wang, M.W., Yan, T.H.: Shannon, C.E.: Programming a computer for playing
Human vs. computer go: review and prospect. IEEE chess. Philos. Mag. (Series 7). 41(314), 256–275
Comput. Intell. Mag. 11(3), 67–72 (2016a) (1950)
Lee, C.S., Wang, M.H., Yen, S.J., Wei, T.H., Wu, I.C., Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L.,
Chou, P.C., Chou, C.H., Wang, M.W., Yan, T.H.: van den Driessche, G., Schrittwieser, J., Antonoglou, I.,
Human vs. computer go: review and prospect – Panneershelvam, V., Lanctot, M., Dieleman, S.,
extended online version. IEEE Comput. Intell. Mag. Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I.,
11(3), 67–72 (2016b) Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T.,
Love, N., Hinrichs, T., Haley, D., Schkufza, E., Hassabis, D.: Mastering the game of go with deep
Genesereth, M.: General Game Playing: Game neural networks and tree search. Nature. 529, 484
Description Language Specification. Technical report, (2016)
Stanford Logic Group (2008) Silver, D., Hubert, T., Schrittwieser, J., Antonoglou, I., Lai,
Lowood, H.: Guest editor’s introduction: perspectives on M., Guez, A., Lanctot, M., Sifre, L., Kumaran, D.,
the history of computer games. An. Hist. Comput. Graepel, T., Lillicrap, T., Simonyan, K., Hassabis, D.:
IEEE. 31(3), 4 (2009) Mastering chess and shogi by self-play with a general
Ma, M., Oikonomou, A., Jain, L. (eds.): Serious Games reinforcement learning algorithm. ArXiv e-prints
and Edutainment Applications. Springer, Huddersfield (2017)
(2012) Simon, H.A.: Computers as chess partners. Science.
McCarthy, J.: Computers, Chess, and Cognition. Springer, 169(3946), 630–631 (1970)
chapter Chess as the Drosophila of AI, pp. 227–237. Smith, A.: The Game of Go: the National Game of Japan.
(1990) C. E. Tuttle Co., Tokyo (1956)
Computer Games and the Evolution of Digital Rights 397

Spencer, D.: Game Playing with Computers. Hayden Book Definition

Co., Rochelle Park (1975)
Strickland, E., Guy, E.: Watson goes to med school. IEEE
Spectr. 50(1), 42–45 (2013) Computer games are all games that are played
Sutton-Smith, B.: Toys as Culture. Gardner Press, using a computer to display the board, generate
New York (1986) events, and provide other players to play with or,
Taube, M.: Computers and game-playing. Science. generally, for any other game relevant aspect. The
132(3426), 555–557 (1960)
Tung, C.: Humans out-play an AI at Texas Hold ‘Em – for entertainment industry is a wide term to encom- C
now’. WIRED Magazine (2015) pass any business generating profit from their
Turing, A.M.: Computing Machinery and Intelligence. activities related to the entertainment sector.
Oxford University Press, Oxford (1950)
Tylor, E.: The History of Games. D. Appleton,
New York (1879)
von Neumann, J., Morgenstern, O.: Theory of Games and
Economic Behaviour, 3rd edn. Princeton University
Introduction
Press, Princeton (1974)
Wang, F.Y., Zhang, J.J., Zheng, X., Wang, X., Yuan, Y.,
Dai, X., Zhang, J., Yang, L.: Where does alphago go: The MITS Altair 8800, released in 1975, is often
from church-turing thesis to alphago thesis and considered to be the first true personal computer
beyond. IEEE/CAA J. Autom. Sin. 3(2), 113–120
(Wolf 2008). Its operating system was created by
(2016)
Weizenbaum, J.: On the impact of the computer on society. two guys from a young company (then still called
Science. 176(4035), 609–614 (1972) “Micro-Soft”) who offered it as a product to MITS
Wiener, N.: Some moral and technical consequences before they had actually written it. These young
of automation. Science. 131(3410), 1355–1358
men (Bill Gates and Paul Allen) soon went on to
(1960)
Yan, J.: Security design in online games. In: Computer also sell IBM an operating system (MS-DOS,
Security Applications Conference, 2003. Proceedings. written in 8086 assembly) for their x86-based
19th Annual, pp. 286–295. (2003) computers. While this does not mark the first
Zermelo, E.: Über eine Anwendung der Mengenlehre
time that hardware and software became two sep-
auf die Theorie des Schachspiels. In: Proceedings
of the Fifth International Congress Mathematics’, arate products, it is arguably the most famous one
pp. 501–504. Cambridge University Press, Cambridge as the renamed company, Microsoft, went on to
(1913) become a giant of the industry. While there are
many fascinating aspects to the time and these
companies, the fact that MS-DOS already used
hidden and undocumented features to disrupt
Computer Games and the competitors’ products (e.g., later MS software
Evolution of Digital Rights products would not work correctly unless the
computer was running on MS-DOS – as opposed
Hanno Hildmann1 and Benjamin Hebgen2 to their competitors’ version of DOS – as its
1
Departamento de Ingenieria de Sistemas y operating system) is the most relevant here.
Automatica, Universidad Carlos III de Madrid, Ever since then, the industry has kept changing
Leganes/Madrid, Spain its business models (Ojala and Tyrvainen 2011)
2
NEC Research Labs Europe, Heidelberg, and how to protect them. Computer games are a
Germany good product family to illustrate and discuss this.
They are by many considered an essential part of
the software and entertainment industry
Synonyms (Hildmann and Hildmann 2012), but one that is
historically understudied (Saarikoski and
Video games; Digital games; Intellectual prop- Suominen 2009). There is no comprehensive
erty; Intellectual property rights; Commercial analysis of the history of video and computer
rights games from a business point of view (Royer
398 Computer Games and the Evolution of Digital Rights

2011), yet. For a general account, we refer to, e.g., different ROM cartridges that would all run on
Purcaru et al. (2014) or Dillon (2016). the same console). As far as home consoles are
concerned, Ralph Baer’s television-based Brown
Box (1966–1967) and Magnavox Odyssey,
Computer Games Driving the Hardware released in 1972 (the year of Pong) (Montfort
Industry and Bogost 2009), are considered landmark hard-
ware when it comes to lineage and impact on
Wolf (2008) and others say that it was the com- design (Nyitray 2011). Three years later, in
puter games that (financially) drove the evolution 1975, Atari started building arcade games for the
from computers the size of warehouses to the PC; private use (Wolf 2008), i.e., home versions.
from the 1975 MITS Altair 8800 to whatever These consoles included Pong, but as they were
model and type is considered state of the art at basically stand-alone units, built on a game-by-
the time the reader is reading this. Whether com- game basis, production took a long time and was
puter games alone deserve the credit for this trans- very costly. In October 1977 Atari released the
formation, which is at the heart of the VCS (Video Computer System) 2600, which
communication and information revolution of could be fed with cartridges of individual games.
the last decades, is arguable; whether computer This product did well and ultimately became the
games have become (and been for 40 years now) a most successful of the early game consoles (Royer
best-selling product is not a question at all. In the 2011). However, financially speaking, it did not
entertainment industry, they have long since do well enough in its initial years. The 1978
replaced traditional board and card games in over- arcade game Space Invaders (by Tomohiro
all profit (Bohannon (2008) estimated gaming to Nishikado), which was licensed by Atari for the
be a $11 billion industry globally in 2008; in 2016 home market, partly rescued them from the
this is the economic impact of the gaming industry losses of 1977–1978 (Montfort and Bogost
to the US GDP alone (Anderton 2017)) with the 2009). Historical reviews such as Glenday
global industry exceeding $90 billion (https:// (2009) rank the game as the top arcade game
tinyurl.com/ya7tfobf (venturebeat.com)). ever, and its arcade version famously caused
The next paragraphs will sketch the century- coin shortages in some countries. The game’s
long development from Charles Babbage’s Ana- 1980 VCS release is considered to have multi-
lytical Engine early in the nineteenth century plied Atari’s console sales.
(Spencer 1975) to the video game culture of
today (Kushner 2011; Williams 2017).
Digital Rights (The Fall of Atari)

From the Arcade to the Living Room What became Atari’s final mistake was that they
(The Rise of Atari) did not protect their developmental investment
into the consoles. At the time a shift in the busi-
This pervasive euphoria for computer games was ness model happened, in the sense that the profit
not always the case. For example, the 1958 game did not come from the sales of the consoles them-
Tennis for Two did not receive wide public atten- selves anymore but from the subsequent sale of
tion or marketing (Nyitray 2011). Until 1972 com- individual games. Unfortunately (for Atari), the
mercial computer games came only in the form of only protection against some guys in a garage
arcade games, sold by companies like Midway, starting to write their own games was the fact
Bally, and Atari (Mazor and Salmon 2009). While that Atari did not offer a programming manual or
Magnavox found itself selling its analog home TV the console blueprints to just anyone.
game Pong in respectable numbers, they decided According to (Levy 2010), Atari “regarded the
to shift their business paradigm from selling the workings of its VCS machine as a secret guarded
consoles to selling the games (i.e., introducing somewhat more closely than the formula for
Computer Games and the Evolution of Digital Rights 399

Coca-Cola.” So when Activision was formed in protecting its hardware to protecting copyright.
1980 (Royer 2011) by some of Atari’s best coders Since the chip was required to be in the cartridge
gone rogue, it marked the eventual end of Atari’s in order for the game to work in the console,
unprecedented success (and the console industry preventing the chip from being copied meant to
as a whole), simply because the soon increasing control the games that could legally be sold for a
number of game-producing companies did not console. It is noteworthy that the design of the
have to pay any royalties for the use of the con- 10NES chip was integral to this, as the similar C
soles on which their games were being played. As approach taken by SEGA failed. Accolade, who
O’Donnell (2009) states, “[a]ny company capable had reverse engineered the SEGA system to be
of determining how the 2600 worked and willing able to circumvent it and publish games for the
to pay for the cost of producing cartridges could Genesis III, ended up winning a court case against
then market their games, which set a low bar for SEGA (O’Donnell 2009).
quality.” This raid on the profit of the console- There are a few differences between the pro-
producing companies (bearing the full financial tection mechanism for the NES (by Nintendo) and
burden and risk of providing the platform for the the approach taken by SEGA to protect their Gen-
games) combined with the sudden flood of medi- esis III:
ocre games (which both disappointed the con-
sumer as well as caused the dumping of prices) 1. Instead of a special chip such as the 10NES,
let to the collapse of the industry in 1983 (Royer SEGA relied on the string SEGA being found
2011). The age of Atari lasted a mere 6 years, but at a specific position in the memory of the
it turned a generation of kids into computer game game cartridge. Writing this string to correct
players, computer (game) programmers (Aspray location in memory required the knowledge of
2004) and, eventually, computer scientists. The what to write where but did not require a spe-
credit for doing this does – of course – not go to cific (and protected) chip.
Atari alone; the company is mentioned as one 2. In addition, when the string was present on a
(admittedly very dominant and rather famous) game cartridge, SEGA had its game console
example of what today is a global multibillion display the text “Produced by or under license
dollar industry. from SEGA Enterprise LTD” before starting the
game. This, however, meant that anyone using
the game including the string without permis-
Copyright Protection and IPR (From sion (i.e., payment of royalties to SEGA) effec-
Nintendo to Today) tively infringed on SEGA’s trademark.
3. Finally, as the string in memory was originally
It took another 2 years and another company to written by SEGA, copying it could be consid-
bring about a change in the market. Nintendo’s ered a breach of the copyright of SEGA’s code.
introduction of the Nintendo Entertainment Sys-
tem (NES) (A number of NES games are emulated While the initial ruling was in favor of SEGA,
and playable online at http://www.virtualnes.com) Accolade successfully appealed and subsequently
in the winter of 1985 became a turning point in the won the case, with one of the arguments being that
history of the industry (O’Donnell 2009) and even the string in memory was tiny in comparison to
has implications for the Digital Millennium Copy- the amount of code written for the actual game and
right Act (DMCA) of 1998. The system required that the copyright breach was therefore negligible.
an authorization chip, the 10NES, to be present in In addition, the clever use of SEGA’s trademark as
the cartridge. Atari tried unsuccessfully to reverse a copy protection mechanism was ruled to consti-
engineer the chip and even went as far as to tute a violation of certain legal restrictions placed
physically disassemble the chip (to no avail). on the use of trademarks. Since then, the industry
The introduction of this chip is considered the (computer games and the computer industry in
moment when the industry changed from general) has continuously changed to ensure the
400 Computer Games and the Evolution of Digital Rights

protection of the companies’ investment into Cross-References

intellectual property (mainly the software).
▶ Computer Games and Artificial Intelligence
▶ Computer Games in Education
Summary and Conclusion ▶ Gamification and Serious Games
▶ Gamification
According to Ojala and Tyrvainen (2011) over the ▶ History of Virtual Reality
last decade, the software industry has taken the ▶ Overview of Artificial Intelligence
shape of a $200 billion software division within a ▶ Serious Online Games for Engaged Learning
$500 billion service industry. Even the latest offline Through Flow
digital rights management (DRM) tools, such as ▶ Social-, Mobile-, and Multi-Player-Games and
Denuvo (https://www.denuvo.com/), fail to ade- Their Impact on Today’s Online Entertainment
quately protect against piracy. One famous exam- Industry
ple of this is the case of Resident Evil 7 in 2017
which was cracked within 5 days (https://tinyurl.
com/yaaluusm (www.dsogaming.com)). Addition- References
ally offline DRM systems can have impacts on the
performance of a game (http://www.sega.com/ Anderton, K.: The business of video games: a multi billion
dollar industry. Forbes. (2017) https://www.forbes.
denuvo) and even block paying customers. This
com/sites/kevinanderton/2017/04/25/the-business-of-
leads to two approaches by the industry: either a video-games-how-much-is-being-spent-and-why-
complete move away from DRM as done by CD infographic/#74108c0f6fe8
Projekt Red’s gaming platform Good-Old-Games Aspray, W.: Computer: A History of the Information
(https://www.gog.com) or the move to a cloud- Machine. Basic-Books, New York (2004)
Bohannon, J.: Flunking spore. Science. 322(5901),
based DRM as done by Valve’s platform Steam 531 (2008)
or others. Tommy Refenes (Super Meat Boy devel- Dillon, R.: The Golden Age of Video Games: The Birth of
oper) states “the fight against piracy equates to a Multibillion Dollar Industry. CRC Press, Boca Raton
spending time and money combating a loss that (2016)
Glenday, C.: Guinness World Records 2009, GUINNESS
cannot be quantified” and “[d]evelopers should
WORLD RECORDS. Random House Publishing
focus on their paying customers and stop wasting Group (2009) New York City, New York
time and money on non-paying customers.” While Hildmann, J., Hildmann, H.: Chapter 8: augmenting initia-
cloud-based online DRM systems made it harder to tive game worlds with mobile digital devices. In: Ma,
make illegal copies of software (and, as Ojala and M., Oikonomou, A., Jain, L. (eds.) Serious Games and
Edutainment Applications. Springer, London (2012)
Tyrvainen (2011) point out, specifically games), Kushner, D.: Betting the farm on games. IEEE Spectr.
they can also lead to a loss of reputation and cus- 48(6), 70–88 (2011)
tomers. While Steam managed to introduce a semi- Levy, S.: Hackers O’Reilly Series. O’Reilly Media, Sebas-
online DRM (allowing for up to 4 weeks in offline topol (2010)
mode) and still be considered a positive influence Mazor, S., Salmon, P.: Anecdotes: magnavox and intel: an
odyssey and the early days of the arpanet. IEEE Ann.
for PC gaming by providing additional services Hist. Comput. 31(3), 64–67 (2009)
(https://tinyurl.com/ycp4c9be (www.cinemablend. Montfort, N., Bogost, I.: Racing the Beam: The Atari
com)), others have failed here. For instance, Video Computer System, Platform Studies. MIT
Ubisoft’s original always-on DRM leads to some Press, Cambridge, MA (2009)
Nyitray, K.: William Alfred Higinbotham: scientist, activ-
backlash from the customers especially when the ist, and com-puter game pioneer. IEEE Ann. Hist.
servers for the DRM are not reachable (https:// Comput. 33(2), 96–101 (2011)
tinyurl.com/y97alctv (kotaku.com)). In this case it O’Donnell, C.: Production protection to copy(right) pro-
leads to a switch in the company policies removing tection: from the 10nes to DVDs. IEEE Ann. Hist.
Comput. 31(3), 54–63 (2009)
the always-on demand for single-player games Ojala, A., Tyrvainen, P.: Developing cloud business
(https://tinyurl.com/yb8w795f (www. models: a case study on cloud gaming. IEEE Softw.
rockpapershotgun.com)). 28(4), 42–47 (2011)
Computer Games for People with Disability 401

Purcaru, B., Andrei, A., Gabriel, R.: Games vs. Hardware. include work at the intersection of games and
The History of PC Video Games: The 80’s. (2014) disability which primarily falls in any of the fol-
Royer, G.: Familiar concepts, unfamiliar territory. IEEE
Ann. Hist. Comput. 33(2), 112 (2011) lowing categories: (i) identifying design consid-
Saarikoski, P., Suominen, J.: Computer hobbyists and the erations or requirements using interviews,
gaming industry in Finland. IEEE Ann. Hist. Comput. questionnaires, or data collected from gaming
31(3), 20–33 (2009) sessions without emphasis on specific games;
Spencer, D.: Game Playing with Computers. Hayden Book
Co, Rochelle Park (1975) (ii) platforms, architectures, or frameworks for C
Williams, A.: History of Digital Games: Developments in creating games; (iii) interfaces or controllers for
Art, Design and Interaction. CRC Press, Boca Raton games; (iv) using hardware and software such that
(2017) gamification is very limited, like use of stars to
Wolf, M.: The Video Game Explosion: A History from
PONG to Playstation and beyond. Greenwood Press, motivate the player; (v) game proposals, stan-
Westport (2008) dards, or designs that are not implemented or
evaluated; (vi) games which were either not eval-
uated or whose evaluation did not involve healthy
subjects, subjects with a disability, or specialists;
(vii) factors affecting acceptance of games; (viii)
Computer Games for People games for training members of educational or med-
with Disability ical staff, such that these games are not designed to
be played by people who need to be evaluated to
Amol D. Mali see if they have a disability or disabled people
Computer Science Department, University of who need to be treated; (ix) educational apps
Wisconsin-Milwaukee, Milwaukee, WI, USA with very limited gamification; (x) challenges in
design, development, or evaluation of games;
(xi) automating or improving fulfillment of tasks
Synonyms in design, development, evaluation, customization,
localization, or enhancement of games; (xii)
Auditory impairment; Autism; Cerebral palsy; reporting on digital-game-related resources or
Down syndrome; Dynapenia; Dysgraphia; their strengths, weaknesses, or limitations; (xiii)
Dyslexia; Dyscalculia; Intellectual disability; use of disability-inspired games to serve areas
Learning disability; Motor disability; Rehabilita- other than disability; (xiv) comparison of digital
tion; Sensory disability; Visual impairment; games with other forms of education or entertain-
Visual memory; Zika virus ment; and (xv) digital games for helping people
with a disability get familiar with assistive
technology.
Introduction

A computer game for people with disability may Games for People with Sensory
be for people with a sensory disability, motor Disability
disability, intellectual disability, learning disabil-
ity, weak communication skills, weak social Most computer games for visually impaired
skills, or a combination of these. This chapter is players give them feedback using audio, haptics,
a concise survey which includes representative or enhanced imagery. Most computer games for
games for each category of disability. Games hearing-impaired players give them feedback
addressing disorders that do not fall in any cate- using a sign language. Moustakas and others
gory of disability are not in the scope of this (2011) report on a framework for real-time com-
survey. The goal of the survey is to make the munication between visually impaired and
readers aware of the types of disability for which hearing-impaired game players in a shared envi-
computer games have been developed. It does not ronment, along with experimental evaluation.
402 Computer Games for People with Disability

Their framework uses gesture recognition, sign- Supermarket (Lanyi et al. 2011) is a game which
language analysis, sign-language synthesis, teaches shopping and money management during
speech analysis and synthesis, and haptic interac- shopping using a shopping list, shopping cart,
tion. Visual information about the gaming envi- shelves with goods with their names, prices, and
ronment is conveyed to the visually impaired images, and a virtual wallet containing banknotes
player via haptic interaction. Semantic informa- and coins. Memobile (Lanyi et al. 2011) is a game
tion about the gaming environment is conveyed to involving important tasks that are typically
the visually impaired player using sound. Verbal performed before leaving the house and through-
input from the visually impaired player is per- out the day, e.g., packing lunch, taking money,
ceived using speech recognition. Sounds of musi- and taking medications. Stress at Work (Lanyi
cal instruments are used to enable the perception et al. 2011) is a game which educates on stress at
of colors using the SeeColor utility. Verbal infor- work. These games were evaluated by people
mation is presented to the hearing-impaired player from three countries who specialized in psychol-
using sign language. The hearing-impaired player ogy, children with special needs, and IT
can provide input to the system through the sign- administration.
language recognizer. The framework communi- Martinez and others (2019) report on
cates the visually impaired player’s input to the SATReLO. It is a tool which allows construction
hearing-impaired player by recognizing the for- of personalized mini-games for helping children
mer player’s speech and synthesizing signs for the with loss of hearing speak a language and write in
speech for the latter player. The framework con- a language. A therapist can select the language
veys the hearing-impaired player’s input to the attributes (articles, pronouns, nouns, objectives,
visually impaired player by recognizing the for- and verbs) to be included in the mini-game. The
mer player’s signs and synthesizing speech for the therapist can also select the theme of the mini-
signs for the latter player. When the visually game. The theme can be animals, school supplies,
impaired player touches a virtual object, he/she or professions. Domino and Sequence Cartoon are
hears the sound synthesized for the object’s color. the mini-games that SATReLO allows a therapist
Training is needed to help visually impaired to customize. Some of the children used in the
players link colors with sounds. The visually evaluation of SATReLO had a cochlear implant or
impaired player and the hearing-impaired player a hearing aid. The rest had normal hearing.
play the game cooperatively, with the odd- A baby infected with Zika virus before birth
numbered steps played by the visually impaired may have a smaller head, joints with limited range
player and the even-numbered steps played by the of motion, seizures, and problems in vision and
hearing-impaired player. Translation between hearing (https://www.cdc.gov/zika/healtheffects/
speech and signs occurs via text. Speech is trans- birth_defects.html). Filho and others (2020)
lated into text and the text is used to generate report on a platform for evaluating executive func-
signs. Signs are translated into text, and speech tions of toddlers and training them. This study
consistent with the text is generated. included 18 toddlers born with Down syndrome
Lanyi and others (2011) have reported on sev- and 16 toddlers born with Zika virus. Let’s Smile
eral serious games for people with learning dis- (Filho et al. 2020) is a game which shows sad
abilities and sensory impairments. A very small white faces which become smiling colorful faces
percentage of such people have jobs. Cheese Fac- when touched. Let’s Blow the Balloon (Filho et al.
tory (Lanyi et al. 2011) is a game which teaches 2020) is a game which shows colored withered
the concepts of fractions and percentages. My balloons tied to a string. The player is expected to
Appearance (Lanyi et al. 2011) is a game involv- keep the button below a balloon pressed, to fill the
ing morning tasks that are typically completed balloon with air. The balloon expands until it
after getting up and before leaving home. 3D touches the top of the screen. When the balloon
Work Tour (Lanyi et al. 2011) is a game which cannot be inflated anymore, the player is expected
simulates the first days at a workplace. VR to move the finger away from the button that was
Computer Games for People with Disability 403

pressed for inflation, and touch the balloon to pop stand on a launch pad and one player has to press
it. The assessment criteria for Let’s Smile included action button on game controller to form their
proportion of hits without repetitions, and propor- group. Voice links to other players are established
tion of precision without error. The assessment immediately. Players can join a mini-game in
criteria for Let’s Blow the Balloon included hold- progress by standing on its launch pad and press-
ing time and reaction time. ing the action button. All avatars move at the same
speed despite the differences in pedaling speeds. C
This avoids skill-based segregation of players.
Games for People with Motor Disability Some of the mini-games have goals only for
group of players, and all players get the same
Burke and others have reported on virtual-reality reward when their group achieves its goal. Ten
games and webcam games for post-stroke reha- youths with cerebral palsy, including seven boys
bilitation of upper limb (Burke et al. 2009). Catch and three girls, played the mini-games from their
(Burke et al. 2009) for bilateral rehabilitation homes over 10 weeks. On average, each player
requires the player to move a physical basket spent 2.75 h/week playing Liberi. All players
with magnetic sensors attached to it, to catch spent the majority of the time playing with others
objects falling in the virtual environment which when at least one other player was available. This
has a virtual basket. Whack a mouse (Burke et al. showed that Liberi was successful in promoting
2009) requires the player to move a hand with a group activities among the players.
sensor attached, to hit the mouse in the virtual E-Wobble (Karime et al. 2011) is an interactive
world with a virtual hammer, in the first level of rehabilitation system for stroke survivors who are
the game. Mouse and dog appear in the second unable to raise the foot at the ankle. The system
level of this game, but the player should not hit the includes a plastic wobble board, an accelerometer,
dog. This is for improving the player’s visual vibration motors, a microcontroller, a wireless
discrimination and selective attention. Rabbit communication module, a sensorized sandal on
Chase – a webcam game (Burke et al. 2009)– is top of the board, and a two-dimensional digital
for rehabilitation of one arm. There is a rabbit golf game with which the wobble board interacts.
which moves between four holes and the player The sandal has cushioning in the area for the toes,
must touch the hole with the rabbit when the with a pressure sensor placed below the cushion to
rabbit stares from it. Arrow Attack (Burke et al. detect the force exerted on it. The player has to
2009) is a webcam game for rehabilitation of both move his/her ankle to move the golf ball. The
arms. The player should touch the right arrow player has to exert pressure on the cushion by
when the right arrow reaches the right box, using flexing the toes, in order to drop the ball into
the right hand. The player should touch the left the hole.
arrow when the left arrow reaches the left box, Zavala-Ibarra and Favela report on design and
using the left hand. Burke and others (Burke et al. development of video games played with a
2009) also report on a game which allows the custom-designed interaction device, to assist in
player to play a virtual vibraphone – an instru- detecting the early signs of dynapenia which is
ment like xylophone, through remote controllers. age-related loss of muscle strength and power
This game is for rehabilitation of wrist and arm. (Zavala-Ibarra and Favela 2012). The interaction
Hernandez and others (2014) present Liberi – a device contains Vernier Hand Dynamometer to
networked, cycling-based game – to help youth measure the user’s grip strength. One game
with motor disabilities socialize while being on requires the player to throw a ball. The player is
special recumbent stationary bicycles. Players given feedback about the amount of strength used
pedal to move their avatars. A player aims using by them. Another game allows the player to
a joystick and invokes game actions with one increase the altitude and the speed of the bird by
button. Liberi consists of six mini-games. Players pressing the controller. The third game asks the
are grouped fast since their avatars just have to player to perform a maximum voluntary
404 Computer Games for People with Disability

contraction using the dynamometer and then player to choose foods to create a well-balanced
maintain their force for as long as possible. The breakfast.
time elapsed between the start of application of Wyeth and others (2014) report on Stomp-a
the maximum force and the drop to half of the floor-based system requiring simple gross-motor
maximum force is the muscle fatigue resistance. actions. Stomp is designed for people with intel-
The player is able to make the fire-extinguishing lectual disability. Players interact with the digital
vehicle hold the water longer by retaining the worlds by triggering the pressure sensors embed-
force applied. ded within a floor mat. There are interactive expe-
Dysgraphia is a nervous-system problem riences that are projected onto the mat. Stomp can
affecting fine-motor skills, causing handwriting be used by one or more people, by stepping,
to be consistently distorted or unclear (https:// stomping, pressing, jumping, or sliding. Players
www.webmd.com/add-adhd/childhood-adhd/ stomp on icons to make their selection. The expe-
dysgraphia-facts). Kariyawasam and others riences provided by Stomp included three musical
(2019) report on a game-based screening-and- experiences, three sports experiences, a painting
intervention tool for dysgraphia, dyslexia, and experience, a paddling experience, a road-safety
dyscalculia. Children in their study wrote Sinha- game, a sheep-herding game, and four arcade-like
lese letters on a mobile screen with a pen. Some of experiences. Players can interact with Stomp
these children had letter dysgraphia. Children’s while sitting, standing, walking, or lying down.
writing was used as data for machine learning. They get visual and auditory feedback. Stomp
A neural network and support vector machine promotes physical and social activity. People
(SVM) were used. Children predicted to have with cerebral palsy, communication disorders,
letter dysgraphia can be initially trained to write mental retardation, autism, or Down syndrome,
letters correctly by showing them animations to used Stomp.
help in following the given paths. Children are Hassan and others report on a game to teach
allowed to write independently in later stages of usage of money to 9- to 14-year-old autistic chil-
intervention. Children were asked to write the dren (Hassan et al. 2011). The game teaches chil-
10 integers ranging from 0 to 9, for getting data dren to recognize banknotes and choose correct
for using machine learning for diagnosing banknotes for purchases. The game also includes
numeric dysgraphia. The authors used a Random communication with the shopkeeper to teach
Forest classifier to predict if a child had numeric social skills to children.
dysgraphia or not. Children predicted to have
numeric dysgraphia can use animations to help
in tracing the given numbers. Games for People with Learning
Disability

Games for People with Intellectual There are differences between intellectual disabil-
Disability, Weak Communication, or ity and learning disability. A person with an intel-
Weak Social Skills lectual disability faces challenges in more than
one area. These areas include communication,
Isasi and others report on a game about healthy completion of activities of daily living, learning,
eating which is playable on an iPad (Isasi et al. speaking, memorizing, foreseeing, and physical
2013). The game is intended for 8- to 12-year-old movement. A person with a learning disability
children with Down syndrome. The game consists may face challenges in one or more areas of learn-
of two mini-games. The first mini-game requires ing. Dyslexia is a learning disorder that affects the
the player to iteratively select ingredients to make ability to read, spell, write, and speak. (https://
a salad. The player is shown two ingredients on www.webmd.com/children/understanding-
each iteration, such that only one of them is suit- dyslexia-basics). Dyscalculia is a brain-related
able for salad. The second mini-game requires the condition which makes basic arithmetic hard to
Computer Games for People with Disability 405

learn (https://www.webmd.com/add-adhd/ aquariums with one fish in each. All fish differ in
childhood-adhd/dyscalculia-facts). Kariyawasam size. The child is expected to remember the size
and others (2019) used a Convolutional Neural and location of each fish. Then fish are removed
Network, KNN (K-Nearest Neighbors algorithm), and aquariums without fish are displayed. The
and audio clips of children’s pronunciations of child is expected to select the aquariums sequen-
words and letters to predict if children were tially such that each aquarium selected earlier had
dyslexic. They report on a multisensory gaming a fish that was smaller than the fish in every C
environment to offer intervention for children aquarium selected later. The study reported in
predicted to be dyslexic. The hard stage requires (Pir et al. 2019) involved 24 8-year-old girls and
the child to pronounce the displayed word. If the 24 8-year-old boys. Of these, 12 girls and 12 boys
child pronounces the displayed word correctly, the were included in the experimental group and the
balloon bursts, and the child is congratulated by remaining 24 children were put in the control
the gaming environment. These authors used the group. The experimental group received interven-
gaming environment to let children show their tion using Neuroland. The control group did not
ability to count numbers, compare numbers, and receive any intervention. The results showed that
add numbers. The authors used counting these mini-games were significantly effective in
accuracy, time spent per counting question, enhancing visual memory and writing skills.
number-comparison accuracy, time spent per
number-comparison question, addition accuracy,
time spent per addition question, and an SVM Games to Promote Awareness of
(Support Vector Machine) classifier, to predict if Disability
a child was affected by dyscalculia or not. The
gaming environment is used to offer intervention Gerling and others (2014) report on four mini-
to children predicted to have dyscalculia. games that the player needs to successfully play
Visual memory is the ability to immediately to arrive at the party. These games are played by a
recall what the symbols, shapes, objects, or person in a real wheelchair such that movements
forms just seen by eye looked like. Visual memory of the wheelchair control movements of the digital
is important for processing short-term memory character. These mini-games are designed to make
into long-term memory (https://www.optome people not using wheelchairs aware of the chal-
trists.org/vision-therapy/guide-vision-and- lenges faced by users of wheelchairs and promote
learning-difficulties/guide-to-visual-information- empathy toward them. One mini-game requires
processing/visual-memory/). A person can have the avatar to cross the street while being in a
poor visual memory despite normal vision and wheelchair when the avatar arrives at an intersec-
hearing. Pir and others (2019) report on tion. Another mini-game requires the avatar to
Neuroland which is an application containing find the items on the shown list while avoiding
10 digital mini-games for improving visual mem- puddles inside a grocery store, and move the arm
ory. Each of these games except two shows letters, to reach the items. Another mini-game requires
words, shapes, or ships, and expects the player to the avatar to pick items from a bookstore with
select an option containing all of what was just multiple floors, using elevators instead of escala-
shown, containing none of what was just shown, tors. The arm needs to be used to grab items.
containing a color not present in what was just Cake, cream, and candles are examples of items
shown, containing a suitable subset of what was to be picked from the grocery store. Birthday
just shown, has color of what was just shown, or is cards and wrapping paper are examples of items
of shape identical to the shape that was just to be picked from the bookstore. Park – one of the
shown. Words Chain displays some words for a mini-games – requires the avatar to come to a park
few seconds, removes them from the screen, and and get three flowers. The avatar must avoid
expects the child to say them in the order in which stairs. After completing the four mini-games, the
they were displayed. Fish Aquarium displays player navigates the streets to arrive at the friend’s
406 Computer Games in Education

house for the birthday party. Twenty-one females Proceedings of the International Conference on Virtual
and 19 males participated in this study. Reality and Visualization (ICVRV),
pp. 172–175 (2019)
Moustakas, K., Tzovaras, D., Dybkjaer, L., Bernsen, N.,
Aran, O.: Using modality replacement to facilitate com-
munication between visually and hearing-impaired
References people. IEEE MultiMedia, 26–37 (2011)
Pir, A., Afshar, L., Maveddat, S.: The effectiveness of a set
Burke, J., McNeill, M., Charles, D., Morrow, P., Crosbie, of Neuroland digital minigames on enhancing visual
J., McDonough, S.: Serious games for upper limb reha- memory and reducing spelling errors in students with
bilitation following stroke. Proceedings of Conference writing problems: A pilot study. Proceedings of the 1st
on Games and Virtual Worlds for Serious Applications, International Serious Games Symposium (ISGS),
pp. 103–110 (2009) pp. 61–67 (2019)
Filho, M., Boato, E., Quesada, A., Moresi, E., Tristao, R.: Wyeth, P., Summerville, J., Adkins, B.: Playful interactions
Evaluation of executive functions of children with for people with intellectual disabilities. ACM Com-
down syndrome and Zika virus using touch-screen puters in Entertainment. 11(3), 2:1–2:18 (2014)
device. Proceedings of the 11th IEEE International Zavala-Ibarra, I., Favela, J.: Assessing muscle disease
Conference on Cognitive Infocommunications related to aging using ambient videogames. Proceed-
(CogInfoCom), pp. 379–386 (2020) ings of the 6th International Conference on Pervasive
Gerling, K., Mandryk, R., Birk, M., Miller, M., Orji, R.: Computing Technologies for Healthcare
The effects of embodied persuasive games on player (PervasiveHealth) , pp. 187–190 (2012)
attitudes toward people using wheelchairs. Proceedings
of ACM Conference on Human Factors in Computing
Systems (CHI), pp. 3413–3422 (2014)
Hassan, A., Zahed, B., Zohora, F., Moosa, J., Salam, T.,
Rahman, M., Ferdous, H., Ahmed, S.: Developing the Computer Games in Education
concept of money by interactive computer games for
autistic children. Proceedings of IEEE International
Symposium on Multimedia, pp. 559–564 (2011) Jule Hildmann1 and Hanno Hildmann2
1
Hernandez, H., Ketcheson, M., Schneider, A., Ye, Z., The University of Edinburgh, Edinburgh, UK
Fehlings, D., Switzer, L., Wright, V., Bursick, S., 2
Departamento de Ingenieria de Sistemas y
Richards, C., Graham, T.: Design and evaluation of a Automatica, Av. Universidad, Universidad Carlos
networked game to support social connection of youth
with cerebral palsy. Proceedings of the 16th Interna- III de Madrid, Madrid, Spain
tional ACM SIGACCESS Conference on Computers
and Accessibility (ASSETS), pp. 161–168 (2014)
Isasi, A., Basterretxea, A., Zorrilla, A., Zapirain, B.: Help- Synonyms
ing children with intellectual disability to understand
healthy eating habits with an iPad-based serious game.
Proceedings of the 18th International Conference on Game-based learning (GBL)
Computer Games (CGAMES), pp. 169–173 (2013)
Karime, A., Al-Osman, H., Gueaieb, W., Alja’am, J., El
Saddik, A.: E-Wobble: An electronic wobble board for
ankle and toe rehabilitation. Proceedings of IEEE Inter- Introduction
national Symposium on Medical Measurements and
Applications, pp. 366–369 (2011) Computers have been used to create and play
Kariyawasam, R., Nadeeshani, M., Hamid, T., Subasinghe, games for basically as long as they are around;
I., Ratnayake, P.: A gamified approach for screening
and intervention of dyslexia, dysgraphia, and according to Nyitray (2011), the first publicly
dyscalculia. Proceedings of the International Confer- accessible computer games appeared as early as
ence on Advancements in Computing (ICAC), 1958, though the term game might be rather
pp. 156–161 (2019) grande for some of the early games. The game
Lanyi, C., Brown, D., Standen, P., Lewis, J., Butkute, V.,
Drozdik, D.: GOET European project of serious games Pong, considered by many as the first, famously
for students with intellectual disability. Proceedings of consisted of the moving of a bar on one side of the
the 2nd International Conference on Cognitive screen to deflect a moving dot (i.e., a ball) back to
Infocommunications (CogInfoCom), pp. 1–6 (2011) the other side of the screen, where another player
Martinez, J.-C., Gutierrez, E., Alvarez, G., Castillo, A.,
Portilla, A., Almanza, V.: Video games to support lan- attempted to do the same. There is evidence that
guage therapies in children with hearing disabilities. traditional board games such as Checkers were
Computer Games in Education 407

developed (Jackson (2000) and Tic-Tac-Toe was 1957). In this context, we will use the definition
implemented (Egenfeldt-Nielsen et al. 2008) as offered by Connolly and Stansfield (2007), who
early as 1951. One of the motivation for using have suggested that computer (games)-based
the then new computing paradigm for such a learning should be seen as “the use of a computer
mundane thing as playing a game was to “convey (games) based approach to deliver, support, and
the message that our scientific endeavors have enhance teaching, learning, assessment, and
relevance for society” (Nyitray 2011). evaluation”. When used appropriately, this C
While many maintain that games do not seem means combining.
to serve any cultural or social function other than Validated approaches from pedagogics, known
to distract (Bogost 2007), we argue in this entry models from psychology and learning theory as
that there is evidence to the contrary. There is little well as adhering to best practices for the design
doubt that games can be beneficial, if used cor- and implementation of computer games and intel-
rectly and by trained exerts. In fact, there is a ligent user interfaces.
vibrant and fast growing community of If this is done correctly computer games have
researchers and practitioners in the field of the potential to constitute a platform through
games-based learning (de Freitas 2011). That which the experts can deliver their teaching in a
being said, it is important to consider the evidence way that makes use of all the befits while mitigat-
from established fields such as psychology and ing a number of normal human traits (such as e.g.,
pedagogics to ensure that the dangers and short- waning attention and lack of motivation).
comings of the games-based learning industry – But the use of games for non-recreational pur-
which today is a billion dollar industry – are not to poses (i.e., as serious games) is certainly not new
be overlooked. As it is to be expected for a rela- (consider that every maneuver is, in a way, a game
tively young field which promises many business and the literature on military maneuvers is more
opportunities, there are many projects that seem to than 2000 years old, e.g., Tzu and Cleary 1988),
take short cuts. Of course the use of computer nor is it restricted to education (Puschel et al.
games alone is not the end to all problems in 2010): business games have been proposed for
pedagogics; professionals and researchers in the research as early as the 1960s (Babb et al. 1966)
field have emphasized (e.g., Hildmann 2010) that and 1970s (Rowland and Gardner 1971). Games
computer technology, while showing a lot of have been used to great success to train complex
potential and offering many benefits, is merely a problem management (Pasin and Giroux 2011)
tool for a teacher. problem solving abilities (Christoph 2006) as
This entry provides the relevant background on well as practical and reasoning skills (Pee 2011).
long established fields such as pedagogics and When used appropriately they can significantly
developmental psychology. The aim is to inform reduce training time and demands on the instruc-
the those interested in designing, implementing or tor (Sandberg et al. 2001; Hildmann and
using computer games for educational purposes Hildmann 2012a, b). In fact, (computer) games
and to highlight the potential benefits of doing have been analyzed from a variety of perspectives,
so. However, the main message is that such both negative (e.g., aggression, violence or gender
games are a supporting technology, and not the stereotyping) and positive (e.g., skills develop-
dominant factor or even the core aspect to teach- ment, engagement or motivation) (Connolly
ing endeavors. et al. 2008).

Computer-Games Based Learning Games-Based Learning (GBL)

Rowland and Gardner (1971) date the use of com- There are many positive aspects of the use of
puters (and in a way, computer games) for educa- games for teaching in the literature. Arguably,
tional purposes back to 1956 (Ricciardi et al. games have never been just a children’s medium
408 Computer Games in Education

(Bogost 2007) and are generally something in for research projects, e.g., Cohen and Rhenman
which people of all ages engage in (Warren et al. (1961) or Rowland and Gardner (1971) and Babb
2011). Games have been shown to inherently drive et al. (1966) report on investigations on using
high motivation levels in those that play them management games or business games
(Malone and Lepper 1987). Repetition is a core (respectively) for research. In recent years a num-
element to many games, which can be used to ber of conferences and academic circles have
embody otherwise boring rehearsal tasks and e.g., focused on this subject and the field is rapidly
large firms and companies have used game-like growing. Today, the teaching and learning para-
settings for decades to implement training session digm is no longer restricted to human-human
for employee training activities. The literature lists interactions: advanced research is now consider-
many areas where games have successfully been ing knowledge transfer between humans and
used as training and simulation tools: military train- machines (from the former to the latter) and e.g.,
ing (Schneider et al. 2005), teaching exact sciences, Arsenio (2004) reports on using simple games to
specifically mathematics (Habgood 2007; Squire assist humans when teaching robots.
et al. 2004; Young and Upitis 1999), training in
software engineering and computer science (Ford
and Minsker 2003; Zhu et al. 2007) as well as Key Concepts of (Computer)
medicine (Beale et al. 2007; Lennon 2006; Games-Based Learning
Roubidoux 2005). Other fields where GBL has
been applied are language education and project- The behavioral activity of engaging in play is
and knowledge management (Johnson and Wu considered by e.g., Brown (1998) to be a funda-
2008; Rankin et al. 2006; Long 2010) and mental basis for development in complex animals,
Christoph (2006); Chua (2005), respectively). on par with the act of sleeping and dreaming. As
This indicates that for members of a society to mentioned above, it is considered a significant
engage in playful activity has the potential to help for the process of maturing from children to
significantly influence the later performance of fully rounded adults and a strongly determining
individuals as well as entire groups. Every mili- factor in the shaping of a functioning member of
tary maneuver (Giles 1974; Leonhard 1994) or society (Bruce 2004). Specifically, video games
every play-acting of, e.g., household situations – are said to have a unique persuasive power and to
as often performed by children – can be seen as a have the potential to support existing social and
game with educational content. cultural positions (Bogost 2007). They can main-
When investigating the aspects that are tain high motivation levels and seem to center
credited with making a gaming experience fun, around a number of fundamental principles
many parallels are found with what researchers (Malone and Lepper 1987). We briefly discuss
such as Gee (2003) and Tiotuico et al. (2008) intrinsic motivation attributed to games and iden-
think makes for a good learning experience. This tify a number of key skills and abilities that are
is consistent with insights from psychology which especially prone to be supported through educa-
have long since accepted that the playing of games tional games.
is an important factor in the early development of
children and young adults. Bruce (2004) showed Intrinsic Motivation
that the act of playing during childhood can Malone and Lepper (1987) offer a detailed argu-
strongly impact social behavior later in life. ment on just how important intrinsic motivation is
Green et al. (2010) report on research that indi- to the designer of educational (computer) games.
cates that playing action video games can have a They identify four individual factors as well as
beneficial impact in decision making. three interpersonal factors that are elementary to
As far back as the 1970s, games have also been providing and maintaining intrinsic motivation
considered in the context of serving as a platform (Table 1).
Computer Games in Education 409

As stated by Connolly et al. (2008), it is safe to world, inviting the player to experiment and to
say that the effect of (computer) games-based take risks.
learning has been analyzed from a variety of both • Challenge and Consolidation: Through repeti-
negative (e.g., aggression, violence or gender tion, the player can master skills and advance
stereotyping) as well as positive (e.g., skills devel- through increasingly challenging stages.
opment, engagement, learning or motivation) • Pleasantly frustrating: Good computer games
perspectives. In all these, as two successive have realistically attainable goals that are, C
comparative studies by Connolly et al. (2007a, b) while achievable, at the outer edge of the
have shown, curiosity, challenge and cooperation players regime of competence.
consistently emerged as the most important moti- • Well-ordered problems: Offering an underly-
vations for playing computer games. ing structure to presented challenges motivates
the player to consider the problems on an
Fundamental Principles of Good Games abstract level. This allows to draw on previous
Besides aiming at important aspects of intrinsic experiences to solve future problems.
motivation identified in the previous section the • System thinking: Good games encourage or
game should aim to include as many as possible of even require players to think about the effect
the fundamental principles of good games. There their decisions have on the course of the game.
is an extensive body of literature that tries to As a result a player considers abstract relation-
identify those principles; Gee (2003, 2004, ships instead of isolated events.
2005) identified the following (Table 2). • Agency: Giving the player a sense of owner-
These principles are: ship over their decisions.

• Identity: Through fictional identities, a game When creating a serious game, the designer
stimulates the player to embrace a new role and should try to include and address as many of the
to take on the corresponding responsibilities. above as possible. In the next section we discuss
Adapting this new role requires learning new key skills that have been shown to be supported
domain knowledge. by GBL. It is unlikely that any one game will
• Interaction: Players experience responses/ target all of these skills equally or even at the
feedback to their actions. same time. The principles listed above should be
• Customization: Customizing a game to the spe- adopted, but which principles should receive the
cific interests of a group elicits the group’s largest attention will be determined by the
interest and provide stimulating challenges. targeted skills.
• Production: Decisions of the players directly
impact events in the game. Key Skills Supported by Games-Based
• Risk Taking: Since the game provides a stand- Learning
alone environment in a virtual world, actions in A (non-exhaustive) list of key skills that have
the game rarely have consequences in the real been shown to be promoted by serious games
has been published by Hildmann and Hirsch
Computer Games in Education, Table 1 The individ-
ual and interpersonal factors considered to be the main
factors provide and maintain intrinsic motivation (Malone Computer Games in Education, Table 2 Fundamental
and Lepper 1987) principles of good games, cf. Gee (2003, 2004, 2005)
Individual factors Interpersonal factors Identity Challenge & Consolidation
Challenge Cooperation Interaction Pleasantly frustrating
Fantasy Competition Customization Well-ordered problems
Curiosity Recognition Production System thinking
Control Risk taking Agency
410 Computer Games in Education

(2008). For a detailed discourse on these the 25 years ago by Randel et al. (1992) found that
reader is referred to Healy (2006) (Table 3). only ≈50% of the considered studies showed any
significant difference between using games and
using conventional instructions. The field has cer-
Considerations, Shortcomings, and tainly evolved since the 1990’s and market ana-
Disadvantages lysts consider games-based learning to be one of
the fastest growing division in the eLearning mar-
Computer games are more and more considered as ket (cf. Connolly et al. 2008), but commercial
tools in the education sector (Pee 2011). This success does not imply the validity of an
interest notwithstanding, there are many negative approach. As Gura (2005) cautions: “[e]ducation
aspects of games acknowledged in the literature is a highly politicized field, [. . .] littered with
such as e.g., the increase of aggressive behavior obstacles to reform and populated by powerful
(Uhlmann and Swanson 2004) or the decrease in individuals with their own pet theories”. The
physical exercise (Liliana Escobar-Chaves et al. fact that there is a market for the idea does not
2010). imply that the idea works; it merely suggests that
Among these negative aspects is also the it is selling well. It is not uncommon that the
proven fact that computer games can be extremely performances and benefits claimed by the market-
addictive (which should be unsurprising since ing departments of major game producers are met
gambling, known to be highly addictive, is often with considerable skepticism by some experts in
embodied by a game). The fact that games can the field (cf. Fuyuno 2007).
become an alternate reality to which the player There is a shortage of solid evidence for the
can escape from real world problems has been validity of using computer games for teaching,
widely discussed, and while this surely has large training and instruction Connolly et al. (2007b).
potential for negative consequences, it is also a Furthermore, a fair number of known shortcom-
property that motivates the use of games as edu- ings are in need of receiving more attention from
cational tools: problems that real life scenarios the community. Examples of such open chal-
and approaches inherently face can often be lenges are the distinct lack of frameworks or
completely ignored in games. guidelines. GBL continues to attract attention by
funding bodies and, to a lesser degree, acceptance
Lack of Supporting Evidence as a mainstream approach for learning and teach-
While we have motivated the benefits of using ing, and the focus should be on guidelines and
computer games in the educational sector, the frameworks to support other advances in the field.
GBL approach is far from being accepted by all However, it is also the case that games for play
practitioners in the field. There are plenty of rea- are often perceived differently from games for
sons to caution against hailing the use of computer serious purposes. When this distinction is made
games as a silver bullet for all educational chal- by the player it can have a significant impact on
lenges. For example, an investigation conducted the willingness to engage in a game and – as a
result – on the benefits gained from playing that
Computer Games in Education, Table 3 A summary
game. If knowing whether a game is meant to be
of the key skills that can be promoted through serious serving a serious purpose affects the performance
games (cf. Hildmann and Hirsch 2008), for a full discus- of the game, then cunning game design and the
sion the reader is referred to Healy (2006) framing of learning activities becomes a signifi-
Problem solving Communication cant factor. This makes objective evaluation even
Analytical thinking Discovery more difficult and once again highlights the
Team working Negotiating importance of applying best practices from the
Social and cultural Logical thinking fields of psychology and pedagogics when engag-
Critical thinking Visualization ing in the development of serious games.
Computer Games in Education 411

Lack of Interdisciplinary Communication Cross-References

The field of games-based learning and serious
games is commonly dominated by experts from ▶ Computer Games and the Evolution of Digital
the computer games industry and commercially Rights
driven by companies. In Hildmann and Hildmann ▶ Gamification
(2009), the authors emphasize the gap between ▶ Gamification and Serious Games
approaches to teaching and learning offered through ▶ Serious Online Games for Engaged Learning C
GBL and established models for human learning Through Flow
(from cognitive or behavioral psychology) or vali- ▶ Social-, Mobile-, and Multi-Player-Games and
dated practices used by teachers in classrooms. In Their Impact on Today’s Online Entertainment
light of the large amount of funding recently offered Industry
for the development of new learning approaches,
there is the danger of overlooking basic needs and
requirements. This problem is well documented in References
the literature: Powell (2003) states that “some edu-
cation experts argue that the scientists who are now Arsenio, A.M.: Cognitive-developmental learning for a
humanoid robot: A caregiver’s gift, Technical Report
developing an interest in the subject are ignoring
AITR-2004-006, MIT Artificial Intelligence Labora-
prior research”, while Gura (2005) goes so far as to tory (2004)
claim that “[u]ntil now, science and educational Babb, E.M., Leslie, M.A., Slyke, M.D.V.: The potential of
research have not mixed well”. While more recent business-gaming methods in research. J. Bus. 39,
465 (1966)
publications increasingly acknowledge the need to
Beale, I.L., Kato, P.M., Marin-Bowling, V.M., Guthrie, N.,
integrate alternative approaches instead of imposing Cole, S.W.: Improvement in cancer-related knowledge
them on the field (e.g., McGonigal (2012): “The following use of a psychoeducational video game for
future will belong to those who can understand, adolescents and young adults with cancer.
J. Adolesc. Health. 41(3), 263–270 (2007)
design and play games”), there is still the issue of Bogost, I.: Persuasive Games: The Expressive Power of
verification and comparison of the usefulness of Videogames. MIT Press, Cambridge, MA/London (2007)
contributions. While the short-term gains from Brown, S. Play as an organizing principle: clinical evi-
heeding the advice of pedagogues, psychologists dence and personal observations, chapter
12, pp. 243–260. Cambridge University Press (1998)
and practitioners in the training sector may be
Bruce, T. Developing learning in early childhood, 0–8
reduced, the long-term benefit (both financially as series. Paul Chapman, London (2004)
well as with regard to the actual performance of the Christoph, N.: The role of metacognitive skills in learning
created products) can only increase. Therefore, to solve problems. Ph.D. thesis, University of Amster-
dam, Amsterdam (2006)
computer scientists and game designers have to
Chua, A.Y.K.: The design and implementation of a simu-
start listening more to the practitioners. At the lation game for teaching knowledge management: special
same time, the education sector is undergoing fun- topic section on knowledge management in asia. J. Am.
damental changes and teachers in a classroom have Soc. Inf. Sci. Technol. 56(11), 1207–1216 (2005)
Cohen, K.J., Rhenman, E.: The role of management games
to accept that old teaching paradigms might have
in education and research. Manag. Sci. 7(2), 131–166
become outdated or may be falling behind what is (1961)
technically possible. Even without challenging any Connolly, T., Stansfield, M.: From eLearning to games-
of the established theories about how children learn, based eLearning: using interactive technologies in
teaching information systems. Int. J. Inf. Technol.
the world in which today’s children grow up is Manag. 6(2/3/4), 188–208 (2007)
vastly different from the world into which senior Connolly, T., Boyle, E., Hainey, T.: A survey of students’
educationalists were born. All sides need to motivations for playing computer games: a compara-
acknowledge that there are contributions from tive analysis. In: Proceedings of the 1st European Con-
ference on Games-Based Learning (ECGBL) (2007a)
other fields, and that only by working together and Connolly, T., Boyle, E., Stansfield, M., Hainey, T.:
fostering interdisciplinary communication can GBL A survey of students’ computer game playing habits.
achieve its full potential. J. Adv. Technol. Learn. 4(4) (2007b)
412 Computer Games in Education

Connolly, T., Stansfield, M., Josephson, J., Lzaro, N., Hildmann, H., Hirsch, B.: Raising awareness for environ-
Rubio, G., Ortiz, C., Tsvetkova, N., Tsvetanova, S.: mental issues through mobile device based serious
Using alternate reality games to support language games. In: 4th Microsoft Academic Days (2008)
learning. In: Proceedings Web-Based Education WBE Jackson, M.: Christopher Strachey: a personal recollection.
2008, Innsbruck (2008) High. Order Symbol Comput. 13, 73–74 (2000)
de Freitas, S.: Technology: game for change (review of Johnson, W.L., Wu, S.: Assessing aptitude for learning
McGonigal (2012)). Nature. 470, 330–331 (2011) with a serious game for foreign language and culture.
Egenfeldt-Nielsen, S., Smith, J., Tosca, S.: Understanding In: Proceedings of the 9th International Conference
Video Games: the Essential Introduction. Routledge, on Intelligent Tutoring Systems, ITS ‘08,
New York (2008) pp. 520–529. Springer-Verlag, Berlin/Heidelberg
Ford Jr., C.W., Minsker, S.: Treez – an educational data (2008)
structures game. J. Comput. Sci. Coll. 18(6), 180–185 Lennon, J.L.: Debriefings of web-based malaria games.
(2003) Simul. Gaming. 37(3), 350–356 (2006)
Fuyuno, I.: Brain craze. Nature. 447, 18–20 (2007) Leonhard, R.: The Art of Maneuver: Maneuver Warfare
Gee, J.: What Video Games Have to Teach Us About Theory and Airland Battle. Presidio Press (1994)
Learning and Literacy. Palgrave Macmillan, Liliana Escobar-Chaves, S., Markham, C.M., Addy, R.C.,
New York (2003) Greisinger, A., Murray, N.G., Brehm, B.: The fun fam-
Gee, J.: Situated Language and Learning: A Critique of ilies study: intervention to reduce children/’s tv view-
Traditional Schooling. Routledge, New York (2004) ing. Obesity. 18, S99–S101 (2010). https://doi.org/10.
Gee, J.: Why Video Games Are Good for Your Soul: 1038/oby.2009.438
Pleasure and Learning. Common Ground, Melbourne Long, G.: A detailed investigation of the applicability and
(2005) utility of simulation and gaming in the teaching of civil
Giles, L.: Sun Tzǔ on the Art of War: The Oldest Military engineering students. PhD thesis, University of Not-
Treatise in the World. Hong Kong Book Co. (1974) tingham (2010)
Green, C.S., Pouget, A., Bavelier, D.: Improved probabi- Ma, M., Oikonomou, A., Jain, L. (eds.): Serious Games
listic inference as a general learning mechanism with and Edutainment Applications. Springer, London
action video games. Curr Biol. 20, 1573–1579 (2010) (2012)
Gura, T.: Educational research: big plans for little brains. Malone, T., Lepper, M.R.: Making learning fun: a taxon-
Nature. 435, 1156–1158 (2005) omy of intrinsic motivations for learning. Aptitude,
Habgood, M.P.J.: The effective integration of digital games Learning and Instruction. 3: Conative and affective
and learning content. Ph.D. thesis, University of Not- process analysis, pp. 223–235 (1987)
tingham (2007) McGonigal, J.: Reality Is Broken: Why Games Make Us
Healy, A.: Does game based learning, based on construc- Better and How They Can Change the World. Random
tivist pedagogy, enhance the learning experience and House (2012)
outcomes for the student compared to a traditional Nyitray, K.: William Alfred Higinbotham: scientist, activ-
didactic pedagogy? Master’s thesis, University of Pais- ist, and computer game pioneer. IEEE Ann. Hist.
ley, Paisley (2006) Comput. 33(2), 96–101 (2011)
Hildmann, J.: Problems are chances in disguise – promot- Pasin, F., Giroux, H.: The impact of a simulation game on
ing social and personal skills through experimental operations management education. Comput.
education in the classroom. Ph.D. thesis, Ludwig- Educ. 57(1), 1240–1254 (2011)
Maximilians-Universit¨at, Munich (2010) Pee, N.C. Computer games use in an educational system.
Hildmann, H., Hildmann, J.: A critical reflection on the Ph.D. thesis, University of Nottingham (2011)
potential of mobile device based tools to assist in the Powell, K.: Science education: spare me the lecture.
professional evaluation and assessment of observable Nature. 425, 234–236 (2003)
aspects of learning or (game) playing. In: Proceedings Puschel, T., Lang, F., Bodenstein, C., Neumann, D.:
of 3rd European Conference on Games Based Learning A service request acceptance model for revenue opti-
(ECGBL09). API, Academic Publishing International, mization – evaluating policies using a web based
Graz (2009) resource management game. In: Proceedings of the
Hildmann, H., Hildmann, J.: Serious Games and Edutain- 2010 43rd Hawaii International Conference on System
ment Applications. In: Ma et al. (2012), chapter 6: Sciences’, HICSS ’10, pp. 1–10. IEEE Computer Soci-
A formalism to define, assess and evaluate player ety, Washington, DC (2010)
behaviour in mobile device based serious games Randel, J.M., Morris, B.A., Wetzel, C.D., Whitehill, B.V.:
(2012a) The effectiveness of games for educational purposes: a
Hildmann, J., Hildmann, H.: Serious Games and Edutain- review of recent research. Simul. Gaming. 23(3),
ment Applications. In: Ma et al. (2012), chapter 8: 261–276 (1992)
Augmenting initiative game worlds with mobile digital Rankin, Y.A., Gold, R., Gooch, B.: Playing for keeps:
devices (2012b) gaming as a language learning tool. In: ACM
Computer Go 413

SIGGRAPH 2006 Educators Program, SIGGRAPH

’06. ACM, New York (2006) Computer Go
Ricciardi, F., Marting, E., Association, A.M. Top Man-
agement Decision Simulation: The AMA Approach
(1957) Kazuki Yoshizoe1 and Martin Müller2
1
Roubidoux, M.A.: Breast cancer detective: a computer Graduate School of Frontier Sciences,
game to teach breast cancer screening to native ameri- The University of Tokyo, Tokyo, Japan
can patients. J. Cancer Educ. 20(1 Suppl), 87–91
(2005)
2
University of Alberta, Edmonton, AB, Canada C
Rowland, K.M., Gardner, D.M. The Uses of Business
Gaming in Education and Laboratory Research, Vol.
BEBR No. 10 of Education: Gaming. University of Synonyms
Illinois, Champaign (1971)
Sandberg, J., Christoph, N., Emans, B.: Tutor training: a
systematic investigation of tutor requirements and an Computer Baduk; Computer Weiqi
evaluation of a training. Br. J. Educ. Technol. 32(1),
69–90 (2001)
Schneider, M., Carley, K.M., Moon, I.-C. Detailed com-
parison of Americas Army game and unit of action Definition
experiments. Technical report. Carnegie Mellon Uni-
versity (2005) Computer Go was an interesting target in AI
Squire, K., Barnett, M., Grant, J.M., Higginbotham, T.: domain because Go was exceptionally difficult
Electromagnetism supercharged!: learning physics
with digital simulation games. In: Proc. 6th Int. for computers among popular two-player zero-
Conf. on Learning Sciences. ACM Press, sum games.
pp. 513–520 (2004)
Tiotuico, N., Kroll-Peters, O., Stelter, T., Odry, D. Game
design: motivation for mobile gaming created by new
technologies. In: Hegering, H.-G., Lehmann, A., Overview
Ohlbach, H.J., Scheideler, C. (eds.) GI Jahrestagung
(1), Vol. 133 of LNI, GI, pp. 505–506 (2008) As widely known, computers are now superior to
Tzu, S., Cleary, T.: The Art of War. Sun Tzu, Shambhala human beings in most of the popular two-player
(1988)
Uhlmann, E., Swanson, J.: Exposure to violent video zero-sum perfect information games including check-
games increases automatic aggressiveness. ers, chess, shogi, and Go. The minimax search-based
J. Adolesc. 27(1), 41–52 (2004) approach is known to be effective for most games in
Warren, S.J., Jones, G., Trombley, A.: Skipping pong this category. Since Go is also one of such games,
and moving straight to world of warcraft: the chal-
lenge of research with complex games used for learn- intuitively minimax search should also work for
ing. Int. J. Web Based Communities. 7(2), 218–233 Go. However, despite the simple rules which had
(2011) changed only slightly in these 2,000 years, Go is
Young, J., Upitis, R.: The microworld of phoenix quest: arguably the last two-player zero-sum game in
social and cognitive considerations. Educ. Inf. Technol.
4(4), 391–408 (1999) which human beings are still superior to computers.
Zhu, Q., Wang, T., Tan, S.: Adapting game technology to The solution to the difficulty of Go was a
support software engineering process teaching: from combination of random sampling and search.
simse to mo-seprocess. In: ‘Proceedings of the Third The resulting algorithm, Monte Carlo tree search
International Conference on Natural Computation –
volume 05’, ICNC ’07, pp. 777–780. IEEE Computer (MCTS), was not only a major breakthrough for
Society, Washington, DC (2007) computer Go but also an important invention for
many other domains related to AI. The strength of
computer Go had rapidly improved since the
invention of MCTS.
Computer Generated This entry consists of introduction to the game
of Go and computer Go topics before and after the
▶ Postproduction in Game Cinematics invention of MCTS.
414 Computer Go

Computer Go, Fig. 1 Left, An empty 19 19 board. Right, middle game position taken from one of the most famous
games in Go history

Game of Go

History of the Game

The game of Go originated in China and had been
played for more than 2,000 years. It is one of the
most popular two-player board games. The game C
is called Go or Igo in Japan, Baduk in Korea, and
A
Weiqi in China. Because the Japanese Go associ-
ation took main part in spreading Go to the world,
Go became the most popular word for the game.
The word Go will be used for this entry. Most of
the players reside in East Asia but in the last B
century it got more popular in the rest of the
world. The population of Go players is thought D
to be approximately 40 million. There are profes-
sional organizations in East Asian countries, and
several hundreds of professional players belong to
these organizations. Computer Go, Fig. 2 Capturing and illegal move

Rules of Go rules. For beginners and short-time games, 9  9

or 13  13 grids are also used (the rule is inde-
Equipment pendent of the size of the grid and it can be played
Go is played on a board with a grid. Players on arbitrary-sized board) (See Figs. 1 and 2).
alternately place one stone at a time on an empty
intersection of the grid. The first player uses black Capturing
stones and the second player uses white stones. Once placed, stones never move on the board.
The aim of the game is to occupy as much terri- Stones get connected in four directions, vertically
tory as possible. A 19  19 grid is used in official or horizontally (do not connect diagonally).
Computer Go 415

A B A B

D E C D

Computer Go, Fig. 3 Suicide moves, eyes Computer Go, Fig. 4 Examples of Ko

Connected stones form a block, and if a block gets Making eyes is important for the game (cf. section
completely surrounded by opponent’s stones, the “Life and Death”).
block will be captured and removed from the There is a variation of rules which allows sui-
board. Capturing opponent stones is often advan- cide of more than one stones (e.g., New Zealand
tageous because it results in greater chances to rules). It gives some effects to theoretical analysis
occupy more territory. but will not be described in details in this entry
Empty intersections adjacent to a block are called because it is rarely used.
liberties. If a block has only one remaining liberty,
the block is in atari. Capturing occurs if an oppo- Ko and Repetition
nent stone occupies the last remaining liberty of a Similar to other board games, Go has a rule about
block. Examples of capturing are shown in Fig. 2. If avoiding repetitions. The simplest and most pop-
black plays on A or B, the white blocks are captured ular case of repetition occurs by capturing an
and removed, as shown on the right side. opponent stone resulting in a stone with only
one liberty. The example is shown in Fig. 4.
Suicide and Eye Black captures a white stone by playing on
It is prohibited to place a stone if the stone (or the A and then white can capture back the black
block which contains the newly placed stone) has stone by playing on B. The stones marked C and
no liberties. In other words, suicidal move is pro- D are also in Ko. To avoid infinite recapturing, a
hibited. For example, white is not allowed to play player must play another move, called Ko threat,
at C in Fig. 2. before capturing back. Ko adds more complexity
However, black is allowed to play at B in Fig. 3 in the game (both in practice and theory) and often
because it can capture the surrounding white makes it more interesting.
block and make liberties for the black stone at There are several variations in repetition
B. In Fig. 3, A, D, and E are all illegal moves for avoiding rules. Super Ko rule prohibits global
black. B and C are illegal moves for white. repetition (which of course includes simple Ko)
A single empty intersection surrounded by the (Super Ko means the global repetition.) For
stones of the same color is called an eye (in Fig. 3, human beings, accurate detection of Super Ko
A and D are white’s eyes and C is a black’s eye). during real games is difficult, and it is excluded
416 Computer Go

two consecutive passes are played. If the game

had ended by passes, the winner is decided by
the score. (Of course, players are allowed to
resign at any moment. The opponent will be
the winner.)
There are two rules for scoring, area scoring
and territory scoring. Area scoring counts the sum
of:

• The number of empty points only one player’s

stones surround
• The number of stones of each player
• Komi points to compensate the advantage of
the first player

Territory scoring counts the sum of:

Computer Go, Fig. 5 Safe (living) blocks • The number of empty points only one player’s
stones surround
from some of the official rules for human tourna- • Minus the number of stones captured by the
ment (e.g., Japanese Go association official rules). opponent
However, computer Go tournaments typically • Komi points to compensate the advantage of
use Super Ko rule because it is not a problem for the first player
computers. There are two types of Super Ko rule.
Situational Super Ko distinguishes the same board The outcome is similar for both rules and the
position if the next player is different and posi- difference rarely affects human players. However,
tional Super Ko does not. how to correctly handle territory scoring is an
interesting topic for computer Go. Area scoring
Life and Death is more computer friendly and used in most com-
If a player owns a group of stones (consisting of pute Go tournament.
one or more blocks) which has two or more eyes,
the group will never be captured by the opponent, Strength of Human Players
unless the owner intentionally fills one of his own Strength of the players is measured by kyu and
eyes (filling own eye is almost always a dan. Human players are given a 25-kyu rank after
terrible move). learning rules. As players improve their strength,
Groups safe from capturing are alive. If a group the number decreases until it reaches 1 kyu.
cannot avoid capturing, it is dead. As the game Players of different ranks can play even games
ends, all stones on the board will be either alive or using handicap stones because having more
dead. The black blocks in Fig. 5 are all alive. Life stones in the opening is more advantageous.
and death is not a part of the rule, but it is a natural The difference between the ranks is used as the
consequence of the rules and the concept is crucial number of handicap stones (e.g., a 5-kyu player
for the game. and a 1-kyu player play with four handicap
stones).
End of Game and Scoring Shodan (which means first dan) is given to
For the game of Go, pass is always a legal players who are 1-stone stronger than 1 kyu, and
move. Players can pass if there is no other then the number increases for stronger players. It
beneficial move remaining. The game ends if normally requires more than 1 year of training to
Computer Go 417

Computer Go, Fig. 6 9  9 board endgame examples

Computer Go, Table 1 Computer strength for two- Computer Go, Table 2 Search space size of two-player
player zero-sum games without Monte-Carlo Tree Search games
(as of 2015)
Game Search space
Game Strength Checkers 1020
Checkers Perfect play is possible Othello 1028
Othello Stronger than human champion Chess 1045
Chess Stronger than human champion Shogi 1070
Shogi Approximately as strong as human Go (19  19) 10172
champion Go (9  9) 1038
Go 9  9 Approximately 3 kyu (based on authors’
guess)
Go Approximately 3 kyu only exception in this category of games
19  19
(Table 1).

reach shodan. The strongest amateur players are Difficulty: Search Space Size
rated approximately 9 dan. Professional players One of the difficulties of the game of Go is the
also use the same word dan, but the difference is enormous search space size. The search spaces of
not measured by the number of handicap stones popular two-player zero-sum games are listed in
(Fig. 6). Table 2 (numbers are from Schaeffer et al. 2014).
The game of Go has the greatest search space size.
Checkers was solved by exhaustive search in 2007.
Computer Go Difficulty Go search space is far beyond the limit of current
(and at least near future) computational power.
Theoretically, minimax search can find the opti- It is empirically known that computers tend to
mal move for two-player zero-sum perfect infor- be stronger for smaller games if the rules are
mation games. For most popular games in this similar. However, there was a small difference in
category, minimax search combined with alpha- the strength of 9  9 Go and 19  19 Go for
beta pruning (e.g., alpha-beta search) actually non-MCTS programs. This fact indicates that the
succeeded in making programs which is at least search space size is not the only reason for the
as strong as human champion (Fig. 1). Go is the difficulty of Go.
418 Computer Go

Difficulty: Evaluation Function

Among the games shown in Table 2, only check-
ers is solved by exhaustively searching the game
13 15
states. For the rest of the games, the search space
is too enormous. Therefore, minimax search 9 12 14 16
prunes unpromising branches based on evaluation
functions. Fast and accurate evaluation functions 5 8 10 11 17
for the games were made using a combination of
1 4 6 7
handcrafted codes and machine learning
techniques. 2 3
Despite the simple rules, Go was the only
exception. It is widely believed that making eval-
uation function for Go is difficult compared to
other two-player games. There are many explana-
tions for the difficulty. Unlike chess, there is no
clue such as the value of the pieces because all
stones are identical. Unlike Othello, focusing on
Computer Go, Fig. 7 Ladder example
important portions of the board (e.g., corner
points or edges) didn’t work. Because it is a
territory-enclosing game, it seems like it is possi- zigzag shape (Fig. 7). Many Go programs use
ble to predict the final territory, but it is only ladder search as one of the tactical components.
possible in the late endgames. Seeking for local
goals such as capturing opponent stones often Semeai
does not help in finding globally good moves. Another tactical problem is the capturing race or
The best evaluation functions developed for semeai, which is a variation of capturing problems.
Go (as of 2014) was either too slow or inaccurate. Capturing race occurs when two groups of different
Minimax search does not work without an evalu- colors are adjacent in an encircled space and can
ation function. A survey paper published in 2002 live only by capturing the other group. Normal
(Müller Jan. 2002) listed research challenges in algorithms for solving two-player games such as
computer Go. The first challenge in the list seems minimax search (alpha-beta search) and proof
very trivial: “Develop a Go program that can number search could be used to solve the problem.
automatically take advantage of greater pro-
cessing power.” It emphasizes the fact that Go Life and Death (Tsumego)
needed a new approach. One of the most important local tactical fights is
the life-and-death problems, also called Tsumego.
(In formal definition, Tsumego means life-and-
Before Monte Carlo Tree Search death problems with only one correct move, but
it is often used as the same meaning.)
Local Tactical Problems Killing (and capturing) a large group of stones
The Go board is large enough to have multiple is advantageous in general. Therefore, in real
local tactical fights. Although there is no guaran- games, it is crucial to correctly analyze life and
tee that locally good moves are globally good death of stones. Alpha-beta search-based solver
moves, blunders in local fights are often fatal. and df-pn (depth-first proof number) search-based
solver are both known to be effective.
Ladders If the problem is enclosed in a small region,
Ladder is a simple but important technique in these solvers are much faster than human players.
Go. Capturing a block after a sequence of ataris However, open-boundary Tsumego is still diffi-
is called a ladder, which normally results in a cult for computers.
Computer Go 419

Theoretical and Practical Analysis territory. The random play sequences until the
endgame is called playout.
Solving Go on Small Boards The basic idea is illustrated in Fig. 8. A given
The smallest size of the board which makes the board position will be evaluated by the average
Go interesting for human players is probably score of the playouts performed from the position.
5  5. Go on 5  5, 5  6, and 4  7 is solved In the figure, black had won 2 out of 3 playouts.
by search (van der Werf 2015). Therefore, the position might be promising for C
black.
Computational Complexity This is an extremely simple idea. All legal
Go using Japanese rules is proved to be moves are evaluated by playouts, and the move
EXPTIME-complete (Robson et al. 1983). The with the highest winning rate will be chosen (the
proofs with Chinese rules, the class is only proved left most branch in Fig. 9). Unsurprisingly,
to be somewhere between PSPACE-hard and one-ply Monte Carlo Go is weak because of a
EXPSPACE. fundamental weakness.
Assume that the playout is purely random
Endgame Theory except avoiding eye-filling moves. If there is a
Since Go is a territory-occupying game, the value threatening move with only one correct reply,
of each move can be described as the amount of the opponent will likely to choose the wrong
territory it will occupy. Combinatorial game the- reply in the playouts. Therefore, such a move
ory (CGT) (Berlekamp and Wolfe 1994) shows will be evaluated highly. The one-ply Monte
how to systematically analyze the values of moves Carlo Go program likes to play direct atari
as a sequence of numerical values and how to moves which are, in most cases, useless moves.
choose the optimal move after these analyses. In short, it tends to choose moves which expect
CGT solves difficult artificial positions better opponents to make blunders.
than human professionals, but there is no program The chance of choosing nonoptimal moves
which actually uses it in the play. will not be zero even given infinite computational
time. The limit of the strength is analyzed when
One-Ply Monte Carlo Go using simple playouts. The winning rate against
Because it was difficult to make good evaluation GNU Go on 9  9 board was approximately
function for Go, there was a different approach 10 %, and it was also extremely weak on
called one-ply Monte Carlo Go. (It was origi- 19  19 boards.
nally called Monte Carlo Go, but to distinguish The first known work was described in an
from Monte Carlo tree search, the term one-ply unpublished report written by Brügmann in 1993
Monte Carlo Go will be used throughout this (Brügmann 1993). There was more sophisticated
entry.) approach based on one-ply Monte Carlo Go. They
Because the number of legal moves decreases, had comparable strengths with other approaches,
it is possible for randomized players to end the but it was clearly not the most successful approach
game naturally according to the rules. If both for Go. However this idea is important because it
players randomly choose one of the legal moves, triggered the invention of the Monte Carlo tree
the game will continue for a long time because search algorithm.
filling own eyes results in repeatedly capturing
large blocks. However, given a simple rule to
avoid filling its own eyes, the game will end in a Monte Carlo Tree Search and Go
reasonably short time (average number of moves Programs
will be approximately the same as the number of
the intersections of the board). In this way it is As described above, one-ply Monte Carlo Go
possible to evaluate the given positions by letting introduced a new way of evaluating the board
random players play both sides and count the position which does not require an evaluation
420 Computer Go

black 2 win
white 1 win

Computer Go, Fig. 8 A 9  9 board evaluation by playout

Computer Go,
Fig. 9 Simplest one-ply black′s turn
Monte Carlo Go
white′s turn

black win playout

white win playout

function. But there was also a fundamental weak- (Coulom et al. 2006). It is widely regarded that
ness. The breakthrough came in the year 2006. the algorithm developed for Crazy Stone by
Coulom is the first MCTS algorithm.
Brief History of MCTS Invention Based on the success of Crazy Stone, Kocsis
Go program Crazy Stone, developed by a French and Csaba Szepesvári submitted the paper about
researcher Rémi Coulom, is the winner of the Upper Confidence applied to Trees (UCT) algo-
9  9 Go division of the 11th Computer Olym- rithm to ECML 2006 Conference (Kocsis and
piad taken place at Turin in 2006. The algorithm Szepesvári 2006). UCT had the proof of conver-
used in Crazy Stone was published at the same gence to the optimal solution which Crazy Stone’s
time in Computers and Games Conference which first approach did not have (explained in section
was one of the joint events with the Olympiad “UCT Algorithm”).
Computer Go 421

Computer Go,
Fig. 10 Enough playouts
on promising branches

black win playout

C

white win playout

Computer Go,
Fig. 11 Expand promising
nodes

black win playout

white win playout

At first, it seemed MCTS works only for small However, at this point, the definition of prom-
boards. However, soon after the UCT paper was ising branch is not clear. The key point of the
published, a Go program named MoGo became algorithm is the selection of promising branches
the first Go program to achieve a shodan on which is explained in the following sections.
19  19 board (Gelly et al. 2006) (on an Internet
Go server, KGS (KGS Go Server 2015)) and Theoretical Background: Multi-armed Bandit
became famous among Go players. The basic approach was surprisingly simple.
However, promising branch has to be decided
Basic Framework of MCTS appropriately. Possibly the simplest approach is
The differences between one-ply Monte Carlo Go to select the branch with the highest mean reward.
and MCTS seem simple. First, more playouts will But it is obviously a bad idea, because if the first
be performed from more promising branches playout of the (unknown) optimal branch had lost,
(Fig. 10). Then if the number of playouts on a it will never be selected again. Therefore, the
leaf node exceeds a threshold, the leaf will be selection method has to give an advantage to
expanded (Fig. 11). With these modifications, branches with small number playouts. More for-
the tree will grow in an unbalanced manner grow- merly saying, for MCTS to be successful,
ing toward the promising parts of the tree. It branches with large confidence interval must be
covers the weakness of the one-ply Monte Carlo given a positive bias. Theories of the multi-armed
Go programs and significantly improved the bandit (MAB) problem gave a solution. MAB is
strength. an old problem which is studied from 1930s.
422 Computer Go

The problem settings are as follows. You MAB — One-ply MC Go

have a certain number of coins and there is a arm — move
coin — playout
slot machine which has a number of arms. Each nu. coins — thinking time
arm returns a reward based on an unknown dis- minimize regret ? optimal move
tribution. The goal is to find a strategy which
minimizes the expected value of cumulative Computer Go, Fig. 12 Relation between MAB and
one-ply Monte Carlo Go
regret. Cumulative regret of a strategy is the
difference between the sum of the expected
reward of the strategy and the sum of the ideal
optimal reward which could be obtained by There is a close relation with MAB and one-ply
pulling the optimal arm every time. (There are Monte Carlo Go (Fig. 12). Each arm is a move,
many different formulations of MAB but this one coin is one playout, and the number of coins
entry focuses on the settings which is related to stands for the amount of the thinking time. The
MCTS and Go.) goal is slightly different, but as explained in the
Intuitively, part of the coins must be used to next section, UCB1 works well if combined with
explore the arm, and the majority of the coins tree search.
should be spent on the optimal arm. This is called
the exploration-exploitation dilemma. UCT Algorithm
Analysis of the theoretically optimal solution is UCT is a tree search algorithm which uses UCB1
already given in 1985 by Lai and Robbins (Lai for branch selection. UCT does the following
and Robbins 1985), but their algorithm was com- procedure repeatedly until a given time limit is
plex and time consuming. Auer et al. proposed a reached or a given number of playouts are
tractable and also optimal (with constant factor performed:
difference) strategy based on upper confidence
bound (UCB) (Auer et al. 2002). They proposed 1. Follow the branch with the highest UCB1
several strategies but here we only introduce value until reaching the leaf node.
UCB1. Algorithm UCB1 chooses the arm with 2. If the number of playouts at the leaf exceeds a
the highest UCB1 value which is defined as given threshold, expand the node.
3. Do one playout.
ln t 4. Update the values of the nodes on the path.
UCB1 ¼ Xi þ C ð1Þ
si
UCT is a generic algorithm which works for
where Xi is the mean reward of i-th arm, si is the various problems, and it also has a proof of con-
number of coins spent for the i-th machine, t is the verging to the optimal solution if the range of the
total number of coins spent so far, and C is a playout reward is in [0, 1]. However, in the same
constant called the exploration constant which is way, as the constant in UCB1, exploration con-
defined based on the range of the reward. For the stant C should be adjusted for UCT also (e.g., to
proof described make Go programs stronger).
p in Auer et al. (2002) to hold,
C should be 2 if the range of the reward is [0,
1]. However, it is also proposed that C should be Reward Definition and Playing Style
adjusted for the target problems to achieve better Crazy Stone attracted the attention of Go pro-
performance. grammers not only with the strength but also
The first term is the mean term and second term with the unique playing style. It won many
is the bias term. While arms with higher mean games by the smallest possible margin by inten-
tend to be chosen, the bias term gives an advan- tionally (it looked like so) playing safe-win
tage to arms with small number of coins. moves.
Computer Go 423

Play aggressively when losing; play safely needs for reading long sequences of moves in
when winning. It was a very difficult task for tactical situations (capturing or life and death is
minimax search-based programs. But MCTS- typical). It is difficult to make playouts correctly
based Go programs naturally acquire this ability. read tactical sequence. This is widely regarded as
It is based on the definition of playout rewards. the remaining weakness of MCTS-based Go
Since Go is a score-based game, it is possible to programs.
use the score itself as the reward. However, if the C
reward is two valued (e.g., 1 for win and 0 for
loss), MCTS tries to maximize the winning prob- Enhancements for MCTS-Based Go
ability, not the score difference. The early version Programs
of Crazy Stone was using the score as the reward,
and the winning rate against GNU Go was in RAVE and AMAF
30–40 % range. After the reward was changed to UCT has a proof of convergence and works fairly
0, 1, it jumped up to higher than 60 %. well, but state-of-the art Go programs (as of 2015)
are not relying on UCT. Practitioners ignored the
Why MCTS Works for Go (Or Weakness theory and replaced the bias term with other terms
of MCTS) using Go knowledge. Rapid Action Value Estima-
MCTS has a generic framework and it drastically tion (RAVE) is one of the most popular techniques
improved Go program strength. But, of course, it used in Go (Gelly et al. 2007).
is not an all mighty algorithm. Theoretical and Occupying a point is often crucial in Go
practical analysis revealed the weakness of regardless of the order of moves. A heuristic tech-
MCTS if the tree has a deceptive structure or trap. nique called All Moves As First (AMAF) heuris-
A trap is a tree where a small number of tic is invented based on this observation. Instead
branches have significantly better (or worse) of forgetting the sequence in playouts, AMAF
values than other branches. If a long sequence updates the values of all moves that appeared in
trap is in the tree, it is highly unlikely for MCTS playout sequences. It is inaccurate but the update
to find the correct solution. In Go the situation speed is improved by a large margin. In RAVE,
typically occurs in a ladder where only one move branches with small number of playouts use
is the correct move and all others are blunders. AMAF-based values, and as the playouts
Early MCTS-based Go programs did actually increases, it is gradually replaced by true values
miss long ladders in real games. of playouts.
A Go proverb says, “if you don’t know ladders,
don’t play Go.” It is impossible to make a strong Playout Enhancements
Go program without correctly recognizing lad- Improving playout quality is the most important
ders. Recent Go programs handle ladders by and subtle part of MCTS-based Go programs.
playouts. As explained later in section “Playout Both handcrafted approach and machine learning
Enhancements,” playouts used in recent Go pro- approach succeed (as of 2014).
grams are far from random. The ladder sequences MoGo had used handcrafted playouts, and it is
in real games are simple and playouts can solve said that program Zen (one of the strongest pro-
them. From the viewpoint of the tree search algo- grams in 2014) also uses at least partly
rithm, the trap is removed by playouts. handcrafted approach. Many other programs use
MCTS is a combination of tree search and different approach. Pattern-based features are
playout. Playout can read simple deep sequences. defined by programmers and the weights are
Tree search can select the best branch from vari- adjusted by machine learning. Typically, game
ous options. If the combination is effective, records played by strong players are used as train-
MCTS works well. However, there are often ing data, and the objective function will be the
424 Computer Go

Computer Go, Fig. 13 Transpositions and UCT

matching rate with the expert moves. In both Transpositions and MCTS
approaches, the playouts will choose more “rea- Game tree of Go is actually not tree but a directed
sonable” moves which makes it possible to solve cyclic graph. Transpositions often occur when
simple tactical situations including ladders. How different sequence of moves results in the same
to make good playout is still not clear because board position. As shown in the left of Fig. 13, it is
playout and tree search are correlated in a com- not trivial to decide the win rate of nodes for
plex manner and theoretical analysis is difficult. DAGs. Efficient handling of transpositions in
MCTS is still an interesting open problem
Progressive Widening (Fig. 13).
To find good moves in game playing, search Go programs uses mainly two ways. One is to
must focus on promising part of the tree. In ignore transpositions and use trees. This is
MCTS, progressive widening method is popu- wasting computational time, but it is possible to
larly used for pruning unpromising part. If the make strong enough programs based on trees.
number of playout at a node is small, only few The other is to record the values separately for
branches will be selected as the target of search. nodes and branches. UCT is proved to converge
As the number of playouts increases, more to the optimal solution if the values stored in
branches are added. nodes are used for mean term and values of the
branches are used for the bias term, as shown in
Parallelization the right of Fig. 13.
Using shared memory parallel, MCTS is common
for strong Go programs. Normal implementation
based on lock mechanism achieves speedup on Implementation Techniques
multi-core machines. It is also known that the Here is a list of common components and tech-
performance could be improved by using lockless niques for modern Go programs:
hash technique.
For distributed memory environment, root par- • Fast data structures for Go board, including
allel approach is used by several strong programs. block and pattern information.
Each compute node independently searches with • Fast pattern matcher including simple 3  3
different random seeds, and a small part of the tree matcher and heuristic features needed in both
is shared among the compute nodes (e.g., tree machine learning phase and playing phase.
nodes with depth 1–3 are shared). It is known to • Machine learning methods.
scale well for up to several dozens of computers. • Zobrist hashing for fast hash value calculation.
Computer Go 425

• Game database used as training data for Computer Go is improving rapidly and it is
machine learning and opening book difficult to predict even in the near future. At
construction. least for some more years, Go is likely to remain
• Time control for playing games in tournament. as one of the most interesting challenges in
• Pondering (thinking while the opponent is game AI.
thinking) and tree (or hash table) reuse.
• Dynamic komi. Especially important for C
handicapped games. Adjust virtual komi to Cross-References
avoid playing too safe (too aggressive) moves.
• Using the results of tactical searches such as ▶ Monte-Carlo Tree Search
capture search or life-and-death search.
• Opening book.
References

Concluding Remarks Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis
of the multi-armed bandit problem. Mach. Learn. 47,
235–256 (2002)
Current Computer Go Strength Berlekamp, E., Wolfe, D.: Mathematical go: chilling gets
N. Wedd maintains a Web page titled “Human- the last point. A K Peters, Wellesley (1994)
Computer Go Challenges” (Wedd 2015). After the Brügmann, B. Monte Carlo Go. Technical report, 1993.
Unpublished draft, http://www.althofer.de/Bruegmann-
invention of MCTS, strength of Go programs
MonteCarloGo.pdf
improved rapidly. From 2012 to 2014, strongest Go Coulom, R.: Efficient selectivity and backup operators in
programs (Zen and Crazy Stone) have played several Monte-Carlo tree search. In: Proceedings of the 5th
4-stone handicapped games against professional International Conference on Computers and Games
(CG’2006). Lecture Notes in Computer Science, vol.
players including former champions (4-stone handi- 4630, pp. 72–83 (2006)
cap means approximately 4-dan difference.) The Gelly, S., Wang, Y., Munos, R., Teytaud, O.: Modification
results include similar number of wins and losses. of UCT with patterns in Monte-Carlo Go. Technical
report 6062, INRIA (2006)
Gelly, S., Silver, D.: Combining online and offline knowl-
Discussion edge in UCT. In: Proceedings of the 24th International
Before the invention of MCTS, Go was regarded Conference on Machine Learning (ICML 2007),
as a grand challenge of game AI research because pp. 273–280 (2007)
of the difficulty. The difficulty of Go led to the Kgs go server. https://www.gokgs.com/. Accessed 12 Feb
2015
invention of an epoch-making algorithm, Monte Kocsis, L., Szepesvári, C.: Bandit based Monte-Carlo
Carlo tree search. Many MCTS-related researches planning. In: Proceedings 17th European Conference
exist both in theory and application and in game on Machine Learning (ECML 2006), pp. 282–293
and nongame domains. Still, Go is the most inten- (2006)
Lai, T.L., Robbins, H.: Asymptotically efficient adaptive
sively studied target for MCTS. allocation rules. Adv. Appl. Math. 6(1), 4–22 (1985)
There are many studies about search algorithm Müller, M.: Computer Go. Artif. Intell. 134(1–2), 145–179
and machine learning, which is combined with (2002)
many implementation techniques. Many Robson, J.M.: The complexity of go. In: IFIP Congress,
pp. 413–417 (1983)
researchers are working how to exploit increasing Schaeffer, J., Müller, M., Kishimoto, A.: Ais have mastered
computational power of recent computers. chess. will go be next? IEEE Spectrum, July 2014.
Recently, at the end of year 2014, the first success van der Werf, E.C.D.: First player scores for mxn go. http://
of deep learning approach for Go was reported. erikvanderwerf.tengen.nl/mxngo.html. Accessed Dec
2015
Deep learning could be the candidate for the Wedd, N.: Human-computer go challenges. http://www.
future breakthrough. It is still in early research computer-go.info/h-c/index.html. Accessed 12 Feb
phase, but the results seem promising. 2015
426 Computer Graphics

well-being. (Regaining skills,

Computer Graphics abilities, or knowledge that may
have been lost or compromised
▶ Planetary Generation in Games as well as helping disabled
people attain, keep, or improve
skills and functioning for daily
living – in line with the
Convention on the Rights of
Computer Graphics, Video Persons with Disabilities
Games, and Gamification (CRPD).).
Impacting (Re)Habilitation, Gesture Movement of the limb, torso, or
Healthcare, and Inclusive Motion whole body within a sensing
Well-Being space.
SoundScapes Contextually specific referring to
Anthony L. Brooks an author-conceived bespoke
Aalborg University, Aalborg, Denmark accessible, inclusive, and
interactive multimedia
environment targeting, through
Synonyms adaptive personalization,
creative expression, and playful
Biofeedback; Gamification; Gesture motion; interactions alongside user
Healthcare; Rehabilitation; SoundScapes; User experiences of enjoyment, fun,
interaction; Video games and entertainment. Utilized as an
alternative intervention concept
in healthcare and rehabilitation to
Definitions improve participation in
treatment programs. The concept
Biofeedback A sensing and response system is informed from the author’s
that sources physiologic human audiovisual art.
data (typically using sensors on- Video Computer graphics and sound
or off-body) mapped to Games elements comprising a “virtual
selectable content (typically environment” within which
digital) as feedback to self- players interact (via an input
inform participants of their input device) with typically animated
consciously, subconsciously, or objects displayed on a
subliminally. Often used in monitoring device for the sake of
treatments to teach patients to entertainment.
modify specific physiologic
functions. Introduction
Gamification Application of game elements
within a typically nongame As Tolstoy stated in “What is Art?” (Tolstoy
context. Contextually, in this case 1995 [1897]) – Art is a human activity consisting
[re]habilitation, healthcare, and in this, that one man [or woman] consciously by
interactive performance art, means of certain external signs, hands on to
targeting participant motivation, others feelings he has lived through, and that
immersion/engagement, and other are infected by these feelings and also
playfully rewarding positive experience them. In the text he states how art is
experiences offering inclusive a form of consciousness, framing in so doing the
Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation 427

essential role of art as a vehicle of communica- positive reactions. In other words, the author-
tion and empathy. developed sensor-based bespoke systems were
On a more recent note than Tolstoy, Grau primarily explored within the author’s work
(2003) posited: . . .ultimately, it is the intellectual of own stage performances at numerous national
vision, transposed into the work step by step with and international festivals (including tele-
technology as its reference, that remains the core vised performances). These explorations included
of a virtual work of art. instances where he directed and produced (e.g., C
This article introduces a body of work Cultural Paralympiad [Atlanta 1996 at Rialto
where the catalyst is creative expression and play- Theatre]; Cultural Olympiad and Paralympiad
ful interactivity. The author’s background as an [Sydney 2000 at Homebush arena]; European
artist is prevalent in how empowerment via em- City of Culture [Copenhagen 1996 at Arken
bodied interaction utilizing digital technologies MoMA and Avignon 2000 at Cafe Nine]; Danish
(predominantly sensor-based [on-body/off-body] NeWave New York at Gershwin, 1999; Scandina-
biofeedback mapped to digital multimedia [audi- vian Museums of Modern Art exhibition tours
tory, visual, robotic stimuli, etc.] and analogue 1995–1999; and other “art-related” settings, e.g.,
content [video feedback, vocals, etc.]) was identi- Roskilde Festival 2000 and more). All instances
fied as a means to supplement traditional interven- were targeted as research resulting in ongoing
tion in specific healthcare treatment programs and learning of system limitations, potentials, and
(re)habilitation. possibilities toward the envisioned applications
Within the work a commercial industry start- beyond solely traditional “art” forms. In other
up was realized from the author’s research, as well words, in the majority of cases – for example,
as international and national funded projects, and inclusive or adjacent to the above listed events –
global acclaim as, e.g., plenary keynotes at lead- demonstration workshops, hands-on tutorials/
ing international conferences, and more. seminars/symposiums, or other accessible show-
This contribution is focused upon sharing case forms were arranged at the author’s initiative
how in the 1990s, for approximately a decade, to present the “alternative art” application,
computer graphics were created as gesture-based i.e., in healthcare, rehabilitation, and therapeutic
interactive games under the author’s gamification training intervention. Such additional events
(including social interaction, creative expression, offered increased research and learning opportu-
and enjoyable play) approach to healthcare and nities including reviews, appraisals, assessments,
rehabilitation intervention. The core of the strat- and evaluations across disciplines, nationalities,
egy is a catalyst fun experience from within an and end users.
openly adaptive interactive environment that can
be tailored for each participant profile and the
treatment program goals. Most recently the work Bespoke Systems Overview: Leading to
has realized a series of publications under the Patent - see Brooks and Sorensen (2005)
theme of “Technologies for Inclusive Well-
Being.” Overviewing and simplifying, the systems
consisted of on-body and off-body systems that
were experimented having differing biosensing
Background profiles. Thus input was sourced ranging, for
example, from inner-body micro-electrical sig-
The work originated as an alternative contem- nals, through limb or whole-body gestural
porary avant-garde “human-at-center” art-related position and motion dynamics, to spatial environ-
creative expressive form (i.e., “human-as-art”). mental signals where human occlusion or signal
Following numerous proof-of-concept and feasi- generation results in system input. Thus, human
bility testing, early apparatus and method were gesture attributes (proprioceptive, kinaesthetic,
realized in performances and exhibitions with and related dynamics) and human state (being,
428 Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation

emotions, etc.) acted as input. Sourced input sig- instrument or tool that could supplement in and
nals are routed to selectable software for map- across rehabilitation and healthcare contexts.
pings (scaling, filtering, etc.) to impact feedback The concept was to explore creativity and play
responsive content (typically digital). This pro- as motivational human modes attempting to
cess routing of the data signals is managed to make the experience of treatment/training more
align desired relationships, i.e., interactive/reac- enjoyable, fun, and stimulating to participate
tive toward a goal of achieving flow, aesthetic within and less mundane, tedious, and boring.
resonance, self-agency, efficacy, and related idio- Once the system had reached certain maturity,
syncratic human attributes via afferent/efferent further reflections and critiques resulted in system
neural feedback loop closure. This causal loop improvements that aligned with external profes-
closure is achieved through the process of sionals who evaluated from a formal and profes-
optimally tailoring system attributes to human sional therapeutic perspective. Over the many
attributes, e.g., where the designed challenge is years, the family and friends of users also evalu-
matching user satisfaction and sense of achieve- ated – albeit in a more non-formal/informal
ment. Individual end-user profile assessment can context.
be either formal (with therapeutic input to realize A motivational intervention (in-action) model
targeted preset steps) or improvised (through sys- and an (on-action) evaluation model were devel-
tem operator’s – usually the author – experiences, oped and published to support practicing profes-
so more impromptu adjustment of change para- sionals and/or home use by families and carers
meters) to impact the system session design or even in self-use Brooks and Petersson (2005).
as experienced by a user. See Brooks Patent
US6893407B1 on method and apparatus.
Exploring Nuances of Differences

Embodied Interaction An ongoing vision was of creating, utilizing, and

exploring digital technologies to explore aspects
The designed embodied interaction considered of interactions such as nuances of differences that
intent and non-intentional input as well as con- may be apparent through dysfunction compensa-
scious and unconscious innate attributes. Through tory requirements (e.g., an augmented sense of
these means control and non-control have been hearing/touch if a person cannot see). An evolving
experimented Brooks (2004, 2018). interest from this vision is in the exemplification
Self-reflections and self-critique included from and integration of such finite sensorial differences
first-person and third-person experiences through and how they can be represented and utilized
which the system developed as a substantiated within the art-related works for wide-audience
open and adaptive entity. As such the system education and enjoyment. In other words, and
(apparatus and method) enabled adoptions of for example, how an “artist” with heightened sen-
various technologies as they appeared as both sory attribute (through loss of other sense) can
input interface apparatus and content toward opti- represent such a sensory nuance so that audience
mizing a used model for treating a range of members who are fully sensory loaded (thus,
patients including those “born with” impairment potentially, not as highly nuanced in a specific
or those who had “acquired” impairment either sense) may appreciate the art and be provoked to
through accident, incident, or disease. reflect on their educating of their evoked sense via
the artwork.
This compensatory approach questioning
Experience as Product comprehension of alternative channelling to aug-
ment training and other benefits from creatively
A goal behind the “human-as-art” performative expressing was apparently original within rehabil-
inquiries with the system was to realize a new itation fields. For example, the applied research
Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation 429

with acquired brain-injured patients in a clinical decrease after an injury – although, your brain
setting questioned how a sense of proprioception will still have the skills stored within, so it is easier
(a participant’s body sense of its parts and relative to relearn them. Even though your strength may
location to its other parts and the effort exerted come back easier, you will still need to spend time
in motion often related to balance and/or neglect) improving your proprioception and kinesthetic
that was damaged could be “trained” through awareness so that you can fully recover.
a patient listening to or seeing where their relative By employing a system that responds to C
position of their own parts of the body are instead motion within an invisible space that can be con-
of feeling it. Additionally, patient dynamics of trolled for data inhabitation and also mapping to
kinesthetic awareness of the position and move- digital content as stimuli offers opportunities in
ment of their parts of the body by means of learning through feeling such as a sense of body
their proprioceptors is targeted through program- position, muscle movement, and weight as felt
mable thresholds in the digital content algorithm through nerve endings.
according to a patient profile. Interactive com- Cognitive aspects are also involved.
puter graphics were thus engaged, alongside
other digital content, as a means to visually inform
a user of system input. Technologies for Inclusive Well-Being

Within the ongoing research, the art informs

Afferent/Efferent Neural Feedback Loop the design and intervention in rehabilitation in
Closure a cross correspondence such that the interven-
tion also informs the art. This is aligned with
Afferent/Efferent Neural Feedback Loop Closure: reconceptualization, reframing, and cross-domain
Proprioception and kinesthetic awareness are key mapping as a bilateral approach that has been
aspects of the concept presented herein. found effective in developing the research to the
Literature informs that the central nervous sys- next level; however, that is not elaborated herein
tem (CNS) receives sensory stimuli as (afferent) as it is a subject of another publication. Suffice to
impulses external to the body. It then sends appro- say that holistically, the research targets societal
priate (efferent) instructions to a person’s muscles impact in (re)habilitation and healthcare under an
and joints on how to react. The brain also receives umbrella titled “Technologies for Inclusive Well-
some messages that cause the body to react uncon- Being” under which a number of publications
sciously. Proprioception is a term referring to the have resulted with the author as lead editor. The
internal messaging (the central nervous system) next section introduces the author’s gamification
driving and controlling motion actions. Proprio- intervention approach.
ceptors are sensors in human joints, muscles, and
fascia, providing information needed to produce
coordinated movement. Kinesthetic awareness Computer Feedback Training Under
refers to our ability to navigate space and the Gamification Approach
awareness of how we move. Kinesthetic aware-
ness and proprioception work as partners to get us Near the time when biofeedback was being
through the movements of our lives from the explored via worn sensor systems, audio-
inside and the outside of the body. Muscle mem- visual computer feedback and a series of robotic
ory is a kinesthetic concept. So many things that light devices were applied under a gamifica-
we do without thinking – such as walking, tion approach to training in (re)habilitation and
whether we do it correctly or not – are kinesthetic healthcare as outlined in the previous section.
experiences based on proprioception, which pro- The purpose of this approach to supplement tra-
vides the awareness of our joints and body in ditional approaches in intervention within training
space. Proprioception and kinesthetic awareness treatment programs was to engage the participant
430 Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation

to enjoy the training (vs. mundane, boring, and

repetitive exercising without any self-reflective
feedback that informs of progression).

Fun, Play, and Creative Expression

Leading to Aesthetic Resonance

Fun, play, and creative expression are keywords

in this approach that stimulate self-efficacy, self-
agency, and a concept coined as aesthetic reso-
nance that was the subject of European projects
around the millennium (see Brooks 2011). In
these externally funded projects, a focus was on
creating systems where there was not the need for
patient preparations such as careful positioning of
sensors, the use of conductive gel to improve
signals, and other invasive aspects. A patient
could simply enter a space, set up an interactive
environment, and move to manipulate digital
responsive content. Initially the content was
audio due to the used MIDI (Musical Instrument
Digital Interface) signal protocol being native to Computer Graphics, Video Games, and Gamification
Impacting (Re)Habilitation, Healthcare, and Inclu-
communication within the music/sound domain. sive Well-Being, Fig. 1 The author’s gesture control of
However, within the applied research sessions, it Martin Lights graphics – SoundScapes at Olympics and
was clear that a wider selection of digital content Paralympics, Sydney 2000 via three-headed infrared sen-
to stimulate participants was required. Due to the sors mapping via MAX to DMX 512 via translation control
interface
author’s background in mainframe computers
(e.g. Honeywell TDC 2000), a decision was
undertaken to investigate computer-based video mentally challenged. Body painting was one
games and interactive graphics (Figs. 1, 2, and 3). of the activities whereby “digital paintings” were
Figure 1 illustrates interactive light gobos and created as graphical images in the computer
Figs. 2 and 3 a “body paint” algorithm developed through participants’ dynamic movements and
under a European project based on the author’s then printed as an A3 picture. The pictures were
research titled CAREHERE (Creating Aesthetic exhibited as shown in Fig. 4 for the duration of
Resonant Environments for Handicapped, Elderly the festival and given as gifts to the authors upon
and Rehabilitation). The algorithm was used cessation. Notable was how, even though the
within the project and beyond, including in the pictures were in abstract forms, on return to
author’s annual workshop hosted by Casa da collect their created artifact following the end
Musica, Porto, Portugal, which was a small part of the festival, the participants each identified
of a larger festival for disabled participants their own creation and collected it from the exhi-
organized by the education team at the venue. bition wall. It was astounding as there were no
Over a 2-week period, a variety of groups with names visible for such identification of the com-
differing profiles attended morning and afternoon puter graphics.
(two workshops daily). Age ranged from young In this simple exercise, it was clear how
children to elderly and across the spectrum a tangible outcome meant so much for the partic-
of dysfunction, both physical – including deaf, ipants. When using auditory feedback, there is
visually impaired, and wheelchair bound – and no tangible outcome unless a recording is made
Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation 431

Computer Graphics,
Video Games, and
Gamification Impacting
(Re)Habilitation,
Healthcare, and Inclusive
Well-Being, Fig. 2 Image
of head and hand painting
by PMLD participant C

Computer Graphics,
Video Games, and
Gamification Impacting
(Re)Habilitation,
Healthcare, and Inclusive
Well-Being, Fig. 3 Full-
body painting based upon
activity-level threshold
mapped to color chart

and given on USB stick or download. The design degree such that their stories, unusually remem-
was thus a success for participants and the orga- bered compared to their other living detail,
nizing staff who attended with the groups. On detailed aspects that her staff did not realize. She
return 4 months later for a conference, the leader explained that it was a moving experience for the
of the elderly home for mentally challenged/dys- families who attended.
functional attendees of the workshop informed Soon after the mid-1990s, investigations of
that the institute organized an exhibition and oral interactive animations and gamification within
presentation event so that the elderly could tell the SoundScapes research were conducted.
their own stories about creating their paintings. This preceded the Personics interactive com-
She explained how their motivation and inspira- puter graphic environment that is
tion had been stimulated to a highly positive introduced next.
432 Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation

Computer Graphics,
Video Games, and
Gamification Impacting
(Re)Habilitation,
Healthcare, and Inclusive
Well-Being,
Fig. 4 Eyesweb body paint
exhibition, Casa da Musica,
Porto, Portugal

Personics iterative design, leading intervention sessions,

and communicating proposed designs and refine-
Experiments within the research led to a first ments with the development team, alongside test-
simple “game” as an animation of an airplane ing and troubleshooting prior to sessions. The
that could take off, fly, and land with control via research personnel included a neuropsychologist,
unencumbered gesture (i.e., no mouse, joystick, a psychologist, a cognitive psychologist, and
or keyboard). The reader should be aware that a physiotherapist that worked under the Danish
this experimental use of games controlled with government Erhvervsfremme Styrelsen (Danish –
gesture technology was before the pervasiveness translated as business development agency) who
of games and gesture control peripherals in the funded the research project titled Humanics
early 2000s. for 9.5 million Danish Kroner (DKK). Another
The simple images on the following pages Danish government body, named Satspuljen,
indicate the further explorations of basic interac- funded the project for an additional one million
tive environments developed with the software Danish Kroner (DKK).
Macromedia Flash (later Adobe Flash) as game Parallel to this project was the aforemen-
content under the commercial company Personics tioned international project. The background of
as animated computer graphics as gamification this project is the research resulted in a European
in (re)habilitation therapy. The development was Union probe (under the European Network for
across the two funded aforementioned national Intelligent Information Interfaces – www.i3net.
and international projects having similar goals. org). This was a funded project titled “The
In one case, the development followed input World Is As You See It” (TWIAYSI) – with the
from the Humanics (the Danish government pro- University of Bristol and a Swedish partner.
ject) research team from the Center for Rehabili- TWIAYSI was developed into a European
tation of Brain Injury [CRBI], this being where the funded Framework V IST Key Action 1 project
clinical location of patient training was based at the supporting the program for applications relating
University of Copenhagen. The team was working to persons with special needs including the dis-
with acquired brain injury patients. Author-led abled and elderly titled CAREHERE (Creating
roles in the project included conceptualizing, Aesthetic Resonant Environments for
Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation 433

Computer Graphics,
Video Games, and
Gamification Impacting
(Re)Habilitation,
Healthcare, and Inclusive
Well-Being, Fig. 5
[Balloon] Computer
graphic with gameplay
receiving arm motion from
C
the participant to reach and
puncture the balloon with
the pin in the animated
hand. Time to complete the
task in upper left

Handicapped, Elderly and Rehabilitation) bmp screenshots depicting start screen and
Funding was approx. €2M. interactions.
Personics was invited by the author to partici- Figure 5 illustrates a balloon game concept
pate in both projects. where a sensor is placed at a specific location
Under the sponsorship of IBM at the according to therapist input. In Hagedorn and
World Congress for Physical Therapy (WCPT) Holm (2010), balance training exercise was for
in Yokohama, Japan, the author presented his each patient to alternate between normal stand-
research paper titled “Virtual Interactive Space ing balancing and toe-standing balancing. Bal-
(V.I.S.) as a Movement Capture Interface Tool loons were popped when each cycle was
Giving Multimedia Feedback for Treatment and completed within the sensing space. Duration of
Analysis” (Brooks, 1999). training depended on patient endurance. The
Approximately a decade later, Hagedorn and number of balloons popped gave indication of
Holm’s (2010) independent randomized inter- training effort.
vention study questioned traditional training ver- Figure 6 was a boxing game tracking the
sus computer feedback training (system resulting patient’s two hands (mapped to the lower boxing
from author’s research). This is reported in the gloves). Scores were archived according to
European Journal of Physical and Rehabilita- performance.
tion Medicine where results state impact gains Figure 7 was a game where the navigation of
of up to 400% illustrate potentials from using a Death Star fighter (Star Wars) was controlled
selected games in Virtual Interactive Space as by patient movement. Guidance through a maze
published in Brooks’ 1999 paper. The 2010 was tasked.
investigation is introduced after the next section Figure 8 was a Dolphin wireframe model that
that presents the simple interactive computer was mapped to two sensors representing horizon-
graphics used. tal and vertical travel. This is used successfully in
the CRBI research where a therapist controlled
one sensor and a patient controlled a second sen-
Personics Computer Graphics sor. Progression for the patient was to use both
sensors to control the full travel of the dolphin.
The following images represent the gameplay Time was recorded for each level and a number
graphics. Notes attempt to describe the gameplay of fish caught. Levels were progressively more
and target in the therapeutic sessions with difficult whereby lethal jellyfish had to be eluded.
434 Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation

Computer Graphics,
Video Games, and
Gamification Impacting
(Re)Habilitation,
Healthcare, and Inclusive
Well-Being, Fig. 6
[Boxer] A two-handed
exercise where sensors
detect dynamic motion of
each hand, which are
mapped to left and right
boxing gloves to strike
opponent who is able to
guard and strike back

Computer Graphics,
Video Games, and
Gamification Impacting
(Re)Habilitation,
Healthcare, and Inclusive
Well-Being, Fig. 7 [Death
Star] Flight simulator where
participant motions control
up-down (y-axis) and left-
right (x-axis) of Death Star
fighter vehicle to prevent
crashes and to reach
targeted goal

Figure 9 was used to task the patient in motion had to exceed a threshold in order to let
dynamic motion aligned with a weightlifter rais- go of the ball. This proved a favorite exercise
ing a dumbbell. Dynamic of motion was tracked for acquired brain-injured unilateral neglect or
within the sensing field. hemispatial neglect patients training a damaged
Figure 10 was themed from the Mission side.
Impossible film where a diamond was stolen. Figure 12 illustrates the tower that was one
Patient activity was through three sensor fields of the animated games used in the Hagedorn and
that would activate, deactivate, or alarm the sys- Holm (2010) study with balance elderly patients.
tem. The mapping of this game was mostly found The number of blocks and difficulty could be
to be too complicated for most patients. changed and archived. Feedback of balance in
Figure 11 illustrates another dynamic motion training was where one leg was lifted from the
computer graphic. In this case the patient’s hand ground – as in stepping actions associated with
Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation 435

Computer Graphics,
Video Games, and
Gamification Impacting
(Re)Habilitation,
Healthcare, and Inclusive
Well-Being, Fig. 8
[Dolphin] Wireframe
dolphin travel controlled by
two sensors mapped to
C
x-axis and y-axis to catch
and eat the dead fish
dropping from top of the
screen while escaping
hazards on each level

Computer Graphics,
Video Games, and
Gamification Impacting
(Re)Habilitation,
Healthcare, and Inclusive
Well-Being, Fig. 9
[Dumbbell] Sensors capture
participant lifting motion
dynamic and range to raise a
corresponding animated
dumbbell held by a
weightlifter

walking activity within the sensing space. The analysis. Physical level-of-difficulty adjustments
height of the built tower indicated training effort. took the form of a thicker foam plate that could
Both legs were trained. additionally be changed for a tilting board; also
The waiter tray game, as illustrated in Fig. 13, glass friction and plate size were adjustable.
was also used in the Hagedorn and Holm (2010) Figure 14 gives a direct feedback to a patient’s
study. Sensors captured body position and balance according to a horizontal line that was
adjusted the tray angle accordingly. Patients required to line up to the central division in the
stood on a firm plate, which was placed upon a graphic.
5 cm depth of dense foam. The numbers of broken The fourth game used in the training was where
glasses were recorded in each session of 2-min balance controlled the position of an animated
duration. A session consisted of two instances empty basket to catch images of fruit falling
of training where both data were included for from a tree. Healthy fruit was collected, while
436 Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation

Computer Graphics, Video Games, and Gamification

Impacting (Re)Habilitation, Healthcare, and Inclu-
sive Well-Being, Fig. 12 [Tower] Motion sensors detect
the moving of virtual blocks from a storage space to an
Computer Graphics, Video Games, and Gamification
adjacent position in order to create a tower
Impacting (Re)Habilitation, Healthcare, and Inclu-
sive Well-Being, Fig. 10 [Migame] Mission Impossible
task where three sensors are used tracking participant
motion to deactivate alarm to raise the glass dome and
reach the diamond target

Computer Graphics, Video Games, and Gamification

Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation, Healthcare, and Inclu-
Impacting (Re)Habilitation, Healthcare, and Inclu- sive Well-Being, Fig. 13 [Waiter] A balance game
sive Well-Being, Fig. 11 [Throw] Training for a dam- where weight on the right or left foot determines the incli-
aged arm (such as in acquired brain injury) where motion nation of the waiter tray. The goal is to keep the glass on the
sensors track dynamic of throw gesture. With sufficient tray and train dynamics. Tray size and friction coefficient
dynamic the hand releases the ball can be adjusted

rotten fruit was not. Each incremental level had transmits feelings through lines, colors, sound,
increasing speed of fruit falling. or words. The feelings embedded in the imagery
start with the creator and the creative process. The
work may take any form, but to be art, the object,
Discussion idea, or installation goes beyond the physical and
contain some form of human experience. Art may
Eber (1997) reflects how a work of art, according be created with any tool, as long as the artist rises
to Tolstoy (1995 [1897]), is sincere, and it beyond that tool into an experiential realm. Many
Computer Graphics, Video Games, and Gamification Impacting (Re)Habilitation 437

object that exists on a wall or behind a glass

but encompasses an infinite number of artistic
possibilities including alternative presentations,
environmental constructions, multisensory sti-
mulation, viewer interactivity, and theatrical
performance.
This text presents a historic perspective of C
the research alongside a review of the basic com-
puter graphics that resulted. The purpose of the
text is to share the narrative while attempting to
inspire next-generation researchers. The goal to
inspire targets championing others to persevere
against adversity, often when being mocked
when pursuing one’s original concept, in this
case a concept that resulted in national and inter-
national funded projects, a patented method and
apparatus, commercial industry product, and
Computer Graphics, Video Games, and Gamification
Impacting (Re)Habilitation, Healthcare, and Inclu- company start-up.
sive Well-Being, Fig. 14 [Balance] Motion sensors com- Early studies explored solely auditory stimulus
bined to perform as a mouse emulator driven by weight as a feedback to user input. Initially the means
distribution. The goal of the participant, typically with of input was hardware-based rocker control
acquired brain injury, was to position the dividing horizon-
tal line along the central balance. Arrows on the y-axis and pedals as typically used by musicians for altering
x-axis act as guides instrument output, e.g., guitar. Subsequently, user
biofeedback signals were sourced either via
on-body or off-body sensing interfaces with dif-
have debated the existence of the creative domain fering profiles. Such profiles have increasingly
with the computer art medium, especially virtual advanced over the years of the research such that
environments (VE). With the tools to create a VE, original interfaces are no longer viable when com-
the artist will learn a new technology that may pared to affordable and available computer game
influence the nature of and how she reaches the peripherals and camera-based solutions.
creative level (see Eber 1997). However, the “communication method and
According to Eber (1997), in addition to the apparatus” patent has been referenced 16 times,
acquisition of new information, the artist who including 12 by the patent examiner.
chooses to work with VEs also has a new set A further example of impact is an independent
of aesthetics to consider, as the final work of art investigation of product resulting from this
is wholly different from that using any other research in a randomized intervention study by
medium. Contrary to the concepts expounded in Hagedorn and Holm (2010) comparing conven-
the popular media, a VE art installation can tional balance training against computer feedback
be more than a display arena for the art of others training with older people. The randomized con-
(e.g., Picasso) or a “shoot 'em up” computer game. trolled 12-week intervention trial was designed
It can be a work of art in and of itself, one that on pre- and post-training evaluations conducted
requires of the artist the same level of abstraction on 35 outpatients of a geriatric falls and balance
into the spirit of creativity as any traditional clinic. Responsive computer graphics, with a vari-
medium demands. How and at what point does ety of selectable game themes, responded to input
the creative process happen for a VE artist in a motion sensors.
world of computer peripherals and code? Results were reported of 400% improvement
Further Eber (1997) states how an art instal- in specific performance; however, in the author’s
lation is a work of art that goes beyond an opinion, the industry is still lacking behind
438 Computer Vision

in training trainers to fully optimize such results Notes The figures (5 – 14) are from the author’s own
within a wide range of interventions so as to archive from employment. All efforts to get permission
have not been responded upon and it is understood that
benefit societally via transfer to activities of the company was closed shortly following the author’s
daily living (ADL). departure and copyright ownership is not listed for these
images. Acknowledgement made in this chapter for the
authors of the images created under the company Personics
who do not name or credit authorship.
Conclusion

This article shares insight of avant-garde art with

References
societal impact that has been recognized by third-
party researchers and educators as pioneering the Brooks, A.L.: Body electric and reality feedback
use of digital technologies with differently abled. loops: virtual interactive space and entertainment.
A focus has been on illustrating the simple com- In: Proceedings 14th International Conference on
puter graphics used as mappable content to give Artificial Reality and Telexistence (ICAT 2004),
pp. 93–98. Advanced Institute of Science and Technol-
feedback selectable from an array of stimuli. This ogy, Seoul (2004)
text informs how the healthcare rehabilitation Brooks, A.L.: SoundScapes: The Evolution of a Concept,
informs the art and correspondingly how the Apparatus and Method where Ludic Engagement in
art informs the healthcare rehabilitation. Virtual Interactive Space is a Supplemental Tool for
Therapeutic Motivation. (PhD) Institut for Arkitektur
The use of multimedia responsive feedback to og Medieteknologi. Aalborg University, Denmark
human input indicates how art (creative expres- (AD:MT, Vol. 57) (2011)
sion) and play (enjoyment and fun) have an Brooks, A.L.: An HCI approach in contemporary
important part to play in both physical and cogni- healthcare and (Re)habilitation. In: Norman, K.,
Kirakowski, J. (eds.) The Wiley Handbook of Human
tive therapy (re)habilitation. While extended Computer Interaction, vol. 2, pp. 923–943. Wiley,
range and dynamics of motion (body, limb, etc.) New York (2018)
are quantifiable, it is also evident through the Brooks A.L., Petersson, E.: Recursive reflection and
comprehension of designed interactions by pro- learning in raw data video analysis of interactive
‘play’ environments for special needs health care.
found and multiple learning disabled (PMLD) In: Proceedings, IEEE HEALTHCOM 2005,
how the concept has potentials beyond what has Enterprise networking and Computing in Healthcare
already been reported. Industry, Busan (2005)
To sum up and in accord with these conclu- Brooks, A.L., Sorensen, C.D.: Communication method
and apparatus. US Patent 6893407 (2005)
sions, it is important to point out how it is increas- Eber, D.E.: The creative process and the making of a virtual
ingly evident that improved systematic evaluation environment work of art. Marilyn Zurmuehlen Work.
is needed in this field to define the specific use Pap. Art Educ. 14(30), 159–163 (1997)
benefits of (computer graphics as) video games Grau, O.: Virtual Art: From Illusion to Immersion.
MIT Press, Cambridge (2003)
in healthcare and rehabilitation intervention. Hagedorn, D.K., Holm, E.: Effects of traditional phys-
Aspects of such evaluation are proposed as incor- ical training and visual computer feedback training
porating increased duration and control of trials in frail elderly patients. A randomized intervention
with improved measurements (randomization, study. Eur. J. Phys. Rehabil. Med. 46(2), 159–168
(2010)
blinding, etc.) and consistency of measurement Tolstoy, L.: What is Art? (Translated by Richard Pevear
tools across investigations including beyond and Larissa Volokhonsky). Penguin, London (1995
actual treatment programs to embrace impact on [1897])
activities of daily living (ADL). However, innate
to such a proposal are the ongoing challenges of
individual human differences that many may con-
sider immeasurable; yet, for improved research
and impact comprehension, it is important to tar- Computer Vision
get optimized research validity and reliability in
order to advance and educate. ▶ Fall Risk Detection in Computer Vision
Conceptual Model of Mobile Augmented Reality for Cultural Heritage 439

Computer Weiqi Conceptual Model of Mobile

Augmented Reality for
▶ Computer Go Cultural Heritage

Ulka Chandini Pendit1, Syamsul Bahrin Zaibon2

and Juliana Aida Abu Bakar2 C
1
Computer-Aided Design Department of Virtual Reality, Faculty of
(CAD) Creative Multimedia, Multimedia University,
Cyberjaya, Selangor, Malaysia
▶ Imagineering Ceramic Pottery Using Computer 2
Institude of Creative Humanities, Multimedia &
Graphics Innovation, School of Creative Industry
Management & Performing Arts, Universiti Utara
Malaysia, Sintok, Kedah, Malaysia

Computer-Aided Industrial
Design (CAID) Synonyms

▶ Imagineering Ceramic Pottery Using Computer Augmented reality; Enjoyable informal learning;
Graphics Mobile augmented reality for cultural heritage;
Mobile augmented reality

Computer-Mediated Reality Definition

▶ History of Augmented Reality Conceptual model of mobile augmented reality

for cultural heritage towards enjoyable informal
learning is a representation that provides, by a
composition of concepts, component of content,
Computing for Smart Toys navigation and user interface design, interactivity,
features, hardware, and process that is appropriate
▶ Toy Computing for enjoyable informal learning at cultural heri-
tage site for mobile augmented reality.

Introduction
Concatenative Sound
Synthesis
Mobile augmented reality (AR) for cultural heri-
tage site has been developed for already a decade
▶ Dynamic Music Generation: Audio Analysis-
(Angelopoulou et al. 2012; Armanno et al. 2012;
Synthesis Methods
Chang et al. 2015; Ciurea et al. 2014; iTACITUS
2007; Kim and Park 2011; Moorhouse et al. 2017;
Seo et al. 2011; Techcooltour 2013; Tussyadiah
et al. 2017; Vlahakis et al. 2002). However, these
Conceptual Blending existing applications lack enjoyable informal
learning concept (Damala 2009).
▶ Foundations of Interaction in the Virtual Real- Enjoyable informal learning is based on inter-
ity Medium pretation theory (informal learning in cultural
440 Conceptual Model of Mobile Augmented Reality for Cultural Heritage

Conceptual Model of Mobile Augmented Reality for Cultural Heritage, Table 1 Frequency of responses of expert
review
Needs very Needs
detailed some Is easy to Didn’t
Items explanation explanation understand respond Total
1. Clarity of terminology
(a) Content structure 0 2 3 2 7
(b)Theory 0 2 3 2 7
(c) Mobile technology 1 1 3 2 7
Some are Some may All are
definitely not be not relevant
relevant relevant
2. Relevancy of proposed components
(a) Media elements 0 0 5 2 7
(b)Activity 0 0 5 2 7
(c) Navigation 0 0 5 2 7
(d) Social interaction 0 2 3 2 7
(e) Games 0 1 4 2 7
(f) Presentation style 0 0 5 2 7
(g) Mobile technology 0 1 3 3 7
Not relevant Relevant
3. Relevancy of proposed theories 0 5 2 7
(a) Multimedia learning theory 1 4 2 7
(b) Mindfulness theory 0 5 2 7
(c) Constructivism theory 0 5 2 7
(d) Situated learning theory 0 5 2 7
(e) Experiential learning theory 0 5 2 7
(f) Collaborative learning theory 0 5 2 7
Yes No
4. The connections of all the theories and 3 1 3 7
components are logical
5. The conceptual model is usable for the 4 0 3 7
development of mobile AR for cultural heritage
toward enjoya ble informal learning
6. In overall, the conceptual model is readable 4 0 3 7

heritage) and enjoyment theory. Enjoyable infor- Therefore, this entry shows how a conceptual
mal learning enables visitor not to feel he/she is model, which provides component for enjoyable
learning, but at the same time, he/she is achiev- informal learning at cultural heritage site (Pendit
ing new knowledge (Ariffin 2009). However, the et al. 2014), can be used to design AR applications
existing mobile AR for cultural heritage site lack for cultural heritage sites. A conceptual model
major components in enjoyable informal learn- represents the key concepts and provides accurate,
ing, namely, navigation and user interface, qual- consistent, and complete representation of con-
ity of content, use of questions, and physical cepts (Churchill 2007; Norman 2014). It helps
orientation (Bellotti et al. 2002; Moscardo developer to develop mobile AR that implements
1996). This is critical as it can make the usage enjoyable informal learning at cultural heritage
of mobile AR as interpretive media to help visitor site that enable visitor to learn at cultural heritage
to learn at cultural heritage site is far from being site in enjoyable way. It helps developer to
practical. develop mobile AR that implements enjoyable
Conceptual Model of Mobile Augmented Reality for Cultural Heritage 441

Conceptual Model of Mobile Augmented Reality for Cultural Heritage, Table 2 Feedback from experts
Experts Comments
Expert (a) Divide “media elements” into two types: passive and active content. Active content is the content that
1 (Spain) includes user interaction, such as: activity, social interaction, and games
(b) More details on the theory to understand their relevancy on supporting the content structure
(c) Put elements in mobile technology into categories: “core technologies for AR” and “necessary devices for
AR.” The terms “sensor” and “mobile technology” are also not proper
(d) Add “taking picture” and “interacting with content” in the “Activity” component
C
(e) The term “chat” in social interaction is not proper
(f) Add virtual views in the presentation style
(g) Strength: the conceptual model is feasible and worthwhile to improve the informal learning experience at
cultural heritage site
Weakness: Most of components have been presented in previous works and novel components in informal
learning are not sufficient
(h) Add details for each component in hierarchy or layers than list the individual elements
(i) Validate the conceptual model through user evaluation
Expert (a) Add more detail explanation on content structure and theory
2 (France) (b) Relationship between components in “mobile technology” is not understandable
(c) Add category for different “function” of component and add “Display” component
(d) “Activities,” “navigation,” and “manipulation” can be added as well as “activity” related to media
elements: “see and hear” in activity component
(e) Add “navigation for museum and indoor cultural heritage environment.”
(f) Add shared view with single display in social interaction
(g) Provide other type of games: 3D puzzle
(h) Provide more explanations in “presentation style”
(i) MLT theory and collaborative learning theory should be linked to other elements in the content structure
(j) Mindfulness theory should consider personal cognitive style and traits of visitor that may influence the
social interaction. Constructivism theory, situated learning theory, and experiential learning theory should be
linked to media elements
Expert (a) Consider HCI theory as AR system should be interactive in real time (Azuma 1997)
3 (Spain) (b) Differentiate between audio and sound in media elements. Also distinguish different types of object: static
and dynamic. The elements also can respond to user interaction
(c) Consider providing a complete map of the site and recommended route for the visit in navigation
(d) Clarify the term “chat.” Differentiate between virtual and real (face to face) interaction in social interaction
(e) Clarify the term “separated augmented view” in presentation style
(f) Add category for different function of each mobile technology component, such, hardware, software, and
process
(g) Conceptual model is well presented and logical. However, it misses the term “interaction” as it is a
fundamental part of AR system. The mobile technology component also needs to be better presented
(h) Consider to add validation/evaluation component in content structure on evaluating the learning process
Expert The proposed model is thorough and detail. I expect the outcomes of this model would be good if the learning
4 (Taiwan) activities can be well-arranged
Expert (a) Useful conceptual framework to inform the design. Mindfulness theory is not familiar but others are well
5 (USA) aligned with the teaching and learning methods possible through AR
(b) The missing major element is the outcome of the variable that will be measured
Expert (a) Proposed elements in content structure are too generic and are applicable to any kind of applications
6 (Malaysia) (b) Connections of all theories ad components are somewhat logical
(c) The conceptual model is partly usable to the development of AR for cultural heritage toward enjoyable
informal learning
(d) The conceptual model is too brief
(e) Expand and detail out specifically about AR and cultural heritage
Expert The conceptual model is good but the scope is too wide that it should focus more on enjoyable learning
7 (Malaysia)
442 Conceptual Model of Mobile Augmented Reality for Cultural Heritage

informal learning at cultural heritage site that evaluations, expert review, field study of enjoy-
enable visitor to learn at cultural heritage site in able informal learning, focus group discussion,
enjoyable way. Since the conceptual model has and review of related conceptual model of mobile
been developed (Pendit et al. 2014), it was then AR for cultural heritage site and mobile guide.
evaluated and revised through series of After all those evaluations, the conceptual model

Conceptual Model of Mobile Augmented Reality for Cultural Heritage, Table 3 Findings of field study of
enjoyable informal learning content at cultural heritage site
No. Category Responses
1 Preferences of media Image, animation, and video
2 Text a. Show point by point
b. Provide big size of font
3 3D model Overlay certain part that is lost
4 3D character Represent the noble people in the past
5 Image a. Overlay certain part that is lost
b. Old pictures with year in chronological order
6 Audio a. Provide history of cultural heritage site
b. Provide history of cultural heritage site in storyline
c. Provide history of cultural heritage site in storyline and the narrator has the
same age with visitors
d. The length of audio should be in 3–5 min
7 Sound Provide ambience of heritage site
8 Animation a. Provide history of cultural heritage site with the noble people as the
character in storyline
b. Length of 3D animation is 5–10 min
9 Video a. Provide video of cultural heritage site with noble people as the character
b. Provide video of cultural heritage site with noble people as character in
narrative storyline
c. Provide video of cultural heritage site with the noble people as the
character in the storyline and the narrator should be of same age with visitors
d. Length of video is 5–10 min
10 Preferences to learn based on No, it is not preferable to learn based on interest
interest
11 Navigation a. Show other interesting places around the cultural heritage site
b. Show the route visitor had visited
c. Show the site based on history in chronological order
12 Activity a. Add/edit information
b. Take picture
c. Create notes
13 Games a. Brain games
b. Adventure games
14 Interaction a. Shaking
b. Blowing
c. Rotating
15 Preferences of AR a. Take picture wearing the costume of noble people using AR technology
b. Take picture with the events of the past using AR technology
16 Things make visitors enjoy at a. Relax
cultural heritage site b. Fresh air
17 Other features Music
Conceptual Model of Mobile Augmented Reality for Cultural Heritage 443

was finally finalized which are presented in the after the component of conceptual model was
end of this entry. reviewed based on user requirement of enjoyable
The topics of this entry are consisted in four informal learning content from field study’s result,
sections, which are introduction, methodology it was sent to focus group discussion. The focus
that tells about validating the conceptual model, group discussion validated the model by joining
the revised conceptual model that becomes the seven experts. Then, after it has been evaluated in
result of validation, and finally, at last, the conclu- focus group discussion, an activity called review of C
sion of the study. the conceptual model of mobile AR for cultural
heritage site and review of mobile guide was com-
pleted to act in answering focus group discussion’s
Methodology feedback about the component of conceptual
model. The result from the review was applied in
Basically, the conceptual model has validated revising the conceptual model of mobile AR for
through two steps: expert review and focus group cultural heritage site toward enjoyable informal
discussion. Expert review involved seven experts learning. At the end, the final version of the con-
to validate model based on review form through ceptual model was produced.
email communication. After the expert review,
before continuing with the focus group discussion, Expert Review
the researcher embarked on a field study of enjoy- In validating the conceptual model, expert review
able informal learning content at cultural heritage was conducted. There were seven experts who had
site in order to respond to review of expert on the reviewed the conceptual model with criteria of
novelty of component of conceptual model. Then, teaching AR/human computer interaction/

Conceptual Model of
Mobile Augmented
Reality for Cultural
Heritage,
Fig. 1 Overview of Mobile AR
conceptual model of mobile
AR for cultural heritage site
Hardware
toward enjoyable informal
Process
learning

Content

Enjoyable Informal Learning

Cultural Heritage Site
Navigation & User Interface Design Physical Orientation
Activity
Interaction
Games
Personalization
444 Conceptual Model of Mobile Augmented Reality for Cultural Heritage

Conceptual Model of Mobile Augmented Reality for multimedia with minimum 5 years of experience.
Cultural Heritage, Table 4 Comments from focus The expert was given a copy of proposed concep-
group experts
tual model with the form that contains check list of
No. Comments items and elements of conceptual model. The
Expert 1 a. Include informal learning theory
thoughts from the experts said that conceptual
b. Emphasized on interactivity and
enjoyable informal learning for content model had terminology that was easy to understand
element component (refer to Table 1). This also applies to the proposed
c. Connection in conceptual model could components which are relevant. Overall, experts
not be seen concluded that the conceptual model is usable in
d. Focus more on enjoyable informal
the development of mobile AR for cultural heritage
learning and the main contribution of
conceptual model site toward enjoyable informal learning.
Expert 2 a. Theories are relevant but have to be The next result displayed written comments
presented more clearly which were sent by two experts who did not fill
b. Add informal learning theory up the form and other five experts who filled up
c. Consider “tangible AR” for terms
the form (refer to Table 2). Three experts
d. Add interaction component
addressed the mobile technology component.
Expert 3 a. Connect the mobile AR technology and
content element component They suggested mobile technology should be pro-
b. Add more elements in content element vided by category to divide different functions of
c. Create general model to be used for each element. Furthermore, they also added some
other field related with mobile AR or elements in the content structure component and
enjoyable informal learning or cultural
heritage site changed the terminologies. Some comments were
Expert 4 a. Connection between all components changed in terms of diction but the meaning of
should be improved comment was maintained.
b. Focus more on enjoyable informal The review was taken into consideration in revis-
learning
ing the conceptual model. In addition, one particular
Expert 5 a. Mobile AR technology component
should be improved comment which mentioned about lack of conceptual
b. The terms should be improved model of novel component of enjoyable informal
c. The connection between mobile AR, learning was responded by conducting a field study
theories, and content element is not clear of enjoyable informal learning content that is elab-
d. The conceptual model is not clear orated in the following section.
enough
Expert 6 a. The theory should be reconstructed in
order to be understood Field Study of Enjoyable Informal Learning
b. Consider to combine content element Content at Cultural Heritage Site
with mobile AR technology In response to the feedback from expert, a field
c. Some terms are not clear, such as, study was conducted at Lembah Bujang Archae-
“provided for each site”
ological Site, Kedah. The purpose is to gather user
d. Combine all components with mobile
AR technology requirements about content of enjoyable informal
e. Differentiate the uniqueness of mobile learning at cultural heritage site and to define
AR technology component from the novel components of proposed conceptual
existing one
model. The questions were taken from the com-
Expert 7 a. Add one element special for cultural
heritage site in content element
ponent of conceptual model, the literature, and
b. Connection between theory and content expert’s feedback. In total, there were five partic-
element seems logical ipants from targeted ten participants from 17 to
c. The term “registration” is not clear 49 years old who participated in the study. The
Expert 8 Create general model consists of three numbers of participants are relatively enough as
main topics: mobile AR, enjoyable
informal learning, and cultural heritage
the researcher has attained similar answer from all
site participants (Creswell 2012).The questions are in
Malay and English language. They are related to
Conceptual Model of Mobile Augmented Reality for Cultural Heritage 445

Conceptual Model of Mobile Augmented Reality for Cultural Heritage, Table 5 Component and element of
conceptual model
Component Description Element Implementation
Hardware Physical components needed for Handheld devices Smart phones
developing mobile AR for cultural Tablet
heritage toward enjoyable informal
learning
Process Steps or actions needed to develop Reconstruction Reconstruct wall of A Famosa into
C
mobile AR for cultural heritage site 3D model by using 3DsMax
toward enjoyable informal learning Registration Align virtual object of wall of
A Famosa in the real world by
tracking user’s position and
orientation of user’s view
Tracking Find location of user by using
sensor-based tracking, vision-
based tracking, and hybrid-based
tracking in A Famosa
Rendering Generate virtual object/scene and
present it to the real environment of
A Famosa
Interaction Shaking, blowing, rotating,
leaning, and nodding the mobile
phone and user’s head to retrieve
information about A Famosa
Content A set of media representation 3D model Reconstruct the lost wall of
which consists of criteria that can Overlay certain part that A Famosa to 3D model
be a guideline to provide enjoyable is lost
informal learning content at 3D character Alfonso d’Alburqueque, Captain
cultural heritage site Represent noble people of Portuguese, as 3D virtual guide
in the past and act as at A Famosa
virtual guide
Text Profile of cultural heritage site that
Provide description in contains information about history
point by point and background information of
cultural heritage site
Image Old picture of the wall of
Overlay certain part that A Famosa, old picture of
is lost A Famosa, overlay picture of
Provide old picture about Alfonso d’Alburqueque, and old
the site, noble people, picture of war between Portuguese
and events and Dutch at A Famosa
Audio Story that tells how A Famosa was
Recorded audio built by Portuguese
presented by narrator
who has the same age
with visitor
In-depth and special
information in 3D
animation and video
Provide audio with
maximum duration in
5 min
Sound Sound of bomb during the war
Provide ambience of the Conversation between inhabitants
site in the past that can Sound of captain’s car
help visitor imagine how
the site was
(continued)
446 Conceptual Model of Mobile Augmented Reality for Cultural Heritage

Conceptual Model of Mobile Augmented Reality for Cultural Heritage, Table 5 (continued)
Component Description Element Implementation
Animation Story about how the war between
Provide in-depth and Portuguese and Dutch long time
special information in ago happened in A Famosa
storyline
Use noble people as
character
Provide animation with
maximum duration in
10 min
Video Story about how the war between
Provide in-depth and Portuguese and Dutch happened in
special information in A Famosa long time ago
storyline
Use noble people as
character
Provide video with
maximum duration in
10 min
Push content All elements of content
The type of content that
appears automatically
when visitor reaches
certain area
Pull content All elements of content
The type of content that
should be retrieved
Navigation Navigation and user interface Provide thematic path Provide theme of cultural heritage
and user design that helps visitor to learn in site based on colonialism era
interface enjoyable way at cultural heritage (Portuguese colonialism, Dutch
design site colonialism, British colonialism)
Provide layered Provide description about the
information structure and construction about
A Famosa and continued by history
of formation of A Famosa
Provide one-tap access Provide “home” button for
for frequent menu accessing the menu
Provide one-handed Provide one-handed control for the
control navigation
Provide clue for scene Provide clue in balloon to tell there
with augmented content is augmented scene in the area
Provide shaped button Provide play button in its shape in
order to make visitor easy to access
Provide quick button to Provide “home” button to the
go to main menu information menu to go to
homepage quickly
Provide big size of font Provide big size of font on the
interface
Provide appropriate size Provide three-fourth content size
of content for the page
Provide enough contrast Provide white background for
between text and black text
background
(continued)
Conceptual Model of Mobile Augmented Reality for Cultural Heritage 447

Conceptual Model of Mobile Augmented Reality for Cultural Heritage, Table 5 (continued)
Component Description Element Implementation
Interactivity Activity: a set of activities that can Take picture Provide option to take picture of
trigger the whole learning process A Famosa
at cultural heritage site by Share information Provide option to share information
integrating the visitors, learning of A Famosa to social media
material, and learning environment (Facebook and Twitter)
C
Edit/add information Provide option to add or edit
information of A Famosa
Create notes Provide option to create notes
about experience of visiting
A Famosa
Save information Provide option to save the
information of A Famosa to
personal device
Interaction Shaking Enable visitor to shake their phone
in order to retrieve the information
about A Famosa
Blowing Enable visitor to blow the wall of
A Famosa to retrieve the
information
Rotating Enable visitor to rotate their phone
to left or right in order to turn the
3D object to the preferred direction
Leaning Enable visitor to lean or move the
3D object to left or right
Nodding Enable visitor to move the 3D
object up and down by nodding
his/her head
Features Personalization: a set of options Historical period Enable visitor to select the cultural
that can be chosen by visitors in heritage site in the range of
order to display the right content to 1819–1900 or 1901–present
fulfill their needs Interest Enable visitor to choose the
cultural heritage site to be visited
based on personal interest
Range of distance Enable visitor to select the cultural
heritage site within the range
(0–5 km, 6–10 km, and 11–15 km)
Language Enable visitor to choose the
language based on their
preferences
Games: the type of games that help Adventure games Treasure hunt games about history
visitor to refresh, stimulate, and (treasure hunt) of A Famosa
make them understand the history Multiple choice quiz Multiple choice question about
history of A Famosa
Physical orientation: is a set of Showing the surrounding Show other cultural heritage site
functions to guide visitor while interested places near A Famosa
learning at cultural heritage site Showing recommended Show the route from airport to
orientation route to the site A Famosa, enable the visitor to
search location of A Famosa
Allowing direction Allow visitor to search for route
inquiry
(continued)
448 Conceptual Model of Mobile Augmented Reality for Cultural Heritage

Conceptual Model of Mobile Augmented Reality for Cultural Heritage, Table 5 (continued)
Component Description Element Implementation
Showing direction with Show direction to A Famosa with
virtual arrows overlay on virtual arrows overlay on real path
real path from Saint Paul
Showing map of the site Show the map of A Famosa and
and location of visitors location of visitor within A Famosa
within the site
Showing provided Show the provided content in
content A Famosa
Showing visited route Show the visited route at Melaka
Heritage Site
Showing the current Show the current position of visitor
position

the content that is appropriate for learning in The Proposed Conceptual Model of Mobile AR
enjoyable way at cultural heritage site, including: for Cultural Heritage Site Toward Enjoyable
types of media, navigation, games, activity, and Informal Learning
interaction. In total, there were nineteen ques- After all the validation process, the conceptual
tions in total with fifteen multiple choice ques- model was comprised to be finalized. There were
tions and two open-ended questions. The six components and twenty-nine elements that is
researcher did the field study by approaching included in the conceptual model based on the
random visitors and interview them about the validation (refer to Fig. 2). The explanation
questions while at the same time demonstrating about component and element of conceptual
mobile AR feature mentioned in the question. model is provided in Table 5.
The responses from participants are provided as The conceptual model focuses on enjoyable
follow (Table 3). informal learning. However, it integrates existing
three main fields: mobile AR, enjoyable informal
learning, and cultural heritage site (refer to Fig. 2).
Focus Group Discussion This becomes the uniqueness of conceptual model
Focus group discussion was conducted to evalu- that is not implemented in the existing conceptual
ate the conceptual model. There were seven model. The proposed conceptual model is
experts who had participated in the focus group. displayed.
The experts were all from Malaysia, and they
should be those have been teaching AR/HCI/Mul-
timedia/Media Studies with a minimum of 5 years
of experience. Conclusion and Discussion
The focus group was started by presenta-
tion about the conceptual model. Then experts This study has produced a conceptual model of
were asked to review the conceptual model mobile AR for cultural heritage site towards
based on the criteria in the review form enjoyable informal learning. The model com-
(refer to Fig. 1). During the discussion, prises three structures, six components and
experts were also allowed to ask questions. twenty nine elements. The structures represent
The process of discussion lasted for one and the main topic of conceptual model, which are,
half hour. It was a beneficial discussion. The mobile AR, enjoyable informal learning, and
result is provided in Table 4. cultural heritage. The component presents the
Conceptual Model of Mobile Augmented Reality for Cultural Heritage 449

Conceptual Model of Mobile Augmented Reality for mobile AR for cultural heritage site toward enjoyable
Cultural Heritage, Fig. 2 Revised conceptual model of informal learning
450 Conduction

main component that are appropriate for Ciurea, C., Zamfiroiu, A., Grosu, A.: Implementing mobile
conducting enjoyable informal learning at cul- virtual ex-hibition to increase cultural heritage visibil-
ity. Inf. Econ. 18(2), 24–31 (2014)
tural heritage site, which are, content, navigation Creswell, J.W.: Educational Research: Planning,
and user interface design, interactivity, features, Conducting, and Evaluating Quantitative and Qualita-
hardware, and process. Lastly, the elements of tive Research, 4th edn. Phi Learning Pvt. Ltd, New
content component are the backbone in realizing Delhi (2012)
Damala, A.: Interaction design and evaluation of mobile
enjoyable informal learning. guides for the museum visit: a case study in multimedia
The conceptual model aims to help developer and mobile augmented reality. Doctoral dissertation,
in developing mobile AR for cultural heritage site Ecole Doctorale EDITE, Paris. https://tel.archives-
toward enjoyable informal learning. It can be ouvertes.fr/tel-00526141/document (2009)
iTACITUS. itacitus.org (2007)
implemented to any platform, to a whole domain Kim, J., Park, C.: Development of mobile AR tour appli-
(mobile AR for cultural heritage site toward cation for the national palace museum of Korea. In:
enjoyable informal learning), and to each domain 2011 International Conference on Virtual and Mixed
(mobile AR, enjoyable informal learning, and cul- Reality: New Trends Volume, Part 1, pp. 55–60.
Springer, Berlin/Heidelberg/Orlando. https://doi.org/
tural heritage site). Furthermore the conceptual 10.1007/978-3-642-22021-0_7 (2011)
model is also hoped to be used and to make a Moorhouse, N., Tom Dieck, M., Jung, T.: Augmented
lasting impact onto the existence of cultural heri- reality to enhance the learning experience in cultural
tage site. heritage tourism: an experiential learning cycle per-
spective. eRev. Tour. Res. 8, 1–5 (2017)
Moscardo, G.: Mindful visitors: heritage and tourism. Ann.
Tour. Res. 23(2), 376–397 (1996). https://doi.org/10.
References 1016/0160-7383(95)00068-2
Norman, D.A.: Some observations on mental model. In:
Gentner, D., Stevens, A.L. (eds.) Mental Models, p. 7.
Angelopoulou, A., Economou, D., Bouki, V., Jin, L., Psychology Press, New York (2014)
Pritchard, C., Kolyda, F.: Mobile augmented reality Pendit, U.C., Zaibon, S.B., Abu Bakar, J.A.: Mobile aug-
for cultural heritage. In: Venkatasubramanian, N., mented reality for enjoyable informal learning in cul-
Getov, V., Steglich, S. (eds.) Mobile Wireless Mid- tural heritage site. Int. J. Comput. Appl. 92(14), 19–26
dleware, Operating Systems and Applications, (2014). http://research.ijcaonline.org/volume92/num
pp. 15–22. Springer, Berlin/Heidelberg (2012) ber14/pxc3895286.pdf
Ariffin, A.M.: Conceptual Design of Reality Learning Seo, B., Kim, K., Park, J.-I.: Augmented reality-based
Media (RLM) Model Based on Entertaining and Fun on-site tour guide: a study in Gyeongbokgung. In:
Constructs. Universiti Utara Malaysia, Kedah (2009). Koch, R., Huang, F. (eds.) The 2010 International Con-
http://etd.uum.edu.my/1521/2/1.Ariffin_Abdul_ ference on Computer Vision – Volume Part 2,
Mutalib.pdf pp. 276–285. Springer-Verlag, Berlin/Heidelberg
Armanno, G., Bottino, A., Martina, A.: SkyLineDroid: an (2011). https://doi.org/10.1007/978-3-642-22819-3_28
outdoor mobile augmented reality application for virtual Techcooltour. http://www.techcooltour.com/en/ (2013)
heritage. International Conference on Cultural Heritage Tussyadiah, I., Jung, T.H., Tom Dieck, M.: Embodiment of
and Tourism, pp. 91–96. http://www.wseas.us/e-library/ wearable augmented reality technology in tourism
conferences/2012/CambridgeUK/CUMNUPEM/ experiences. J. Travel Res, pp 1–37 (2017). ISSN
CUMNUPEM-14.pdf (2012) 0047-2875
Azuma, R.T.: A survey of augmented reality. Presence. Vlahakis, V., Ioannidis, N., Karigiannis, J., Tsotros, M.,
6(4), 355–385 (1997). http://pdf.thepdfportal.net/ Gounaris, M.: Virtual reality and information technol-
PDFFiles/26651.pdf ogy for archaeological site promotion. In: Proceedings
Bellotti, F., Berta, R., Gloria, A.D., Margarone, M.: User of the Fifth International Conference on Business Infor-
testing a hypermedia tour guide. IEEE Pervasive mation Systems (BIS02), Poznan (2002)
Comput. 1(2), 33–41 (2002). https://doi.org/10.1109/
MPRV.2002.1012335
Chang, Y.-L., Hou, H.-T., Pan, C.-Y., Sung, Y.-T., Chang,
K.-E.: Apply an augmented reality in a mobile guid-
ance to increase sense of place for heritage places.
Educ. Technol. Soc. 18(2), 166–178 (2015) Conduction
Churchill, D.: Towards a useful classification of learning
objects. Educ. Technol. Res. Dev. 55(5), 479–497
(2007). https://doi.org/10.1007/s11423-006-9000-y ▶ Game Development Leadership Tips
Constrained Edges and Delaunay Triangulation 451

Conform

▶ Abstraction and Stylized Design in 3D Ani-

mated Films: Extrapolation of 2D Animation
Design
C

Constrained Edges and

Delaunay Triangulation
Constrained Edges and Delaunay Triangulation,
1 2
Fig. 1 Delaunay Triangulation
Simena Dinas and Héctor J. Martínez
1
Facultad de Ingeniería, Universidad Santiago de
Cali, Cali, Colombia
2
Universidad del Valle, Cali, Colombia

Synonyms

Delaunay tessellations; Delaunay Triangulation;

Delone tessellations

Definition Constrained Edges and Delaunay Triangulation,

Fig. 2 Constrained Delaunay Triangulation
A Delaunay Triangulation is a triangle net in
which every triangle satisfies the Delaunay con-
dition: the circumcircle of each triangle includes
only the vertices of the triangle. In other words,
the circumcircle does not contain any vertex of
other triangles (van Kreveld 2014). Constrained
edges in a Delaunay Triangulation have been used
as fixed edges into the triangulation. Constrained
Delaunay Triangulation (CDT) and Delaunay
Constrained Triangulation (DCT) are extensions
of Delaunay Triangulation including constraints
widely studied. The Fig. 1 represents a Delaunay
Triangulation and the Figs. 2 and 3 depicts a CDT Constrained Edges and Delaunay Triangulation,
and a DCT, respectively. Fig. 3 Delaunay Constrained Triangulation

inserting constraints into a Delaunay Triangula-

Introduction tion two different methods: stable and unstable.
Stable methods begin with a Delaunay Triangula-
An algorithm with optimal time to compute CDT tion, inserts a constrained edge, and produces
was firstly proposed by Chew (1987), whereas another Delaunay Triangulation. Contrarily, an
Rognant et al. (1999) defined for the process of unstable method does not produce a Delaunay
452 Constrained Edges and Delaunay Triangulation

Triangulation, for instance, CDT is an unstable Constrained Delaunay Triangulation

method. Some CDT algorithms were documented
by Agarwal et al. (2005); Shewchuk and CDT were introduced independently as General-
Brown (2015). ized Delaunay Triangulation (Lee and Lin 1986)
and as Obstacle Triangulation by Chew (1987).
Constrained Delaunay Triangulation Restricted Delaunay Triangulation was the name
Given a set of n1 vertices V (n1) and a set of m used for others (Anglada 1997). A triangle is
constrained edges E(m), whose endpoints are Constrained Delaunay if its interior does not inter-
V (n2), a CDT is a net of nonoverlapping triangles sect any input segment, and its circumcircle does
whose set of vertices is V (n) ¼ V (n1) U V (n2), all not enclose vertices visible from the interior of the
constrained edges E(m) are included as edges of triangle. Nevertheless, an edge is Constrained
the triangulation and it is as Delaunay as possible. Delaunay if it does not cross any input segment,
(V (n1) may be Ø.) The strong lines in the Fig. 2 and it has a circumcircle which does not enclose
represent the constrained edges. vertices visible from the interior of the edge.
Several authors have worked on CDT; for
Delaunay Constrained Triangulation instance, Agarwal et al. (2005) worked on the
Given a set of n1 vertices V (n1) and a set of m efficiency for input-output data based on the con-
constrained edges E(m), whose endpoints are struction process to optimize the use of the mem-
V (n2), a DCT is a net of nonoverlapping triangles ory. Shewchuk (2008) explored the weighted
whose set of vertices is V (n) ¼ V (n1) U V (n2) U V Constrained Delaunay Tetrahedralizations
(n3), where V (n3) is a set of n3 virtual vertices, all whereas Dinas and Bañón (2012) proposed to
constrained edges E(m) are included as edges of the take data from images; the construction of
triangulation, and it is a Delaunay Triangulation. Delaunay Triangulation uses a cloud of points
(V (n1) and V (n3) may be empty.) Only the strong from images and a set of edges E captured from
red lines are included in a DCT as constrained images are the constrained edges for Constrained
edges, then the strong black lines are ignored. Delaunay Triangulations. Dinas and Martínez
Delaunay Triangulation can use fixed edges, (2019) worked on the constrained edges and
which are usually called constrained edges or Kinetic Delaunay Triangulation as a collision
briefly constraints. Delaunay Triangulation must detection approach. In contrast, Domiter and
be called CDT (for more details, see Chew (1987); Zalik (2008) worked on accelerating the location
van Kreveld (2014)). In a CDT, the priority is the process for inserting new vertices, whereas Nam
constrained edges; it means, the triangulation et al. (2009) worked on accelerating the algorithm
must include the constrained edges. In other and Eder et al. (2018) worked on the para-
words, the most important is preserve constrained llelization of the polygons with holes in
edges rather than the Delaunay structure. Conse-
quently, the result is a triangulation including all
the constrained edges even though some triangles
do not pass the Delaunay condition (Chew 1987).
In contrast, for DCT the most important is to
conserve the Delaunay nature rather than guaran-
tee the constrained edges (Rognant et al. 1999;
Dinas and Bañón 2012). Thus, the result is a
Delaunay Triangulation with as many as possible
constrained edges (some constrained edges can be
ignored or excluded of the final triangulation).
However, there are a several ways to preserve
the constrained edges and the Delaunay structure,
Constrained Edges and Delaunay Triangulation,
but it requires virtual vertices. In the Fig. 4, the Fig. 4 Delaunay Constrained Triangulation and virtual
virtual vertices are enclosed in red circles. vertices
Constrained Edges and Delaunay Triangulation 453

Constrained Delaunay. Additionally, refinement the Shewchuk and Brown (2015) is close related
methods have been used to improve CDT Algo- to the Chew’s implementation of deleting a vertex
rithms (Engwirda and Ivers 2015). in linear time.

Divide and Conquer

Constrained Delaunay Triangulation Chew (1987) proposed a divide and conquer
Algorithms (CDT) algorithm for CDT. The set of vertices V (n) is C
divided into vertical rectangles, which contains
The input is a set of constrained edges usually exactly one vertex. Additionally, there are cross-
called a Planar Straight-Line Graph (PSLG) and ing lines inside the rectangle that represent the
the set of vertices of the triangulation. The output constrained edges. Thus, the triangulation is
is a CDT where every edge is locally Delaunay constructed by using the crossing lines and the
except for constrained edges. constrained edges. Finally, the strips are merged
to complete the CDT; it takes O(n log n) time
Incremental Construction (Chew 1987).
Anglada’s algorithm (Anglada 1997) starts with a
Delaunay Triangulation that is constructed using Plane Sweep
the set of vertices V (n), and then localizes the Domiter and Zalik (2008) proposed an algorithm
constrained edge in the triangulation. focused on the main bottleneck of the algorithm:
locating the triangle in which a new vertex falls;
Incremental Insertion thus, the cost of inserting an edge into a CDT
Based on Anglada’s algorithm (Anglada 1997), decreases. The worst time complexity of the pro-
the proposal of Nam et al. (2009) accelerates the posed algorithm was O(n2).
point location by using Skvortsov’s algorithm and
a uniform grid.
Delaunay Constrained Triangulation
Randomized Incremental Construction (DCT)
A randomized implementation based on
Anglada’s algorithm (Anglada 1997) was pro- DCT produces stable method because the final
posed by Agarwal et al. (2005). This triangulation guarantees the Delaunay structure
approach includes in the Delaunay Triangula- and respects constrained edges, at the same time.
tion the endpoints of the constrained edges, it Stable methods can be constructed by inserting
means, the endpoints of the constrained new vertices in the triangulation referred as Vir-
edges are used to construct the Delaunay tual Vertices. It is also known as Conforming
Triangulation. Delaunay Triangulation; its edges are all locally
Delaunay. As DCT is a Delaunay Triangulation,
Randomized Incremental Segment Insertions the construction time is related to the Delaunay
Typical implementations for segment insertion time, but the number of virtual vertices depends
take O(kn2), where n is the number of input verti- on the selected method. Additionally, Shewchuk
ces and k is the number of input segments. (2008) reported Conforming Delaunay Triangula-
Inserting a constrained edge into a CDT takes tion. The proposed algorithms accept in the input
O(n log n + n log2 k) (Agarwal et al. 2005). This small angles, and then new vertices are included
algorithm deletes the edges intercepted by the to increase the quality by eliminating small
constrained edge, inserts the constrained edge in angles. In the worst case, the input needs!(n2) to
the cavity, and retriangulates it. An improved O(n2.5).
algorithm proposed by Shewchuk and Brown Rognant et al. (1999) defined stable and
(2015) decreases the complexity to O(n log2 k) unstable methods according to the result: they
by computing the segment location procedure are stable if they produce a Delaunay
before the segment insertion procedure. Finally, Triangulation.
454 Constrained Edges and Delaunay Triangulation

Virtual Vertices Triangulation. An advantage of this triangulation

is that their construction does not require addi-
A virtual vertex is a vertex, which does not belong tional vertices and the disadvantage of this trian-
to the initial set of vertices V (n) of the triangula- gulation is that it is not always a Delaunay
tion; it is created to divide constrained edges into Triangulation. To guarantee both, the constrained
smaller pieces with the purpose of creating a DCT. edges and the Delaunay Triangulation, it is possi-
Several authors have been working on inserting ble to convert a CDT into a DCT by inserting new
vertices to the triangulations by different methods; vertices inside the triangulation. In the next sec-
for instance, the Steiner Points by Engwirda and tions are briefly documented the following
Ivers (2015). Virtual vertices use the constrained methods: densification, dichotomy, orthogonal
edges; nevertheless, its creation or selection projection, intersection, bisection, and trisection
depends on the applications. For instance, virtual methods.
vertices can be created by the intersection between
a pair of constrained edges or between a Densification
constrained edge and a Delaunay edge. According Densification method is constructed by a set of
to the kind of intersection, it can be classified into steps that should be applied to each constrained
three groups: (i) An X intersection, it is necessary edge inside the triangulation (see Fig. 5). The first
to divide two constrained edges into four new step is to find in the set of vertices V (n) the nearest
constrained edges, where each constrained edge vertex to the constrained edge. After, to calculate
has a unique vertex and one virtual vertex. (ii) A the length d from this vertex to the constrained
T intersection, it happens when an endpoint of a edge; d is used to divide the constrained edge into
constrained edge is involved in the intersection. It m small constrained edges with length of at least
requires to split only one edge into two edges, but d. That is to say, at least m – 1 of the m new small
it is not required to create a new virtual vertex, and constraints have size d (Faugeras et al. 1990);
(iii) an L intersection, which does not require however, it can be modified to include
virtual vertices. m segments with equal length. This method can
produce a big number of virtual vertices; however,
the main advantage of this method is that it does
Delaunay Constrained Triangulation not require the inCircle test for the new the
Methods triangles.

In this section, the set of methods that can be used Dichotomy

to construct a DCT is explained. A DCT is created by dividing each constrained
edge into small constrained edges; the size of each
Constrained Delaunay Triangulation segment can be defined by the author or can be
CDT respects constrained edges, and the resulting randomly selected, as is shown in Fig. 6. The
triangulation is as close as possible to a Delaunay densification method guarantees the existence of
Triangulation; it means, there is not guaranteed the right set of virtual vertices of the dichotomy
that the resulting triangulation is a Delaunay method. Compared to the densification method, it

Constrained Edges and

Delaunay Triangulation,
Fig. 5 Densification
method for Delaunay
Constrained Triangulation
Constrained Edges and Delaunay Triangulation 455

Constrained Edges and

Delaunay Triangulation,
Fig. 6 Dichotomy method
for Delaunay Constrained
Triangulation

C
Constrained Edges and
Delaunay Triangulation,
Fig. 7 Orthogonal
projection method for
Delaunay Constrained
Triangulation

Constrained Edges and

Delaunay Triangulation,
Fig. 8 Intersection method
for Delaunay Constrained
Triangulation

is equally reliable; it is computationally less Intersection

expensive and requires less additional vertices. The result of overlapping a Delaunay Triangulation
The advantage of this method is that it defines and be constrained edges is shown in Fig. 8; the
the constrained edge partition as simple as possi- intersection between the constrained edges create
ble and it does not produce more virtual vertices the virtual vertices. This method can be used to
than necessary for the triangulation to be insert constrained edges into a Delaunay Triangu-
Delaunay; the disadvantage is that it requires the lation. When the Delaunay Triangulation is
inCircle test for the new triangles. Rognant et al. unknown, constructing it can be more expensive
(1999) reported a brief explanation. than using another method; however, it is reliable.
A fast way to create the intersection vertices is to
Orthogonal Projection find a pair of vertices on opposite sides that sur-
In Fig. 7, the orthogonal method is done for each round the constrained edge, and then calculate the
vertex located around a constrained edge. Each intersection vertex between the constrained edge
of these vertices is orthogonally projected on the and the segment formed by the pairs of vertices
constrained edge; consequently, it produces an surrounding the constrained edge. Finally, com-
edge that is orthogonal to the constrained edge. plete the triangulation by connecting the virtual
Thus, each virtual vertex is the orthogonal pro- vertices with the surrounding vertices (Rognant
jection of the vertex on the constrained edge. et al. 1999). This approach is based on the
This method is reliable; the orthogonal projec- Anglada’s algorithm to insert segments in a CDT
tion is not a complex procedure; additionally, the (Anglada 1997).
number of new virtual vertices depends on the
number of vertices that are around to the Bisection
constrained edge (Rognant et al. 1999). The Bisection method is based on a divide and conquer
shortcoming of this method is that the cost of approach depicted in Fig. 9: it is simple to imple-
calculating the virtual vertices is ment, and the inCircle test must verify the result.
computationally high. The first step is to bisect the constrained edge and
456 Constrained Edges and Delaunay Triangulation

Constrained Edges and

Delaunay Triangulation,
Fig. 9 Bisection method
for Delaunay Constrained
Triangulation

Constrained Edges and

Delaunay Triangulation,
Fig. 10 Trisection method
for Delaunay Constrained
Triangulation

locate a virtual vertex. Then connect the virtual shortcoming of this method is the necessity of
vertex with the set of vertices surrounding the evaluating the result after each division. Other
constrained edge. Finally, evaluate the triangles methods as densification and intersection methods
by the inCircle test, if any triangle does not satisfy produce correct DCT and do not require evaluation;
the circumcircle condition, then bisect the piece of however, their procedure to locate the virtual ver-
the constrained edge again until each triangle sat- tices is not as straightforward as the used for bisec-
isfies the circumcircle condition. The advantage of tion method. The number of virtual vertices
this method is that it defines a simple constrained produced by the bisection and trisection are
edge partition and it does not produce more virtual expected to have similar behavior, even though
vertices than necessary for the triangulation to be the bisection method requires more levels of parti-
Delaunay. The shortcoming of this method is the tion, but the trisection method produces more vir-
necessity of evaluating the result after each divi- tual vertices per partition.
sion. Other methods as densification and intersec-
tion methods produce correct DCT and do not
require evaluation; however, their procedure to Conclusion and Discussion
locate the virtual vertices is not as straightforward
as the used for bisection method. Delaunay Constrained Triangulation and
Constrained Delaunay Triangulation have
Trisection remarkable difference between them. Both trian-
This method is based on a divide and conquer gulations can be used to represent constrained
approach (see Fig. 10) it is simple to implement, edges in a triangulation; however, the use depends
and the inCircle test must verify the result. The first on the purpose. While CDT is flexible with
step is to trisect the constrained edge and locate two Delaunay Triangulation, it is much stricter with
virtual vertices. Then connect the virtual vertices constrained edges. In contrast, DCT is flexible
with the set of vertices surrounding the constrained with constrained edges but stricter with a result
edge. Finally, evaluate the triangles by the inCircle that passes the conditions of Delaunay Triangula-
test, if any triangle does not satisfy the circumcircle tion. The inclusion of artificial vertices called
condition, then trisect the piece of the constrained virtual vertices can help to guarantee both: the
edge again until each triangle satisfies the circum- Delaunay structure and de constrained edges.
circle condition. The advantage of this method is Finally, reviewed methods are based on virtual
that it defines a simple constrained edge partition vertices: densification, orthogonal projection,
and it does not produce more virtual vertices than dichotomy, and intersection, and the proposed
necessary for the triangulation to be Delaunay. The methods: bisection and trisection.
Constructing Game Agents Through Simulated Evolution 457

Cross-References International Conference on Information Visualisation,

pp. 147–152. IEEE Computer Society, Washington,
DC (1999). https://doi.org/10.1109/IV.1999.781551
▶ Delaunay Triangulation Shewchuk, J.R.: General-dimensional constrained
▶ Modeling and Mesh Processing for Games Delaunay and constrained regular triangulations, I:
▶ Teaching Computer Graphics by Application Combinatorial properties. Discret. Comput. Geom. 39
▶ UV Map Generation on Triangular Mesh (1–3), 580–637 (2008). https://doi.org/10.1007/
s00454-008-9060-3
Shewchuk, J.R., Brown, B.C.: Fast segment insertion and
C
incremental construction of constrained Delaunay tri-
References angulations. Comput. Geom. 48, 554–574 (2015).
https://doi.org/10.1016/j.comgeo.2015.04.006
Agarwal, P.K., Arge, L., Yi, K.: I/o-efficient construction van Kreveld, M.: Delaunay Triangulation and Tetra-
of constrained Delaunay triangulations. In: Brodal, G., hedralization. Department of Information and Comput-
Leonardi, S. (eds.) Algorithms – ESA 2005, vol- ing Sciences, Faculty of Science, Utrecht University
ume 3669 of Lecture Notes in Computer Science, (2014)
pp. 355–366. Springer, Berlin/Heidelberg (2005).
https://doi.org/10.1007/11561071_33
Anglada, M.V.: An improved incremental algorithm for
constructing restricted Delaunay triangulations. Comput. Constructing Game Agents
Graph. 21(2), 215–223 (1997). https://doi.org/10.1016/
S0097-8493(96)00085-4. Graphics Hardware
Through Simulated Evolution
Chew, L.P.: Constrained Delaunay triangulations. In: Pro-
ceedings of the Third Annual Symposium on Computa- Jacob Schrum1 and Risto Miikkulainen2
tional Geometry, SCG 1987, pp. 215–222. ACM, 1
Department of Mathematics and Computer
New York (1987). https://doi.org/10.1145/41958.41981
Dinas, S., Bañón, J.M.: Representing extracted edges from
Science, Southwestern University, Georgetown,
images by using constrained Delaunay triangulation. TX, USA
2
In: Proceedings of IADIS International Conference Department of Computer Science, University of
Applied Computing, pp. 1–5 (2012) Texas at Austin, Austin, TX, USA
Dinas, S., Martínez, H.J.: Constrained edges on kinetic
Delaunay triangulation. Revista Ibérica de Sistemas e
Tecnologias de Informação. 22, 174–186 (2019)
Domiter, V., Zalik, B.: Sweep-line algorithm for Synonyms
constrained Delaunay triangulation. Int. J. Geogr. Inf.
Sci. 22(4), 449–462 (2008). https://doi.org/10.1080/
13658810701492241
Evolutionary agent design; Evolutionary algo-
Eder, G., Held, M., Palfrader, P.: Parallelized ear clipping rithms; Evolutionary computation; Evolutionary
for the triangulation and constrained Delaunay triangu- machine learning; Neuroevolution
lation of polygons. Comput. Geom. 73, 15–23 (2018).
https://doi.org/10.1016/j.comgeo.2018.01.004
Engwirda, D., Ivers, D.: Size-optimal Steiner points for
Delaunay-refinement on curved surfaces. CoRR, Definition
abs/1501.04002 (2015)
Faugeras, O. D., Le Bras-Mehlman, E., Boissonnat, J.-D.: Construction of game agents though simulated
Representing stereo data with the delaunay triangulation.
Artif. Intell. 44(1–2), 41–87 (1990). https://doi.org/10.
evolution is the use of algorithms that model the
1016/0004-3702(90)90098-K biological of process of evolution to develop the
Lee, D.-T., Lin, A.K.: Generalized Delaunay triangulation behavior and/or morphology of game agents.
for planar graphs. Discret. Comput. Geom. 1, 201–217
(1986). https://doi.org/10.1007/BF02187695
Nam, N.M., Kiem, H.V., Nam, N.V.: A fast algorithm for
constructing constrained Delaunay triangulation. In: Introduction
International Conference on Computing and Commu-
nication Technologies, RIVF 1909, pp. 1–4 (2009). Computer game worlds are often inhabited by
https://doi.org/10.1109/RIVF.2009.5174607
Rognant, L., Chassery, J.-M., Goze, S., Planès, J.-G.: The
numerous artificial agents, which may be helpful,
Delaunay constrained triangulation: The Delaunay sta- neutral, or hostile toward the player or players.
ble algorithms. In: Proceedings of the 1999 Common approaches for defining the behavior of
458 Constructing Game Agents Through Simulated Evolution

such agents include rule-based scripts and finite performance. Sometimes a certain number of top
state machines (Buckland 2005). However, agent performers are selected directly (a technique
behavior can also be generated automatically known as elitism), but selection is generally a
using evolutionary computation (EC; Eiben and random process that merely favors high-
Smith 2003). EC is a machine-learning technique performing individuals, while still allowing
that can be applied to sequential decision-making some poor-performing, but lucky, individuals to
problems with large and partially observable state be chosen. This random element is one way of
spaces, like video games. maintaining diversity in the evolving population
EC can create individual agents or teams, and and is generally important to the long-term suc-
these agents can be opponents or companions of cess of evolution.
human players. Agents can also be evolved to play In order for evolution to progress, some of the
games as a human would, in order to test the slots in the new population must be filled by
efficacy of EC techniques. EC can even create results of recombination or mutation. Recombina-
game artifacts besides agents, such as weapons. tion creates a new solution to the problem by
The reason EC is so flexible is that it requires little combining components of solutions that were
domain knowledge compared to traditional selected from the old population. Generally, two
approaches. It is also capable of discovering sur- solutions from the old population, called parents,
prising and effective behavior that a human expert are selected and recombined to create a new solu-
would not think to program. If applied intelli- tion, a child or offspring, via simulated crossover,
gently, this approach can even adapt to humans which models the process of genetic crossover
in a manner that keeps providing interesting and that is a major benefit in biological sexual repro-
novel experiences for players. This article focuses duction. In addition, some of these offspring
mostly on discovering effective opponent behav- undergo mutation before joining the new
ior (since that is the focus of most research), population.
although examples of other applications are also Mutation operations are applied with low prob-
given when appropriate. ability and generally result in small changes to a
candidate solution. It is also possible, and com-
mon, for mutation to be applied directly to mem-
Evolutionary Computation bers of the old population to generate new
solutions, which can also fill slots in the new
EC models the process of Darwinian evolution by population. Mutation without recombination
natural selection (Darwin 1859) for the purpose of models asexual reproduction.
generating solutions to difficult embedded prob- The new population of candidate solutions is
lems. Initially, a random collection of candidate labelled the next generation of the evolutionary
solutions, called the population, is generated and process. The new population now also undergoes
evaluated in a task within some environment. evaluation and is subject to selection, recombina-
Because of randomness in how the population tion, and mutation, which leads to yet another
was generated, there will be variation in the per- generation, and so on. Because recombination
formance of different candidate solutions. At this and mutation keep creating new individuals, this
point a new population is generated from the old process is able to search the space of possible
population using a mixture of selection, recombi- solutions in parallel, and because selection favors
nation, and mutation. high-performing individuals, this search will
Selection is based on Darwin’s concept of nat- gradually focus on the best solutions in the search
ural selection, by which fitter individuals enjoy space. As such, the evolutionary process is
higher reproductive success. It involves identify- repeated until some stopping criteria is reached,
ing members of the population that perform best, such as the attainment of a desired level of perfor-
typically through a fitness function that maps can- mance, or the end of a preset number of
didate solutions to numeric measures of generations.
Constructing Game Agents Through Simulated Evolution 459

A major benefit of this process is that it is different scripted opponents could be used, either
general: it can be applied to any domain in in one trial or across the course of several, in order
which there is a measure of fitness/performance to make the discovered behavior more robust in
that allows certain solutions to be identified as the face of different opponents.
being better than others. Scripted controllers could be rudimentary yet
still pose an interesting challenge for an evolved
controller to overcome. However, scripted oppo- C
Evolution in Games nents may have weaknesses that evolution can
discover and exploit. Such behaviors may result
Games are typically full of numeric scores and in a high score, even though they may be
metrics that can easily be used as a means of uninteresting or easily defeatable for human
measuring agent performance. Each possible players. Fortunately, the evolutionary approach
agent is a candidate solution to the problem of can be generalized and extended into a process
how to behave in the game world. Several differ- that discovers good behaviors in an absolute
ent representations for such agents are discussed sense. This process is coevolution.
later, but even given such a representation, there
are different ways of evaluating an agent’s Coevolution
performance. Coevolution occurs when individuals in a popu-
Although most game agents are ultimately lation are evaluated with respect to other evolved
designed to interact with humans, having humans individuals. Such individuals can come from the
evaluate all candidate solutions is seldom feasible same or different populations and can be evalu-
because it is difficult for humans to maintain focus ated in tasks requiring cooperation or competition.
and evaluate solutions consistently. Completely A prominent example of competitive coevolution
automated approaches are more commonly used, within a single population is Fogel’s (2002)
but sometimes humans can also be incorporated evolved checkers player, Blondie24. Blondie24
into the process. was evolved by an evolutionary algorithm that
pitted evolved players from a single population
Evolution in Stationary Worlds against each other. The players that did a better job
A simple approach to evolving agent behavior is of defeating other members of the same popula-
to have an evolved agent interact only with a static tion had higher fitness and were used to create
or stationary world. Such worlds may have no more offspring for the next generation. The best
other agents in them or may only have agents individual after many generations used the name
with fixed control policies. A world is stationary Blondie24 on an online checkers service and was
if it and its agents do not adjust or adapt to what found to be highly competitive against the human
occurs during evaluation. In other words, the players it faced.
probability of experiencing certain outcomes in Although checkers is a traditional board
certain situations remains the same. game, the same coevolutionary process can be
An example of an agent evolving in a station- used in video games where bots are needed to
ary world without other agents is a racecar con- fill in for human players. First-person shooter
troller on a track without other cars. This process (FPS) games, like the Unreal Tournament and
can produce skilled racing behavior (Cardamone Quake franchises, fit this model because during
et al. 2009). To add to this agent the ability to the deathmatch mode of play (a free-for-all
interact with other racecars, a scripted component competition between agents trying to kill each
could be added to the controller that takes over other for points), all agents in the game have the
when other cars are near, thus combining scripted same in-game representation and available
and evolved components. Another option is to action set, making it straightforward to evolve
evolve a racecar controller in an environment such agents with a single homogeneous
filled with scripted opponent cars. A variety of population.
460 Constructing Game Agents Through Simulated Evolution

When the representations and available actions absolute sense. Instead, they may simply get bet-
of different classes of agents are different from ter with respect to each other in ways that a human
each other, it makes more sense to evolve separate player will find bizarre or incompetent. Such
populations for each type of agent and define their improvements may go through alternating cycles
fitnesses in relation to each other. For example, because they lead to behavior that beats the cur-
fighting games, like the Street Fighter and Tekken rent prevalent opponent behavior but has a weak-
franchises, pit two characters against each other in ness against another easily discovered opponent
direct one-on-one competition and generally fea- behavior. The trick with coevolution is to discover
ture a variety of characters. Therefore, the abilities behavior that incorporates all of the strengths
of the two players may be completely different while avoiding all of the weaknesses available
from each other. within the population’s range of possible
For example, assume that the goal of coevolu- behaviors.
tion is to discover skilled controllers for Ryu and In some domains, performance that is good in
Guile in Street Fighter (at least, each controller an absolute sense will be achieved automatically.
will become skilled with respect to its particular In others, it may be necessary to keep evaluating
opponent). In this scenario, there is a population each population against an archive of defeated
of Ryu controllers and a population of Guile con- opponents to assure that agents never lose the
trollers: each evaluation is a match between a ability to overcome opponents their ancestors
member of each population in which performance could defeat.
depends on the amounts of damage dealt and Although coevolution can give rise to behavior
received by each controller (there are various that is intelligent in an absolute sense, it is hard to
ways to evaluate performance with respect to implement correctly. However, agent behavior
these two scores). Any improvement in the per- only needs to be interesting with respect to
formance of individual Ryu controllers will come human players, and there are also ways to evolve
at the expense of Guile controllers, because the agent behavior by including humans in the loop.
two populations are in direct competition. When
set up correctly, this process will result in an Evolving with Humans in the Loop
evolutionary arms race, encouraging each popu- As mentioned before, the main challenges to
lation to find new ways to overcome the other. evolving against humans are that they have a
However, there are many potential pitfalls to limited ability to maintain focus for long periods
this process. For example, because each member of time and that they are not consistent in their
of each population is different, evaluations of the evaluations.
Ryu population will not be consistent if each Ryu A computer can usually run many evaluations
controller faces a different member of the Guile between computer opponents very quickly, but all
population. There is a risk of a mediocre Ryu evaluations with a human must occur in real time.
controller receiving a high performance rating After many such evaluations, a human is likely to
simply because it was paired with a poor Guile become fatigued and be unwilling to expend the
controller. This problem can be somewhat miti- necessary effort to evaluate agents properly. Nat-
gated if every member of each population faces urally, this tendency also makes evaluations
off against several members of the other popula- inconsistent. However, fatigue is less likely to
tion, and overall performance depends on perfor- occur if it is possible to evaluate many agents at
mance in all evaluations. However, performance once, or if the population is sufficiently small.
will only be completely consistent if the set of Fatigue can also be avoided if a prolonged evalu-
opponents for each population is the same, and ation process is simply the point of the game.
picking an appropriate set of opponents is For example, the commercial Creatures (Grand
challenging. et al. 1997) series of games is centered around
Unfortunately, if the set of opponents is chosen raising artificial creatures called Norns. Superfi-
poorly, the two populations will not improve in an cially, the game looks like a virtual pet-style
Constructing Game Agents Through Simulated Evolution 461

game, but among many other AI techniques several different humans or split between humans
applied in the game is support for evolution. The and the computer.
creatures the player raises grow, mature, and seek Sharing evaluations with the computer means
mates. The Creatures games take place in open- that the computer still evaluates the majority of
ended worlds in which the fun comes from nur- candidate solutions in the usual way, using a
turing and interacting with Norns. However, these computer-controlled opponent as a stand-in for
lengthy interactions influence when and with a human player. This process could in fact be C
whom each Norn mates and therefore influence carried out for many generations, only occasion-
the direction evolution takes in the creation of new ally letting a human face the best evolved agents.
Norns. If performance against the human is comparable
The model used in the Creatures games is to performance against the computer-controlled
interesting and unique, but too slow and time stand-in, then evolution is on the right track.
intensive to be useful in most other genres. Incon- Otherwise, data on how the human plays can
sistency in human evaluations is also not terribly be collected and used to adjust the behavior of
relevant in the Creatures games because variation the stand-in. These adjustments can be made
and novelty in the results of evolution are part of using supervised learning techniques, or by
the fun of the game. Additionally, there is no set evolving the stand-in to emulate human play
goal that the evolved Norns are supposed to better. However, such a system is complex, and
achieve, but the game is entertaining precisely a great deal of effort is required to make sure all
because of the variety it produces. of the separate components successfully
Another manner in which a human may be an interact.
inconsistent evaluator is due to a human’s ten- A conceptually simpler way to distribute eval-
dency to learn and adapt: a human player that uations is across many human players. Although
changes strategy mid-generation will evaluate using different human players makes inconsis-
members of the same generation differently, tencies in evaluation even more likely, there will
which would likely give an advantage to agents at least not be any systematic tendency toward
evaluated before the human adopted a new generating behaviors that are inappropriate for
strategy. human consumption: if any human can exploit
However, human adaptation is also a potential an agent’s failings, then it will eventually be
benefit. Inconsistent evaluations may add noise weeded out of the population. Furthermore, dis-
to the evolutionary process, but in the long run a tributing evaluations across many humans is made
human or set of humans who evaluate artificial easier by the Internet: specifically, tools such as
agents will settle on strategies that suit their Amazon’s Mechanical Turk and massively multi-
computer opponents. However, if the humans player online (MMO) games.
adapt and improve, then the evolved agents In fact, although the MMO model has not yet
should improve as well. In fact, if this improve- been used to evolve agent behaviors specifically,
ment happens in real time, then the resulting EC has succeeded in the MMO video game Galac-
experience is more exciting and engaging for tic Arms Race (Hastings et al. 2009). This space-
the human player. based action shooter game evolves diverse
Therefore, the primary challenge to evolving weapons for users to find and equip on their space-
agents with humans in the loop is in generating ships. The weapon preferences of all users deter-
new and interesting behaviors quickly enough to mine the fitness of weapons. The most popular
keep humans engaged. In general, having one weapons are more likely to create offspring, i.e.,
human be responsible for evaluating all individ- new weapons that players are given when they
uals puts an unreasonable burden on that individ- defeat certain enemies. A similar model could
ual, so methods that keep humans in the loop need apply for enemy agents in many MMO worlds,
to distribute evaluations in novel ways. These with enemies that are more successful in combat
evaluations can either be distributed among with human players being considered more fit,
462 Constructing Game Agents Through Simulated Evolution

and giving rise to increasingly challenging off- Evolved Representations

spring. Such a process has the potential to give
rise to new types of games in which all agents When constructing agents for games via evolu-
evolve and adapt based on a community of human tion, each candidate solution is a means of
players. representing an agent. Often, this representation
needs only account for the behavior of the agent,
Evolving Humanlike Behavior because its form is often fixed by the constraints of
Pitting evolved agents against human opponents the game. However, diverse agent morphology
will assure that incompetent behaviors are weeded can also be evolved. Regardless, there are a vari-
out of the population. However, simply discover- ety of representations that can be used to suit the
ing skilled behavior is not always a problem. needs of any particular game.
Because artificial agents are differently embodied
than human-controlled avatars, they may have Parameter Tuning
unfair access to skills that are difficult for humans The simplest way to incorporate evolution into
to develop, which in some cases means that they traditional agent design is via parameter tuning.
quickly become too skilled to be good opponents If there is an existing controller for an agent whose
for humans. For example, in Unreal Tournament behavior is influenced by some key parameters,
2004, artificial agents can be programmed using a then these parameters can be optimized using
system called Pogamut (Gemrot et al. 2009). It is evolution (typically via genetic algorithms or evo-
easy for these agents to shoot their weapons with lution strategies).
pinpoint accuracy: evolution thus creates skilled For example, a hand-coded controller for an
agents, albeit in a way that human players find agent in an FPS may have numeric parameters
frustrating and inhuman. indicating which weapon to favor, depending on
However, evolution can still be applied in the agent’s distance from its opponents. Similarly,
these situations. Agents can be evolved to max- such an agent may have several distinct control
imize performance, but under restrictions similar modules, like attack, retreat, and explore, and
to those experienced by humans. The ability of might decide which one to use based on numeric
such an agent to behave in a humanlike manner features such as its own health and its distance
was demonstrated in the 2007–2012 BotPrize from enemies and items. Evolved parameters then
competition. The purpose of the competition specify the exact thresholds for each feature, indi-
was to develop bots for Unreal Tournament cating when one module is used instead of
2004 that human players would mistake for another.
humans at least 50 % of the time. The bot UT^2 The strength of parameter tuning depends on
achieved this goal with evolved combat behavior the strength of the underlying controller. For a bad
(Schrum et al. 2012). The key idea was to opti- controller, no amount of parameter tuning may be
mize the behavior under humanlike restrictions: able to help. Similarly, a very good controller may
the more quickly it was moving and the farther its not be very difficult to tune, resulting in quick but
targets were, the less accurate it was. These small improvements in performance. In order for
restrictions forced the bot to evolve humanlike evolution to reach its full potential, the evolved
movement patterns in order to have skilled representation needs to exist within a search space
behavior. that is rich enough to contain skilled solutions that
This example demonstrates how the abilities a human designer either would not consider, or
available to an evolved agent have a strong influ- would have difficulty creating.
ence on the range of behaviors that are likely to be
evolved. These abilities are in turn influenced by Rule-Based Scripts
the type of controller evolved for the agent. Rule-based scripts are a common approach to
A variety of controllers that can be evolved to specifying the behaviors of agents in commercial
produce game agents are discussed next. games. Typically, considerable effort and person-
Constructing Game Agents Through Simulated Evolution 463

hours go into designing scripts for each agent in For any given game, the specific functions that
the game. Simple agents can have simple scripts, can be used in evolved trees need to be specified
but scripts for opponents must be complicated in by the programmer. The types of functions used
order for the game to be challenging and depend on how the evolved trees are used to
interesting. control an agent. Evolved trees could be straight-
Scripts generally consist of a list of rules, and forward function approximators made up of
each rule consists of a trigger and a corresponding purely mathematical functions using agent sen- C
action or sequence of actions. Triggers and actions sors to provide numbers. However, trees with
may also be parameterized. Evolution can easily arbitrarily complex functions can also be evolved.
rearrange blocks of information and search the For example, functions can have side effects that
parameter spaces of each rule and trigger. Of directly lead to agent action or that alter a stored
course, the process can be difficult if there is a memory structure whose contents can influence
large number of basic triggers and actions. future function evaluations.
One game genre in which opponents have a GP can also be used to evolve behavior trees.
large range of possible actions is real-time strat- Such trees hierarchically decompose behavior
egy (RTS) games. Because the computer oppo- into a collection of tasks that are prioritized and
nent must control a collection of agents in a large then executed only if certain triggers are satisfied.
space, the number of actions available is massive. In fact, a behavior tree can be thought of as a
Therefore, it makes more sense to reason about hierarchical rule-based script. Behavior trees
behavior at a higher level. Given a set of high- were initially developed for the commercial
level actions, or tactics, to choose from, a rein- release of Halo 2 (Isla 2005) and have since
forcement learning technique called dynamic been evolved in Unreal Tournament 2004 using
scripting can be used to select the best tactic for Pogamut (Kadlec 2008).
each situation, leading to improved behavior. In GP can also be used as part of a developmental
its basic form, this technique is still limited by the process: the evolved programs are executed to
preprogrammed tactics available to the agent. create some other structure that is actually used
However, dynamic scripting can be combined to control the agent. Such a process more closely
with evolution that generates new tactics. This emulates the creation of complex organisms from
process has been successfully applied to Wargus, DNA. With GP, an evolved program tree can be
a clone of the very popular Warcraft II RTS game used to create the structure and weights of a neural
(Ponsen et al. 2006). network (Gruau et al. 1996) or simply be queried
Since commercial game designers are already to fill in the weights of a predefined network
comfortable using scripts, evolving scripts is a architecture (Togelius et al. 2009). Neural net-
straightforward way to combine existing industry works have their own set of advantages as agent
knowledge with cutting-edge AI techniques. control mechanisms, which are discussed next.
However, there are also evolvable representations
that are potentially more powerful, but less well Neuroevolution
known in the game industry. The human brain is a neural network made up of
neurons that connect to each other via synapses
Genetic Programming and communicate via electrical signals. An artifi-
Genetic programming (GP) is a technique for cial neural network is an abstraction of this idea
evolving computer programs, or more accurately that transmits numerical values in place of electri-
subroutines, that are often represented as trees. cal signals, and neuroevolution is the process by
Each internal node is a function call whose which artificial neural networks are evolved to
branches are input parameters, and leaves are solve problems.
either constants, or functions with no parameters. There are many neural network models
These functions with no parameters provide sen- (Haykin 1999), but the most common is a multi-
sor values from the agent to the program. layer perceptron (MLP), consisting of input
464 Constructing Game Agents Through Simulated Evolution

neurons, output neurons, and hidden neurons in easier. HyperNEAT networks are evolved with
between. Each neuron is connected to every neu- NEAT, but with extra activation functions possi-
ron in the next layer, and a continuous activation ble in the neurons to capture symmetries and
function, typically a sigmoid, transforms the repeated patterns in the domain. Most impor-
numerical signals accumulated in each neuron. tantly, each evolved network is used to create
MLPs are universal function approximators, another network, which becomes the actual con-
assuming the correct number of neurons/layers is troller of an agent. This is another example of a
available, so they are useful in defining agent developmental process (cf. section “Genetic Pro-
behavior. MLPs can be trained by supervised gramming”). A benefit of this process is that it
learning if labelled training data is available, but becomes feasible to generate very large, but effec-
this is seldom the case when defining agent behav- tive, controller networks from small evolved net-
ior in games. works. In fact, HyperNEAT has been effectively
MLPs typically have their architecture applied to simulated RoboCup Soccer Keepaway
(number of neurons in each layer) fixed before (Verbancsics and Stanley 2010) and general game
learning, and in such a setting there is a known playing of Atari games (Hausknecht et al. 2012)
number of synaptic weights in the network. Dis- using controller networks whose input layers were
covering the weights for such networks is there- linked to 2D grids spanning the entire visual dis-
fore a special case of parameter tuning. Although play. Such massive networks are difficult to
intelligent behavior can be learned using MLPs, evolve when each connection weight must be
the large number of parameters can make it diffi- learned individually.
cult to learn particularly large MLPs. HyperNEAT is known to produce regular net-
An alternative approach is NeuroEvolution of works with repeating patterns. However, these
Augmenting Topologies (NEAT; Stanley and networks are not inherently modular (though tech-
Miikkulainen 2002), which does not produce niques to encourage such modularity exist; Hui-
MLPs. Rather, NEAT networks can have neurons zinga et al. 2014). Modularity is useful because a
connected to each other in an arbitrary topology. challenging problem can be broken down into
All networks start evolution with a minimal topol- smaller components that are easier to learn.
ogy with no hidden neurons. The networks in the Breaking up a controller into several distinct sub-
population gradually complexify across genera- controllers is a useful way to achieve multimodal
tions as new neurons and links are added via behavior, i.e., behavior that consists of distinct
mutations, which allows for convoluted, but modes subjectively different from each other.
effective topologies. In fact, by beginning the Such behavior is necessary in many games,
search in a small space with few links, it is often because different strategies often require different
possible to find very effective simple networks actions, such as attacking, retreating, searching,
with fewer links than an MLP with the same hiding, etc.
number of inputs and outputs. Such multimodal behavior can be discovered
A variant of NEAT that allows a team of agents with neuroevolution through architectures that
to learn in real time (rtNEAT; Stanley et al. 2005) support multiple distinct output modules. Such
was actually the driving force behind a machine- modules can exist in the initial population or be
learning game called Neuro-Evolving Robotic added by a mutation operator called module muta-
Operatives (NERO), in which the player takes on tion (Schrum and Miikkulainen 2014). This tech-
the role of a virtual drill sergeant to train robot nique was applied to Ms. Pac-Man, and the
soldiers that learn via neuroevolution. NEAT has evolved networks discovered both expected
since then been applied to many other video games. modes of behavior – such as a mode for fleeing
An extension to NEAT called HyperNEAT threat ghosts and a mode for chasing edible
(Stanley et al. 2009) can exploit the geometry of ghosts – and unexpected modes of behavior,
a state space to make learning certain behaviors such as one for dodging ghosts after luring them
Constructing Game Agents Through Simulated Evolution 465

near a power pill, so that when the ghosts became Conclusion

edible they would be easier to eat.
So far, only means of evolving complex con- Evolutionary computation is a powerful machine-
trollers have been discussed. However, it is possi- learning technique that has been used to discover
ble to go beyond evolving controllers and evolve skilled and interesting agent behavior in many
the bodies of agents as well. domains. Video game agents can be evolved to
play the game as a human would, to serve as C
Morphology opponents for human players, or can be evolved
EC can be used to create many types of struc- in a context where interacting with the evolution-
tures beside function approximators. The ary process is the point of the game.
Evolved Virtual Creatures (EVCs; Sims 1994; Despite the ability of evolution to discover
Lessin et al. 2014) community has developed diverse and interesting agent behaviors, the com-
ways of evolving interesting creature morphol- mercial games industry has not yet harnessed the
ogies, often using graph-based encodings. power of evolution (and other advanced AI tech-
These encodings allow for arbitrary numbers niques). This article provides a useful starting
of limbs and joints arranged in novel ways. point for understanding what can be done with
Sometimes these morphologies mimic those of evolution in games and also points out some areas
real-world organisms, but more unusual mor- of untapped potential.
phologies can also emerge; the strange quality
of such morphologies would lend itself well to a
game filled with aliens, robots, or other bizarre Cross-References
creatures.
Given the body, a means of controlling it is ▶ Machine Learning
required. Specifically, engaging and disengaging
the existing joints and/or artificial muscles will
cause parts of the body to move, which can lead References
to complex behavior if done properly. Sometimes
simple repetitive control signals, as from a sine Buckland, M.: Programming Game AI by Example. Jones
and Bartlett Learning. Plano, Texas (2005)
wave, can lead to interesting behavior given the
Cardamone, L., Loiacono, D., Lanzi, P. L.: Evolving com-
right morphology. Naturally, a human designer petitive car controllers for racing games with
could also step in and provide the behavior for neuroevolution. In: Proceedings of the 11th Annual
an evolved morphology. Conference on Genetic and Evolutionary Computation,
(GECCO’09), pp. 1179–1186. ACM, New York (2009)
However, EVCs can also have their control
Darwin, C.: On the Origin of Species by Means of Natural
routines embedded into their morphologies. In Selection or the Preservation of Favored Races in the
particular, specific sensors situated on an EVC Struggle for Life. Murray, London (1859)
can be linked to its muscles and joints. Internally, Eiben, A.E., Smith, J.E.: Introduction to Evolutionary
Computing. Springer, Berlin (2003)
these connections can be wired in a manner sim-
Fogel, D.B.: Blondie24: Playing at the Edge of AI. Morgan
ilar to a neural network or electrical circuit, mean- Kaufmann, San Francisco (2002)
ing that sensor values may be aggregated and/or Gemrot, J., Kadlec, R., Bida, M., Burkert, O., Pibil, R.,
serve as inputs to functions, whose outputs are Havlicek, J., Zemcak, L., Simlovic, J., Vansa, R.,
Stolba, M., Plch, T., Brom, C.: Pogamut 3 can assist
passed on until they eventually determine muscle
developers in building AI (not only) for their video-
and joint behavior. Such controllers have been game agents. Agents Games Simul. LNCS 5920, 1–15
evolved to run, jump, swim, grab objects, chase (2009)
after a light source, and fight or flee different Grand, S., Cliff, D., Malhotra, A.: Creatures: Artificial life
autonomous software agents for home entertainment.
opponents. These skills could serve as the build-
In: Proceedings of the 1st International Conference on
ing blocks for more complex and interesting game Autonomous Agents, AGENTS’97, pp. 22–29. ACM,
agents. New York (1997)
466 Construction Management Processes in a Digital Built Environment, Modelling

Gruau, F., Whitley, D., Pyeatt, L.: A comparison between Togelius, J., Karakovskiy, S., Koutnik, J., Schmidhuber, J.:
cellular encoding and direct encoding for genetic neural Super Mario evolution. In: Proceedings of the IEEE
networks. In: Proceedings of the 1st Annual Confer- Symposium on Computational Intelligence and Games,
ence on Genetic Programming, GP’96, 81–89. MIT CIG’09, pp. 156–161. IEEE, Piscataway (2009)
Press, Cambridge, MA, USA (1996) Verbancsics, P., Stanley, K. O.: Transfer learning through
Hastings, E.J., Guha, R.K., Stanley, K.O.: Automatic con- indirect encoding. In: Proceedings of the 12th Annual
tent generation in the Galactic Arms Race video game. Conference on Genetic and Evolutionary Computation,
IEEE Trans. Comput. Intell. AI Games 1(4), 245–263 GECCO’10, pp. 547–554. ACM, New York (2010)
(2009)
Hausknecht, M., Khandelwal, P., Miikkulainen, R., Stone,
P.: HyperNEAT-GGP: a HyperNEAT-based Atari Gen-
eral Game Player. In: Proceedings of the 14th Annual
Conference on Genetic and Evolutionary Computation, Construction Management
GECCO’12, pp. 217–224. ACM, New York (2012) Processes in a Digital Built
Haykin, S.: Neural Networks, a Comprehensive Founda- Environment, Modelling
tion. Prentice Hall, Upper Saddle River (1999)
Huizinga, J., Mouret, J.-B., Clune, J.: Evolving neural
networks that are both modular and regular: Hyper- Wilfred Matipa, Raj Shah, Sam-Odusina
NEAT plus the connection cost technique. In: Proceed- Temitope and Dianne Marsh
ings of the 16th Annual Conference on Genetic and Liverpool John Moores University, FET, SBE,
Evolutionary Computation, GECCO’14, pp. 697–704.
ACM, New York (2014)
Liverpool, UK
Isla, D.: Managing complexity in the Halo 2 AI system. In:
Proceedings of the Game Developers Conference,
GDC’05, San Francisco (2005) Synonyms
Kadlec, R.: Evolution of Intelligent Agent Behaviour in
Computer Games. Master’s thesis, Charles University
in Prague, Czech Republic (2008) Building information modelling of construction;
Lessin, D., Fussell, D., Miikkulainen, R.: Adapting mor- Building product modelling
phology to multiple tasks in evolved virtual creatures.
In: Proceedings of the 14th International Conference on
the Synthesis and Simulation of Living Systems,
ALIFE’14. MIT Press, Cambridge, MA (2014) Definition
Ponsen, M., Muñoz-avila, H., Spronck, P., Aha, D.W.:
Automatically generating game tactics via evolutionary The modelling of construction process can be
learning. AI Mag. 27(3), 75–84 (2006)
defined as a procedure of collecting tangible and
Schrum, J., Karpov, I.V., Miikkulainen, R.: Humanlike
Combat Behavior via Multiobjective Neuroevolution, intangible data that is needed for a successful
pp. 119–150. Springer, Berlin (2012) operationalization of construction activities. For
Schrum, J., Miikkulainen, R.: Evolving multimodal behav- decades, construction process modelling has
ior with modular neural networks in Ms. Pac-Man. In:
largely remained manual and or partially informa-
Proceedings of the 16th Annual Conference on Genetic
and Evolutionary Computation, GECCO’14, tion technology centric. This article discusses the
pp. 325–332. ACM, New York (2014) use of digital information modelling tools as a
Sims, K.: Evolving virtual creatures. In: Proceedings of the medium for modelling construction processes as
21st Annual Conference on Computer Graphics and
a way of improving performance in the built
Interactive Techniques, SIG-GRAPH’94, pp. 15–22.
ACM, New York (1994) environment.
Stanley, K.O., Miikkulainen, R.: Evolving neural networks
through augmenting topologies. Evol. Comput. 10(2),
99–127 (2002)
Stanley, K. O., Bryant, B. D., Miikkulainen, R.: Evolving
Introduction
neural network agents in the NERO video game. In:
Proceedings of the IEEE Symposium on Computa- The worldwide realization of the potential benefits
tional Intelligence and Games, CIG’05. IEEE, associated with the deployment of building infor-
Piscataway (2005)
mation modelling (BIM) (Jamil and Fathi 2018)
Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-
based encoding for evolving large-scale neural net- has created, inter alia, a destructive force of unre-
works. Artif. Life 15(2), 185–212 (2009) alistic framing and hype (Fox 2014) attributable to
Construction Management Processes in a Digital Built Environment, Modelling 467

stifling process modelling of construction activi- clumsiness associated with modelling of informa-
ties. On one hand scholars claim that BIM has tion using manual methods.
proven and outstanding results in construction
processes by enhancing knowledge sharing with
regard to a building or facility throughout its life Challenges of Modelling Contextual and
cycle from the conceptual design to facility man- Social Competencies Necessary for
agement (Jamil and Fathi 2018; Khosrowshahi Managing Construction Processes C
and Arayici 2012; Aranda-Mena et al. 2009;
Lindkvist 2015). On the other hand, scholars “If it isn’t broke, don’t fix it” – construction
believe that as innovative as BIM strategies industry mantra
could be (Murphy 2014), the construction indus- As the worldwide construction industry
try still requires a massive effort with regard to undergoes a paradigm shift driven by the adoption
modelling information related to processes perti- of BIM (Eadie et al. 2015), some owners and
nent to delivery of projects (Fox 2014; Motawa operators are yet to discover the added value of
and Almarshad 2015; Bosch et al. 2015). Cur- BIM in their respective areas of interest, espe-
rently the construction industry uses pervasive cially the information management within their
data modelling tools such as digital cameras, tab- organizations (Bosch et al. 2015). Anecdotal evi-
let computers and other handheld gadgets, drones, dence shows that experienced construction man-
and the like (Aziz et al. 2017) to capture data on agement workforce have a rule of thumb that
building product and processes. This implies that prefers proven construction processes to new
the industry uses varied digital gadgets to capture ones because the former have served the industry
and generate data of varied sort at any stage in the over the years need not be changed for the sake of
construction process; hence, the industry remains change than the later. Meaning that before intro-
largely fragmented in the way it handles construc- ducing change, there should be evidence showing
tion data (The Infrastructure and Projects that new approaches could outperform current
Authority 2016). While the technology to capture ones. It could, therefore, be argued that modelling
building product data is fully developed, there is a process information using BIM requires robust
challenge related to modelling tacit knowledge scrutiny and evidence-based data that could
from a section of experienced construction man- sway industry decision-makers to adopt it.
agement workforce. The introduction of the BIM According to Kassem et al. (2015), BIM is the
strategy of construction delivery, as is the case process of generating, storing, managing,
with Level 2 BIM initiative of the UK government exchanging, and sharing building information in
of 2016 (The Infrastructure and Projects Authority an interoperable and reusable way. The critical
2016), essentially takes away the most familiar issue has been to operationalize the interoperable
and user-friendly means of modelling knowledge know-how that underpins decision-making on
and ideas (paper, pen, and the like) on construc- construction processes so as to ease the justifica-
tion processes. The unintended consequences of tion and estimation of the resource consumable at
using BIM have been the massive loss of knowl- a given time period of the project. Even though
edge because current technological systems are there is a perception that BIM uptake would con-
ill-prepared to capture data from highly experi- tinue increasing in the short term, enforced Level
enced workers who are disinterest in a gadgetry- 2 BIM (BS 1192-4 2014) coupled with client
driven work environment. This article, therefore, demand has been the cardinal drivers of BIM
explains the necessity of construction processes thus far (Eadie et al. 2015). However, the increase
modelling and how it should be implemented in the uptake of BIM in its current form does not
using international standards. Without the use of address the challenges of modelling processes that
international standards, the construction indus- are critical to smooth operationalization of con-
tries miss-out on the benefits of operating in a struction projects. Construction processes, by
digital work environment because of the nature, encompass technical, contextual, and
468 Construction Management Processes in a Digital Built Environment, Modelling

behavioral (or social) competencies that are nec- (2012) opined that BIM implementation is a
essary to achieve project excellence (International major change management task, involving diver-
Project Management Association (IPMA) 2018). sity of risk areas. The identification of the chal-
It implies that one should manage people, pro- lenges and barriers is therefore an imperative
cesses, resources, results as well as project pur- precondition of this change process. While
pose. However, for construction process generic preconditions have been explored over
specialists to model the necessary data from the the past decade, there is a gap in addressing fun-
desired level of competencies, they would have to damentals of process modelling with regard to
utilize and go beyond current BIM standards and construction processes and the competencies of
guidelines (BS 1192-4 2014; BS 1192:2007+A2 managing projects. The major factor has been
2016; ISO 10303-1 1994; ISO 16739 2013). Such the “abstract” nature of the data types needed to
an achievement is desired, but highly unrealistic model processes.
as at 2018 because the industry largely hovers on Despite the realization of the need to revisit the
Level 2 BIM under the COBie code of practice modelling of construction processes, BIM imple-
(Eadie et al. 2015; BS 1192-4 2014; Specialist mentation strategies tend to focus on organiza-
Engineering Contractors’ (SEC) 2014). There- tional, cultural, and management-related action
fore, the challenge for modelling construction points. For instance, Khosrowshahi and Arayici
process information lies with the top-down BIM (2012) proposed three structured patterns to sys-
initiative strategy because it focuses more on strat- tematically tackle technology, process, and people
egy and less on data process modelling. issues in BIM implementation. These are organi-
zational culture, education and training, and infor-
Frailty of Top-Down BIM Initiatives mation management (Khosrowshahi and Arayici
The full benefits of BIM for construction manage- 2012). Similarly, Aranda-Mena et al. (2009) iden-
ment could only be realized when the industry tified mechanisms of informing project manage-
moves to a fully interoperable model (Level ment practice on the business benefits of building
3 BIM of COBie code of practice) – by file sharing information modelling (BIM) adoption. They
system or by database system, with the capability found that there was a need for “shared under-
to model technical and commercial data for the standing on business drivers to adopt BIM for
whole project (Specialist Engineering Contrac- managing the design and construction process of
tors’ (SEC) 2014). Such an aspiration could be building projects ranging from small commercial
realized if construction management research to high-rise” (Aranda-Mena et al. 2009). Motawa
could redirect the energy on process data model- and Almarshad (2015) opined that the “next gen-
ling using international standards (BS 1192-4 eration of BIM should seek to establish the con-
2014; ISO 10303-1 1994; ISO 16739 2013). Cur- cept of Building Knowledge Modelling (BKM)”.
rently, construction processes have unlimited They argued that BIM applications in construc-
approaches through which they could be tion, including those for asset management, had
modelled mainly because of the varied nature of mainly been used to ensure consistent information
data types (ISO 10303-1 1994). For instance, con- exchange among the stakeholders (Motawa and
struction process model would not only depict Almarshad 2015). Yet BKM is needed to utilize
schedule data but also contract documentation to knowledge management (KM) techniques into
model legal information necessary to building models to advance the use of these sys-
operationalize the project (Jamil and Fathi tems (Motawa and Almarshad 2015). Motawa and
2018); contracts to model risks and responsibili- Almarshad (2015) proposed a “BIM module to
ties of project stakeholders (Construction Industry capture relevant information and case-based rea-
Council (CIC) 2018); and the expert knowledge to soning (CBR) module to capture the operational
model best practice from construction managers knowledge of maintenance activities”. The struc-
(Khosrowshahi and Arayici 2012; Kähkönen and ture of the CBR module was based on analysis of a
Rannisto 2015). Khosrowshahi and Arayici number of interviews and case studies conducted
Construction Management Processes in a Digital Built Environment, Modelling 469

with professionals working in public BM depart- construction processes, and (vi) models based on
ments (Motawa and Almarshad 2015). The sug- international standards.
gestion departs from the international initiative
articulated in (ISO 16739 2013); the critical Perception of Innovation in Modelling
issue is to address the challenge at data model Construction Process Data
level using Express modelling language. Even The initial theme was to assess the perception of
though the current BIM strategies promote data innovation in modelling construction process C
sharing, knowledge is kept by those who possess data. By innovation, it was envisaged that BIM
it, mainly because their decisions are not easily could be perceived as a vehicle for modelling
modelled due to lack of mechanisms to do that. technical as well as commercial data (Murphy
Project management practices in construction 2014). The results indicated that there is a strong
refer to, inter alia, process-based modelling in perception that BIM was addressing strategic
order to plan and control the delivery of projects competencies at the expense of technical and
as efficiently as possible and within health and operational processes. Discourse on modelling
safe working environment. The ISO standard for construction processes was only perceived to
IFC, using STEP modelling language, has imply “Gantt charts” and schedule related data.
established the route way to neutral modelling of In reality, there are a myriad of key competencies
building information of all sorts, including both for construction process which would have been
explicit and tacit knowledge. However, the chal- modelled, but the means to model them is limited
lenge has been the creation of user-friendly inter- at best.
faces that could model tacit knowledge, within a
dynamic construction management environment. Value Addition of Data Models to
Construction Processes
According to Bosch et al. (2015), owners and
Approach to Developing a Procedure of operators are presumably yet to discover the
Modelling Construction Processes added value of BIM for maintenance and infor-
mation management within their organizations.
The main approach to developing a procedure of Construction processes were more complex than
modelling construction processes involved collec- building maintenance; hence, it was critical to
tion of information from practitioners and aca- explore “sources of inefficiency and ineffective-
demics (Saunders et al. 2009; May 2011), ness” of BIM. Bosch et al. (2015) found that
whereby questions were asked with regard con- current added value of BIM in the operations
struction processes and the detailed explanation stage was marginal, mainly because of the chal-
and justification (Rosenthal 2016). The model lenges of alignment between the supply of and
information came from practitioners in the United demand for information and the context-
Kingdom (UK) (Strata.com 2017; Rose et al. dependent role of information (Bosch et al.
2015; Sounderpandian 2008). Professional con- 2015). Therefore, the second question related to
struction personnel were asked to explain how the perception that data models could add value to
BIM implementation strategies address modelling construction processes. The rationale for this
of construction management processes with a question was to assess how BIM was assessed
view to improving the capturing both tangible with relation to existing construction processes.
and intangible information. Six themes have Practitioners felt that value addition was
been used to explain, and they are (i) perception represented through 4D simulation of schedule
of innovation in modelling construction process and resource planning. The same reason was
data, (ii) value addition of data models to con- cited by Li et al. (2015). The industry struggles
struction processes, (iii) perception of ultimate to capture data models with information from the
benefits of data models, (iv) modelling systems wider supply chain on construction objects. This
information, (v) modelling sustainable weakness is compounded by the scanty nature of
470 Construction Management Processes in a Digital Built Environment, Modelling

construction and commercial data because the Modelling Sustainable Construction

nature of this type data is mainly abstract and Processes
dynamic. For BIM to make a positive impact on sustainable
construction, it would have to facilitate the model-
Perception of Ultimate Benefits of Data ling to Level 3 (BS 1192-4 2014; Specialist Engi-
Models neering Contractors’ (SEC) 2014; Alwan and
Lindkvist (2015) opined that “BIM offers a holis- Gledson 2015). In such a situation, the role of
tic approach to building projects across a number modelling standards such as COBie cannot be
of practices”. However, projects have a relatively underestimated (Alwan and Gledson 2015).
short-term benefit of using BIM with 80 per cent Therefore construction practitioners were asked
of the cost of an asset spent in operations. The if they were modelling construction processes
ultimate benefit of BIM is when the information that enhanced sustainability in the industry. Con-
which the end user inherits from building projects struction practitioners felt that they had not
creates value that occurs over a long period of reached the stage where they could model all
time (Lindkvist 2015). Therefore, the questions elements and processes to promote sustainable
regarding the perception of ultimate benefit of construction. This shows that there is a realization
data model were aimed at assessing if construc- that process of instantiating particular classes
tion practitioners had a long-term view of BIM. within existing data schema such as COBie has
The response shows that 83% felt that construc- been challenging, meaning that models that claim
tion management processes are critical in assuring to contain robust building data have less informa-
the asset owner throughout the life cycle of the tion than they purport to possess.
product. However, there was an overemphasis on
the schedule, resource management, and commer- Models Based on International Standards
cial management. Construction practitioners were asked about the
role of international modelling standards for cap-
Modelling Systems Information turing construction process models. Alwan and
Practitioners were asked how the deal with data Gledson (Alwan and Gledson 2015) assessed the
models for various building systems necessary use of COBie data schema for building asset
to operate a facility. Love et al. (2015) argued modelling. Hence, the question on the role of
that BIM models emphasized the integration of standards was critical to this research. The
software packages for architectural, structural, response shows that construction practitioners
heating ventilation and air conditioning, and had a strategic awareness of the role that interna-
hydraulics, mainly because such elements have tional standards on data models played in the
scale, geometry, and can be visualized. They construction industry. However, the perception
argued that there were many systems with no of standards was attributed to regulating how the
scale or geometry and that they could not be industry and its stakeholders conducted them-
visualized (Love et al. 2015); yet they are selves in the BIM working environment. The
equally vital to the development of facilities. response indicates that there has been a gap as to
Therefore, practitioners argued that they buy how industry could contribute to the use of data
off-the-shelf data models for systems informa- models such as ISO 10303-1 (1994) and ISO
tion. The response shows that just like construc- 16739 (2013), which focus on building product
tion practitioners found it hard to model systems and process representation.
information, the same goes for construction
management processes. It is prudent to learn Discussion of the Procedures for Modelling
from systems information models (SIM) and Construction Process
use a system of isolating classes that have not Currently, the industry should rely on BIM as a
been instantiated. way of modelling construction processes because
Construction Management Processes in a Digital Built Environment, Modelling 471

of the ability to capture both tangible and intangi- modelling construction processes; this should
ble data. However, there is a challenge with the be a focus of researchers within that field (Day
current BIM schema because its construction 2011).
management processes largely remain outside Therefore, the goal for improving modelling
data models because of the abstract nature of the construction process needs to focus on meta-
data that represents the said processes. For exam- analysis of the pertinent data in Level 3 BIM
ple legal information of a construction project (Noor and Yi 2018); else the status quo is likely C
cannot be modelled as an “object,” yet it is critical to have a negative impact on future construction
to the construction process. Legal information is workers because knowledge is being lost through
of abstract data type, critical to the management of a natural process such as retirement of the most
stakeholder responsibilities, but has challenges to experience people (Sfakianaki 2015), whose
model within the BIM environment (Olatunji and knowledge cannot be captured and shared with
Akanmu 2015). Kähkönen and Rannisto (2015) new entrants to construction management.
argued that construction project management is
heavily built around document control and relat-
ing events such as change orders, submittals, Conclusion
transmittals, and requests for information; hence,
there is a reliance on electronic data/document As the construction industry continues to claim
management systems (EDMS). benefits for adopting BIM, the modelling of
However, it could be argued that EDMS offer a construction management processes and their
limited solution to modelling construction pro- embedded knowledge remain stagnant. This is
cesses. This implies that EDMS do not fully mit- mainly because of the abstract nature of the data
igate software interoperability challenges of a that typify construction processes. As a result,
typical construction process. EDMS do suffer construction processes are laden with a myriad
from the lack of software interoperability, because of undocumented tasks, events and decisions
it only speeds up the manual way of working and related to logistics, health and safety, and the
does not fundamentally address interoperability as impact on delivery schedules and the supply
per international BIM standards. Davies et al. chain and the milestones. Such challenges can-
(2017) opined that the potential for BIM to not be ignored any longer for the industry to
improve processes is documented, but few pro- move to Level 3 BIM, or else the BIM working
jects realize that potential. environment would continue being insignifi-
This is because construction processes are cant in the operational life of facilities. Strate-
laden with a myriad of undocumented tasks, gic protocols for implementing BIM have an
events, and decisions related to logistics, health insignificant drive on the process of data
and safety (Riaz et al. 2017), and the impact on modelling. Making BIM mandatory does not
delivery schedules, and the supply chain and the address the challenges of modelling construc-
milestones. Such challenges cannot be ignored tion processes.
any longer for the industry to move to Level
3 BIM (Specialist Engineering Contractors’
(SEC) 2014) (SEC, 2014), or else the BIM work- Cross-References
ing environment would continue being insignif-
icant in the operational life of facilities ▶ Interactive Computer Graphics and Model-
(Edirisinghe et al. 2017). It can be argued that View-Controller Architecture
adopting strategic protocols for implementing ▶ Open Source 3D Printing, History of
BIM has an insignificant drive on the process of ▶ Technologies for the Design Review Process
data modelling (Mei et al. 2017). Making BIM ▶ Virtual Reality Systems, Tools, and
mandatory does not address the challenges of Frameworks
472 Construction Management Processes in a Digital Built Environment, Modelling

References ISO 16739: Industry Foundation Classes (IFC) for Data

Sharing in the Construction and Facility Management
Alwan, Z., Gledson, B.J.: Towards green building perfor- Industries Technical Committee: ISO/TC 184/SC
mance evaluation using asset information modelling. 4 Industrial Data. ISO, Geneva (2013)
Built Environ. Proj. Asset Manag. 5(3), 290–303 Jamil, A.A.H., Fathi, M.S.: Contractual challenges for
(2015) BIM-based construction projects: a systematic review.
Aranda-Mena, G., Crawford, J., Chevez, A., Froese, T.: Built Environ.Proj. Asset Manag. (2018). https://doi.
Building information modelling demystified: does it org/10.1108/BEPAM-12-2017-0131
make business sense to adopt BIM? Int. J. Manag. Kähkönen, K., Rannisto, J.: Understanding fundamental
Proj. Bus. 2(3), 419–434 (2009) and practical ingredients of construction project data
Aziz, Z., Zainab Riaz, Z., Arslan, M.: Leveraging BIM and management. Constr. Innov. 15(1), 7–23 (2015)
big data to deliver well maintained highways. Facili- Kassem, M., Kelly, G., Dawood, N., Serginson, M.,
ties. 35(13/14), 818–832 (2017) Lockley, S.: BIM in facilities management applica-
Bosch, A., Volker, L., Koutamanis, A.: BIM in the opera- tions: a case study of a large university complex.
tions stage: bottlenecks and implications for owners. Built Environ. Proj. Asset Manag. 5(3), 261–277
Built Environ. Proj. Asset Manag. 5(3), 331–343 (2015)
(2015) Khosrowshahi, F., Arayici, Y.: Roadmap for implementa-
BS 1192:2007+A2: Collaborative Production of Architec- tion of BIM in the UK construction industry. Eng.
tural, Engineering and Construction Information –Code Constr. Archit. Manag. 19(6), 610–635 (2012)
of Practice. The British Standards Institution, London Li, H., Chan, G., Skitmore, M., Huang, T.: A 4D automatic
(2016) simulation tool for construction resource planning: a
BS 1192-4: Collaborative Production of Information Part case study. Eng. Constr. Archit. Manag. 22(5), 536–550
4: Fulfilling Employer’s Information Exchange (2015)
Requirements Using COBie –Code of Practice. The Lindkvist, C.: Contextualizing learning approaches which
British Standards Institution, London (2014) shape BIM for maintenance. Built Environ. Proj. Asset
Construction Industry Council (CIC): Building Informa- Manag. 5(3), 318–330 (2015)
tion Modelling (BIM) Protocol Second Edition: Stan- Love, P.E.D., Zhou, J., Matthews, J., Sing, C.-P., Carey, B.:
dard Protocol for Use in Projects Using Building A systems information model for managing electrical,
Information Models. CIC, London (2018) control, and instrumentation assets. Built Environ. Proj.
Davies, K., McMeel, D.J., Wilkinson, S.: Making friends Asset Manag. 5(3), 278–289 (2015)
with Frankenstein: hybrid practice in BIM. Eng. May, T.: Social Research: Issues, Methods and Process,
Constr. Archit. Manag. 24(1), 78–93 (2017) 4th edn. McGraw Hill – Open University Press, Maid-
Day, M.: BIM is likely to become mandatory for enhead (2011)
public projects. Architects' J. 233(1), 24–25, 2p Mei, T., Wang, Q., Xiao, Y., Yang, M.: Rent-seeking
(2011) behavior of BIM- and IPD based construction project
Eadie, R., Browne, M., Odeyinka, M., McKeown, C., in China. Eng. Constr. Archit. Manag. 24(3), 514–536
McNiff, S.: A survey of current status of and per- (2017)
ceived changes required for BIM adoption in the Motawa, I., Almarshad, A.: Case-based reasoning and
UK. Built Environ. Proj. Asset Manag. 5(1), 4–21 BIM systems for asset management. Built Environ.
(2015) Proj. Asset Manag. 5(3), 233–247 (2015)
Edirisinghe, R., London, K.A., Kalutara, P., Aranda-Mena, Murphy, M.E.: Implementing innovation: a stakeholder
G.: Building information modelling for facility man- competency-based approach for BIM. Constr. Innov.
agement: are we there yet? Eng. Constr. Archit. Manag. 14(4), 433–452 (2014)
24(6), 1119–1154 (2017) Noor, B.A., Yi, S.: Review of BIM literature in construction
Fox, S.: Getting real about BIM: critical realist descriptions industry and transportation: meta-analysis. Constr.
as an alternative to the naïve framing and multiple Innov. (2018). https://doi.org/10.1108/CI-05-2017-0040
fallacies of hype. Int. J. Manag. Proj. Bus. 7(3), Olatunji, O.A., Akanmu, A.: BIM-FM and consequen-
405–422 (2014) tial loss: how consequential can design models be?
International Project Management Association (IPMA): Built Environ. Proj. Asset Manag. 5(3), 304–317
IPMA standards – project excellence baseline (PEB). (2015)
In: IPMA, Amsterdam. https://www.ipma.world/pro Riaz, Z., Parn, E.A., Edwards, D.J., Arslan, M., Shen, C.,
jects/standard/ (2018) . Accessed 18 Aug 2018 Pena-Mora, F.: BIM and sensor-based data manage-
ISO 10303-1: Industrial automation systems and integra- ment system for construction safety monitoring. J Eng
tion – product data representation and exchange – part Desig Technol. 15(6), 738–753 (2017)
1: overview and fundamental principles. In: Technical Rose, S., Nigel Spinks, N., Canhoto, A.I.: Management
Committee: ISO/TC 184/SC 4 Industrial data. ISO, Research: Applying the Principles. Routledge – Taylor
Geneva (1994) Francis, London (2015)
Contemporary Computer Shogi 473

Rosenthal, M.: Methodology matters: qualitative research

methods: why, when, and how to conduct interviews Contemporary Computer
and focus groups in pharmacy research. Curr Pharm
Teach Learn. 8(4), 509–516 (2016) Shogi
Saunders, M., Lewis, P., Thornhill, A.: Research Methods
for Business Students, 5th edn. Pearson Education Takenobu Takizawa1, Takeshi Ito2,
Limited, Harlow (2009) Takuya Hiraoka3 and Kunihito Hoki2
Sfakianaki, E.: Resource-efficient construction:
rethinking construction towards sustainability.
1
Faculty of Political Science and Economics, C
World J Sci Technol Sust Develop. 12(3), Waseda University, Tokyo, Japan
2
233–242 (2015) Department of Communication Engineering and
Sounderpandian, A.: Complete Business Statistics, Informatics, The University of Electro-
7th edn. McGraw-Hill/Irwin (2008). New York
Specialist Engineering Contractors’ (SEC): First Steps Communications, Chofu, Tokyo, Japan
3
to BIM Competence A Guide for Specialist Con- HEROZ, Inc., Osaka, Japan
tractors, Published by the Specialist Engineering
Contractors’ (SEC) Group in collaboration with
the BIM Academy at the University of Northum-
bria. https://www.thefis.org/wp-content/uploads/ Synonyms
2016/09/BIM-Guide-2014-1.pdf (2014). Accessed
18 Aug 2018 Chess variant; Japanese chess
Strata.com: Total Number of Quantity Surveyors in the
United Kingdom (UK) from 2011 to 2016 (in 1,000).
https://www.statista.com/statistics/319241/number-of-
quantity-surveyors-in-the-uk/ (2017). Accessed 12 Jan Definition
2017
The Infrastructure and Projects Authority: “Govern- Computer shogi is a field of artificial intelligence
ment Construction Strategy 2016–20”, The Infra-
structure and Projects Authority (IPA) March involving the creation of software programs
2016. HM Treasury and Cabinet Office, London capable of playing shogi, the Japanese form of
(2016) chess.

Introduction
Constructionism
Computer shogi was first developed in late 1974
by Takenobu Takizawa and his research group. It
▶ PBL-Based Industry-Academia Game Devel-
has been steadily improved by researchers and
opment Education
commercial programmers using game tree making
and pruning methods, opening and middle game
databases, and feedback from research into tsume-
shogi (mating) problems. The strength of com-
Constructivism puter shogi has been measured by watching and
studying many games between computer pro-
▶ PBL-Based Industry-Academia Game Devel- grams and professional players and has reached
opment Education that of top-level human players. In the remainder
of the article, section “Computer-Computer
Games” describes the history of computer-
computer games. Section “Computer Shogi
Consumers Players” describes the programs that played
them, and section “Computer-Human Games”
▶ Game Prosumption describes the history of human-computer games.
474 Contemporary Computer Shogi

Computer-Computer Games 2. The CSA imposes no restrictions on the hard-

ware or software of any WCSC entrant or on
Yoshiyuki Kotani and Takenobu Takizawa jointly persons entering the hardware or software.
established the Computer Shogi Association 3. The CSA supports the interchange of ideas
(CSA, Takada 2014) in 1986. This organization among hardware/software developers who
started organizing computer shogi tournaments enter their products in WCSC tournaments.
called the World Computer Shogi Championships
(WCSCs), in 1990 (Takizawa and Grimbergen Table 1 shows a summary of WCSC results.
2001). The tournaments are supported by Nihon Ten programs have won the tournaments. Kana-
Shogi Renmei (the Japan Shogi Association or zawa Shogi has won five times, IS-Shogi and
JSA). Gekisashi four times each, YSS three times,
The following are the policies of WCSCs Bonanza and GPS Shogi twice each, and Eisei
declared on January 23, 2012. Meijin, Morita Shogi, Bonkras, and Apery once
each. The 22nd WCSC was held on May 3–5,
1. The WCSC tournaments are held for the pur- 2012. Forty-two teams (including one invited)
pose of determining the strongest computer entered, with GPS Shogi winning the champion-
shogi program at the time under conditions of ship for the second time. The 23rd WCSC was
fair and impartial operation. held on May 3–5, 2013. Forty teams (including

Contemporary Computer Shogi, Table 1 Results of the world computer shogi championships
No. Date Number of Participants Winner Second Third
1 1990.12.2 6 Eisei Meijin Kakinoki Morita
2 1991.12.1 9 Morita Kiwame Eisei Meijin
3 1992.12.6 10 Kiwame Kakinoki Morita
4 1993.12.5 14 Kiwame Kakinoki Morita
5 1994.12.4 22 Kiwame Morita YSS
6 1996.1.20–21 25 Kanazawa Kakinoki Morita
7 1997.2.8–9 33 YSS Kanazawa Kakinoki
8 1998.2.12–13 35 IS Kanazawa Shotest
9 1999.3.18–19 40 Kanazawa YSS Shotest
10 2000.3.8–10 45 IS YSS Kawabata
11 2001.3.10–12 55 IS Kanazawa KCC
12 2002.5.2–5 51 Gekisashi IS KCC
13 2003.5.3–5 45 IS YSS Gekisashi
14 2004.5.2–4 43 YSS Gekisashi IS
15 2005.5.3–5 39 Gekisashi KCC IS
16 2006.5.3–5 43 Bonanza YSS KCC
17 2007.5.3–5 40 YSS Tanase Gekisashi
18 2008.5.3–5 40 Gekisashi Tanase Bonanza
19 2009.5.3–5 42 GPS Ootsuki Monju
20 2010.5.2–4 43 Gekisashi Shueso GPS
21 2011.5.3.–5 37 Bonkras Bonanza Shueso
22 2012.5.3–5 42 GPS Puella alpha Tsutsukana
23 2013.5.3–5 40 Bonanza Ponanza GPS
24 2014.5.3–5 38 Apery Ponanza YSS
Kanazawa is the successor to Kiwame
Puella alpha is the successor to Bonkras
Contemporary Computer Shogi 475

Contemporary Computer Shogi, Table 2 22nd WCSC final results (May 5, 2012)
Rank Player 1 2 3 4 5 6 7 Pt SB MD
1 GPS Shogi 6+ 5+ 8+ 7+ 3+ 2 4+ 6.0 17.0 12.0
2 Puella alpha 5+ 7+ 6 4 8+ 1+ 3+ 5.0 16.0 9.0
3 Tsutsukana 8+ 6+ 7 5+ 1 4+ 2 4.0 11.0 6.0
4 Ponanza 7+ 8+ 5 2+ 6+ 3 1 4.0 11.0 5.0
5 Shueso 2 1 4+ 3 7+ 8 6+ 3.0 9.0 3.0 C
6 Gekisashi 1 3 2+ 8+ 4 7+ 5 3.0 8.0 2.0
7 YSS 4 2 3+ 1 5 6 8+ 2.0 5.0 0.0
8 Blunder 3 4 1 6 2 5+ 7 1.0 3.0 0.0

Contemporary Computer Shogi, Table 3 23rd WCSC final results (May 5, 2013)
Rank Player 1 2 3 4 5 6 7 Pt SB MD
1 Bonanza 5+ 8+ 6+ 2 7 4+ 3+ 5.0 16.0 10.0
2 Ponanza 8+ 7+ 5 1+ 6+ 3 4+ 5.0 15.0 9.0
3 GPS Shogi 7+ 6+ 8+ 5+ 4 2+ 1 5.0 14.0 8.0
4 Gekisashi 6+ 5 7+ 8+ 3+ 1 2 4.0 11.0 5.0
5 NineDayFever 1 4+ 2+ 3 8+ 7 6 3.0 10.0 4.0
6 Tsutsukana 4 3 1 7+ 2 8+ 5+ 3.0 6.0 2.0
7 Shueso 3 2 4 6 1+ 5+ 8 2.0 8.0 0.0
8 YSS 2 1 3 4 5 6 7+ 1.0 2.0 0.0

one invited) entered, with Bonanza winning the which finished eighth, in the last game, Ponanza
championship for the second time. The 24th would have been the winner. The final round
WCSC was held on May 3–5, 2014. Thirty-eight results are shown in Table 3.
teams entered, with Apery winning the champi- The 24th WCSC tournament had 38 program
onship for the first time. entrants, with Apery being declared the winner.
The 22nd WCSC tournament had 42 computer Both Apery and Ponanza won five games and lost
players (including one invited). GPS Shogi was two, but Apery got the nod on tiebreak, even
the winner, but if Ponanza had beaten GPS Shogi though YSS beat both of them. Ponanza was
in the last game, then Tsutsukana, Ponanza, or thus runner-up as it had been the year before,
Puella alpha would have been the winner. The while YSS had to settle for equal third place.
final round results are shown in Table 2. The YSS program entered the WCSC tourna-
The 23rd WCSC tournament had 40 computer ments 23 times and never finished worse than
players (including one invited), and Bonanza was equal eighth in any of them. Both Ponanza and
the winner. After the preliminaries had narrowed NineDayFever would have won the tournament if
the field down to eight, those eight played a round they had won their last game. The final round
robin (each player playing the other seven players results are shown in Table 4 (Takizawa 2014).
once) to determine the winner. The results were
close, with the top three programs losing twice,
while every program won at least once. This indi- Computer Shogi Players
cates that the top programs were comparable in
strength. For example, GPS Shogi would have Most computer shogi programs carry out minimax
been the winner if it had won the last game. If tree searches enhanced by techniques in computer
Shueso, which finished seventh, had beaten YSS, chess and by other completely new ideas. In this
476 Contemporary Computer Shogi

Contemporary Computer Shogi, Table 4 24th WCSC final results (May 5, 2014)
Rank Player 1 2 3 4 5 6 7 Pt SOS SB MD
1 Apery 4 6+ 7+ 5+ 3 2+ 8+ 5.0 23.0 15.0 10.0
2 Ponanza 5+ 7+ 6+ 8+ 4+ 1 3 5.0 23.0 14.0 10.0
3 YSS 6 5 4 7+ 1+ 8+ 2+ 4.0 24.0 13.0 8.0
4 NineDayFever 1+ 8+ 3+ 6 2 7+ 5 4.0 24.0 12.0 7.0
5 Gekisashi 2 3+ 8+ 1 7 6+ 4+ 4.0 24.0 11.0 7.0
6 Bonanza 3+ 1 2 4+ 8+ 5 7 3.0 25.0 8.0 4.0
7 Tsutsukana 8+ 2 1 3 5+ 4 6+ 3.0 25.0 7.0 3.0
8 N4S 7 4 5 2 6 3 1 0.0 28.0 0.0 0.0

section, “The Art of Computer Shogi” describes machine learning of evaluation functions in
the art of computer shogi, and “Brief Description WCSC and won the competition. The learning
of the 24th WCSC Winner Program” gives a brief method consists of subsequent steps: prepare a
description of the 24th WCSC winner program, set of grandmaster game records, a search func-
Apery. tion, and a linear weighted evaluation function
and decide the initial weights in the function.
The Art of Computer Shogi Give positions in game records to the search func-
Many computer shogi programs use alpha-beta tion and compute agreements between the search
pruning, PVS (principal variation search), quies- results and the recorded moves. If more search
cence search, aspiration search, null move results agree with recorded moves, then the
(forward) pruning, futility pruning, killer heuris- weights are probably better. To adjust the weights,
tic, history heuristic, iterative deepening, transpo- iterative procedures of numerical minimization
sition hash tables, and singular extension, adopted techniques are carried out. In this way, the learn-
from chess programs. A minimax tree for shogi ing procedures optimize the weights to control the
usually has a larger branching factor in the end- minimax tree search results.
game than in the opening. The row branching Figure 1 illustrates such a learning procedure
factor on average is about 80, and it is known as applied to chess. Assume the game position has
that the factor is effectively reduced to about three legal moves and a grandmaster has just
3 by using pruning techniques. played a move to reach child position A. The
procedure carries out a tree search to examine
Large-Scale Optimization for Evaluation Functions the three child positions that could have been
Heuristic search is a powerful method in artificial reached on the move and obtains search value
intelligence. In the case of chess, it is capable of 1 for position A, 3 for position B, and 7 for posi-
deciding a plausible move after expanding a large tion C. In this case, the tree search recognizes that
minimax game tree with heuristic leaf evalua- C is the best, B is the second best, and A is
tions. The quality of such heuristic evaluations is unfortunately the worst. Under this circumstance,
crucial for making strong computer chess players. the learning procedure adjusts the value of the leaf
Researchers have made substantial efforts in a evaluation rooted at A higher, and those rooted at
quest to create effective evaluation functions by B and C lower.
using machine learning techniques in various Such learning methods were investigated in
games (Fürnkranz 2001). Shogi is probably one computer chess because the game scores of
of the more interesting games to see successful grandmaster games were a common way to learn
examples of machine learning of evaluation chess, not only for computer players but also
functions. human players. Assume that one has a position
In 2006 Bonanza, a program developed by reached in a game score and the desired move is
Kunihito Hoki, demonstrated a practical use of the one that was actually played. A chess program
Contemporary Computer Shogi 477

Contemporary
Computer Shogi, 7 Game position
Fig. 1 An illustration of ove
m
evaluation learning ired
procedure Des

A B C Child positions
C
Tree search

1 3 7 Leaf evaluation

Increase Decrease

has an evaluation function e(p,w), where p is the m0 and p.d to get better learning performances.
game position and the feature weight vector w con- Furthermore, Tesauro (2001) used a continuous
tains the parameters to be adjusted. Now consider approximate of the step function. Although it
a simple intuitive goal: make the results of a seemed that such learning methods would be
one-ply search agree with the desired move, able to adjust hundreds of weights to have reason-
where the search selects the highest evaluation able values, fully automated learning of the chess
value. Thus, w should be adjusted so that the evaluation functions still remains a challenging
desired move has the highest evaluation of all goal. For example, developers have reported that
the moves. This goal can be written as a minimi- the majority of the features and weights in Deep
zation problem with an objective function: Blue were created/tuned by hand (Campbell
et al. 2002).
J H ðw Þ ¼ m0
H ðeðp:m0 , wÞ  eðp:d, wÞÞ ð1Þ It turned out that such machine learning by
using grandmaster games was also useful in
Here, p.m is the child position reached after move shogi. Hoki et al. proposed a method, Minimax
m, d is the desired move, index m0 runs for all legal Tree Optimization (MMTO, Hoki and Kaneko
moves except d, and H(x) is the Heaviside step 2014), to learn the evaluation function of a prac-
function, i.e., H(x) equals 1 if x  0 and 0 other- tical alpha-beta search program. They used grid-
wise. Because this objective function counts the adjacent update, equality constraint, and l1 regu-
number of moves that have an evaluation value larization to achieve scalability and stability. Their
greater than or equal to that of the desired move, a objective function with modified gradient vectors
better w can be found by minimizing Eq. 1. was to be able to optimize the values of 13 shogi
In chess, several studies have been made on the pieces with reasonable numerical accuracy and to
basis of the objective function Eq. 1 (e.g., see adjust 40 million parameters for strength. Hoki
Meulen 1989). However, numerical minimization had proposed an earlier version of MMTO in
seemed to present practical difficulties. To over- 2006 (Hoki et al. 2006) and implemented it in
come such difficulties, Marsland (1985) used the shogi program Bonanza, winning the WCSC
some continuous functions instead of the non- championships in 2006 and 2013. After 2006, it
continuous step function H(x) so that the gradient became famous and has often been referred to as
vector would help to reduce the function value the “Bonanza method.” Recent computer shogi
numerically. Moreover, Hsu et al. (1990) used players have evaluation functions, where the
evaluations of leaf positions of the principal var- weights are learned from professional players’
iations instead of using the direct evaluations of p. game records.
478 Contemporary Computer Shogi

Many machine learning techniques that do not played individually. They also proposed a pseu-
require grandmaster game records have also been dorandom number (PRN) ensemble method. In
applied to shogi. However, the adjustment of the this method, the ensemble is built using multiple
full weights in the evaluation function remains a copies of one base program, and each copy is
challenging goal. The studies that have been diversified by adding random numbers to the eval-
published so far have adjusted only piece values uation function of the base program.
or a small part of the feature weights in the eval- They researched these methods where a
uation functions. machine chose a move automatically without
human intervention. Here, they defined “consul-
Consultation Algorithm tation” as a process that generates one answer on
Much research that has compared problem- the basis of conclusions obtained by using two or
solving by groups with that by individuals has more different thinking processes. They consid-
been carried out in the field of cognitive science. ered various methods of “consultation” by com-
Shaw conducted an experiment that compared the puters and concluded that the “majority voting
efficiency of problem-solving by a group and that system” might be one of the simplest systems, in
by individuals in 1932, using a missionaries-and- which a majority opinion is adopted from various
cannibals problem (Shaw 1932). Many opinions. Although the system was very simple,
researchers used simple logic problems of this they showed its effectiveness in their experimen-
type from the 1940s to the 1950s and found that tal results.
groups were able to outperform individuals. These Sugiyama et al. (2011) examined another
results supported the old proverb that “two heads approach, one of combining multiple programs.
are better than one.” In carrying out this approach, they used a new
Althöfer et al. (2003) have carried out studies selection rule that selects the player that yields the
since 1985 on chess or the game of Go in research highest evaluation value. They called this method
on the selection of moves in thought games. They the “optimistic ensemble system” and reported
proposed a system called 3-Hirn that consists of that it often outperformed the “majority voting
two computer programs and a human chess system” when multiple computer programs were
player, where the human selects a move from the prepared by using the PRN ensemble method.
programs’ outputs. They demonstrated that the Hoki et al. (2014) examined these ensemble
system enabled the programs to improve their systems in computer chess. The results of their
ratings by about 200 points. They carried out experiments showed that both the “majority vot-
almost the same experiments in Go or other ing system” and the “optimistic ensemble system”
games, and through them they demonstrated the were efficient in computer chess.
system’s efficiency. Two advantages of the “consultation” method
From this, one gets the idea that an ensemble of are known. First, it can use a loosely coupled
game programs may be able to play a better move computing environment. Because it is simple
than an individual program does. Although many and fault tolerant, it can always be an alternative
sophisticated ensemble-based systems in com- when a large-scale distributed environment is
puter science have been built with the aim of available for strength. Second, it has a high degree
achieving better performance, designing such sys- of generality and is orthogonal to other parallel
tems in computer games still remains a search techniques. It can function efficiently with
challenging task. multiple different programs even when each pro-
One of the methods that could be used to build gram utilizes parallel search techniques.
such an ensemble-based system in shogi is the Research on consultation systems in games is
majority voting method. Obata et al. (2011) still in the infancy stage. So far, the effectiveness
reported that majority voting in three famous pro- of simple majority voting has been examined only
grams (YSS, GPS Shogi, and Bonanza) produced for shogi and chess, where computer players per-
better games than any of the three programs form minimax tree searches, and in Go, where
Contemporary Computer Shogi 479

computer players perform Monte Carlo tree players also often use a hard-coded function
searches. dedicated to finding one- or three-ply mate
sequences.
Realization Probability Search
When computer players start to play a shogi game, Brief Description of the 24th WCSC Winner
moves previously made by professional players Program
have already been collected and categorized and The Apery program, which was developed by C
their probabilities calculated in the program. Takuya Hiraoka, had a Stockfish-like search func-
These moves include capturing and recapturing tion (Stockfish is a strong open source chess pro-
pieces, promoting a rook or bishop to gain a gram). Over the past 3 or 4 years, the effectiveness
material advantage, checking, and so on. During of the chess search function in shogi has become
the course of the game, the players control tree famous, notwithstanding the differences in the
expansions based on these probabilities, i.e., if the rules between the two games. Using chess search
multiplied probability of a move history from the techniques, Apery achieves performance
root to the current position is higher than a thresh- improvements by using additional functions such
old, then the program searches deeper. Tsuruoka as a one-ply mate search function.
et al. (2002) proposed this algorithm and Apery also has a Bonanza-like evaluation
implemented it in the Gekisashi program, and function. It evaluates shogi positions by using a
with it they won WCSC titles in 2002, 2005, three-piece square table that evaluates all combi-
2008, and 2010. nations of a king and other two-piece locations.
Because the combinations always contain one or
Distributed Parallel Search two kings, the evaluation function is sensitive to
The first computer shogi program using a multi- the distance between kings and other pieces. In
processor system was the Super Shogi program shogi, the distance from a king is a vital feature.
developed by Hisayasu Kuroda in 1997, which Most shogi pieces have limited mobility, and
used an eight-computer system. In 2012, the GPS those that are located far from the kings are for
Shogi program developed by Tetsuro Tanaka and the most part useless, especially in the endgame.
Tomoyuki Kaneko et al. used 320 processors Since the source codes of Bonanza version
(666 cores in total) and won the 22nd WCSC 4 appeared online in 2009, it seems that the eval-
title (Kaneko and Tanaka 2012). uation function’s performance has advanced in
both computational speed and evaluation accu-
Tsume-Shogi (Mating Problem) Solver racy. Hiraoka believes that Apery’s evaluation
In addition to ordinary minimax searches, a function does not reach the highest level and has
computer player often uses an additional some room to improve. One of the difficulties in
search function dedicated to finding a long creating such a highest-level evaluation function
mate sequence. The techniques used in is that machine learning of evaluation functions
tsume-shogi functions derive from studies on requires a tremendous amount of computational
solving such mating problems. Unlike in resources.
chess, in shogi the number of possible Hiraoka believes that Apery’s rating is
moves in the endgame is the same as the 200–250 points below that of the top programs
number of possible moves in the middle and considers that Apery won the 24th WCSC for
game. Therefore, an efficient AND/OR tree a number of reasons. First, the tournament con-
search algorithm are needed for solving cluded with a round robin of eight entrants, so
tsume-shogi problems. Recent state-of-the-art only seven games determined the tournament
solvers employ the df-pn search algorithm winner. Second, on the basis of a “pawn toss”
proposed by Ayumu Nagai, which is capable method, Apery moved first four times and second
of solving most existing tsume-shogi prob- three times. This is important because the player
lems (Kishimoto et al. 2011). Computer moving first has the initiative. Statistics show that
480 Contemporary Computer Shogi

in professional shogi, the player who moves first six-game match, the strongest computer shogi
wins about 52 % of the time; Apery’s program was only a little stronger than an average
corresponding percentage playing against other club player. The top programs reached the profes-
programs is 56 %. Third, a relatively short time sional 4-dan level in 2010 and have now reached
control was used in the tournament (25 min for the the top human-player level. Table 5 summarizes
whole game with no time-shortage countdowns), the results of computer-human games that have
and while the top programs utilized a loosely been played to date.
coupled computing environment, Apery did not. The first game between a female professional
Because Apery was free from distributed comput- shogi player and a computer shogi program was
ing overheads, the short time control increased its an exhibition game played on July 29, 2001 at the
winning chances. Fourth, Apery played the open- Mind Sports Olympiad in Japan. Each player had
ings quite strongly since its opening “book” 15 min for the whole game plus a 60-s time-
contained the moves computer players had previ- shortage countdown. Yamato Takahashi, the pro-
ously played that were available in floodgate fessional player, moved first and won her game
(http://wdoor.c.u-tokyo.ac.jp/shogi/). On the other against IS-Shogi. After the game she said, “IS-
hand, the other entrants did not seem to focus on Shogi’s strength is about 1-dan in the opening,
the best moves as given in opening books. 4- or 5-dan in the middle game, and 3- or 4-dan
in the endgame. For the game as a whole it is not
1- or 2-dan but 3- or 4-dan.”
Computer-Human Games On September 19, 2005, Hokkoku Shimbun
sponsored a game between the TACOS program
In 1997, when IBM’s “Deep Blue” program beat and Takanori Hashimoto, an 8-dan professional.
the world chess champion Gary Kasparov in a Hashimoto eventually won, but TACOS had the

Contemporary Computer Shogi, Table 5 Game results: professional shogi players vs. computer shogi programs
Time control
Date Event Professional human player Computer Winner (countdown)
2007.3.21 Daiwa Shoken Hai Akira Watanabe (Ryu-O Bonanza Human 2 h (60 s)
Special Game titleholder)
2011.12.21 Den-O-Sen Practice Kunio Yonenaga (Lifetime Bonkras Computer 15 min (60 s)
Game Kisei titleholder)
2012.1.14 First Shogi Kunio Yonenaga (Lifetime Bonkras Computer 3 h (60 s)
Den-O-sen Kisei titleholder)
2013.3.23 Second Shogi Koru Abe (4-dan) Shueso Human 4 h (60 s)
2013.3.30 Den-O-Sen Shin’ichi Sato (4-dan) Ponanza Computer
2013.4.6 Kohei Funae (5-dan) Tsutsukana Computer
2013.4.13 Yasuaki Tsukada (9-dan) Puella draw
alpha
2013.4.20 Hiroyuki Miura (9-dan) GPS Shogi Computer
2013.12.31 Den-O-Sen Rematch Kohei Funae (5-dan) Tsutsukana Human 4 h (60 s)
2014.3.15 Third Shogi Den-O- Tatsuya Sugai (5-dan) Shueso Computer 5 h (60 s) (chess
2014.3.22 Sen Shin’ya Sato (6-dan) Yaneura-O Computer clock)
2014.3.29 Masayuki Toyoshima YSS Human
(7-dan)
2014.4.5 Taku Morishita (9-dan) Tsutsukana Computer
2014.4.12 Nobuyuki Yashiki (9-dan) Ponanza Computer
2014.7.19–20 Third Shogi Den-O- Tatsuya Sugai (5-dan) Shueso Computer 8 h (60 s) (chess
Sen Rematch clock)
Contemporary Computer Shogi 481

advantage in the middle game. After this the JSA game plus a 60-s time-shortage countdown. The
prohibited professional players from playing programs had taken the first five places at the 22nd
against computer players in front of an audience WCSC. Table 1 shows the match results. It had
without its authorization. In the same year, been predicted that the human players would win
Gekisashi was invited to an amateur Ryu-O tour- the match with four wins and one loss, but the
nament. Placing 16th, it was evaluated as being prediction was wrong; the programs took the
comparable in strength to top-level amateur shogi match with three wins, one draw, and one loss. It C
players. was a major surprise that the very high-ranking
On March 21, 2007, a game was played professional Hiroyuki Miura (9-dan) was defeated
between Bonanza and Akira Watanabe, holder of by GPS Shogi (Takizawa 2013).
the prestigious Ryu-O championship title. Spon- The third Den-O-Sen (July 2014), like the sec-
sored by Daiwa Securities, it was the first official ond, matched five human players and five com-
game between a professional shogi player and a puter programs. The programs had taken first
computer player since the abovementioned through fifth places at the first Den-O tournament
Hashimoto-TACOS game in 2005. It was a very (a November 2013 all-computer tournament
close game, with Watanabe eventually winning. sponsored by Dwango and the JSA), with
Watanabe recently said that he was lucky to win Ponanza the winner. Again it was predicted that
because in a critical position, a professional the human players would win the match with four
human player can find a winning move relatively wins and one loss, this time because the program-
easily, but this is not so for computer shogi mers had not changed their programs after the
programs. Den-O tournament and the professional players
On October 11, 2010, Ichiyo Shimizu, one of had had a chance to study them. But again the
the top female professional players, lost a game prediction was wrong; this time the programs took
against the computer shogi system Akara2010. the match with four wins and one loss.
The game was sponsored by Komazakura (JSA
Ladies Professional Players Group), the Informa-
tion Processing Society of Japan, and the Univer- Summary
sity of Tokyo. Each player had 3 h for the whole
game plus a 60-s time-shortage countdown. The The top computer shogi programs have already
Akara2010 system employed the majority voting come close to the top human-player level. About
method using four existing programs combined 10 years ago, Yoshiharu Habu, holder of the pres-
with a distributed search method (Hoki tigious Meijin championship title, predicted that
et al. 2013) to enable it to use a large number of the top programs would get close to the top
computers. human-player level in 10 years. His words were
The first Den-O-Sen was held on January prophetic. Many professional players now under-
14, 2012. This was a game played between a stand how strong top computer programs have
retired professional, the late Kunio Yonenaga, become, as do many people who have seen the
and the Bonkras computer program that had won results of the Den-O-Sen competitions or read
the 21st WCSC. The game was sponsored by the about them in newspapers.
JSA, Dwango, and Chuokoron-Shinsha, Inc. Each Computer shogi programs have become the
player had 3 h for the whole game plus a 60-s helpful partners of professional players, who
time-shortage countdown. Taking advantage of now use them in their studies. For example, 8-
having the initiative that comes with moving dan Daisuke Nakagawa observed the 18th WCSC
first, Bonkras won the game. and also the exhibition game between the Tanase
The second Den-O-Sen, sponsored by Dwango Shogi program (the tournament runner-up) and
and the JSA and held in March–April 2013, the top amateur player Yukio Kato, which Tanase
matched five human players and five computer Shogi won. After studying this game, Nakagawa
programs. Each player had 4 h for the whole won his next three games in professional
482 Context

competition, including one against Akira Hsu, F.-H., Anantharaman, T.S., Campbell, M.S.,
Watanabe, one of today’s very top-ranked players. Nowatzyk, A.: Deep thought. In: Marsland, T.A.,
Schaeffer, J. (eds.) Computers, Chess, and Cognition,
Another example is 9-dan Toshiyuki Moriuchi, pp. 55–78. Springer, New York (1990)
who studied Ponanza’s moves and used them to Kaneko, T., Tanaka, T.: Distributed game-tree search based
beat the superior Yoshiharu Habu in their 2013 on prediction of best moves. IPSJ J. 53, 2517–2524
match for the Meijin championship title. (2012)
Kishimoto, A., Winands, M., Müller, M., Saito, J.-T.:
The game of chess as played between humans Game-tree search using proof numbers: the first twenty
is still vibrant and active, although computer chess years. ICGA J. 35, 131–156 (2011)
programs are now stronger than the strongest Marsland, T.A.: Evaluation function factors. ICGA J. 8,
human players. The same thing is likely to happen 47–57 (1985)
Obata, T., Sugiyama, T., Hoki, K., Ito, T.: Consultation
in the next 5 years for the game of shogi. How- algorithm for computer shogi: move decisions by
ever, while computer programs have come closer majority. In: van den Herik, J., et al. (eds.) Computer
to unraveling the deepest mysteries of chess than and Games 2010. Lecture Notes in Computer Science,
they have of shogi, the fathomless depths of both vol. 6515, pp. 156–165. Springer, Berlin (2011)
Shaw, M.E.: Comparison of individuals and small groups
games will continue to challenge researchers in in the relational solution of complex problems.
the future. Am. J. Psychol. 44, 491–504 (1932)
Sugiyama, T., Obata, T., Hoki, K., Ito, T.: Optimistic selec-
tion rule for ensemble approach to improving strength
of Shogi program. In: van den Herik, J. (ed.) Computer
Cross-References and Games 2010. Lecture Notes in Computer Science,
vol. 6515, pp. 156–165. Springer, Berlin (2011)
▶ Computer Go Takada, J.: The computer shogi association web page.
http://www.computer-shogi.org/index_e.html (2014).
Accessed 31 Dec 2014
Takizawa, T., Grimbergen, R.: Review: computer Shogi
References and Further Reading through 2000. In: Marsland, T.A., Frank, I. (eds.)
Computers and Games. Lecture Notes in Computer
Althöfer, I., Snatzke, R.G.: Playing games with multiple Science, vol. 2063, pp. 433–442. Springer, Berlin
choice system. In: Schaeffer, J., et al. (eds.) Computer (2001)
and Games. Lecture Notes in Computer Science, vol. Takizawa, T.: Computer Shogi 2012 through 2014. In:
2883, pp. 142–153, Springer, Berlin (2003) Proceedings of the 19th Game Programming Work-
Beal, D.F., Smith, M.C.: Temporal difference learning shop, Kanagawa, Japan, pp. 1–8 (2014)
applied to game playing and the results of application Takizawa, T.: Computer shogi programs versus human
to shogi. Theor. Comput. Sci. 252, 105–119 (2001) professional players through 2013. In: Proceedings of
Campbell, M., Hoane Jr., A.J., Hsu, F.-h.: Deep blue. Artif. the 18th Game Programming Workshop, Kanagawa,
Intel. 134, 57–83 (2002) Japan, pp. 98–101 (2013)
Fürnkranz, J.: Machine learning in games: a survey. In: Tesauro, G.: Comparison training of chess evaluation func-
Fürnkranz, J., Kubat, M. (eds.) Machines That Learn to tions. In: Fürnkranz, J., Kubat, M. (eds.) Machines That
Play Games, pp. 11–59. Nova, Commack (2001) Learn to Play Games, pp. 11–59. Nova, Commack
Hoki, K., Kaneko, T.: Large-scale optimization for evalu- (2001)
ation functions with minimax search. J. Artif. Intell. Tsuruoka, Y., Yokoyama, D., Chikayama, T.: Game-tree
Res. 49, 527–568 (2014) search algorithm based on realization probability.
Hoki, K.: Optimal control of minimax search results to ICGA J. 25, 145–152 (2002)
learn positional evaluation. In: Proceedings of the van der Meulen, M.: Weight assessment in evaluation
11th Game Programming Workshop, Kanagawa, functions. In: Beal, D. (ed.) Advances in Computer
Japan, pp. 78–83 (2006) Chess 5, 81–89. North-Holland, Amsterdam (1989)
Hoki, K., Kaneko, T., Yokoyama, D., Obata, T., Yamashita,
H., Tsuruoka, Y., Ito, T.: Distributed-Shogi-System
Akara 2010 and its demonstration. Int. J. Comput. Inf.
Sci 14, 55–63 (2013)
Hoki, K., Omori, S., Ito, T.: Analysis of performance of Context
consultation methods in computer chess. J. Inf. Sci.
Eng. 30, 701–712 (2014) ▶ Game Writer’s Dilemma: Context vs. Story
Counter-Strike Global Offensive, an Analysis 483

Contextual Learning and Counter-Strike Global

Teaching Offensive, an Analysis

▶ Challenge-Based Learning in a Serious Global Daniel Kasnick2, Sam Romershausen2 and

Game Newton Lee1,2
1
Institute for Education, Research, and C
Scholarships, Los Angeles, CA, USA
2
Vincennes University, Vincennes, IN, USA

Convergence
Synonyms
▶ Cross-cultural Game Studies
Esports; First-person shooter; Free-to-play; Loot
box; Mod

Definition
Conversion
First-person-shooter (FPS) ¼ a genre of games
▶ Virtual Reality Stereo Post-Conversion After where the camera focuses on a gun (or other
Effects Workflow weapon) where the objective is to eliminate spe-
cific targets.
E-sports ¼ a sports competition with video
games as the focus rather than traditional sports.

Convolutional Neural
Network (CNN) History of Counter-Strike Global
Offensive
▶ American Sign Language Detection
Counter-Strike: Global Offensive (often abbrevi-
ated as CS:GO) is a team-based online first-
person-shooter for Windows, OS X, Linux,
Xbox 360, and Playstation 3 platforms in 2012.
CS:GO was developed and published by Valve
Cooking Games (Moore 2018). Counterstrike was originally a
mod for a game, also developed by Valve, called
▶ On Computer Games About Cooking
Half-Life. A mod is an unofficial change to a
video game created by players of the game.
Mods can range from changing the way a game
looks to changing the way a gamemode would
normally work, in essence creating its own unique
Corona identity.
Since the early 2000s, Counter-Strike has
▶ Protection Korona: A Game Design on evolved into its own standalone game franchise.
Covid-19 CS:GO. In 2018, Valve changed CS:GO’s pricing
484 Covid-19

method, switching to a free-to-play model by ones being skin trading. Skins are just customiz-
using revenue from cosmetic items to supplant able wraps that several artists develop. They can
the up-front cost. Free-to-play games allow access be obtained from loot boxes. A loot box is an item
to a large part of the gameplay activities without that holds several other random items that can be
up-front monetization. In-game purchases for cos- used to change an item’s appearance, in-game
metic items offset the lack of up-front revenue. equipment, and other gameplay elements. It is
Counter-Strike’s most popular feature is its often used as a way to monetize a game further.
online matchmaking. It is a 5v5 team-based Some forms of Loot Boxes have been speculated
game that involves two sides, called the Terrorists to be a form of gambling. Some exceedingly rare
and Counter Terrorists. The Terrorists are respon- and exclusive items have been sold on markets for
sible for planting a bomb and ensuring that it over $30,000 USD.
explodes, while the Counter Terrorists are respon- Counter-Strike: Global Offensive was well
sible for defending bomb sites and defusing the received by critics at the time of release, with
bomb. Players do not respawn after they die. Each most ranging from 8/10 to 9.5/10. It has a Meta-
team must strategize and/or counteract the oppos- critic score of 83 and a user score of 7.5
ing team’s plays and be able to break through (Metacritic 2012). After an update in 2013, an
enemy lines or keep their defenses up; however, economy around the in-game skins cultivated sev-
the fast pace of Counter Strike means that con- eral websites that allowed users to gamble their
stantly being able to communicate efficiently and skins, including underaged players. Valve took
thoroughly to teammates can prove to be a chal- steps in the following months to dissuade gam-
lenge. For example, if a team’s takeover strategy bling sites from operation. In 2018, several coun-
failed, the team has to quickly think of a new tries passed antiloot box laws as they determined
strategy in order to adapt, rise, and overcome. that they were gambling; Valve changed the game
There are many problems that a team must solve in the Netherlands and Belgium to comply with
on the fly, but are not directly told. Some include these laws by preventing the opening of loot
communication, playing as a team, doing your boxes.
part in the match, etc. There are many scenarios
that make the game enticing enough that most
seasoned veterans return season after season; the Cross-References
team that either kills all other players on the
opposing team or blows/defuses the bomb wins ▶ First-Person Shooter Games, a Brief History
the round. CS:GO has a large competitive scene.
There are massive competitive tournaments with
many different professional Counter Strike References
players that play for sometimes millions of dol-
lars. These tournaments, such as DreamHack, Metacritic: Counter-strike: Global offensive. https://www.
metacritic.com/game/pc/counter-strike-global-offen
ESL, and more, hold some of the most viewed
sive (21 August 2012)
e-sports tournaments. Moore, B.: Counter-strike: Global offensive (for PC)
CS:GO allows players to host their own servers review. https://www.pcmag.com/reviews/counter-
that have their own modded game modes. Other strike-global-offensive-for-pc (13 December 2018)
fan-created game-modes include bunny-hop,
which is hopping from platform to platform to
reach the end in the fewest number of mistakes.
There are some other game modes that have gath- Covid-19
ered an audience, but run on small private servers.
There are other features that a large part of CS: ▶ Protection Korona: A Game Design on
GO’s player base partake in. One of the major Covid-19
Cross-cultural Game Studies 485

cross-cultural lenses has potential to expand new

Creating Graphics findings and knowledge on video games and
culture.
▶ Teaching Computer Graphics by Application

Culture in Game Studies

C
Credit Skipping Game studies are increasingly focusing on game
culture in diverse local contexts (Liboriussen and
▶ Speedrunning in Video Games Martin 2016). Whether physically or in cyber
space, in this era of digital transformation, video
games, their creation and distribution, content and
gaming experiences are consistently crossing cul-
Cross-cultural Game Studies tural boarders, showcasing the complexity associ-
ated with understanding games and their
Yukiko Sato surrounding phenomena. This obvious integration
Cygames, Inc., Shibuya, Tokyo, Japan and transmission of culture and games opened
new perspectives to study and clarify the differ-
ences and similarities of games and culture as well
Synonyms as its transition in several regions, which is signi-
fied by an increase in cross-cultural aspects in
Comparative studies; Convergence; Cross- game studies. This entry introduces the concept
cultural studies; Game studies; Globalization; of culture and games, and current research on
Localization games and culture conducted through the lens of
cross-cultural studies.
The concept of culture is notoriously hard to
Definitions define. Culture has been considered from a variety
of academic viewpoints and theories, character-
Cross-cultural game studies aim to compare and ized by several layers, levels, or dimensions
clarify differences and similarities of games and (Hofstede 2011; Schwartz 1999), and scientists
culture across two or more cultures on a regional have discussed how cultures evolve through glob-
or national level. The elements of culture in game alization (Reinecke and Bernstein 2013). The cul-
studies are understood as a shared way of think- ture of video games has been shaped from various
ing, behavior, language, value, belief, and gaming interdisciplinary perspectives, focusing on game
practices, which surround game cultures in the cultures understood as games as cultural artifacts,
form of games as cultural artifacts, or as parts of or as parts of subcultures formed by specific gam-
subcultures formed by specific communities. Cur- ing communities, defined by social practices, and
rent cross-cultural game studies investigate and through the discourses of the video game industry,
compare game contents, game markets, the gam- media, and academia (Shaw 2010). Mäyrä (2010)
ing industry and production, player behavior as views culture as a shared way of thinking, behav-
well as game and player interaction. Methods ior, values, and gaming practices among individ-
range from qualitative analyses, such as compar- uals in the micro- and macro-levels of society.
ative case studies, literature reviews, empirical Such cultures result in forming subcultures,
interviews, and text analysis, to quantitative which are visible aspects of specific language
methods of surveys, statistical tests, and natural use, behavior, and beliefs among a particular com-
language processing. The perspective of studying munity. In terms of levels of games and culture,
games and their surrounding contexts from Elmezeny and Wimmer (2018), p. 82) introduce
486 Cross-cultural Game Studies

three levels of defining game cultures: 1) the Media 2020). Game contents produced in one
macro-level focuses on the overall culture of culture are enjoyed in other cultures, for example
games, gamers, and gameplay in regional or games developed in East Asia are consumed by
national fields worldwide, 2) the meso-level Western audiences (Consalvo 2006), influenced
focuses on culture among multiple games or com- by localization processes (Mangiron and
munities with unified characteristics, such as PS4 O’Hagan 2006). Game-related research so far
gamers and Nintendo gamers, and 3) the micro- has utilized a cross-cultural perspective to com-
level focuses on culture of a specific game or pare and understand the differences and similari-
community, such as World of Warcraft culture or ties between game markets, production, design,
German FIFA culture. and user experience. In this background, this entry
In contemporary society, video games are the introduces the strands of cross-cultural game
latest form of cultural expression, helping us to research conducted on a macro level, which com-
understand modern culture and society (Muriel pare the overall differences and similarities
and Crawford 2018). As games are becoming a among characteristics of culture of games,
major cultural influence, game studies evolved not gamers, and gameplay on a regional or national
only as a field focusing on games, playing, and level. This entry also discusses current themes and
related phenomena (Mäyrä 2008, p. 11) but also challenges of cross-cultural game studies.
hold the power to critically analyze power rela-
tions in our culture and society by studying game Video Game Market
contents, audiences, and production processes As the global video game market continues to
(Nieborg and Hermes 2008). Current game stud- grow, researchers study the history, expansion,
ies are formed by six major domains: 1) studying and emergence of the games industry in global
games as artifacts or texts (e.g., Mäyrä 2008; and local game markets and explains how the
Schell 2008; Juul 2011), 2) analysis of player innovation of games and technology influence
behavior (Charles et al. 2005; Braun et al. 2016), our contemporary and future culture and society.
3) globalization and localization of the games Dating back to the invention of digital games in
industry (Consalvo 2006), 4) game development the 1950s, the hit and crash of home consoles
(Engström 2020), 5) gaming culture (Muriel and from the Magnavox Odyssey (1972) to the Atari
Crawford 2018), and 6) interaction between game 2600 (1977), the competition between Nintendo,
and player (Caroux et al. 2015). However, few Sega, and Sony in the early 1980s, joined by
have examined games and related phenomena Microsoft in 2001, and mobile games proliferat-
from a cross-cultural perspective, that is, sought ing in the present, researchers have been studying
the difference and similarities among game mar- the process of how the game markets changes our
kets, game production and design, game content, culture (Wolf 2008, 2012). Looking back at the
as well as the meaning and perception on games history of fast-moving video games leads to the
by game designers and players in geographically realization of how conventional media, entertain-
and socially different contexts. ment, and technology converge to create a new
Looking at the current situation in concern of phenomenon and a new part of culture in our
video games, game developers and platforms society (Wolf 2008).
from different cultures, such as Tencent (China), Market analysts such as Newzoo (2020) track
Sony (Japan), EA (America), and Ubisoft the developments in the rapidly changing global
(France), compete on a global level, targeting games market. They provide comparisons of mar-
local and global audiences (Newzoo 2020). ket shares and growth for different regions. Unity
Games are frequently developed through collabo- Technologies (2020) releases detailed reports
ration across culturally diverse regions. Death focusing on mobile gaming trends. Computer
Stranding was for example created by the promi- Entertainment Supplier’s Association (2020) also
nent Japanese game designer Hideo Kojima col- delivers annual white papers to report the gaming
laborating with Western actors and artists (Game trends in the world.
Cross-cultural Game Studies 487

Within academia, Johns (2006) closely studies current (console) video game industry is a hybrid
and compares the expansion of hardware and soft- that mixes Japanese and American businesses
ware production networks of game markets, practices and cultures (Consalvo 2006, p.118).
focusing on three supra-regional sections, North As studies on globalization and transition of
America, Europe, and Asia-Pacific. Ip (2008) major game industries continue, other researchers
investigates the games market chronologically, argue the importance to study smaller regions of
to see how game titles and platforms created in game development. Studying the characteristics C
different cultures evolved through convergence of of game development in three different regions
technology and content. However, as Šisler et al. in the Nordic region, India, and China, Toftedahl
(2017) state, current research on games and et al. (2016) point out that the current understand-
focuses on particular regions, while smaller ings of video game development have been asso-
regions are ignored. As the games market further ciated mostly with Japan and the USA, raising
globalizes, partially through online and concerns to the standardization of the develop-
smartphone technologies, there are needs to sys- ment practices of video game development and
tematically analyze differences and similarities in limitations to understand the development process
game markets from a broader perspective to in other regions. This shows one challenge of
understand how games and their meaning change cross-cultural games research. Therefore,
in our globalizing society. researchers capable of cross-regional and
industry-academia collaborative research need to
Video Game Industry and Production take the lead to conduct cross-cultural game
Studies on game production, industry, and cultural development research in order to show the differ-
influence is one of the earliest themes focused on ences and similarities of global and local game
in cross-cultural games research. As it is a chal- industries, contributing to the next generation of
lenge for academics to access information from game development.
the games industry (Engström 2019), this field of
study sheds light upon how games are designed Game Content
and developed, how developers are structuralized, As game developers seek to sell their games glob-
and how industry across different regions merge ally, the process of adapting products, contents,
to disseminate games in a different cultures. Pre- and services to suit the players in each region
vious comparative research heavily focuses on the plays a major role. Carlson and Corliss (2011)
difference between Eastern and Western game show how cultural differences influence game
development, especially on the differences production and game content. Hence, localization
between Japan and the West. Compared to West- is essential to enable video games to be played in a
ern game design, Aoyama and Izushi (2003) note similar way in different markets (O’Hagan 2009).
the pervasiveness of the manga and anime indus- Arguably, it is important to ensure that players
try in the Japanese games industry. The similari- enjoy a game created in another culture in the
ties between Western countries and differences of context of their own culture (Mangiron and
the Japanese games industry are further studied by O’Hagan 2006). Localization involves making
comparing the evolution and transition of the decisions about culturally appropriate images,
video games industry in the USA, the UK, and character designs, translation, gameplay mechan-
Japan. Johns (2006) claims that North American ics, the technical nature of software, and negotia-
and European game companies became closer tion of national regulatory boards (Carlson and
through internationalization, while Japanese Corliss 2011). Case studies on cultural differences
game companies grew isolated, creating games focusing on the translation of the original game
influenced by the manga and anime industry. In and its localized version are conducted by
contrast, Consalvo (2006) also explores the com- O’Hagan and Mangiron (2004) and Mangiron
plex transnational relations between the Japanese and O’Hagan (2006), arguing that the cultural
and American games industries, arguing that the differences impact the characterization of the
488 Cross-cultural Game Studies

main characters. Carlson and Corliss (2011) pro- of American and Chinese participants.
vide comprehensive examples of what kind of Researchers have also investigated behavior
game contents are altered when media travel focusing on online resources. Šisler et al.
across national borders, for example how game (2017) focus on user behavior through social net-
designers consider to create a generic character works, analyzing what fans liked on Facebook,
instead of a specific one, alter the speed of char- based on an original method of Normalized Social
acters movements to fit cultural preferences, as Distance, calculating the distances between vari-
well as change how blood is displayed in a game ous social groups. Zagal and Tomuro (2013) com-
due to rating board regulations. Ng (2006) studies pare Japanese and American user reviews by
how game contents differ between Asian regions, utilizing statistics and natural language pro-
stating that players add new context to games, and cessing. On the other hand, qualitative research
also to the culture itself, by consuming and investigated players’ experiences more closely by
hybridizing with other popular culture. analyzing player data through text analyses. For
example, Brückner et al. (2019) utilize a
Player Behavior and Experience Grounded Theory Approach to compare German
Analyzing the interaction of game and player and Japanese professional and user reviews to
within their respective contexts has been a central identify differences between German and Japa-
task of game studies (Mäyrä 2008). In cross- nese player.
cultural games research, researchers have been In terms of regions, cross-cultural research
analyzing the difference of users’ playstyles, studies heavily involve participants in the USA
their perception of games. Game user related stud- and Germany, followed by other countries. As
ies can be sorted by two axes of 1) research Šisler et al. (2017) mention, few research projects
methods, that is, qualitative or quantitative and concentrate on studying game cultures in Eastern
2) target regions, that is, Asia-West or Northern Europe or the Middle East. As such, many possi-
America-Western European, with the exception of bilities to comparative user studies across differ-
a study on game culture and players in the ent cultures and regions are open and researchers
Middle East. should unite as well as systemize their methods to
In quantitative studies, large-scale surveys are exhaustively define the differences and similari-
conducted and statistically analyzed. Bialas et al. ties among video game players.
(2014) examine the playstyles of users in eight
different Western countries (Australia, Canada,
Finland, Germany, the Netherlands, Sweden, the Summary
United Kingdom, and the United States) playing
Battlefield 3 with statistical ANOVA tests. Ćwil Cross-cultural game studies aim to clarify and
and Howe (2020) combine statistical tests of chi- compare differences and similarities of games,
square, correlation, and MANOVA to identify game markets, game production and design, as
differences of gamer identities between players well as the meaning and perception on games by
in the USA and Poland. Researchers also designed game designers and players across two or more
itemized behavior measurements of video game cultures on a regional or national level. Culture in
players and test the measurements’ validity game studies is understood as games as cultural
among different cultures. Koban and Bowman artifacts or as parts of subcultures, consisting of a
(2020) utilize a five-factor Video Game Demand shared way of thinking, behavior, values, lan-
Scale (VGDS) to measure demands of video guage, belief, and gaming practices among spe-
games players in Germany based on the results cific groups or individuals. Methods utilized in
of measurements in the USA. Kahn et al. (2015) such cross-cultural studies include qualitative
propose a new scale to examine player motivation analyses of case studies, literature reviews,
Cross-cultural Game Studies 489

interviews, and text analysis, as well as quantita- JRPGs in Germany and Japan. Trans. Digit. Games
tive methods of surveys and statistical tests, net- Res. Assoc. 4(3), 209–243 (2019). https://doi.org/10.
26503/todigra.v4i3.105
work analysis, and natural language processing. Carlson, R., Corliss, J.: Imagined commodities: video
Previous cross-cultural game studies mainly focus game localization and mythologies of cultural differ-
on the major fields of game markets, game pro- ence. Games Cult. 6(1), 61–82 (2011). https://doi.org/
duction and design, game content, and player 10.1177/1555412010377322
behavior as well as game and player interaction.
Caroux, L., Isbister, K., Le Bigot, L., Vibert, N.: Player-
video game interaction: A systematic review of current
C
However, more close analysis on granular fields is concepts. Comput. Hum. Behav. 48, 366–381 (2015).
yet to be conducted, such as the game market of https://doi.org/10.1016/j.chb.2015.01.066
mobile games and application platforms, regula- Charles, D., McNeill, M., McAlister, M., Black, M.,
Moore, A., Stringer, K., Kücklich, J., Kerr, A.: Player-
tions, the role of game designers, game mecha- centred game design: Player modelling and adaptive
nisms, detailed analysis of game content, and digital games. In: Proceedings of DiGRA 2005 confer-
player perception. Moreover, most studies have ence (DiGRA ‘05), pp. 285–298. DiGRA (2005)
the tendency to compare the regions of the West Computer Entertainment Supplier’s Association: CESA
games white paper. https://www.cesa.or.jp/survey/
and Asia or the USA and Japan to be in particular; book/hakusho.html (2020). Accessed 16 Nov 2020
however in this global era, where cultures Consalvo, M.: Console video games and global corpora-
diversely cross with one another, researchers tions: Creating a hybrid culture. New Media Soc. 8(1),
should aim to study and compare regions on indi- 11 7 – 1 3 7 ( 2 0 0 6 ) . h t t p s : / / d o i . o r g / 1 0 . 11 7 7 /
1461444806059921
vidual levels, to seek the transition and mixture of Ćwil, M., Howe, W.T.: Cross-cultural analysis of gamer
games and their surroundings. Such cross-cultural identity: A comparison of the United States and Poland.
game studies lead to clarifying the changes of our Simul. Gaming. 51(6), 785–801 (2020). https://doi.org/
contemporary understanding of culture in the 10.1177/1046878120945735
Elmezeny, A., Wimmer, J.: Games without frontiers:
globalized and digitized world. A framework for analyzing digital game cultures com-
paratively. Media Commun. 6(2), 80–89 (2018).
https://doi.org/10.17645/mac.v6i2.1330
Cross-References Engström, H.: GDc vs. DiGRA: Gaps in game production
research. In: Proceedings of the 2019 DiGRA interna-
tional conference (DiGRA ‘19). DiGRA (2019)
▶ Games and the Magic Circle Engström, H.: Game development research. The Univer-
sity of Skvde, Skovde (2020)
Game Media: 5 greatest video game collabs. https://news.
game.co.uk/video-gamecollabs/ (2020). Accessed
References 9 Nov 2020
Hofstede, G.: Dimensionalizing cultures: The Hofstede
Aoyama, Y., Izushi, H.: Hardware gimmick or cultural model in context. Psychol. Cult. 2(1) (2011). https://
innovation? Technological, cultural, and social founda- doi.org/10.9707/2307-0919.1014
tions of the Japanese video game industry. Res. Policy. Ip, B.: Technological, content, and market convergence in
32(3), 423–444 (2003). https://doi.org/10.1016/S0048- the games industry. Games Cult. 3(2) (2008). https://
7333(02)00016-1 doi.org/10.1177/1555412008314128
Bialas, M., Tekofsky, S., Spronck, P.: Cultural influences Johns, J.: Video games production networks: Value cap-
on play style. In: 2014 IEEE conference on computa- ture, power relations and embeddedness. J. Econ.
tional intelligence and games, pp. 1–7. IEEE (2014). Geogr. 6(2), 151–180 (2006). https://doi.org/10.1093/
https://doi.org/10.1109/CIG.2014.6932894 jeg/lbi001
Braun, B., Stopfer, J.M., Müller, K.W., Beutel, M.E., Juul, J.: Half-real: Video games between real rules and
Egloff, B.: Personality and video gaming: comparing fictional worlds. MIT Press, Cambridge, MA (2011)
regular gamers, non-gamers, and gaming addicts and Kahn, A.S., Shen, C., Lu, L., Ratan, R.A., Coary, S., Hou,
differentiating between game genres. Comput. Hum. J., Meng, J., Osborn, J., Williams, D.: The trojan player
Behav. 55, 406–412 (2016). https://doi.org/10.1016/j. typology: A cross-genre, cross-cultural, behaviorally
chb.2015.09.041 validated scale of video game play motivations.
Brückner, S., Sato, Y., Kurabayashi, S., Waragai, I.: Comput. Hum. Behav. 49, 354–361 (2015). https://
Exploring cultural differences in game reception: doi.org/10.1016/j.chb.2015.03.018
490 Cross-cultural Studies

Koban, K., Bowman, N.D.: Further validation and cross- Game development in Nordic countries, China and
cultural replication of the video game demand scale. India. In: Proceedings of the 2016 DiGRA international
J. Media Psychol. (2020). https://doi.org/10.1027/ conference (DiGRA ‘16), DiGRA (2016)
1864-1105/a000280 Unity Technologies: The 2020 mobile game monetization
Liboriussen, B., Martin, P.: Regional game studies. Game report. https://create.unity3d.com/2020-mobile-game-
Stud. 16(1) (2016). Accessed 28 Dec 2020. http:// monetization-report (2020). Accessed 9 Nov 2020
eprints.nottingham.ac.uk/39228/1/liboriussen Wolf, M.J.: The video game explosion: A history from
Mangiron, C., O’Hagan, M.: Game localisation: PONG to Playstation and beyond. Greenwood Press,
Unleashing imagination with “restricted” translation. Westport (2008)
J. Specialised Translat. 6, 10–21 (2006). Accessed Wolf, M.J.: Before the crash: Early video game history.
26 Dec 2020 Wayne State University Press, Detroit (2012)
Mäyrä, F.: An introduction to game studies. SAGE Pub- Zagal, J.P., Tomuro, N.: Cultural differences in game
lishing, Thousand Oaks (2008) appreciation: A study of player game reviews. In:
Mäyrä, F.: Gaming culture at the boundaries of play. Game FDG 2013 conference proceedings, pp. 86–93 (2013).
Stud. 10(1), 1–6 (2010). Accessed 28 Jan 2021 Accessed 28 Jan 2021
Muriel, D., Crawford, G.: Video games as culture: Consid-
ering the role and importance of video games in con-
temporary society. Routledge, Oxon, UK/New York
(2018)
Newzoo: Global games market report. https://resources.
newzoo.com/hubfs/Reports/2020_Free_Global_ Cross-cultural Studies
Games_Market_Report.pdf (2020). Accessed 9 Nov
2020
Ng, B.W.M.: Street fighter and the king of fighters in Hong ▶ Cross-cultural Game Studies
Kong: A study of cultural consumption and localization
of Japanese games in an Asian context. Game Stud.
6(1) (2006). Accessed 11 Jan 2021. http://www.
gamestudies.org/0601/articles/ng
Nieborg, D.B., Hermes, J.: What is game studies anyway? Cross-culturalism
Eur. J. Cult. Stud. 11(2), 131–147 (2008)
O’Hagan, M.: Towards a cross-cultural game design: an
explorative study in understanding the player experi- ▶ Diversity in Gaming and the Metaverse
ence of a localised Japanese video game. J. Specialised
Translat. 11, 211–233 (2009)
O’Hagan, M., Mangiron, C.: Games localization: When
arigato gets lost in translation. In: New Zealand
game developers conference proceeding, Crossplatform
pp. 57–62 (2004)
Reinecke, K., Bernstein, A.: Knowing what a user likes:
A design science approach to interfaces that automati- ▶ Redesigning Games for New Interfaces and
cally adapt to culture. MIS Q. 37(2), 427–453 (2013). Platforms
https://doi.org/10.25300/MISQ/2013/37.2.06
Schell, J.: The art of game design: A Book of Lenses.
Elsevier/Morgan Kaufmann, San Francisco (2008)
Schwartz, S.H.: A theory of cultural values and some
implications for work. Appl. Psychol. 48(1),
23–47 (1999). https://doi.org/10.1111/j.1464-0597. Crowd Animation
1999.tb00047.x
Shaw, A.: What is video game culture? Cultural studies and ▶ Crowd Simulation
game studies. Games Cult. 5(4), 403–424 (2010).
https://doi.org/10.1177/1555412009360414
Šisler, V., Švelch, J., Šlerka, J.: Video games and the
asymmetry of global cultural flows: The game industry
and game culture in Iran and the Czech Republic. Int. Crowd Evacuation
J. Commun. 11, 3857–3879 (2017) Accessed 2020
Nov. 16
Toftedahl, M., Berg Marklund, B., Engström, H., Back- ▶ Crowd Evacuation Using Simulation
lund, P.: Global influences on regional industries: Techniques
Crowd Evacuation Using Simulation Techniques 491

macroscopic, data-driven, and hybrid. After that

Crowd Evacuation Using we present the methods based on psychological
Simulation Techniques factors and multi-cues. The crowd simulation
techniques can be integrated into a crowd evacu-
Sai-Keung Wong ation system to simulate the interaction among
National Chiao Tung University, Hsinchu, Taiwan individuals and crowds. The following topics are
route computation and crowd evacuation tech- C
niques. Finally, we conclude this chapter and
Synonyms highlight future research directions.

Crowd evacuation; Crowd simulation; Macro-

scopic simulation; Microscopic simulation; Vir- Crowd Behavior Analysis
tual environments
Crowd behaviors (Sakour and Huosheng 2017) can
be classified into three types: (1) individual,
Definitions (2) interactions between individuals, and (3) crowd.

Crowd Evacuation Simulation: Using simula- 1. Individuals can make a good decision or bad
tion techniques to simulate the motion of crowds decision in escaping from dangerous regions.
in evacuation in virtual environments. They may be panic if the situation is out of
control. Experienced individuals can calm
down and help others.
Introduction 2. Interactions between individuals: Inexperi-
enced individuals may follow others even
Crowd evacuation is important in building design, though the others may make a wrong decision.
road infrastructure design, and city planning. Panic individuals may need the help from
A wide range of techniques have been proposed others to calm down. Individuals may share
for crowd evacuation. The major aims of the stud- ideas with each other to build up their own
ies on crowd evacuation include: (1) simulating cognitive map.
the individual and crowd behaviors, (2) identify- 3. Crowd: A crowd consists of individuals. Thus,
ing the potential problems of building structures, a crowd may increase its density as more indi-
(3) the effects of obstacles and exits, (4) optimal viduals join it. However, a higher risk level
route computation. In an emergency evacuation, may be caused due to a degrading comfort
uncontrolled actions are observable in a massive level of the individuals.
crowd due to the influences of individuals. How- A successful crowd evacuation technique
ever, there are ethical issues to perform real life should capture the essential behaviors of individ-
experiments. Therefore, using mathematical uals and crowds. We can model crowds, groups,
models and computer simulations are essential in and individuals for scripted, reactive, and guided
studying crowd evacuation. The major goal of behaviors (Musse and Thalmann 2001).
crowd evacuation simulation is that we would
like to find the appropriate solutions to reduce
fatality in emergency evacuation. Crowd Simulation
The organization of the rest of this chapter is as
follows. In the beginning, we present crowd Crowd simulation techniques can be classified
behavior analysis. Then crowd simulation tech- into macroscopic, microscopic, data-driven and
niques are presented, including microscopic, hybrid.
492 Crowd Evacuation Using Simulation Techniques

Macroscopic Crowd Simulation @ri @ @fi @

 þ r gðrÞf 2i þ
The macroscopic techniques focus on the @t @x i @x @x
aggregate behaviors of crowds, evaluation of
@fi
building structures and performance. The  ri gðrÞf 2i ðrÞ
@x
computation cost is relatively lower than the
microscopic techniques. In macroscopic ¼0 ð1Þ
crowd simulation, the individuals are grouped
together. There are no interactions between and
individuals in the same group. The motion
of pedestrians can be modeled as a kind of 1
gðrÞf i ðrÞ ¼ ,i
fluid. @fi
2
@fi
@x þ @y
We can derive a nonlinear, time-dependent
equations for pedestrian flow based on three ¼ 1, ⋯, N: ð2Þ
hypotheses (Hughes 2002) as follows:
One possible choice for fi(r) is that fi(r) is linear
1. Hypothesis 1: The speed of pedestrians of a in r, such as fi(r) ¼ βi(A  Br), i ¼ 1,   , N.
single type in multiple type flow is determined Here, βi, A, B should be selected appropriately.
by the function f(r) but where r is the total A crowd of various types of pedestrians walk
density rather than the density of a single toward different objectives or with various speed
pedestrian type. relationships (Hughes 2003), which are not
2. Hypothesis 2: A potential field exists for observed in a classical fluid. The shear force
each pedestrian type such that pedestrians between the pedestrians and ground underneath
move at right angles to lines of constant should be considered.
potential.
3. Hypothesis 3: Pedestrians seek the path that Microscopic Crowd Simulation
minimizes their (estimated) travel time, but In microscopic crowd simulation, a crowd is
temper this behavior to avoid extremely high treated as a set of particles (or agents). Each
densities. agent has its own attributes, including position,
velocity, destination, and psychological factors.
The pedestrians are grouped into different Furthermore, agents can have interactions among
types. Furthermore, The different groups each other. Although the computation cost is rel-
make their way to one of a finite number of atively higher than macroscopic techniques, the
objectives. Let ri be the density of pedestrians detailed simulation result can be achieved such as
of particular type i. The total density is r ¼ panicked individuals.
N
i¼1 ri where N is the number of pedestrian
,
The simulation space for simulating crowd can
types. Then the flow of a particular type i of be continuous or discrete. In a continuous space,
pedestrians is ri fi(r), where fi(r) is a specified the positions of agents are determined by their
function of total pedestrian density for pedes- current positions and velocities. However, in a
trians of type i. Assume that ’i is the potential discrete space, the simulation space is discretized
for type i of pedestrians moving over the (x, y) into a grid and the agents can move only to their
floor plane. Denote g(r) as the factor related neighboring cells. We give examples for these two
to the discomfort of the crowd at a given simulation spaces as follows.
density, and (x, y,t) denotes the horizontal For a continuous space, the new position
space and time coordinates. For each type p(t + Δt) of an agent is updated based on its
i of pedestrians, the governing equations are current position p(t), velocity v(t), and time step
as follows: size Δt, i.e.,
Crowd Evacuation Using Simulation Techniques 493

pðt þ DtÞ ¼ pðtÞ þ vðtÞDt, ð3Þ where G(i, j) is the set of neighboring cells of (i, j);
nk‘ and wk‘ indicate whether the grid cell (k, ‘) is
where t is the current time. The velocity of the occupied or there is an obstacle, respectively.
agent is affected by factors such as the velocities We can combine cellular automata with game
of neighboring agents, crowd density, and theory to perform crowd evacuation (Zheng and
obstacles. Cheng 2011). The main idea is to apply game
For a discrete space, an agent can jump to a theory for determining the movement directions C
position of a set of fixed positions. Cellular autom- of agents. The agents may consider to cooperate
ata are a popular class of discrete models. In a or adopt a competitive manner while they evacu-
cellular automaton model, it has a grid of regular ate. The following three results are obtained:
cells and there is at most one agent at a grid cell at a
time. An agent can move from its current grid cell 1. A longer evacuation time is required and also
to a neighboring grid cell within a simulation step. the frequency of cooperation reduces for
Thus, for a regular grid with square cells, there are higher degree of emergency.
at most eight neighboring cells for one grid cell. 2. Hyper-rationality inhibits cooperation and
Discrete Techniques. We use a theatre to illus- delays evacuation times, which may lead to
trate this kind of techniques. The space of the crowd disaster.
theatre is discretized as a regular grid with square 3. The frequency of cooperation increases for
cells. We can construct a floor field by assigning higher degree of imitation among evacuees
higher probability to aisle regions than seat but also leads to longer evacuation times.
regions in a theatre (Yang et al. 2010). At an exit Continuous Techniques. Helbing and
area, an information board shows the real time Molnar (1995) proposed a force-based model
information (e.g., exit density) to the agents to to simulate the behaviors of pedestrians. The
aid them make movement decision. A grid cell surrounding environment and agents exert
dimension is 0.5 m  0.5 m and it has a total forces on an agent. The main idea is to compute
attraction value which is used for computing a the total force on an agent. The forces should
probability of an agent to move to it. An agent be considered as influences of the surrounding
has a higher chance to move to a neighboring cell objects to the agent. We consider that the net
with higher total attractive value. The total attrac- force exerting on an agent is f. Then the veloc-
tion value of a cell (i, j) is computed as ity is updated as

N ij ¼ eks ðS1 þS2 Þþkr Rþkd D , ð4Þ vðt þ DtÞ ¼ v þ

f
Dt, ð6Þ
m
where S1 is the attraction of exit position, S2 is the
where m is the mass of the agent. To determine the
attraction of aisle region; R is the repulsive force
velocities of the agents, we can also compute
between agents or between agents and obstacles;
feasible movement directions of agents based on
D is the influence factor of neighboring agents
reciprocal velocity obstacles (Van den Berg et al.
(due to psychology); ks, kd, kd are the weights for
2008). The possible movement direction of an
setting the dominant roles of the parameters. R is a
agent is computed based on the positions and
non-positive value and D is a non-negative value.
velocities of the other agents. The feasible direc-
Thus, R and D play a negative and positive effect
tion is picked for avoiding collisions. Further-
for choosing the cell, respectively. The transition
more, a collision prediction scheme can be
probability to cell (i, j) is
performed to evaluate the possible future colli-
sions of each agent (Karamouzas et al. 2009).
N ij
pij ¼ , ð5Þ Each agent adapts a new route as early as possible
ðk,‘Þ∈Gði,jÞ k‘  ð1  nk‘ Þ  wk‘
N
in order to minimize the influence of others. In
494 Crowd Evacuation Using Simulation Techniques

general, the agent avoids collision with another navigation field can guide a large amount of
agent if they may collide in the earliest contact agents to move to a common destination (Tsung-
time. The result indicates that the approach leads Yu Tsai et al. 2017). However, if the guidance
to shorter and less curved paths. paths intersect with each other or congestion
To achieve biomechanically energy-efficient occurs, we can adjust them by applying particle
and collision-free trajectory, we can apply the swarm optimization to achieve better traveling
principle of the least effort to determine the veloc- time of the agents (Wong et al. 2015). A fitness
ities of the agents (Guy et al. 2010). The instanta- function is used to evaluate the quality of the
neous power (P) spent by a walking agent is guidance paths. The fitness function for a finite
computed as generation period is constructed as follows. All
the agents should move to their destinations
P ¼ e s þ e w k vk 2 , ð7Þ before the simulation process is terminated. Let
N G be the number of agents not reaching the
where v is the instantaneous velocity, and es destination before the simulation is finished.
(measured in J/Kg/s) and ew (measured in Js/Kg/ Denote a particle position x (i.e., a guidance
m2) are constant. Assume that the mass of the path). Then the fitness function is as follows:
agent is m. The total consumed metabolic energy
of the agent while walking along a path is com- F1 ðxÞ ¼ B N G ⁎ T þ aS , ð9Þ
puted as

where S ¼ ‘s Dt
T
, T¯ is the average traveling time, ‘
2
E¼m es þ ew kvk dt: ð8Þ is the average interaction distance of agents, s¯ is
the average desired speed of agents, and α is a
The function E is the subject to be minimized constant. S¯ is the average extra traveling time for
to obtain the path of the agent. all agents. We would like to adjust the guidance
To implement the concept about “right of path so that all the agents should reach the desti-
way,” we can assign agents of different priorities nation. Therefore, we set B N G ¼ 1 for N G ¼
(Curtis et al. 2013). The agents with lower priority 0; and B N G ¼ 1þ N G for N G  1. The term ‘s
give way to those with higher priority. This is represents the extra average traveling time of
crucial at narrow regions. If there is no “right of agents per simulation time step due to collision
way” mechanism, the agents may block at a nar- resolution for agents. The term DtT
is the average
row passage. simulation time steps for agents to move to their
It is efficient to compute the local views of destinations. The guidance paths are adjusted iter-
agents to determine the movement of the agents atively until the fitness value reaches a minimum
(Fu-Shun Li and Sai-Keung Wong 2016). Each value (Wong et al. 2015).
agent has the local view of its neighborhood
region and it picks the feasible direction that Data-Driven and Hybrid Techniques
costs the least effort in term of the turning direc- To calibrate crowd models and group models
tion of the agent. Furthermore, the speed of agents (Kang Hoon Lee et al. 2007), we can adopt the
is also affected by the local density. For higher evolutionary optimization to compute parameters
crowd density, the speed of an agent is slower. from real videos (Johansson et al. 2007). To
Conversely, if the crowd density is low, the agent achieve better calibration results, we can classify
can move at its desired speed. The agents can also the types of crowd based on steering contexts
follow some guidance paths to move. (e.g., groups crossing and chaos) (Boatright et al.
A navigation field can be constructed based on 2015). In this way, we can capture the main char-
the guidance paths. Each cell of the navigation acteristics of crowds in each steering context.
field has a movement direction and the agent at Based on the characteristics of a steering context
that cell follows the direction to move. The for a crowd, the best crowd-simulation technique
Crowd Evacuation Using Simulation Techniques 495

can be employed to simulate the crowd. Further- cellular automata can be adopted. Furthermore,
more, we can measure the density of a crowd to controlled dynamic exit signs can be employed
evaluate the similarity between simulation results for evacuation routes (Desmet and Gelenbe 2014).
and real crowd motion in videos (Lerner et al. To compute evacuation routes for three-
2009). The density measure takes into account dimensional networks, we can employ the
the local crowd densities surrounding a subject pseudo-polynomial-time dynamic programming
agent. Based on the density measure, we can algorithm (Tang et al. 2014). C
compute similarity scores for the simulation For evenly distributed agents in an area, we can
results and then adjust the calibrated parameters. compute evacuation routes by using the genetic
algorithm (Abdelghany et al. 2014). Furthermore,
Psychological and Multi-Cue Methods we can apply an evolutionary algorithm to com-
To achieve realistic simulation of high-density pute the best region assignment that describes
crowd, we should combine physical force; agent how agents in regions are assigned to exits of a
personalities; and psychological, physiological, building or an open area (Jinghui Zhong et al.
and geometrical rules to simulate the local motion 2015). A particle encodes the region assignment
of autonomous agents (Pelechano et al. 2007). and the subregions are encoded as chromes. Muta-
Furthermore, to handle a large amount of heterog- tion and crossover are performed to update the
enous agents, we can adopt a multi-agent para- particles based on the result obtained from a cel-
digm in a distributed simulator (Dimakis et al. lular automata-based simulation.
2009). Psychological parameters in crowd models An alternative way to compute evacuation
make crowds exhibit various personalities and routes is to apply agent-based simulation. It is
emotions (Durupınar et al. 2016). A virtual agent applicable to an environment which can be
consists of features that determine cognitive, per- represented as a road network and a set of obstacles
ceptual, and psychological characteristics. The (Sai-Keung Wong et al. 2016, 2017). Agents move
agents react according to the interaction between along the roads. Along each road, there is a division
the features and environment stimuli. We can point which divides the agents on the road into two
integrate different cues including sound percep- groups moving in the opposite directions. If the
tion, multi-sense attention, and understanding of agents move to a junction, they are split into
environment semantics, to enhance the realism of smaller groups and move along the subsequent
crowd (Kapadia et al. 2015). road segments. The approach computes the divi-
sion points such that the agents around the division
points have similar average evacuation times.
Evacuation Route Computation

There are mathematical models to compute opti- Crowd Evacuation and Building Design
mal routes (Hamacher and Tjandra 2002). An
environment is represented as a graph which con- There are a variety of concerns for crowd evacu-
sists of a set of nodes and a set of edges. The nodes ation. There can be agents of different kinds, such
and edges are the intersections and roads of the as children, parent, and authorities (Tsai et al.
environment. The models require the input of the 2011). It is shown that it is effective for a few
movement speed of a crowd and road capacities. leaders guiding evacuees to safety regions
To evaluate whether a building is well (Pelechano and Badler 2006). There may be a
designed, we can compute the travel time of a negative impact if there were too many leaders
crowd (Thompson and Marchant 1995). The qual- (Yi Ma et al. 2016). Furthermore, the evacuation
ity of an exit assigned to an agent can be evaluated strategies should depend on the types of interac-
based on earliest arrival flows, maximum flows, tion, grouping restrictions, and environment fac-
and minimum cost flows (Dressler et al. 2010). To tors (e.g., hazardous areas) (Rodriguez and Amato
make the problem tractable, the methods based on 2010). Members of the same group should remain
496 Crowd Evacuation Using Simulation Techniques

within a predefined distance of others. The mem- based techniques are devised from the micro-
bers can help each other. Agents may have various scopic and macroscopic techniques. Data-driven
degree of knowledge about the environment. techniques can be employed to calibrate the
We can adopt appropriate methods to perform parameters of various crowd-simulation tech-
evacuation. To perform evacuation in buildings niques. Route-computation techniques are also
caused by fire, we can adopt dynamic network crucial. The best routes can be transmitted to
flows (Hadzic et al. 2011). Regions which are devices of people via wireless communication,
not reachable are removed from the graph. The such as mobile phones (Inoue et al. 2008).
effects of temperature, spread of smoke speed, and It would be compelling to improve the overall
CO (carbon oxide) in fire scenes should also eval- coordination opportunity and evacuation result
uated (Hai-Rong Wang et al. 2014). People should via cooperation of individuals. Efficient and accu-
avoid congestion and move to large space. Fur- rate prediction methods should be developed to
thermore, due to guidance signs, people may prevent disaster from occurring. Due to ethical
move together. Thus the movement speed should issues, Sadiyoko et al. (2012) modeled the psy-
be carefully monitored to avoid congestion. chological behaviors of human based on a psy-
To guide the design of a building, we can chological dynamic model in order to understand
evaluate the effects of the placement of pillars how people interact with each other. The models
and doors (Berseth et al. 2015). Let p be the set of psychological behaviors can be applied to
of parameters of a scene configuration. A crowd robots. For example, the emergence of chaos
flow for a specific scenario with parameterization observed in a simulation result of robots can be
p is defined as useful in predicting when a real crowd will occur
with the same chaos. Consequently, the chaos for
jAc j human could be avoided.
f ðpÞ ¼ , ð10Þ
tavg

where, tavg ¼ a∈A

ta
, |Ac| is the destination, ta is Cross-References
j Aj
the time taken by agent a to reach the destination,
▶ Crowd Simulation
tavg is the average completion time of agents, |A| is
the cardinality of set A. The goal of the study was
to find p which minimizes f(p). The local naviga-
References
tion abilities of the agents affected the results for
different kinds of steering techniques. Hence, Abdelghany, A., Abdelghany, K., Mahmassani, H.,
evacuation techniques based on statistical analysis Alhalabi, W.: Modeling framework for optimal evacu-
are highly demanded. Bidirectional traffic pat- ation of large-scale crowded pedestrian facilities. Eur.
J. Oper. Res. 237(3), 1105–1118 (2014)
terns are greatly affected by door widths.
Berseth, G., Usman, M., Haworth, B., Kapadia, M.,
Faloutsos, P.: Environment optimization for crowd
evacuation. Comput. Anim. Virtual. Worlds. 26(3–4),
Summary 377–386 (2015)
Boatright, C.D., Kapadia, M., Shapira, J.M., Badler, N.I.:
This chapter covers a variety of simulation tech- Generating a multiplicity of policies for agent steering
in crowd simulation. Comput. Anim. Virtual. Worlds.
niques for crowd evacuation. There are two cate- 26(5), 483–494 (2015)
gories of crowd simulation: macroscopic and Curtis, S., Zafar, B., Gutub, A., Manocha, D.: Right of way.
microscopic. Macroscopic-based techniques are Vis. Comput. 29(12), 1277–1292 (2013)
useful in computing the global behaviors of Desmet, A., Gelenbe, E.: Capacity based evacuation
with dynamic exit signs. In: Pervasive Computing and
crowds. To model the individual behaviors and Communications Workshops (PERCOM Workshops),
interactions between individuals, microscopic- 2014 I.E. International Conference, pp. 332–337
based techniques should be employed. Hybrid- (2014)
Crowd Evacuation Using Simulation Techniques 497

Dimakis, N., Filippoupolitis, A., Gelenbe, E.: Distributed 2007 ACM SIGGRAPH/Eurographics Symposium on
building evacuation simulator for smart emergency Computer Animation, pp. 109–118 (2007)
management. Comput. J. 53(9), 1384–1400 (2009) Kapadia, M., Pelechano, N., Allbeck, J.: Virtual Crowds:
Dressler, D., Groß, M., Kappmeier, J.P., Kelter, T., Steps Toward Behavioral Realism. Morgan & Claypool
Kulbatzki, J., Plmpe, D., Schlechter, G., Schmidt, M., Publishers, San Rafael (2015)
Skutella, M., Temme, S.: On the use of network flow Karamouzas, I., Heil, P., van Beek, P., Overmars, M.H.:
techniques for assigning evacuees to exits. Proc. Eng. A predictive collision avoidance model for pedestrian
3, 205–215 (2010)
Funda Durupınar, Uğur Güdükbay, Aytek Aman, Norman
simulation. In: Motion in Games, Lecture Notes in
Computer Science, pp. 41–52. Springer, Berlin (2009)
C
I. Badler. IEEE Transactions on Visualization and Lerner, A., Chrysanthou, Y., Shamir, A., Cohen-Or, D.:
Computer Graphics (Volume: 22, Issue: 9, Sept. 1 Data driven evaluation of crowds. In: Proceedings of
2016), pp. 2145–2159. the 2nd International Workshop on Motion in Games,
Fu-Shun Li, Sai-Keung Wong: Animating agents based on pp. 75–83 (2009)
radial view in crowd simulation. In: Proceedings of the Musse, S.R., Thalmann, D.: Hierarchical model for real
22nd ACM Conference on Virtual Reality Software time simulation of virtual human crowds. IEEE Trans.
and Technology, pp. 101–109 (2016) Vis. Comput. Graph. 7(2), 152–164 (2001)
Guy, S.J., Chhugani, J., Curtis, S., Dubey, P., Lin, M., Pelechano, N., Badler, N.I.: Modeling crowd and trained
Manocha, D.: Pledestrians: A least-effort approach to leader behavior during building evacuation. IEEE
crowd simulation. In: Proceedings of the 2010 ACM Comput. Graph. Appl. 26(6), 80–86 (2006)
SIGGRAPH/Eurographics Symposium on Computer Pelechano, N., Allbeck, J.M., Badler, N.I.: Controlling
Animation, pp. 119–128 (2010) individual agents in high-density crowd simulation.
Hadzic, T., Brown, K.N., Sreenan, C.J.: Real-time pedes- In: Proceedings of the 2007 ACM SIGGRAPH/
trian evacuation planning during emergency. In: Tools Eurographics Symposium on Computer Animation,
with Artificial Intelligence (ICTAI), 2011 23rd IEEE pp. 99–108 (2007)
International Conference, pp. 597–604 (2011) Rodriguez, S., Amato, N.M.: Behavior-based
Hai-Rong Wang, Qing-Guang Chen, Jian-Bo Yan, Zhi evacuation planning. In: Robotics and Automation
Yuan, Dong Liang: Emergency guidance evacuation (ICRA), 2010 I.E. International Conference,
in fire scene based on pathfinder. In: Proceedings of pp. 350–355 (2010)
the 2014 7th International Conference on Intelligent Sadiyoko, A., Riyanto, T. B., Mutijarsa, K.: The propaga-
Computation Technology and Automation, tion of psychological variables in crowd: Simulation
pp. 226–230 (2014) results. In: 2012 Sixth Asia Modelling Symposium,
Hamacher, H.W., Tjandra, S.A.: Mathematical modelling pp. 59–64 (2012)
of evacuation problems – A state of the art. In: Pedes- Sai-Keung Wong, Yu-Shuen Wang, Pao-Kun Tang, Tsung-
trian and Evacuation Dynamics, Springer, Berlin, Yu Tsai: Optimized Route for Crowd Evacuation. In:
pp. 227–266 (2002). Pacific Graphics Short Papers pp. 7–11 (2016)
Helbing, D., Molnar, P.: Social force model for pedestrian Sai-Keung Wong, Yu-Shuen Wang, Pao-Kun Tang, Tsung-
dynamics. Phys. Rev. E. 51, 4282–4286 (1995) Yu Tsai: Optimized evacuation route based on crowd
Hughes, R.L.: A continuum theory for the flow of pedes- simulation. Comput. Vis. Media. 3(3), 243–261 (2017)
trians. Transp. Res. B Methodol. 36(6), 507–535 Sakour, I., Huosheng, H.: Robot-assisted crowd
(2002) evacuation under emergency situations: A survey.
Hughes, R.L.: The flow of human crowds. Annu. Rev. Robotics. 6(2), (2017)
Fluid. Mech. 35, 16982 (2003) Tang, H., Elalouf, A., Levner, E., Cheng, T.C.E.: Efficient
Inoue, Y., Sashima, A., Ikeda, T., Kurumatani, K.: Indoor computation of evacuation routes on a three-
emergency evacuation service on autonomous naviga- dimensional geometric network. Comput. Ind. Eng.
tion system using mobile phone. In: Universal Com- 76, 231–242 (2014)
munication, 2008. ISUC ’08. Second International Thompson, P.A., Marchant, E.W.: A computer model
Symposium, pp. 79–85 (2008) for the evacuation of large building populations. Fire.
Jinghui Zhong, Wentong Cai, Linbo Luo: Crowd evacua- Saf. J. 24(2), 131–148 (1995)
tion planning using cartesian genetic programming Tsai, J., Fridman, N., Bowring, E., Brown, M., Epstein, S.,
and agent-based crowd modeling. In: Proceedings Kaminka, G., Marsella, S., Ogden, A., Rika, I.,
of the 2015 Winter Simulation Conference, Sheel, A., et al.: Escapes: Evacuation simulation with
pp. 127–138 (2015) children, authorities, parents, emotions, and social
Johansson, A., Helbing, D., Shukla, P.K.: Specification of comparison. In: The 10th International Conference on
the social force pedestrian model by evolutionary Autonomous Agents and Multiagent Systems – vol. 2,
adjustment to video tracking data. Adv. Complex pp. 457–464 (2011)
Syst. 10(Suppl 02), 271–288 (2007) Tsung-Yu Tsai, Sai-Keung Wong, Yi-Hung Chou, Guan-
Kang Hoon Lee, Myung Geol Choi, Qyoun Hong, Jehee Wen Lin: Directing virtual crowds based on dynamic
Lee: Group behavior from video: A data-driven adjustment of navigation fields. Comput Anim Virtual
approach to crowd simulation. In: Proceedings of the Worlds, pp. 7–11 (2017)
498 Crowd Simulation

Van den Berg, J., Lin, M., and Manocha, D.: Reciprocal or video games. Believable virtual crowds are
velocity obstacles for real-time multi-agent navigation. the key of success for virtual therapies treating
In: Robotics and Automation, 2008. ICRA 2008. IEEE
International Conference on Robotics and Automation, agoraphobia. Real-time 3D crowds could popu-
pp. 1928–1935. Pasadena, IEEE (2008) late virtual cities and virtual worlds can be found
Wong, S.-K., Tang, P.-K., Li, F.-S., Wang, Z.-M., Yu, S.-T.: on the Internet.
Guidance path scheduling using particle swarm opti- Realistic-looking, believable-behaving, and
mization in crowd simulation. Comput. Animat. Virtual
Worlds. 26(3–4), 387–395 (2015) real-time rendered virtual crowds are challenging.
Yang, L., Zhu, K., Liu, S. Cellular automata evacuation At the individual scale, virtual agent must look
model considering information transfer in building realistic, i.e., the 3D models are textured and
with obstacles. In: Peacock, R.D., Kuligowski, E.D., lighted. They are goal-directed behaving. People
Averill, J.D. (eds). Pedestrain Dynamics and Evacua-
tion. Springer, Berlin 2010 don’t walk in the streets freely, i.e., going in a
Yi Ma, Kwok Kit Yuen, Wai Ming Lee: Effective leader- random direction each time they encounter an
ship for crowd evacuation. Physica A Stat. Mech. Appl. obstacle. Usually they walk with a goal in mind:
450, 333–341 (2016) going to work, shopping. At the crowd level, each
Zheng, X., Cheng, Y.: Modeling cooperative and compet-
itive behaviors in emergency evacuation: A game- virtual character should be unique. Except if you
theoretical approach. Comput. Math. Appl. 62(12), are surrounded by pairs of twins dressed similarly,
4627–4634 (2011) in real life everybody has different morphology
and clothes. Crowd dynamics should be
respected. Virtual characters avoid each other to
not collide. Flows are created naturally in a dense
Crowd Simulation crowd. Another important aspect of crowd is the
number of virtual characters. One can start speak-
Daniel Thalmann ing of crowd if at least one hundred agents are
Institute for Media Innovation, Nanyang rendered. Massive crowds can count several thou-
Technological University, Singapore, Singapore sands of characters. Real time adds the constraint
that virtual characters are simulated, animated,
and rendered at frame rates that allow user
Synonyms interactions.
Most approaches are application specific,
Crowd animation focusing on different aspects of the collective
behavior, using different modeling techniques.
Employed techniques range from those that do
Definition not distinguish individuals such as flow and net-
work models in some of the evacuation simula-
Process of simulating the movement and/or the tions, to those that represent each individual as
behavior of a large number of entities or being controlled by more or less complex rules
characters. based on physical laws, chaos equations, or
behavioral models in training systems or socio-
logical simulations.
Introduction

Computer-generated crowds of virtual humans Historic Background

gain a lot of attention these years. Applications
are multiple. Accurate crowd simulation is Human beings are arguably the most complex
required for risks and evacuations planning. The known creatures; therefore, they are also the
entertainment industry is demanding of realistic- most complex creatures to simulate.
looking crowd of virtual characters in order to A behavioral animation of human (and humanoid)
create amazing scenes in movies, commercials, crowds is based on foundations of group
Crowd Simulation 499

simulations of much more simple entities, notably templates are instantiated several times. For each
flocks of birds (Reynolds 1987) and schools of instance, one texture is randomly chosen within
fish (Tu and Terzopoulos 1994). The first proce- the template’s available set. Then, color and shape
dural animation of flocks of virtual birds was variety techniques are applied so that instances of
shown in the movie by Amkraut, Girard, and a same template and using the same texture are
Karl called Eurhythmy, for which the first concept still different.
was presented at the Electronic Theater at Previous work on color variety is based on the C
SIGGRAPH in 1985. In his pioneer work, Reyn- idea of dividing a human template into several
olds (1987) described distributed behavioral body parts, identified by specific intensities in
model for simulating aggregate motion of a flock the alpha channel of the template texture. At
of birds. Brogan and Hodgins (1997) simulated runtime, each body part of each character is
group behaviors for systems with significant assigned a color in order to modulate the texture.
dynamics. Bouvier and Guilloteau (1996) pre- Although these methods offer nice results from a
sented a crowd simulation in immersive space reasonable distance, they produce sharp transi-
management and a new approach of particle sys- tions between body parts.
tems as a generic model for simulations of For large crowds, a common approach consists
dynamic systems. Musse and Thalmann (2001) in modifying separately the height of the human
presented a hierarchical model for real-time sim- body and its shape. The height of a human tem-
ulation of virtual human crowds. Their model is plate can be modified by scaling its skeleton
based on groups, instead of individuals: groups (Fig. 1). For each new skeleton, a global scale
are more intelligent structures, where individuals factor is randomly chosen within the given
follow the groups’ specification. Groups can be range. Then, the associated new scale for each of
controlled with different levels of autonomy: its bones is deduced. Short/tall skeletons mixed
guided crowds follow orders given by the user in with broad/narrow shoulders are thus created. The
runtime; programmed crowds follow a scripted skin of the various skeletons also needs adapta-
behavior; and autonomous crowds use events tion. Each vertex of the original template is
and reactions to create more complex behaviors. displaced by each joint that influences it.
O’Sullivan et al. (2002) described a simulation of
crowds and groups with level of details for geom-
etry, motion, and behavior. Decision systems are
generally applied to simple reactive behaviors
such as collision avoidance because of the com-
putational cost of implementing existing rational
models with a crowd of virtual people.

Variety

To generate thousands of individuals, a naive

approach is to design as many humans as there
are people in the crowd. Obviously, such an
approach is impossible, since it would require
armies of designers and an infinite memory. The
common and more reasonable approach is to use
human templates. A human template is a virtual
human defined by its skeleton, its mesh, which is
skinned to the skeleton, and its set of textures. To
create large crowds, a small group of human Crowd Simulation, Fig. 1 Changing the height
500 Crowd Simulation

For the shape, the human mesh is modified automatically infer one value for each vertex of
using three steps: (1) An area called FatMap (see the template’s mesh. Each of these values, called a
Fig. 2) is automatically painted on each individ- fatWeight, is attached to the vertex as an addi-
ual; when the creation of the FatMap is complete, tional attribute. (2) It is computed in the direction
the grayscale values at each texel are used to the vertices are moved when scaled; for this, we
compute the scaling direction of each vertex as the
weighted normal of the bones influencing
it. (3) Once the direction of the body scaling is
computed for each vertex, the actual scaling can
take place. The extent to which we scale the body
is defined by a fatScale, randomly chosen within a
predefined range.

Accessories

Accessorizing crowds offers a simple and efficient

alternative to costly human template modeling.
Accessories are small meshes representing ele-
ments that can easily be added to the human
template original mesh. Their range is consider-
able, from subtle details, like watches, jewelry, or
glasses, to larger items, such as hats, wigs, or
backpacks, as illustrated in Fig. 3. Distributing
accessories to a large crowd of a few human
templates varies the shape of each instance and
Crowd Simulation, Fig. 2 FatMaps – Dark areas repre-
sent regions more influenced by fat or muscles modifica- thus makes it unique. We can distinguish three
tion, while lighter parts are less modified types of accessories.

Crowd Simulation,
Fig. 3 Population with
accessories: bags, hats,
glasses
Crowd Simulation 501

The first group of accessories does not neces- in the domain of navigating crowds, i.e., working
sitate any particular modification of the animation with locomotion animations:
clips played. They simply need to be correctly
“placed” on a virtual human. Each accessory can 1. Variety can be introduced in the animation by
be represented as a simple mesh, independent generating a large amount of locomotion
from any virtual human. First, let us lay the prob- cycles (walking and running) and idle cycles
lem for a single character. The issue is to render (like standing, talking, sitting, etc.) that we C
the accessory at the correct position and orienta- morphologically adapt for each template. For
tion, accordingly to the movements of the locomotion clips, walk and run cycles can be
character. generated from a locomotion engine based on
The second group of accessories we have iden- motion capture data.
tified is the one that requires slight modifications 2. Precomputed animation cycles can be aug-
of the animation sequences played, e.g., the hand mented with upper body variations, like having
close to the ear to make a phone call or a hindered a hand on the hip or in a pocket.
arm sway due to carrying a heavy bag. 3. Finally, procedural modifications can be
Concerning the rendering of the accessory, we applied at runtime on locomotion animations
still keep the idea of attaching it to a specific to allow crowds to wear complex accessories
joint of the virtual human. The additional diffi- as mentioned earlier.
culty is the modification of the animation clips to
make the action realistic. If we want a virtual The principal component analysis (PCA)
human to carry a bag for instance, the animation method is often used to represent the motion
modifications are limited to the arm sway and capture data in a new, smaller space. As the
maybe a slight bend of the spine to counterweight first PCs (principal components) contain the
the bag. If it is a cell phone accessory that we want most variance of the data, an original method-
to add, we need to keep the hand of the character ology is used to extract essential parameters
close to its ear and avoid any collision over the of a motion. This method decomposes the
whole locomotion cycle. PCA in a hierarchical structure of sub-PCA
The third category of accessories is the one spaces. At each level of the hierarchy, an
which needs a specific animation; we can con- important parameter of a motion is extracted
sider in this category handicapped people using and a related function is elaborated, allowing
crutches, skating, and scooter. We may consider not only motion interpolation but also extrap-
accessories that have their motion but linked to olation. Figure 4 shows an example of
the motion of the pedestrian like: a wheelbarrow, PCA-based locomotion.
a caddy. Accessories may also have their own
animation like a dog with a lash. Another cate-
gory of accessories are the ones requiring more Path Planning and Navigation
than one person to carry them, for example,
furniture. Path planning is an important and challenging task
Figure 3 shows examples of accessories. in crowd simulation, which helps each agent to
find the path to its individual goal. The path plan-
ning problem has been widely explored by the
Animation Variety robotics community. Although the multiple-
agent path planning has been addressed for coop-
A second important factor, although less para- erative tasks of multiple robots, it is still a chal-
mount is their animation. If they all perform the lenge to solve the path planning problem for large
same animation, the results are not realistic crowds in real time, especially for large-scale
enough. We can consider three techniques to crowds. Because the methods used for robots are
vary the animation of characters while remaining usually exponential in the number of robots,
502 Crowd Simulation

Crowd Simulation,
Fig. 4 PCA-based walking
models

which are too expensive to be adopted in crowd see each other. Inspired from Voronoi dia-
simulation. grams, Pettré et al. (2006) presented a novel
Four types of methods are popular for path approach to automatically extract a topology
planning and navigation: from a scene geometry and handle path plan-
ning using a navigation graph. The environ-
1. Social force models. Helbing’s social force ment is usually discretized into a fine regular
model (Helbing et al. 2000) is one of the most grid in the potential field method.
influential models in agent-based motion plan- 4. Potential fields. The method (e.g., Treuille
ning. This model considers each agent as a et al. 2006) produces a potential field from
particle subject to long-ranged forces induced the addition of a static field (goal) and a
by the social behavior of individuals. The dynamic field (modeling other people). Each
movement of agents can be described with a pedestrian then moves against the gradient
main function which determines the physical towards the next suitable position in space
and social forces, similar to Newtonian (a waypoint) and thus avoids all obstacles.
mechanics. The social force model is capable
of describing the self-organization of several
observed collective effects of pedestrian Collision Avoidance
behavior.
2. Probabilistic roadmaps. Benefiting from Except the topological model of the environment
motion planning algorithms in robotics, geo- and path planning, collision avoidance is another
metric representation of probabilistic challenging problem to be addressed. The colli-
roadmaps (PRM) can also be used for path sion avoidance techniques should be efficient
planning in crowd simulation. PRM was enough to prevent a large number of agents from
applied to solve the problem of determining a bumping into each other in real time. The greatest
collision-free path between a starting configu- difficulty of collision avoidance is from the
ration of the robot and a goal configuration. absence of other agents’ current velocities. Fur-
3. Visibility graphs. A visibility graph is used for thermore, the agents are not able to communicate
the path planning for large numbers of virtual to coordinate their navigation. A common solu-
agents. The visibility graph connects together tion to this problem is to assume that the other
vertices of the environment if and only if they agents are dynamic obstacles whose future
Crowd Simulation 503

motions are predicted as linear extrapolations of of virtual humans based on the groups’ behavior
their current velocities. The agent then selects a information. The sociological effects modeled in
velocity that avoids collisions with the extrapo- the presented rules are:
lated trajectories of other agents. This is the idea
of velocity obstacle. Considering the case in • Grouping of individuals depending on their
which each agent navigates independently with- interrelationships and the domination effect
out explicit communication with other agents, van • Polarization and the sharing effects as the C
den Berg et al. (2008) propose a new concept, the influence of the emotional status and domina-
“reciprocal velocity obstacle,” which takes into tion parameters
account the reactive behavior of the other agents • Relationship between autonomous virtual
by implicitly assuming that the other agents make humans and groups
a similar collision avoidance reasoning. This con-
cept can be applied to navigation of hundreds of Environment modeling is closely related to
agents in densely populated environments behavioral animation. The purpose of the models
containing both static and moving obstacles for of the environment is to facilitate simulation of
real-time simulation. entities dwelling in their surrounding environ-
ments. Believability of virtual creatures can be
greatly enhanced if they behave in accordance
Crowd Behavior with their surroundings. To make crowd move-
ments more realistic, the first important step is to
The behavior of people in a crowd is a fascinating identify the main places where many people tend
subject: crowds can be very calm but also rise to to go, i.e., places where there is a lot of pedestrian
frenzy; they can lead to joy but also to sorrow. It is traffic. It can be a shopping mall, a park, a circus,
quite a common idea that people not only behave etc. Adding meta-information to key places in an
differently in crowd situations but that they environment has been achieved in many
undergo some temporary personality change different ways.
when they form part of a crowd. Most writers in A recent effort in improving the crowd behav-
the field of mass- or crowd psychology agree that ior has been focused on creating groups. In our
the most discriminating property of crowd situa- everyday life, it is rare to observe people in an
tions is that normal cultural rules, norms, and urban scene walking all by themselves. Indeed, it
organization forms cease to be applicable. For is easy to notice that pedestrians often evolve in
instance, in a panic situation the normal rule of groups of two or more. For this reason, we intro-
waiting for your turn and the concomitant organi- duce an additional and optional layer to our
zation form of the queue are violated and thus motion planning architecture. This layer takes
become obsolete. care of creating small groups of people, which
A simple method for describing the crowd try to remain close to each other during simula-
behavior is through group interrelationships. Vir- tion. Figure 5 shows an example of crowd.
tual actors only react in the presence of others,
e.g., they meet another virtual human, evaluate
their own emotional parameters with those of the Crowd Simulation, What’s Next
other one, and, if they are similar, they may walk
together. The group parameters are specified by Crowd simulation is generally seen as the process
defining the goals (specific positions which each of simulating the movement of a large number of
group must reach), number of autonomous virtual entities or characters, and key issues seem to be
humans in the group, and the level of dominance path planning and collisions. But huge crowds
from each group. This is followed by the creation generally don’t walk; 90 % of crowd images on
504 Crowd Simulation

Crowd Simulation,
Fig. 5 Crowd simulation

Google Images show static crowds. Crowds are There are other scenes we did not see until now
not only moving, and even when they move, they in simulations like large restaurants or crowded
can run or even swim in a crowded pool. Most buses.
path planning algorithms for walking won’t work Most crowds are composed of people with a
for swimming. goal; but on a Sunday afternoon, many people
A lot of research has focused on the collective wander without specific goals.
social behavior of people at social gatherings, Natural motivations should be introduced to
assemblies, protests, rebellions, concerts, sporting simulate more complex and realistic situations.
events, and religious ceremonies, but there are For example, in an airport, people should not
very few simulations showing such case studies. just check in, go to the security, then the gate, as
Behaviors are very situation dependent; for in most simulations. They should be able to go
example, people in a city generally walk and to restaurants, cafés, shops, and toilets
stop only to watch events or chat with people. according to their internal motivations. Such
Many people in public parks will sit down in the models exist for individuals or small groups,
grass or on public seats. but the problem is that it will be extremely CPU
In terms of appearance, research has focused intensive to introduce them to large crowds.
on shape, size, skin color, and accessories. But, More details may be found in (Thalmann and
we should see more by representing children, Musse 2012).
babies, old people, and handicapped people. We
should also mix people with cars, bicycles, etc.
Currently, individuals in crowds can carry Cross-References
accessories; what we don’t see is crowds manip-
ulating objects, open doors, eating, bringing ▶ Character Animation Scripting Environment
objects from one place to another, and exchanging ▶ Crowd Evacuation Using Simulation
objects. Techniques
Cybersickness 505

References
Cultural Diversity
Bouvier, E., Guilloteau, P.: Crowd simulation in
immersive space management. In: 3rd
▶ Diversity in Gaming and the Metaverse
EUROGRAPHICS Workshop on Virtual Environ-
ments, Monte Carlo (1996)
Brogan, D., Hodgins, J.: Group behaviors for systems with
significant dynamics. Autonom. Robot. 4, 137–153 Cultural Heritage C
(1997)
Helbing, D., Farkas, I., Vicsek, T.: Simulating dynami-
cal features of escape panic. Nature 407, 487–490 ▶ 3D Game Asset Generation of Historical Archi-
(2000) tecture Through Photogrammetry
Musse, S.R., Thalmann, D.: A hierarchical model for real ▶ Mixed Reality, Gamified Presence, and Story-
time simulation of virtual human crowds. IEEE Trans.
Vis. Comput. Graph. 7(2), 152–164 (2001)
telling for Virtual Museums
O’Sullivan, C., Cassel, J., Vilhjálmsson, H., Dingliana, J.,
Dobbyn, S., Mcnamee, B., Peters, C., Giang, T.: Levels
of detail for crowds and groups. Comput. Graph. Cultural Identity
Forum 21(4), 733–741 (2002)
Pettré, J., de Heras Ciechomski, P., Maim, J., Yersin, B.,
Laumond, J.-P., Thalmann, D.: Real-time navigating ▶ Preserving the Collective Memory and Re-cre-
crowds: scalable simulation and rendering: research ating Identity Through Animation
articles. Comput. Anim. Virtual World. 17(3–4),
445–455 (2006)
Reynolds, C.W.: Flocks, herds, and schools: a distributed
behavioral model. Comput. Graph. 21(4), 25–34
(1987)
Cyber Security
Thalmann, D., Musse, S.R.: Crowd simulation. 2nd edn.
Springer (2012) ▶ Audio and Facial Recognition CAPTCHAs for
Treuille, A., Cooper, S., Popovic, Z.: Continuum crowds. Visually Impaired Users
ACM. Trans. Graph. 25(3), 1160–1168 (2006)
Tu, X., Terzopoulos, D.: Artificial fishes: Physics, locomo-
tion, perception, behavior. In: Computer Graphics.
ACM SIGGRAPH’94 Conference Proceedings, vol.
28, pp. 43–50. ACM, Orlando (July 1994) Cybersecurity Attacks
van den Berg, J., Patil, S., Sewall, J., Manocha, D., Lin, M.:
Interactive navigation of multiple agents in crowded ▶ Anti-phishing Attacks in Gamification
environments. In: Proceedings of the 2008 Symposium
on Interactive 3D Graphics and Games, Redwood City
(2008)
Cybersickness

Keith Nesbitt1 and Eugene Nalivaiko2

Crowdsourcing, Scientific 1
School of Electrical Engineering and
Games Computing, University of Newcastle, Callaghan,
NSW, Australia
▶ Games in Science 2
School of Biomedical Sciences and Pharmacy,
University of Newcastle, Callaghan, NSW,
Australia

Crowdsourcing: User-
Generated Content Synonyms
▶ Gamification in Crowdsourcing Applications Simulator sickness
506 Cybersickness

Definitions form of motion sickness, called “space adaptation

syndrome” that occurs in exposure to zero-gravity
Cybersickness is an uncomfortable side effect conditions (Davis et al. 2015).
experienced by users of immersive interfaces While there are definite relationships between
commonly used for Virtual Reality. It is associated the symptoms experienced in cybersickness, sim-
with symptoms such as nausea, postural instabil- ulator sickness, and motion sickness, they may
ity, disorientation, headaches, eye-strain, and provoke slightly different clusters of symptoms
tiredness. that can help differentiate the three conditions
(Kennedy et al. 1993). In one of the largest early
studies of symptoms, available data from
Cybersickness 10 United States Navy flight simulators, using
1,119 pairs of preexposure and postexposure
Cybersickness is a relatively common, unwanted scores from self-reported data on motion sickness
side effect of immersive interfaces that causes symptoms, were analyzed using a series of factor
a broad range of unpleasant symptoms such as analyses to identify 16 principal symptoms clus-
nausea, headaches, disorientation, and tiredness. tered into the three main categories of oculomotor,
More serious symptoms, such as postural instabil- disorientation, and nausea (Kennedy et al. 1993).
ity, although less common, can also result from The oculomotor cluster includes the symptoms of
prolonged exposure to virtual interfaces. eyestrain, difficulty focusing, blurred vision, and
Cybersickness is typically experienced by sta- headache (Kennedy et al. 1993). The disorienta-
tionary users that perceive that they are moving in tion cluster includes dizziness and vertigo, while
a virtual scene. This stationary reality and the the nausea cluster includes stomach awareness,
associated compelling experience of self-motion increased salivation, and burping (Kennedy et al.
is believed to underlie the condition (Webb and 1993). A further dimension of cybersickness
Griffin 2003). By contrast, simulator sickness was related to the “sopite syndrome” has also been
first found in pilots who underwent extended characterized, and this includes symptoms of
training in flight simulators that tried to provide drowsiness, yawning, disengagement, and nega-
a real sense of vehicle movement using motion tive affect (Lawson and Mead 1998).
platforms (Kennedy et al. 1993). It is likely that The actual cause of cybersickness is not known
there are discrepancies between the simulator’s and the underlying physiological mechanisms are
actual motion and the expected motion of the uncertain. The three most prominent theories for
virtual vehicle that contribute to simulator the cause of cybersickness are postural instability
sickness. theory, poison theory, and sensory conflict theory
Cybersickness and simulator sickness share (LaViola 2000).
similar symptoms with motion sickness although The postural instability theory is based on the
the conditions are caused by exposure to slightly idea that the main goal of humans is to maintain
different situations. Younger children, aged postural stability in the environment (Riccio and
between 4 and 12, are more prone to motion Thomas 1991). Therefore, prolonged postural
sickness and indeed susceptibility to this condi- instability results in cybersickness symptoms
tion in childhood can be a good indicator of and the longer the instability, the more severe
susceptibility to cybersickness (Golding 1998). the symptoms are likely to be (LaViola 2000).
Motion sickness can be brought on by travelling By contrast, poison theory suggests an evolution-
in any type of moving vehicle including cars, ary survival mechanism comes in to play when the
buses, trains, aircraft, boats, and submarines and user experiences sensory hallucinations consistent
may also be induced on an amusement ride, a with ingesting some type of poison (Bouchard
spinning chair or simply by using a swing at a et al. 2011). It is suggested that patterns of visual
playground. Astronauts may experience a related and vestibular stimuli may trigger motion
Cybersickness 507

sickness by accidentally activating brain sensors but are not limited to, fatigue, hangovers, and the
for detecting toxins (Treisman 1997). flu (LaViola 2000).
However, the most longstanding and popular The ranges of device factors that impact
explanation for cybersickness is known as the on cybersickness are lag, flicker, calibration,
sensory conflict theory (Cobb et al. 1999; field of view, and general ergonomics. One
Kolasinski 1995; LaViola 2000). This theory problem with most stereoscopic displays is
describes the conflicts of two key sensory systems known as accommodation-convergence conflict C
engaged in virtual environments, namely, the (Durlach and Mavor 1994). In the real world,
visual and vestibular senses (Kolasinski 1995). our eyes converge to look at near objects while
They provide information about an individual’s also focusing (accommodating) at the same dis-
orientation and perceived motion, and it is the tance on the object. However, in Virtual Reality,
mismatch of these senses that can frequently while the eyes will still converge to look at a
occur in Virtual Reality. For example, the vestib- virtual object, the focus needs to be on the plane
ular system may be telling the individual that their of the display itself rather than the object. Other
body is stationary, while the visual system characteristics of head-mounted displays that may
is telling them that their body is moving, causing be relevant are the vertical and horizontal field of
a sensory mismatch (Howarth and Costello 1997). view, the resolution, the contrast, the luminance,
Unfortunately, like the other theories, the and color characteristics of the display (Durlach
sensory conflict theory lacks predictive power in and Mavor 1994).
determining how severe the symptoms of cyber- The refresh rate of the display can also be
sickness will be relative to any virtual experience. important factor as any lag in the system can
Furthermore, these various theories still fail to contribute to cybersickness symptoms (LaViola
explain why, given identical virtual experiences, 2000). Lag occurs when there is a delay between
some individuals get sick and others do not. an individual’s action (e.g., turning a steering
While the underlying mechanisms that cause wheel) and the system’s reaction. Real time graph-
cybersickness are still not completely understood, ical displays that can operate at around 50–60 Hz
there has been more success in identifying some are critical. Efficient tracking of head movements
of the factors known to impact on users develop- is also critical as people expect their view to
ing symptoms. These factors are varied and relate change promptly when they move their head.
to individual differences, variations in the devices Any errors in accuracy of tracking of head move-
being used, the task being performed, and the ment can likewise impact on cybersickness.
design of the virtual environment. Display flicker, a visible fading between video
Individual factors that impact on cybersickness frames, is also related to visual refresh rate.
include age, gender, race, illness, and posture. Flicker is not only distracting but also causes eye
Children in the 2–12 age range have the greatest fatigue (Kolasinski 1995). Flicker fusion is an
susceptibility to cybersickness, and this rapidly important property of the device and is even
decreases from the ages of 12–21 and beyond more critical for wider fields of view as peripheral
(Kolasinski 1995). Thus, older people are less vision is more sensitive to flicker (LaViola 2000).
susceptible to symptoms. In relation to gender, The perception of flicker is another factor that can
women have a wider field of view which increases also vary between individuals.
the likelihood of flicker perception and this in Due to differences in physical characteristics,
turn increases their susceptibility to cybersickness poor system calibration can also increase cyber-
(LaViola 2000). Research has also shown that sickness symptoms. For example, interpupillary
female hormones can affect susceptibility distance, which is the distance between the centers
(Kolasinski 1995). For all users, any underlying of the pupils of both eyes, varies between individ-
illness increases an individual’s susceptibility to uals (Kolasinski 1995). As stereoscopic displays
cybersickness. These physical conditions include, require each eye to receive a slightly offset view
508 Cybersickness

of the virtual world, this offset needs to corre- understanding of the complexity of changing
spond as closely as possible to the users own scenes in a virtual environment (Smith
specific interpupillary distance. As such appropri- et al. 2017).
ate calibration is required for each individual. As it stands, cybersickness still provides an
General ergonomic factors also need to be consid- obstacle to the wide spread adoption and commer-
ered when designing immersive systems. For cial development of technologies associated with
example, heavy and poor fitting headsets can Virtual Reality. Of particular advantage would
cause physical discomfort, and restricted move- be better quantitative measures for predicting
ment from cables can cause further distractions a user’s susceptibility to cybersickness and reli-
from the virtual experience (McCauley and able methods for detecting and measuring symp-
Sharkey 1992). Furthermore, head-mounted dis- toms such as the nausea associated with
plays impact on the normal inertia characteristics the condition. Most historical studies of cyber-
of the head, generating unusual forces during head sickness have relied on subjective self-reporting
movements that can also directly impact on cyber- of the severity of symptom conditions (Ames et al.
sickness (Durlach and Mavor 1994). The posture 2005; Cobb et al. 1999; Dennison et al. 2016;
of the individual, possibly related to the postural Gianaros et al. 2001; Golding 1998; Kennedy
instability theory, is also important. For example, et al. 1993; Nesbitt et al. 2017).
sitting is a safer posture for users than standing as The Pensacola Motion Sickness Question-
this reduces any demand on postural control naire (Kellogg et al. 1965) based on 27 previ-
(Kolasinski 1995). ously identified issues (Hardacre and Kennedy
Cybersickness can also be influenced by the 1963) is recognized as one of the earliest subjec-
specific task the user is performing in the environ- tive measures designed for assessing motion
ment. The main task factors include the level of sickness (Bouchard et al. 2011). This work led
control the user has and the duration of the task. to the development of the Pensacola Diagnostic
Participants who have good control in a virtual Index (Graybiel et al. 1968). The Pensacola
environment can better predict future motion and Diagnostic Index score is calculated by summing
are found to be less susceptible to cybersickness. an individual’s ratings on various scales related
By contrast, users with no control over the virtual to the symptoms of dizziness, headache, warmth,
environment lack the same level of predictability sweating, drowsiness, salivation, and nausea.
about the environment and are thus more prone to After a major study analyzing the factors relevant
symptoms (Kolasinski 1995). Longer exposure to simulator sickness, an alternative 16-item
times to Virtual Reality also increase the likeli- Simulator Sickness Questionnaire was devel-
hood and severity of cybersickness and lead to the oped (Kennedy et al. 1992). Another widely
need for longer adaptation periods. Indeed, using used survey instrument is the Nausea Profile
brief exposures to virtual environments is one way (Muth et al. 1996). Like the Simulator Sickness
to improve the speed of adaptation (Kolasinski Questionnaire, the Nausea Profile is distin-
1995; McCauley and Sharkey 1992). guished from approaches such as the Pensacola
As the illusion of motion is one of the factors Diagnostic Index in that it examines symptoms
that induce cybersickness, the design of the virtual along multiple dimensions. Another multivariate
world, in terms of visual complexity and amount questionnaire was developed to measure the
of motion associated with the visual stimuli, symptoms associated with the subscales of gas-
is also an important factor that might be controlled trointestinal, central, peripheral, and sopite-
(So et al. 2001). For example, during the design of related symptoms (Gianaros et al. 2001). More
virtual experiences, optical flow might be used as recently, the Virtual Reality Symptom Question-
a measure the complexity of motion in a stream of naire (Ames et al. 2005) was developed specifi-
video frames (Ali 2013; Beauchemin and Barron cally for investigating symptoms that result from
1995). Likewise, a measure such as approximate Virtual Reality viewing using technology such as
entropy can be used to provide a general head-mounted displays.
Cybersickness 509

Generally, subjective approaches for measur- thermoregulation (Nalivaiko et al. 2014) that
ing cybersickness symptoms elicit an individual’s manifest as dilatation in the cutaneous vascular
ratings on scales that relate to either the propensity bed and reduction in thermogenesis; it is quite
to be susceptible to simulator, motion, or cyber- likely that “cold sweating” is a part of this ther-
sickness, or the experience of sickness under moregulatory response. The dilatation of cutane-
provocation conditions. Subjective measurements ous vessels during provocative motion has been
may be impacted by systematic biases and confirmed in experimental animals C
psychological factors (Jahedi and Méndez 2014), (Ngampramuan et al. 2014) and thus appears to
and thus there is a trend to devise more objective be a cross-species real-time marker of motion
measures of cybersickness (Bouchard et al. 2011; sickness (Nalivaiko et al. 2014).
Bruck and Watters 2011; Cowings et al. 1986; It appears that objective signs of cyber-
Gavgani et al. 2017; Kim et al. 2005; Nalivaiko sickness resemble those of other types of motion
et al. 2015). These objective measures are of sickness; it is however not known whether subtle
particular interest in training domains where inde- differences exist, similar to differences in symp-
pendent identification of susceptibility, or affect- toms between motion sickness, cybersickness,
edness, can assist in maximizing training and simulator sickness. In the future, it is likely
experiences, minimizing postexposure effects, that both objective and subjective approaches
and optimizing content to reduce provocation. will be used to help design better devices and
Objective approaches focus on the measure- experiences in Virtual Reality that improve
ment of physiological responses to cybersickness understanding of this complex condition while
(Kim et al. 2005). The key physiological changes helping to minimize the commercial impact of
include sweating, alterations in gastric myoelec- cybersickness. Further detail around cyber-
tric activity and in cardiac vagal tone, an increase sickness can be found in a recent systematic
in the delta-power of the EEG, and a rise of review (Davis et al. 2014).
plasma vasopressin (Stern et al. 2011). In one
study, 16 electrophysiological parameters where
collected while subjects navigated a virtual Cross-References
environment (Kim et al. 2005). During this prov-
ocation, some parameters increased (gastric ▶ Perceptual Illusions and Distortions in Virtual
tachyarrhythmias, eye blink rate, skin conduc- Reality
tance, respiratory sinus arrhythmia, and delta- ▶ Redirected Walking
power of the EEG), while other decreased (heart ▶ Spatial Perception in Virtual Environments
period, fingertip temperature and photo-
plethysmographic signal, and EEG beta-power).
Of those changes, several (gastric tachyarrhyth- References
mias, eye blink rate, respiration rate, respiratory
sinus arrhythmia, and heart rate) had significant Ali, S.: Measuring flow complexity in videos. In: Proceed-
ings of the 2013 I.E. International Conference on
positive correlation with the subjective score of
Computer Video, 1097–1104, IEEE (2013)
cybersickness. In a further study, immersion in a Ames, S.L., Wolffsohn, J.S., McBrien, N.A.: The develop-
virtual environment resulted in an increase of the ment of a symptom questionnaire for assessing virtual
low-frequency but not high-frequency compo- reality viewing using a head-mounted display. Optom.
Vis. Sci. 82(3), 168–176 (2005)
nents of the heart rate variability (Ohyama et al. Beauchemin, S.S., Barron, J.L.: The computation of opti-
2007). In conjunction with the previously men- cal flow. ACM Comput. Surv. 27(3), 433–466 (1995)
tioned work, this may indicate that cybersickness Bouchard, S., Robillard, G., Renaud, P., Bernier, F.:
is associated with an increase of the cardiac sym- Exploring new dimensions in the assessment of virtual
reality induced side effects. J. Comput. Inf. Technol.
pathetic outflow.
1(3), 20–32 (2011)
It is less known but well documented that Bruck, S., Watters, P.A.: The factor structure of cyber-
motion sickness causes disturbances in sickness. Displays. 32(4), 153–158 (2011)
510 Cybersickness

Cobb, S., Nichols, S., Ramsey, A., Wilson, J.: Virtual for quantifying simulator sickness. Int. J. Aviat.
reality-induced symptoms and effects (VRISE). Psychol. 3(3), 203–220 (1993)
Presence Teleop. Virt. 8(2), 169–186 (1999) Kim, Y., Kim, H., Kim, E., Ko, H., Kim, H.: Characteristic
Cowings, P.S., Suter, S., Toscano, W.B., Kamiya, J., changes in the physiological components of cyber-
Naifeh, K.: General autonomic components of motion sickness. Psychophysiology. 42(5), 616–625 (2005)
sickness. Psychophysiology. 23(5), 542–551 (1986) Kolasinski, E.M.: Simulator sickness in virtual environ-
Davis, S., Nesbitt, K., Nalivaiko, E.: A systematic review ments. Technical report 1027. United States Army
of cybersickness. In: Proceedings of Interactive Research Institute for Behavioral and Social Sciences.
Entertainment (IE2014), Newcastle, Australia. ACM, http://www.dtic.mil/dtic/tr/fulltext/u2/a29586.pdf.
New York (2014). https://doi.org/10.1145/2677758. (1995). Accessed 9 Jan 2018
2677780 LaViola Jr., J.J.: A discussion of cybersickness in virtual
Davis, S., Nesbitt, K., Nalivaiko, E.: Comparing the onset environments. ACM SIGCHI Bull. 32(1), 47–56
of cybersickness using the Oculus Rift and two virtual (2000)
roller coasters. In: Pisan, Y., Nesbitt, K. Blackmore, K. Lawson, B.D., Mead, A.M.: The sopite syndrome revisted:
(eds.) Proceedings of the 11th Australasian Conference drowsiness and mood changes during real or apparent
on Interactive Entertainment (IE 2015) Sydney, motion. Acta Astronaut. 43, 181–192 (1998)
Australia. CRPIT, 167, ACS, 3–14 (2015) McCauley, M., Sharkey, T.: Cybersickness: perception of
Dennison, M.S., Wisti, A.Z., D’Zmura, M.: Use of physi- self-motion in virtual environments. Presence Teleop.
ological signals to predict cybersickness. Displays. 44, Virt. 1(3), 311–318 (1992)
42–52 (2016) Muth, E.R., Stern, R.M., Thayer, J.F., Koch, K.L.: Assess-
Durlach, N.I., Mavor, A.S.: Virtual Reality: Scientific and ment of the multiple dimensions of nausea: the nausea
Technological Challenges. National Academies Press, profile (NF). J. Psychosom. Res. 40, 511–520 (1996)
Washington, DC (1994). https://doi.org/10.17226/ Nalivaiko, E., Rudd, J.A., So, R.H.Y.: Motion sickness,
4761 nausea and thermoregulation: the “toxic” hypothesis.
Gavgani, A.M., Nesbitt, K.V., Blackmore, K.L., Nalivaiko, Temperature. 1(3), 164–171 (2014)
E.: Profiling subjective symptoms and autonomic Nalivaiko, E., Davis, S.L., Blackmore, K.L., Vakulin, A.,
changes associated with cybersickness. Auton. Nesbitt, K.V.: Cybersickness provoked by head-
Neurosci. 203, 41–50 (2017) mounted display affects cutaneous vascular tone,
Gianaros, P.J., Muth, E.R., Mordkoff, J.T., Levine, M.E., heart rate and reaction time. Physiol. Behav. 151,
Stern, R.: A questionnaire for the assessment of the 583–590 (2015)
multiple dimensions of motion sickness. Aviat. Space Nesbitt, K., Davis, S., Blackmore, K., Nalivaiko, E.:
Environ. Med. 72(2), 115–119 (2001) Correlating reaction time and nausea measures with
Golding, J.F.: Motion sickness susceptibility questionnaire traditional measures of cybersickness. Displays. 48,
revised and its relationship to other forms of sickness. 1–8 (2017)
Brain Res. Bull. 47, 507–516 (1998) Ngampramuan, S., Cerri, M., Del Vecchio, F., Corrigan, J.J.,
Graybiel, A., Wood, C.D., Miller, E.F., Cramer, D.B.: Kamphee, A., Dragic, A.S., Rudd, J.A.,
Diagnostic criteria for grading the severity of acute Romanovsky, A.A., Nalivaiko, E.: Thermoregulatory
motion sickness. Aerosp. Med. Res Lab. 39, 453–455 correlates of nausea in rats and musk shrews.
(1968) Oncotarget. 5(6), 1565–1575 (2014)
Hardacre, L.E., Kennedy, P.: Some issues in the develop- Ohyama, S., Nishiike, S., Watanabe, H., Matsuoka, K.,
ment of a motion sickness questionnaire for flight stu- Akizuki, H., Takeda, N., Harada, T.: Autonomic
dents. Aerosp. Med. 34, 401–402 (1963) responses during motion sickness induced by virtual
Howarth, P., Costello, P.: The occurrence of virtual simu- reality. Auris Nasus Larynx. 34(3), 303–306 (2007)
lation sickness symptoms when an HMD was used as a Riccio, G.E., Thomas, A.S.: An ecological theory of
personal viewing system. Displays. 18(2), 107–116 motion sickness and postural instability. Ecol. Psychol.
(1997) 3(3), 195–240 (1991)
Jahedi, S., Méndez, F.: On the advantages and disadvan- Smith, S.P., Blackmore, K.L., Nesbitt, K.V.: Using optical
tages of subjective measures. J. Econ. Behav. Organ. flow as an objective metric of cybersickness in virtual
98, 97–114 (2014) environments. Paper presented at the Australasian Sim-
Kellogg, R.S., Kennedy, R.S., Graybiel, A.: Motion sick- ulation Congress 2017 (ASC 2017), Sydney, 28–31
ness symptomatology of labyrinthine defective and Aug 2017. http://hdl.handle.net/1959.13/1346847.
normal subjects during zero gravity maneuvers. Accessed 9 Jan 2018
Aerosp. Med. 36, 315–318 (1965) So, R.H., Ho, A., Lo, W.T.: A metric to quantify virtual
Kennedy, R.S., Fowlkes, J.E., Berbaium, K.S., scene movement for the study of cybersickness: defini-
Lilienthal, M.G.: Use of a motion sickness history tion, implementation, and verification. Presence
questionnaire for prediction of simulator sickness. Teleop. Virt. 10(2), 193–215 (2001)
Aviat. Space Environ. Med. 63, 588–559 (1992) Stern, R.M., Koch, K.L., Andrews, P.: Nausea: Mecha-
Kennedy, R.S., Lane, N., Berbaum, K., Lilienthal, M.: nisms and Management. Oxford University Press,
Simulator sickness questionnaire: an enhanced method New York (2011)
Cyborg 511

Treisman, M.: Motion sickness, an evolutionary hypothe-

sis. Science. 197, 493–495 (1997) Cyberworld
Webb, N.A., Griffin, M.J.: Eye movement, vection, and
motion sickness with foveal and peripheral vision.
Aviat. Space Environ. Med. 74(6), 622–625 (2003) ▶ Virtual Reality and Robotics

C
Cyberspace Cyborg

▶ Diversity in Gaming and the Metaverse ▶ Virtual Reality and Robotics

D

Dark Side of Gamification playing games. Players assume the role of a char-
acter in their unique world. Combat is done in real
▶ Gamification Ethics time as opposed to turn-based games.

Dark Souls III Action-Adventure

Dark Souls Role-Playing Game

▶ Dark Souls Through the Lens of Essential Dark Souls III is an adventure and action role-
Experience playing game developed by From Software and
published by Bandai Namco. Dark Souls III came
out in Japan on March 24, 2016, and later to the
rest of the world on April 12, 2016. It is both a
Dark Souls III, an Analysis single and multiplayer game that was released on
Xbox One, PlayStation 4, and other consoles with
Matthew Clark2 and Newton Lee1,2 an M rating. Dark Souls III is the third game in the
1
Institute for Education, Research, and Dark Souls franchise.
Scholarships, Los Angeles, CA, USA In Dark Souls III, the player fights their way
2
Vincennes University, Vincennes, IN, USA through countless enemies as they progress
through the game. Dark Souls III takes pride in
being one of the most difficult games that gamers
Synonyms can find. The gore and character designs heavily
aims for an adult audience.
Action role-playing game; Adventure game This is a game that has very few cutscenes;
instead, the game relies on in-game item descrip-
tions and dialogue with the NPC’s to tell the story.
Definitions Dark Souls III is set in the fallen country of
Lothric. In the beginning of this world, the only
Adventure game ¼ A game genre where the beings in existence were the immortal dragons.
players are motivated by exploration, the After the discovery of fire, multiple types of life
unknown, and solving puzzles. began to come out of the darkness.
Action role-playing game ¼ a game genre This started the cycle of fire and darkness. At
combining elements of both action and role- the start of the game, the players are awakened by
© Springer Nature Switzerland AG 2024
N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
514 Dark Souls III, an Analysis

Dark Souls III, an Analysis, Fig. 1 Character creation also lets players customize almost every aspect of a character

the sound of a bell that is warning that the fire is in flasks. They give portions of health back. Players
danger of going out. This leads them to a building can collect in game items to increase the number
where the fire is located. What they find is that of flasks they have and how much health they will
only one of five of the Lords of Cinder remains to restore. The magic in this game has the same type
keep the fire lit. Their job is to go and defeat the of system but they are another type of flask called
other lords and bring them back to the fire. The ashen estus flasks. The catch is that the health
Lords of Cinder are Ludleth of Courland, Abyss flasks and magic flasks are connected, meaning
Watchers, Aldrich the Devourer of Gods, Yhorm that players can only have seven total flasks at
the Giant, and Lothric the Younger Prince. once. The ratio of health to magic flasks can be
In Dark Souls III, the player starts the game in a adjusted in-game at the blacksmith.
character creation screen (Fig. 1). Here they The stamina system in Dark Souls III is set so
choose an origin that gives them stats that are that as a player runs, stamina will deplete. If one
based on the origin. An example of this would stops running, their stamina will fill back
be that a knight will have default stats that are up. Stamina also depletes each time they use a
suited for melee combat, like strength. weapon or dodge (Fig. 2). The amount of stamina
Dark Souls III has bonfires that are placed a player has can be increased by leveling up their
around the map. These bonfires are lit by walking stats.
up to one and lighting them. These bonfires are In Dark Souls III, when a player dies, they will
save points which will reset health and estus start to become hollow. The more they die, the
flasks. These bonfires also act as a fast travel more hollow they will become. Hollowing is an
system to go from one bonfire to another. effect that will gradually start to lower the max
The combat in this game is based on figuring amount of health they will have. This will also
out the patterns and attacks the enemies use to start to make their character look more and more
dodge, block, or counter their attacks. The health decayed. The way they can reverse the effect is to
system in the game uses what are called estus find embers that are scattered around the map.
Dark Souls III, an Analysis 515

Dark Souls III, an Analysis, Fig. 2 The picture above is boxes are usable items and weapons will be displayed. The
to show the games interface. In the top left corner is the red bottom right corner has a counter showing the number of
health bar, blue magic bar, and green stamina bar. In the souls collected
bottom left corner, there are equipment boxes. Inside these

Once a player uses one, their health’s max level to upgrade stats or to get new equipment. Souls
will go back up and their character will look can also be used to upgrade the equipment that
normal again. This state is called being embered. they already have.
Hollowing also prevents multiplayer. If a player To upgrade the equipment, players have to find
dies, their loot will be dropped where they died. If in-game items that match the specific weapon they
they die again before returning to collect the loot, are trying to upgrade. Along with these items,
they will vanish. players will need a certain amount of souls to
Dark Souls III has a mechanic in the game that upgrade as well. A piece of equipment can be
limits the amount of items players can have on upgraded a total of ten times. Each time the equip-
their person by weight. The less weight players ment will get stronger but the amount of items they
have, the faster their character can move. This is need to upgrade the equipment will also increase.
shown to the player in a percentage. If they are Dark Souls III has a unique style of multi-
below 30% of max weight capacity, they are faster player. There are items in the game that can be
than normal. The normal weight range is between used to make a mark on the ground. This mark can
30 and 70%. At this weight, they are moving at the be seen by other players in their game if they are in
normal speed of their character. At 70% or higher, the area. The player will have to go and accept the
the character will slow down drastically. Move- player so he can join the other player’s game.
ment speed and dodging speed will go down and They can put down a red or white mark. A red
stamina will regenerate slower. If it is over 100%, mark will bring them to another's world as an
they will no longer be able to run or dodge. The enemy and a white mark will bring them as an
game calls this state being overburdened. A player ally. There is a feature where a player can invade
can increase their max carrying capacity by level- another's game. This will make them unable to
ing their stats. progress unless the defeat the invaders or they are
In this game, the currency, called souls, is used defeated themselves.
to buy equipment or upgrades. Souls are dropped The levels in Dark Souls III are designed so
by enemies. These are traded in to different NPC’s that players fight through enemies to clear out the
516 Dark Souls III, an Analysis

area and collect all of the items in the area. If Metacritic rated the game 89 out of 100. Dark
players die while clearing the area, the enemies Souls III has no notable controversies.
that they killed will respawn. Each level has a boss There have been many improvements to the
at the end that they have to beat to continue. There franchise since the first game. The second game
are some optional bosses that do not need to be improved the hollowing system by adding the
beaten to continue. These bosses are found in lowering of the health cap. Dark Souls III further
optional levels that branch off from the main improved this system by adding dark sigils.
levels. As players fight through the levels, the The more sigils a character has, the more the
enemies’ difficulty increases, as does the number hollowing affects them. Dark Souls III is also
of souls they drop. The items that they drop and the first game in the series to introduce the magic
the items they can pick up from the world will also bar. In the previous games, spells were limited use
increase in value. and could be restored at a bonfire. The games also
Dark Souls III had great public reviews. had graphic upgrades from game to game. The
Gamespot.com rated Dark Souls III as an 8/10. pictures from top left to right (Fig. 3) show the
PC Gamers rated the game a 94 out of 100. graphical upgrades.

Dark Souls III, an Analysis, Fig. 3 There are two games Souls franchise and was developed by the same developers.
made by the same developer that are similar to these Nioh and Lords of the Fallen are games that are also similar
games. The first is the game that was released before the to Dark Souls III. These two games are made by different
original Dark Souls, and it is called Demon Souls. The next developers and have different plots, but they have the same
game called Bloodborne is also very similar to the Dark style of clear an area and then fight a boss gameplay
Dark Souls RPG Through the Lens of Challenge 517

References Dark Souls ¼ An action RPG released in 2011,

which has gained infamy in the gaming industry
BANDAI NAMCO Entertainment America – More fun for for its high difficulty.
everyone! (n.d.). Retrieved August 25, 2018, from
https://www.bandainamcoent.com/games/dark-souls-
iii#news
Davenport, J.: Dark Souls 3 review. (2018, August 26). Introduction
Retrieved August 26, 2018, from https://www.
pcgamer.com/dark-souls-3-review/
Dark Souls is a notoriously difficult video game
Dark Souls 3 boss: How to beat Vordt of the Boreal Valley.
that alienates most people from trying it. This D
(2016, April 05). Retrieved August 26, 2018, from
https://www.vg247.com/2016/03/05/dark-souls-3- article explains what makes Dark Souls a good
boss-how-to-beat-vordt-of-the-boreal-valley/ game through the description of its gameplay
Dark Souls 3: Character Class and Burial Gift Guide.
(2016, April 11). Retrieved August 25, 2018, from
systems, and how its difficulty makes it a more
https://www.usgamer.net/articles/dark-souls-3- worthwhile experience.
character-class-and-burial-gifts-guide From Software, the developer of Dark Souls,
Dark Souls III.: (n.d.-a). Retrieved August 25, 2018, from didn’t always have the booming success that it has
http://darksouls.wikia.com/wiki/Dark_Souls_III
today. It was originally formed in 1986 as a devel-
Dark Souls III.: (n.d.-b). Retrieved August 26, 2018, from
https://www.gamespot.com/dark-souls-iii/reviews/ oper for office software, but in 1994 it jumped
Dark Souls III.: (2016, April 12). Retrieved August onto the gaming scene with King’s Field for the
26, 2018, from http://www.metacritic.com/game/ PlayStation, a first-person dungeon crawler RPG
playstation-4/dark-souls-iii
Perez, M.: Every item on Dark Souls 3’s intimidating
that saw the player explore dungeons, solve puz-
Status screen, explained. (2016, April 28). Retrieved zles, and fight monsters. King’s Field retained the
August 25, 2018, from https://www.polygon.com/ level of difficulty that the Souls series would
2016/4/28/11528364/dark-souls-3-status-screen- eventually be known for, but a lot of the difficulty
explain-thrust-hollowing-weight-damage-absorption
Which Is Better?: Dark Souls 3 vs. Bloodborne. (2016,
came from the bizarre controls and obtuse puz-
April 25). Retrieved August 26, 2018, from https:// zles. King’s Field would spawn a few more
www.gamerevolution.com/features/12469-which-is- sequels on PlayStation 1 and 2, but it never had
better-dark-souls-3-vs-bloodborne the success that other competing RPGs like the
Final Fantasy series did (Ciolek 2015).
In 2009, Demon’s Souls was released to not
much fanfare. It was originally only released in
Dark Souls RPG Through the Japan, but Western players were able to import it
Lens of Challenge and find a difficult but rewarding experience. This
convinced Atlus to publish the game in the West,
Seth Gaither2 and Newton Lee1,2 where it found immediate success. Demon’s Souls
1
Institute for Education, Research, and is the first Soulslike game From Software ever
Scholarships, Los Angeles, CA, USA released, and many of its gameplay systems
2
Vincennes University, Vincennes, IN, USA would be refined and implemented into Dark
Souls (Ciolek 2015).
Dark Souls was released in 2011 for the Xbox
Synonyms 360 and PlayStation 3. It received universal
acclaim from critics and was cemented as a hall-
Action RPG; Soulslike mark of gaming. As of 2022, the Souls series has
sold 27 million copies (Hannah 2021).

Definition
Plot
RPG ¼ A roleplaying game where the player
controls the actions of a character in an immersive Making sense of the plot to Dark Souls can be a
virtual world. daunting task for newcomers. Aside from the
518 Dark Souls RPG Through the Lens of Challenge

opening cutscene that plays when the player starts currency. The player has one chance to return to
a new playthrough, the game doesn’t explain its the spot where they died in order to retrieve them,
story very well. Every secret about the game’s but if they die again before that happens, their
world and its lore must be found by the player, souls are gone forever.
whether that be through character interactions or If the player character was human before they
by reading item descriptions. This adds a level of died, they will become hollow. Hollowing will
immersion unmatched by most other games, as change the player’s appearance to that of a zombie
the player is learning with their character, but it’s and disables the summoning of NPCs or other
entirely possible to beat the game without under- players for bosses. On the other hand, it also pro-
standing the plot. tects the player from invaders, other players with
The player controls the Chosen Undead, an their own copies of Dark Souls whose only goal is
undead warrior prophesized to prolong the Age to kill the player. Depending on the situation,
of Fire by ringing the Bells of Awakening. How- being hollow can be either a detriment or a benefit.
ever, this prophecy is misleading, as many undead The world of Dark Souls is massive, but linear.
have undertaken the pilgrimage and failed. Those It wouldn’t be inaccurate to compare the structure
that have failed became hollow and lost their of the map to that of a Metroidvania, as the player
sanity. The Chosen Undead is not actually chosen slowly unlocks more of the map as they play, and
by anyone, rather, it could be said that the Chosen they have the option to use shortcuts to access
Undead is whoever is able to ring the bells first different areas quickly. Players can complete cer-
(“Dark Souls Lore Explained!” 2020). tain objectives out of order, but they still must be
done in order to beat the game. For example, at the
beginning of the game the player is tasked with
Gameplay ringing two bells: first, one in Undead Burg, and
then one in Blighttown. Depending on how they
Dark Souls has combat like most 3D Zelda games: start, the player could ring these bells in any order
the player can attack, dodge, and lock on to ene- they want, but they must ring both before they can
mies, and a lot of time is spent hiding behind a do anything else.
shield. What sets Dark Souls apart from Zelda,
however, is resource management. Dark Souls has
a stamina bar that drains with nearly every action A Reputation for Difficulty of Dark Souls
the player can perform. This includes sprinting,
attacking, and dodging. This means that if the Dark Souls has built up a reputation of being one
player spams the attack button until they run out of the hardest games one could ever beat. Many
of stamina, they won’t have any stamina left to gamers are intimidated from trying Dark Souls, or
dodge an incoming attack until the bar refills. any other Soulslike games, due to this difficulty.
Patience, timing, and careful stamina usage are Despite this, the Souls series has plenty of devoted
important in order to succeed in Dark Souls. fans that sing its praises, and Dark Souls is seen as
Along the journey, the player will find bonfires. one of the best games ever made.
Bonfires act as checkpoints for the player, and Dark Souls is not an easy game. It’s likely that
allows them to replenish their health, magic, and first-time players will die hundreds of times before
Estus Flask, a magical healing drink that the they see the ending, so some players may begin to
player can use a limited number of times before feel frustrated if they are not making progress,
resting at a bonfire. However, bonfires will also which isn’t helped by the fact that the player will
respawn all previously defeated enemies except lose all their souls every time they die. Unless the
for bosses. lost souls were enough for the player to level up,
When the player dies, they respawn at the last its generally not worth going back for them. It’s
bonfire they visited and they lose all their souls, also recommended to spend all souls before ven-
which act as both experience points and a turing into a new area or fighting a boss.
Dark Souls Through the Lens of Essential Experience 519

The game also doesn’t always make it clear Cross-References

where players are supposed to go or how certain
mechanics work. Progression is linear, especially ▶ Dark Souls Through the Lens of Essential
early on, but it’s still possible to miss important Experience
items if the player doesn’t explore. If the player is ▶ Dark Souls III, An Analysis
confused about where to go next, it might be best
to look up the solution online or they might be
wandering around for hours. References
As the player progresses, they will find ene- D
mies with bigger health pools, and they might feel Ammerman, J.: Dark souls named ‘Ultimate Game of All
Time’ by Golden Joystick awards. Game Rant, 25 Nov.
that they are not doing enough damage. There is a
2021. https://gamerant.com/dark-souls-ultimate-game-
blacksmith named Andre at the end of the Undead all-time-golden-joystick-awards/ (2021)
Parish that will upgrade weapons and armor. Ciolek, T.: The history of from software. IGN, 16 Mar.
While it’s possible to beat the game without ever 2015. https://www.ign.com/articles/2015/03/16/the-
history-of-from-software (2015)
visiting Andre, the player will be weak, and their
Dark Souls Lore Explained!: YouTube, uploaded by
weapons and armor will eventually break. It is VaatiVidya, July 10, 2020. https://www.youtube.com/
recommended that the player backtracks to this watch?v¼McXJj7sjcZ0 (2020)
area occasionally to make sure their gear is up Hannah, S.: Every Fromsoftware Soulsborne game, ranked
according to number of sales. ScreenRant, ScreenRant,
to date.
10 Aug. 2021. https://screenrant.com/fromsoftware-
If the player is stuck on a boss, they have the soulsborne-games-most-copies-sold/ (2021)
option to summon an NPC or another player, MacDonald, K.: Dark souls review. IGN, 30 Sept. 2011.
though NPC summon signs are often hidden and https://www.ign.com/articles/2011/09/30/dark-souls-
review (2011)
there isn’t a guarantee other players will be online.
If summoning was fruitless, it might be worth
grinding or traveling to a different area, if
possible.
Dark Souls Through the Lens
of Essential Experience
Reception
Michael Phillips2 and Newton Lee1,2
1
Dark Souls is unlike most action RPGs that came Institute for Education, Research, and
before it. It can be ruthlessly difficult sometimes, Scholarships, Los Angeles, CA, USA
2
but the gameplay is tightly designed and balanced Vincennes University, Vincennes, IN, USA
in a way where the player won’t get stuck if they
explore every option they have. The satisfaction
of overcoming a difficult area or boss also makes Synonyms
continued play more rewarding than many other
action RPGs as well. The experience of playing Dark souls; Essential experience; Player
Dark Souls has stuck with many gamers. Keza experience
MacDonald of IGN gave it a 9 out of 10, saying
that it was “one of the most thrilling, most fasci-
nating and most completely absorbing experi- Definition
ences in gaming” (MacDonald 2011).
At the 2021 Golden Joystick Awards, Dark Lens of essential experience – A critical thought
Souls won the award for Ultimate Game of All process by applying the concept of player experi-
Time, beating out many other immensely popular ence in scrutinizing or analyzing a game
games like Doom, Minecraft, and The Legend of Former Disney game designer Jesse Schell
Zelda: Breath of the Wild (Ammerman 2021). stated that every memorable experience has
520 Dark Souls Through the Lens of Essential Experience

some key features that define it and make it special Milan Cathedral in Italy. As the player progresses
(Schell 2015). Without an impactful experience, through this lustrous world, they encounter many
how would video games be popular mediums of challenging bosses as they venture forward to
entertainment? The lens of essential experience succeed Lord Gwyn to fulfill the prophecy of
encourages game designers to create a whole returning the Lost Soul to the First Flame.
new world, whether familiar or alien, to witness Defeating bosses is the main objective, but the
something that its consumers could not find player must first explore each map as well as learn
through any other means. We humans experience the lore around the areas they encounter. The
things every single day, yet some of us still seek player virtually gets to undergo this world without
video games to fill our inquisitive minds. One limits as it is an open-world game; this means the
exemplification of games that portray excellent player gets to experience the game as they please,
player experience is Dark Souls, a third-person not being directly pointed in the right directions.
action role-playing game developed by In fact, the player is not spoon-fed with the quests
FromSoftware and published by Namco Bandai they must undertake. The player as a character
Games in 2011 for PlayStation 3 and Xbox 360. must decide on how to commence their adventure.
Dark Souls is known for its impossible level of There are many different routes the player may
difficulty, but it still managed to become a classic take to complete the game. This is an interesting
within the gaming community’s collections. idea as it means that their characters get to adven-
Dark Souls answers the three questions for the ture forward as they so choose.
lens of essential experience: “What experience do Among just experiencing the world alone, they
I want the player to have?”, “What is essential to have the option of inviting friends to ally them on
that experience?”, and “How can the game capture their journey or being invaded by enemy players
that essence?”. As stated before, Dark Souls is who are also playing the game. Friends enable an
known for its unbearable difficulty. How could a even higher level of difficulty because the bosses
game so difficult deliver such a memorable expe- increase their stats with each new ally. Enemy
rience? It is simple: the value of experiencing the players all have the objective of killing the player
massive world, bosses, and lore helps make this a on their adventure. In Dark Souls the player must
game to play. These elements are essential to approach bosses and the world around them with
being able to have an impactful experience. The an open mind, learn the patterns of the bosses, and
difficulty itself is the most unique experience a explore alternative pathways around the world all
person can have; without it, this game would not while cooperatively traversing with friends or
have been as popular of a game as it is. being attacked by foes.
The player must learn and observe the world All these elements of Dark Souls capture the
that Dark Souls sets in front of them. The only essence of experience. If any one of these attri-
reason Dark Souls is considered difficult may be butes had been excluded from the final build of the
because the player is thrown into an unfamiliar game, it would not have done as well as it did. The
environment. Of course, this is not a game that a lens of essential experience is an important lens
player can just jump into and expect positive for creating a masterpiece, and Dark Souls clearly
results; it’s up to the player to figure out shows how, if applied correctly, someone can
the mechanics, the boss’s attack patterns, and the develop a phenomenal video game.
map they are thrown into. The player takes the
role of a handcrafted character that they may
create themselves. The player has free reign over
the character that they assemble; this means the References
player gets to experience the world around them in
Bischoff, D.: Dark Souls Review. (2011). https://www.
a whole new light. Game director and producer
gamerevolution.com/review/53046-dark-souls-review
Hidetaka Miyazaki modeled various places in the MacDonald, K.: Dark Souls review. (2011). https://www.
game after real-world locations including the ign.com/articles/2011/09/30/dark-souls-review
Data Gloves for Hand and Finger Motion Interactions 521

Mitchell, R.: Dark Souls review: Brick walls and pancakes. tracking of the user’s hands and fingers. Since
(2011). https://www.engadget.com/2011/10/03/dark- direct sensing is employed, there are no environ-
souls-review-brick-walls-and-pancakes/
Schell, J.: The Art of Game Design: a Book of Lenses, mental restrictions and timely, highly reliable data
2nd edn. CRC Press, New York (2015) can be collected. Motion interactions are then
implemented through real-time analysis and rec-
ognition of the user’s hand and finger postures and
gestures. The employed nonoptical sensing
Data Baking method is particularly suitable for interactions of
deafblind users through an extended Malossi D
▶ Plug-in-Based Asset Compiler Architecture alphabet.

Introduction
Data Compiler
The first report of a finger motion tracking data
▶ Plug-in-Based Asset Compiler Architecture glove (DeFanti and Sandin 1977) appeared in
1977 and was followed by various research efforts
and implementations of glove-based input for
hand and finger motion digitization (Sturman
Data Cooking and Zeltzer 1994; Dipietro et al. 2008). The direct
tracking of finger motions pursued by this
▶ Plug-in-Based Asset Compiler Architecture approach is of particular interest in areas where
gloves are considered indispensable. Data gloves
can be employed, for example, in medical train-
ing, where surgical gloves are mandatory for
Data Gloves for Hand and many medical procedures. For adequate experi-
Finger Motion Interactions ence, such data gloves should match the physical
properties of the surgical gloves and provide very
Kamen Kanev1, Hidenori Mimura1 and similar touch and feel. To achieve this, novel
Patrick C. K. Hung2 materials are nowadays specifically designed to
1
Research Institute of Electronics, Shizuoka meet the requirements of the target application
University, Hamamatsu, Japan domain. In addition, recent technological
2
Faculty of Business and IT, Ontario Tech advancements are employed for implementing
University, Oshawa, ON, Canada finger motion tracking sensors that blend into the
glove fabric and allow for seamless integration.
The early wired glove designs (DeFanti and
Synonyms Sandin 1977) employed embedded optical sensors
constructed from light emitters and photocells.
Deafblind communication interfaces; Extended The components of the optical sensors were
Malossi alphabet; Gesture-based interactions; connected by flexible rubber tubes that bended
Hand and finger motion tracking; Motion and following the motions of the human fingers
posture analysis (Zimmerman 1985). In result, the light reaching
the photocells decreased proportionally to the
tube bend and thus controlled the output voltage
Definition of the photocell. A later method (Harvill et al.
1992) employing optical fibers with an intention-
Data gloves are wearable devices with incorpo- ally damaged surface that attenuated the light
rated sensors that allow for motion and posture propagation proportionally to the bend lead to
522 Data Gloves for Hand and Finger Motion Interactions

the construction of more flexible and easier to CNT sheets are fabricated by stacking and shrink-
handle optical sensors. More advanced fiber- ing the CNT Webs (Inoue et al. 2011).
optic sensors based on double cladding fiber Experimental CNT strain sensors are
(Ivanov and Chertoriyskiy 2015) were also manufactured by placing the CNT sheet on a flat
employed in high-end gloves such as the Virtual and smooth substrate (e.g., glass) in a direction
Programming Languages (VPL) Dataglove parallel to the stretching direction and impregnat-
(Zimmerman et al. 1986). ing it with elastomeric resin. This process requires
Capacitive and resistive bend sensors (Neely a rubber-like elastomeric resin with low elasticity
and Restle 1977) on the other hand were used in and low-loss properties such as polycarbonate-
consumer-grade products such as the Nintendo urethane (PCU) and segmented poly-
Power Glove (Gardner 1989). Note that some tetramethylene ether glycol-urethane (PTMGU)
bendable resistive sensors for wired gloves were to enhance the contraction properties of the sub-
produced by applying conductive inks to a flexi- strate (Suzuki et al. 2016).
ble substrate (Langford 1996). However, such To further stabilize the contraction behavior of
bendable sensors tend to produce an asymmetric the sensor, an elasticity-assist layer is applied on
response since conductive inks on a flexible sub- top of the elastomer resin surrounding the CNT
strate are more sensitive to expansion than to bundle. Since the resistance naturally increases
compression. with strain due to the cracking of the CNT sheet
Sensing of the bending is often insufficient when network, obtained elasticity can be adjusted by
high fidelity finger motion tracking and the detec- changing the number of layers in the employed
tion of subtle finger movements are targeted. For CNT sheets. This enables the construction of
this, stretchable sensors employing either capacitive stretchable sensors with predefined resistance
(Hirata et al. 2015; Tairych and Anderson 2019; that allows for more accurate motion tracking.
Glauser et al. 2019) or resistive (Firouzeh and Note that the sensor resistance is proportional
Paik 2015; Sbernini et al. 2016; Lee et al. 2018) to the applied tensile strain that increases with the
technologies appear to be more suitable. applied force. The temporal strain changes are
closely followed by the variation of the strain
sensor resistance that can exceed 200%. The
Advanced Carbon Nanotube (CNT) excellent rapid response of the sensors has been
Based Sensors confirmed through a large number of resistance
measurements during sensor extensions and con-
Rapid-response widely stretchable sensors are tractions at different frequencies (Suzuki et al.
indispensable for the detection of the subtle 2016). This has also confirmed the high linearity
human hand and finger motions. Those are spe- of the resistance variation with respect to the strain
cialized sensors derived from the recent research that further increases the motion and posture
on the synthesis of vertically aligned ultralong tracking precision.
multiwalled carbon nanotubes (CNT) using iron Different versions of the specialized stretch-
chloride powder (Inoue et al. 2008). The unique able CNT-based sensors have been commercial-
process essential for the reliable and cost-effective ized and are currently employed in Yamaha data
fabrication of the CNT material embedded in such gloves (Suzuki et al. 2016).
high-fidelity motion tracking sensors is as
follows.
A 2.1-mm long CNT array is first grown by Applications and Implementation
conventional thermal chemical vapor deposition Examples
on a quartz surface with a single gas flow of
acetylene for 20 min. It is then spun into a yarn, Data gloves employing CNT sensors discussed in
thus converting the three-dimensional array into a the previous section are capable of high fidelity
horizontally aligned web. Afterward, well-aligned motion tracking (Gelsomini et al. 2021). The
Data Gloves for Hand and Finger Motion Interactions 523

Conclusions

The early data gloves incorporated optical sensors

consisting of light emitters and photocells
connected with flexible rubber tubes. This allo-
wed for tracking of the finger motions with a
limited precision through measurements of the
attenuated light reaching the photocells. In more
advanced applications where higher fidelity finger D
motion tracking is required, stretchable sensors
are deemed more appropriate, and active research
Data Gloves for Hand and Finger Motion Interac- is continuing in this field.
tions, Fig. 1 A data glove with embedded highly stretch-
able CNT sensors connected to a dedicated wireless
communication device
Cross-References

Yamaha data gloves, for example, are employed ▶ Interactive Augmented Reality Pop-Up Book
for tracking of the subtle hand and finger with Natural Gesture Interaction for Handheld
motions during the play of different musical ▶ Object Manipulation Using Real Hand Gesture
instruments. Note that such motion tracking for Augmented Reality Interior Design
has to be implemented with a minimal burden ▶ Tracking Techniques in Augmented Reality for
to the artist so that the free natural movements of Handheld Interfaces
the hands and the fingers of the player are ▶ Virtual Hand Metaphor in Virtual Reality
unimpeded. To facilitate this, a wireless data
glove version has been developed as shown in
Fig. 1. References
Data gloves have been employed in a number
of research projects, beginning with an experi- DeFanti, T., Sandin, D.J.: Sayre Glove Final Project
Report. US NEA R60-34-163 Final Project Report
mental control of a robot hand in exergaming (1977)
and implementation of hand-based interactions Demoe, M., Uribe-Quevedo, A., Salgado, A.L., Mimura,
with mobile robots (Demoe et al. 2020). This has H., Kanev, K., Hung, P.C.K.: Exploring data glove and
been extended with a more general model for data robotics hand exergaming: lessons learned 2020. In:
IEEE 8th International Conference on Serious Games
glove-based human-robot communications and Applications for Health (SeGAH), pp. 1–8, Van-
including experimental support for deafblind couver (2020). https://doi.org/10.1109/SeGAH49190.
through an extended version of the Malossi alpha- 2020.9201747
bet (Gelsomini et al. 2022). There is work in Dipietro, L., Sabatini, A.M., Dario, P.: A survey of glove-
based systems and their applications. IEEE Trans. Syst.
progress on the integration of optical and data Man Cybern. Part C Appl. Rev., 461–482 (2008)
gloves input for improved sign language analysis Firouzeh, A., Paik, J.: The design and modeling of a novel
and interpretation through machine learning resistive stretch sensor with tunable sensitivity. IEEE
(Raavi et al. 2022). Sens. J. 15, 6390–6398 (2015)
Gardner, D.L.: Inside story on: the power glove (Cover).
With respect to medical applications and train- Des. News. 45(23), 63 (1989)
ing, data gloves have been instrumental in Gelsomini, F., Hung, P.C.K., Kapralos, B., Uribe-
researching the user experience aspects in wear- Quevedo, A., Jenkin, M., Tokuhiro, A., Kanev, K.,
able multi-device applications designed for health Hosoda, M., Mimura, H.: Specialized CNT-based sen-
sor framework for advanced motion tracking. In: The
systems (Salgado et al. 2021) and in conducting 54th Hawaii International Conference on System Sci-
experiments related to psychomotor-based virtual ences (HICSS-54), Symposium: Computing in Com-
medical training (Wilcocks et al. 2021). panion Robots and Smart Toys, pp. 1898–1905. Grand
524 Data Processor

Wailea, Maui (2021). https://doi.org/10.24251/HICSS. Suzuki, K., Yataka, K., Okumiya, Y., Sakakibara, S., Sako,
2021.231 K., Mimura, H., Inoue, Y.: Rapid-Response, Widely
Gelsomini, F., Tomasuolo, E., Roccaforte, M., Hung, P., Stretchable Sensor of Aligned MWCNT/Elastomer
Kapralos, B., Doubrowski, A., Quevedo, A., Kanev, Composites for Human Motion Detection. ACS Sen-
K., Makoto, H., Mimura, H.: Communicating with sors (2016)
humans and robots: a motion tracking data glove for Tairych, A., Anderson, I.: Capacitive stretch sensing for
enhanced support of deafblind. In: The 55th Hawaii robotic skins. Soft Robot. 6 (2019)
International Conference on System Sciences Wilcocks, K., Perivolaris, A., Kapralos, B., Quevedo, A.,
(HICSS-55), pp. 2056–2064. Grand Wailea, Maui Jenkin, M., Kanev, K., Mimura, H., Hosoda, M., Alam,
(2022). https://doi.org/10.24251/HICSS.2022.259 F., Doubrowski, A.: Work-in-progress: a novel data
Glauser, O., Panozzo, D., Hilliges, O., Sorkine-Hornung, glove for psychomotor-based virtual medical training.
O.: Deformation capture via soft and stretchable sensor In: 2021 IEEE Global Engineering Education Confer-
arrays. ACM Trans. Graph. 38, 1–6 (2019) ence (EDUCON), pp. 1318–1321, Vienna (2021).
Harvill, Y.L., Zimmerman, T.G., Grimaud, J.G.: Motion https://doi.org/10.1109/EDUCON46332.2021.
sensor which produces an asymmetrical signal in 9453962
response to symmetrical movement. US Patent 5 097 T. G. Zimmerman, “Optical Flex Sensor,” US Patent 4 542
252, 17 Mar 1992 291, September 17, 1985.
Hirata, I., Nakamoto, H., Ootaka, H., Tada, M.: The flex- Zimmerman, T., Lanier, J., Blanchard, C., Bryson, S.,
ible interface using a stretch sensor. Proc. Manuf. 3, Harvill, Y.: A hand gesture interface device. ACM
845–849 (2015) Sigchi Bull. 17, 189–192 (1986)
Inoue, Y., Kakihata, K., Hirono, Y., Horie, T., Ishida, A.,
Mimura, H.: One-step grown aligned bulk carbon
nanotubes by chloride mediated chemical vapor depo-
sition. Appl. Phys. Lett. 92 (2008)
Inoue, Y., Suzuki, Y., Minami, Y., Muramatsu, J.,
Shimamura, Y., Suzuki, K., Ghemes, A., Okada, M., Data Processor
Sakakibara, S., Mimura, H., Naito, K.: Anisotropic
carbon nanotube papers fabricated from multiwalled ▶ Plug-in-Based Asset Compiler Architecture
carbon nanotube webs. Carbon. 49 (2011)
Ivanov, O., Chertoriyskiy, A.: Fiber-optic bend sensor
based on double cladding fiber. J. Sens., 1–6 (2015)
Langford, G.B.: Flexible potentiometer. US Patent 5 583
476, 10 Dec 1996
Lee, H., Cho, H., Kim, S., Kim, Y., Kim, J.: Dispenser Data Visualization
printing of piezo-resistive nanocomposite on woven
elastic fabric and hysteresis compensation for skin- ▶ Artistic Data Visualization in the Making
mountable stretch sensing. Smart Mater. Struct. 27
(2018) ▶ Immersive Visualizations Using Augmented
Neely, J.S., Restle, P.J.: Capacitive bend sensor. US Patent Reality and Virtual Reality
5 610 528, 11 Mar 1977
Raavi, R., Kanev, K., Hung, P.C.K.: Integration of optical
and data gloves input for improved sign language anal-
ysis and interpretation through machine learning. In:
The 8th International Symposium toward the Future of
Advanced Research in Shizuoka University (ISFAR- Data Visualization of Mental
SU2022), p. 52, Shizuoka (2022) Health Issues
Salgado, A., Fung, B., Hung, P., Mimura, H., Kanev, K.,
Tokuhiro, A., Uribe-Quevedo, A.: User experience
aspects in wearable multi-device applications designed
▶ Indigenous Knowledge for Mental Health, Data
for health systems: lessons learned. In: The 6th Inter- Visualization
national Symposium on Biomedical Engineering
ISBE2021, pp. 212–213, Hamamatsu (2021)
Sbernini, L., Pallotti, A., Saggio, G.: Evaluation of a
Stretch Sensor for Its Inedited Application in Tracking
Hand Finger Movements, pp. 1–6. IEEE International DCI
Symposium on Medical Measurements and Applica-
tions (MeMeA), Benevento (2016)
Sturman, D.J., Zeltzer, D.: A survey of glove-based input. ▶ Engaging Dogs with Computer Screens: Ani-
IEEE Comput. Graph. Appl. 14(1), 30–39 (1994) mal-Computer Interaction
Dead Space Through the Lens of Resonance 525

1. What is it about my game that feels powerful

Dead Space Through the Lens and special?
of Resonance 2. When I describe my game to people, what
ideas get them really excited?
Devon Myers2 and Newton Lee1,2 3. If I had no constraints of any kind, what would
1
Institute for Education, Research, and this game be like?
Scholarships, Los Angeles, CA, USA 4. I have certain instincts about how this game
2
Vincennes University, Vincennes, IN, USA should be. What is driving those instincts?
D
Dead Space is a survival horror franchise that
Synonyms was published and released by Electronic Arts
(EA). There are three games in the franchise.
Survival horror Dead Space was released in 2008 for the Xbox
360, Playstation 3, and Windows PC. Dead Space
2 was released in 2011 for the Xbox
Definitions 360, Playstation 3, and Windows PC. Dead
Space 3 was released, once again, for the Xbox
Survival Horror – a subgenre of action-adventure 360, Playstation 3, and Windows PC in 2013.
and horror genres where players are frail and are Rather than making a player feel powerful, the
given few weapons, forcing them to approach games instead tried to make them feel as weak as
obstacles through problem-solving rather than possible with methods such as starting the game
combat. with a very small health. Story wise, the player
character is one of the only humans left on a
spaceship full of aliens trying to kill them in
Introduction several gruesome ways. Progressing through the
story gives players new weapons and abilities
The Lens of Resonance is to be mainly associated while also maintaining the very fragile health
with how powerful a game makes a player feel bar, which gives them a very good sense of
and if it gives a good consistency of this power power while also severely limiting abilities with
throughout the game. If the player were to feel like their health.
they are barely making a difference within Dead Space made itself different from other
the game it may cause the player to feel as if the games in the horror genre at the time by having the
journey is pointless and may discontinue the game game’s setting be in space. The vast emptiness of
due to this. This can be achieved through many space can be very unnerving even when the game
ways, such as completing a certain is at a lull. It can also make the player wonder
campaign mission, giving a character powerful what other aliens are out there and where they
attributes, or even giving the character a powerful came from. Other than the horror aspect, players
music ensemble that gives the player a sense of loved the look, lore, and the difficulty in killing
what events are important. Giving players a sense the aliens.
of power can go all the way back to Pong, one of As in several other horror games, Dead Space
the very first, if not the very first game. It gave the has several constraints to make the game more
player a feeling of power after winning a game, interesting, such as having limited ammo and a
bragging rights. very weak melee attack that makes players second
In Jesse Schell’s book The Art of Game guess their actions in a game where reactions are
Design: A Book of Lenses, the Lens of Resonance everything. Players that plan poorly or act rashly
states that “to use the Lens of Resonance, you can find themselves in a situation where they run
must look for hidden power” (Schell 2019). The out of ammo and have to resort to their weak
Lens of Resonance asks the following questions: melee attack or find a way to bypass an alien
526 Deafblind Communication Interfaces

obstacle entirely. If the creators did happen to

have no constraints, then there could be aspects Decoupling Game Tool GUIs
such as having unlimited ammo, an increased from Core Editing Operations
health bar, and possibly even gorier animations.
If these were implemented into the game, it would Nicusor Nedelcu
make the game too easy to beat and defeat the 7thFACTOR Entertainment Studios, Brasov
horror purpose. Other horror genre aspects County, Romania
include overly graphic and displeasing animations
from the aliens which would not be suitable for
the weak-willed. Synonyms
EA knew that they wanted to bring a new
breath of fresh air to the horror genre while keep- Decoupling systems; Plugin system; Command
ing a constant pace of action within the game as system; Editor architecture; Game level editor
well. But as the franchise developed, EA’s drive
changed from survival horror to add more action
genre aspects to Dead Space. Due to this influence Definition
from the action genre, the horror aspects were
overshadowed at some points during Dead A software architectural method of decoupling the
Space 3. This was a common complaint from game editing operations from the actual GUI of a
several critics and the playerbase. Statistics show game level editor.
that people much prefer the first and second games Since the early days of video game development
over the third, and this is again because of when the programmer had to write the code and
decreased focus on horror while maximizing the design and create the levels without the aid of a
importance of the action sequences. game editor, the tools to create games have evolved
into the must-have game development software we
use today. Now the level editors are built into the
development kits, and the developer’s job is much
References
easier but still filled with potential pitfalls.
Schell, J.: The Art of Game Design: A Book of Lenses, In the past few years, it has become common to
3rd edn. A K Peters/CRC Press (2019) decouple game level editor operations and func-
tionality from game-specific features, so that the
editor can be reused for more games and game
types. The same thing has happened on the game
engine side: engines have become more and more
Deafblind Communication flexible and reusable.
Interfaces But problems remain. One big issue with game
level editors is complexity and manageability.
▶ Data Gloves for Hand and Finger Motion Once you have added many features to the editor,
Interactions it will grow in source code size and complexity
and become harder and harder to maintain and
extend. Another problem is that you have to
choose a GUI toolkit to create your interface.
That can become a headache if you ever decide
Decision-Making to switch to another GUI toolkit, since many
editing operations are tied in with the UI code
▶ 2-Simplex Prism as a Cognitive Graphics Tool itself.
for Decision-Making To address the issue of changing GUI toolkits in
▶ Character Artificial Intelligence these fast and ever-shifting times, we present a
Decoupling Game Tool GUIs from Core Editing Operations 527

Decoupling Game Tool

GUIs from Core Editing
Operations, Fig. 1 The
editor ecosystem

method of decoupling the visual user interface code In Fig. 1, we can see the entire editor
from the non-GUI editing operations code in the ecosystem:
game level editor or other tools. By separating the The editor GUI can be developed using any UI
UI from core editing functions, you can change SDK/API, and it can have its own plug-ins. For
to another GUI toolkit in no time, leaving the example, subeditors like the model editor, cinematic
editing operation code almost untouched. The editor, scene editor, material editor, etc. can be hosted
decoupling operation can be accomplished via by the main editor, and we can even run them as
C++ editor core functionality code and various separate tools. Each tool will implement its own UI
editor user interfaces using GUI toolkits like Qt, functionality and will call commands by their name
MS WinForms, WPF, MFC, HTML5/JavaScript, and parameter values (arguments). The editor core
or even a command line editor UI, all using the will search its registered command list and dispatch
same editor functionality code as a common hub. the call to the appropriate plug-in command.
Communication between the editor functions We can also have an editor network layer,
and the visual interface is achieved through a which waits for tools to connect to it and simply
command system (basically the command pat- dispatches command calls and sends back their
tern). We will also explore the architecture of a results. There are various other methods of com-
plug-in system using this command communica- munication between the GUI and the editor core;
tion approach. these methods use IPC (inter-process communi-
cation) such as pipes, DDE, and shared memory
or files, but sockets are supported on all platforms,
Editor Ecosystem so they are the obvious first choice.

The editor is split into two major logical parts:

Editor Core C++ API
• Nonvisual, consisting of the editor core, plug-
ins, and their commands (no GUI) Now let’s get to the nuts and bolts of the actual
• Visual, created using the UI toolkit of your code. First we will declare our editor C++ inter-
choice, which will call the commands provided face to be used by the plug-ins. We are going to
by the plug-ins and editor core expose plug-in and command methods and
528 Decoupling Game Tool GUIs from Core Editing Operations

interfaces, a simple history (undo/redo) system, (Simplified Wrapper and Interface Generator),
and an event system (used for triggering events in e.g., having too many dependencies in the C++
the editor, plug-ins can register themselves as code would complicate things for the SWIG con-
event sinks to receive or trigger events). verter, sometimes failing to properly create a
Let’s start with the building block interfaces wrapper for other languages.)
related to commands, undo, events, and other After we define our simple types like uint32,
primitive constructs. We use a self-contained, we define a Handle union to be used as a pointer
independent header file, with only pure interfaces, transporter between the calling application and the
not relying on external headers so it can be easily editor core internals. This will keep things sim-
wrapped or converted to other languages. (It’s pler, since the user can’t use the pointer itself
especially important to keep the interfaces simple. anyway (see Listing 1).
If you were using something like SWIG

Listing 1. A Generic Handle Container.

// A generic handle, used to pass pointers in command parameters without

having to know the pointer type

union Handle
{
Handle()
: hVoidPtr(NULL)
{}
explicit Handle(int32 aVal)
: hInt(aVal)
{}
explicit Handle(int64 aVal)
: hInt64(aVal)
{}
explicit Handle(void* pVal)
: hVoidPtr(pVal)
{}
int32 hInt;
int64 hInt64;
void* hVoidPtr;
};

We will also need a Version structure to be used will have for the editor API and plug-in versions
in the various version comparisons/validations we (see Listing 2).

Listing 2. A Generic Version Holder Structure.

// A version structure, holding version information for plug-in

// or editor
Decoupling Game Tool GUIs from Core Editing Operations 529

struct Version
{
Version();
Version(uint32 aMajor, uint32 aMinor, uint32 aBuild);
bool operator <¼ (const Version& rOther) const;
bool operator >¼ (const Version& rOther) const;
Version& operator ¼ (const char* pVerStr);
uint32 major, minor, build;
}; D

After this, a central point of the editor core API commands and also by the main editor skeleton
is the main editor interface (see Listing 3), which application, which will manage those plug-ins.
will provide command, plug-in, and event
methods to be used by plug-ins and their

Listing 3. The Main Editor Core Interface.

// The editor main interface

struct IEditor
{
enum ELogMsgType
{
eLogMsg_Info,
eLogMsg_Debug,
eLogMsg_Warning,
eLogMsg_Error,
eLogMsg_Fatal
};
virtual ~IEditor(){}
virtual Version GetVersion() const ¼ 0;
virtual void PushUndoAction(IUndoAction* pAction) ¼ 0;
virtual bool CanUndo(uint32 aSteps ¼ 1) ¼ 0;
virtual void Undo(uint32 aSteps ¼ 1) ¼ 0;
virtual void Redo(uint32 aSteps ¼ 1) ¼ 0;
virtual void ClearHistory(int32 aSteps ¼ -1) ¼ 0;
virtual bool RegisterCommand(IPlugin* pPlugin,
TPfnCommand pCmdFunc,
const char* pCmdName) ¼ 0;
virtual bool UnregisterCommand(TPfnCommand pCmdFunc) ¼ 0;
virtual bool IsCommandRegistered(
const char* pCmdName) ¼ 0;
virtual bool RegisterEvent(IEvent* pEvent) ¼ 0;
virtual bool UnregisterEvent(IEvent* pEvent) ¼ 0;
virtual bool IsEventRegistered(
const char* pEventName) ¼ 0;
virtual bool TriggerEvent(IEvent* pEvent,
530 Decoupling Game Tool GUIs from Core Editing Operations

IEventSink::ETriggerContext aContext,
void* pUserData) ¼ 0;
virtual void CallEventSinks(IEvent* pEvent,
void* pUserData) ¼ 0;
virtual bool RegisterEventSink(
IEventSink* pEventSink) ¼ 0;
virtual bool UnregisterEventSink(IEventSink* pEventSink) ¼ 0;
virtual IParameterValues* CreateParameterValues() ¼ 0;
virtual IParameterDefinitions*
CreateParameterDefinitions() ¼ 0;
virtual bool Call(
const char* pCommandName,
IParameterValues* pParams) ¼ 0;
virtual void WriteLog(
ELogMsgType aType,
const char* pModule,
const char* pFormat, . . .) ¼ 0;
};

This is the main editor interface at a glance. Its commands in the editor’s ecosystem and provide
methods are quite self-explanatory, the most used information about these commands through a
methods being the Call(. . .) method, which is manifest file associated with the plug-in’s DLL.
used to execute commands by their name and A core editor plug-in consists of two files:
requires a parameter “bag” (optional), and the
IParameterValues interface, created before the • A C++ DLL file, the plug-in code (Example.
call by the user using the CreateParameterValues dll)
() method and then filling up the parameter values • A manifest file (Example.plugin.xml), having
for the command to use. the same base file name as the plug-in’s DLL
(Example), containing information about it

Plug-ins Listing 4 shows an example of a plug-in

manifest file.
The plug-ins are DLLs loaded by the core editor
DLL. Each plug-in will expose and register its

Listing 4. Plug-in Manifest File.

<plugin
name¼"Example"
description¼"The example editor plugin"
author¼"Nicusor Nastase Nedelcu"
url¼"http://some.com"
guid¼"31D91906-1125-4784-81FF-119C15267FC3"
version¼"1.0.0"
minEditorVersion¼"1.0.0"
maxEditorVersion¼"2.0.0"
Decoupling Game Tool GUIs from Core Editing Operations 531

icon¼"example.png"
unloadable¼"true">
<dependencies>
<depends nameHint¼”OtherPlugin”
guid¼”DAA91906-1125-4784-81FF-319C15267FC3” />
<depends nameHint¼”SomeOtherPlugin”
guid¼”F51A2113-1361-1431-A3EA-B4EA2134A111” />
</dependencies>
<commands> D
<command name¼”get_some_thing”
info¼”This command get something”>
<param name¼”someParam1” type¼”int32”
info¼”this is parameter 1” />
<param name¼”someParam2” type¼”float”
info¼”this is parameter 2” />
</command>
</commands>
</plug-in>

Of course, you can choose any format for the its commands. Only the manifest file holds that
manifest file, like JSON or a custom text format. information.
The important thing is that the plug-in’s DLL does Plug-ins can be located in a directory structure
not contain any information about the plug-in or as shown in Listing 5.

Listing 5. Example of Plug-in and Editor Directory Structure.

\Plugins
\Example1
Example1.dll
Example1.plugin.xml
\Example2
Example2.dll
Example2.plugin.xml
EditorCore.dll (the editor code library)
EditorUI.exe (the main editor application)

One reason we have chosen to store the plug-in editor configurations for lighting artists, program-
information inside external files is that plug-ins mers, or level designers, and these configuration
can be listed (with all their details) in the editor’s files can be shared among users.
plug-in manager without being loaded into mem- As you can see from the plug-in manifest, we
ory. In this way, we can avoid loading some plug- have added information about the name, descrip-
ins we do not need to load but still have informa- tion, author, and other useful properties but also
tion about them. For example, there can be special about the plug-in’s dependencies (other plug-in
532 Decoupling Game Tool GUIs from Core Editing Operations

GUIDs). Optionally, there should be information the discovery of incorrect command calls during
about the commands, like name, description, development.
parameters, and return values, since we do not For plug-in identification, we will use a GUID
store this information in the C++ source files. in the form shown in Listing 6.
This information can be used by a debug layer to
check the command syntax at runtime and help

Listing 6. The GUID Structure Used to Identify Plug-ins.

// A plug-in unique ID, in a GUID form (see more about Microsoft

// GUID) and online/offline GUID generators
struct PluginGuid
{
PluginGuid();
// construct the guid from several elements/parts
// example:
// as text: 31D9B906-6125-4784-81FF-119C15267FCA
// as C++: 0x31d9b906, 0x6125, 0x4784, 0x81,
// 0xff, 0x11, 0x9c, 0x15, 0x26, 0x7f, 0xca
PluginGuid(uint32 a, uint16 b, uint16 c, uint8 d,
uint8 e, uint8 f. uint8 g, uint8 h, uint8 i,
uint8 j, uint8 k);
bool operator ¼¼ (const PluginGuid& rOther) const;
// convert a GUID string to binary
// string format: "11191906-6125-4784-81FF-119C15267FC3"
bool fromString(const char* pGUID);
uint32 data1;
uint16 data2;
uint16 data3;
uint8 data4[8];
};

We will use the interface shown in Listing 7 to

get information about the discovered plug-ins
(gathered from the plug-in manifest files).

Listing 7. The Interface That Describes a Plug-in (from the Plug-in Manifest).

struct IPluginInfo
{
virtual ~IPluginInfo(){}
virtual const char* GetName() const ¼ 0;
virtual const char* GetDescription() const ¼ 0;
virtual const char* GetAuthor() const ¼ 0;
virtual const char* GetWebsiteUrl() const ¼ 0;
Decoupling Game Tool GUIs from Core Editing Operations 533

virtual PluginGuid GetGuid() const ¼ 0;

virtual Version GetVersion() const ¼ 0;
virtual Version GetMinEditorVersion() const ¼ 0;
virtual Version GetMaxEditorVersion() const ¼ 0;
virtual const char* GetIconFilename() const ¼ 0;
virtual bool IsUnloadable() const ¼ 0;
virtual PluginGuidArray GetPluginDependencies()
const ¼ 0;
}; D

The plug-in interface methods are easy to under- The simple creation process of new plug-ins
stand, but we can say more about the GetMinEdi- and commands should be the crux of this system;
torVersion() and GetMaxEditorVersion(). These thus, coding new command sets hosted in the
methods are used to check whether the plug-in can plug-ins should be straightforward. In the editor
be loaded into the current editor and help avoid core API, there is an interface each plug-in must
loading plug-ins that are not supposed to run under implement on its side, called IPlugin, as shown in
newer or older editor versions. Listing 8.

Listing 8. The Interface to Be Implemented by a Plug-in.

struct IPlugin
{
virtual ~IPlugin(){}
virtual void Initialize(IEditor* pEditor) ¼ 0;
virtual void Shutdown() ¼ 0;
virtual bool IsCommandEnabled(TPfnCommand pCmdFunc)¼ 0;
};

Commands A command executes only non-GUI-related

code so it will not deal with the GUI functions
We will create the editor core as a C++ DLL. This itself, only engine calls and game data. The
handles the loading of plug-ins that are exposing GUI code will take care of visual representa-
editing commands. The GUI will call the com- tion for the user, and it will call the available
mands using only the core editor interfaces (see commands.
Fig. 2). The plug-ins will expose their set of com-
The command system is designed as an mands, but they will have nothing to do with the
RPC-like (remote procedure call) architecture, GUI itself. You can create a separate plug-in sys-
where commands are actually functions that are tem for the editor’s GUI. This is where the true
called with arguments and return one or more decoupling kicks in, the editor core plug-ins being
values. The call itself can be made directly using just “buckets” of non-GUI-related commands
the editor core C++ API or the UI editor applica- with the editor GUI using those commands.
tion connecting sockets to an editor core server, There is no need for a 1:1 match between the UI
transmitting the command call data and then functions and the commands. You only need to
receiving the returned values. expose the basic/simple commands, which should
534 Decoupling Game Tool GUIs from Core Editing Operations

Decoupling Game Tool

GUIs from Core Editing
Operations, Fig. 2 The
command system diagram

be generic enough to be used by multiple UI tools define the parameter type, direction, and descrip-
in various situations. tion. This information is read from the plug-in’s
manifest file, but it’s optional since the calling of
commands is accomplished through a parameter
Command Parameters set that is aware of the data types at the moment of
setting the values. In Listing 9 we declare the
When calling the commands, we have the option IParameter interface.
to send parameters to them, and for this we need to

Listing 9. The Command Parameter Interface.

struct IParameter
{
enum EDataType
{
eDataType_Unknown,
eDataType_Int8,
eDataType_Int16,
eDataType_Int32,
eDataType_Int64,
eDataType_Float,
eDataType_Double,
eDataType_Text,
Decoupling Game Tool GUIs from Core Editing Operations 535

eDataType_Handle
};
enum EDirection
{
eDirection_Input,
eDirection_Output,
eDirection_InputOutput
};
virtual ~IParameter(){} D
virtual const char* GetName() const ¼ 0;
virtual const char* GetDescription() const ¼ 0;
virtual EDataType GetDataType() const ¼ 0;
virtual EDirection GetDirection() const ¼ 0;
virtual bool IsArray() const ¼ 0;
};

The IParameter interface is implemented by To keep the parameter information in one

the editor core DLL, so plug-in developers do place, we declare an IParameterDefinitions inter-
not need to care about the implementation, only face, which holds the parameter information list
what methods it provides, such as the name of the for a command as seen in Listing 10.
parameter, description, type, direction (if it’s an
in/out parameter), and whether the parameter is an
array of the type specified.

Listing 10. The Parameter Definition Container Interface.

struct IParameterDefinitions
{
virtual size_t GetCount() const ¼ 0;
virtual IParameter* Get(size_t aIndex) const ¼ 0;
virtual bool Add(
const char* pName,
IParameter::EDataType aDataType,
const char* pDescription,
IParameter::EDirection aDirection,
bool bArray) ¼ 0;
};

When calling the commands, we need to parameters, like #define extravaganza or tem-
pass the parameters. For this, we will use an plates to declare several command call forms
IParameterValues value “bag,” which can with from one to ten parameters in their dec-
set/get parameters and store the values. You laration. Listing 11 shows the parameter value
can use other approaches for passing interface.
536 Decoupling Game Tool GUIs from Core Editing Operations

Listing 11. The Parameter Value Container Interface.

// Parameter values container, used to pass and receive

// in/out parameter values from a command call
struct IParameterValues
{
virtual ~IParameterValues(){}
virtual void SetInt8(const char* pParamName,
int8 aValue) ¼ 0;
virtual void SetInt16(const char* pParamName,
int16 aValue) ¼ 0;
virtual void SetInt32(const char* pParamName,
int32 aValue) ¼ 0;
virtual void SetInt64(const char* pParamName,
int64 aValue) ¼ 0;
virtual void SetFloat(const char* pParamName,
float aValue) ¼ 0;
virtual void SetDouble(const char* pParamName,
double aValue) ¼ 0;
virtual void SetText(const char* pParamName,
const char* pValue) ¼ 0;
virtual void SetHandle(const char* pParamName,
Handle aValue) ¼ 0;
virtual void SetInt8Array(const char* pParamName,
Int8Array aArray) ¼ 0;
virtual void SetInt16Array(const char* pParamName,
Int16Array aArray) ¼ 0;
virtual void SetInt32Array(const char* pParamName,
Int32Array aArray) ¼ 0;
virtual void SetInt64Array(const char* pParamName,
Int64Array aArray) ¼ 0;
virtual void SetFloatArray(const char* pParamName,
FloatArray aArray) ¼ 0;
virtual void SetDoubleArray(const char* pParamName,
DoubleArray aArray) ¼ 0;
virtual void SetTextArray(const char* pParamName,
TextArray aArray) ¼ 0;
virtual void SetHandleArray(const char* pParamName,
HandleArray aArray) ¼ 0;
virtual int8 GetInt8(
const char* pParamName) const ¼ 0;
virtual int16 GetInt16(
const char* pParamName) const ¼ 0;
virtual int32 GetInt32(
const char* pParamName) const ¼ 0;
Decoupling Game Tool GUIs from Core Editing Operations 537

virtual int64 GetInt64(

const char* pParamName) const ¼ 0;
virtual float GetFloat(
const char* pParamName) const ¼ 0;
virtual double GetDouble(
const char* pParamName) const ¼ 0;
virtual const char* GetText(
const char* pParamName) const ¼ 0;
virtual Handle GetHandle( D
const char* pParamName) const ¼ 0;
virtual Int8Array GetInt8Array(
const char* pParamName) const ¼ 0;
virtual Int16Array GetInt16Array(
const char* pParamName) const ¼ 0;
virtual Int32Array GetInt32Array(
const char* pParamName) const ¼ 0;
virtual Int64Array GetInt64Array(
const char* pParamName) const ¼ 0;
virtual FloatArray GetFloatArray(
const char* pParamName) const ¼ 0;
virtual DoubleArray GetDoubleArray(
const char* pParamName) const ¼ 0;
virtual TextArray GetTextArray(
const char* pParamName) const ¼ 0;
virtual HandleArray GetHandleArray(
const char* pParamName) const ¼ 0;
// delete all parameter values
virtual void Clear() ¼ 0;
// get the number of parameters this list holds
virtual size_t GetCount() const ¼ 0;
// get the data type of parameter at given index
virtual IParameter::EDataType GetDataType(
size_t aIndex) const ¼ 0;
// get the direction of parameter at given index
virtual IParameter::EDirection GetDirection(
size_t aIndex) const ¼ 0;
// get the data type of parameter at given index
virtual const char* GetName(size_t aIndex) const ¼ 0;
// is this parameter an array at given index?
virtual bool IsArray(size_t aIndex) const ¼ 0;
};

To avoid memory fragmentation due to fre- The actual command is a callback function
quent command calls, you would ideally manage receiving a parameter values set and is declared
the parameter values through a memory pool. as shown in Listing 12.
538 Decoupling Game Tool GUIs from Core Editing Operations

Listing 12. The Command Callback Function Type.

typedef void (*TPfnCommand)(IParameterValues* pParams);

For debugging and auto-documentation pur- description from the plug-in manifest file, plus
poses, the editor core API can provide detailed the command callback function pointer, as
command information through the ICommand shown in Listing 13.
interface, which can hold the command

Listing 13. The Command Information Provider Interface.

struct ICommand
{
virtual ~ICommand(){}
virtual const char* GetName() const ¼ 0;
virtual const char* GetDescription() const ¼ 0;
virtual const char* GetIconFilename() const ¼ 0;
virtual TPfnCommand GetCommandFunc() ¼ 0;
virtual const IParameterDefinitions*
GetParameterDefinitions() const ¼ 0;
};

Direct Editor API Command Calls tool for another language like C# (SWIG). To call
the commands in C++, we can use the code shown
You can call the editor core interface for executing in Listing 14.
commands directly from C++ or use a wrapper

Listing 14. How to Call a Command.

// create a parameter values bag

IParameterValues* pParams ¼
pEditor->CreateParameterValues();
// set some parameter values
pParams->SetInt32(“someParam1”, 123);
pParams->SetText(“someName”, “Elena Lenutza Nedelcu”);
pParams->SetText(“someOtherName”, “Dorinel Nedelcu”);
// the actual command call
pEditor->Call(“someCommandName”, pParams);
// retrieve the return values
float fRetVal ¼ pParams->GetFloat(“returnSomeValue”);
int someNum ¼ pParams->GetInt32(“otherValue”);
Decoupling Game Tool GUIs from Core Editing Operations 539

Remote Editor API Command Calls though C# can also be supported using Mono on
platforms other than Windows). This editor will
We can use sockets for calling the commands be an empty skeleton that contains a plug-in man-
remotely, since they’re cross-platform and rela- ager dialog and nothing else, since all the func-
tively easy to use from any language or environ- tionality will be brought in by the plug-ins. Once
ment. On the editor core DLL side, we will have a again we need to emphasize the separation of the
network server executable, and on the editor UI plug-in systems. They are two systems, one for the
side, we will have a network client sending and UI and one for the editor core commands. UI plug-
receiving command data. ins will use the commands found in the editor core D
Communication can be accomplished through plug-ins (see Fig. 1 at the beginning of this arti-
reliable UDP or TCP. For a local editor on the cle). The main UI editor can even do without a
same machine, TCP would be okay even for LAN plug-in system if it’s so intended, but the editor
scenarios. If you are not so keen on using TCP core command plug-ins will still exist.
because you consider it slow, UDP should suffice
to send commands. All logic remains the same in Implementing a Plug-in with Commands
this networked scenario, but this setup opens the To ensure that you have a simple way of
doors to online collaboration of multiple clients implementing new commands, the method of
operating on the same data on the server. We’re declaring commands and plug-ins must be
not going to discuss this here, since it’s a subject straightforward.
for a whole article (a challenging and interesting In the editor core API, the IPlugin is the inter-
one!). face a plug-in must implement. To help rapid
Networked editing is also feasible for plug-in development, you can write a series of
debugging and remote in-editor live tutorials. macros. In our sample plug-in, implementing a
few commands would look like the code shown
in Listing 15.
Putting It All Together

Implementing the Main Editor Application

The editor can be implemented in Qt (just an
example, chosen for its cross-platform support,

Listing 15. A Sample Plug-in Implementation.

#include “EditorApi.h”
void example_my_command1(IParameterValues* pParams)
{
// get our calling parameter values
int numberOfHorses ¼
pParams->GetInt32("numberOfHorses");
std::string dailyMessage ¼
pParams->GetText("dailyMessage");
// do something important here for the command. . .
// return some parameter values
pParams->SetDouble("weightOfAllHorses", 1234.0f);
pParams->SetText("userFullName", "Iana Lomos");
}
void example_my_command2(IParameterValues* pParams)
540 Decoupling Game Tool GUIs from Core Editing Operations

{
// now here we’ll try to grab an array
FloatArray magicFloats ¼
pParams->GetFloatArray("magicFloats");
for (size_t i ¼ 0; i < magicFloats.count; ++i)
{
float oneMagicFloat ¼ magicFloats.elements[i];
// do something majestic with the float. . .
}
// we do not need to return any value now
}
BEGIN_PLUGIN
void Initialize(IEditor* pEditor)
{
REGISTER_COMMAND(example_my_command1);
REGISTER_COMMAND(example_my_command2);
}
// used to check if a command is disabled at that time
// can be helpful for UI to disable buttons in toolbars
// or other related visual feedback
bool IsCommandEnabled(TPfnCommand pCmdFunc)
{
return true;
}
void Shutdown()
{
}
END_PLUGIN

Note that BEGIN_PLUGIN and Be sure to name the commands in a way that
END_PLUGIN are macros hiding the start/end avoids conflicts. Usually some sort of group
of the IPlugin interface implementation. The Ini- naming, like the name of the plug-in and the
tialize method is called when the plug-in is loaded actual command action name, should be
into the editor. We are also registering the plug- enough, like assets_reload, assets_set_tag,
in’s commands by just referring invoking the assets_delete, or if you prefer camel-case style,
global functions example_my_command1 and Assets_SetTag.
example_my_command1. The Shutdown method The generated plug-in will be named example.
is called when the plug-in is unloaded (no need to dll and will be accompanied by its manifest file,
call the unregister commands; this can be tracked example.plugin.xml. Of course the plug-in must
and executed by the editor core itself, since it export a CreatePluginInstance global function so
knows the IPlugin pointer when the commands the editor core can load it and instantiate the
are registered). The IsCommandEnabled method IPlugin implementation.
is used to verify whether a command has the status
of “enabled” so it can be called/executed.
Decoupling Game Tool GUIs from Core Editing Operations 541

Events themselves as event sinks, as shown in

To make the plug-ins aware of events occur- Listing 16.
ring in the editor ecosystem, they can register

Listing 16. An Event Sink, Which Can Be Implemented by the Plug-ins.

// Every plug-in can register its event sink so it can

// receive notifications about events happening in the D
// editor ecosystem, coming from other plug-ins or the
// editor core itself
struct IEventSink
{
// When the event sink call is received, before, during or
// after the event was consumed
// The eTriggerContext_During can be used to have
// lengthy events being processed and many triggered to
// update some progress bars
enum ETriggerContext
{
eTriggerContext_Before,
eTriggerContext_During,
eTriggerContext_After
};
virtual ~IEventSink(){}
virtual void OnEvent(IEvent* pEvent,
ETriggerContext aContext,
void* pUserData) ¼ 0;
};

The IEventSink::OnEvent method is called event sinks. The method receives a pointer to the
whenever an event is triggered by other plug-ins triggered event interface (see Listing 17).
or their commands and broadcast to the registered

Listing 17. An Event, Implemented by the Trigger Code.

// An event is triggered when certain actions are happening

// in the editor or its plug-ins. For example we can have an
// event at Save level or an object moved with the mouse
struct IEvent
{
virtual ~IEvent(){}
virtual const char* GetName() ¼ 0;
virtual void OnTrigger(void* pUserData) ¼ 0;
virtual void* GetUserData() ¼ 0;
};
542 Decoupling Game Tool GUIs from Core Editing Operations

Listing 18 shows how to trigger an event.

Listing 18. Creating, Registering, and Triggering an Event.

// we declare an event
struct MyEvent: IEvent
{
virtual const char* GetName()
{
return “MyCoolEvent”;
}
// this will be called when the event is triggered,
// before being broadcast to all the event sinks
// so the event can even modify the user data
virtual void OnTrigger(void* pUserData)
{
// modify or store the pUserData
m_pData ¼ pUserData;
}
virtual void* GetUserData()
{
return m_pData;
}
uint8 m_pData;
} s_myEvent;
// we register an event (usually in the Initialize method
// of the plug-in)
...
REGISTER_EVENT(&s_myEvent);
...
// in some command, we trigger the event
void my_command(IParameterValues* pParams)
{
uint8* pSomeData;
// ....... do things with pSomeData
g_pEditor->TriggerEvent(
&s_myEvent,
IEventSink::eTriggerContext_After,
pSomeData);
}

In some plug-ins, an event sink registered for a

particular event would be notified of the event
being triggered, as shown in Listing 19.
Decoupling Game Tool GUIs from Core Editing Operations 543

Decoupling Game Tool GUIs from Core Editing Operations, Fig. 3 Skeleton editor UI application and settings
dialog, with the plug-in manager (made with Qt)

Listing 19. Creating and Registering an Event Sink.

// declare our event sink

struct MyEventSink: IEventSink
{
void OnEvent(IEvent* pEvent,
ETriggerContext aContext,
void* pUserData)
{
// is this the event we’re looking for?
if (!strcmp(pEvent->GetName(), “MyCoolEvent”))
{
uint8* pEventData ¼ pEvent->GetUserData();
// . . .do things when that event was triggered
}
}
} s_myEventSink;
// inside the plug-in’s Initialize method, register
544 Decoupling Systems

// the event sink

...
pEditor->RegisterEventSink(&s_myEventSink);
...

In Fig. 3, you can see a demo application of this

system, with the editor skeleton having just one Deep Learning Algorithms for
menu item, and a setting dialogue where plug-ins 3D Reconstruction
are managed.
Junzi Yang and Ajune Wanis Ismail
Mixed and Virtual Reality Research Lab,
Conclusion Vicubelab, School of Computing, Faculty of
Engineering, Universiti Teknologi Malaysia,
Decoupling the UI from core editing functionality Johor Bahru, Malaysia
helps the development and fast feature-set itera-
tion of game creation tools, since fewer hard-
coded dependencies and monolithic schemes are Synonyms
used. The tools can be extended and used for a
wide range of projects, the editor itself being quite 3D Reconstruction; Deep Learning; Human
agnostic to the game type and even the engine Detection
used. The solution presented here can be
implemented in many ways, from the command
system interfaces to the UI or plug-in system. In Definition
all cases, one thing remains constant: the use of
UI-independent editing operations is separated 3D human reconstruction is an important research
from the tools’ GUI using the command layer. topic in VR/AR content creation, virtual fitting,
I hope this article inspires you to make the right human-computer interaction, and other fields.
choices when creating extensible, elegant solu- Deep learning theory has made important
tions for your game development tools. achievements in human motion detection, recog-
nition, tracking, and other aspects, and human
motion detection and recognition is an important
link in 3D reconstruction. In this entry, the deep
Decoupling Systems learning algorithms in recent years, mainly used
for human motion detection and recognition, are
▶ Decoupling Game Tool GUIs from Core reviewed, and the existing methods are divided
Editing Operations into three types: CNN-based, RNN-based, and
GNN-based. At the same time, the main stream
data sets and frameworks adopted in the refer-
ences are summarized. The content of this entry
Deep Learning provides some references for the research of 3d
reconstruction of human motion.
▶ Deep Learning Algorithms for 3D Reconstruc-
tion
▶ Deep Reinforcement Learning in Virtual Envi- Introduction
ronments
▶ Human Interaction in Machine Learning (ML) The earliest study of human movements was pro-
for Healthcare posed by Johansson (1973). The motion of the
▶ Machine Learning for Computer Games main joints is described by a few bright spots
Deep Learning Algorithms for 3D Reconstruction 545

against a dark background. The effectiveness of algorithm is introduced in detail. Then, according
the kinetic-geometric model for visual vector to different reconstruction methods, the latest
analysis is verified. This study opens the door research progress is introduced from two aspects
for human motion analysis. Since almost all of direct reconstruction and two-stage reconstruc-
human behavior understanding needs to be based tion. Finally, the framework of deep learning and
on accurate motion reconstruction, human motion the data set used in the research literature are
reconstruction is a hot topic in the field of com- introduced.
puter vision. At present, the 3D reconstruction
technology of human motion has been widely D
applied in human-computer interaction, costume Deep Learning Fundamentals
design, virtual fitting, and games.
From the technical point of view, there are For different types of data and problems, people
many deep learning algorithms for 3D reconstruc- have studied all kinds of neural network structural
tion of human motion detection, and different models. Now, the mainstream methods of deep
algorithms have different processing methods. learning technology in 3d reconstruction of
Based on the standard RNN model typically human motion detection are mainly CNN, RNN,
used for human motion, Martinezet et al. (2017) and GNN as the basic framework or their combi-
develop a sequence-sequence model with residual nation, and has achieved remarkable results.
connections. Its performance is better than the Figure 1 shows the overall structure of
early human motion prediction work and achieves convolutional neural network, which is mainly
good results. Li et al. (2020) propose a dynamic composed of convolution layer, activation func-
multi-scale graph neural network (DMGNN) tion, pooling layer, and full connection layer.
which is adaptive during training and a multiscale LeCun et al. first proposed Convolutional Neural
graph computational unit (MGCU). Despite the Network (CNN) in 1998 (LeCun et al. 1998). The
continuous improvement of deep learning algo- structure of LeNet network is divided into eight
rithms, motion pose estimation has been a recog- layers, which mainly uses the principle of image
nized problem for researchers in the study of local correlation to process image data. AlexNet is
computer vision. A good attitude estimation a Convolutional Neural Network developed by
method needs to be robust to occlusion and defor- Krizhevsky et al. In the ILSVRC competition of
mation, stable to changes caused by factors such that year, the error rate of Top5 is reduced to
as illumination and clothing, and the human body 15.315%. Compared with Lenet-5, AlexNet uses
is a hinged object with different attitudes, so it is a deeper network structure, with 5 convolution
difficult to keep absolute static. layers, 3 full connection layers, 60 million param-
This paper reviews the 3D reconstruction of eters, and 65,000 neurons. AlexNet uses two Gpus
human motion detection based on deep learning for calculation, which greatly improves the com-
algorithm. Firstly, the foundation of deep learning puting efficiency. The sigmoid function and Tanh

Deep Learning Algorithms for 3D Reconstruction, Fig. 1 Schematic diagram of CNN (LeCun et al. 1998)
546 Deep Learning Algorithms for 3D Reconstruction

function are replaced with the non-saturated non- time. Therefore, RNN can only deal with the
linear function ReLU function (Krizhevsky et al. problems of short sequence dependence. LSTM
2012). Since AlexNet, deep learning has enjoyed a (Hochreiter and Schmidhuber 1997) is an
Renaissance. In the following years, various improvement on RNN. Different from the cyclic
Convolutional Neural Network models based on layer in the basic structure of RNN, LSTM uses a
the basic structure of AlexNet spring up, such as gate control mechanism in the memory unit and
VGGNet, GooleNet, and ResNet (He et al. 2016). combines short-term memory with long-term
The proposal of ResNet is an improvement on the memory. It can learn the content with long time
degradation of deep network structure, which is a dependence and alleviate the problem of gradient
milestone event in the history of CNN image pro- explosion and disappearance to a certain extent.
cessing. Since then, the research focus of the aca- GRU is improved on the basis of LSTM. It has the
demia has changed from how to improve the same effect as LSTM, but it is improved in struc-
accuracy of the neural network to how to achieve ture. GRU (Cho et al. 2014) simplifies the three
the same accuracy with less parameters and calcu- “gates” of LSTM structure to two “gates”. To
lation. SqueezeNet is a typical example (Iandola prevent the gradient from disappearing and
et al. 2016). With the deepening of deep learning exploding, IndRNN (Li et al. 2018) introduces
research, more CNN models and design ideas have Relu as the activation function, and separates the
been adopted into the network model design of 3D neurons in the layer, which can also build a deeper
human motion reconstruction, which greatly pro- and longer network and make the network learn
motes the development of 3D human motion for a long time. Dual-path Recursive Neural Net-
reconstruction technology. work (DPRNN) (Luo et al. 2020) is an effective
RNN (Recurrent Neural Network) is used to and simple way to organize RNN layers in deep
process sequence data. The difference between structures to make RNN model long sequences.
CNN and RNN is that a directional loop is formed The experimental result shows that replacing one-
between neurons, in which the hidden state at the dimensional CNN with DPRNN in TasNet can
last moment and the input at this moment are both improve the experimental results (Fig. 3).
the input of neurons, so the network can remem- Scarselli et al. (2008) first proposed the con-
ber the information at the previous moment. The cept of graph neural network in their entry. In the
structure of circulation unit is shown in Fig. 2. entry, they used neural network on graph structure
RNN is unidirectional propagation. Based on this, data. GNN also has many limitations and is suit-
bidirectional RNN (Schuster and Paliwal 1997) is able for shallow structures, most with no more
proposed. In the process of training RNN, due to than three layers. Graph Convolutional Neural
the repeated multiplication of weights, the prob- Network (GCN) (Kipf and Welling 2016) summa-
lems of gradient disappearance and explosion are rizes the convolutional operation from grid data to
easy to occur, and it is difficult to learn for a long graph data, which is a combination of CNN and

Deep Learning Algorithms for 3D Reconstruction, Fig. 2 Schematic diagram of RNN (Schuster and Paliwal 1997)
Deep Learning Algorithms for 3D Reconstruction 547

graph topology structure, and implements multi- technologies have their own characteristics.
layer stacking. When constructing GCN, two Depending on the specific properties of these
methods, spectral method and non-spectral methods, how to apply these deep learning
method, are usually followed. The Graph Atten- methods to 3D reconstruction of human motion
tion Network (GAT) (Busbridge et al. 2019) intro- is extremely important. Based on the types of deep
duces the attention mechanism based on GCN. learning methods, this section elaborates 3D
Masked self-attentional layers is introduced to reconstruction of human motion detection from
improve the disadvantages of graph convolution. the following three aspects: (i) CNN based,
Assigning corresponding weights to different (ii) RNN based, human motion reconstruction. D
adjacent nodes requires neither matrix operation
nor prior knowledge of the graph structure. The CNN Approach
model has better performance and is more robust CNN is the most widely used neural network
to disturbances. Relational Graph Attention Net- among all the neural networks, and it is also the
works (Busbridge et al. 2019) applies the Atten- first one used for 3D reconstruction of human
tion mechanism to graph convolution and adds motion detection. It can process images and any
relational information to the model, thus kind of data that can be converted into a similar
extending the non-relational graph attention image structure. Tompson et al. (2014) propose to
mechanism. Relational Graph Attention Net- use CNN to make pose estimation and use heat
works is an extension of GAT and has broader map to regression the key points. The method
applications. Aiming at the problems of GCNs, optimizes the prediction results by using the struc-
Self-Supervised Semantic Alignment for Graph tural relations between key points and markov
Convolution Network (SelfSAGCN) is proposed. random field. LeCun’s team proposes a novel
Identity Aggregation and Semantic Alignment are architecture in which refined models are cascered
its two key approaches. This algorithm reduces with the latest CNN models, including an effec-
over-smoothing and enhances the similarity tive “position refinement” model that can be
between unlabeled features and labeled features trained to estimate joint offset positions in small
of the same class. Experimental results show that areas of the image (Tompson et al. 2015). Stacked
the algorithm is better than other methods in var- hourglass networks is a cascade of funnel-like
ious classification tasks (Yang et al. 2021). neural networks, each of which acts as an encoder
and decoder to extract features and generate heat
map results (Newell et al. 2016). In recent years,
3D Reconstruction with Deep Learning many studies on human pose estimation (single or
multiple) have been based on this basic network
3D reconstruction of human motion detection is a structure, as well as another network structure,
very complicated process. As discussed earlier, OpenPose. OpenPose is an open source project
the mainstream approaches to deep learning at Carnegie Mellon University based on models

Deep Learning
Algorithms for 3D
Reconstruction,
Fig. 3 Graph
Convolutional Neural
Network (Kipf and Welling
2016)
548 Deep Learning Algorithms for 3D Reconstruction

Deep Learning Algorithms for 3D Reconstruction, Fig. 4 Multi-Resolution Sliding-Window with Overlapping
Receptive Fields (Brau and Jiang 2016)

from three papers. One of the papers (Cao et al. and spatial features of skeleton data in human
2017) describes the 2D pose detection method motion recognition tasks (Yang et al. 2020). In
PAF (Part Affinity Fields) in multi-player images, the attention subnetwork of spatial dimension, the
which first detects various points and then con- author uses LSTM network to learn the relation-
nects them with individuals to realize real-time ship between the nodes of the current frame and
detection of multiple people. Ke et al. (2018) the nodes of the previous frame, form a currently
improve the recent deep convolution and input attention map frame node data, and auto-
deconvolution hourglass model in four key points, matically find the current frame data of skeleton
and develop a robust multi-scale structural per- points, which has the greatest impact on action
ceptual neural network for human pose estima- recognition. In the attention subnetwork of the
tion. Figure 4 shows deep learning architecture time dimension, the author uses the LSTM net-
consists of three main components, a work to learn the relationship between the current
convolutional neural network (CNN), a camera frame and the previous frame, to form the atten-
projection and bone length computation layer, tion map of the current input frame data, and
and a 3D pose prior network (Brau and Jiang automatically learn which video frames contribute
2016). the most to action recognition. Zhang et al. (2017)
propose an adaptive recursive neural network
RNN Approach (RNN) based on LSTM structure, instead of
The dynamic characteristics and context- relocating the skeleton based on human defined
dependent information of action can be captured prior criteria. This allows the network itself to
by using recurrent neural network (RNN). Based adapt from one end to the other to the most appro-
on layered bone input, a multi-layer RNN frame- priate point of view. Xiang and Fox (2017) inte-
work is proposed by Du et al. (2019). In the grate DA-RNN’s with KinectFusion (Newcombe
algorithm, the human body is divided into five et al. 2011) for semantic 3D scene reconstruction.
parts, and then each part is input into five subnets KinectFusion was known as an accurate real-time
for training. In the end, the extracted features are mapping of indoor scenes in variable lighting
input into a single layer perceptron to determine conditions (Fig. 5).
the action category. Based on the RNNs with Multistage convolutional neural network
Long Short-Term Memory (LSTM), an atten- (CNN) has made advanced achievements in real-
tional mechanism is proposed to learn temporal izing single image human posture estimation, but
Deep Learning Algorithms for 3D Reconstruction 549

Deep Learning
Algorithms for 3D
Reconstruction,
Fig. 5 The pixel labels
provided by the RNN are
integrated into the 3D
semantic map (Newcombe
et al. 2011)

its application to video requires a lot of calcula- He, K., Zhang, X., Ren, S., & Sun, J.: Deep residual
tion, and there will be performance degradation learning for image recognition. In Proceedings of the
IEEE Conference on Computer Vision and Pattern
and jitter. A new recursive LSTM model is pro- Recognition, pp. 770–778 (2016)
posed for video pose estimation (Luo et al. 2018). Hochreiter, S., Schmidhuber, J.: Long short-term memory.
Artacho and Savakis (2020) proposed UniPose Neural Comput. 9(8), 1735–1780 (1997)
and Unipos-LSTM architectures for single image Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K.,
and video pose estimation respectively. This Dally, W. J., Keutzer, K.: SqueezeNet: AlexNet-level
accuracy with 50x fewer parameters and< 0.5 MB
structure can better understand the context infor- model size. arXiv preprint arXiv:1602.07360 (2016)
mation in the framework and help to estimate the Johansson, G.: Visual perception of biological motion and
posture of the subject more accurately. a model for its analysis. Percept. Psychophys. 14(2),
201–211 (1973)
Ke, L., Chang, M. C., Qi, H., & Lyu, S.: Multi-scale
structure-aware network for human pose estimation.
References In Proceedings of the European Conference on Com-
puter Vision (ECCV), pp. 713–728, (2018).
Artacho, B., Savakis, A.: Unipose: Unified human pose Kipf, T. N., Welling, M.: Semi-supervised classification
estimation in single images and videos. In Proceedings with graph convolutional networks. arXiv preprint
of the IEEE/CVF Conference on Computer Vision and arXiv:1609.02907. (2016)
Pattern Recognition, pp. 7035–7044 (2020) Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet clas-
Brau, E., Jiang, H.: 3d human pose estimation via deep sification with deep convolutional neural networks.
learning from 2d annotations. In: 2016 Fourth Interna- Adv. Neural Inf. Proces. Syst. 25, 1097–1105 (2012)
tional Conference on 3D Vision (3DV), pp. 582–591. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-
IEEE (2016) based learning applied to document recognition.
Busbridge, D., Sherburn, D., Cavallo, P., Hammerla, N.Y.: Proc. IEEE. 86(11), 2278–2324 (1998)
Relational graph attention networks. arXiv preprint Li, S., Li, W., Cook, C., Zhu, C., Gao, Y.: Independently
arXiv:1904.05811 (2019) recurrent neural network (indrnn): Building a longer
Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi- and deeper rnn. In Proceedings of the IEEE conference
person 2d pose estimation using part affinity fields. In on computer vision and pattern recognition,
Proceedings of the IEEE Conference on Computer pp. 5457–5466 (2018)
Vision and Pattern Recognition, pp. 7291–7299 (2017) Li, M., Chen, S., Zhao, Y., Zhang, Y., Wang, Y., Tian, Q.:
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Dynamic multiscale graph neural networks for
Bougares, F., Schwenk, H., Bengio, Y.: Learning 3d skeleton based human motion prediction. In
phrase representations using RNN encoder-decoder Proceedings of the IEEE/CVF Conference on Com-
for statistical machine translation. arXiv preprint puter Vision and Pattern Recognition, pp. 214–223
arXiv:1406.1078 (2014) (2020)
Du, B., Peng, H., Wang, S., Bhuiyan, M.Z.A., Wang, L., Luo, Y., Ren, J., Wang, Z., Sun, W., Pan, J., Liu, J., . . . Lin,
Gong, Q., et al.: Deep irregular convolutional residual L. Lstm pose machines. In Proceedings of the IEEE
LSTM for urban traffic passenger flows prediction. Conference on Computer Vision and Pattern Recogni-
IEEE Trans. Intell. Transp. Syst. 21(3), 972–985 (2019) tion, pp. 5207–5215 (2018)
550 Deep Reinforcement Learning

Luo, Y., Chen, Z., Yoshioka, T.: Dual-path rnn: efficient

long sequence modeling for time-domain single- Deep Reinforcement Learning
channel speech separation. In: ICASSP 2020–2020
IEEE International Conference on Acoustics, Speech in Virtual Environments
and Signal Processing (ICASSP), pp. 46–50. IEEE
(2020) Feng Lin1,2 and Hock Soon Seah2
Martinez, J., Black, M. J., Romero, J.: On human motion 1
China-Singapore International Joint Research
prediction using recurrent neural networks. In Proceed-
ings of the IEEE Conference on Computer Vision and Institute, Singapore, Singapore
2
Pattern Recognition, pp. 2891–2900 (2017) School of Computer Science and Engineering,
Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Nanyang Technological University, Singapore,
Kim, D., Davison, A.J., et al.: Kinectfusion: Real-time Singapore
dense surface mapping and tracking. In: In 2011 10th
IEEE International Symposium on Mixed and Aug-
mented Reality, pp. 127–136. IEEE (2011)
Newell, A., Yang, K., Deng, J.: Stacked hourglass net- Synonyms
works for human pose estimation. In: European confer-
ence on computer vision, pp. 483–499. Springer, Cham
(2016) Deep learning; Deep reinforcement learning;
Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Humanoid avatar; Reinforcement learning;
Monfardini, G.: The graph neural network model. Unmanned aerial vehicle; Virtual reality
IEEE Trans. Neural Netw. 20(1), 61–80 (2008)
Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural
networks. IEEE Trans. Signal Process. 45(11),
2673–2681 (1997) Definitions
Tompson, J.J., Jain, A., LeCun, Y., Bregler, C.: Joint train-
ing of a convolutional network and a graphical model Deep learning (DL) is a special type of machine
for human pose estimation. Adv. Neural Inf. Proces.
Syst. 27, 1799–1807 (2014) learning (ML) (Yu et al. 2012; De et al. 2018) in
Tompson, J., Goroshin, R., Jain, A., LeCun, Y., Bregler, C.: which an artificial neural network (ANN) is uti-
Efficient object localization using convolutional net- lized to map a set of input parameters into a set of
works. In Proceedings of the IEEE Conference on outputs. DL works with labeled datasets as train-
Computer Vision and Pattern Recognition,
pp. 648–656 (2015) ing samples which usually involve complex, high-
Xiang, Y., Fox, D.: Da-rnn: Semantic mapping with data dimensional raw input data (Lou et al. 2018). It
associated recurrent neural networks. arXiv preprint needs less manual feature engineering than prior
arXiv:1703.03098 (2017) ML methods (Liang et al. 2018). DL has enabled
Yang, Y., Wang, J., Liu, T., Lv, X., Bao, J.: Improved long
short-term memory network with multi-attention for significant progress in many application fields
human action flow evaluation in workshop. Appl. Sci. such as computer vision, games, and virtual and
10(21), 7856 (2020) augmented reality (Khiew et al. 2019).
Yang, X., Deng, C., Dang, Z., Wei, K., Yan, J.: In artificial intelligence (AI) game designs,
SelfSAGCN: Self-Supervised Semantic Alignment for
Graph Convolution Network. In Proceedings of the behavior of the in-application non-player charac-
IEEE/CVF Conference on Computer Vision and Pat- ter (NPC) is a key issue. It will largely affect the
tern Recognition, pp. 16775–16784 (2021) overall quality of the game user experience.
Zhang, P., Lan, C., Xing, J., Zeng, W., Xue, J., Zheng, N.: Instead of designing specific NPC algorithms,
View adaptive recurrent neural networks for high per-
formance human action recognition from skeleton data. the NPC agent can be trained with reinforcement
In Proceedings of the IEEE International Conference learning (RL) in the entire behavior system by
on Computer Vision, pp. 2117–2126 (2017) configuring the desired game environment and a
set of hyperparameters. RL is conceptualized as a
Markov decision process (MDP). Refer to Fig. 1,
an Agent at an execution step t is defined by a
Deep Reinforcement Learning State St. The Agent arbitrarily takes an Action At,
which results in a change of the Environment. The
▶ Deep Reinforcement Learning in Virtual Environment provides the Agent with a feedback
Environments Reward/Penalty score Rt þ 1 for its next execution
Deep Reinforcement Learning in Virtual Environments 551

step t þ 1, and transits the Agent to next State applications, DRL ANN is implemented with a
St þ 1 according to the environment dynamics. In specific reinforcement learning policy according
a trial-and-error manner, the Agent attempts to to the problems to be solved. A machine learning
learn a policyfrom observations to actions, in agent is rewarded for actions that achieve the
order to maximize/minimize its expected sum target outcome and penalized if not. The DRL
of Reward/Penalty scores. applications in two case studies are demonstrated.
In practice, an RL policy is dependent on a
high-dimensional vector of inputs. Deep rein-
forcement learning (DRL) is to implement a DL Case Study 1 D
network with an RL policy. In this study, DRL is
to solve the problem of an NPC learning to make The conventional agricultural unmanned aerial
decisions in the AI game. In particular, DRL vehicle (AUAV) is based on the principles of
allows an NPC agent to make decisions from aerodynamics and atmospheric turbulence, and
high-dimensional and unstructured input data its flight control system involves a complex pro-
without manual engineering of the state space. cess and needs a long learning curve. It requires
The implemented DRL algorithms are able to the operators to be familiar with the aerodynamic
take in a very large set of inputs, for example, all characteristics of the AUAV, as well as the agri-
pixels in a video game, and decide what actions to cultural tasks such as spraying; the operators have
take to optimize an objective, for example, max- also to be vulnerable to external meteorological
imizing the game score. and geographical environments which are con-
straint by the climate conditions; and worse, fail-
ures in the operations often lead to crashes of the
Applications of Deep Reinforcement drones, which is costly.
Learning in Virtual Reality To largely reduce the dependence on the phys-
ical settings for the AUAV tasks, with references
DRL proves to be a promising tool for real-time to our other VR autonomous movement controls
feedback to virtual reality (VR) and intelligent (Zhao et al. 2021a, b) and path finding algorithms
virtual environments (IVE) (Wang et al. 2019a, (Zhang et al. 2021a), DRL is applied into a VR
b, 2020; Kumar et al. 2016; Liu et al. 2020). In the AUAV. Modeling based on our four-rotor AUAV,

Deep Reinforcement
Learning in Virtual
Environments,
Fig. 1 Reinforcement
learning for an agent to
adapt to the environment
552 Deep Reinforcement Learning in Virtual Environments

and targeting at complex external meteorological body part (Wu et al. 2004; Cai et al. 2016, 2017).
and geographical environments as well as irregu- By locking or limiting certain X, Y, and Z motion
lar plants, the VR AUAV flight is simulated and and angular motion, it prevents the joints from
controlled using its machine learning agent. moving erratically in the VR game.
Briefly, In the DRL processes for the VR game, as
shown in Fig. 3 (lower), the humanoid agent acts
(a) A momentum balance model (MBM) of four- based on the observations generated and is given a
rotor AUAV and its intelligent runtime control reward correspondingly (Zhang et al. 2021b;
system is designed. Ming et al. 2021). The various attributes of the
(b) Virtual flight controls of the four-rotor AUAV humanoid agent such as the number of actions to
by graphical user interface (GUI), including be taken are defined by the behavior component
take-off, landing, forward flight, left/right (Leong et al. 2020). Briefly,
turn, are implemented with its flight parame-
ters for optimization in real time. (a) Before training is undertaken, the humanoid
(c) DRL for the roam function is implemented to agent possesses a Learning Behavior where
acquire the flight attitude and position infor- the attributes are undefined. At the end of the
mation of the drone, as well as simulation of training session, an ANN file is generated. By
the control of agricultural spraying and sow- attaching the trained ANN file to the human-
ing with particle systems, a VR simulation oid agent, the initial Learning Behavior trans-
shown in Fig. 2. fers into Inference Behavior and the attributes
become defined.
(b) These attributes are passed on to the master
Case Study 2 controller which calculates the offset between
the rotation, angular momentum, center of
A VR golf training environment is designed, with mass of the agent’s body parts and trainer’s
two humanoid avatars, one representing the body parts. The observations made by the
trainer and the other for the machine learning master controller are used by the agent com-
agent. Configurable joints and rigid bodies are ponent to compute the corresponding reward.
added to the body parts of the humanoid avatar, (c) The difference in these values between the
as shown in Fig. 3 (upper). This allows for physics agent and trainer are tracked as observa-
to be added to the character as well as movements tions. During training, the values can be
of the character to be customized by configuring seen from the master controller script
the X, Y and Z motion and angular motion of each attached to the humanoid agent. Through

Deep Reinforcement
Learning in Virtual
Environments,
Fig. 2 DRL for VR AUAV
flight controls in IVE of
agricultural spraying
Deep Reinforcement Learning in Virtual Environments 553

Deep Reinforcement Learning in Virtual Environments, Fig. 3 DRL for VR golf swing
554 Deep Reinforcement Learning in Virtual Environments

RDL, the agent continuously makes deci- discriminant analysis. Comput. Biol. Med. 99,
sions that will minimize the difference 123–132 (2018)
Liu, X., Shen, Y., Liu, J., Yang, J., Xiong, P., Lin, F.:
between these values. Parallel spatial-temporal self-attention CNN based
(d) The reward function incentivizes the agent to motor imagery classification for BCI. Front. Neurosci.
move closely to the trainer’s action. It is 14, 587520 (2020)
obtained by adding the respective reward Lou, C., Pang, C., Jing, C., Wang, S., He, X., Liu, X.,
Huang, L., Lin, F., Liu, X., Wang, H.: Dynamic balance
earned from the observations made. measurement and quantitative assessment using wear-
able plantar-pressure insoles in a pose-sensed virtual
environment. Sensors. 18(12), 4193 (2018)
Conclusions Ming, R.T.R., Feng, C., Seah, H.S., Lin, F.: Movability
assessment on physiotherapy for shoulder periarthritis
via fine-grained 3D ResNet deep learning. In: SPIE
In the two case studies on VR applications, DRL Proceedings of International Forum on Medical Imaging
has been proven to be effective for training intel- Asia (IFMIA’21), Taiwan (Online), 24–27 January 2021
ligent and autonomous in-game agents. Neverthe- Wang, Y.C., Zhang, Q., Lin, F., Goh, C.K., Seah, H.S.:
PolarViz: a discriminating visualization and visual ana-
less, while a virtual policy in RL is easy to realize lytics tool for high-dimensional data. Vis. Comput.
compared with its counterpart in real-world, the 35(11), 1567–1582 (2019a)
virtual environments with complex physics are Wang, Q., Shou, G., Liu, Y., Lin, F., Seah, H.S.: Converg-
yet to be researched. This is critical in scenarios ing mobile edge computing and wireless access for
virtual reality. In: SPIE Proceedings of International
when precision of the agent is seen as the priority Workshop on Advanced Image Technology
in the learning process (Zhao et al. 2021c). More (IWAIT’19), Singapore, 7–9 January 2019b
robust and efficient reinforcement learning algo- Wang, J., Ji, B., Lin, F., Lu, S., Lan, Y., Cheng, L.:
rithms should be investigated for real-time sys- A multiple pattern complex event detection scheme
based on decomposition and merge sharing for massive
tems of Virtual Reality and Intelligent Virtual event streams. Int. J. Distrib. Sens. Netw. 16(10) (2020).
Environments. https://doi.org/10.1177/1550147720961336
Wu, Z.K., Lin, F., Seah, H.S., Chan, K.Y.: Evaluation of
difference bounds for computing rational Bezier curves
and surfaces. Comput. Graph. 28(4), 551–558 (2004)
References Yu, J., Lin, F., Seah, H.S., Lin, Z.: Image classification by
multimodal subspace learning. Pattern Recogn. Lett.
Cai, J., Lin, F., Seah, H.S.: Graphical Simulation of 33(9), 1196–1204 (2012)
Deformable Models. Springer International Publishing Zhang, H., Lim, S.F., Lin, F., Li, L., Seah, H.S.: Ball flight
Switzerland (2016)., ISBN 978-3-319-51030-9, ISBN path determination in VR table tennis. In: SPIE Pro-
978-3-319-51031-6 (eBook) ceedings of International Workshop on Advanced
Cai, J., Lin, F., Lee, Y.T., Qian, K., Seah, H.S.: Modeling Image Technology (IWAIT’21), Japan (Online),
and dynamics simulation for deformable objects of 5–6 January 2021a
orthotropic materials. Vis. Comput. 33(10), Zhang, J., Liu, M., Xiong, P., Du, H., Zhang, H., Lin, F.,
1307–1318 (2017) Hou, Z., Liu, X.: A multi-dimensional association
De, J., Zhang, X., Lin, F., Cheng, L.: Transduction on information analysis approach to automated detection
directed graphs via absorbing random walks. IEEE and localization of myocardial infarction. Eng. Appl.
Trans. Pattern Anal. Mach. Intell. 40(7), Artif. Intell. 97, 104092 (2021b)
1770–1784 (2018) Zhao, K., Lin, F., Seah, H.S.: Steering autonomous animals
Khiew, J.B., Tan, P.H., Lin, F., Seah, H.S.: VR puzzle room in VR hunting. In: SPIE Proceedings of International
for cognitive rehabilitation. In: SPIE Proceedings of Workshop on Advanced Image Technology
International Forum on Medical Imaging Asia (IWAIT’21), Japan (Online), 5–6 January 2021a
(IFMIA’19), Singapore, 7–9 January 2019 Zhao, K., Lin, F., Seah, H.S.: Collective intelligence of
Kumar, A., Lin, F., Rajapakse, J.C.: Mixed spectrum anal- autonomous animals in VR hunting. In: IEEE VR 2021
ysis on fMRI time-series. IEEE Trans. Med. Imag. Workshop on 3D Content Creation for Simulated
35(6), 1555–1564 (2016) Training in Extended Reality (IEEE VR21
Leong, M.C., Prasad, D.K., Lee, Y.T., Lin, F.: Semi-CNN TrainingXR), Lisbon, Portugal (Online), 27 March –
architecture for effective spatio-temporal learning in 3 April 2021b
action recognition. Appl. Sci. 10(2), 557 (2020) Zhao, K., Lin, F., Seah, H.S.: Reinforcement learning for
Liang, Y., Sun, L., Ser, W., Lin, F., Thng, S.T.G., Chen, Q., quadruped locomotion. In: Computer Graphics Interna-
Lin, Z.: Classification of non-tumorous skin pigmenta- tional 2021 (CGI’21), Geneva (Online), 6–10 September
tion disorders using voting based probabilistic linear 2021c
Delaunay Triangulation 555

Introduction
Defamiliarization
A triangulation is a connection of vertices by
▶ Design of Alienation in Video Games edges, which form a set of non-overlapping tri-
angles (Sinclair 2016). The most known triangu-
lations in the literature are: Greedy Triangulation
(Dickerson et al. 1994), Triangulation of Garey
Dehumanization (Garey et al. 1978), Radial Sweep (Hjelle and
Dæhlen 2006), and Delaunay Triangulation D
▶ Uncanny Valley in Virtual Reality (de Berg et al. 2008).
The Delaunay Triangulation for the set of ver-
tices V(n) satisfies four properties: (i) local-empty
circle: for each circumcircle Cijk created for a
Delaunay Tesselations Delaunay triangle tijk, there is not any vertex vl
of other triangle falling inside the circumcircle
▶ Delaunay Triangulation Cijk (de Berg et al. 2008), (ii) Maximize the min-
imum angles, (iii) a Delaunay Triangulation is
unique (Verbree 2010), and (iv) the boundary of
a Delaunay Triangulation for a set of vertices V(n)
Delaunay Tessellations is the convex hull of V(n).
A Delaunay Triangulation has a dual graph
▶ Constrained Edges and Delaunay Triangulation called a Voronoi diagram, which is formed by
using circumcenters of Delaunay triangles
(or tetrahedra), thus some authors have taken
advantage of this to explore both geometrical
Delaunay Triangulation structures (Watson 1981; Chew 1990; Agarwal
et al. 2015; Allen et al. 2016). The most common
Simena Dinas1 and Hector J. Martínez2 methods to construct a Delaunay Triangulation
1
Facultad de Ingeniería, Universidad Santiago de are Lawson method (Lawson 1977), Bowyer
Cali, Cali, Colombia method (Bowyer 1981) and Watson method
2
Universidad del Valle, Cali, Colombia (Watson 1981).
For the set of vertices V(n) shown in Fig. 1a, it
is created a Delaunay Triangulation (see Fig. 1b).
Synonyms The convex hull for the set of vertices V(n) is
depicted in Fig. 1c. Whereas Fig. 1d depicted the
Delaunay Tesselations; Delone Tesselations; circumcircles, and Fig. 1e shows the centers for
Delone triangulation each circumcircle. Finally, the Voronoi diagram is
represented in Fig. 1f.

Definition Properties
A Delaunay Triangulation has to satisfy four prop-
Delaunay Triangulation is a geometrical structure erties: Local empty circle, maximize the minimum
widely used in Computational Geometry. It was angles, uniqueness, and boundary (convex hull).
proposed by Boris Nikolaevich Delone 1. Local empty circle: A circumcircle is a
(1890–1980) in 1934. Basically, given a set of n unique circle passing through all vertices of a
vertices V(n) (n  3) a Delaunay Triangulation is a triangle in a Delaunay Triangulation DT(V(n)).
net of non-overlapping triangles whose set of This circle contains no other vertex from the set
vertices is V(n). of vertices V(n), (see Fig. 2) (van Kreveld 2014).
556 Delaunay Triangulation

There are two levels to prove this property: (i) an Fig. 2b shown that the triangles does not satisfy
edge eij is Delaunay if and only if there is not any the property.
vertex of the triangulation that falls inside the Maximize the minimum angles: For each
minimum circle that passes through both vertices quadrilateral in Delaunay Triangulation DT(V(n)),
of the edge. And, (ii) a triangle tijk is Delaunay if two possible triangulations can be produced; the
and only if there is not any vertex of the triangu- triangulation that maximizes the minimum of the
lation that falls inside the circumcircle of the tri- six internal angles is the correct triangulation (see
angle. A triangulation is Delaunay if and only if Fig. 3). It means, for each quadrilateral in a Delaunay
each edge or each triangle is Delaunay. Local Triangulation, there are two different triangulations
Empty Property is shown in Fig. 2a, whereas with two triangles each one; let ta and tb be

Delaunay Triangulation, Fig. 1 Delaunay triangulation and voronoi (a) Delaunay Points (b) Delaunay Triangulation
(c) Convex Hull (d) Delaunay Circuncircles (e) Delaunay Circumcenters (f) Voronoi Diagram

Delaunay Triangulation, Fig. 2 The local empty circle property (a) Delaunay Triangles (b) No Delaunay Triangles
Delaunay Triangulation 557

triangulations. Let αta and αtb be the smallest angle of Quality in a Delaunay Triangulation
the triangulations ta and tb, respectively. The The following list shows a set of features that can
Delaunay Triangulation includes the biggest angle ensure high quality in a Delaunay Triangulation
between αta and αtb (see Fig. 3b). For a Delaunay (van Kreveld 2014):
triangulation, Fig. 3a shows an invalid angles con-
struction. A correct angle construction is depicted in • It does not have long edges. Long edges pro-
Fig. 3b, which maximizes the minimum of the six duce degeneracies; they are good in the bound-
internal angles (Mandal and Agarwal 2011). ary of the triangulation.
Uniqueness: Delaunay Triangulation DT(V • It does not have triangles with both, short and D
(n)) is unique for a set of vertices V(n), except long edges. The combination of long and short
for a set of four cocircular vertices (Khanimov and edges produces needles and caps.
Sharir 2015) (see Fig. 4a). Similarly, Delaunay • It does not have triangles with very small
Tetrahedralization DT(V(n)) is unique for a set of angles; these triangles can be needles or needle
vertices V(n), except for a set of five co-spherical and caps.
vertices; for instance: a box (see Fig. 4b). • It does not have triangles with obtuse angles.
Those examples are based on regular shapes; Caps and needle and caps have these type of
however, irregular polygons or polyhedral can angles.
form a no-unique triangulation, for instance: uni- • It does not have vertices with high degree.
form triangular prisms and uniform rectangular A vertex connected to a high number of verti-
prisms, amongst others. Additionally, this excep- ces is related with small angles, and, conse-
tion can be extended to upper number of quently, they are degeneracies.
co-circular and co-spherical vertices. • The combination of both long-short edges
Boundary: The borders of a Delaunay Trian- and small-obtuse angles produces degenera-
gulation DT(V(n)) is the convex hull of V(n) cies; it means needles and/or caps in two
(see Fig. 1c). dimensions.

Delaunay Triangulation, Fig. 3 Bad and good angles in a Delaunay Triangulation (a) Bad angles in a Delaunay
Triangulation (b) Good angles in a Delaunay Triangulation

Delaunay Triangulation, Fig. 4 No-unique Delaunay triangulation and tetrahedralization (a) No-unique Delaunay
triangulation (b) No-unique Delaunay tetrahedralizations
558 Delaunay Triangulation

Number of Triangles in a Delaunay Evaluation Tests

Triangulation Even though Delaunay Triangulation has to fulfill
According to van Kreveld (2014), a Delaunay the four properties described previously, the cir-
Triangulation from n vertices – or a DT(V(n)) – cumcircle criterion is the base of the evaluation of
have at least n  2 and at most 2n  5 Delaunay the structure; the circumcircle criterion forces the
triangles; moreover, it has at least 2n  3 and at other properties.
most 3n  6 Delaunay edges. According to de The inCircle test verifies if a pair of adjacent
Berg et al. (2008) a Delaunay Triangulation from triangles is Delaunay triangles; otherwise, the
n vertices has 2n  2  k triangles and 3n  3  k common edge between adjacent triangles has to
edges, where k is the number of vertices in the be flipped to become a Delaunay triangles (Biniaz
Convex Hull. and Dastghaibyfard 2012). Similarly, the
inSphere test proves the condition for a pair of
Advantages of Delaunay Triangulation tetrahedra; however, the change in the structure is
The following list shows the main advantages of a not as straightforward as flipping an edge.
Delaunay Triangulation (Mys 2004): Let I be a set of indices and let tijk i, j, k ∈ I; i 6¼ j;
i 6¼ k be a triangle in counter-clockwise (CCW)
• It has elegant theoretical foundations and it has orientation in the plane. Let Cijk be the circumcir-
a wide range of application. cle of the triangle tijk. Let vl l ∈ I  {i, j, k} be a
• The convex hull of the set of vertices is a part of vertex. Then, the inCircle test is defined as
a Delaunay Triangulation. follows:
• The line connecting each point to its nearest
neighbor must be in a Delaunay Triangulation. ¼ 0; #l ∈Cijk : #l falls on Cijk :
• It produces a continuous surface, and it has an inCircle tijk , #l > 0; #l is outside Cijk :
irregular connectivity. < 0; #l is inside Cijk :
• The construction does not depend on the
starting point nor the dispersion of the points; In general, the inCircle test can be simplified in
it is predictable, repeatable, and equally easy an operation as follows:
(or hard) to construct for all applications.
• It is a method to cover a surface of a polygon xi yi x2i þ y2i 1
using triangles.
xj yj x2j þ y2j 1
• It is a common method to represent Triangu- inCircle tijk , #l ¼
lated Irregular Networks (TIN). xk yk x2k þ y2k 1
• It generates a mesh automatically. xl yl x2l þ y2l 1

Disadvantages of Delaunay Triangulation The orientation test verifies the order of the
Some disadvantages of working with Delaunay vertices that define the triangle, it means, to cal-
Triangulations include (Mys 2004): culate a Delaunay Triangulation, a circumcircle
test is required; however, the triangle has to have a
• It does not maximize the minimum angle in positive orientation (counter-clockwise orienta-
three dimensions, even though, it does in two tion). The orientation test is given by the follow-
dimensions. ing expression:
• When there is a mesh, the triangulation not
always preserves it. orientation 2D tijk
• It produces degeneracies in two and three ¼ 0; the vertices are collinear:
dimensions. > 0; the triangle tijk has a positive ðCCWÞ orientation:
• The type of tetrahedra with degeneracies in
< 0; the triangle tijk has a negative ðCWÞ orientation:
three dimensions is high (nine types) compared
with degeneracies in two dimensions (three
where
types).
Delone Tesselations 559

xi yi 1 de Berg, M., Cheong, O., van Kreveld, M., Overmars, M.:

Chapter 9, Delaunay triangulations – Height interpola-
orientation 2D tijk ¼ xj yj 1 tion. In: Computational Geometry: Algorithms and
xk yk 1 Applications, 3rd edn, pp. 191–218. Springer,
New York (2008). https://doi.org/10.1007/978-3-540-
ðx i  xk Þ ðyi  yk Þ 77974-2
¼ Dickerson, M.T., Drysdale, R.L.S., McElfresh, S.A.,
xj  xk yj  yk Welzl, E.: Fast greedy triangulation algorithms. In:
Proceedings of the Tenth Annual Symposium on Com-
putational Geometry, SCG’94, pp. 211–220. ACM,
New York (1994). https://doi.org/10.1145/177424. D
Conclusion and Discussion 177649
Garey, M.R., Johnson, D.S., Preparata, F.P., Tarjan, R.E.:
A Delaunay Triangulation is a triangle net in Triangulating a simple polygon. Inf. Process. Lett. 7,
175–180 (1978)
which every triangle satisfies the Circumcircle Hjelle, Ø., Dæhlen, M.: Triangulations and Applications
condition: the circumcircle of each triangle (Mathematics and Visualization). Springer, New York/
includes only the vertices of the triangle. In other Secaucus (2006). https://doi.org/10.1007/3-540-
words, the circumcircle does not contain any ver- 33261-8
Khanimov, M., Sharir, M.: Delaunay triangulations of
tex of other triangles. If both the inCircle and the degenerate point sets. CoRR, abs/1510.04608 (2015)
Orientation tests are the algorithmically mannered Lawson, C.L.: Software for c1 surface interpolation. In:
to probe this condition for all the triangles in a Rice, J. (ed.) Mathematical Software III, pp. 161–194.
triangulation, then it can be labelled as Delaunay Academic, New York (1977)
Mandal, C., Agarwal, S.: Online delaunay triangulation
Triangulation. using the quad-edge data structure. In: Wyld, D.,
Wozniak, M., Chaki, N., Meghanathan, N., Nagamalai,
D. (eds.) Advances in Computing and Information
Technology, volume 198 of Communications in Com-
puter and Information Science, pp. 132–141. Springer,
Cross-References Berlin/Heidelberg (2011). https://doi.org/10.1007/978-
3-642-22555-0_15
▶ Constrained Edges and Delaunay Triangulation Mys, R.: Delaunay triangulation (dt). Lehrstuhl für
▶ Modeling and Mesh Processing for Games Informatik 10 (Systemsimulation). Friedrich-
Alexander-Universität Erlangen-Nürnberg, Erlangen
▶ Teaching Computer Graphics by Application
(2004)
▶ UV Map Generation on Triangular Mesh Sinclair, D.: S-hull: a fast radial sweep-hull routine for
delaunay triangulation. CoRR, abs/1604.01428 (2016)
van Kreveld, M.: Delaunay Triangulation and Tetra-
hedrilization. Department of Information and Comput-
References ing Sciences, Faculty of Science, Utrecht University,
Utrecht (2014)
Agarwal, P.K., Kaplan, H., Rubin, N., Sharir, M.: Kinetic Verbree, E.: Delaunay tetrahedralizations: honor
voronoi diagrams and delaunay triangulations under degenerated cases. In: The International Archives of
polygonal distance functions. Discrete Comput. the Photogrammetry, Remote Sensing and Spatial
Geom. 54(4), 871–904 (2015). Available at https:// Information Sciences – ISPRS 2010, pp. 69–72. Berlin
doi.org/10.1007/s00454-015-9729-3 (2010)
Allen, S.R., Barba, L., Iacono, J., Langerman, S.: Incre- Watson, D.F.: Computing the n-dimensional delaunay tes-
mental voronoi diagrams. CoRR, abs/1603.08485 sellation with application to voronoi polytopes.
(2016) Comput. J. 24(2), 167–172 (1981). https://doi.org/10.
Biniaz, A., Dastghaibyfard, G.: A faster circle-sweep 1093/comjnl/24.2.167
delaunay triangulation algorithm. Adv. Eng. Softw.
43(1), 1–13 (2012). https://doi.org/10.1016/j.
advengsoft.2011.09.003
Bowyer, A.: Computing dirichlet tessellations. Comput.
J. 24(2), 162–166 (1981). https://doi.org/10.1093/
comjnl/24.2.162
Chew, L.P.: Building Voronoi Diagrams for Convex Poly-
Delone Tesselations
gons in Linear Expected Time. Technical Report. Dart-
mouth College, Hanover (1990) ▶ Delaunay Triangulation
560 Delone Tessellations

Introduction
Delone Tessellations
The technology era has been growing very
▶ Constrained Edges and Delaunay Triangulation quickly, and it leads to the new industrial revo-
lution which is also known as Industry 4.0. The
inventions of new technologies like virtual real-
ity, 3D printer, and the Internet have greatly
Delone Triangulation influenced different sectors of the world econ-
omy. The digital technology sector in the UK has
▶ Constrained Edges and Delaunay Triangulation grown tremendously despite the economic crisis
▶ Delaunay Triangulation in 2008. Based on the Tech Nation Report in
2018, the UK digital tech sector is worth nearly
£184bn to the economy, a rise from £170bn in
2016 (Cahill et al. 2018). However, Job Market
Report 2017 has shown that the growth of the
Depth of Field digital economy and the emergence of new tech-
nologies has led to skills shortages and increased
▶ 3D-Rendered Images and Their Application in demand for graduates with the right qualifica-
the Interior Design tions (Dice 2017).
These advancements led educational institu-
tions to be interested in possible changes that
could involve the changes in curriculum which
Design Framework for can help prepare the students to face the industry
Learning to Support Industry after they graduate. Industry 4.0 implies that the
4.0 world is globalized, automatized, virtualized,
networked, and flexible. This section briefly
Sin Ying Tan1, Dhiya Al-Jumeily1, Jamila maps the education with industrial revolution. It
Mustafina2, Abir Hussain1 and Yuanyuan Shen1 is believed that education has to follow the pace of
1
Liverpool John Moores University, Liverpool, industrial revolution. The following section maps
UK the education with industrial revolutions.
2
Kazan Federal University, Kazan, Russia

Mapping Education with Industrial

Synonyms Revolutions

Education model; Education pedagogy; Fourth The evolution of the industry which progresses
Industrial Revolution; Industry 4.0; Learning ana- from First Industrial Revolution (Industry 1.0) to
lytics; Learning framework Fourth Industrial Revolution (Industry 4.0) can be
used as a sign to show how important it is that
education should also be developed and evolved
Definition from Education 1.0 towards Education 4.0. The
industry has undergone a process of digital trans-
Learning analytics is defined as the measurement, formation which exposes the education to chal-
gathering, analysis, and reporting of data about lenges and opportunities of meeting the needs of
learners and their environments, for the aim of the fast-growing industry. The changes in the
understanding and improving learning process industry development (industrial revolution) will
and the contexts in which learning occurs also cause changes to the development of educa-
(Siemens and Baker 2012). tion (education revolution) which means certain
Design Framework for Learning to Support Industry 4.0 561

skills are required which are not exactly the same electronic devices replaced the electric-based pro-
as the skills that were required before. Therefore, duction machines (Khan 1987). The Third Indus-
the current and future education structure should trial Revolution (Industry 3.0) initiated the
not only focus on training knowledge-based skill telecommunication industry, and this implies that
labor but also emphasize on cultivating innovative as communication became easier, people could
talent to meet the current demand of the industry. access and create content anywhere, any time.
Before and During Industry 1.0: Before the However, there are some countries that are still
industrial revolution, education started by focus- applying the approach used in Education 1.0
ing only on teaching the elite classes and educat- which only involves one-way learning process D
ing boys. Education was taught informally before (where students are passive and they are instilled
it transformed by focusing on scientific research. with the most essential or basic academic knowl-
Most education started with the dominance of edge and skills and character development
religions. The industrial revolution caused wide- (Petrina 1998; Gerstein 2014) and Education 2.0
spread change in all aspects of society. Therefore, (an approach used where students are active
many motivated individuals could easily take learners and gain knowledge by formulating and
advantage of the many economic opportunities solving their own problems (McWilliam 2009;
of the situation. Research study also added in Gerstein 2014), while other countries have started
that modern economic growth depends on the developing new education models that reflect the
growth of useful knowledge (Tang and Werner increased use of technology and enable increas-
2017). Therefore, this emphasizes the importance ingly flexible, experimental, and fairer learning
of gaining knowledge through education. Industry environments which lead to the introduction of
1.0 marked the beginning of industrialization the new era, Education 3.0. Students started shar-
which led to the demand for mass education, ing their knowledge with peers who have different
education for ordinary people especially from skills and levels of knowledge to co-create new
lower classes. This was the time when there was knowledge. This shows that technology was used
a demand in the workforce fit for the industrial to assist students’ learning process and helped
sector. More schools were built and the new con- them in shaping the content, location, and ways
cept “free education” was introduced during that in which they learn.
period (Robinson 2011). Industry 4.0: Due to the new industry era, the
Industry 2.0: This industry began with electri- technology grows rapidly, and there are so many
fication cycle when electricity became the primary new inventions. However, Lortie (2002) argued
source of power since twentieth century (Hughes that at the beginning of twenty-first century, the
1993). During the Second Industrial Revolution education does not change at a rapid pace as
(Industry 2.0), the graduates were viewed as ill- the structures of education are still the same as in
prepared line assembly workers as the inventions the twentieth century. The existence of sensors
of machines helped a lot in mass production. Skill and IOT can actually indicate an early sign of the
undoubtedly played an important role in techno- use of wearables-assisted teaching, learning, and
logical innovation and adoption (Greenwood training devices. Recently, there is a limited num-
1997). People did not know how to work efficiently ber of research studies that use smartphones and
until Frederick Taylor and Henry Ford proposed sensors to study the factors that affect the aca-
workplace methods and just-in-time and lean demic performance of students. It was found out
manufacturing principles to optimize the work- that there are many other factors that affect their
force and improve their quality and output. performance, and this leads to complexity in
Industry 3.0: It is also known as the informa- research. As Education 4.0 indicates the world is
tion revolution or the digital revolution. This is the complex, standardization approach that was used
era of production automation when there is an to be applied in Education 1.0 had to be eradi-
increasing use of electronics in industrial process cated. It is also added that standardized learning
and commerce and computer-programmed methods cannot deliver what the current and
562 Design Framework for Learning to Support Industry 4.0

future education needs when it comes to cope with datasets. It focuses on the educational challenge
complexity (Wallner 2012). This is true because in optimizing opportunities for learning. Siemens
everyone is different and unique, and everyone has first defined LA in his blog post as the use of
can stand out if one can manage to discover intelligent data, learner-produced data, and analy-
his/her own method of effective learning. This sis models to discover information and social
can be done through the assistance of current connections and to predict and advise on learning
technologies invented in Industry 4.0. (Siemens and Baker 2012). The definition was
The comparison between education evolutions then refined in international conferences, and the
and industry revolution are summarized in Society for Learning Analytics Research
Table 1. The table shows that the technology (SOLAR) then defined LA as the measurement,
advances rapidly, revolving from Industry 1.0 to gathering, analysis, and reporting of data about
the current era Industry 4.0, but the old educa- learners and their environments, for the aim of
tional models used in previous eras are still being understanding and improving learning process
applied in the fast-moving technology era. This and the contexts in which learning occurs.
implies that there is a gap between each education LA is a new research field that is often associ-
era and technology era. Education is moving ated with technology-enhanced learning. The
slower than the industry. It has to move faster in improvement of teaching and learning using LA
order to catch up with the fast-growing demands, has led to the use of the term “Big Data” in the
and in fact, it should be faster than the industry so education field. Moreover, in this context, big data
that the supply of skills and talents can match with in education which is also known as educational
the demand of the industry. data is what drives new methods to be used in LA
(Siemens and Baker 2012). They stated that the
specific features of data contributed to the differ-
Different Types of Analytics in Education ent methods are playing a prominent role in edu-
cational data mining (EDM) and LA. LA is an
Learning analytics (LA) is a powerful tool which emerging research field that studies strategies on
gives practical insights on the variables in the how to enhance student and faculty performance,

Design Framework for Learning to Support Industry 4.0, Table 1 Summary of comparison between industrial
revolution and education evolution
Types of Before Eighteenth Nineteenth Twentieth Twenty-first century
revolution eighteenth century century century (Connected World)
(Age) century (Industrial Age) (Knowledge (Digital Age)
(Agricultural Age)
Age)
Industry Before 1.0 2.0 3.0 4.0 (Smart automation)
Industry 1.0 (Mechanization) (Electrification (Automation)
cycle)
The views Industrial Assembly line As ill-prepared As Lack of skills and talents as
of industry Age has not workers assembly line co-workers required (Evangelinos and
on started yet workers Holley 2016; Manpower
graduates Group 2018)
Education 1.0 1.0 1.0 1.0 1.0
2.0 (Ideally) 2.0 2.0 2.0
3.0 (Ideally) 3.0 3.0
4.0 (Ideally) 4.0 (Still emerging)
Implication – There is a gap between each education era and technology era. Education is moving
slower than the industry. It has to move faster in order to catch up with the fast-growing
demands, and in fact, it should be faster than the industry so that the supply of skills and
talents can match with the demand of the industry
Design Framework for Learning to Support Industry 4.0 563

how to identify the needs of struggling learners, academic modules and student strengths and
how to improve accuracy of prediction, etc. weakness can be identified and appropriately
(Larusson and White 2014). In addition, LA rectified.
helps to explore in what areas of the curriculum Educational data mining (EDM) is a field
that can be improved. This complies with the which is often used together with LA as they
definition of LA which is “the interpretation of a present similar notions. However, both focus on
wide range of data produced by and gathered on different types of challenges. EDM deals more
behalf of students in order to assess academic with the technical challenge while LA focuses
progress, predict future performance, and spot more on educational challenge. EDM stresses on D
potential issues” (Liñán and Pérez 2015). It can extracting value from an enormous pool of edu-
be clearly seen that there are different definitions cational data. The phrase “data mining” in EDM
provided for the term LA. However, these defini- has already suggested that it is very technical. It is
tions focus on transforming educational data into concerned with developing methods to explore
actionable data to help improve the learning the unique types of educational data and identify
process. the patterns in order to have a better understanding
The term academic analytics (AA) is always of the students and the environments in where
interchangeably used with LA. However, there is they learn (Romero et al. 2010).
a distinction between these two terms. In order to Quite similar to LA, the objective of EDM is to
identify the difference between LA and AA, it is analyze the data to understand how students learn
vital to know the origin of AA. This term was first and to make predictions based on the analysis.
used to describe the business intelligence applica- Unlike EDM, LA further includes other methods,
tion tools and practices in higher education (Chatti such as statistical and visualization tools or social
et al. 2012). The authors further refer business network analysis (SNA) techniques and puts them
intelligence to an analytics tool that involves the into practice for studying their actual effective-
processes like collecting, storing, analyzing, and ness on the improvement of teaching and learning.
providing access to data to help enterprise users EDM focuses more on classification, association,
make informed business decisions. Therefore, to and other data mining techniques in
derive the definition from business intelligence analyzing data.
and to place it in educational context, it can be It is hard to separate LA and adaptive learning
said that AA is more emphasized on political or analytics (ALA) because they both foster
economic aspect of education as it helps to technology-supported, learner-centered educa-
improve learning opportunities and educational tion. LA is often associated with adaptation.
results at institutional, regional, and international Adaptation means making adjustments in a par-
levels (Ferguson 2012; Daniel 2015). The author ticular environment to accommodate the individ-
further added that AA benefits funders, adminis- ual differences. Adaptivity and adaptability are
trators, marketing, government, and education the common terms that are related to adaptation.
authorities (Ferguson 2012). Like LA, AA also Adaptivity refers to a process where the data about
analyzes enormous data sets with statistical tech- learner is used in a system controlled way,
niques and predictive modelling, but the purpose whereas adaptability means the system supports
is to help in decision-making unlike LA which is end-user modifiability providing student control
used to help in improving learning process (Mavroudi et al. 2018).
(Daniel 2015). A common idea behind those adaptive educa-
To narrow the definition down, AA focuses on tional systems is that, based on the information
student retention and graduation rates that actually about the learner and the current context, an
affect the institutional level (Campbell et al. appropriate adaptation method should be chosen
2007). Issues of detecting students that are at to adapt the presentation of the course material to
risk are often related to AA (Chatti et al. 2012). the individual learner. ALA focuses on learner
In AA research, operational activities relevant to modeling as the core for achieving adaptive and
564 Design Framework for Learning to Support Industry 4.0

personalized learning environments, which will positive attitude in pursuing education. Moreover,
be able to consider the heterogeneous needs of Lepper et al. (2005) and Pérez-López and Contero
learners and provide them with tailored learning (2013) claim that academic achievement also
experience suited for their unique needs. influences the intrinsic motivation. Attitude in
learning is considered very essential as it influ-
ences the academic achievement (Cai et al. 2017).
Motivation to Learn

Conventional educational pedagogy which Big Data Analytics, Academic Analytics,

adheres to the “one-size-fits-all” approach charac- Learning Analytics Process
teristic of traditional education systems can no
longer be used in this fast-changing era. The moti- Big data analytics (BDA) in the education context
vation and capacity to learn independently is cru- is also known as LA. LA is more specific and only
cial to personalization which is the pedagogy for used in educational context, whereas BDA can be
Education 4.0, because it reduces dependence on applied in different sectors. Thus, the stages
the teacher and traditional class-based styles of involved in BDA are a bit different from the stages
instruction. In order to produce individuals that involved in LA and AA.
thirst for learning, self-reflection and motivation Data Collection and Pre-processing: The first
are required to be reinforced (Leadbeater 2008). stage in BDA, LA, and AA processes involves
In addition, an education model will tend to collecting a vast amount of data. In LA and AA,
undermine the possibility of independent learning data are collected from the various educational
and developing skills if there is a lack of motiva- environments and systems, whereas data in Big
tion, because it can lead to a decrease in the levels Data can be collected from different sources and
of learning (Saveedra and Opfer 2012). As it is they are not necessarily sourced from the educa-
important to foster motivation for independent tional context (Daniel 2015). These data often
learning, research emphasizes the importance of come in various attributes and formats.
the teacher’s role in motivating learners and find- Researchers need to first determine the format of
ing ways for them to build intrinsic motivation data, the amount of data to be used, and choose
(Meyer et al. 2010). There are a few theories techniques to retrieve that data (Campbell
which contribute to the learning motivation of et al. 2007).
learners. Social cognitive theory reflects the As the collected data are available in vast
importance of the relationship between behavior, amount, data have to be managed and pre-
environment factor, and personal factor as these processed. These data are analyzed to discover
three factors are interconnected, and it is also interesting and useful patterns that can be used
known as cause and effect relationship (Bandura to retrieve useful information. In LA process, data
1989). This theory also depicts what caused the is transformed into suitable format by using dif-
people to behave certain ways and provides basic ferent methods such as data cleaning, data inte-
interventions (Bandura 1997). It is also found gration, data transformation, data reduction, data
out that environment factor can influence people modelling, user and session identification, and
and can be divided into social environment path completion.
(family and friends) and physical environment Statistics tools are recommended to be used
(comforts) (Bandura 1997). These factors are during this stage to handle large quantities of
also known to affect the students’ learning data (Chatti et al. 2012). This stage is vital when
process. it comes to computing the description for the data
Research also found that the challenge, curios- as it gives the end users a reflection on what has
ity, control, and fantasy are the key factors to happened.
trigger up intrinsic motivation (Ryan and Deci Analytics and Actions: The next stage of the
2000). These factors can boost will power and process in LA, AA, and BDA is known as
Design Framework for Learning to Support Industry 4.0 565

analytics. Each technique used to analyze data predictors or criteria, modifying the variables of
depends on the type of selected data and the goal analysis, or selecting a new analytics method.
of the research. Different LA techniques like data
mining techniques, statistics and mathematics,
text mining, semantics and linguistic analysis, Existing Frameworks
visualization social network analysis, qualitative
analysis, and gamification can be applied to ana- LA Framework: Previous literature studies have
lyze and explore the data in order to discover proposed different frameworks for LA. A refer-
hidden patterns or insights that can help to provide ence model for LA is proposed based on four D
a more effective learning experience (Khalil and dimensions, namely, data and environments
Ebner 2016). (what?), stakeholders (who?), objectives
The following stage in all three LA, AA, and (why?), and methods (how?) (Chatti et al.
BDA processes includes actions on the informa- 2012). Based on the reference model, various
tion that have been analyzed such as monitoring, challenges and research opportunities are
analysis, prediction, intervention, assessment, suggested by reviewing recent publications on
adaptation, personalization, recommendation, LA and its related fields based on the proposed
and reflection. Taking actions is the primary aim reference model. Findings showed that (1) cen-
of the whole analytics process (Daniel 2015) tralized web-based learning systems represent
which also means answering questions that leads the most widely used data source for LA,
to the stage of data collection (Campbell et al. (2) most of the current LA applications are ori-
2007). A model is built at the end of this stage ented toward intelligent tutors or researchers/
and improvements can be seen as the result. These system designers, (3) the most commonly
actions can be executed either manually or auto- applied objectives are adaptation and monitor-
matically which may include linking, connecting ing/analysis, and (4) the most frequently used
correlating different data sets to be able to obtain LA techniques are classification and prediction.
insight that is supposed to be conveyed by these The authors that proposed the generic model
data. In LA process, actions can be prescriptive as actually extended the reference model by adding
it can help students to be successful (Khalil and two extra dimensions which are the external lim-
Ebner 2016). On the other hand, the complexity of itations and internal limitations (Greller and
BDA occurs when the management and the anal- Drachsler 2012). The authors also argued that
ysis of the largely diverse data becomes a complex the critical problem zones, and some potential
process (Daniel 2015). dangers to the beneficial exploitation of educa-
Refining/Post-processing/Visualization: The tional data, should be explored in order to provide
last stage of LA and AA processes is a bit different a more comprehensive and useful guide for setting
from the last step in BDA process. The last step in up LA services to support the learners, teachers,
the BDA process actually involves presenting the and institutions.
analyzed data into an interpretable and integrated While the other proposed frameworks are only
information to help inform the decision of stake- focusing on LA generally, a different framework
holders (Daniel 2015), whereas the last stage of that combines adaptation and LA was proposed
LA and AA process includes continuous improve- and considered the same dimensions as the previ-
ment of analytics exercise (Chatti et al. 2012; ously mentioned frameworks (what, who, why,
Campbell et al. 2007). This stage takes place and how) but added two more dimensions which
when the actions taken in the previous stage and are when and where (McWilliam 2009). This
the processed results are evaluated. As stated in framework suggested the following points:
Chatti et al. (2012), this stage may include com- (1) key data related to the context and tools used,
piling new data from additional data sources, (2) objectives, (3) stakeholders, (4) application
improving the data set, determining new criteria area and context, (5) time-related aspects, and
required for the new iteration, identifying new (6) use of LA and adaptation.
566 Design Framework for Learning to Support Industry 4.0

DigComp Framework and EntreComp Vries 2017). LA should be designed based on

Framework by European Commission: theoretical models and needs of students should
DigComp framework is a framework that consists be well understood. Feedback of LA is essential
of five dimensions which are competence areas, for learning, and it is important to identify which
competences, proficiency level, examples of kind of feedback is suitable for which kind of
knowledge skills and attitudes, and example of student.
applicability to purpose (Ferrari 2013). The com- Feedback of LA leads to the question on the
petence areas which include information, commu- quality assurance of LA. The quality assurance of
nication, content creation, safety, and problem- LA services is questioned as they might only meet
solving are the main components of this frame- the expectations of certain stakeholders (e.g.,
work (Vuorikari et al. 2016). This framework is managers) while overlooking those who are the
being applied in the education sectors and proven most important (e.g., students) (Liñán and Pérez
to be helpful in investigating the digital attitudes, 2015).
skills, and development needs of healthcare stu- Although DigComp framework is being
dents (Evangelinos and Holley 2016). Besides widely used for strategic support for policy mak-
that, this framework is also used to examine fac- ing by European Union member, none has
tors predicting lower secondary school students’ displayed strategies that can be used with different
digital competence (Hatlevik et al. 2015). This stakeholders to develop digital competence espe-
framework also covers three different proficiency cially in the aspect of problem-solving (Balula
levels and specified indicators for the develop- 2016). The author also said that it is vital to
ment of each digital competence. highlight that DigComp framework is descriptive
Similar to DigComp framework, EntreComp rather than prescriptive, and therefore, this frame-
framework is also developed by the European work is always prone to be revised and updated,
Commission to establish a bridge between the and each update takes a long time as it involves
world of education and employment and to be many stakeholders to reach a consensus.
used as a reference by any initiative which aims EntreComp framework has not yet been adapted
to encourage entrepreneurial learning. Like to or tested in real settings, and it is a result of a
DigComp framework, EntreComp has compe- robust research methodology which involves
tence areas which are ideas and opportunities, experts’ consultation and input. Therefore, it will
resources, and putting strategies into action. also take time for the framework to be updated.
Each area includes 5 competences which sums A framework that suits education and industry
up to 15 competences along an eight-level pro- needs should be able to be updated quickly to
gression model (Bacigalupo et al. 2016). suit the needs as education evolution has been in
Issues Concerned with Existing Framework: a slow pace while technology is growing very
There are several concerns that should be taken rapidly (Table 2).
into consideration when it comes to designing
LA. Researchers will also need to investigate
problems faced by learners in different environ- Conclusion and Discussion
ments and what success looks like from the per-
spective of learners (Ferguson 2012). The There are studies on LA framework, but there is a
analytics process should be transparent so that limited number of studies (prior to our knowl-
the learners can respond with feedback that can edge) focusing on students’ motivation in learning
be used to refine the model and to be able to see in the design of LA framework. Attitude is impor-
how their data are used. The other researchers also tant because the quality of framework will not
added that LA can only be effective if the outcome even matter if the students do not have motivation
can generate insights into the pedagogical didactic to learn. Therefore, the design of the framework
consequences for both learning and teaching prac- should start from focusing on the learners’ moti-
tice (Ferguson 2012; Van den Bogaard and De vation to learn (attitude) before focusing on other
Design Framework for Learning to Support Industry 4.0 567

Design Framework for Learning to Support Industry 4.0, Table 2 Summary on the strengths and limitations of
different frameworks
Types of
framework Strengths Limitations
LA Provide a more comprehensive and useful guide Quality assurance (only meet the expectations of
framework for setting up LA services to support the learners, certain stakeholders)
teachers, and institutions which is believed to
cover all related aspects: (1) key data related to the
context and tools used, (2) objectives
(3) stakeholders, (4) application area and context, D
(5) time-related aspects, and (6) use of LA and
adaptation
EntreComp Used as a reference by any initiative which aims Has not yet been adapted to or tested in real
framework to encourage entrepreneurial learning settings and it is a result of a robust research
methodology which involves experts’
consultation and input
DigComp Helpful in investigating the digital attitudes, This framework is always prone to be revised and
framework skills, and development needs of students updated, and each update takes a long time as it
Used to examine factors predicting students’ involves many stakeholders to reach a consensus
digital competence

elements to help improve their learning process. Framework. Publication Office of the European
To date, the education analytics such as LA, Union, Luxembourg (2016)
Balula, A.: The use of DigComp in teaching and learning
EDM, and so on have focused predominantly on strategies: a roadmap towards inclusion. In: Proceed-
analyzing data systematically gathered in educa- ings of the 7th International Conference on Software
tional settings, which at the tertiary level includes Development and Technologies for Enhancing Acces-
factors of prior academic performance, demo- sibility and Fighting Info-Exclusion, Vila Real, Portu-
gal pp. 275–282. ACM (2016)
graphic data, such as age and gender, and data Bandura, H.: Human agency in social cognitive theory.
gathered by logs recording student behavior in Am. Psychol. 44(9), 1175 (1989)
online learning environments. Although the initial Bandura, A.: Self-Efficacy: The Exercise of Control. Free-
results in the literature studies are encouraging man, New York (1997)
Cahill, F., Windsor, G., Sorotos, H.E., Cousins, L., Boga,
across a variety of analysis techniques, there is S., Mohamed, S.B.: Connection and collaboration:
scope for more research investigating the contri- powering UK tech and driving the economy. Tech
bution of additional data such as psychometric Nation. https://technation.io/wp-content/uploads/
data that could be gathered by tertiary education 2018/05/Tech-Nation-Report-2018-WEB-180514.pdf
(2018). Accessed 25 July 2018
providers, as well as how this data should be Cai, S., Chiang, F.K., Sun, Y., Lin, C., Lee, J.J.: Applica-
modelled to enhance the current algorithmic tions of augmented reality-based natural interactive
models of student learning, and offer actionable learning in magnetic field instruction. Interact. Learn.
feedback on the learning process. Therefore, in Environ. 25(6), 1–14 (2017)
Campbell, J.P., DeBlois, P.B., Oblinger, D.G.: Academic
order to address this issue, it is important to under- analytics: a new tool for a new era. EDUCAUSE Rev.
stand the factors that affect the students’ learning 42(4), 40 (2007)
progress so that more interventions can be done to Chatti, M.A., Dyckhoff, A.L., Schroeder, U., Thüs, H.:
help to solve the problems of talent shortage. A reference model for learning analytics. Int.
J. Technol. Enhanc. Learn. 4(5–6), 318–331 (2012)
Daniel, B.: Big data and analytics in higher education:
opportunities and challenges. Br. J. Educ. Technol.
References 46(5), 904–920 (2015)
Dice: Job Market Report 2017. Dice Publishing. http://
Bacigalupo, M., Kampylis, P., Punie, Y., Van den Brande, assets.theitjobboard.com/JE/uk/Dice_Job_Market_Re
G.: EntreComp: The Entrepreneurship Competence port_2017.pdf (2017). Accessed 25 July 2018
568 Design Framework for Learning to Support Industry 4.0

Evangelinos, G., Holley, D.: Investigating the digital liter- 18_lo%206_25_18_FINAL.pdf (2018). Accessed
acy needs of healthcare students when using mobile 20 Sept 2018
tablet devices. EAI Endorsed Trans. e-Learn. 160, 60– Mavroudi, A., Giannakos, M., Krogstie, J.: Supporting
67 (2016) adaptive learning pathways through the use of learning
Ferguson, R.: Learning analytics: drivers, developments analytics: developments, challenges and future oppor-
and challenges. Int. J. Technol. Enhanc. Learn. 4 tunities. Interact. Learn. Environ. 26(2), 206–220
(5–6), 304–317 (2012) (2018)
Ferrari, A.: DIGCOMP: A Framework for Developing McWilliam, E.: Teaching for creativity: from sage to guide
and Understanding Digital Competence in Europe. to meddler. Asia Pac. J. Educ. 29(3), 281–293 (2009)
JRC83167 Scientific and Policy Report. Publications Meyer, B., Haywood, N., Sachdev, D., Faraday, S.: Inde-
Office of the European Union, Seville, Spain (2013) pendent Learning: Literature Review. Research Report
Gerstein, J.: Moving from Education 1.0 through Educa- No. DCSF-RR051. Department for Children, Schools
tion 2.0 towards Education 3.0. In: Blaschke, L.M., and Families, Nottingham (2010). www.gov.uk/govern
Kenyon, C., Hase, S. (eds.) Experiences in Self- ment/uploads/system/uploads/attachment_data/file/
Determined Learning, pp. 83–98. Createspace Inde- 222277/DCSF-RR051.pdf. Last accessed 8 Apr 2018
pendent Publishing Platform, California (2014) Pérez-López, D., Contero, M.: Delivering educational mul-
Greenwood, J.: The Third Industrial Revolution: timedia contents through an augmented reality applica-
Technology, Productivity, and Income Inequality tion: a case study on its impact on knowledge
(No. 435). American Enterprise Institute, Washington, acquisition and retention. Turk. Online
DC (1997) J. Educ. Technol. 12(4), 19–28 (2013)
Greller, W., Drachsler, H.: Translating learning into num- Petrina, S.: Multidisciplinary technology education. Int.
bers: a generic framework for learning analytics. J. Technol. Des. Educ. 8, 103–138 (1998)
J. Educ. Technol. Soc. 15(3), 42 (2012) Robinson, K.: Out of Our Minds: Learning to Be Creative.
Hatlevik, O.E., Guðmundsdóttir, G.B., Loi, M.: Examining Wiley, West Sussex (2011)
factors predicting students’ digital competence. J. Inf. Romero, C., Ventura, S., Pechenizkiy, M., Baker, R.S.:
Technol. Educ. Res. 14, 123–137 (2015) Handbook of Educational Data Mining. CRC Press,
Hughes, T.P.: Networks of Power: Electrification in West- London (2010)
ern Society, pp. 1880–1930. JHU Press, Baltimore Ryan, R.M., Deci, E.L.: Intrinsic and extrinsic motivations:
(1993) classic definitions and new directions. Contemp.
Khalil, M., Ebner, M.: What is learning analytics Educ. Psychol. 25(1), 54–67 (2000)
about? A survey of different methods used in Saveedra, A., Opfer, V.: Teaching and Learning 21st Cen-
2013–2015. In: The Smart Learning Conference, tury Skills: Lessons from the Learning Sciences.
Dubai, UAE, 294–304. Dubai: HBMSU Publishing A Global Cities Education Network Report. Asia Soci-
House (2016) ety, New York (2012). http://asiasociety.org/files/rand-
Khan, R.N.: The Third Industrial Revolution: An Eco- 0512report.pdf. Accessed 8 Apr 2018
nomic Overview. Impact of Science on Society, vol. Siemens, G., d Baker, R.S.: Learning analytics and educa-
146, pp. 115–122. Taylor and Francis/UNESCO, Lon- tional data mining: towards communication and collab-
don (1987) oration. In: Proceedings of the 2nd International
Larusson, J.A., White, B.: Learning Analytics: From Conference on Learning Analytics and Knowledge,
Research to Practice, vol. 13. Springer, Heidelberg pp. 252–254. ACM, Vancouver, British Columbia,
(2014) Canada (2012)
Leadbeater, C.: What’s Next? 21 Ideas for 21st Century Tang, M., Werner, C.H. (eds.): Handbook of the Manage-
Learning, Innovation Unit, London (2008) ment of Creativity and Innovation: Theory and Prac-
Lepper, M.R., Corpus, J.H., Iyengar, S.S.: Intrinsic and tice, pp. 225–229. World Scientific, London (2017)
extrinsic motivational orientations in the classroom: Van den Bogaard, M.E.D., De Vries, P.: Learning analytics
age differences and academic correlates. is about learning, not about analytics. A reflection on
J. Educ. Psychol. 97(2), 184 (2005) the current state of affairs. In: 45th Annual SEFI Con-
Liñán, L.C., Pérez, Á.A.J.: Educational data mining and ference, Terceira Portugal, Lisbon (2017)
learning analytics: differences, similarities, and time Vuorikari, R., Punie, Y., Carretero Gomez, S., Van den
evolution. Int. J. Educ. Technol. High. Educ. 12(3), Brande, G.: DigComp 2.0: The Digital Competence
98–112 (2015) Framework for Citizens. Update Phase 1: The Concep-
Lortie, D.: Schoolteacher: A Sociological Study. Univer- tual Reference Model. JRC27948 Scientific and Policy
sity of Chicago Press, Chicago (2002) Report. Office of the European Union, Luxembourg
Manpower Group: Talent Shortage Survey 2018- Solving (2016)
the Talent Shortage: Build, Buy, Borrow and Bridge. Wallner, T.: The return of the human. In: Jeschke, S., Hees,
Manpower Group Publishing. https://go. F., Richert, A., Trantow, S. (eds.) Prethinking Work –
manpowergroup.com/hubfs/TalentShortage%202018 Insights on the Future of Work, pp. 19–20. Lit-Verlag,
%20(Global)%20Assets/PDFs/MG_TalentShortage20 Hamburg (2012)
Design of Alienation in Video Games 569

Playwright Bertolt Brecht argues that “The

Design of Alienation in Video modern theatre mustn’t be judged by its success
Games in satisfying the audience’s habits but by its suc-
cess in transforming them” (Brecht 1964). A term
Kemal Akay1 and Cansu Nur Simsek2 coined by him, “alienation” is an effect that
1
Unity Technologies, Copenhagen, Denmark intends to disrupt the performative characteristic
2
Kadir Has University, Istanbul, Turkey of a play to render the act of performance abnor-
mal by underlining the conventions of the play,
rather than its content. The goal is to raise self- D
Synonyms awareness among the audience, make the theatri-
cal devices visible, and as Brecht states, change
Alienation; Defamiliarization; Mechanics the way an audience engages with a text.
According to Brecht, theater should attempt
to stimulate social change by being provocative
Definition and didactic and also it should address their reason
rather than their senses (Brecht 1991). Brechtian
In this entry, it is proposed that game design- theater aims to break down the illusion of reality
ers can utilize hardware systems and the struc- with the alienation effect which makes the audi-
ture of gameplay to highlight a game’s formal ence critically aware of itself as audience and of
characteristics for “alienation” effect. Draw- the play as artifice. The audience is encouraged to
ing inspiration from Bertolt Brecht’s notion of observe their own daily circumstances in a new
“Verfremdungseffekt,” or alienation effect, and light.
using MDA framework (Hunicke et al. 2004), traits When applied to video games, the concept of
between alienation effect and techniques used in alienation can further enhance the quality of an
specific video games are analyzed to define what audience’s interaction with the medium (Dunne
this article calls as “gamic alienation.” Through the 2014). However, the studies between alienation
reading of gameplay moments in various games effect and video games are not new to the field.
that make use of gamic alienation, a continuum For instance, it is known that horror game genre
that consists of two important concepts for creating can use this effect to defamiliarize the player from
this effect is identified: hardware signifiers and the game; but with rare exceptions, the purpose is
software signifiers. Throughout the entry, it is not to urge self-reflexivity, instead to convey spe-
discussed how the use of these signifiers is cific aesthetic effects (Kirkland 2007). Therefore,
implemented in a range of video games, from it is important to acknowledge that the use of
triple-A games such as Metal Gear Solid to indie alienation effect may not always lead to critical
games such as DLC Quest. play behavior.
Technical glitches, lags, and bugs can also
be considered as alienating effects (Galloway
Introduction 2006). In addition, loading screens or in-game
interfaces such as pause screen or saving slots
In the area of video games, the player has similar- can be interpreted as mechanical systems that
ities with the audience of a theatre play. The player detract the player from core gameplay (Dunne
is an active agent of the software, in which 2014). However, that analysis is beyond the
she provides the input, but concurrently, she is scope of this entry. It is important to acknowledge
also the audience of play. The participatory struc- that this article examines “gamic alienation” that
ture of video games allows the players to become is intentionally constructed by game designers in
spectators of their own engagement with the sys- order to break player’s engagement with a game.
tem. This cyclic process of input and output can In this entry, the term alienation effect is not
lead to changes in player’s approach and behavior. necessarily used in the political (i.e., Marxist
570 Design of Alienation in Video Games

aesthetics) sense used by Brecht in his discussion the objectives of MDA framework is to work as a
of epic theatre. However, an emphasis is placed on bridge to close the gap “between game design and
the critical attitude that is aimed to encourage the development, game criticism, and technical game
audience (in this context, players) to adopt. In research” (Hunicke et al. 2004).
particular, formal game structures that are in the In MDA, mechanics refer to formal aspects
pursuit of inducing reflection in terms of aes- of game design, namely, set of rules, for instance,
thetics and refer to its own mechanics are AI behavior, physics, winning condition, health
highlighted. In that regard, this entry uses a pre- system, etc., Dynamics is about how the game
existing framework Mechanics, Dynamics, and responds to actions player take during gameplay
Aesthetics (which will be referred to as “MDA”) such as commanding an army, controlling a gang,
and by mechanics, “particular components of the fighting enemies, or simply navigational stuff like
game, at the level of data representation and algo- walking and running. All the rules have to work
rithms” are meant (Hunicke et al. 2004). Aes- together and give proper feedback to the player.
thetics are defined as “the desirable emotional Aesthetics, on the other hand, defines player’s expe-
responses evoked in the player, when she interacts rience and main interests in the game. For example,
with the game system” and dynamics describe a player can start playing a game to get under the
“the runtime behavior of the mechanics acting skin of a superhero, to beat a friend’s high-score in a
on player inputs and each others’ outputs over competitive game, or to enter a fantasy world.
time” (Hunicke et al. 2004). Thus, “gamic alien- The mechanics work in concert to make the
ation” can be described as the technique in which dynamics, which generate the game’s aesthetics
the output of gameplay points out the mechanics (see Fig. 1). Game designers and players meet a
of the game, and it is intrinsically designed for the game from the opposite sides. At a fundamental
purpose of encouraging reflection among players. level, players start to experience a game through
The rest of the entry is structured as follows. aesthetics and game designers initially construct
The terminology behind MDA framework in rela- their ideas with the mechanics, which in turn
tion to “gamic alienation” is further explained. translates to player experience.
Then, two categories, namely, hardware signifiers Similar to a playwright, a game designer can
and software signifiers, that demonstrate the ways create the set of rules but has a little amount of
game designers use alienation techniques are ana- control over how the act of play will be enacted by
lyzed. These categories are explored with differ- the players. In that regard, MDA framework
ent examples. Finally, the entry shares its shows how a game designer can affect player’s
concluding marks and ends with suggestions for experience, but it is the player who crafts the
other works in future. gameplay experience.
A player needs to start with the experience
crafted by the game in order to understand the
Mechanics, Dynamics and Aesthetics in mechanics of the game. And mechanics are built
Relation to Gamic Alienation on top of two layers: hardware and software (see
Fig. 2). As it is demonstrated in the following
MDA framework provides a detailed terminology to sections, video games can make different use of
analyze how each component influence designer’s mechanics with hardware signifiers and software
choices and influences player’s experience. One of signifiers in order to invoke “gamic alienation.”

Design of Alienation in Video Games, Fig. 1 MDA as seen by designer and player
Design of Alienation in Video Games 571

In the next section, the first part of the contin- Psycho Mantis can lift all the items and objects
uum, hardware signifiers, is discussed (Fig. 3). in the room and throw them to the player’s avatar,
Solid Snake. All the attempts by the player to hurt
Psycho Mantis are failed and the player loses all
Hardware Signifiers the control over the controller because of Mantis’s
psychokinetic abilities.
In first Metal Gear Solid (MGS) (1997), the game The only way player can defeat Psycho Mantis
features a boss battle with an enemy called Psycho is to unplug the controller’s cable from the con-
Mantis. No details about the character are pre- sole’s port and plug into the second one. By D
sented before the encounter, except the clue in his switching the controller’s port, player regains
name. When the player first confronts him, a its control over Solid Snake and since Psycho
cutscene intervenes the gameplay, and during that Mantis cannot read her mind anymore, the player
cutscene, Psycho Mantis starts to comment on the can easily beat him.
save files of the player, depending on which system This instance of gameplay moment shows how
the game is being played. If MGS (1997) is played mechanics can be used to create a “gamic alien-
on PlayStation, then Psycho Mantis will comment ation.” Before presenting the actual challenge, an
on Konami-based save files (“You like Castlevania, in-game character refers to non-in-game elements
don’t you?”). If it is played on Nintendo (which is such as save data from other games, or explicit
called as Metal Gear Solid: The Twin Snakes reference to the controller. This gameplay
(2004) and is the remake of the original game), moment occurs at a point where player is already
then there are other specific games he is able to familiar with aesthetics and dynamics of the
recognize, which are Castlevania: Symphony of the game, and by using hardware signifiers, the
Night (1997), Super Mario Bros. (1985), and Eter- mechanics in the game encourage the player to
nal Darkness: Sanity’s Requiem (2002). If there is question mechanics themselves and therefore, it
no save file that the system can detect, Psycho requires a critical reflection of the medium. The
Mantis says (“Your memory is empty”). Moreover, game defamiliarizes player from the game itself.
Psycho Mantis also comments on how many times This kind of alienation that makes use of hard-
the player has saved the game until that point of ware signifiers is also used in Nintendo DS game,
gameplay or how he is acting towards enemies in The Legend of Zelda: Phantom of Hourglass
the game (e.g., using violence or stealth). After (2007). One specific puzzle involves critical
that, he orders the player to put his controller on thinking and requires the self-awareness of the
the floor and he starts to move the controller by player in order to overcome the challenge. As it
using vibration feature. is known to all, Nintendo DS features two split
When this sequence is over, the player starts to screens. And in this puzzle, the player needs to
fight with Psycho Mantis. During the fight, press the seal on a map from upper-screen to
bottom-screen. The player needs to close her DS
and open it again in order to complete the task.
Another prominent example that makes use of
this approach is Boktai: The Sun is in Your Hand
(2003). A Game Boy Advance title directed by the
Design of Alienation in Video Games, Fig. 2 The designer of Metal Gear Solid (1997), Hideo
relationship between hardware, software and mechanics Kojima, Boktai: The Sun is in Your Hand (2003)

Design of Alienation in Video Games, Fig. 3 A continuum to show the categorization of gamic alienation
572 Design of Alienation in Video Games

introduces a vampire hunter called Django and the taken by player. Unexpectedly, the game plays
game asks the player to be outdoors and get sun- with narrative conventions in so far that in most
light. The game asks this because the game’s car- of the cases, the descriptions of the narrator con-
tridge included a daylight sensor (Plunkett 2017) flicts actions taken by the player.
and you can save the energy from sunlight in The game starts in an office room and in the
batteries and later, Django can use that energy for very beginning, the player finds himself in a room
his weapon to play against his enemies in the dark. with two doors. The narrator tells that “When
Boktai: The Sun is in Your Hand (2003) is simply Stanley came to a set of two open doors, he
another title that induces reflection on the game’s entered the door on his left.”
medium. Even though the game explicitly men- Evidently, it depends on the player to choose
tions that only sunlight can be used to save energy, which door she wants to enter. Choosing right
some players found out creative solutions to get door breaks the narrative progression of narrator
sunlight without going outdoors (Plunkett 2017). and tells the player to return back to her path.
This is just another proof that raising a critical Otherwise, narration continues smoothly until to
approach via mechanics can lead to unexpected a certain point where something that narrator has
gameplay styles and offer different aesthetics. not planned appears.
The conflict between narrator’s comments and
player’s action is the first and major type of alien-
Software Signifiers ation effect used in The Stanley Parable (2013).

In the previous section, the use of hardware sig- Demonstration of Game’s Production Process
nifiers within the context of “gamic alienation” is The second type of alienation in the game is about
examined. The three titles that are covered dem- showing the scenes and assets behind the game’s
onstrated how mechanics can be used to refer to production process. There is one particular part in
physical space outside game’s world. the game where a female narrator takes over and
In this section, different approaches to create teleports Stanley to a museum so that he can
alienation with the use of software signifiers are explore the game’s production process. In this
identified, in particular through the reading of The museum level, player can see the credits, content
Stanley Parable (2013). Unlike the examples that from different endings, scale model of levels in
use hardware signifiers, The Stanley Parable the game, the details about Steam’s Greenlight
(2013) has been selected since it provides a wide process, and the assets that was removed in the
range of “gamic alienation” moments that are final version of the game. But this is rather an
distinct than the ones already analyzed. In no interesting use of alienation as these assets take
particular order, these approaches created with place in the very final build of the game, even
the use of software signifiers can be listed as: though they are claimed to have been removed.
For example, in one of the rooms in the museum,
– Conflict between narrator’s comment and player sees “Warzone” scene and it is written that:
player’s actions Early in development, we designed an ending
– Demonstration of game’s production process where Stanley would end up on a battlefield fighting
– Reference to other game worlds aliens. The action game would become sentient and
would wage war against the Narrator. We realized
shortly after starting to build it that it was far too
Conflict Between Narrator’s Comment and jokey and on-the-no for the tone of the game.
Player’s Actions
Stanley Parable is an indie game that includes a Thus, players are essentially playing a level
voice-over narrator who tries to tell a story by within the game where they are able to see behind
commenting on the actions and the decisions the scenes of development and understand the
Design of Alienation in Video Games 573

intentions of game designers. This level suspends messages implied through mechanics, it plays an
the gameplay and playfully refers to its own arti- important role to perceive software signifiers.
ficiality by using software signifiers to create
alienation effect.
Conclusion
Reference to Other Game Worlds
The third type of “gamic alienation” through the In this entry, the concept of alienation is examined
use of software signifiers can be described with through the reading of several games. The article
the references to other game worlds. These refer- initially draws inspiration from Brecht’s alien- D
ences can either be visual analogies to game envi- ation effect, but it is primarily based on MDA
ronments of other titles or replication of framework. The entry identifies two important
mechanics from other games. terms: hardware signifiers and software signifiers.
The Stanley Parable (2013) accommodates And games can be placed differently on this
levels from other titles such as Minecraft (2009) continuum.
and Portal (2007). There is a special ending in the The most important conclusion of gamic alien-
game where narrator gives up on player and ation is the games that use this effect want their
instead of convincing Stanley to follow his orders, players to take a critical stance towards the game
he loads example levels from Minecraft (2009) itself and become self-aware. This is done by
and Portal (2007), respectively. changing player’s focus from the avatar’s actions
Another title that serves as a good example to and events taking place on the screen and by
explain this category is DLC Quest (2012). The drawing attention to game’s mechanics, including
title of the game refers to Downloadable Content the hardware that the game software is running on.
(DLC), and it is a satirical critique of games that Nonetheless, there are issues that need to
promote DLC packages. DLC Quest (2012) does be addressed in a future research. First important
not incorporate the mechanics from other titles point is that the use of gamic alienation does
literally but alludes their implications within its not always lead to critical play. Regardless
own mechanics. of game designer’s clear intentions, the gameplay
DLC Quest (2012) also includes DLCs, but experience can lead to hours of frustration among
unlike other games, DLCs in this game can be players, and it can eventually cause them to leave
purchased freely. The game is based on a 2d the game. It is worth analyzing why this issue
platformer genre, but it starts without any anima- occurs and how it can be addressed.
tion or sound. A merchant in the first level is met, Secondly, the parameters to determine a
reminiscent of the ones in an RPG game, and he game’s place on the hardware/software signifier
informs the player that he needs to collect coins to continuum are not clear. Further analysis is nec-
buy in-game components like sound, animation, essary to formulate the attributes of a game in the
enemies, and weapons. Obviously, the game aims pursuit of improving continuum’s structure. New
to critically refer latest industry trends for moneti- categories can be introduced if necessary.
zation models used by triple-A studios. The results listed above are only limited to the
Presenting game’s essential components such reading of few games. As such, they are not
as sound and animation as a separately sold DLC, exhaustive and therefore requires validation
required to be bought by the player to further through empirical research and examination of a
progress in the game, elevates its meaning to a wider range of video games. In addition, new
critical level. Hence, the player is considered as an emerging technologies such as VR and AR pro-
active agent who has a background knowledge of vide more opportunities for using alienation effect
the DLC issues. Although it is possible to play the in different ways. This entry shows the scarcity of
game without understanding the self-referential good examples that make use of hardware
574 Design Review Process

signifiers for creating gamic alienation. The find- Minecraft: [video game] Mojang, Mojang (2009)
ings in this entry can provide some insights on Plunkett, L.: The Kojima game that made you play in the
sun. Kotaku. https://kotaku.com/the-kojima-game-
alienation concept and potentially inspire game that-made-you-play-in-the-sun-1796303870 (2017).
designers to create new alienation effects. Accessed 27 Nov 2017
Lastly, instead of purely focusing on immer- Portal: [video game] Valve, Valve (2007)
sion, video games can embrace the technical attri- Super Mario Bros: [video game] Nintendo R&D4,
Nintendo (1985)
butes it inherits as a medium and incorporate them The Legend of Zelda: Phantom of Hourglass: [video game]
to provide enriching play experiences. Just as Nintendo EAD, Nintendo (2007)
Brecht wanted to encourage the audience to ques- The Stanley Parable: [video game] Galactic Cafe, Galactic
tion the events on the stage, why don’t the video Cafe (2013)
games motivate players to be self-aware and take
more action beyond the screen?

Design Review Process

Cross-References
▶ Technologies for the Design Review Process
▶ Cognitive Psychology Applied to User Experi-
ence in Video Games
▶ Game Player Modeling
Destiny and Destiny 2, an
Analysis of an FPS
References
Kyle McCarter2, Brandon Coker2 and
Boktai: The Sun is in Your Hand: [video game] Konami Newton Lee1,2
Computer Entertainment Japan, Konami (2003) 1
Institute for Education, Research, and
Brecht, B.: Alienation Effects in Chinese Acting. Brecht on
Theatre: The Development of an Aesthetic. (J. Willett,
Scholarships, Los Angeles, CA, USA
2
Trans.) Hill and Wang (1964) (Original work published Vincennes University, Vincennes, IN, USA
1936)
Brecht, B.: Brecht on Theatre, Hill and Wang, p. 121.
Methuen, London (1991)
Castlevania: Symphony of the Night: [video game] Konami
Synonyms
Computer Entertainment Tokyo, Konami (1997)
DLC Quest: [video game] Going Loud Studios, Going Multiplayer first-person shooter; Video Games
Loud Studios (2012)
Dunne, D.J.: Brechtian alienation in videogames. Press
Start. 1(1), 79–99 (2014)
Eternal Darkness: Sanity’s Requiem: [video game] Silicon Definitions
Knights, Nintendo (2002)
Galloway, A.: Gaming: Essays on Algorithmic Culture. First-person shooter (FPS)¼ a genre of games where
University of Minnesota Press, Minneapolis (2006)
Hunicke, R., Leblanc, M., Zubek, R. MDA: a formal
the camera focuses on a gun (or other weapon) where
approach to game design and game research. the objective is to eliminate specific targets.
In: Proceedings of the Challenges in Games AI Work-
shop, Nineteenth National Conference of Artificial
Intelligence, AAAI Press, San Jose (2004) 2
Kirkland, E.: The self-reflexive funhouse of Silent Hill.
Multiplayer First-Person Shooter (FPS)
Converg.: Int. J. Res. New Media Technol. 13(4),
403–415 (2007) Destiny is a multiplayer first-person shooter fran-
Metal Gear Solid: [video game] Konami Computer Enter- chise developed by Bungie. Destiny 1 was
tainment Japan, Konami (1997)
Metal Gear Solid: The Twin Snakes: [video game] Konami
published by Activision, while Destiny 2 was
Computer Entertainment Japan/Silicon Knights, self-published by Bungie. Destiny 1 was released
Konami (2004) for the Xbox 360, Xbox One, Playstation 3, and
Destiny and Destiny 2, an Analysis of an FPS 575

Playstation 4 in 2014. Destiny 2 was released for a lot of time collecting armor and weapons; many
the Xbox One, Playstation 4, and Windows in hope to achieve to collect a full armory.
2017 while getting a Stadia release in 2019. The If there is a point of contention in Destiny, it
most recent expansion, Beyond Light came out in would be their multiplayer matchmaking, called
November 2020. All games are rated Teen. the Crucible in Destiny 2. In the first game, players
Destiny’s target audience is anyone who could pick a game mode, but not a map. In Destiny
enjoys a science-fiction first-person shooter. The 2, the player is unable to pick a game mode or map
gameplay and mechanics are similar to other pop- to play on. This leaves players who wish to only
ular first-person shooters but with a little hint of play one mode a bit out of luck; they have to hunt D
difference when it comes to abilities. Destiny 2 is for a custom game and hope that they find a lobby
similar to other first-person shooters, such as the that fits them if they do not already have a group of
Halo and Call of Duty games. It has the same like-minded players.
basic controls, but the right bumper provides an There is also one major controversy with the
extra method of attacking, usually a melee option. series: the DLC content and the game itself. In the
The Destiny universe takes place in the Milky first game, Bungie did not release the whole game
Way, several thousands of years after the present right way. Instead they decided to slowly release it
day. The bulk of the game takes place in a giant through DLC release. It was not until their final
open world where the player travels from planet to DLC that the game’s mechanics were fully realized.
planet trying to find the Vanguard, high ranking Some players believe that Destiny 2 is an
members of the same race as the player character, improvement from the original Destiny, but
to rally together in hopes of reclaiming their home some others opine that the original game is supe-
from the alien race known as the Cable, who are rior (Heather 2021 & Stevryu et al. 2017). To
led by the warlord named Ghaul. In the beginning compare and contrast the two versions, we can
of the first game, they come to Earth and try to examine the following:
steal the Traveler’s Light, who is a moon-sized
god that granted the power of the light to human- 1. Player Base: Although Destiny is very popular,
ity and its allies. Destiny 2 has a much larger player base with
In terms of the HUD, there is a health bar on the over 20 million players.
top middle that only appears after the player takes 2. Weapons: Players can carry three weapons in both
damage. The bottom left lists of the player’s Destiny and Destiny 2. The major advantage of
ammo, grenade, melee ability, and super energy Destiny 2 is that all weapons are already fully
bar. The top left houses a radar (when available). leveled up, unlike Destiny where players have to
Players can compete in several player versus spend time and resources to level up a weapon.
enemy modes to earn loot. Loot ranges from com- Destiny 2 also provides more exotic weapons cat-
mon, uncommon, rare, legendary, and exotic. egorized into kinetic, energy, and power weaponry.
They all drop from enemies corpses form of an 3. Story: Unlike the original Destiny, every story
engram, which is a dodecahedron that is colored and plotline in Destiny 2 has a clear beginning,
to show rarity. From the most common to rare the middle, and end.
colors are white, green, blue, purple, gold/yellow. 4. Character Development: Both Destiny and
There are a few other ways to get rare loot, which Destiny 2 offer three character classes, namely,
include getting them from stories, bounties, Titan, Hunter, and Warlock. In Destiny, players
quests, doing a raid which is the hardest player can customize the subclasses. In Destiny 2,
versus enemy quest players can do, or doing customization is replaced by new special abil-
multiplayer. Engrams hold weapons or armor ities for each character.
that players can use to upgrade their light level. 5. Armor: Destiny 2 allows customization of
Light level is a character’s armor and weapon armor with mods and perks.
rating; basically how much damage they can 6. Content: Compared to the original Destiny,
take and deal to enemies. The player base spends Destiny 2 has more worlds to explore, more
576 Destiny and Destiny 2, an Analysis of an FPS

raids and dungeons, and more competitive Destiny 2 provides additional Guided Games
modes like Gambit and Crucible. feature for solo players to team up with others
7. Hybrid PvP/PvE: Destiny 2 offers a hybrid to complete missions.
PvP/PvE mode that does not exist in the orig-
inal Destiny. In conclusion, Destiny 2 is certainly an
8. Raids: Both Destiny and Destiny 2 require a improvement over the original Destiny. Neverthe-
team of six players to complete a raid mission. less, every player is entitled to his or her own

Destiny and Destiny 2, an Analysis of an FPS, Destiny 1 Destiny 1

Destiny and Destiny 2, an Analysis of an FPS, Destiny 2 Destiny 2

Detecting and Preventing Online Game Bots in MMORPGs 577

opinion as far as what game he or she likes virtual items to them. Among the various types of
the most. games, MMORPGs (Massively Multiplayer
Online Role Playing Games) make up one of the
most popular genres.
References As online games gain economic and social
importance, various forms of threats emerge.
Heather. June 2, 2021. Destiny vs Destiny 2 – Which is A variety of methods have developed to parasitize
Better?. https://geekvibesnation.com/destiny-vs-
and gain unfair advantages in online games. In
destiny-2-which-is-better/
this chapter, we focus on cheating actions using D
Stevryu, Saniyaga, JonRyan-IGN. 5 Sep 2017. Destiny VS
Destiny 2 Differences. https://www.ign.com/wikis/ the game bot, which is one of the most prevailing
destiny-2/Destiny_VS_Destiny_2_Differences reasons why users get banned from the game
company.
Game users cheat to level up and accumulate
cyber assets in an easy and fast way without
Detecting and Preventing sufficient effort. Game items and game money
Online Game Bots are critical to increasing the survivability of
in MMORPGs in-game characters by improving their power
and reputation. In particularly, MMOPRGs are
Huy Kang Kim and Jiyoung Woo designed such that players take prescheduled
Graduate School of Information Security, Korea courses to achieve high-level characters and
University, Seongbuk-Gu, Seoul, Republic of become rich in cyber assets. These courses require
Korea the users to spend a considerable amount of time
on repetitive play.
To skip these time-consuming processes for
Synonyms achieving high-level characters and acquire more
cyber assets within a short period of time, users
Cheating; Detection; Game bot; MMORPG begin to cheat. One of the most frequently used
tools for cheating in online games is the game bot.
The game bot enables users to cheat in a conve-
Definition nient way by automatically performing the
required actions. A typical game bot is an auto
Game users cheat to level up and accumulate program that plays the game instead of a human.
cyber assets in an easy and fast manner without Some users are eager to achieve a high level
sufficient effort. One of the most widely used tools within a short period of time, so they buy virtual
for cheating in online games is the game bot, goods or higher-level accounts by paying real
which enables users to cheat in a convenient way money. Game items and currency gained through
by automatically performing the required actions. game play can be sold and monetized into real
Therefore, game companies employ various secu- currency. Real money trading of virtual goods is
rity solutions for the detection and prevention of also one of reasons why players cheat. The ille-
game bots. gitimate activity of gathering virtual goods in
online games primarily for real money is called
gold farming (Davis 2009). Gold farming is one of
Introduction the most problematic issues in online games
because gold farming is not only performed at an
Online gaming is one of the successful Internet individual level but also by a factory-sized illegal
services. In the past few years, online games have group. “Gold farming groups” are industrialized
become popular and have been generating huge organizations that gather and distribute virtual
profits. Online game companies generate profits goods for capital gain in the online gaming
by charging users a subscription fee and selling world (Keegan et al. 2010). Real money trade by
578 Detecting and Preventing Online Game Bots in MMORPGs

gold farming groups has now become a social Literature Review

problem in many countries. Real money trade
leads to tax evasion, illegal labor in developing Technology Trends
countries, and damage to the game company. Game bot detection methods have evolved over
Gold farmers mainly use game bots for a large- the years in the following order:
scale operation, thus saving on labor costs.
Cheating in online games is no longer a per- • Client-side detection and network side detec-
sonal problem of a player and causes damage to tion (first generation)
other players since online games maintain a mas- • Server-side detection (second generation)
sive set of players who form social relationships • Surgical strike (third generation)
with other players. • Proactive detection (advanced surgical strike)
Cheating causes an unfair competition, spoils
other players’ fun, and makes other players lose The first-generation methods are signature-
their motivation for the play. For example, bot based methods. Client-side bot detection includ-
users play the game much longer without a ing antivirus programs and CAPTCHA
break, while playing continuously for a long (Completely Automated Public Turing test to tell
time is difficult for human players. Since the Computers and Humans Apart) is a first-
activities and extraordinary abilities of game generation method. The first generation of com-
bots are noticeable to other players in the game mercial products could be detoured with reverse
world, they can make users feel that the game is engineering. In addition, CAPTCHA can annoy
unfair and cause users to leave the game. For users.
companies, the detection and prevention of The second generation is the server-side
cheating incur a significant expenditure. When detection. This method mainly focuses on
players use a game bot, they can reach the highest distinguishing between a bot player and a normal
level in a shorter period of time than that expected player by analyzing server-side log files. As gold
by the game designers. Then, the users at the farming becomes industrialized, the detection of
highest level have no more game contents to all bot users is difficult to implement because of
enjoy and leave the game. Consequentially, the huge number of bot users. It is not efficient
cheating causes various losses to the game com- because gold farming groups provide standby
pany. It reduces the item sales amount, and even- characters against being banned by the game com-
tually the number of players, and shortens the pany. Against the endless arm race battle between
game lifecycle. gold farming groups and game companies, we
Online game companies realize the seriousness need a more efficient and selective detection
of the damage caused by a gold farmer group; they method. The third-generation method is a surgical
detect game bots and then ban their accounts or strike policy (Woo et al. 2011, 2013a, b). This
IPs to neutralize the gold farming group. They method examines a gold farming group’s ecosys-
actively hire monitoring personnel, called GM tem and performs banning to maximize the ban-
(Game Masters), and deploy security solutions ning effect while minimizing the banning
on both the client side and the network side. In operation.
addition, they adopt log analysis systems to detect The current approaches in previous research
game bot users. rely only on the analysis of behavior patterns.
Here, we will discuss some literature on state- When the players’ activity patterns are distin-
of-the-art game bot detection and prevention guishable from the majority, these players are
methods, with the aim to fully understand the suspected to be malicious users. Most game
current prevention techniques and to advance companies take an action that bans bot users’
countermeasures against the use of game bots in accounts when they repetitively use a
online games. game bot.
Detecting and Preventing Online Game Bots in MMORPGs 579

Previous detection methods are reactive; they Network-Side Detection Methods

detect bot users only after users have repetitively Network-side detection methods are designed to
exhibited malicious behavior. For a proactive and look into network traffic or network packets gen-
preventative response against malicious users, the erated by the client when the client communicates
social network analysis-based detection methods with the game server. This type of method detects
including the study on the contagion over the network traffic burstiness and anomalies in the
social network are used. command timing, response time, and traffic inter-
val. Some methods are designed to detect different
Client-Side Bot Detection Methods reactions of humans and game bots to changes in D
Client-side methods are focused on signature- network conditions such as protocol changes or
based, client-side BOT detection, such as anti- the occurrence of traffic lags.
virus programs and CAPTCHA. PunkBuster When a company changes its network proto-
(http://www.punkbuster.com) was the first col, game bots lose their connection and pause to
attempt to avoid cheating on the client side. This update their programs so that they can be fitted to
tool monitors a client machine looking for any the changed protocol. When a company generates
abnormality and sends a screenshot of the client a traffic lag by design, human players react to the
to the game server. Khanh (2010) proposed a change sensitively, for example, by continuously
specifically designed module called GameGuard, clicking their mouse or keyboard buttons until the
which is added to the client to guard against pos- connection resumes, whereas game bots are not
sible cheating attempts from game players. sensitive.
GameGuard hides security-related modules so The network-side solution to a nonclient bot is
that hacking tools cannot see them, by using cer- to ask a question to the user who can give a right
tain system techniques to interfere with some answer when she uses the client software provided
operations of Windows systems. It also places by the game company. The nonclient bot does not
hooks in the Windows kernel mode to examine operate on the client software, so it cannot provide
new processes entering the system and check the right responses for specialized questions.
whether these processes are hacking attempts Similarly, most network-side solutions mainly
or not. adopt a method that frequently changes the net-
CAPTCHA requests the answers that can be work protocol or applies cryptography to encrypt/
easily solved by humans but are difficult for bots decrypt network transmission.
to solve. Yampolskiy and Govindaraju (2008) For the P2P game, the game company facili-
proposed the integrated testing procedure as part tates a method for participants to reach a consen-
of the game step performed by the player during sus on the current state of the game in a way that
the game in order to distinguish bots from legiti- prevents malicious individuals and groups from
mate human players. Golle and Ducheneaut cheating. Securing the protocol that delivers mes-
(2005) demonstrated the CAPTCHA test embed- sages when peers communicate is a solution to
ding in the game world in order to minimize the this security issue.
disruption compared to the out-of-band use of The NEO (GauthierDickey et al. 2004) proto-
CAPTCHAs. col is designed to prevent protocol-level cheating
However, the first generation of commercial in P2P games. This protocol enables to accom-
products could be detoured with reverse engineer- plish the exchange of update information among
ing. In addition, the client-side solution has other players in an authenticated manner. It verifies the
drawbacks. Client-side solutions run on the client identity of the sender and registers updates in
computer. This causes a load on the client com- advance to ensure that every party has chosen its
puter and can cause any inconvenience to users. update for a round before it may read the updates
CAPTCHA can annoy users when they receive of the other players. It also attempts to ensure
questions while they are playing. game state consistency by confirming which
580 Detecting and Preventing Online Game Bots in MMORPGs

updates have been received by which peers. The User behaviors in the gaming world include
details of the NEO protocol (Hu and Liao 2004) major activities such as move and play. Other
are as follows: It divides the time into uniform socializing behaviors are also a good source for
intervals, called rounds, in which each player game bot detection. MMOPRGs are designed to
sends an update to all the other players. Each make people interact with others in order to com-
update is encrypted, and in the following round, plete difficult tasks and then level up their charac-
the players send the corresponding key to each ters and have fun playing the game through such
other. interactions. User behaviors can be categorized
Corman et al. (2006) invented a more secure into sole behaviors and social behaviors. The
protocol called SEA (Secure Event Agreement). main sole behaviors are movement and play. In
The authors investigated the drawback of the MMORPGs, a play encompasses combat, harvest,
NEO protocol and then proposed an improved and healing. Social behaviors mainly include
version of this protocol. They focused on the party play, communication, trade, and community
protocol for communications rendering informa- activity.
tion such as the position, direction, and action of
virtual entities. The SEA protocol signs an entire • Party play means that two or more players
event message and binds the message to a partic- form a group to undertake quests or missions
ular round and group. together. Users in party play typically share
Most of the network-side countermeasures are experience points, money, and items acquired
based on a cryptographic methodology to guaran- upon completion of successful quests. Most
tee network traffic integrity and confidentiality. To MMORPGs are designed to require
secure network traffic, it is necessary to encrypt party play.
and decrypt the packets. However, this requires a • Players in the game communicate with other
considerable amount of computing power, and the players by sending text messages or e-mails.
error in this process can spread to all concurrent In the gaming world, players can trade
user connections. This can cause single-point-of- items. In general, players exchange items
failures, so game companies are reluctant to adopt with other items of equivalent value or
strong network-side cryptographic methods. money. Trade patterns provide a good clue
to detect abnormalities, particularly gold
Server-Side Detection Methods farming groups.
Server-side methods use the log collected in the • In general, players maintain a friend list for
server of the company. The company records user easy co-play and communication. Player
behaviors as a log in the database. Game bots A sends a friend request to player B. When
display repeated and biased patterns in their player B accepts the request from player A,
actions differing from human players. Techni- they become friends. They show up in the
cally, server-side methods adopt data mining- friend list of the other party.
based or statistics-based methods. First, these • Players organize a guild to socialize or achieve
types of methods extract the feature set from the a similar long-term goal.
log. Then, classification using the feature set is
performed; classifiers are automatically built Previous works focused on movement patterns
through learning from data by using data mining and sole play patterns because they used simula-
or statistical methods. tion data obtained by operating game bots. This is
Feature extraction is a critical component of an alternative method used when a real game log
behavior-based detection techniques. First, we is not available. Bots are programmed, so their
build a taxonomy of behavior-based features that movement and play patterns are repetitive and
classifies previous research in a systematic way. regular. Bot detection models have been pro-
Then, we review previous works based on this posed based on this fact. Social behaviors have
taxonomy. recently been adopted in research. Social
Detecting and Preventing Online Game Bots in MMORPGs 581

behaviors cannot be obtained in the absence of the sequences of bots and human players.
the cooperation of a game company. Fortunately, Since bots can make restricted choices when
several studies provide the analysis results of they play the game, they show limited and
large-scale real data under the cooperation with repetitive behaviors. The authors implemented
game companies. their proposed detection tool on a server-side
Now, we will review key papers in detail. and client-side instance and an in-game client-
Movement-based methods use the fact that side add-on that is capable of monitoring its
most bots have preconfigured moving behaviors own event horizon.
while humans have random moving patterns • Lee et al. (2014) proposed a measure that D
(Kesteren et al. 2009; Mitterhofer et al. 2009). expresses the extent to which a player behaves
repetitively. They analyzed the full action
• The longest common path (LCP) is a good sequence of users on a big data analysis plat-
indicator that measures the regularity in move- form. The differences between the action
ment patterns. Bots take same paths repeti- sequences of normal users and bot users were
tively, so they have high LCP, while human determined.
players show random behavior and thus have
low LCP. Furthermore, bot users can turn the With respect to social behaviors, party play,
bot program on and off, which results in a high communication, and trade have been explored to
variation in regularity. Human players will identify the differences between game bots and
have low regularity and a low variation in human players.
regularity.
• Kang et al. (2013) focused on party play, which
Play patterns are widely used for is a group play with several characters, for
distinguishing between a bot behavior and a game bot detection. They pointed out that the
human behavior. game bot forms a party play with a limited
number of players, usually other game bots,
• Chung et al. (2013) considered all types of and has limited actions biased towards
behaviors that can be observed in the game collecting game money in the party play.
world. They categorized play patterns into Bat- • Kang et al. (2012) proposed a chatting-based
tle, Collect, and Move; specified battle behav- bot detection model. They retrieved chatting
iors as Hunting, Attack, Hit, Defense, contents and derived features by using text
Avoidance, and Recovery; and built a feature mining techniques from the chatting contents.
set by using such specified behaviors. The They also derived entropy-based features,
features can be directly retrieved from the chatting-pattern based features, and text fea-
game log. Further, they developed the feature tures. The proposed detection model assumes
set from raw features in order to represent how that game bots communicate with other bots
efficiently a user plays the game. The devel- through limited messages that humans find
oped features include combat ability, collecting difficult to understand.
pattern, and movement pattern.
• Christensen et al. (2013) examined the race To detect factory-sized illegal groups that oper-
duration from both the client side and the ate numerous game bots, i.e., gold farming
server side, and the gap between the duration groups, an understanding of the ecosystem of a
measured on both sides. They suspected a very gold farming group with respect to the trade pat-
short duration time, long duration, and incon- terns is required. The followings are some surgical
sistent duration time between the client side strike approaches.
and the server side.
• Platzer et al. (2011) analyzed the sequence of • Woo et al. (2011) and Kwon et al. (2015),
play patterns and identified the differences in respectively, identified the ecosystem of gold
582 Detecting and Preventing Online Game Bots in MMORPGs

farming groups, as shown in the following More advanced methods are based on the con-
figure. Gold farmers repeatedly hunt monsters tagion process in the players’ social networks.
and harvest craft materials to earn game money This generation method is in an early stage.
and to save time. The collected items and game
money are delivered to the merchant charac- • Woo et al. (2013a, b) and Blackburn
ters; merchant characters then sell these items et al. (2014) showed that a social contagion
for game money. The game money from gold of game bot usage can develop. The first and
farmers and the acquired money by the item the third study showed that contagion
trade through merchant characters transfer to between players in a social network exists.
banking characters. Banking characters pos- The second study proposed a modeling
sess most of the game money in the GFG to method to track the contagion process. This
sell the game money efficiently. modeling method is based on an epidemic
• Ahmad et al. (2011) considered player charac- model. Modeling of the diffusion process
teristics and items and detected contraband enables one to predict the future diffusion
networks in terms of players and items. They process and to estimate the likelihood of an
analyzed clandestine social networks of devi- individual’s bot adoption.
ant players in MMOGs (Massively Multiplayer • Ahmad et al. (2013) proposed label propaga-
Online Games) along with a network of con- tion to detect gold farmers. The proposed
traband items sold by these players. model initially sets the label that indicates the
• Oh et al. (2013) utilized the fact that game bots gold farmers, and then, propagates these labels
and human players form social networks in over highly connected networks, such as the
contrasting ways. They constructed a social mentoring network, housing-trust network,
network on the basis of mentoring relation- and trade network. To reduce false positives
ships. They derived features from the play in which normal users are misjudged as bot
and the trade and added social network-based users, the authors also considered user
features. These authors proposed new features similarity.
based on eigenvector centrality to capture the
social influence. The game company adopts a negative policy
• Keegan et al. (2010) and Ahmad et al. (2011) for bot users since banning accounts often causes
studied the clandestine trade and trust networks a legal issue between the user and the game com-
of gold farmers, respectively, and described pany. If the game company selectively targets
how gold farmers try to obfuscate their inter- users for banning, it can ban users for minimizing
action patterns in these networks to evade the compliance risk and maximizing the banning
detection. effect at the same time. Identifying influentials in
• Keegan et al. (2011) discussed the usefulness the diffusion process of game bot usage will solve
of studying clandestine networks in the virtual this issue.
world and their applications to studying their
counterparts in the offline world. • Ki et al. (2014) provided the analysis results on
• Blackburn et al. (2014) introduced an interest- a contagion of game bot usage and identified
ing platform, the Steam Community, an online the influentials in the contagion process. Iden-
social network built on top of the world’s dom- tifying the influentials in the diffusion of mali-
inant digital game delivery platform. They cious behaviors and understanding the
performed a thorough social network analysis diffusion process of the malicious behaviors
on the players’ social relationships and interac- is particularly important, as it will give the
tions. They found that the cheaters’ network game company a new opportunity to act pro-
position is largely indistinguishable from that actively and preventively against malicious
of fair players. users.
Detecting and Preventing Online Game Bots in MMORPGs 583

Conclusions Third, game bots are evolved to avoid detec-

tion, so behavior-based methods become outdated
This survey summarized how the game bot detec- in a short span of time. More features that reflect
tion methods have evolved in the recent years. It the cheater’s psychology and the behavior
further developed a taxonomy according to the lifecycle can be used for generating more genetic
data source for countermeasures and introduced models. In addition, behavior features are gener-
the state-of-the-art literature on the game bot ally game-specific, so generic features also should
detection and prevention methods. The game bot be considered.
detection methods have been developed from This survey will hopefully boost researchers’ D
client-side detection and network-side detection interests in this area of study, help in determin-
methods to server-side detection methods. Client- ing the research agenda, and finally, lead to
side security solutions that are required to be the development of countermeasures against
installed on the client computer often cause colli- game bots.
sions in the operating system, resulting in user
inconvenience. Network-side detection methods
such as network traffic monitoring or network Cross-References
protocol changes cause a network overload and
lags in the game play, a significant annoyance in ▶ Secure Gaming: Cheat-resistant Protocols and
the online gaming experience. Server-side Game History Validation
methods that are mainly based on log mining of
user behaviors produce highly accurate rate and
effective rules to detect game bots and do not References
interfere with the game play. With the evolution
of game bot usage, from being used by individual Ahmad, M.A., Keegan, B., Sullivan, S., Williams, D.,
Srivastava, J., Contractor, N.: Illicit bits: Detecting
users to gold farming groups, server-side methods and analyzing contraband networks in massively multi-
have been advanced to detect gold farming player online games, privacy, security, risk and trust
groups. Further, the detection methods have (passat). IEEE third international conference on and
become more proactive, getting hints from the 2011 I.E. third international conference on social com-
puting (SocialCom), pp. 127–134 (2011)
contagion process of the game bot usage on a
Ahmad, M.A., Keegan, B., Roy, A., Williams, D.,
social network that can be observed in online Srivastava, J., Contractor, N.: Guilt by association?
games. Network based propagation approaches for gold farmer
Through this survey, several research gaps detection, advances in social networks analysis and
mining (ASONAM). IEEE/ACM International Confer-
have been identified.
ence on, pp. 121–126 (2013)
First, the surgical strike methods for the detec- Blackburn, J., Kourtellis, N., Skvoretz, J., Ripeanu, M.,
tion of gold farming groups have not been Iamnitchi, A.: Cheating in online games: a social net-
explored sufficiently thus far. As gold farming work perspective. ACM Trans. Internet Technol.
(TOIT) 13(9) (2014)
increases, efficient and effective detection Christensen, J., Cusick, M., Villanes, A., Veryovka, O.,
methods are needed and the current literature has Watson, B., Rappa, M.: Win, Lose or Cheat: The Ana-
much room for improvement. lytics of Player Behaviors in Online Games, Technical
Second, the server-side methods mostly focus report (North Carolina State University. Dept. of Com-
puter Science), 1–7 (2013)
on feature extraction and apply existing machine
Chung, Y., Park, C.-y., Kim, N.-r., Cho, H., Yoon, T., Lee,
learning algorithms to build automatic classifiers. H., Lee, J.-H.: Game bot detection approach based on
However, an algorithm specialized for game bot behavior analysis and consideration of various play
detection has not been developed thus far. An styles. ETRI J. 35, 1058–1067 (2013)
Corman, A.B.., Douglas, S., Schachte, P., Teague, V.:
algorithm that reflects the characteristics of game A secure event agreement (sea) protocol for peer-to-
bot users or gold farming groups may be more peer games, availability, reliability and security. The
effective than the existing algorithms. First International Conference on. 8 (2006)
584 Detection

Davis, R.: Welcome to the new gold mines. The Guardian. Platzer, C.: Sequence-based bot detection in massive multi-
5, (2009) player online games. In: Information, Communications
GauthierDickey, C., Zappala, D., Lo, V., Marr, J.: Low and Signal Processing (ICICS) 2011 8th International
latency and cheat-proof event ordering for peer-to- Conference on, pp. 1–5 (2011)
peer games. In: Proceedings of the 14th International Woo, K., Kwon, H., Kim, H.-c., Kim, C.-k., Kim, H.K.:
Workshop on Network and Operating Systems Support What can free money tell us on the virtual black mar-
for Digital Audio and Video, pp. 134–139 (2004) ket? ACM SIGCOMM Comput. Comm. Rev. 41,
Golle, P., Ducheneaut, N.: Preventing bots from playing 392–393 (2011)
online games. Comput. Entertain. 3, 1–10 (2005) Woo, J., Kim, H. K.: Survey and research direction on
Hu, S.-Y., Liao, G.-M.: Scalable peer-to-peer networked online game security. In Proceedings of the Workshop
virtual environment. In: Proceedings of 3rd ACM at ACM SIGGRAPH Asia. pp. 19–25 (2012)
SIGCOMM Workshop on Network and System Sup- Woo, J., Kang, A.R., Kim, H.K.: Modeling of bot usage
port for Games, pp. 129–133 (2004) diffusion across social networks in MMORPGs. In:
Kang, A.R., Kim, H.K., Woo, J.: Chatting pattern based Proceedings of the Workshop at SIGGRAPH Asia,
game bot detection: do they talk like us? KSII Trans. pp. 13–18 (2013a)
Internet Inf. Syst. 6, 2866–2879 (2012) Woo, J., Kang, A.R., Kim, H.K.: The contagion of mali-
cious behaviors in online games. In: Proceedings of the
Kang, A.R., Woo, J., Park, J., Kim, H.K.: Online game bot
ACM SIGCOMM 2013 Conference on SIGCOMM,
detection based on party-play log analysis. Comput.
pp. 543–544 (2013b)
Math. Appl. 65, 1384–1395 (2013)
Yampolskiy, R.V., Govindaraju, V.: Embedded non-
Keegan, B., Ahmed, M.A., Williams, D., Srivastava, J., interactive continuous bot detection. Comput. Enter-
Contractor, N.: Dark gold: statistical properties of clan- tain. 5, 7 (2008)
destine networks in massively multiplayer online
games, social computing (SocialCom). In: IEEE Sec-
ond International Conference on, pp. 201–208 (2010)
Keegan, B., Ahmed, M.A., Williams, D., Srivastava, J.,
Contractor, N.: Sic Transit Gloria Mundi Virtuali?:
Promise and peril in the computational social science
of clandestine organizing. In: Proceedings of the 3rd Detection
International Web Science Conference, vol. 24 (2011)
Kesteren, M., Langevoort, J., Grootjen, F.: A step in the ▶ Detecting and Preventing Online Game Bots in
right direction: Bot detection in Mmorpgs using move- MMORPGs
ment analysis. In: Proceedings of the 21st Belgian-
Dutch Conference on Artificial Intelligence (2009)
Khanh, Van Nguyen, G.: A Windows-based software archi-
tecture for protecting online games against hackers. In:
Proceedings of the 2010 Symposium on Information
and Communication Technology, pp. 171–178 (2010) Diagram
Ki, Y., Woo, J., Kim, H.K.: Identifying spreaders of mali-
cious behaviors in online games. In: Proceedings of the
Companion Publication of the 23rd International Con- ▶ Unified Modeling Language (UML) for Sight
ference on World Wide Web Companion, pp. 315–316 Loss
(2014)
Kwon, H., Mohaisen, A., Woo, J., Kim, Y., Kim, H.K.:
Crime scene reconstruction: online gold-farming net-
work analysis, under review. In: IEEE Transactions on
Information Forensics & Security 2015, pp. 1–11
(2015) Dichromate Gelatin (DCG)
Lee, J., Lim, J., Cho, W., Kim, H.K.: In-Game action
sequence analysis for game bot detection on the big ▶ Holography as an Architectural Decoration
data analysis platform. In: Proceedings of the 18th Asia
Pacific Symposium on Intelligent and Evolutionary
Systems, vol. 2, pp. 403–414 (2014)
Mitterhofer, S., Platzer, C., Kruegel, C., Kirda, E.: Server-
side bot detection in massive multiplayer online games.
IEEE Secur. Priv. 7, 29–36 (2009) Diegetic Interfaces
Oh, J., Borbora, Z.H., Sharma, D., Srivastava, J.: Bot
detection based on social interactions in MMORPGs,
Social Computing (SocialCom). In: 2013 International ▶ Game Interface: Influence of Diegese Theory
Conference on, pp. 536–543 (2013) on the User Experience
Digital Games for Animals 585

Prerequisites for Digital Play in Animals

Diffraction Microscopy
Humans are far from the only species known to
▶ Holography, History of engage in playful behavior. Wirman et al. (2011)
argue that “all mammals play, including humans,”
and play-like behaviors have also been identified
in birds, reptiles, fish, and invertebrates such as
Diffusion-Reaction Problems octopuses (Burghardt 2005). While the evolution-
ary dynamics and psychological mechanisms of D
▶ Lattice Boltzmann Method for Diffusion-Reac- play are not yet fully understood, it is known that
tion Problems in both humans and other animals, playing is a
crucial prerequisite for physiological, psycholog-
ical, and social development (Bekoff 1972) and
play deprivation can result in serious develop-
Digital Collectibles
mental deficiencies in adulthood (Hol et al.
1999). Based on this evidence, scholars such as
▶ NFT Games
Sutton-Smith (1997: 218) have advocated for a
definition of play which encompasses both
humans and animals.
Digital Communication Animals are also known to interact with technol-
ogy in a variety of ways. Many pet owners have
▶ Life-Size Telepresence and Technologies observed their animal companions watch television,
and animals’ engagement with screen media can be
meaningful. Research suggests, for example, that
dogs can recognize other dogs on the screen and
Digital Games distinguish them from other species using visual
cues alone (Autier-Dérian et al. 2013).
▶ Computer Games and the Evolution of Digital Service animals are often trained to use tech-
Rights nology. One example is “canine augmentation
technology,” developed to foster more efficient
dog–human communication in search and rescue
operations (Ferworn et al. 2006). Digital technol-
Digital Games for Animals ogy also plays a major role in animal cognition
research, with great apes such as the male bonobo
Mikhail Fiadotau Kanzi being trained to use a touchscreen interface
Centre of Excellence in Media Innovation and to communicate with humans in a lexigram-based
Digital Culture (MEDIT), Tallinn University, language (Greenfield et al. 2008). More recently,
Tallinn, Estonia underwater acoustic touchscreens have been
developed to facilitate human–dolphin communi-
cation (Herzing 2016).
Definitions Digital games for animals use technology as a
vehicle for animals’ natural playfulness.
Digital games aimed at animal players are a grow-
ing area in both entertainment and research,
enabled particularly by the mass availability of Examples
touchscreen technology. These games rely on ani-
mals’ natural proclivity for play and can facilitate Whereas some digital games for animals are com-
playful interactions between species. mercial products targeting pet owners, others are
586 Digital Games for Animals

developed for research purposes or for use in can relieve stress and depression in home alone
specialized facilities such as animal rehabilitation dogs (Geurtsen 2014). Another issue digital play
centers and zoos. Of the former variety, most can address is animals’ sedentary lifestyle (Pons
games appear to target cats and feature a similar et al. 2014). One example is Feline Fun Park, a
mechanic: tracking and tapping an object moving game for cats which has an automatic mode where
on the touchscreen. Friskies Jitterbug and Cat the level of challenge is adjusted based on the
Alone are the better known of many such titles. animal’s activity level (Young et al. 2007).
A number of similar games, such as Jolly Dog, Digital play can also benefit human–animal
also exist for dogs. interactions. Some digital games, such as Canine
Research-based games and games used in spe- Amusement and Training, are designed to facili-
cialized settings often focus on enriching the lives tate dog training (Wingrave et al. 2010). Other
of captive animals, targeting species as diverse as digital games, such as the human–orangutan col-
parrots (Woodman 2014), penguins (Westerlaken laborative game deployed at Melbourne Zoo, aim
2017), and orangutans (Webber et al. 2017). Many to increase visitors’ empathy for animals while
such games promote cross-species play. One providing the latter with environmental enrich-
example is Pig Chase, an experimental game ment (Webber et al. 2017). Another project,
developed in the Netherlands, which involves Apps for Apes, included an iPad donation cam-
farm pigs interacting with a large touchscreen paign for captive orangutans as a way of raising
installed in their pen. The objective is for a pig public awareness about orangutan survival in the
to use its snout to touch a moving ball, which is wild (Smith 2011).
remotely controlled by a human player. When As an area of research, digital play in animals
touched, the ball fires off colorful sparks. The can be regarded as part of the wider domain
game keeps track of each pig’s individual perfor- of animal–computer interaction (Baskin and
mance and has a leaderboard which can be viewed Zamansky 2015) whose aims and concerns it
by the human on their tablet. The aim of the shares. These include improving animals’ life
project is to relieve the pigs’ boredom, which is expectancy and quality, assisting working animals
a major issue in intensive farming, as well as in their legal functions, deepening our understand-
facilitate cross-species play and, through it, ing of animal cognition, and fostering better
reduce “the distance between farming practices communication between species (Mancini 2011).
and the general public” (Meijer 2016: 71). Other Studying animals’ digital play may also help us
examples of cross-species digital play have better understand the psychology and the
involved orangutans (Wirman et al. 2011; Webber roles of play – including in our own species
et al. 2017), cats (Westerlaken and Gualeni 2014), (Wirman 2013).
dogs (Wingrave et al. 2010), and hamsters (Cheok
et al. 2011).
Challenges

Rationale Despite their potential, digital games for animals

present a number of challenges and concerns,
One of the main reasons to explore digitally medi- most notably of an ethical nature. Play is meant
ated play in animals is its potential to improve their to be a voluntary and autotelic (intrinsically moti-
lives. Digital games can provide cognitive stimula- vating) activity (Denzin 1975), yet this is not
tion and enrichment to captive animals, mitigating always the case with digital games for animals,
the issue of boredom and facilitating cognitive which in many settings are reward-based (Wirman
development (Baskin and Zamansky 2015). Exper- et al. 2011). It is thus often unclear whether the
imental evidence suggests playing digital games animal is genuinely playing or exhibiting trained
Digital Games for Animals 587

behavior (Baskin and Zamansky 2015). Some References

breeds of dogs display strong predatory patterns
when engaging with digital games, suggesting the Autier-Dérian, D., Deputte, B.L., Chalvet-Monfray, K.,
Coulon, M., Mounier, L.: Visual discrimination of spe-
animals may be trying to hunt, with the failure to
cies in dogs (Canis familiaris). Anim. Cogn. 16(4),
capture the prey resulting in frustration (Baskin 637–651 (2013). https://doi.org/10.1007/s10071-013-
et al. 2015). 0600-8
Another concern is the short- and long-term Baskin, S., Zamansky, A.: The player is chewing the tab-
let!: towards a systematic analysis of user behavior in
physiological and behavioral effects of digital
play in animals (Westerlaken and Gualeni 2014).
animal-computer interaction. In: Proceedings of the D
2015 Annual Symposium on Computer-Human Inter-
As many other aspects of animals’ technology action in Play, pp. 463–468. ACM (2015). doi:https://
use, these require further research. doi.org/10.1145/2793107.2810315
Baskin, S., Anavi-Goffer, S., Zamansky, A.: Serious
A more technical challenge lies in the fact that
games: is your user playing or hunting? In:
modern technology is overwhelmingly designed Chorianopoulos, K., Divitini, M., Baalsrud Hauge, J.,
for human use and is not easily accessible to other Jaccheri, L., Malaka, R. (eds.) Entertainment Comput-
animals. Most mammals, for example, have ing – ICEC 2015. Lecture Notes in Computer Science,
vol. 9353, pp. 475–481. Springer, Cham (2015). https://
dichromatic color vision, as opposed to the tri-
doi.org/10.1007/978-3-319-24589-8_43
chromatic vision in humans (Neitz et al. 1989), Bekoff, M.: The development of social interaction, play,
meaning that they are able to distinguish between and metacommunication in mammals: an ethological
a more limited range of colors. Avians such as perspective. Q. Rev. Biol. 47, 412–434 (1972)
Burghardt, G.M.: The Genesis of Animal Play: Testing the
parrots can recognize images on a flat-panel dis- Limits. MIT Press, Cambridge, MA (2005)
play but not a cathode-ray-tube screen, due to the Cat Alone: [digital game] Galbro, Inc. (2015). Available
latter’s flickering (Woodman 2014). Many ani- from: https://play.google.com/store/apps/details?
mals, including dogs and cats, also have a differ- id¼com.galbro.cataloneandroid&hl¼en
Cheok, A.D., Tan, R.T.K.C., Peiris, R.L., Fernando,
ent field of vision and depth perception compared O.N.N., Soon, J.T.K., Wijesena, I.J.P., Sen, J.Y.P.:
to humans (Pons et al. 2017), making standard Metazoa Ludens: mixed-reality interaction and play
tablet screens suboptimal for their use. Addition- for small pets and humans. IEEE Transactions on Sys-
ally, most touchscreens are neither durable tems, Man, and Cybernetics-Part A: Systems and
Humans. 41(5), 876–891 (2011). https://doi.org/10.
enough nor responsive to claws, rendering stan-
1109/TSMCA.2011.2108998
dard tablets nearly unsuitable for cat and dog Denzin, N.K.: Play, games and interaction: the contexts of
use (McGrath 2015), despite the proliferation childhood socialization. Sociol. Q. 16, 458–478 (1975)
of “feline” and “canine” games on these devices. Ferworn, A., Sadeghian, A., Barnum, K., Rahnama, H.,
Pham, H., Erickson, C., Ostrom, D., Dell’Agnese, L.:
Thus a crucial challenge when designing a Urban search and rescue with canine augmentation
digital game aiming at animals is to ensure it technology. In: System of Systems Engineering, 2006
uses “species appropriate” (McGrath 2009) tech- IEEE/SMC International Conference. IEEE (2006).
nology and undergoes sufficient user testing doi:https://doi.org/10.1109/SYSOSE.2006.1652317
Friskies Jitterbug: [digital game] Nestle Purina PetCare
with actual animals to mitigate the developers’ (2014). Available from: https://www.gamesforcats.com
inevitable anthropocentrism (Westerlaken and Geurtsen, A.: An experiment in animal welfare informat-
Gualeni 2014). ics: effects of digital interactive gameplay on the psy-
chological welfare of home alone dogs. Master
of Science Thesis, Leiden University (2014).
Available from: http://mediatechnology.leiden.edu/
Cross-References images/uploads/docs/geurtsen-an-experiment-in-
animal-welfare-informatics.pdf
▶ Cognitive Psychology Applied to User Experi- Greenfield, P.M., Lyn, H., Savage-Rumbaugh, E.S.: Proto-
language in ontogeny and phylogeny: combining deixis
ence in Video Games and representation. Interact. Stud. 9(1), 34–50 (2008.)
▶ Cybersickness Available from: http://greenfieldlab.psych.ucla.edu/
▶ Psychological Game Design Evolution_and_Primate_Studies_files/04gre.pdf
588 Digital Geometry Processing

Herzing, D.L.: Interfaces and keyboards for human- 2014.-Felino_The-Philosophical-Practice-of-Making-

dolphin communication: what have we learned. Anim. an-Interspecies-Videogame.-PCG2014.pdf
Behav. Cogn. 3(4), 243–254 (2016). https://doi.org/10. Wingrave, C.A., Rose, J., Langston, T., LaViola Jr, J.J.:
12966/abc.04.11.2016 Early explorations of CAT: canine amusement and
Hol, T., Van den Berg, C.L., Van Ree, J.M., Spruijt, B.M.: training. In: CHI’10 Extended Abstracts on Human
Isolation during the play period in infancy decreases Factors in Computing Systems. pp. 2661–2670. ACM
adult social interactions in rats. Behav. Brain Res. 100 (2010)
(1–2), 91–97 (1999). https://doi.org/10.1016/S0166- Wirman, H.: The playing other and what we cannot help
4328(98)00116-8 learning from the study of animal play. Presented at
Jolly Dog: [digital game] Crashinvaders. (2016). Available DIGRA 2013 conference, Atlanta (2013). Available
from: http://crashinvaders.com from: http://www.hannawirman.net/wirman_
Mancini, C.: Animal-computer interaction: a manifesto. digra2013.pdf
Interactions. 18(4), 69–73 (2011). https://doi.org/10. Wirman, H., Smits, W., Yu, G., Yuen, W.: Defeated by an
1145/1978822.1978836 orangutan? Approaching cross-species gameplay. Pre-
McGrath, R.E.: Species-appropriate computer mediated sented at DiGRA 2011 conference, Utrecht (2011).
interaction. In: CHI’09 Extended Abstracts on Human Available from: http://www.hannawirman.net/
Factors in Computing Systems, pp. 2529–2534. ACM wirman_digra2011_abstract.pdf
(2009). doi:https://doi.org/10.1145/1520340.1520357 Woodman, C.: Successful intuitive computer interfaces for
birds, and other forays into giving parrots electronic
McGrath, R.: Species-Inappropriate Touch Screens
enrichment. Presented at the International Association
[online], https://robertmcgrath.wordpress.com/2015/
of Avian Trainers and Educators Conference (2014).
09/12/species-inappropriate-touch-screens/ (2015)
Av a i l a b l e f r o m : h t t p : / / fl y i n g w i t h p e n c i l s .
Meijer, E.: Animal deliberation: from farm philosophy to runningwithpencils.com/ForaysintoGivingParrotsElec
playing with pigs. Krisis. 1, 71–73 (2016.) Available tronicEnrichment.pdf
from: http://krisis.eu/animal-deliberation-from-farm- Young, J.E., Young, N., Greenberg, S., Sharlin, E.: Feline
philosophy-to-playing-with-pigs/ fun park: a distributed tangible interface for pets and
Neitz, J., Geist, T., Jacobs, G.H.: Color vision in the dog. owners. In: Video and Adjunct Proceedings of the 5th
Vis. Neurosci. 3, 119–125 (1989) International Conference on Pervasive Computing,
Pons, P., Jaen, J., Catala, A.: Animal ludens: building Toronto (2007). Available from: http://hci.cs.
intelligent playful environments for animals. In: Pro- umanitoba.ca/publications/details/feline-fun-park-a-
ceedings of the 2014 Workshops on Advances in Com- distributed-tangible-interface-for-pets-and-owners
puter Entertainment Conference. ACM (2014). doi:
https://doi.org/10.1145/2693787.2693794
Pons, P., Jaen, J., Catala, A.: Towards future interactive
intelligent systems for animals: study and recognition
of embodied interactions. In: Proceedings of the 22nd
International Conference on Intelligent User Interfaces.
pp 389-400. ACM (2017). https://doi.org/10.1145/ Digital Geometry Processing
3025171.3025175
Smith, J.: Apps for apes: Orangutans want iPads for
Christmas. New Scientist. 212, 69–71 (2011). https:// ▶ Modeling and Mesh Processing for Games
doi.org/10.1016/S0262-4079(11)63173-4
Sutton-Smith, B.: The Ambiguity of Play. Harvard
University Press, Cambridge, MA (1997)
Webber, S., Carter, M., Sherwen, S., Smith, W., Joukhadar,
Z., Vetere, F.: Kinecting with Orangutans: zoo visitors’
empathetic responses to animals? Use of interactive
technology. In: Proceedings of the 2017 CHI Confer- Digital Humans
ence on Human Factors in Computing Systems.
pp. 6075–6088. ACM (2017). doi:https://doi.org/10. ▶ 3D Avatars in Virtual Reality Experience
1145/3025453.3025729
Westerlaken, M.: Uncivilizing the future: imagining non-
speciesism. Antaeus. 4(1), 53–67 (2017.) Available
from: https://muep.mau.se/handle/2043/24019
Westerlaken, M., Gualeni, S.: Felino: The philosophical
practice of making an interspecies videogame. Pre-
sented at the Philosophy of Computer Games Confer-
ence, Istanbul (2014). Available from: http://
Digital Image
gamephilosophy.org/download/philosophy_of_com
puter_games_conference_2014/Westerlaken_Gualeni- ▶ Rendering Equation
Digital Images Using Heuristic AdaBoost Haar Cascade Classifier Model 589

framework and made improvements to the origi-

Digital Images Using Heuristic nal framework keeping with the advancements of
AdaBoost Haar Cascade technology. The most recent application of the
Classifier Model, Detection of framework was via GPU architecture and combin-
Partially Occluded Faces ing Viola-Jones’ method with a neural network.
The contribution of Viola and Jones is only
Tulasii Sivaraja and Abdullah Bade surpassed by Lienhart and Maydt in 2002 when
Mathematics, Graphics and Visualization they extended the set of Haar-like features used in
Research Group (MGRAVS), Faculty of Science the framework. Their contribution of rotating the D
and Natural Resources, Universiti Malaysia Haar-like feature by 45 and also a center-
Sabah, Kota Kinabalu, Sabah, Malaysia surround feature provided the cascade with more
features to compute without increasing the com-
putation time (Lienhart and Maydt 2002).
Synonyms

Face detection; Face recognition; Haar cascade State-of-the-Art Work

classifier; Image processing; Occluded faces
Classification of Face Detection Techniques
Face detection approaches can be grouped into
Definition two distinct approaches: feature based and image
based. Feature-based approaches in face detection
Detection of partially occluded faces in digital requires the extraction of facial features from an
images using AdaBoost Haar cascade classifier image and comparing it with a knowledge base of
is a viable technique of face detection if the cas- face features, whereas image-based approaches
cade training procedure is modified. attempts to get the best match while comparing
training images against testing images (Modi and
Macwan 2014). Figure 1 illustrates the classifica-
Introduction tion of face detection algorithms.
The feature-based approaches consist of active
Face detection is one of the more popular applica- shape models (ASMs), low-level analysis, and
tions of object detection in computer vision. The feature analysis. Active shape models were devel-
computer uses a series of mathematical algorithms, oped by Tim Cootes and Chris Taylor in 1995,
pattern recognition, and image processing to iden- where a shape of an object in the form of a statis-
tify faces from an image or video input. Over the tical model is used to find the object in a new
years, the technology of detecting faces has evolved image by the means of iterative deformation to
proportional to its usage in various applications. fit the new object (Cootes et al. 2000). Low-level
The most known algorithm for face detection was analysis is a feature-based approach that uses
introduced by Viola and Jones in 2001. They pro- visual features such as edges, gray scale levels,
posed a framework that produces real-time face color information, motion, and general measures
detection by the means of a novel image represen- to differentiate facial features of a face with its
tation known as integral image and incorporated the surroundings in an image (Singh et al. 2017).
Haar basis functions that was used in the general Meanwhile, feature analysis refers to the usage
framework of object detection (Papageorgiou et al. of the geometrical facial features to locate faces
1998) with AdaBoost and implemented it in a cas- irrespective of illumination, pose, and angles.
cade structure, creating a boosted cascade of weak There are two methods of implementing feature
Haar-like feature classifiers (Viola and Jones 2001). analysis in face detection, which are feature
Since the introduction of the Viola-Jones searching, such as Viola-Jones method and con-
framework, various researchers have adapted the stellation method.
590 Digital Images Using Heuristic AdaBoost Haar Cascade Classifier Model

Digital Images Using

Heuristic AdaBoost Haar
Cascade Classifier
Model, Detection of
Partially Occluded Faces,
Fig. 1 Classification of
face detection algorithms

Meanwhile, image-based approaches aim to method, there are some modifications that have
improve the unpredictability of the various been done on the original algorithm that are worth
feature-based approaches. The strategy this mentioning. In 2002, the Viola-Jones method was
approach utilizes is treating face detection as a enhanced by using an extended set of Haar-like
pattern recognition problem, whereby the specific features which rotates the Haar-like features by
application of face knowledge is steered clear 45 that improved the false positive rates
from and the issue is tackled as learning to recog- (Lienhart and Maydt 2002). The initial Viola-
nize face patterns from examples (Hjelmås and Jones method has gone through various enhance-
Low 2001). The fundamental approach of ments and modification. Later, a new variant of
image-based methods relies on training proce- AdaBoost known as asymmetric AdaBoost was
dures that classify the samples as face and non- introduced (Viola and Jones 2002). Then another
face classes. The straightforward approach is form of boosting was introduced when FloatBoost
illustrated by template matching. Most of the was used in a detector-pyramid architecture to
methods rely on an exhaustive search done by a detect and recognize faces (Zhang et al. 2002).
window scanning algorithm in an attempt to In 2003, researchers introduced a boosting chain
detect faces. However, there are various methods to enhance the cascade model for object detection
under the image-based approach such as linear which was later used for face detection (Xiao et al.
subspace methods, neural networks, and statisti- 2003). A novel variant of Viola-Jones that uses a
cal approaches that are able to accomplish the task nested cascade structure for multi-view face
of face detection successfully. detection proved to be successful for detection of
Since this entry aims to improve face detection faces from various viewpoints (Huang et al.
of partially occluded faces using the Viola-Jones 2004). In 2005, a similar approach as Viola-
Digital Images Using Heuristic AdaBoost Haar Cascade Classifier Model 591

Jones was introduced. However, it used joint industry has only increased with the age of tech-
Haar-like features for classifiers (Mita et al. nology. Face detection now plays an important
2005). It became increasingly important to detect role in face recognition and biometric authentica-
faces that are non-frontal or multi-view faces in tion. Additionally, face detection is also exten-
images and video. A research put forth the idea of sively used in the software embedded in digital
using width-first search (WFS) tree structure to cameras. Most digital cameras and cameras on
improve the performance of speed and accuracy mobile phones use face detection to detect and
of the detection by implementing a vector focus on the faces to reduce blurriness. Mean-
boosting algorithm based on the real AdaBoost while, social networks and social applications D
algorithm, and a piecewise function with LUT was such as Facebook and Snapchat use face detection
used to aid with the weak classifiers (Huang to further enhance user experience (Rajawat et al.
et al. 2005). 2017). These application and usage of face detec-
Other hybridization of the Viola-Jones method tion further motivates researchers to optimize and
includes using color information whereby skin address the issues faced in face detection as this
detection is used by computing the skin percent- directly impacts the industrial application of face
age (Mahmoud et al. 2011). Another popular var- detection.
iation is to combine the Viola-Jones method with a
neural network, where the neural network is used
to classify faces with non-faces processing stage Open Issues in Face Detection
(Da'san et al. 2015). Additionally, the Viola-Jones
method can be integrated with other known The challenges encountered in the field of face
methods such as shape constraints (Cristinacce detection and recognition in digital images are
and Cootes 2003), low-level saliency (Cerf et al. commonly expressed as A-PIE, which represents
2008), and with corner points using Shi-Tomasi aging, poses, illumination, and expression
detector (El Kaddouhi et al. 2017). Recent devel- (Mahalingam et al. 2014). Humans are able to
opment of the Viola-Jones detector involves identify faces almost instantaneously despite the
implementation via graphics processing unit changes that occur due to aging; however, com-
(GPU), multi-threaded central processing unit puters do not have this ability. The difficulty
(CPU), and field-programmable gate array encountered due to aging is still being regarded
(FPGA). The proposed usage of FPGA hardware as a problem as the physical changes of a person
architecture design with AdaBoost face training over different periods of time can be subjected to
and detecting algorithm improved the perfor- external factors such as injuries and cosmetic sur-
mance of the face detection algorithm (Lai et al. geries. In terms of poses, the degree of head rota-
2007). In 2011, researchers proposed a technique tion will affect the amount of facial features
that uses GPU computing to implement a modi- available for detection. Illumination also plays a
fied Viola-Jones framework (Devrari and Kumar part in the detection of faces. If minimum lighting
2011). Moreover, the study done to compare the is available in the image, then it becomes a chal-
implementation of the Viola-Jones framework in a lenge to differentiate the change in pixel value
single-threaded CPU, a multi-threaded CPU, and which is required to detect features of a face.
a GPU implementation using CUDA shows the Finally, in terms of emotion, the changes in
GPU implementation to be the fastest (Krpec and human expression would not be recognized easily
Němec 2012). if the face detection model is not trained to recog-
nize a variation of emotion, as the general struc-
ture of the face changes when a person
Application of Face Detection experiences different emotions. Additionally, par-
tial occlusion poses a problem in face detection as
Over the years, the popularity of face detection well. Some of the A-PIE challenges can also be
has yet to cease. The usage of face detection in the considered as a partial occlusion problem as well.
592 Digital Images Using Heuristic AdaBoost Haar Cascade Classifier Model

Researchers began noticing that partial occlusion boosting parameters will affect the detection rates
of faces affects the overall detection rate of the of partially occluded faces in images. The integra-
algorithm. Most of the face detection techniques tion of optimal tree depth value and boosting
work by searching facial features such as mouth, parameter values in the cascade training will create
nose, and a set of eyes in images in a joint search. a heuristically boosted cascade model. The heuris-
Therefore, occlusions such as sunglasses and tically boosted cascade classifier model is then
scarves hinder the detection of faces which con- used in the architecture of the any feature-based
sequently affects the performance of the detection approach of face detection in order to detect par-
algorithms. tially occluded faces in images.
The detection of faces in images are made In order to achieve the best results of face detec-
difficult with the limited visibility of faces due to tion, the input image has to go through some pre-
various factors such as pose variation, poor light- processing steps, which are resizing and converting
ing, and occlusions due to hats, scarfs, hair, and the color space from RGB to grayscale. These two
foreign objects. To accurately detect these par- steps are taken to ensure that the best experimental
tially occluded faces in images without using arti- results are obtained for the detection of partially
ficial intelligence (AI) requires an approach that occluded faces in images. Since grayscale images
does not sacrifice too much in terms of computa- are used to train the cascade, therefore, when exe-
tion time and false positive and negative rates. cuting the face detection algorithm, it is best to use
Despite all these problems, the Haar cascade grayscale images. Furthermore, the computation
model can be used to improve the detection rates time of a grayscale image is much faster compared
of partially occluded faces with proper calibration. to an image that uses three channels. The dimen-
If the model is trained with sufficient images that sions of width and height used in the training of the
has a variation of occlusion and A-PIE characteris- model are usually quite small. Hence, to accom-
tics, the model will be able to detect faces in said modate this, the input image has to be resized to be
problematic conditions. Additionally, the sensitiv- twice as small as the original image. Then the
ity of the Haar cascade classifier model training trained cascade model is used to detect faces in
also influences the rates of detection and false the image. The detection process involves a sub-
positives that would be given in the face detection. window sliding over the preprocessed input image
in various scales using the cascade model to eval-
uate the probability of the sub-window containing
Heuristic Haar Cascade Classifier Model a face. In the output, the detected faces or objects
in the image have rectangles drawn over them.
A heuristic Haar cascade classifier can be created Figure 2 illustrates the flow of training the heuristic
by setting different training parameters that within cascade classifier model and the detection of faces
the threshold of the Haar cascade classifier model in images.
from Lienhart and Maydt. The parameters that The effectiveness of the heuristic cascade clas-
influence the detection rates of faces are the depth sifier model can be tested using ROC curves,
of the decision tree, minHitRate and maxFalseA- confusion matrix, and comparison tests. The
larmRate. These parameters determine the number ROC curve is used to analyze the accuracy of
of features or weak classifiers selected in a partic- the model whereby it can be inferred that the
ular stage, in addition to the performance of the closer the curve is to the top left corner of the
cascade in terms of detection rates. Moreover, graph, the more accurate the cascade model. From
minHitRate and maxFalseAlarmRate also influ- the confusion matrix, the F1 score can be calcu-
ence the value of the acceptance ratio break value. lated. This score is the ratio between accuracy and
Once the acceptance ratio break value is achieved, precision. The Haar cascade classifier models
the heuristic cascade model is created; otherwise, a tested for the detection of partially occluded
new stage of cascade is added when the faces needs to have a balanced result between
maxFalseAlarmRate is reached. The value of the accuracy and precision due to the nature of the
Digital Images Using Heuristic AdaBoost Haar Cascade Classifier Model 593

GENERATION OF HEURISTIC CASCADE CLASSIFIER MODEL

set parameter values:

INPUT OUTPUT
• depth of CART tree
Training • minHitRate heuristic cascade classifier
Images maxFalseAlarmRate model
• acceptance Ratio D
break value

FACE DETECTION METHOD

INPUT INPUT

Input images heuristic cascade classifier

model

resize images

convert image to Process of detecting

grayscale faces in images

OUTPUT

Image with rectangle over

detected faces

Digital Images Using Heuristic AdaBoost Haar Cascade Classifier Model, Detection of Partially Occluded
Faces, Fig. 2 Method of Implementation

features available. For the detection of partially Figure 3 shows the result of three sample
occluded faces in images, one of the factors that images being tested with three different Haar cas-
would influence the efficiency of the classifier cade classifier models which are Haar_frontalfa-
model is the depth of the CART tree. The ideal ce_default, Haar_frontal_face_alt, which are
tree depth would ensure that sufficient feature classifier models by Lienhart and Maydt, and a
information is preserved in to enable quick detec- heuristic Haar cascade classifier model trained to
tion without underfitting the CART tree. How- detect partially occluded faces.
ever, having a high value of tree depth would not From Fig. 3, it can be observed that the
be ideal in the case of detecting partially occluded heuristic cascade works better in detecting par-
faces in images as a lot of facial features would be tially occluded faces where it was able to rec-
discarded due to overfitting the tree. Additionally, ognize facial features under less favorable
the detection rates of a CART tree with a low tree conditions.
depth will end up taking too much time as it would Table 1 shows the confusion matrix analysis
have to loop through more feature information preformed to test the accuracy and sensitivity of
from the tree. the cascade classifier models.
594 Digital Images Using Heuristic AdaBoost Haar Cascade Classifier Model

Digital Images Using Heuristic AdaBoost Haar Cas- Haar cascade classifier model; (b) Haar_frontalfa-
cade Classifier Model, Detection of Partially ce_default; (c) Haar_frontal_face_alt
Occluded Faces, Fig. 3 Sample images: (a) heuristic

Digital Images Using Heuristic AdaBoost Haar Cascade Classifier Model, Detection of Partially Occluded
Faces, Table 1 Confusion matrix analysis
Heuristic cascade Haar_frontalface_default Haar_frontalface_alt
Accuracy 0.6525 0.5290 0.4843
Recall 0.6945 0.5336 0.5199
Precision 0.8683 0.9839 0.8759
F1 score 0.7717 0.6919 0.6525

The heuristic cascade classifier model is ratio of sensitivity and specificity of the heuristic
trained with using images that have various levels cascade classifier model is 10.9% and 16.7% bet-
and types of occlusion. Additionally, the training ter compared to the other two cascade classifier
parameter values are calibrated to enhance the models Haar_frontalface_default and Haar_fron-
detection of partially occluded faces. The accu- talface_alt, respectively. This means that the heu-
racy and true positive rate or recall of the heuristic ristic cascade classifier model is superior in
cascade classifier model is much higher compared detecting partially occluded faces when compared
to both the other cascade models. However, the to the other two cascade classifier models pro-
precision of Haar_frontalface_default and vided by Lienhart and Maydt.
Haar_frontalface_alt is much better than the heu- Additionally, by visually comparing the detec-
ristic cascade classifier model. Despite this, the tion of each cascade model, it is observed that the
Digital Images Using Heuristic AdaBoost Haar Cascade Classifier Model 595

heuristic cascade classifier model performs better El Kaddouhi, S., Saaidi, A., Abarkan, M.: Eye detection
than the other two classifier models in detecting based on the Viola-Jones method and corners points.
Multimed. Tools Appl. 76, 23077–23097 (2017)
partially occluded faces in digital images. The heu- Hjelmås, E., Low, B.: Face detection: a survey. Comput.
ristic cascade classifier model outperforms the Vis. Image Underst. 83, 236–274 (2001)
Haar_frontalface_default and Haar_frontal_face_alt Huang, C., Al, H., Wu, B., Lao, S.: Boosting nested cas-
classifier models contributed by Lienhart and Maydt cade detector for multi-view face detection. In: Pro-
by 23.66% and 21.7%, respectively, in detecting ceedings of the 17th International Conference on
Pattern Recognition, 2004. ICPR 2004, Vol. 2,
partially occluded faces in images. pp. 415–418 (2004)
D
Huang, C., Ai, H., Li, Y., Lao, S.: Vector boosting for
rotation invariant multi-view face detection. In: Tenth
IEEE International Conference on Computer Vision
Conclusion (ICCV’05), Vol. 1, pp. 446–453 (2005)
Krpec, J., Němec, M.: Face detection CUDA accelerating.
A heuristically trained Haar cascade classifier In: ACHI 2012, The Fifth International Conference on
Advances in Computer-Human Interactions,
model can be used in order to detect partially pp. 155–160 (2012)
occluded faces in digital images. By training the Lai, H., Savvides, M., Chen, T.: Proposed FPGA hardware
Haar cascade classifier model with various levels architecture for high frame rate (100 fps) face detec-
and types of occlusion, it increases the probability tion using feature cascade classifiers. In: 2007 First
IEEE International Conference on Biometrics: Theory,
of detecting partially occluded faces in images. Applications, and Systems (2007)
During the training phase, several training param- Lienhart, R., Maydt, J.: An extended set of Haar-like
eters can be heuristically determined in order to features for rapid object detection. In: Proceedings.
further enhance the detection of partially occluded International Conference on Image Processing
(2002)
faces in images.
Mahalingam, G., Ricanek, K., Albert, A.: Investigating the
Periocular-based face recognition across gender trans-
formation. IEEE Trans. Inf. For. Secur. 9, 2180–2192
Cross-References (2014)
Mahmoud, T., Abdel-latef, B., Abd-El-Hafeez, T., Omar,
A.: An effective hybrid method for face detection. In:
▶ Facial Recognition and Emotion Detection in Proceedings of the Fifth International Conference on
Environmental Installation and Social Media Intelligent Computing and Information Systems, Cairo,
Applications pp. 263–268 (2011)
Mita, T., Kaneko, T., Hori, O.: Joint Haar-like features for
face detection. In: Tenth IEEE International Conference
on Computer Vision (ICCV’05), Vol. 1, pp. 1619–1626
References (2005)
Modi, M., Macwan, F.: Face detection approaches: a sur-
Cerf, M., Harel, J., Einhäuser, W., Koch, C.: Predicting vey. Int. J. Innov. Res. Sci. Eng. Technol. 3,
human gaze using low-level saliency combined with 11107–11116 (2014)
face detection. Adv. Neural Inf. Proces. Syst. 20, Papageorgiou, C., Oren, M., Poggio, T.: A general frame-
241–248 (2008) work for object detection. In: Sixth International Con-
Cootes, T., Baldock, E.R., Graham, J.: An introduction to ference on Computer Vision (IEEE Cat.
active shape models. In: Image Processing and Analy- No.98CH36271), Vol. 1 (1998)
sis, pp. 223–248. Oxford University Press (2000) Rajawat, A., Pandey, M., Rajput, S.: Low resolution face
Cristinacce, D., Cootes, T.: Facial feature detection using recognition techniques: A survey. In: 2017 3rd Interna-
AdaBoost with shape constraints. In: Proceedings of tional Conference on Computational Intelligence &
the British Machine Vision Conference 2003 (2003) Communication Technology (CICT) (2017)
Da'san, M., Alqudah, A., Debeir, O.: Face detection using Singh, N., Daniel, A., Chaturvedi, P.: Template matching
Viola and Jones method and neural networks. In: 2015 for detection & recognition of frontal view of human
International Conference on Information and Commu- face through Matlab. In: 2017 International Conference
nication Technology Research (ICTRC), pp. 40–43 on Information Communication and Embedded Sys-
(2015) tems (ICICES) (2017)
Devrari, K., Kumar, K.: Fast face detection using graphics Viola, P., Jones, M.: Rapid object detection using a boosted
processor. Int. J. Comput. Sci. Inf. Technol. 2, cascade of simple features. In: Proceedings of the 2001
1082–1086 (2011) IEEE Computer Society Conference on Computer
596 Digital Preservation

Vision and Pattern Recognition. CVPR 2001, Vol.

1 (2001) Digitally Encoded Surfaces
Viola, P., Jones, M.: Fast and robust classification using
asymmetric adaboost and a detector cascade. In:
Advances in Neural Information Processing Systems, ▶ Tangible Surface-Based Interactions
pp. 1311–1318 (2002)
Xiao, R., Zhu, L., Zhang, H.: Boosting chain learning for
object detection. In: Proceedings Ninth IEEE Interna-
tional Conference on Computer Vision, Vol. 1,
pp. 709–715 (2003) Direct Digital Manufacturing
Zhang, Z., Zhu, L., Li, S., Zhang, H.: Real-time multi-view
face detection. In: Proceedings of Fifth IEEE Interna- ▶ 3D Printing, History of
tional Conference on Automatic Face Gesture Recog-
nition, pp. 149–154 (2002)

Direction

Digital Preservation ▶ Game Development Leadership Tips

▶ 3D Game Asset Generation of Historical Archi-

tecture Through Photogrammetry
Disney Toontown Online, a
Massively Multiplayer Online
Role-Playing Game
Digital Process Newton Lee
Institute for Education, Research, and
▶ Pipeline of 2D Vector Animation in Television Scholarships, Los Angeles, CA, USA
Series
Vincennes University, Vincennes, IN, USA

Synonyms
Digital Production
Action adventure game; MMORPG; Multiplayer
▶ Pipeline of 2D Vector Animation in Television game
Series

Definitions

MMORPG ¼ Massive multiplayer online role-

Digital Sports playing game is a multiplayer game designed to
be played online simultaneously by a large num-
▶ Diversity in Gaming and the Metaverse
ber of players.
Multiplayer game ¼ a game that is designed
for multiplayer mode where two or more players
are expected throughout the entire gameplay.
Digital Storytelling Action adventure game ¼ a game that com-
bines core elements from both action game and
▶ Narrative Design adventure game genres.
Disney Toontown Online, a Massively Multiplayer Online Role-Playing Game 597

Introduction The game was designed mainly for children (7 and

up), but adults also enjoyed Toontown’s nostalgic
Disney Toontown Online was a massively multi- elements, humor, and game play.
player online role-playing game (MMORPG)
developed by The Walt Disney Company (Madej
and Lee 2020). It was first released in 2003 and it Toontown’s Backstory
allowed players to create their own cartoon ava-
tars called “Toons” and explore a virtual world Toontown’s story begins on the outskirts of the
filled with various activities, quests, and place cartoon characters call “home.” On this day, D
challenges. world-famous billionaire Scrooge McDuck
In Toontown Online, players could battle decides to pay a visit to his favorite employee,
against comical villains called the “Cogs” who eccentric inventor Gyro Gearloose.
were trying to take over the world of Toontown. Gyro has posted a warning sign on his labora-
Players could team up with other Toons to defeat tory door that says “KEEP OUT.” Scrooge
the Cogs using a variety of gag weapons such as ignores it, opens the door, and calls out “Gyro?”
cream pies, squirt guns, and anvils. He walks pass a vast array of laboratory equip-
Toontown Online was popular among both ment, test tubes, and microscopes. Gyro is
children and adults due to its colorful graphics, nowhere to be found. “I wonder where Gyro
fun gameplay, and social features. It was known could be?” thinks Scrooge. In the back of Gyro’s
for its focus on teamwork and collaboration, laboratory, Scrooge is stunned by what he sees
encouraging players to work together to complete and cries out, “Sufferin’ catfish! A g-g-giant
quests and defeat the Cogs. The game was shut robot!”
down in 2013 after a successful ten-year run, but it Scrooge then thinks to himself, “Hmm. . .
has since been revived by a group of fans who A giant robot. . . Perfect. . . Why, a thing like this
have created their own version called Toontown could be a really big help to the citizens of
Rewritten. Toontown. . . And make me a big pile of money!”
However, Gyro has left a huge note on the
robot that says “DO NOT TOUCH!” Scrooge is
A Different MMORPG upset. “Do not touch?” he thinks, “Phooey! That
can’t mean me! After all, I paid for it. . .. Let’s get
Toontown Online is based not in the traditional my investment up and running!”
quest, fantasy, or science fiction themes of most Scrooge proceeds to connect the broken
MMORPGs. Instead, Toontown is a whimsical wires – the blue one to the red one. The giant
virtual world in which Toons, cartoon characters robot instantly comes to life!
of the 1940s variety, inhabit six neighborhoods. “All systems ready!” says the giant robot in a
The neighborhoods are happy, colorful places and deep voice. Scrooge begins to worry, “Great
each is associated with a classic Disney character. heavens to Betsy! What have I done?”
As an element of conflict in the game, mean Cogs, His fears are fulfilled as the giant robot stomps
out-of-control business robots, have come to town toward the control panel, pushes the red buttons,
and are ruining this happy environment by mak- pulls the lever, and manufactures hundreds upon
ing it a black and white city of skyscrapers and hundreds of evil robots called Cogs. They come in
businesses. The game enlists players to become various sizes, shapes, and abilities to inflict evils
Toons to save the town. Toons talk with each other in Toontown. Scrooge panics, “Oh my heavenly
through word balloons and take their world back days! This can’t be good. . .”.
by using traditional cartoon practical jokes and The story ends with Gyro’s invention running
slapstick comedy such as cream pies, seltzer bot- amok. Scrooge has placed himself and all of
tles, and whistle blowers against the Cogs, who Toontown in danger. Can anyone stop this army
have no sense of humor and cannot take a joke. of robots? “Toontown needs your help now!”
598 Disney Toontown Online, a Massively Multiplayer Online Role-Playing Game

Becoming a Toon phrases that automatically adapt to the player’s

objectives and status in the game. The set menus
To save Toontown from the evil Cogs, the player eliminate game jargon that may not be under-
has to become a Toon. Each account holder can standable by new players and helps novice typists
create up to six Toons in the virtual world. A menu who may have difficulty with grammar and spell-
of characteristics such as height, weight, color, ing. A particular feature is that the phrases are
and fashion style are available to make each friendly and encourage engagement and coopera-
Toon an individual character that reflects a tion between players.
player’s preferences. SpeedChat is sufficient to convey a player’s
Naming characters in games has often been feelings (e.g., happy or sad) and simple thoughts
abused, and it is almost impossible to create a (e.g., follow me, let’s play a game) that are enough
filter that would prevent players from using inap- to play the Toontown game without hindrance.
propriate names that might be offensive to some The friendly nature of the word sets helped to
of the audience. Instead of relying on an auto- overcome shy players’ inhibitions. Overall, the
mated “naughty list” filter, Toontown Online lack of open chat helped the players focus more
employs a name generator that gives a player on the gaming tasks at hand rather than spending
millions of possible combinations based on a time on chitchatting and digression.
choice of title, first name, and last name from a Open chat is available in Toontown between
suggested list. The generated list provides names “true friends.” True Friends are those who are
that are in keeping with the humorous nature of friends in actual life and may want to chat more
Toontown. To avoid the problem of inappropriate extensively when playing online. To activate open
names, players are offered a list of titles and first chat between two Toons, a randomly generated
and last names from which to choose their avatar’s secret password that is valid for only 48 h is given
name. If the player chooses, they may enter their to each of them. Friends share their secret pass-
own name, which goes through client and server- word over the phone, via email, or instant mes-
side filters and is reviewed by a Disney customer saging; each friend must then enter the correct
representative before acceptance. password in Toontown before it expires. True
Once a player has made and named her/his own Friend chat is filtered for inappropriate language
Toon, they are ready to enter Toontown and meet and the chat is incomprehensible to other Toons.
and play with other Toons. In keeping with the In addition to making chatting safe as well as
positive, upbeat tone of the site, if Toons join on a friendly and engaging, Toontown offers many
special day such as New Year’s Day, they are shared activities to help with making friends and
greeted in the playground with a display of becoming a part of the community: there are mini
fireworks. games to play, opportunities to go fishing, and
ongoing Cog battles to join. Socializing is encour-
aged by the simple way in which newbies (newer
Safe and Friendly Socializing players) can join play: They could hop on a trolley
car and meet other players going to a game or
Toontown encourages socializing and incorpo- simply walk up to a battle that may be in progress
rated in the game are features that facilitate social- and be instantly included without any chat. Elders
ization that is friendly, safe, and fun. (more experienced players) are rewarded when
Communicating through chat is a necessity in a they help out a newbie in battle against the
multiplayer game; in most MMORPGs it is Cogs; because the rewards received in a game
unrestricted. To protect kid’s privacy and ensure are proportional to the contribution made, elders
online safety, Toontown Online features are not penalized in battle when a newbie joins.
SpeedChat, a menu-based chat system that pro- This positive approach facilitates newer players’
vided safe player-to-player text interaction. Kids inclusion in the gaming community.
choose what they want to say from an extensive Other features were included to make
set of preselected context-sensitive words and Toontown a friendly environment: Portable holes
Disney Toontown Online, a Massively Multiplayer Online Role-Playing Game 599

allowed players to teleport immediately to wher- announced in The Toontown Online Times
ever their friends are located, there is no player-to- daily newspaper.
player battling or any opportunity to steal or hoard
items, and, instead of getting hurt or dying, Toons
become sad and go back to the playground to Toontown Innovation and Panda3D
become happy again.
Toontown Online was designed as “a safe social
game that was simple to learn yet challenging to
Collecting and Cooperating to Save master” (Mine et al. 2003). When released, D
Toontown Toontown proved to be innovative in many ways:

Toontown Online offers many mini games for the • It provided a safe online environment for chil-
players to win the “jelly beans” necessary to make dren thanks to the Toon name generator,
purchases of cartoon weapons to fight the evil SpeedChat, and True Friends.
robots. Jellybeans are the main components in • It encouraged both competitiveness (in the
the game’s system of barter. mini games) and cooperation (in fighting
Some of the popular mini games are maze, tag, against evil). There is no “winner takes all”
memory, treasure dive, slingshot, ring, cannon, mentality even in a competitive mini game.
jungle vine, and tug-o-war. Most of these games • It minimized violence in the gameplay through
are action adventure in nature to keep the adrena- the use of cartoon slapstick weapons such as
line flowing and maintain a quicker pace in the cream pies and seltzer bottles instead of guns
overall mood of Toontown. Although most of the and swords, and by using robots that are
mini games are competitive, there is no “winner mechanical rather than humans. Because
takes all” mentality. If the player exerts some Toons never die, but rather become sad, issues
amount of effort, chances are he/she will win of physical pain, blood, and death didn’t arise
some jellybeans, although perhaps not as many as they do in adult MMORPGs.
as the winner. For newbies there is a series of
training tasks to follow to learn how to win Toontown Online was built on an open-source
jellybeans, explore the virtual world, and destroy cross-platform Panda3D game engine created by
the evil robots. Toons battled Cogs with coopera- The Walt Disney Company. Panda stands for Plat-
tion and comic rather than violent action. form Agnostic Networked Display Architecture
Cogs have different names such as Pencil which provides the portability and flexibility for
Pusher, or Tightwad, according to their different Toontown to run on any operating system. Built
abilities, and they are classified into different on an efficient low-level C++ engine, Panda3D
levels according to their power. Regardless of expressive scene graph architecture gave
their abilities, high-level Cogs are more danger- designers tools for creating diverse and dynamic
ous than low-level robots. It often takes simulta- worlds for Toons to inhabit while its interpreted
neous actions from a team of Toons to destroy a scripting language gave programmers flexibility
high-level Cog. When fighting against the evil to prototype software rapidly and debug game
robots or Cogs, Toons need to cooperate with logic quickly.
each other.
Once all the Cogs have been evicted from a
building and eliminated, the ugly grey building Public Reception
is magically transformed into a happy place
with bright vibrant colors. The team of Toons Although Toontown Online was designed as a
that accomplished this task is rewarded by hav- nonviolent game alternative for kids aged 7–12,
ing each player’s Toon name displayed on the adults enjoyed the game as much as kids did,
wall of fame inside the building. In addition, making Toontown Online family entertainment
the top Toons and their accomplishments are similar to a Disney theme park where kids and
600 Disorientation

parents have fun together. The monthly subscrip-

tion model with reduced rates for longer member- Distal Pointing
ships proved acceptable to the audience and
profitable for the company. A basic level of play ▶ Raycasting in Virtual Reality
is available free on the site so anyone interested ▶ Virtual Pointing Metaphor in Virtual Reality
can make a Toon avatar, enjoy walking about, and
play a mini game or two to see if they like the
environment. In addition to the online subscrip-
tion plans, prepaid subscription cards (one or
more months) were made available at retail outlets Distance Matching
to allow sampling of premium content.
Toontown Online quickly gained enormous ▶ Motion Matching: Data-Driven Character Ani-
popularity with a high retention rate after the mation Systems
initial free three-day trial. In mid-2005, Toontown
had nearly 10,000 people playing simultaneously
during the busiest times. In May 2007, indepen-
dent Internet research firm Comscore estimated
Distance Misperception
Toontown Online had nearly 1.2 million users.
Disney also produced versions of Toontown
▶ Perceptual Illusions and Distortions in Virtual
Online for the UK, France, Spain, Japan, South-
Reality
east Asia, and Brazil.

Cross-References
Distance Underestimation
▶ Online gaming scalability
▶ Panda3D ▶ Spatial Perception in Virtual Environments

References

Madej, K., Lee, N.: Disney Stories: Getting to Digital, Distant Object
2nd edn. Springer Nature, Switzerland (2020)
Mine, M.R., Shocket, J., Hughston, R.: Building a mas- ▶ 3D Selection Techniques for Distant Object
sively multiplayer game for the millions: Disney’s
Toontown online. ACM Comput. Entertain. 1(1), Arti- Interaction in Augmented Reality
cle 06 (2003)

Distributed Architectures for

Disorientation Online Multiplayer Games

▶ Spatial Perception in Virtual Environments ▶ Online Gaming Architectures

Display Holography Distributed Multiplayer Game

▶ Holography as an Architectural Decoration ▶ Distributed Simulation and Games

Distributed Simulation and Games 601

event queue. Game engines run a game loop, which

Distributed Simulation and dictates the minimum time step between two
Games sequential events. Typically, game optimizations
make sure that only events related to objects
Jouni Ikonen which need to be inspected at that time are
Software engineering, Lappeenranta University of inspected and they can be also prioritized based
Technology, Lappeenranta, Finland on the importance of the object (e.g., visible to
player, affect game play fairness, and correctness).
In practice that can mean that less important events D
Synonyms are placed to an event queue further apart in time.

Distributed multiplayer game, Simulation,

Synchronization Game Simulation in Distributed
Environment

Definition Sharing of a game environment between mul-

tiple players with their own game devices
Distributed game simulation is a single simula- adds another dimension. There is usually some
tion, which has been distributed over multiple uncertainty between the game (simulation) states
processors and interconnected via a network. in each device, which is caused by the message
The game entities are synchronized in a manner, exchange latency. The impact of this latency
which try to ensure that the game is playable and increases with the demand of more frequent
fair, despite network latencies. game state updates. For example, in an ice hockey
game, each player plays the game in their own
computer, which is modelling the game simula-
Introduction tion, and controls one of the ice hockey players.
The game state in each computer should not differ
Many computer games present dynamically mod- too much to make the play experience acceptable.
elled game environments, which are dynamic in the In distributed simulation, there are two ap-
sense that they change over time. Changes to the proaches for managing uncertainty and ensur-
environment are typically initiated by events, ing correctness of the simulation: conservative
which can be created by users (e.g., players) or (Chandy and Misra 1979) and optimistic
other events. Simulation in a computer game uses (Jefferson 1985) approaches. In a conservative
discrete time, as the state of the environment is approach, each simulation agent (or game client)
inspected in discrete points in time. This is called reports how far in time they have progressed.
discrete event simulation. Inspection of the states They can only progress within some preselected
does not necessarily have to be inspected at even time boundaries of each other. Conservative ap-
intervals, but when something occurs that can proaches are not generally used in action games
change a state in the system. Events are placed in because they are limited by the speed of the
an event queue and ordered by the timestamp of slowest client (or the longest latency). In optimis-
their scheduled execution. An example can be a tic approaches, each client can simulate the game
simulated airplane, which is scheduled to land at as fast as they can, but if an error is detected they
time t, for which an event landing is added to the have to do a rollback, i.e., they have to undo
event queue. Time t will be the next point of time things that went wrong in the simulation, and
when the airplane has to be inspected again. How- correct game states, which can cause more roll-
ever, if something happens in the system that can backs. These synchronization problems can be
affect the plane before landing, then an earlier seen, e.g., as an opponent avatar changing posi-
placed event might have to be removed from the tion suddenly. Game interfaces try to hide these
602 Distributed Virtual Environment, DVE

synchronization errors, e.g., by using start move- ▶ Virtual World, a Definition Incorporating Dis-
ment animations, which can lead to failed action tributed Computing and Instances
as needed.
As the network is one of the critical resources,
it is important to try to limit the frequency of References
synchronizations and the volume of exchanged
messages. Methods like dead reckoning (Pantel Aldridge, D.: I shot you first: Networking the gameplay of
HALO: REACH. Game Developers Conference (2011)
and Wolf 2002), in which the next position of
Bettner, P., Terrano, M.: 1500 archers on a 28.8: Network
an object is estimated from current movement, programming in Age of Empires and beyond. In The
are used to reduce needed update frequency. 2001 Game Developer Conference Proceedings, San
Analysis of synchronization problems with their Jose, CA (2001)
Chandy, K.M., Misra, J.: Distributed simulation: a case
solutions are available for the games Halo: Reach
study in design and verification of distributed pro-
(Aldridge 2011) and Age of Empires (Bettner and grams. IEEE Trans. Softw. Eng. SE-5, 440–452
Terrano 2001). (1979). https://doi.org/10.1109/TSE.1979.230182
Extensive work has been done in standard- IEEE Std 1278.1: IEEE 1278.1–2012 - Standard for Dis-
tributed Interactive Simulation - Application protocols
ization to connect distributed simulators, espe-
(2012)
cially related to serious gaming. High-level IEEE Std 1516: IEEE Std 1516–2010 - IEEE Standard
architecture (HLA) (IEEE Std 1516 2010) is an for Modeling and Simulation (M&S) High Level
interoperability standard to integrate a number of Architecture (HLA)– Framework and Rules (2010)
Jefferson, D.R.: Virtual time. ACM Trans. Program.
separate simulations into one and used, for exam-
Lang. Syst. 7, 404–425 (1985). https://doi.org/10.
ple, in military war games. An alternative standard 1145/3916.3988
is Distributed Interactive Simulation (DIS) (IEEE Pantel, L., Wolf, L.C.: On the suitability of dead reckoning
Std 1278.1 2012). schemes for games. In: 1st Workshop on Network and
System Support for Games, pp. 79–84. ACM Press,
New York (2002)

Conclusion and Discussion

Impact of the network is an aspect that should

be considered from the beginning of a multiplayer Distributed Virtual
game design. In many cases in the early 1990s, Environment, DVE
game developers produced games, which were
later decided to include Internet multiplayer pos- ▶ Peer-to-Peer Gaming
sibility. The complexity of adding the multiplayer
option turned out to be more challenging than it
was pre-seen and typically took considerably
more time to get working than was expected. Diversity in Gaming and the
These problems have not disappeared today, and Metaverse
game designers have to consider how the synchro-
nization between the game simulations is done in Stefania Bau and Rachel Power
sufficient way to ensure good user experience and Init Esports, Lewes, DE, USA
scalability.

Synonyms
Cross-References
Cross-culturalism; Cultural diversity; Cyber-
▶ Secure Gaming: Cheat-Resistant Protocols and space; Digital sports; Electronic sports; Esports;
Game History Validation Ethnic inclusiveness; Multiculturalism; Multi-
▶ Serious Games verse; Virtual reality
Diversity in Gaming and the Metaverse 603

Definition diverse populations are more connected than ever

before. People interact globally through many
The metaverse is a digital reality that combines different platforms including social media, com-
aspects of social media, online gaming, aug- munication applications, and online gaming
mented reality (AR), virtual reality (VR), and (esports). With a more integrated society, cultural
cryptocurrencies to allow users to interact virtu- differences come to the surface which allows peo-
ally (Folger 2021). ple to learn about different traditions, values, and
Gaming or Esports are competitive digital tour- belief systems. These tools that connect the world
naments (sports) held online, in real life, and also can bring people together based on a common D
in the metaverse. interest, and this is a powerful way to build brid-
Diversity (Diversity and Inclusion Definitions) ges and discourage prejudice.
is the range of human differences, including Ideally, everyone can play in this virtual world
but not limited to race, ethnicity, gender, gender of esports and gaming where they can truly
identity, sexual orientation, age, social class, become what/who they want regardless of physi-
physical ability or attributes, religious or ethical cal, economic, or other disadvantages. At the heart
values system, national origin, and political of esports is an inclusive ecosystem. Unfortu-
beliefs (n.d.). nately, the reality turns ugly when people are not
accountable for their digital actions. The intercon-
nectivity and the possibility to hide behind a
Introduction screen can fuel cyber bullying when some people
exploit the anonymity of a gamer-tag (a player’s
This entry is a commentary on the current state of chosen name while gaming) to be abusive to
diversity in the digital world and what opportuni- others without consequence. Gaming communi-
ties the future may hold in the new ecosystem ties tend to be a tight nit group because of their
(metaverse). shared passion. The social connection is important
The human fascination with a virtual world is and can be difficult for a person that does not fit in
not a new concept. Video games have been around with the group. When an ecosystem is not wel-
since 1947 (Early history 2021; Wikimedia coming and safe, it does not breed diversity.
Foundation 2021) but the first game that really Whereas on one side, esports is about entertain-
launched the industry can be attributed to physi- ment and fun, there is an ugly side of the virtual
cist William Higinbotham who created a simple world that lacks the checks and balances to pre-
tennis game in October 1958 that became a big hit vent people from behaving badly. However, a
at the Brookhaven National Laboratory open solution may be just around the corner. There are
house (Tretkoff 2008; October 1958: Physicist many companies that are working on creating
invents first video game n.d.). In the early days, programs that can help solve this issue with a
a video game was a computer-generated program digital identity for every individual. The goal is
intended to deliver fun to the user that was to develop accountability and regulations, so the
performing certain tasks. Today the scope remains virtual world can be safe for everyone as the
the same, but the level of engagement is almost metaverse becomes more mainstream.
limitless. From one user interacting with a com- The term metaverse (Frey 2021) was first used
puter to today’s virtual worlds, the digital gaming in the 1992 novel “Snow Crash” by Neal Stephen-
evolution has moved quickly into our lives with son. In 2003, Linden Lab built an on-line platform
both positive and negative results. called “Second Life” where people could create
With the rapid growth of the World Wide Web an avatar (digital persona) and have a second life
and access to information, people are becoming in a virtual world. In 2011, author Ernest Cline
more educated, more connected, and more gave a then-new interpretation of a virtual world
informed. Although there still exists a digital in his novel (Cline 2011) “Ready Player One.”
divide or gap in access for some demographics, Cline tells the story of how a videogame character
604 Divide

comes to life in the year 2045 and therefore owns index.php?title=Early_history_of_video_games&

its position in the virtual world. He imagined what oldid=1060108429
Folger, J.: Metaverse. Investopedia. (2021). Retrieved
it would be like for a player to experience a January 2, 2022, from https://www.investopedia.com/
transformative participation where the player/ metaverse-definition-5206578
character was not anymore just “behind the Frey, T.: The history of the metaverse. Futurist Speaker.
screen” but was sort of “within the screen.” (2021). Retrieved January 2, 2022, from https://
futuristspeaker.com/future-trends/the-history-of-the-
Visionaries had already started putting together metaverse/
the building blocks of this imaginary digital October 1958: Physicist invents first video game: Ameri-
world and soon after mainstream companies can Physical Society. (n.d.). Retrieved January 2, 2022,
pivoted to align with this emerging technology. from https://www.aps.org/publications/apsnews/
200810/physicshistory.cfm#:~:text¼In%20October%
New decentralized currencies have been intro- 201958%2C%20Physicist%20William,Brookhaven%
duced as the world started its next phase of a 20National%20Laboratory%20open%20house
digital revolution. In 2020, while in the midst of Simpson, J. December 2008 Update. (2008). Retrieved
a global pandemic, people across the globe were October 13, 2022, from https://www.oed.com/public/
update0812/december2008-update
stuck at home with limited ways to socialize, so Tretkoff, E. October 1958: Physicist Invents First Video
they turned to the digital world to escape reality. Game. American Physical Society. (2008). Retrieved
The concept of a metaverse became truly January 2, 2022, from https://www.aps.org/publica
mainstreamed in 2021 when Facebook’s parent tions/apsnews/200810/physicshistory.cfm#:~:text=In
%20October%201958%2C%20Physicist%20William,
company changed its name to Meta. Brookhaven%20National%20Laboratory%20open%
What part diversity can play in all of this? 20house
Imagine a completely new world in a digital Wikimedia Foundation: Early history of video games.
form created by people from different back- Wikipedia. (2021). Retrieved January 2, 2022, from
https://en.wikipedia.org/wiki/Early_history_of_video_
grounds. In this world, everyone will have their games
own avatar that can perform actions one could
have only dreamed of before. Now imagine that
all of these avatars have a digital identity, so
everyone can feel safe and welcome. The meta- Divide
verse is an opportunity to create an ecosystem
where everyone is equal. It is also a new frontier ▶ Academic and Video Game Industry “Divide”
to explore proof of concept, test new research, or
prototype new products to then be reproduced in
real life. The opportunities are endless. With the
power of connectivity, people from less privileged DLC
communities can have a chance of making a mark
in a world with a lower barrier of entry. Society ▶ Emotion in Games
has a new chance to do this right and, hopefully,
everyone will embrace it. Full inclusivity is now
just a click away. Only time will tell.
Documentary Games

References ▶ Political Game Design

Cline, E. (2011). Ready Player One. Crown Publishers.

Diversity and Inclusion Definitions: Ferris State University.
(n.d.). Retrieved January 2, 2022, from https://www. Dog Technology
ferris.edu/administration/president/DiversityOffice/
Definitions.htm
Early history of video games. Wikipedia. (2021). Retrieved ▶ Engaging Dogs with Computer Screens: Ani-
January 2, 2022, from https://en.wikipedia.org/w/ mal-Computer Interaction
Dōjin Game 605

characters from popular culture make up much

Dogs of dōjin scene; yet original works are also com-
mon (Lamerichs 2016). A significant proportion
▶ Engaging Dogs with Computer Screens: Ani- of dōjin works are erotic or pornographic in nature
mal-Computer Interaction (Watabe 2013), with many fan works seeking to
extend the fictional worlds of popular culture into
the realm of erotic fantasy.
By contrast to Japan’s mainstream gaming,
Dōjin Game which revolves largely around game consoles D
and to a lesser extent amusement arcades, dōjin
Mikhail Fiadotau games, also referred to as dōjin sofuto, are primar-
Centre of Excellence in Media Innovation and ily designed for the personal computer (Hichibe
Digital Culture, Tallinn University, Tallinn, and Tanaka 2016). Most of these games belong to
Estonia one of a handful of typical genres such as visual
novels, role-playing games, and scrolling
shooters. Thematically and aesthetically, they
Synonyms usually cater to established niches such as bishōjo
(male-oriented games featuring beautiful girls),
Doujin Game otome (games for young women focusing on rela-
tionship building, usually with a handsome male
character), yaoi (games depicting homoerotic
Definitions relationships between male characters, but gener-
ally intended for female consumption), and so on.
Dōjin games (alternatively romanized as doujin) While there are many dōjin shops in Japan,
are digital games made by individuals or small specializing in dōjin manga (dōjinshi), games,
teams of enthusiasts, referred to as dōjin circles. and music, dōjin works can also be found in
Alongside other media such as manga comics, mainstream manga and game shops. However,
literature, anime, and music, these games are the most visible venue for dōjin work dissemina-
part of the wider dōjin scene, which originated in tion is the semi-annual Comic Market convention
Japan and which revolves around a shared appre- in Tokyo (Lam 2010). Smaller local events and
ciation of popular culture. conventions also play an important role.
Despite originating from Japan, dōjin culture
has extended to other countries in the region such
Overview as China (Si 2015), Taiwan (Chiang and Lo 2012)
and South Korea (Yoon 2015). Moreover, it has
Kenkyūsha’s New Japanese-English Dictionary had considerable cultural impact in the West as
(2003) explains the term dōjin (同人) as being well, leading creators to appropriate some of its
short for dōshi-no hito (同志の人), or “like- genres and conventions (Lamerichs 2016).
minded person, kindred spirit.” The word dōjin
is generally used to refer to small groups (sākuru,
“circles”) of individuals brought together by their History
shared tastes in popular culture, who produce self-
published works such as literature, manga comics, While dōjin games are a relatively recent addition
digital games, literature, and/or music. Although to the wider dōjin culture, the culture itself has its
most dōjin producers are hobbyists, some are pro- origins in the late nineteenth–early twentieth cen-
fessional or semi-professional artists, writers, tury dōjin literary circles (Morishita 1980). These
game designers, etc. (Hichibe and Tanaka 2016). circles produced self-published literary magazines
Fan works (nijisōsaku) based on worlds and such as Bungakukai and Shinshichō, which were
606 Dōjin Game

highly influential in Japan’s pre-war literary scene distance themselves from mainstream gaming
(Encyclopedia Nipponica 1994). In the post-war (Garda and Grabarczyk 2016), many dōjin
period, some of these magazines began to include games are in fact fan works based on mainstream
manga; dedicated dōjinshi manga magazines, games. Additionally, dōjin games are one part of a
such as Shōtarō Ishinomori’s Bokujū Itteki, were close-knit transmedia continuum also including
also increasingly published (Kálovics 2016). Self- manga, anime, literature, music, etc.
published manga gradually became a fixture at The lack of opposition towards mainstream
science fiction conventions, and in 1972, Nihon gaming and the close integration with other
Manga Taikai, a dedicated manga convention, media means that some dōjin games achieve
was held for the first time, with some booths wide recognition beyond both the niche of dōjin
selling dōjinshi magazines (Iizuka 2017). culture and the cultural form of game. One exam-
However, the biggest impetus to the develop- ple is Higurashi no Naku Koro ni (“Higurashi:
ment of dōjin culture was the emergence of the When They Cry”), which started in 2002 as a
Comic Market (or Comiket) in 1975 (Lam 2010). horror visual novel sold at the Comic Market but
Whereas the first grassroots convention attracted has now evolved into a transmedia franchise com-
32 dōjin circles and about 700 attendees, today prising an anime series, a manga, a live action
Comiket brings together around 500,000 partici- film, multiple novels, and an action game
pants on average who buy and sell dōjin manga, (Wheeler 2011), reflecting how interconnected
novels, and games, as well as engage in cosplay Japan’s “media mix” (Steinberg 2012) is.
and other activities. The integration of dōjin content with main-
The appearance of home computers such as stream media and major distribution networks is
MSX and NEC PC-88 in the 1980s enabled hob- also interesting from another perspective: that of
byist developers to create their own computer copyright. The co-existence of mainstream works
games, which began to find their way into dōjin and fan-made “parodies” (often of an erotic
culture. Visual novel Ningyo no Namida (“The nature) on manga and game shop shelves can
Mermaid’s Tears”), released in 1984 by Teikoku hardly be explained through the Western copy-
Soft, is considered to be the first dōjin game to be right framework. Such a status quo, albeit precar-
exhibited at a Comiket. By the end of 1980s, ious from a legal standpoint (Noppe 2010), offers
computer games had become a staple at dōjin a vision of how copyright holders can interact
conventions. In 1988, Pasoket, a spinoff of with fan producers – and how creativity can
Comiket focusing on games, began to take place emerge at the intersection of independent produc-
in cities across Japan on a monthly basis (Hichibe tion and fandom.
2013).
More recently, the advent of digital distribution
has further widened the reach of dōjin games,
Cross-References
many of which are now sold (or distributed for
free) online, with an increasing number found on
▶ Indie Game
global distribution platforms such as Steam and
▶ Visual Novel
Apple’s App Store (Hichibe and Tanaka 2016).

References
Specificity and Significance
Chiang, Y.-H., Lo, T.-Y.: A study of Doujinshi product
Dōjin games are often compared to the Western design. In: Proceedings of the 59th Annual Conference
phenomenon of indie games (Picard 2013; of the JSSD, pp. 223–224 (2012)
Consalvo, M.: Atari to Zelda: Japan’s Videogames in
Consalvo 2016), but there are important differ-
Global Contexts. MIT Press, Cambridge (2016)
ences between the two. While indie games stress Dōjin. In: Koine, Y. (ed.) New Japanese-English Dictio-
the artistic independence of their creators and nary, 5th ed. Kenkyūsha, Tokyo (2003)
Domain-Specific Choices Affecting Design Effort in Gamification 607

Dōninzasshi. In: Nihon Dai Hyakka Zensho [Encyclopedia

Nipponica]. Shōgakukan, Tokyo (1994). [In Japanese] Domain-Specific Choices
Garda, M.B., Grabarczyk, P.: Is every indie game indepen-
dent? Towards the concept of independent game. Game Affecting Design Effort in
Stud. 16 (2016) http://gamestudies.org/1601/articles/ Gamification
gardagrabarczyk
Hichibe, N.: Gēmusangyō seichō no kagi toshite no Amol D. Mali1 and Kompalli Jwala Seethal
jishuseisaku bunka [Independent production culture as
a key to the growth of game industry], PhD thesis Chandra2
1
(2013) [In Japanese]. http://t2r2.star.titech.ac.jp/rrws/ Computer Science Department, University of
file/CTT100674911/ATD100000413/ Wisconsin-Milwaukee, Milwaukee, WI, USA D
Hichibe, N., Tanaka, E.: Content production fields and 2
Microsoft Corporation, Redmond, WA, USA
Doujin game developers in Japan: non-economic
rewards as drivers of variety in games. In: Pulos, A.,
Lee, S.A. (eds.) Transnational Contexts of Culture,
Gender, Class, and Colonialism in Play, pp. 43–80. Synonyms
Palgrave Macmillan, London (2016)
Iizuka, K.: Comparison of “minicomi” and “doujinshi”.
Bull. Fac. Humanit., Seikei University. 52, Choices in games; Edutainment; Game design;
89–107 (2017) [In Japanese] Game production effort; Gamification
Kálovics, D.: The missing link of shojo manga history: the
changes in 60s shojo manga as seen through the mag-
azine Shukan Margaret. J. Kyoto Seika University. 49,
3–22 (2016) Definitions
Lam, F.-Y.: Comic market: how the world’s biggest ama-
teur comic fair shaped Japanese Dōjinshi culture. Gamification: It is the process of modifying an
Mechademia. 5, 232–248 (2010) activity which does not include elements of a
Lamerichs, N.: Euromanga: hybrid styles and stories in
transcultural Manga production. In: Brienza, C. (ed.) game, to allow the performer of the modified
Global Manga: “Japanese” Comics Without Japan? activity to (i) improve knowledge or skill using
pp. 75–94. Routledge, London (2016) the feedback from the mechanism built into the
Morishita, S.: Shin Dōjinzasshi Nyūmon [New Introduc- modified activity while being entertained through
tion to Dōjin Magazines]. Kōseisha, Tokyo
(1980) [In Japanese] text, pictures, animations, video, or audio,
Noppe, N.: Dōjinshi research as a site of opportunity for (ii) dynamically earn rewards, (iii) win, or
manga studies. In: Berndt, J. (ed.) Comics Worlds & the (iv) be dynamically ranked, by fulfilling relevant
World of Comics: Towards Scholarship on a Global conditions, such that the overall experience of the
Scale, pp. 115–131. International Manga Research
Center, Kyoto (2010) performer of the modified activity is different
Picard, M.: The foundation of geemu: a brief history of from the experience of getting points, grades,
early Japanese video games. Game Stud. 13 certificates, gifts, payments, or prizes based on
(2013) http://gamestudies.org/1302/articles/picard paid/volunteer work, shopping, or assessment
Si, W.: From emotional to rational: the interaction of
Doujin group and formation of ritual. Cont. Youth similar to that in schools or colleges.
Res. 4 (2015) [In Chinese]
Steinberg, M.: Anime’s Media Mix: Franchising Toys and
Characters in Japan. U of Minnesota Press, Minneapo-
lis (2012)
Introduction
Watabe, K.: Book review: Mizuko Ito, Daisuke Okabe, and
Izumi Tsuji, Eds. Fandom unbound: Otaku culture in a Santos and others (Santos et al. 2018) report on
connected world. Spectator. 33, 71–74 (2013) http:// how requirements analysis, test planning, test
cinema.usc.edu/spectator/33.1/10_Watabe.pdf
Wheeler, J.: The Higurashi code: algorithm and adaptation
design, test execution, and test closure for com-
in the otaku industry and beyond. Cinephile. 7, puter games differ from these activities for soft-
26–30 (2011) http://cinephile.ca/files/Vol7No1-FOR% ware which is not a game. Though there is lot of
20JEFF/Vol7No1-FOR%20JEFF.pdf theoretical and empirical work on estimating the
Yoon, W.-H.: The reinterpretation of comic-animation by
content users – the reproductions in Korean Cosplay
effort needed for development, testing, and main-
culture. Cartoon Ani. Stud. 41, 487–510 (2015). https:// tenance of software, estimation of effort for
doi.org/10.7230/KOSCAS.2015.41.487. [In Korean] designing, developing, and testing computer
608 Domain-Specific Choices Affecting Design Effort in Gamification

games remains underexplored. Identifying Level 3 has distinguished citizens. The gover-
domain-specific choices affecting game-design nance of the municipality may publicly reward
effort has several uses. Design influences the the winner of Leaderboard of Distinguished Citi-
effort for development, testing, and documenta- zens each year.
tion. This affects budget of the project, quality of Butler and Ahmed (2016) present a game to
the game, and the temporal schedule of game motivate students to achieve computer-science-
production. Indicators of game-deign effort can learning goals. The first level of the game intro-
be used in predicting if the game can be produced duces stacks and queues. The second level of the
while meeting budgetary, temporal, and quality- game includes Selection Sort, which is a sorting
related requirements. This helps in adjusting algorithm. A player is a cadet at a space academy
scope of the game. Since design of computer and has his/her own robot to control and custom-
games is usually open-ended because of game ize while performing missions.
design being highly creativity-based, theory of A list of key design choices in gamifying the
computer games continues to evolve. Identifying process of motivating people to achieve a goal is
domain-specific choices in design of computer provided next.
games is also useful for refining the theory of
computer games that captures breadth, depth, (i) Goals for which to motivate the player, e.g.,
and diversity in this field. This article identifies blood donation, monetary donation, study-
many of the key domain-specific design choices in ing for an international competitive exami-
games for motivating people, teaching people to nation, volunteering at a retirement home,
recognize objects, and teaching a language, engi- and volunteering at a public library
neering design, computer programming, or algo- (ii) Number and types of activities that directly
rithm development, and shows how identifying motivate the player to achieve the relevant
domain-specific choices in game design can be goal by issuing rewards that the player can
useful. show others if he/she wishes (e.g., an email
sent to friends or relatives of the player when
the player donates the correct quantity of
blood at a location where it is most needed
Gamification for Motivating is a reward that directly motivates the player
to donate blood in virtual world, and it is
Bianchini and others (Bianchini et al. 2016) hoped that the player will be motivated to
present a web-based mobile platform for donate blood in the real world after playing
e-participation of administrators and citizens. this game)
This platform allows citizens and administrators (iii) Number and types of activities that indirectly
to submit and share problems, initiatives, and motivate the player to achieve the goal, e.g.,
solutions. Other citizens within the same munici- taking a tour of a hospital in digital world
pality can add their own solutions. Citizens obtain showing suffering of patients in need of
points for submitting proposals and solutions. blood
A citizen gets an extra point for each detail (iv) Number and types of activities to include to
added to the solution and each vote received on teach undesirable consequences of overdo-
the solution. Administrators can award additional ing the activities that the game motivates the
points. Leaderboard of New Solutions, Leader- player to perform
board of Completed Solutions, Leaderboard of
Citizens, and Leaderboard of Proposals are used
to assign black, silver, and gold medals to citizens. Gamifying Object Recognition
An administrator can assign a blue medal. Medals
enable citizens to change their level. Level 1 has Teaching players to recognize objects has numer-
interested citizens. Level 2 has active citizens. ous advantages, e.g., players can use the ability to
Domain-Specific Choices Affecting Design Effort in Gamification 609

recognize objects in a domain in shopping, Gamifying Language Learning

repairing machines, supplying parts, and
reporting diagnosis of dysfunctional machines. A gamification of Mandarin learning is presented
A list of key design choices in gamification of in (Heryadi and Muliamin 2016). They divided
object recognition is provided next. the learning material among the following catego-
ries: (i) Greetings, (ii) Family, (iii) General con-
(i) Types of objects to include in the game, versation, (iv) Colors, (v) Numbers, (vi) Days and
e.g., plants and animals months, (vii) General vocabulary, (viii) Food, and
(ii) Objects of chosen types to include in the (ix) Animals. D
game, e.g., cabbage, spinach, and carrots A list of key design choices in gamification of
(iii) Shapes, designs, colors, sizes, and orienta- language learning is provided next.
tions of objects, e.g., red cabbage and green
cabbage (i) Languages to teach
(iv) Lighting conditions in the region (ii) Native languages to include in the game
containing the object/objects to be (iii) Number of letters or symbols of the alpha-
recognized bet to show at a time
(v) Whether to simultaneously show multi- (iv) Number of words to show at a time
ple objects for the player to recognize (v) Number of sentences to show at a time
or not (vi) Number of words in a sentence
(vi) Whether to show a part of the object to be (vii) Order in which to show letters, symbols,
recognized instead of the entire object words, or sentences
(vii) Whether to allow objects to touch each (viii) Whether to show incomplete words or
other or partially block the view of other sentences or not
objects or not, if multiple objects are to be (ix) Number of letters to exclude from words
shown simultaneously and number of words to exclude from
(viii) Whether to show damaged objects or not sentences, when the player is expected to
(ix) Types of damage to show enter missing letters or words
(x) Whether to include objects that are not (x) Number of writing styles to use to gener-
damaged but have an undesirable appear- ate letters, symbols, words, and sentences
ance or not, e.g., cups with dust that can be (xi) Number and types of accents in the audio
removed by washing clips containing pronunciations of letters,
(xi) Whether to include irrelevant objects or not symbols, and words
(xii) Maximum number of objects to show (xii) Minimum number of letters in words,
simultaneously minimum number of words, minimum
(xiii) Number of images of an object to show number of sentences, minimum number
(xiv) Whether to show multiple images of an of words in individual sentences, or the
object simultaneously or sequentially lower limit on the total number of words
(xv) Whether to show three-dimensional version the player is expected to type or speak, if
of an object or its 2D projections, e.g., view the player is asked to type or speak as
from top, view from bottom, view from much as possible in the given time
front side, and view from rear side (xiii) Number of irrelevant symbols, letters,
(xvi) Types of clues and number of additional words, pictures, or sentences to include
clues to give to help in recognizing the for the player to identify the relevant ones
object, e.g., sound made when the object (xiv) Types of pictures and number of pictures
falls on wood or concrete, changes in the to include if the player is expected to
object with variation in temperature, and match them with text, type relevant text,
changes in the object due to chemical or speak in response to the shown picture/
reactions pictures
610 Domain-Specific Choices Affecting Design Effort in Gamification

(xv) Number and types of pictures to include if materials, incorrect assumptions, incorrect
the player is expected to order them based units, and incorrect variables, constants,
on the words in the given sentence or and mathematical operators in equations
based on the given sentences (vi) Number and types of steps to exclude, if
(xvi) Number and types of words whose mean- the player is expected to detect that some
ings to include to create a verbose text that steps are missing, and is expected to
the player is expected to make concise by provide them
replacing the meanings with (vii) Amount and types of information to
corresponding words exclude from individual steps if the player
(xvii) Number of words, sentences, and clusters, is expected to detect incompleteness
and clustering criteria, when the player is within individual steps and make the
expected to group words or sentences steps complete by providing the missing
based on the given criteria, e.g., putting information
synonyms together, putting words or (viii) Number and types of choices to provide to
sentences related to airports together, etc. the player at a time when the player is
(xviii) Number of clusters and number of words taught to design by having him/her make
or sentences in individual clusters, when choices incrementally
the player is expected to identify the (ix) Number and types of designs or design
criteria used to create the clusters steps to include to have the player rate,
(xix) Number and types of words, sentences, or classify, or order them
paragraphs to include in the text that the (x) Number and types of design questions and
player is expected to summarize number and types of answers to include
(xx) Minimum and maximum number of char- when the player is expected to match
acters, words, or sentences in the player’s questions with answers
answers to free-response questions (xi) Whether to limit the game to teaching to
(xxi) Themes for free-response questions produce feasible designs or include opti-
mal designs too
(xii) Factors to include besides safety and dura-
Gamifying Engineering Design bility, e.g., manufacturability, sustainabil-
ity, and cost
A list of key design choices in gamification of (xiii) Number and types of differences to intro-
engineering design is provided next. duce between two designs or steps, when
the player is expected to compare them
(i) Whether to expect the player to design an and find as many differences as possible
entire product or a part of it (xiv) Number and types of facts about designs
(ii) Products and parts to design fully or to include if short questions are to be
partially posed to the player
(iii) Number of design steps to show to the (xv) Number of products or parts to include in
player at a time, if the player is taught by the set of options when the player has to
having him/her reorder the given steps choose the products or parts the given
(iv) Number of irrelevant steps to show to the design steps are relevant to, from the
player at a time, if the player is taught by given options
having him/her remove irrelevant steps (xvi) Number and types of opportunities to
from the given steps include in a correct design for the player
(v) Number and types of mistakes to intro- to detect and improve the design
duce within individual steps, if the player (xvii) Number and types of steps in the design of
is expected to correct mistakes in individ- different products or different parts to pre-
ual steps, e.g., incorrect choice of sent to the player as if all of these steps are
Domain-Specific Choices Affecting Design Effort in Gamification 611

for designing only one product or only one (xi) Number and types of steps to exclude, if
part, for the player to associate the steps the player is expected to detect that some
with different parts or products steps are missing, and is expected to
(xviii) Number and types of designs or design provide them
steps to present to the player when the (xii) Amount and types of information to
player is expected to identify why they exclude from individual steps if the player
are ordered, rated, or clustered the way is expected to detect incompleteness
they are in the challenge within individual steps and make the
steps complete by providing the missing D
information
Gamifying Algorithm Development (xiii) Number and types of choices to provide to
the player at a time when the player is
A list of key design choices in gamification of taught to design an algorithm by having
algorithm development is provided next. him/her make choices incrementally
(xiv) Number and types of algorithms or steps
(i) Types and number of algorithms to in algorithms or steps in algorithm analy-
include in the game sis to include, and have the player rate,
(ii) Whether to design challenges based on an classify, or order them
entire algorithm or a part of it (xv) Number and types of design or analysis
(iii) Whether to expect the player to design an questions and number and types of
entire algorithm or a part of it answers to include, when the player is
(iv) Whether to expect the player to analyze an expected to match questions with answers
entire algorithm or a part of it (xvi) Whether to limit the game to teaching to
(v) Type of analysis to include in a challenge design sound algorithms or include algo-
(e.g., completeness, optimality, time com- rithm optimization too
plexity, and space complexity) (xvii) Factors to include besides soundness, e.g.,
(vi) Number and types of steps of an algorithm ease of modification to handle extra inputs
to show to the player at a time, for or inputs of different types, ease of modi-
analyzing them fication to produce a solution meeting
(vii) Number and types of subtasks within an additional or different requirements, and
algorithm to present to the player for ease of modification to find more than one
him/her to provide a part/parts of the solution or count the number of possible
algorithm solutions
(viii) Number of steps of an algorithm to show (xviii) Number and types of differences to intro-
to the player at a time, if the player is duce between two designs or steps, when
taught by having him/her reorder the the player is expected to compare them
given steps and find as many differences as possible
(ix) Number of irrelevant steps to show to the (xix) Number and types of facts about algo-
player at a time, if the player is taught by rithms to include if short questions are to
having him/her remove irrelevant steps be posed to the player
from the given steps (xx) Number of problems to include in the set
(x) Number and types of mistakes to intro- of options when the player has to choose
duce within individual steps, if the player the problems the given algorithm steps are
is expected to correct mistakes in individ- relevant to
ual steps, e.g., incorrect choice of data (xxi) Number and types of opportunities to
structures, syntactic errors, incorrect include in a correct algorithm or analysis
mathematical operators, variables, or con- of an algorithm, for the player to detect
stants, and incorrect type of statement and improve
612 Domain-Specific Choices Affecting Design Effort in Gamification

(xxii) Number and types of steps from different (viii) Types of program analysis to include in a
algorithms or their analyses to present to challenge (e.g., guarantee of finding a
the player as if they are steps for only one solution, optimality, dealing with invalid
algorithm or its analysis, for the player to inputs, comments for understanding and
associate the steps with corresponding modification, running time, and memory
algorithms or analyses needed)
(xxiii) Whether to include parallel algorithms (ix) Number and types of steps of a program
or not to show to the player at a time, for
(xxiv) Number and types of steps or algorithms analyzing them
to present to the player to parallelize (x) Number and types of subtasks fulfilled
(xxv) Number of parallelizable steps to include by a program to present to the player for
along with inherently-sequential steps, him/her to provide the corresponding
when the player is expected to classify part/parts of the program
each step as parallelizable or inherently (xi) Number of steps of a program to show to
sequential the player at a time, if the player is taught
(xxvi) Number and types of algorithms or their by having him/her reorder the
parts to present to the player when the given steps
player is expected to identify why they (xii) Number of irrelevant steps to show to the
are ordered, rated, or clustered the way player at a time, if the player is taught by
they are in the challenge having him/her remove irrelevant steps
from the given steps
(xiii) Number and types of mistakes to intro-
Gamifying Programming duce within individual steps, if the player
is expected to correct mistakes in individ-
Carreno-Leon and others (Carreno-Leon et al. ual steps, e.g., incorrect choice of data
2018) present the use of gamification in a course structures, syntactic errors, incorrect
on introduction to programming. In the first level, mathematical operators, variables, or con-
the student is given only those cards that appear in stants, and incorrect type of statement
solution. In the second level, the student is given (xiv) Number and types of steps to exclude, if
more cards with pseudocode fragments than the player is expected to detect that some
needed, but the number of cards needed in solu- steps are missing, and is expected to
tion is disclosed. In the third level, the student is provide them
given more cards than needed and not told how (xv) Amount and types of information to
many are needed in solution. exclude from individual steps if the
A list of key design choices in gamification of player is expected to detect incomplete-
programming is provided next. ness within individual steps and make
the steps complete by providing the
(i) Programming paradigms missing information
(ii) Programming language/languages (xvi) Number and types of choices to provide
(iii) Features of programming language/lan- to the player at a time when the player is
guages to cover in the game taught to write a program by having
(iv) Number and types of programming tasks him/her make choices incrementally
(v) Whether to design a challenge based on (xvii) Number and types of programs or steps
an entire program or a part of it from programs or steps from program
(vi) Whether to expect the player to provide analysis to include, to have the player
an entire program or a part of it rate, classify, or order them
(vii) Whether to expect the player to analyze (xviii) Number and types of design, implemen-
an entire program or a part of it tation, or analysis questions, and number
Domain-Specific Choices Affecting Design Effort in Gamification 613

and types of answers to include, when (xxix) Number and types of programs or their
the player is expected to match questions parts to present to the player, when the
with answers player is expected to identify why they
(xix) Whether to limit the game to teaching to are ordered, rated, or clustered the way
write correct programs or include pro- they are in the challenge
gram optimization too
(xx) Factors to include besides correctness,
e.g., ease of modification to handle Usefulness of Identifying
extra inputs or inputs of different types, Domain-Specific Choices D
ease of modification to produce a solu-
tion meeting additional or different Domain-specific choices like the ones we identi-
requirements, ease of modification to fied for six domains, directly or indirectly affect
find more than one solution or count the the effort needed to design characters, actions,
number of possible solutions, ease of feedback, rules, challenges, hints, rewards, pun-
modification to handle other data struc- ishments, gaming world, challenge-adaptation
tures, and ease of rewriting in a different mechanisms for keeping the player motivated
programming language and interested, surprises, levels, fairness-
(xxi) Number and types of differences to intro- enforcement mechanisms, cheating-detection
duce between two programs or steps mechanisms, elements relevant to players with
from programs, when the player is special needs, and a multiplayer version. They
expected to compare the two and find as also directly or indirectly affect the effort needed
many differences as possible for localization. Identifying domain-specific
(xxii) Number and types of facts about pro- choices is useful in several ways. These choices
gramming to include if short questions allow us to use counting techniques from discrete
are to be posed to the player mathematics to quantify some components of the
(xxiii) Number of problems to include in the set gaming experience. Identifying these choices
of options when the player has to choose allows us to compare them to know which choices
the problems the given program steps are have a bigger impact on the effort needed for
relevant to game design, development, and testing. This
(xxiv) Number and types of opportunities to helps in deciding which choices to use. Identify-
include in a correct program or correct ing domain-specific choices can be useful in
analysis of a correct program, for the speeding up game design, development, and test-
player to detect and improve ing, by reusing design, code, and test plans of a
(xxv) Number and types of steps from different game from domain D0 to create a game in domain
programs or their analyses to present to D00 when D00 and D0 have similar domain-specific
the player as if they are steps from one choices.
program or its analysis, for the player to
associate the steps with corresponding
programs
(xxvi) Whether to include parallel programs References
or not
Bianchini, D., Fogli, D., Ragazzi, D.: Promoting citizen
(xxvii) Number and types of steps or programs
participation through gamification. In: Proceedings of
to present to the player to parallelize 9th Nordic Conference on Human-Conference Interac-
(xxviii) Number of parallelizable steps to include tion (NordiCHI), pp. 1–4 (2016)
along with inherently-sequential steps, Butler, S., Ahmed, D.: Gamification to engage and moti-
vate students to achieve computer science learning
when the player is expected to classify goals. Proceedings of International Conference on
each step as parallelizable or inherently Computational Science and Computational Intelli-
sequential gence, pp. 237–240 (2016)
614 Dopamine

Carreno-Leon, M., Sandoval-Bringas, A., Alvarez-

Rodriguez, F., Camacho-Gonzalez, Y.: Gamification Dynamic Game Balancing
technique for teaching programming. In: Proceedings
of IEEE Global Engineering Education Conference
(EDUCON), pp. 2009–2014 (2018) ▶ Meta Artificial Intelligence and Artificial Intel-
Heryadi, Y., Muliamin, K.: Gamification of M-learning ligence Director
Mandarin as second language. In: Proceedings of the
1st International Conference on Game, Game Art, and
Gamification (ICGGAG), pp. 1–4 (2016)
Santos, R., Magalhaes, C., Capretz, L., Correia-Neto, J.,
da Silva, F., Saher, A.: Computer games are serious Dynamic Music
business and so is their quality: particularities of soft-
ware testing in game development from the perspec-
tive of practitioners. In: Proceedings of the 12th ▶ Adaptive Music
ACM/IEEE International Symposium on Empirical
Software Engineering and Measurement (ESEM),
pp. 1–10 (2018)
Dynamic Music Generation:
Audio Analysis-Synthesis
Methods
Dopamine
Gilberto Bernardes and Diogo Cocharro
▶ Video Game Trolls and Dopamine Withdrawal INESC TEC and University of Porto, Faculty of
Engineering, Porto, Portugal

Doujin Game Synonyms

▶ Dōjin Game Adaptive music systems; Audio collage; Audio

mosaicing; Concatenative sound synthesis; Inter-
active music systems

Down Syndrome
Definitions
▶ Computer Games for People with Disability
Dynamic music generation systems create ever-
different and changing musical structures based on
formalized computational methods. Under scope is
a subset of these methods which adopt musical
DSDV audio as a strategy to formalize musical structure
which then guides higher-level transformations to
▶ Simulation and Comparison of AODV and be synthesized as new musical audio streams.
DSDV Protocols in MANETs

Introduction

Dynamic Difficulty Technologies which adhere to nonlinear, as

Adjustment opposed to fixed, storytelling are becoming per-
vasive in the digital media landscape (Lister et al.
▶ Meta Artificial Intelligence and Artificial Intel- 2003). In this context, methods for dynamically
ligence Director generating music have been prominently and
Dynamic Music Generation: Audio Analysis-Synthesis Methods 615

increasingly adopted in games, virtual and aug- architecture (e.g., compression) (Verfaille and
mented reality, interactive installations, and Arfib 2001). Recent advances in the hierarchical
360 video. Their adoption is motivated by a wide analysis and generation of musical audio structure
range of factors from computational constraints have expanded the transformations beyond the
(e.g., limited memory) to enhanced interaction sample level towards music processing (e.g., auto-
with external actuators and artistic endeavor. matic remixing).
Dynamic music generation systems are typi- Analysis-synthesis systems for dynamic gen-
cally driven by formalized or algorithmic methods eration of musical audio mimic fundamental per-
whose history is interleaved with modern comput- ceptual and cognitive human functions in a D
ing (Nierhaus 2009). This article reviews a subset threefold component architecture of machine lis-
of these systems which adopt musical audio as a tening, learning, and composing (see Fig. 1).
source to formalize musical structure algorithmi- Machine listening and learning are two
cally, which is then used to guide the generation of intertwined components which primarily adopt
new musical streams by synthesizing sub-audio bottom-up (or content-driven) processing
clips from the musical audio source – an approach methods to infer hierarchical structure from
addressed as musical audio analysis-synthesis. audio samples. It comprises two main tasks:
The remainder of this entry details a typical multilevel (beat, downbeat, phrase, and section)
architecture of a generative musical audio segmentation and the description of its temporal
analysis-synthesis system (section “From Sound structure. Inferred information tends to be
to Musical Audio Analysis-Synthesis Systems”) represented as graphs, whose nodes represent seg-
and presents an overview of its applications sce- mented musical structures, and directed pairwise
narios (section “Applications”). links their temporal relations (see Fig. 2). The
resulting representation provides a robust means
for computational structure discovery, notably by
From Sound to Musical Audio finding redundant information across the temporal
Analysis-Synthesis Systems structure which can be clustered according to some
(perceptually guided) similarity metric (see Fig. 2).
Audio analysis-synthesis methods break down a Typically, each node or segmented musical
sound into some essential, measurable attributes audio structure is represented in the system by a
(e.g., amplitude or pitch) to guide sound trans- feature vector, i.e., a set of numerical features that
formations during (re-)synthesis (Jehan 2005). result from measurable musical attributes, such as
Historically, these transformations exist at the pitch, loudness, and percussiveness. The choice of
sample level and follow an adaptive audio effect such attributes, the metrics used to compare them,

Machine Listening Machine Learning Machine Composing

musical audio generative

ANALYSIS
source model

transformation

new musical
SYNTHESIS
audio stream

Dynamic Music Generation: Audio Analysis-Synthesis Methods, Fig. 1 Architecture of a musical audio analysis-
synthesis system for dynamic generation
616 Dynamic Music Generation: Audio Analysis-Synthesis Methods

Attribute 2

Attribute 1

Perceptual threshold Audio Segments

Directed pairwise temporal relations

Dynamic Music Generation: Audio Analysis-Synthe- data points, represented as circles, whose locations are
sis Methods, Fig. 2 Illustration of a graph model of defined by their attributes in the descriptor space. Small
musical audio structure. The space is defined by segment distances denote higher (perceptual) similarity in
(perceptual) attributes (reduced in this illustration to two the descriptor space. The circumferences define a percep-
dimensions). Musical audio segments are understood as tual threshold used to cluster similar-sounding segments

and the perceptual threshold used to define the Applications

maximum degree of dissimilarity among seg-
ments are crucial to the graph design. All these We present a user-centered perspective of applica-
variables are commonly set manually by the users tions for dynamic generation of musical audio
due to their subjective nature and implications in through analysis-synthesis approaches. Our twofold
the resulting musical output of the system (i.e., fuzzy categorization is rooted in the nature of
synthesis). To a certain extent, the choice of such processed signals and distinguishes (natural or syn-
variables is akin to the role of sketches and raw thetic) sound textures and soundscapes from
material in the compositional process. For a com- music. A distinction is made concerning the struc-
prehensive comparison of these variables and ture of the audio signal content (e.g., noisy
their implications in the musical results, please vs. pitched or nonmetric vs. metric) rather than any
refer to Norowi et al. (2017). discrimination based on artistic merit, since both
If the resulting structural model has sufficient categories (music and soundscapes) can be under-
redundancy, machine composing can then tra- stood as the product of creative-oriented practices.
verse the directed links of the graph to create The generation of sound textures and
ever-different and changing musical streams soundscapes using analysis-synthesis methods
which retain the (higher-level) structure of the has been applied in audio postproduction for tele-
audio source captured by the model. The output vision and film as well as in sound design for
of an analysis-synthesis systems is musical col- games and interactive installations. It mostly
lages made of rearranged segments or audio snip- tackles the pervasive problem of extending a
pets from the original or source audio. The given audio clip in postproduction whenever the
structure of the output musical audio can be prerecorded audio does not cover the entire dura-
understood as a variation of particular features tion of a scene (Frojd and Horner 2007; Bernardes
temporal evolution of the source, thus retaining et al. 2016). Moreover, some analysis-synthesis
some level of similarity while allowing its inde- soundscape systems also enable to procedurally
terminate temporal expansion. generate highly controlled nuances that match
Dyslexia 617

external actuators (Bernardes et al. 2013; Schwarz with the audio oracle algorithm. In: Proceedings of the
and Schnell 2010). For example, in a game International Computer Music Conference, pp. 140–146
(2016). http://openaccess.city.ac.uk/15424/
engine, the player behavior can be mapped to Frojd, M., Horner, A.: Fast sound texture synthesis using
soundscape parameters, such as density of events overlap-add. In: International Computer Music Confer-
and spectral richness, to enhance the playability ence (2007)
through symbiotic relations across modalities. Jehan, T.: Creating Music by Listening. PhD thesis, Mas-
sachusetts Institute of Technology (2005)
The generation of music using analysis- Lamere, P.: The infinite jukebox. www.infinitejuke.com
synthesis methods has been mainly applied as (2012). Accessed 24 May 2016
(online) performance or (offline) computer- Lister, M., Dovey, J., Giddings, S., Grant, I., Kelly, K.: New D
assisted composition tools. In the first scenario, Media: A Critical Introduction. Routledge, London (2003)
Nierhaus, G.: Algorithmic Composition: Paradigms of
it has been highly explored in interactive music Automated Music Generation. Springer Science &
systems for human-machine improvisation, where Business Media, Wien (2009)
co-improvising machines aim to capture and emu- Norowi, N.M., Miranda, E.R., Hussin, M.: Parametric
late the musician’s style (Schwarz et al. 2006; factors affecting concatenative sound synthesis. Adv.
Sci. Lett. 23(6), 5496–5500 (2017)
Assayag et al. 2006; Surges and Dubnov 2013; Pachet, F., Roy, P., Moreira, J., d’Inverno, M.: Reflexive
Pachet et al. 2013; Einbond et al. 2016). The loopers for solo musical improvisation. In: Proceedings
second scenario has been highly explored to of the SIGCHI Conference on Human Factors in Com-
guide the search for variations of a given musical puting Systems, pp. 2205–2208. ACM (2013)
Schwarz, D., Schnell, N.: Descriptor-based sound texture
audio excerpt in the realm of entertainment tech- sampling. In: Proceedings of the Sound and Music
nologies or composition of a given user-defined Computing, pp. 510–515 (2010)
musical audio (Jehan 2005; Bernardes et al. 2013; Schwarz, D., Beller, G., Verbrugghe, B., Britton, S.: Real-
Lamere 2012; Davies et al. 2014). time corpus-based concatenative synthesis with
CataRT. pp. 279–282. Montreal (2006)
Surges, G., Dubnov, S.: Feature selection and composition
using pyoracle. In: Ninth Artificial Intelligence and Inter-
Cross-References active Digital Entertainment Conference, pp. 19 (2013)
Verfaille, V., Arfib, D.: A-dafx: Adaptive digital audio
effects. In: Proceedings of the Workshop on Digital
▶ Audiogame Audio Effects, pp. 10–14 (2001)
▶ Procedural Audio in Video Games
▶ Sonic Interactions in Virtual Environments

Dynapenia
References
▶ Computer Games for People with Disability
Assayag, G., Bloch, G., Chemillier, M., Cont, A., Dubnov,
S.: Omax brothers: a dynamic topology of agents for
improvisation learning. In: Proceedings of the 1st ACM
Workshop on Audio and Music Computing Multime- Dyscalculia
dia, pp. 125–132. ACM (2006)
Bernardes, G., Guedes, C., Pennycook, B.: EarGram: An ▶ Computer Games for People with Disability
Application for Interactive Exploration of Concatenative
Sound Synthesis in Pure Data, pp. 110–129. Springer,
Berlin (2013)
Bernardes, G., Aly, L., Davies, M.E.P.: Seed: Dysgraphia
resynthesizing environmental sounds from examples.
In: Proceedings of the Sound and Music Computing
Conference (2016) ▶ Computer Games for People with Disability
Davies, M.E.P., Hamel, P., Yoshii, K., Goto, M.: Auto-
mashupper: automatic creation of multi-song music
mashups. IEEE/ACM Trans. Audio Speech Lang. Pro-
cess. 22(12), 1726–1737 (2014). https://doi.org/10. Dyslexia
1109/TASLP.2014.2347135. ISSN 2329-9290
Einbond, A., Schwarz, D., Borghesi, R., Schnell, N.: Introduc-
ing catoracle: corpus-based concatenative improvisation ▶ Computer Games for People with Disability
E

Earphones Educational Game Abzû and

the Lens of Fun Learning
▶ Immersive Auralization Using Headphones
Kyle McCarter2 and Newton Lee1,2
1
Institute for Education, Research, and
Scholarships, Los Angeles, CA, USA
2
Editor Architecture Vincennes University, Vincennes, IN, USA

▶ Decoupling Game Tool GUIs from Core

Editing Operations Synonyms

Abzû; Educational games; Fun learning

Education Model
Definition
▶ Design Framework for Learning to Support
Industry 4.0 Educational games – Games that are explicitly
designed with educational purposes by helping
players learn a concept, a historical event, or a
skill as they play.
Education Pedagogy Lens of fun learning – A critical thought pro-
cess by applying the concept of fun learning in
▶ Design Framework for Learning to Support scrutinizing or analyzing a game.
Industry 4.0 Learning should be fun. How does an educa-
tional game like Abzû hold up when scrutinized
by the lens of fun learning? Released in 2016,
Abzû is an adventure video game developed by
Educational Game Giant Squid Studios and published by 505 Games
for PlayStation 4, Xbox One, Nintendo Switch,
▶ MEEGA+, Systematic Model to Evaluate Edu- and Microsoft Windows. Abzû is named after the
cational Games freshwater god in the Ancient Mesopotamian reli-
▶ Unified Modeling Language (UML) for Sight gion (Butterworth 2016; Gaston 2016; McElroy
Loss 2016; Nava 2016).
© Springer Nature Switzerland AG 2024
N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
620 Educational Games

Abzû is similar to Journey – a 2012 indie minimum. The game makes it fun to explore.
video game developed by thatgamecompany The player can swim around in current streams
and published by Sony Computer Entertain- with all the fish, look through caves, and explore
ment for PlayStation 3 that year. Like Journey, the unknown sea where they woke up in. The
the player starts the game waking up in a game also teaches about environmental protec-
strange place without any memory. So the tion. When there is no pollution, the player can
player starts exploring to find something that search under deep sea. However, when the water
will help tell them who they are and why they is polluted, the player could only see the shallow
are there. The major difference between Abzû areas. Toxic water also kills all the fish that live in
and Journey is the environment. In Journey, the the affected areas.
player is on a planet with a barren desert, The graphics in the game is quite pleasing in
whereas in Abzu the planet is completely cov- spite of limited budget on animation from a game
ered in water. development company. Nevertheless, it would be
In the beginning the player sees a giant area of more fun if there are more ways to interact with
water, and there is almost no land in sight except the fish in addition to sonar chimes and being
for a few islands here and there. The camera then able to grab onto large marine animals and ride
begins to zoom in on what appears to be a scuba them for a short distance. All in all, Abzû is a
diver floating with his/her face in the water. At good and elegant exemplification of a well-
maximum zoom, the scuba diver wakes up, looks designed game when scrutinized with the lens
around, and then decides to dive under to explore. of fun learning.
While traveling the diver meets many different
types of fish, even ones that are said to be extinct.
The types of fish are based on tens of thousands of References
real-life marine creatures. Their movements are
modeled after the behavior of real fish. Artificial Butterworth, S.: Holy Diver. (2016). https://www.
gamespot.com/reviews/abzu-review/1900-6416489/
intelligence is used to create real-time interaction
Gaston, M.: Abzu review. Wetter is better. (2016). https://
among marine animals, the diver, and the www.eurogamer.net/articles/2016-08-02-abzu-review
environment. McElroy, J.: Abzu review. (2016). https://www.polygon.
The player soon begins to notice that there are com/2016/8/2/12290330/abzu-review-ps4-pc-
playstation-4
not many fish and to wonder where they could
Nava, M.: Interview with the Creative Director of ABZÛ.
be. When the player encounters a coral reef-type (2016). https://80.lv/articles/interview-with-the-
object and interacts with it, the object begins to creative-director-of-abzu/
summon some fish back into the sea. As the
player continues the adventure and interacts
with different objects in a given area, it brings it
back to life by restoring all the fish in that area of Educational Games
the game. The reason the fish were gone was
because they were killed off by a giant evil ▶ Educational Game Abzû and the Lens of Fun
robotic otherworldly being. Once the robot is Learning
destroyed at the end of the game, the entire ▶ Games in Science
ocean and its marine life is fully restored to ▶ Transformational Games
peace and harmony.
Through the lens of fun learning, we ask our-
selves what parts of a game are fun and conducive
to learning and what parts need to be more fun and Educational Simulation
more conducive to learning. In Abzû, a player
learns about the sea and marine life at the ▶ Immersive Technologies for Medical Education
Educational Virtual Reality Game Design for Film and Animation 621

is mostly observed that those were the ones who

Educational Virtual Reality knew the existing concepts and former works very
Game Design for Film and well and therefore were successful in creating
Animation new ones. The educational VR game Cinevoyage
is designed to be a supporting tool for those edu-
Oytun Kal1 and Yavuz Samur2 cational institutions which aim to transfer the gen-
1
Game Design, Bahcesehir University, Istanbul, eral knowledge and skills on cinematic
Turkey storytelling to their students, who might become
2
Computer Education and Instructional the creative artists of the future.
Technologies, Bahcesehir University, The term Digital Educational Game (DEG)
E
Istanbul, Turkey refers to any kind of digital game that is designed
with the purpose of educating the player in a
specific context (Connolly 2012). The design
Synonyms methodologies, the platforms they are built for,
or their genres can differentiate from one DEG to
AR: Augmented reality; DEG: Digital educational another significantly. Another term serious games
game; GBL: Game-based learning; VR: Virtual is also used commonly as an alternative to digital
reality educational games. But serious games imply
mostly business-related simulations with gamified
elements (George 2017). Therefore, DEG is the
Definition preferred term within this article to describe the
educational VR game Cinevoyage and other
This article examines the design methodology of games that inspired its design process.
the educational VR game Cinevoyage, which was Game-Based Learning (GBL) refers to any
designed by the authors, both experienced as kind of learning activity that is supported with
instructors and creators in film and animation game mechanics (Connolly et al. 2008). These
and educational game design industries. games or gamified learning experiences do not
Cinevoyage is designed as an educational tool necessarily have to be in digital formats (Prensky
for filmmaking and animation students with the 2001). In recent years, GBL refers to the usage
aim of enhancing their knowledge and skills on of both DEGs and analog games for educational
cinematic storytelling. In this article the term cin- purposes. These games can differ from each
ematic storytelling refers to telling a story with other in great aspects. Cinevoyage is an attempt
moving images with respect to disciplines like to introduce GBL approach to educational institu-
scriptwriting, film grammar, cinematography, tions that aim to teach cinematic storytelling.
and editing. Thus, any kind of film and digital Virtual Reality (VR) refers to the technology
video making and drawn or computer-generated and the audio-visual products that present an alter-
animations can be considered as a product of native simulated reality to the user. The technol-
cinematic storytelling, if the created content is ogy of VR is rapidly changing but within this
aimed at telling a story in any way, which can research, VR refers to a sensory deception within
create emotional responses in the audience. Edu- audio-visual field by using specifically built head-
cational institutions teaching cinematic storytell- sets. The degree of immersiveness and suspense
ing in any form attempt to build a curriculum of disbelief in VR experiences might differ in
that teaches certain fundamental concepts to great aspects depending on many factors. Some
their students. However, like in any art style, of these factors are the processing power of the
there have always been revolutionary artists who supporting computer systems, the field of view of
broke the existing rules and created new concepts the headsets, or the audio-visual design of the
and methodologies in certain artistic disciplines. It virtual world.
622 Educational Virtual Reality Game Design for Film and Animation

Introduction purpose of increasing the knowledge of the stu-

dents on film history, visual language, mise-en-
As educational technologies become more acces- scène and improving their skills on working with
sible to the masses, more scholars and designers professional cameras, lighting equipment, and
from various disciplines are motivated to design other crew and cast members under real life-like
efficient DEGs in learning environments (Garris shooting conditions?
et al. 2002). However, it is important to keep in
mind that media created for educational purposes
does not guarantee a better learning outcome than Overview
the traditional teaching approaches. The famous
debate between the scholars Clark and Kozma The design methodology of the educational VR
hints that it is the duty of every responsible instruc- game Cinevoyage is constructed from six different
tional designer to ask the question whether the DEG design methodologies. These studies are:
same educational objective can be reached effi- “The FIDGE Model” (Akilli and Cagiltay 2005);
ciently without the use of expensive educational “Narrative GBL objects for story-based DEGs”
media (Clark 1994, pp. 22–23). Wouters et al. (Göbel et al. 2009); “Tools and methods for
(2009) suggested that the learning outcomes, efficiently designing serious game” (Marfisi-
which can be attained with educational games can Schottman et al. 2010); “A New Methodology of
be classified in four different ways. These are cog- Design and Development of Serious Games”
nitive learning outcomes (divided into knowledge (Barbosa et al. 2014); “Design methodology for
and cognitive skills), motor skills, affective learn- educational games based on interactive screen-
ing outcomes and communicative learning out- plays” (Prieto et al. 2015); “Development of
comes. As demonstrated in Bonde et al. (2014), a design heuristics for DEGs for school children of
virtual reality educational game can increase the 7 to 11 Years Old” (Khanana 2016). A summary
motivation and can be effective in achieving these of each methodology and how they were used
learning outcomes if the media usage is combined during the design process of Cinevoyage is pre-
with traditional lectures with instructors (p. 696). sented in the methodologies section.
This article gives an overview of the research, Overall design of Cinevoyage is explained
design, and ongoing development processes of briefly in the research design section, where
an educational VR game Cinevoyage, which is examples of DEGs, educational VR games, appli-
the result of a master’s thesis study from game cations with positive learning outcomes, and
design department. The objective of the game is inspirational cinematic storytelling works were
to increase the knowledge of the students on film also mentioned. During the research process, the
grammar, frame composition, storyboard creation, opinions of scholars and students were asked via
mise-en-scène and to improve their skills on work- an online survey. Their suggestions for designing
ing with professional cameras and lighting equip- an educational VR game for cinematic storytelling
ment under real life-like shooting conditions. For were also evaluated in the findings section.
this purpose, this article reviews the design meth-
odologies and opinions of researchers, filmmakers,
and game designers on how to transfer cinematic Significance of the Study
storytelling knowledge to students in a more effec-
tive and fun way through a DEG. Although there are some existing or underdevel-
opment applications for PC, mobile, and VR
platforms like Cinedesigner, Tvori, Mindshow,
Research Question Source Filmmaker, Film Engine, Vpet, Tiltbrush,
Oculus Medium, Limitless VR, Modal VR, Ani-
What would be the most efficient game design mator by Daydream, all of which might be helpful
solution for an educational VR game, with the in teaching different aspects of cinematic
Educational Virtual Reality Game Design for Film and Animation 623

Educational Virtual
Reality Game Design for
Film and Animation,
Fig. 1 Labster: equipment
price comparison (EdTech
Europe 2014 Innovation
Showcase: Labster)

storytelling, Cinevoyage is probably the first edu- Labster (Fig. 1), which provides a practicing
cational VR game design proposal that is purely area in a virtual science lab with simulated ver-
designed for educational purposes that covers sions of very expensive equipment, Cinevoyage is
many topics from curriculums of film and anima- also designed with a similar purpose for film and
tion schools. Thus, there is a need for further animation schools. With implementing
research and development to increase the effective- Cinevoyage in their curriculum, the educational
ness of educational games for film and animation. institutes can reduce the cost of equipment
Cinevoyage combines useful mechanics and significantly.
design elements from effective examples by intro- Additionally, Cinevoyage can simulate differ-
ducing the student to a fun and engaging learning ent types of filmmaking locations in VR.
environment, which is supported with a narrative Depending on the given tasks in the scenario, the
storyline, where the student can interact with students will be able to practice in studios with
famous filmmakers from film history. The game artificial lights or in exterior locations with natural
design allows the student to learn from film- lights, in day and night conditions or under vari-
makers like Méliès, Welles, Kubrick, Hitchcock, ous weather conditions like rain, wind, and snow.
Kurosawa, Fellini, Trier, Cuaron, etc., by visiting The recent technological developments created
them in the virtual filmmaking environments also new debates like whether VR may be the end
based on the real sets from their works. This of cinema as we know it. Although the number of
might allow the student to experience a simulated VR video productions is increasing every year,
face-to-face lecture by one of her/his role models VR still does not seem to threaten the classical
from the history of filmmaking. Thanks to this film and animation productions as we know it.
interaction, the increasing emotional response by Nevertheless, these new technologies might intro-
the student can boost motivation and strengthen duce new filmmaking methods for the future
the effectiveness of learning. filmmakers. The democratization of technology
One of the main reasons that Cinevoyage is allows many independent creative people from
designed as a VR educational game is that it around the world to design and develop their
may provide a virtual simulated practice environ- own short films, animations, games, and various
ment for film and animation students. The stu- types of applications. We might witness that vir-
dents can increase their theoretical knowledge tual filmmaking can be one of the main methods
and practical skills on certain equipment like pro- of creating cinematic storytelling works in the
fessional film cameras, different types of lighting, future (Morozov 2008). Cinevoyage can also be
and additional supportive grip equipment like considered as an early example that hints what
dollies, cranes, and many more. Like in the exam- kind of virtual filmmaking tools can be used by
ple of the gamified educational VR application future filmmakers. For this purpose, in addition to
624 Educational Virtual Reality Game Design for Film and Animation

the narrative based system with levels, it is con- second, the instructional designer must decide on
sidered to implement a free-to-play area within the the educational subject depending on her/his
game, where the student can act as the director of experience. In the case of Cinevoyage this target
their own scene, where they can choose the equip- group can be considered as film and animation
ment, props, actors, coworkers and the style of the students and the subject can be defined as cine-
material freely in addition to task-based levels. matic storytelling. In the third step the instruc-
The content in Cinevoyage, which consists of tional designer should make a literature review
the equipment, props, characters, virtual loca- to investigate whether the selected educational
tions, and tasks, can be extended depending on content is suitable for a GBL application. If
the development process and the demand in the supporting evidence is found, then designer
long term. The design proposal in this article should specify the pedagogical objectives of the
should be considered as a brief glimpse of what educational game for the selected target group.
Cinevoyage is aimed to become at the end of After this step, the designer should collect data
the continuing iterative design and development from experts within these fields by asking their
process. opinions and recommendations via primary
research tools like surveys or interviews. This
step was followed carefully for Cinevoyage too.
Methodologies Some of the answers to certain open-ended ques-
tions in the survey can be found in the findings
This section of the article reviews six different section. The designer is also responsible in ana-
design methodologies for educational games, all lyzing the technical aspects of the game develop-
of which are listed in chronological order from the ment tools; in the case of Cinevoyage this step
oldest to the newest. Their most strong and should include both the software and the hardware
relevant aspects were analyzed and used as help- side due to the technical requirements of VR
ful resources during the design process of headsets with various alternatives. The designer
Cinevoyage. should also analyze existing games to decide
Although the FIDGE-model design methodol- which genre, platform, and game elements are
ogy was proposed by Akilli and Cagiltay (2005), most suitable for the targeted educational game.
the strategies offered by the researchers in the As explained in the following sections
pre-analysis, analysis, and development phases Cinevoyage is decided to be point and click
were beneficial during the research and design adventure style, narrative VR game with interac-
process of Cinevoyage. The word FIDGE in the tive 3D worlds and characters, which is supported
title stands for “Fuzzified Instructional Design with fantastical elements in harmony with real
Development of Game-like Environment” for historical anecdotes. Table 1 gives a summary of
learning (p. 7). The researchers emphasize the the analysis and design and development
fuzzy logic principle by citing supporting research approaches of the proposed methodology.
about it, which basically refers to nonlinear, Second methodology proposed by Göbel et al.
dynamic approximate and sometimes irrational (2009) is based on an educational game project in
reasoning of human beings. This approach the field of technology-enhanced learning of geog-
allowed the researcher to create a design method- raphy with the aim of harmonizing different features
ology, which is more dynamic and flexible com- and objectives of the storytelling, learning, and
pared to unrealistic strict methodologies, which gaming approaches as seen in Fig. 2. The design
can be time consuming (p. 3). The researchers methodology of the game 80 Days is based on
determined these following issues in the pre- several principles. The designers used the
analysis section, which “addresses the needs of storytelling as an instrument for suspenseful knowl-
novice instructional designers” (Akilli and edge transfer by implicating an emotional
Cagiltay 2005, pp. 9, 15); first the instructional immersive dramaturgy. The gaming aspects of the
designer should decide on her/his target group; teaching material were intended to create a fun,
Educational Virtual Reality Game Design for Film and Animation 625

Educational Virtual Reality Game Design for Film

and Animation, Table 1 Summary of the FIDGE
model (Akilli and Cagiltay 2005)
Issue Its property
Participants All of actively participating learners and
experts
Team Multidisciplinary, multi-skilled, game-
player experience
Environment Socio-organizational, cultural
Process Dynamic, non-linear, fuzzy, creative,
enriched by games’ and simulations’
elements (fantasy, challenge, etc.) E
Change Continuous, evaluation-based
Evaluation Continuous, iterative, formative and
summative, fused into each phase
Management Need for a leader and a well-planned
and scheduled time management
Technology Suitable, compatible Educational Virtual Reality Game Design for Film
Use By (novice/expert) instructional and Animation, Fig. 2 Technology-enhanced learning
designers and educational game (Göbel et al. 2009)
designers for game-like learning
environments and educational games
according to the GameFlow model (Sweetser and
Wyeth 2005) and instructional game design
criteria.
motivational, interactive, and explorative experi- In the fourth study “A New Methodology of
ence. For the learning part, they have emphasized Design and Development of Serious Games”
the assessment and learning success by imple Barbosa et al. (2014) propose a new methodology
menting, motivating, and engaging mechanisms of design and development of educational games,
effectively. To prevent a too linear and non- which are enriched with mini-games for having
interactive plot-based approach in storytelling, the the potential of providing engaging and motivat-
designers have used the Hero’s Journey storytelling ing experiences. This methodology suggests that a
model as a template in a flexible way with variations main game section with missions and independent
they made in the “The Road of Trials” step for tasks like mini games, puzzles, or quizzes serve to
ludological purposes as is seen in Fig. 3 (Göbel transfer knowledge and skill that run in a parallel
et al. 2009). Because of the emphasis on using the way without distracting the player too much from
Hero’s Journey story template in a balanced way, the engaging narrative elements (Barbosa et al.
which allows the player freedom to play instead of a 2014, p. 3). In the light of this study, mini-levels
boring and too linear story structure, this methodol- are implemented in Cinevoyage too (Fig. 5).
ogy convinced the researchers and designers of The fifth methodology study to design educa-
Cinevoyage to write the game script in accordance tional games based on interactive screenplays
with the same template. by Prieto et al. (2015) was again emphasizing
The third methodology study by Marfisi- the importance of creating a detailed scenario in
Schottman et al. (2010) describes the production accordance with educational objectives, which
of a DEGs in seven steps (Fig. 4). For Cinevoyage will allow the player a fun and emergent learning
the first three steps are successfully completed. experience with the support of designed educa-
Currently the researchers and designers are work- tional game. During the script writing process of
ing together to evaluate the software solutions for Cinevoyage, Prieto’s methodology functioned as
building the mechanics and dynamics of the first a strong guide (Fig. 6).
play-testable prototype of the game. Once com- Finally, the dissertation study by Khanana
pleted, the prototype will be evaluated by scholars (2016) demonstrates that the old “GameFlow
626 Educational Virtual Reality Game Design for Film and Animation

Master of Two Worlds

Meeting with Goddess

Crossing 2. Threshold
Crossing 1. Threshold

Refusal of the Return

The Ultimate Boon
The Road of Trials
Refusal of the call
Call to adventure

Freedom ot Live
Normal world

Departure Inititation Return

Educational Virtual Reality Game Design for Film and Animation, Fig. 3 Hero’s Journey story model (left)/linear
and modular story units (right) (Göbel et al. 2009)

1 2 3 4 5 7
6
Specification of Choice of the General Searching for Detailed Specifications
Pedagogical
the pedagogical Serious Game description of software description of for
quality control
objectives model the scenario components the scenario subcontractors

cognitive expert pedagogical expert storyboard writer

graphic designer,
& artistic director
& domain experts sound manager, actors...

Educational Virtual Reality Game Design for Film and Animation, Fig. 4 The seven steps for designing SGs. From
tools and methods for efficiently designing serious games (Marfisi-Schottman et al. 2010)

Educational Virtual
Reality Game Design for LM11 LM21 LMn1
Film and Animation,
Fig. 5 Diagram of the
methodology; a game with
several levels and the
learning mechanisms Level 1 Level 2 Level n
associated to each layer
(Barbosa et al. 2013)

LM1n LM2n LMnn

Educational Virtual Reality Game Design for Film and Animation 627

Educational Virtual Reality Game Design for Film and Animation, Fig. 6 Methodology based on interactive
screenplay (Prieto et al. 2015)

model,” which was structured by Sweetser and goals, good explanations, appropriate chal-
Wyeth (2005) as a design and evaluation tool lenge levels, nondistractive audiovisual charac-
for educational games in accordance with the ters, environment and interface, and
flow theory by Csikszentmihalyi (1990), can informative in-game feedback.
still be very effective in the field of educational
games. The researcher’s main purpose in this
study is to propose a simplified, fun design and Research Design
evaluation methodology for students
7–11 years old. “The gaps of GameFlow Gaming Objectives
model” in this study presents a deep analysis The objective of the game is to start and complete
of the eight elements of GameFlow model, a journey through the history of cinema, while
which are concentration, challenge, player learning the fundamentals of film and animation
skills, control, clear goals, feedback, immer- making. During the game, the player visits various
sion, and social interaction (Khanana 2016, film sets from different epochs of film history,
pp. 88–89). Even though Cinevoyage is not where s/he interacts with famous filmmakers,
designed for this young age group the pilot who give the player various tasks to proceed fur-
study results by Khanana were instrumental in ther to next levels, where the difficulty but also the
making the design decisions like the clear reward values increase.
628 Educational Virtual Reality Game Design for Film and Animation

Pedagogical/Educational Objectives in relation to their curriculum and learning

This game is designed as an educational game for objectives.
film and animation students. Learning occurs by
listening, observing, and doing, while the players Failure States
explore the virtual environments and interact with If the player cannot succeed in completing a mis-
the characters. The difficulty and the complexity sion, s/he can quit that certain mission and prac-
of the knowledge and skills that should be tice with the equipment in a free practice and
acquired with Cinevoyage increase as the player learning area. The storyline allows to switch to a
reaches higher levels by completing given tasks different mission if the player finds it too difficult.
successfully. Normally, the player must complete all the tasks
In Cinevoyage the player should be able to to finish the game; however, the instructor of the
reach the following objectives: educational institute can lead their students to play
certain levels while skipping some other levels
To be able to operate the simulated versions of which are more relevant to their learning
professional cameras, various lighting and grip curriculum.
equipment like dolly, crane, Steadicam, etc., Another important aspect that needs to be
while demonstrating theoretical knowledge regarded carefully is that the current VR technol-
on general cinematography ogy can cause health problems like nausea, blurry
To become more familiar with terms like ISO, vision, or headache if played for very long dura-
shutter speed, aperture, frame per second, tions (Merhi et al. 2007). This aspect of VR tech-
depth of field, color temperature, various nology is also important to remind the students
lenses not to play all levels at once, which might take
To be able to read and understand the following hours. A typical level of Cinevoyage can be
diagrams to complete the tasks: passed in 5–20 min. To prevent playtimes longer
– Mise-en-scène diagram (showing the move- than 30 min, which can be detrimental to the
ment of the actors) health of the student, a time limit is also
– Camera diagram (showing the place, angle implemented in the design. Accordingly, if the
(s), lens, and movement) player fails to pass a level in 30 min, s/he can be
– Lighting diagram (showing the place and declared unsuccessful and can only continue after
type of the equipment) a certain break period.
– Storyboards (showing the framing, shot
scales, camera movement) Game Levels
To be able to work in different filmmaking condi- Research on educational games demonstrates
tions like in interior and exterior locations with that they are most effective if the educational
various lighting and weather conditions, which content is supported with regular lectures (Bonde
can be both natural or artificial et al. 2014). Therefore, it is intended to design
Cinevoyage in such a way that the instructor can
Victory Conditions choose and lead their students to play certain
In Cinevoyage the player interacts with levels in accordance with their curriculum.
characters from film history to learn about the On the other hand, the level design of the game
task/assignment, which s/he must solve. If the is constructed with references to Hero’s Journey
player successfully completes the task, s/he gets story design template. In Cinevoyage the player
immediate feedback from the character and experiences a journey with a guide, which can
gains the artifact, which enables the player to travel beyond time and space. Cinevoyage pro-
continue her/his journey to higher levels of the gresses through several increasingly difficult and
game. As indicated in the “Game Levels” sec- complex levels. In each level the player must
tion, the instructor of the educational institute demonstrate her/his knowledge to perform the
can decide which levels the student should play given tasks successfully.
Educational Virtual Reality Game Design for Film and Animation 629

If the player successfully completes a level, Méliès as the main mentor of the player, who
s/he can travel to the next level. The levels address will be able accompany the player throughout
various learning objectives starting from easy to the levels. His character will reflect his abilities
hard. The player will travel chronologically as an inventor and magician. His invention
through levels, which will begin in the early Cinemagica, which was inspired from actual
periods of film history around 1900s and will Méliès short films, functions as a magical inven-
continue till 2017 and beyond. The complete list tory box that can keep all kinds of equipment and
of levels and the specific educational objectives of necessary tools within, will be with the player
each level are briefly summarized in the Tables 2 through all levels.
and 3. Ongoing development phase for a play-
E
testable Cinevoyage prototype is going to consist Game Narrative
the first five levels. The narrative and visual As explained in the methodologies section, the
details of the first five levels are given in the main story structure of Cinevoyage is based on
narrative, storyboard, and flowchart sections. the Hero’s Journey story template, which is
As explained in the methodologies section, the claimed to be one of the most occurring story
mini-games are also considered as an effective template throughout the human history. As the
way of knowledge transfer and skill improvement methodology examples show, Hero’s journey is
without affecting the game narrative negatively. used efficiently in many educational games by
Therefore, some of the above-mentioned learning allowing the players to go through an emotional
objectives will be achieved with the support of journey, while increasing their knowledge and
mini-games, puzzles, or quizzes. improving their skills. The following storyboard
and flowchart sections give a glimpse of the game
The Characters script for the first five levels.
During the gameplay of Cinevoyage the player
will be able to interact with simulated 3D modeled Storyboard
characters from filmmaking history. In addition to The storyboard template (Fig. 8), which is used to
the above listed famous filmmakers, who will visualize the first five levels of Cinevoyage
both give tasks and instruct the player during the (Figs. 9 and 10), is a combination of two different
game, there will be additional 3D modeled char- templates, which are created by VR developers
acters for cast and crew members. As a part of the McCurley (2016) and Leitch (2017), both are
narrative, the designers decided to use George based on Mike Alger’s presentation with the title

Educational Virtual Reality Game Design for Film and Animation, Table 2 Levels and learning objectives of
Cinevoyage (0–5)
Level Movie Scene Filmmaker Learning Objectives
0_Introduction – Introduction: George Tutorial for basic mechanics to move,
Cinemagica Méliès select & interact
1_The Bridge Photo shooting of Movement Eadweard Fundamentals of photography; exposure,
the horse (1878) Muybridge iso, aperture, shutter speed
2_Action! Arrival of Train at La Arrival Lumière Shot scales, exposure, iso, aperture
Ciotat (1895) Brothers
3_Crosscut The Great Train Train station Edwin Parallel editing, special effects
Robbery (1903) S. Porter
4_Montage Kuleshov a + b1 ¼ x Lev Montage theory
Experiment (1919) a + b2 ¼ y Kuleshov
a + b3 ¼ z
5_Wake Up Dr. Caligari (1920) Cesare’s Robert Lighting, framing
Awakening Wiene
630 Educational Virtual Reality Game Design for Film and Animation

Educational Virtual Reality Game Design for Film and Animation, Table 3 Levels and learning objectives of
Cinevoyage (6–20)
Level Movie Scene Filmmaker Learning objectives
6_Retrofuturo Metropolis City Fritz Lang Studio shooting, lighting, special effects
(1927) miniature
7_The Prodigy Citizen Kane: Childhood Orson Storyboard reading, frame composition,
(1941) scene Welles blocking
8_Eastlight Seven Samurai Flag scene Akira Frame composition, camera movement
(1954) Kurosawa
9_The Master Vertigo (1958) Vertigo Alfred Focal length, camera movement
effect Hitchcock
10_Dreamman 8 1/2: (1963) Asa Nisi Federico Black & white lighting
Masa Fellini
11_End of 2001: A Space Final room Stanley Shot/reverse shot, lighting, colors
Beginning Odyssey (1968) scene Kubrick
12_Houston, we Barry Lyndon Interior, Stanley Zoom, depth of field, lighting, lens
have a Problem! (1975) candlelight Kubrick aperture
13_Depth Stalker (1979) Tunnel Andrei Composition, depth of field, camera
scene Tarkovsky movement
14_Track the Fall Wings of Desire Fall of an Wim Shot scales, tracking shot
(1987) angel Wenders
15_The Bridge Matrix (1999) Bullet time Watchowskis Green screen, special effects, bullet time
effect
16_Location or Dogville (2003) Introduction Lars von Mise-en-scène, camera movement
Not Trier
17_Stopmotion Corpse Bride Meeting Tim Burton Stop motion, frame composition, lighting
(2005) Emily
18_Plansequence Children of Men Car scene Alfanso Advanced plan sequence, scene blocking,
(2006) Cuaron shot scales
19_Back to Station Hugo (2011) Train crash Martin Special effects with green screen
Scorsese
20_Resolution The Revenant Church Alejandro Composition, Imax cameras, crowded
(2016) G. Iñárritu scene experience, snowy weather

“VR Interface Design Previsualisation Methods” the famous characters the player is going to meet
(2015). As shown in Fig. 7 designing for VR and interact; about the references to actual movies
requires to pay attention to certain details like from film history; about the objects that the player
the distances between the objects and the player will acquire and the actions that occur when
to provide a comfortable VR experience for interacting with them; about the vehicles like the
the users, which can be planned during the Cinetrain, with which the player is going to travel
storyboarding process. within the game; about the user interface like the
The storyboard combines the written informa- camera display; and about the tasks that needs to
tion from the narrative script of the game with the be fulfilled to pass a certain level.
visual information that represents a basic version
of how the game world, the characters, and the Flowchart
objects should look and how the player is going The flowchart is another important part of a Game
to interact with them. The storyboard of the first Design Document, which functions as a commu-
five levels of Cinevoyage shown in Figs. 9 and 11 nication platform among the game designers, pro-
gives the information about where in the virtual grammers, and audio-visual asset creators. The
world the first-person player will be located; about flowchart for the first five levels of Cinevoyage
Educational Virtual Reality Game Design for Film and Animation 631

Educational Virtual Reality Game Design for Film and Animation, Fig. 7 VR viewing distance (virtualrealitypop.
com)

Annotations
1 4
God’s Eye View
1 Top down view of whole world
5 6

First Person POV

2 Front-facing view inside HMD
7
First Person POV (180° Rotation)
3 Rear facing view inside IMD
8

4 Main Content Zone∗

Field of view –94°
2
Camera Position
5
4 Dot indicates camera placement centered in world

6 Dividing Line
Dotted line separates Main Content & Curiosity Zones
9 9

7 Curiosity Zone∗
Rear-facing view in God’s Eye View & inside HMD

180° Rotation
8
Arrows indicate rotation to Curiosity Zones
3

9 Peripheral Zone∗
7 Field of view ∼102° (∼204° with maximum head tum)

9 9
8

Educational Virtual Reality Game Design for Film and Animation, Fig. 8 Annotations for VR storyboard template
(virtualrealitypop.com & medium.cinematicvr.org)

in Fig. 11 depicts what the player will be interface that will guide the player to select the
experiencing throughout the gameplay. From relevant menus like the settings, lecture videos, or
this flowchart one can acquire information about the new game menu; about the place and the
the beginning process of the game; about the user number of cut scenes; about the characters the
632 Educational Virtual Reality Game Design for Film and Animation

Educational Virtual Reality Game Design for Film and Animation, Fig. 9 Cinevoyage storyboard (levels 0–2)

player is going to meet and interact within the Thirty-four out of 92 people, who responded the
game; about the given tasks and the free play online survey, claimed that they have experience
modes; about the game objects that should be as an instructor or teacher in film making, video
acquired and what to do with them; about the production, photography, and similar branches.
number and passing conditions of the levels; and Some response examples by these people to
about the main game mechanics like to teleport the open-ended question “In which ways could
from one place to another. you benefit from the educational VR game
Cinevoyage in your lectures?” can be seen in
Table 4.
Findings As is seen Table 4 there are many participants
who emphasized the importance of gaining prac-
The deep literature review for this study enabled tical experience within the simulated filmmaking
the designers to determine which elements of environment. Even the participants, who had little
which design methodologies will be used while or no experience with VR environments can easily
designing Cinevoyage. After a collaborative work imagine how it would benefit the students, when
of game designers and instructional designers, the they can play with virtual filmmaking equipment
finished Game Design Document of Cinevoyage without the fear of making mistakes. Considering
and an online survey was shared with a selected the fact many educational institutes cannot afford
group of people to gather suggestions and recom- the professional level film equipment due to
mendations on the general design of the game. the high costs and security risks Cinevoyage is
Educational Virtual Reality Game Design for Film and Animation 633

Educational Virtual Reality Game Design for Film and Animation, Fig. 10 Cinevoyage storyboard (levels 3–5)

designed with the purpose of enabling the stu- motor skills, perceptual and cognitive skills,
dents to work with virtual filmmaking equipment and physiological outcomes is quasi-
(Sheffield 2001). Additionally, Cinevoyage aims experiment, followed by randomized control
to simulate the social aspects of filmmaking too trials (RCT), surveys, and qualitative designs.
like the interaction with the director or other cast On the other hand, to study the affective, moti-
and crew members on a film set, which was con- vational, and social outcomes of the games,
sidered as an important factor within the survey researchers used mostly surveys even though
results (Table 4). these were typically to secondary interest
(Connolly et al. 2012). To measure the affective
and motivational outcomes of the DEGs some
Conclusion and Discussion researchers used questionnaires. Jennett et al.
(2008) used a subjective questionnaire to distin-
According to the study performed by Connolly guish immersiveness levels of games based on
et al. (2012) the most frequently occurring cognitive involvement, emotional involvement,
outcome with educational games was knowl- real-world dissociation, challenge, control, and
edge acquisition and content understanding. eye movements. Fu, Su, and Yu’s study (2009)
The methods of measuring the outcomes of the found that Sweetser and Wyeth’s GameFlow
educational games differed from study to study. model (2005) can be used effectively in evalu-
The most popular study design for collecting ating the motivational features of educational
empirical data on knowledge acquisition, games.
 634

Educational Virtual Reality Game Design for Film and Animation, Fig. 11 Flowchart for Cinevoyage (levels 0–5)
Educational Virtual Reality Game Design for Film and Animation
Educational Virtual Reality Game Design for Film and Animation 635

Educational Virtual Reality Game Design for Film and Animation, Table 4 Response examples to the survey
question “In which ways could you benefit from an educational VR game in your lectures?”
“It could function as a simulation game. If the class that is being taught needs practical skills, it could be used to
practice that skill and improve on it without worrying about failure. It would allow the user to learn the skill in a safe, but
realistic environment.”
“By creating a simulated interactive environment, you can adjust light, distance, lens, camera height, angle etc. in
comparison with various options.”
“Cinevoyage could function as a simulation game. If the class that is being taught needs practical skills, it could be used
to practice that skill and improve on it without worrying about failure. It would allow the user to learn the skill in a safe,
but realistic environment.”
“By creating an interactive environment simulation with this VR game, students can learn quickly to adjust elements like
the time, light, distance and lens angle, camera height, angle etc. in combination with each other.” E
“Learning camera and lens, editing, light, and even script writing, can become faster and more fun with the help of a VR
game like “Cinevoyage”. The burden of collecting visual material for the lecturer would be less with the support of these
games. In an interactive virtual film studio, the student can feel like a real cameraman, who takes her/his directions from
the director, which would allow her/him to experience theory and the practice at the same time instead of a passive
listening experience”

The first working prototype of Cinevoyage is Moreover, as the study by Zhang and Zhao
planned to be tested within 2018 in some educa- (2017) exemplifies successfully that AR is also
tional institutes. The research and design team is very promising to support education. Therefore,
planning to make both similar quasi- the design team of Cinevoyage is also
experiments with control groups to compare experimenting with technologies like ARkit for
the learning outcomes of cinematic storytelling Apple devices and ARCore for Android
lectures with the support of Cinevoyage versus smartphones and how they can be used for cine-
classical lecture methods. Additionally, the matic storytelling education too.
game will be evaluated both by instructors and
students according to evaluation surveys and
questionnaires like Game Features Test, Instruc- Cross-References
tional Game Survey, and GameFlow model to
collect secondary data on motivational and ▶ Augmented Learning Experience for School
social outcomes, levels of immersion, and Education
engagement of the students during gameplay ▶ Gamification and Serious Games
(Samur 2012). ▶ Interactive Augmented Reality to Support
Cinevoyage is currently under development Education
with a team of script writers, game developers,
3D and instructional designers for room-scale VR
platforms with high-end PC support like Oculus References
Rift, HTC Vive. Additionally, a smaller mobile
VR version for platforms like Samsung Gear VR Akilli, G.K., Cagiltay, K.: An instructional
design/development model for the creation of game-
and Google Daydream are in consideration, which
like learning environments: the FIDGE model. In:
are planned for quick prototyping sessions. As Affective and Emotional Aspects of Human-Computer
can be seen within this article Cinevoyage has a Interaction: Game-Based and Innovative Learning
very wide scope and it can be expanded as the Approaches, vol. 1, pp. 93–112 (2005). Amsterdam,
Netherlands: IOS Press. http://ocw.metu.edu.tr/mod/
demand occurs. Currently, funding opportunities resource/view.php?id=1411
for a sustainable design and development process Alger, M.: Visual design methods for virtual reality. https://
of Cinevoyage are being investigated by the drive.google.com/file/d/0B19l7cJ7tVJyRkpUM0hVYmx
researchers too. JQ0k/view (2015). Accessed 20 Nov 2016
636 Educational Virtual Reality Game Design for Film and Animation

Barbosa, A.F.S., Pereira, P.N.M., Dias, J.A.F.F., Silva, G.M.: medium.cinematicvr.org/a-storyboard-for-virtual-reali

A new methodology of design and development of ty-fa000a9b4497#.3jkh3n6ml (2017). Accessed 5 Jan
serious games. Int. J. Comput. Games Technol. 2014, 2017
Article ID 817167, 8p (2014). https://doi.org/10.1155/ Marfisi-Schottman, I., George, S., Tarpin-Bernard, F.:
2014/817167 Tools and methods for efficiently designing serious
Bonde, M.T., Makransky, G., Wandall, J., Larsen, M.V., games. Paper presented at the 4th Europeen conference
Morsing, M., Jarmer, H., Sommer, M.O.A.: Improving on games based learning ECGBL2010, Copenhagen,
biotech education through gamified laboratory simula- Denmark. Retrieved from http://free.iza.free.fr/articles/
tions. Nat. Biotechnol. (2014). https://doi.org/10.1038/ Marfisi_ECGBL_2010.pdf (2010 Oct)
nbt.2955 McCurley, V.: Storyboarding in Virtual Reality [online].
Clark, R.E.: Media will never influence learning. https://virtualrealitypop.com/storyboarding-in-virtual-
Educ. Technol. Res. Dev. 42(2), 21–29 (1994). https:// reality-67d3438a2fb1#.it5h6q6w6 (2016). Accessed
doi.org/10.1007/BF02299088 12 Dec 2016
Connolly, T.M., Stansfield, M.H., Hainey, T.: Develop- Merhi, O., Faugloire, E., Flanagan, M., Stoffregen, T.:
ment of a general framework for evaluating games- Motion sickness, console video games, and head-
based learning. In: Proceedings of the 2nd European mounted displays. Hum. Factors. 45(9), 920–935
Conference on Games-Based Learning. Universitat (2007)
Oberta de Catalunya, Barcelona (2008) Morozov, A.: Machinima learning: prospects for teaching
Connolly, T.M., Boyle, E.A., MacArthur, E., Hainey, T., and learning digital literacy skills through virtual film-
Boyle, J.M.: A systemic literature review of empirical making. In: Luca, J., Weippl, E. (eds.) Proceedings of
evidence on computer games and serious games. EdMedia: World Conference on Educational Media
Comput. Educ. 59(2), 661–686 (2012). https://doi. and Technology 2008, pp. 5898–5907. Vienna, Aus-
org/10.1016/j.compedu.2012.03.004. Retrieved from tria: Association for the Advancement of Computing in
https://www.researchgate.net/publication/230628116 Education (AACE) (2008). https://www.learntechlib.
Csikszentmihalyi, M.: Flow: The psychology of optimal org/p/29201/
experience. Harper & Row, New York (1990). http:// Prensky, M.: Digital Game-Based Learning. McGraw-Hill,
www.bates.edu/purposeful-work/files/2015/03/Csiksz New York (2001)
enthmihalyi-1990.pdf Prieto, R., Medina-Medina, N., Patricia Paderewski, P.,
Fu, F.-L., Su, R.-C., Yu, S.-C.: EGameFlow: a scale to Gutiérrez: Design methodology for educational games
measure learners’ enjoyment of e-learning games. based on interactive screenplays. Paper presented at
Comput. Educ. 52, 101–112 (2009) Cosecivi 2015, Barcelona, Spain (2015). Retrieved from
Garris, R., Ahlers, R., Driskell, J.E.: Games, motivation, https://www.researchgate.net/publication/279763791
and learning: a research and practice model. Simul. Samur, Y.: Measuring Engagement Effects of Educational
Gaming. 33(4), 441–467 (2002) Games and Virtual Manipulatives on Mathematics.
George, P.: Gamification and serious games. Encyclopedia Doctoral dissertation, Virginia Polytechnic Institute
of Computer Graphics and Games, pp. 1–4 (2017). and State University. Retrieved from http://scholar.lib.
https://link.springer.com/referenceworkentry/10.1007/ vt.edu/theses/available/etd-05072012-185722/unrestri
978-3-319-08234-9_90-1 cted/Samur_Y_D_2012.pdf (2012)
Göbel, S., Rodrigues, A.deC., Mehm, F., Steinmetz, R.: Sheffield, S.L.: Streetwise: rethinking motion picture arts
Narrative game-based learning objects for story-based education. J. Film Video. 53(1), 20–24 (2001)
digital educational games. In: Michael, D. (ed.) Sweetser, P., Wyeth, P.: GameFlow: a model for evaluating
Kickmeier-Rust: Proceedings of the 1st International player enjoyment in games. ACM Comput. Entertain.
Open Workshop on Intelligent Personalization and 3(3), Article 3A (2005)
Adaptation in Digital Educational Games, pp. 43–53 Wouters, P., van der Spek, E., van Oostendorp, H.: Current
(2009), Graz, Austria. http://ftp.kom.tu-darmstadt.de/ practices in serious game research: a review from a
papers/GdMS09_533.pdf learning outcomes perspective. In: Connolly, T.M.,
Jennett, C., Cox, A.L., Cairns, P., Dhoparee, S., Epps, A., Stansfield, M., Boyle, E.A. (eds.) Games-Based Learn-
Tijs, T., et al.: Measuring and defining the experience of ing Advancements for Multi-Sensory Human Com-
immersion in games. Int. J. Hum. Comput. Stud. 66(9), puter Interfaces: Techniques and Effective Practices.
641–661 (2008) https://doi.org/10.4018/978-1-60566-360-9.ch014. IGI
Khanana, K.: Development of Design Heuristics for Global, Hershey and London (2009). http://www.cs.
Digital Educational Games for School Children of uu.nl/docs/vakken/b3elg/literatuur_files/Wouters.pdf
7 to 11 Years Old. Doctoral dissertation, University of Zhang, Y. & Zhao Q.: Interactive augmented reality to
Leicester. Retrieved from https://lra.le.ac.uk/bitstream/ support education. Encyclopedia of Computer
2381/37526/1/2016KHANANAKPhD.pdf (2016) Graphics and Games, pp. 1–8 (2017). https://link.
Leitch, A.: A Storyboard for Virtual Reality. But What springer.com/referenceworkentry/10.1007/978-3-319-
Does a Storyboard Look Like in VR?. https:// 08234-9_82-1
EEG as an Input for Virtual Reality 637

Introduction
Edutainment
Ability to move objects with the mind has always
▶ Domain-Specific Choices Affecting Design been an intriguing subject for humans. The idea of
Effort in Gamification being able to interact with our environment with-
▶ Public Health Education via Computer Games out direct physical contact has been at the center
of the works of fiction as well. The concept of
Force in Star Wars film series is an excellent
example of it, which helps certain characters
who can use it to perform telekinesis (Bouzereau
E
EEG as an Input for Virtual 1997). Despite that any parapsychological effort
Reality has failed to show any credible real-world evi-
dence for the practice of telekinesis, which is
Oğuz Orkun Doma based on illusion, science has finally developed
Architectural Design Computing, Istanbul new brain–computer interface (BCI) technologies
Technical University, Istanbul, Turkey that allow humans to interact with their environ-
ment solely concentrating with their minds. Elec-
troencephalography (EEG) is one of these BCI
Synonyms technologies. EEG is monitoring the electrical
activity of the brain, measuring voltage fluctua-
BCI, brain–computer interface; EEG, electroen- tions from electrodes placed on the scalp (Fisch
cephalography; Virtual reality and Spehlmann 1999). EEG has been studied
widely as a part of neurology, psychology, and
marketing studies, and it is commonly used in the
industry fields like education, rehabilitation,
Definitions entertainment, and user experience design
(Wolpaw et al. 2002; Finkelstein et al. 2010;
Brain-computer interfaces (BCI) use electro- Mulert and Lemieux 2009; Friedman 2015).
physiological measures of brain functions to With the recent developments in wearable tech-
send inputs to a computer from a new non-mus- nologies and EEG headsets, they became more
cular channel (Wolpaw et al. 2002). In this entry, affordable and available for casual end users.
the use of electroencephalography (EEG) is Therefore, EEG is increasingly used as a swift
introduced as a BCI input for virtual reality and unique BCI input, which enables user inter-
(VR). AVR mini-game is developed to showcase action via brain waves or “thoughts” in virtual
the use of EEG as an input in VR. With EEG, environments (Li et al. 2017).
using the predefined brainwave patterns that are
defined via EEG as a set of commands, users can
interact with their environments in VR dynami- VR and EEG
cally, through the changes in their emotional
mood and concentration. This does not only Virtual reality is a different realm where all the
introduce an alternative input for VR, in which experience takes place beyond the physical realm.
the physical body’s integration is somewhat Thanks to the current persuasively photorealistic
restricted due to the technological limitations, CGI technologies, increasing processing speed,
but also enables authentic digital realm experi- improved refresh rate on displays, and affordabil-
ences which would be impossible in the physical ity of the user hardware, virtual reality has been a
world. popular subject once again (Jerald 2015). Every
638 EEG as an Input for Virtual Reality

day more and more VR headsets are being sold, (Baudrillard 1999). By using the authentic new
and more VR content is being developed. There- opportunities of this hyperreality, it is possible to
fore, VR is far beyond being an experimental represent the human body in the virtual space in
technology but becoming an essential part of novel ways that it cannot be represented in the
how we interact with the digital realm. Several physical space and enable novel interactions that
universities and laboratories have used EEG as a are not possible in the physical reality. It also
control input for virtual environments and video delivers more enhanced inputs faster into the vir-
games successfully (Lecuyer et al. 2008). Using tual environments (Li et al. 2017). While bodily
EEG in VR makes the experience even more existence in VR is a critical one, using brain waves
immersive. as an interface between the brain and the virtual
Transferring the user’s inputs and movements to space will enable manipulation, interpretation, and
VR as comprehensive as possible ensures an recreation of that space only using the brain itself.
immersive experience. These inputs are mostly In the project that is mentioned in this article,
within the possibilities of what humans already EEG was used to create a dynamic interaction via
have in the physical world: movements, control- neural controls for a VR video game experience.
lers, haptics, eye tracking, and vocal commands, to The players can dynamically interact with the VR
name a few. This gap between physical and virtual environment with the changes in their emotions
adds to the ambiguous manifold of reality and and mental concentration.
virtual realities. This critical duality also leads to
new opportunities, which weren’t available before
the digitally manufactured realities. Virtual does Method
not necessarily need to be an epigone of the reality.
As Baudrillard put it, simulation creates its own This project makes use of two technologies: VR
hyperreality, beyond its referential reality and EEG. The player experiences the VR video

EEG as an Input for Virtual Reality, Fig. 1 Brain-Computer Interface (EEG) and Virtual Reality pipeline of the project
EEG as an Input for Virtual Reality 639

game created in CRYENGINE via Oculus Rift color for the environmental lighting in the VR
VR headset. The core mechanic of the designed game.
gameplay is simple; the player walks through a In a previous study, users wearing Emotiv EEG
segmented tunnel, trying to elevate each segment headset were able to drive an automobile with
to form a flight of stairs to reach a higher platform. EEG commands (Göhring et al. 2013). Therefore,
Meanwhile, the interior lighting of the tunnel the precision of EEG is considered to be well
changes based on the player’s emotions (Fig. 1). enough as a controller input for the VR game.
Emotiv EPOC+ EEG headset monitors the
player’s brain activities and digitizes it as a com-
puter input. Emotiv’s dedicated software EPOC Results and Conclusion E
Control Panel enables certain neural patterns,
which the player needs to define it beforehand, Using EEG as a brain–computer interface for VR
to be read as an input for the VR video game. In provides a new range of unique possibilities and
this project, the players define EEG commands interactions, which would be impossible in the
such as push/pull and resize objects in EPOC physical world. Making a user move an object
Control Panel. As seen in Fig. 2, these commands with EEG in the physical world would require a
are used as controller inputs by CRYENGINE. lot more effort than moving some object in a video
EPOC Control Panel also streams the excitement, game in VR. Also, wearing EEG headsets is not
engagement, and meditation levels of the player to very fashionable yet, but it is less bothersome
CRYENGINE, which is converted into an RGB when the user is already tethered in

EEG as an Input for Virtual Reality, Fig. 2 The players elevate each segment by concentrating on the push/pull
commands they have defined
640 EEG Signal

VR. Considering the existing bodily integration Lecuyer, A., Lotte, F., Reilly, R.B., Hirose, M., Slater, M.:
and movement limitations of virtual reality tech- Brain–computer interfaces, virtual reality, and video-
games. Computer. 41(10), 66–72 (2008)
nologies, figuratively connecting the brain Li, S., Leider, A., Qiu, M., Gai, K., Liu, M.: Brain-based
directly to the computer by the use of EEG as an computer interfaces in virtual reality. In: 2017 I.E.
input for VR looks very promising for the future 4th International Conference on Cyber Security and
applications. Cloud Computing, pp. 300–305. CSCloud,
New York (2017)
Mulert, C., Lemieux, L.: EEG–fMRI: Physiological
Basis, Technique, and Applications. Springer, Berlin/
Heidelberg (2009). https://link.springer.com/book/10.
Cross-References 1007/978-3-540-87919-0#about. Accessed 28 May
2016
▶ Biosensing in Interactive Art: A User-Centered Wolpaw, J. R., Birbaumer, N., McFarland, D. J.,
Pfurtscheller, G., & Vaughan, T. M.: Brain-computer
Taxonomy interfaces for communication and control. In: Clinical
▶ Brain Signals as a New Biometric Authentica- neurophysiology, 113 (6), pp. 767–791. (2002).
tion Method Using Brain-Computer Interface
▶ Emotion-Based 3D CG Character Behaviors
▶ Foundations of Interaction in the Virtual Real-
ity Medium
▶ History of Virtual Reality EEG Signal
▶ Locomotion in Virtual Reality Video Games
▶ Spatial Perception in Virtual Environments ▶ Brain Signals as a New Biometric Authentica-
▶ Virtual Reality Game Engines tion Method Using Brain-Computer Interface
▶ Color Detection Using Brain Computer
Interface
References

Bouzereau, L. (ed.): Star Wars: The Annotated Screen-

plays. Titan Books, London (1997)
Baudrillard, J.: Simulacra and Simulation. Michigan EEG, Electroencephalography
University Press, Michigan (1999)
Finkelstein, S., Nickel, A., Barnes, T., Suma, E.:
Astrojumper: motivating children with autism to exer- ▶ EEG as an Input for Virtual Reality
cise using a VR game. In: CHI’10 Extended Abstracts
on Human Factors in Computing Systems,
pp. 4189–4194. ACM, New York (2010)
Fisch, B.J., Spehlmann, R.: Fisch and Spehlmann’s EEG
Primer: Basic Principles of Digital and Analog EEG. Electronic Sports
Elsevier Health Sciences, Amsterdam (1999)
Friedman, D.: Brain-computer interfacing and virtual real-
ity. In: Nakatsu, R., Rauterberg, M., and Ciancarini, ▶ Diversity in Gaming and the Metaverse
P. (eds.) Handbook of Digital Games and Entertainment ▶ Online Gaming Industry Evolution, Monetiza-
Technologies, pp. 1–22. Springer, Singapore (2015). tion, and Prospects
https://link.springer.com/referenceworkentry/10.1007/
978-981-4560-52-8_2-1. Accessed 10 Nov 2017
Göhring, D., Latotzky, D., Wang, M., Rojas, R.: Semi-
autonomous car control using brain–computer inter-
faces. In: Lee S., Cho H., Yoon KJ., Lee J. (eds.)
Intelligent Autonomous Systems 12. Advances in Intel- Embodied Agent - Autonomous
ligent Systems and Computing, vol 194. Springer, Ber- Digital Character
lin/Heidelberg (2013). https://link.springer.com/
chapter/10.1007/978-3-642-33932-5_37. Accessed
28 May 2016
▶ Interacting with a Fully Simulated Self-
Jerald, J.: The VR Book: Human-Centered Design for Balancing Bipedal Character in Augmented and
Virtual Reality. Morgan & Claypool, New York (2015) Virtual Reality
Emotion in Games 641

Introduction
Embodiment
Emotion is how we feel and how connected we get
▶ Gamification and Social Robots in Education to someone or something. In life we cry over the
ones we have lost or if we are mad we will be
angry at life or at someone. Believe it or not, some
people who play video games can get emotional
on a character’s death or dying so much that they
Embodiment in Digital Games want to break the game controller. (Isbister, 2016)
Character growth is what most players like to
E
▶ Player-Avatar Link: Interdisciplinary Embodi- see in an intense story-driven game. Some exam-
ment Perspectives ples of games that show emotion well are Halo,
Mass Effect, Red Dead Redemption, Call of Duty,
and Gears of War. These games present great
character growth but they also let their favorite
characters die. For example, Dom, in Gears of
Emotion Detection War, is the brother in arms who dies in the third
game, sacrificing himself to save his friends and to
▶ Facial Recognition and Emotion Detection in be with his family. In Call of Duty, many memo-
Environmental Installation and Social Media rable likable characters die, including Soap,
Applications Ghost, Sandman, Harper, and Mason. In Mass
Effect, we can build our own character from
body and facial features, and transfer them to
other mass effect games so we can continue the
adventure. However, the character ends up dying
Emotion in Games at the end of the third game, causing trauma for
some players.
Ryan Hilderbrand2, Sam Romershausen2 and Here are some examples of emotion in games:
Newton Lee1,2 (Freeman, 2004; Yannakakis & Paiva, 2014)
1
Institute for Education, Research, and Emotion: Fear.
Scholarships, Los Angeles, CA, USA A player is playing a game and hears sounds
2
Vincennes University, Vincennes, IN, USA that cannot be explained, shadows in the distance,
or have a restrictive camera view.
Emotion: Joy.
Synonyms A player gets a gun or armor piece that is
extremely valuable as a reward for completing a
DLC difficult trial; they are probably going to be filled
with joy.
Emotion: Panic.
Definition Games can induce panic in ways that are tied to
story in gameplay. The second chapter of Super
Emotion in games – a feeling such as happiness, Paper Mario has the player exploring a mansion
love, fear, anger, or hatred while a person is for Merlee, an important character. After a few
playing a computer game. mishaps while exploring the estate, including get-
DLC – downloadable content, often called ting thrown in a dungeon by Merlee’s assistant,
DLC, is extra content made for a game Mario and company search the basement to find
after release. It can be given out freely or an imposter Merlee. While some players could
monetized. assume from context clues that Merlee’s assistant
642 Emotional Congruence in Video Game Audio

was the imposter, players could not predict that synthesis and signal processing, and automated
the assistant, who calls herself Mimi, would repurposing of existing music matching the
reveal her true form by having her head spin expected emotion of the players.
around her neck and sprouting spider legs.
From here, Mimi chases Mario throughout the
basement until he can find the real Merlee. Cross-References
Despite having a creepy appearance, Mimi is
also threatening: She is invincible, can shoot ▶ Emotion-Based 3D CG Character Behaviors
projectiles, and can follow players through
doors. This scenario changes a platformer
where a character can take their time into a situ- References
ation where players need to think fast to solve
puzzles to get to the end of a maze. Freeman, D.: Creating emotion in games: the craft and art
of emotioneering™. Computers in Entertainment
Emotion: Frustration.
(CIE). 2(3), 15–15 (2004)
Some games are designed to make the player Isbister, K.: How Games Move Us: Emotion by Design.
frustrated with difficult challenges. Some players Mit Press (2016)
play for the clout of being able to make it through Williams, D., Lee, N.: Emotion in Video Game
Soundtracking. Springer (2018)
a difficult level.
Yannakakis, G. N., Paiva, A.: Emotion in games. Hand-
Emotion: Sadness. book on affective computing, 459–471, (2014)
A game that does this well is Destiny 2 with its
DLC, Forsaken. In the base campaign, the player
gets the feeling of hopelessness. The central hub
that was enjoyed in the main game is suddenly
under fire and can no longer be accessed, aiming Emotional Congruence in
to make the player feel uncomfortable. Video Game Audio
Emotion: Vengeance.
Vengeance is not always so predictable. In Duncan A. H. Williams1, Peter I. Cowling1 and
Forsaken, a main character from the Vanguard, Damian T. Murphy2
1
Cayde 6, dies by the hands of Uldren Sov. The Digital Creativity Labs, Department of
goal from thereon out is to kill Uldren Sov by Computer Science, University of York, York, UK
2
hunting him down the entire campaign. The Department of Electronic Engineering,
player does get a feel for vengeance but the hole University of York, York, UK
that Cayde 6 has left for the player will never be
repaired.
Emotion Driven by Audio and Music. Synonyms
The emotional impact of audio and music
cannot be understated. (Williams & Lee, Affect; Music
2018) Combining images with audio enhances
the impact of a multimodal experience during
gameplay. Unlike traditional composition such Definition
as film music, game music mirrors the nonlinear
narrative of gameplay. Player-dependent Video game audio is more challenging in many
actions can change the narrative and thus the regards than traditional linear soundtracking.
emotional characteristics required in the Soundtracking can enhance the emotional impact
soundtrack. Video games uses various tech- of gameplay, but in order to preserve immersion, it
niques such as algorithmic composition, auto- is important to have an understanding of the
mated emotion matching from biosensors, mechanisms at work when listeners respond to
motion captures, emotionally targeted speech audio emotionally.
Emotional Congruence in Video Game Audio 643

Introduction of emotional responses to music. A popular model

is the two-dimensional (or circumplex) model of
Video game soundtracking presents a number of affect (Russell 1980). This plots positivity
unique challenges in contrast to traditional linear (valence) on a horizontal axis and activation
soundtracking (e.g., in television or film). Many strength (or arousal) on the vertical axis. Thus, a
solutions are in use to address the most common player state with high valence and low arousal
problem: dynamic soundtrack creation in might be described as calm, peaceful, or simply
response to gameplay action, but these often happy. This approach has the advantage of being
approach the problem from the point of view of, able to quantify the emotional response – we can
for example, procedural audio techniques (Collins have a sliding scale for both axes and perhaps a
E
2009). One of the major reasons to include particular emotional coordinate for a player at a
soundtracking is to enhance the emotional given point in the narrative of the game. This
response of the player, for example, to accentuate approach also facilitates some direct mapping to
danger, success, failure, and other elements of a given soundtrack which matches the player
gameplay (Berndt 2011). Depending on the type state. However, this type of model is
of game, there may be established musical gram- problematic. For example, let us consider a state
mars to convey such things, and thus emotional which is very negative and also very active (low
congruence is vitally important in maintaining valence and high arousal). How might a player in
player immersion (Arsenault 2005). Defining such a condition be described – afraid? Or angry?
what we mean by “emotion” is important here, Both are very active, negative states, but both are
as perceptual science often refers to a number of quite different types of emotional response. So
synonymous terms (including mood and affect). three-dimensional models of emotion have also
These are sometimes distinguished between been proposed, for example, including dominance
according to the length and specifics of the feel- as a third axis (in which case, afraid might be a
ing, with mood being longer lived, for example passive response, at one end of the scale in the
(Williams et al. 2014). If a player is in danger, do third axis, and angry would be the dominant
they feel threatened, excited, afraid, or angry? Are response at the opposite end of the same scale)
any of these terms quantifiable (is there a medium (Mehrabian 1996). This three-dimensional model
level of fear that might be increased as the level of and models like it have also come under criticism
danger in the game increases?) and are they dis- when adapted to use with musical stimuli, and
tinct and in a linear relationship to one another multidimensional, music-specific models have
(if fear increases, will excitement also increase?)? recently been used (Scherer 1995).
These are difficult questions to answer but of vital
importance for the game sound designer if suc-
cessful emotionally congruent soundtracking is to Challenges and Future Technology
be achieved. For our purposes we will consider
terms including affect, mood, and emotion as One of the most important issues when consider-
being interchangeable, and rather than referring ing emotional congruence in video game
to affective states or moods, we will refer simply soundtracking is the distinction between an emo-
to emotion or emotional responses. tion which the music communicates and an emo-
tion which the player actually feels (Gabrielsson
2002). Imagine that the player is in a part of the
How Is Emotion in Music Measured? narrative which requires generically happy sound-
ing music. The tune which is selected by the audio
There are established methods for evaluating engine might resemble circus music, but in this
emotional responses in traditional psychology particular case, the player has a phobia of the
and cognitive sciences (Schubert 1999). These circus and in particular of clowns. The music
can be, and have been, adapted to the evaluation may then create the opposite of the intended
644 Emotional Congruence in Video Game Audio

emotional state in the player. Similarly, there is a the Indiana Jones series and, perhaps most
growing body of evidence which suggests that famously, the Monkey Island series of games).
when sad music is played to a listener who are in iMuse implemented two now commonplace solu-
a similar emotional state, the net effect can actu- tions, horizontal re-sequencing and vertical
ally be that the listener’s emotional response is re-orchestration, both of which were readily
positive, due to an emotional mirroring effect implementable due to the use of MIDI orchestra-
which releases some neurosympathetic responses tion as a structural representation of the music
(Molnar-Szakacs and Overy 2006). There is some soundtrack, rather than a definitive (i.e., recorded
research suggesting that music has the power to be and rendered) digital audio file.
perceived as a sympathetic listener and to make In the future, we might see an optimized solu-
people in negative emotional states feel “listened tion, combining machine learning approaches to
to.” Therefore, giving generically sad music to the composition with an individual’s own selection of
player at a particular point in the narrative might music or the use of biophysiological measures of
also be inappropriate. Beyond this, almost every- emotion to manipulate a soundtrack to best max-
one has slightly different tastes in music, includ- imize the intended, induced emotional response in
ing preferences for certain genres, performers, and an individual gamer on a case-by-case basis.
even specific songs (Kreutz et al. 2008). These These solutions sound far-fetched at the time of
individual differences are very challenging for the writing, but due to the increase in wearable
game audio designer to account for, but the biosensing technology, and the ever-decreasing
greatest challenge remains that of adaptability to cost of more complicated associated technology
nonlinear narrative changes (changes under the (facial recognition, electroencephalography, gal-
control of the player or other game agents). vanic skin response), such technology may well
Early solutions such as looping material can become commercially viable in the world of game
become repetitive and ultimately break player audio soon.
immersion. Branching strategies, wherein differ-
ent music cues are multiplexed at narrative
breakpoints, can drastically increase the composi- References
tional complexity required in the audio design
strategy (Lipscomb and Zehnder 2004). An alter- Arsenault, D.: Dark waters: spotlight on immersion. Game-
On North America 2005 Conference Proceedings,
native might be to apply procedural audio tech-
pp. 50–52 (2005)
niques, which have been used with varying Berndt, A.: Diegetic music: new interactive experiences.
degrees of success in game sound effect sequenc- Game Sound Technology and Player Interaction Con-
ing and design. However, for music tasks, the cepts and Development, pp. 60–76 (2011)
Collins, K.: An introduction to procedural music in video
computational cost involved in procedural gener-
games. Contemp. Music. Rev. 28(1), 5–15 (2009)
ation can be large – for example, if long sequences Gabrielsson, A.: Emotion perceived and emotion felt: same
of pre-rendered audio are required for particular or different? Music. Sci. 5(1 Suppl), 123–147 (2002)
narrative sequences with various nonlinear prop- Kreutz, G., Ott, U., Teichmann, D., Osawa, P., Vaitl, D.:
Using music to induce emotions: influences of musical
erties, in response to gameplay and intended emo-
preference and absorption. Psychol. Music. 36(1),
tional state. Such solutions can also require a great 101–126 (2008)
deal of compositional complexity (with only lim- Lipscomb, S.D., Zehnder, S.M.: Immersion in the virtual
ited savings in this regard over branching strate- environment: the effect of a musical score on the video
gaming experience. J. Physiol. Anthropol. Appl. Hum.
gies) but have been used successfully in some Sci. 23(6), 337–343 (2004)
instances, for example, LucasArts iMuse system Mehrabian, A.: Pleasure-arousal-dominance: a general
(Strank 2013) which was a dynamic music framework for describing and measuring individual
streaming system which initially used MIDI files differences in temperament. Curr. Psychol. 14(4),
261–292 (1996)
with varying degrees of transformation to imply
Molnar-Szakacs, I., Overy, K.: Music and mirror neurons:
characters and emotional states. This system was from motion to‘e’motion. Soc. Cogn. Affect. Neurosci.
used to accompany role-playing games (including 1(3), 235–241 (2006)
Emotion-Based 3D CG Character Behaviors 645

Russell, J.A.: A circumplex model of affect. J. Pers. emotions in virtual space. This article espe-
Soc. Psychol. 39(6), 1161 (1980) cially focuses on the intelligent agents commu-
Scherer, K.R.: Expression of emotion in voice and
music. J. Voice. 9(3), 235–248 (1995) nicating with a human, i.e., an agent as an
Schubert, E.: Measuring emotion continuously: validity intelligent user interface with abilities to under-
and reliability of the two-dimensional emotion-space. stand human’s emotions. This topic contains
Aust. J. Psychol. 51(3), 154–165 (1999) diversified research fields: Intelligent Agent,
Strank, W.: The legacy of iMuse: interactive video game
music in the 1990s. Music Game, pp. 81–91 (2013) Intelligent Virtual Environment, and Affective
Williams, D., Kirke, A., Miranda, E.R., Roesch, E., Daly, Computing.
I., Nasuto, S.: Investigating affect in algorithmic com-
position systems. Psychol. Music. 43, 831–854 (2014)
E
Introduction

Nowadays, the user interface connecting a

Emotional Contagion human to a computer has come to play a more
important role in our daily life. Since the user
▶ Simulation of Emotional Crowd and interface takes multimodal styles, such as voice
Applications input/output interface and haptic feedback inter-
face, many researches about Human Computer
Interaction (HCI) (Wong and Horace 2008) have
been actively made. Especially, Intelligent
Emotional Intelligence User Interface (IUI) (Sullivan and Tyler 1991)
which can sophisticatedly interact with a human
▶ Animation and Neurocinematics: Visible Lan- by using Artificial Intelligence (AI) technologies
guage of E-motion-S and Its Magical Science or Computational Intelligence (CI) technologies
has a role to promote human-computer commu-
nications. IUI with an ability to understand
human’s emotions enables to make their rela-
Emotion-Based 3D CG tionships more closely and friendly. On the
Character Behaviors other hand, the researches of Intelligent Agent
in a virtual environment also have developed
Kosuke Kaneko1 and Yoshihiro Okada2 with AI and Artificial Life (AL) fields. Non-
1
Cyber Security Center, Kyushu University, Player Character (NPC) in a video game is an
Fukuoka, Japan appropriate example as the intelligent agent.
2
Innovation Center for Educational Resource, This article focuses on researches about an intel-
Kyushu University Library, Fukuoka, Japan ligent agent with feelings and emotions as IUI
and introduces theories and technologies related
to them.
Synonyms This article is organized as follows. The next
section introduces popular researches about
Affective computing; Intelligent argent; Intelli- emotional theories and emotion models which
gent user interface; Intelligent virtual environ- are frequently referred by Affective Computing
ment; Human-computer interaction (Picard 1995) researches. Then, in the Section of
“Intelligent Agents with Feelings and
Emotions”, several related researches about
Definition intelligent agents with feelings and emotions
are introduced, e.g., Intelligent Virtual Environ-
Emotion-based 3D CG character behaviors are ment (IVE), emotion recognitions, and so
the various actions of intelligent agents with on. Afterward, Section 4 explains future visions
646 Emotion-Based 3D CG Character Behaviors

and state-of-the-art technologies about Affective vertical dimension describes intensity of the
Computing researches. The section especially primary emotions: the upper side means a high
focuses on Deep Learning technologies, which intensity state of a primary emotion and the
is one of the hot topics in AI and CI fields, and lower one means a low intensity state of it. As
considers its possibilities to be adopted to the different emotion theories from these, there are
intelligent agent. Finally, Section 5 concludes another approach to represent emotions as the
this article. relationship between a physical response and
mental feeling. Several different theories, e.g.,
James-Lange’s theory, Cannon-Bard’s theory,
Emotion Theories and Emotion Models and Schacter-Singer’s Two-factor theory, are in
the concept but the theories would be able to be
Emotion theories and emotion models are divided into two concepts: Appraisal Theory
discussed from viewpoints of various research (Emotion Drives Behavior) or Constructivist
fields. One of the dominant theories is Discrete Theory (Behavior Drives Emotion). In the case
Emotion Theory. The theory considers emotions that a human encounters a bear, in Appraisal
as discrete elements. One of the well-known Theory, he trembles because he felt fear; mean-
researches in this theory is Ekman et al. while, in Constructivist Theory, he feels fear
(1982). They classified human’s emotions into because he trembled. The OCC (Ortony, Clore,
six discrete categories, Joy, Sadness, Anger, and Collins) model represents these relation-
Fear, Disgust, and Surprised, based on their ships between emotions and behaviors (Ortony
facial expression researches. As another model et al. 1998). Because the model is possible to
of categorizing emotions, Parrott represented predict an appropriate emotion in given situa-
emotions as a tree-structured list which consists tions, the most of researches about agents’
of three types of categories: Primary emotion, behaviors use this model for implementing
Secondary emotion, and Tertiary emotions agents’ emotion and behavior models. Figure 2
(Parrott 2001). The upper image in Fig. 1 depicts the OCC model. The model describes a
describes the list. Another dominant theory is hierarchy structure and 22 types of emotion
Dimensional Emotion Theory. The theory con- categories. The hierarchy has three branches
siders emotions as a combination of several about Events, Agents, and Objects in its situa-
psychological dimensions. Therefore, each tion. The model explains why an emotion occurs
emotion is not separated but continuously allo- from the situation by retrieving these branches
cated on two or three dimensions. Posner et al. according to its situations. For example, con-
introduced emotions represented as a two- sider a situation where there are two agents in
dimensional circular space which has two axes a virtual environment, i.e., agent A and agent B,
of Arousal and Valence (Posner et al. 2005). and predict an appropriate emotion of agent A in
Emotion states are allocated in the circular the following situations.
space according to the values of the axis. The
left-lower image in Fig. 1 depicts a circular style • Find a wallet that had been lost (consequences
model. Plutchik’s wheel of emotions is a popu- of events)
lar example of the three dimensional emotion • The wallet is a thing of agent B (consequences
model (Plutchik 2001). The wheel model is of others)
described as the right-lower image in Fig. 1. • Agent B smiles (desirable for other)
He defined eight primary emotions. Each emo-
tion has an opposite emotion: Joy-Sadness, In these situations, the OCC model built-in
Trust-Disgust, Fear-Anger, and Surprise-Antic- agent A indicates happy-for agent B.
ipation. The primary emotions are allocated on These introduced emotion theories are used for
the circle by degrees of similarity. Emotions constructing emotion models for intelligent
allocated outside of the circle are mixture emo- agents. The emotion models become a trigger to
tion of each neighbor primary emotions. The cause the agent’s behaviors.
Emotion-Based 3D CG Character Behaviors 647

Emotion-Based 3D CG Character Behaviors, Fig. 1 Emotion models: the upper image is Parrott’ model, the left-
lower one is Posner et al. model, and the right-lower one is Plutchik’s model
648 Emotion-Based 3D CG Character Behaviors

Emotion-Based 3D CG Character Behaviors, Fig. 2 OCC (Ortony, Clore, and Collins) model

Intelligent Agents with Feelings and • Have the ability to represent and reason over
Emotions knowledge
• Have “social intelligence” for effectively
The term of Intelligent Agent is defined in interacting with other Artificial General Intel-
various interpretations around AI and CI fields. ligent agents
According to the book written by Smith et al.,
Kasabov in 1998 defined the characteristics an Russell and Norvig also defined an agent as
intelligent agent should exhibit as followings anything that can be viewed as perceiving its
(Smith et al. 2009). environment through Sensors and acting upon
that environment through Effectors (Russell and
• Plan, learn, and improve through interaction Norvig 1995). The agent model is described in
with the environment (embodiment) Fig. 3. In the case of a humanoid robot, the
• Adapt online and in real time Sensors would be eye-cameras for recognizing
• Learn quickly from large amounts of data things and ultrasonic sensors for detecting dis-
• Accommodate new problem solving rules tances from other things and the Effectors would
incrementally be arms and legs for influencing their environment
• Have memory-based exemplar storage and which would be real-space. This article treats
retrieval capacities Intelligent Agent as a software program which
• Have parameters to represent short- and long- has ability to autonomously behave according to
term memory, age, forgetting, etc. changing conditions in its environment including
• Be able to analyze itself in terms of behavior, Virtual and Real space by using the Sensors and
error, and success the Effectors. In the virtual space, intelligent
Emotion-Based 3D CG Character Behaviors 649

used the 24 emotions based on the OCC model

to express agent’s emotions by using facial
animations. The model was applied to their pro-
totype application which has a dialogue system
and a talking head with synchronized speech and
facial expressions. As researches for the Effectors
which express agents’ feelings, Raouzaiou’s
facial expression research is useful (Raouzaiou
et al. 2002). Raouzaiou et al. defined basic facial
expressions of six emotions based on the Ekman’s
Emotion-Based 3D CG Character Behaviors, E
Fig. 3 Russell’s and Norvig’s intelligent agent model emotion theory using Facial Animation Parame-
ters of the MPEG-4 specification. Egges and
Thalmann’s research about an emotion model
agents perceive information from their virtual and body animations with its emotions for a 3D
environment and behave according to their character can be available to implement intelligent
purposes. Such virtual environment containing agents in virtual environment (Egges and
intelligent agents is called Intelligent Virtual Thalmann 2005). The authors introduced an idle
Environment (IVE). Aylett and Luck defined motion engine which was adopted 2D emotion
IVE as combination of intelligent techniques and space with activation-evaluation axis and can gen-
tools, embodied in autonomous creatures and erate realistic looking animations according to its
agents, together with effective means for their emotional state. Several researches tried to
graphical representation and interaction of vari- develop an authoring system to build these
ous kinds (Aylett and Luck 2000). Namely, IVE is emotion model and agents’ behaviors. BAAP
a study field of merging Virtual Reality (VR), AI, (Behavioral Animation Authoring Platform)
and Artificial Life (AL) fields. As a research about developed by Li et al. is one of such systems
agents with feelings and emotions in IVE, Liu1 (Li et al. 2010). They implemented agent’s
and Pan introduced an emotion model of 3D emotion model by using 2D emotion model
virtual characters in IVE (Liu and Pan 2005). which contains six emotions, Happiness, Sadness,
The virtual characters have the ability to perceive Anger, Fear, Surprise, and Disgust, and agent’s
current environment conditions in their virtual behavior was controlled by a Behavior Tree in
space, e.g., a forklift is closing to them, and they which a leaf node represents a behavior of the
can express their emotion by their body pose agent. The tree structure was implemented based
movements and their facial expressions. A lot on the OCC model. The authoring system can
more IVE researches and applications appear in customize an agent’s personality by operating an
the survey paper written by Aylett and Cavazza interface of the behavior tree. They introduced a
(2001). Researches for constructing an emotion story scene as an experimental result in where two
model to operate the Effectors are important different stories are generated by changing agents’
theme to implement an intelligent agent. Egges’ personalities. As another approach, Loyall et al.
approach can be helpful to build the emotion (2004) developed an authoring system which can
model and the agent’s behaviors as the Effectors build a 3D character’s personality including its
(Egges et al. 2003). The authors provided a emotion. The system controls their 3D character
generic model for personality, mood, and emotion by script languages. Popescu et al. developed an
simulation for conversational virtual humans. emotion engine for a NPC in a video game
The personality and emotion model was devel- (Popescu et al. 2014).
oped based on the five personality factors, i.e., By changing the perceptive target of the
Openness, Conscientiousness, Extraversion, Sensors from virtual-world to real-world, the
Agreeableness, and Neuroticism, and one good- intelligent agents can be adopted to IUI. Namely,
bad mood dimension. Additionally, the authors the agent can percept humans’ behaviors such as
650 Emotion-Based 3D CG Character Behaviors

facial expressions, conversations, and gesture intelligent agents to play an active role. The agents
poses, and act in the virtual-world according to might be able to become a navigator on maps and
the real-world conditions by operating the a supporter for our learnings. We can feel these
Effectors. In a case of the intelligent agent with possibilities of intelligent agents from the survey
feelings and emotions as IUI, the functionalities paper of Virtual Learning Environment (VLE) by
of the Sensors and Effectors can be handled Pan et al. (2006). Although it is different from VR,
as those of emotion detections and recognitions the concepts of the intelligent agents and method
from humans’ behaviors. Azcarate’s recognition to build emotion models can be adopted to Reality
method classified seven emotions, Happy, field, e.g., robotics technology. A robot which has
Surprised, Angry, Disgusted, Afraid, Sad, and sensors used to understand humans’ emotions and
Neutral, from facial expressions by using Naive abilities to express his/her feelings is one of the
Bayes Classifier (Azcarate et al. 2005). Castellano ideal intelligent agents and can become a good
et al. recognized human emotions from gestures partner for human beings. The affective robot
(Castellano et al. 2007). They focused on a veloc- research of Breazeal shows us the possible future
ity, acceleration, and fluidity of the hand’s visons (Breazeal 2003).
barycenter in continuous gestures. As a research Because IUI is closely related to AI and CI
of intelligent agents as IUI, Kaneko and Okada fields, in the future as well as current situation,
introduced a system for the agents who under- the researches and technologies about intelligent
stand human emotion and express the emotion agents as IUI will evolve with these fields. In AI
by facial expression (Kaneko and Okada 2014). and CI fields, these days Deep Learning (Deng
The agent can receive voice input data from a and Dong 2013) becomes one of the hot topics.
microphone and convert it to text data. The text Deep Leaning is a set of machine learning algo-
data was interpreted by an emotion word database rithms using Artificial Neural Network (ANN)
and the agent express the emotion as a facial piled in multilayers. Figure 4 depicts one of the
expression. In video game fields, such Affective Deep Learning architectures: Deep Boltzmann
Computing research is called Affective Gaming Machine. The advantage of the algorithm is that
(Gilleade and Dix 2005). Bacivarov and Corcoran we need not extract feature values for pattern
tried to apply facial expressions to video games recognitions: the algorithm extracts them. The
(Bacivarov and Corcoran 2009). A lot more algorithm treats raw data, e.g., pixel data, as
researches and applications about Affective input data for input nodes of ANNs in which
Gaming are summarized in the survey paper feature values are extracted and the extracted
written by Kotsia et al. (2013). feature values as output data of the ANNs
are used for input data of another ANNs in
another layer. ANN structures as the learning
Future Visions and State-of-the-art model have been devised in resent researches.
Technologies Deep Boltzmann Machines (DBM), Deep Belief
Networks (DBN), and Convolutional Neural
In the future, intelligent agents with feelings and Networks (CNN) are popular architectures in the
emotions as IUI will play increasingly important Deep Learning field. A lot more Deep Learning
roles to connect a human and a computer in architecture appears in the document written by
various application fields. In Virtual Reality LISA lab (LISA lab 2015). Deep Learning has
(VR) fields, we might spend our daily life for in powerful possibilities to advance the intelligent
IVE by using a head mount display and haptic agent technologies. The remainder of this section
devices. A 3DCG character as an intelligent agent, focuses on the state-of the-art researches about
in the situation, will become a helper for psycho- Deep Learning which can be adopted to the intel-
logical therapies and physical rehabilitations. ligent agents as IUI.
Augmented Reality (AR) and Mixed Reality In the video game research field, there are
(MR) can be available in another field for the several interesting researches which have
Emotion-Based 3D CG Character Behaviors 651

Emotion-Based 3D CG Character Behaviors, Fig. 4 A concept of Deep Learning architecture: the RBM in the figure
means Restricted Boltzmann Machine

potential to be adopted to intelligent agent behav- recognizing human emotions. Neagoe et al. intro-
iors. Mnih et al. introduced a Deep Learning duced several Deep Learning model for facial
research in which a machine trained by Reinforce- emotion recognition (Neagoe et al. 2013). The
ment Learning method played Atari 2600 games research focused on models based on CNN and
and got high scores than a human expert in several DBN. The models recognized seven emotion cat-
games (Mnih et al. 2013). The authors used raw egories, Happiness, Sadness, Surprise, Anger,
pixel data as input data for CNN and apply its Disgust, Fear, and Neutral, by using facial images
output data to Q-Leaning, which is a kind of in JAFFE database. The emotion recognition
Reinforcement Learning method. Although the results got high scores than the results of other
research target is not the case of in 3D VR benchmark algorithms: Nearest Neighbor (NN),
world, the method might be hints for developing Support Vector Machine (SVM) with RBF kernel,
intelligent agent behaviors in video games. As a and SVM with linear kernel. Albornoz et al. intro-
research about 3D video games, the approach of duced an effective method to classify seven
Min et al. provides interesting results to think emotions, Anger, Boredom, Disgust, Fear, Joy,
behavior models for the intelligent agent. The Sadness, and Neutral, from speech utterances
authors introduced a goal recognition framework data by using RBM and DBN which obtained
for Open World game (Min et al. 2014). The better scores than a multilayer perceptron
research used action log data of game player in a classifier (Albornoz et al. 2014). Although
first-person viewpoint game as training data set Neverova et al. approach which detects gestures
for Stacked Denoising Autoencoders (SdA), by using Deep Learning method is not for emotion
a kind of Deep Learning architecture. The action detection, the concepts and methods might be
log data are categorized into five parts: Action applicable to the human emotion detection
Type, Action Argument, Location, Narrative (Neverova et al. 2014). Martínez et al. investi-
State, and Previously Achieved Goals. The gated the utility of Deep Learning approaches
research obtained results of outperforming the for modeling affects comparing several types of
previous goal recognition approach based on CNNs. The authors tried to find emotional mani-
Markov logic networks. The concept to apply festations of Relaxation, Anxiety, Excitement, and
player’s action data in VR to machine thinking Fun form skin conductance signals and blood
and decisions will be useful to develop intelligent volume pulses while a user play a video game
behaviors of human-like agents. (Martínez et al. 2013). Emotion recognition from
Away from researches around intelligent agent Electroencephalogram (EEG) by using Deep
behaviors, then, we focus on researches around Learning method is one of the futuristic
Affective Computing for detecting and approaches. Jirayucharoensak et al. applied
652 Emotion-Based 3D CG Character Behaviors

power spectral densities of 32 channel EEG sig- Aylett, R. and Luck, M.: Applying artificial intelligence to
nals to their recognition method based on Stacked virtual reality: intelligent virtual environments. Applied
Artificial Intelligence, 14 (1), 3–32, (2000)
Autoencoders which classified nine emotion Azcarate, A., Hageloh, F., van de Sande, K. and Valenti, R.:
states: Happy, Pleased, Relaxed, Excited, Calm, Automatic facial emotion recognition. University of
Distressed, Miserable, Depressed, and Neutral Amsterdam, http://staff.science.uva.nl/~rvalenti/pro
which are allocated in Valence-Arousal dimen- jects/mmis/Automatic%20Facial%20Emotion%20Rec
ognition.pdf (2005)
sional emotion model (Jirayucharoensak Bacivarov, I. and Corcoran, P.M.: Facial expression model-
et al. 2014). ing using component AAM models – gaming applica-
These state-of-the-art technologies about agent tions. Games Innovations Conference, 2009. ICE-GIC
behaviors, emotion detections, and emotion 2009. International IEEE Consumer Electronics
Society’s, 1–16, (2009)
recognitions will become useful hints to develop Breazeal, C.: Emotion and sociable humanoid robots.
intelligent agents with feelings and emotions as International Journal of Human-Computer Studies,
IUI as future works. 59(1–2), 119–155, (2003)
Castellano, G., Villalba, S.D. and Camurri, A.:
Recognising human emotions from body movement
and gesture dynamics: affective computing and intelli-
Conclusion gent interaction, Lecture Notes in Computer Science,
4738, 71–82, (2007)
This article introduced several researches related Deng, L. and Dong, Y.: Deep learning methods and appli-
cations. Foundations and Trends in Signal Processing,
to intelligent agent works as user interfaces which 7(3–4), 197–387, (2013)
can understand humans’ emotions and can behave Egges, A., Thalmann, N. M.: Emotional communicative
against them. Recently, these fields become more body animation for multiple characters. In Proceedings
hot tops because several breakthrough researches of the First International Workshop on Crowd Simula-
tion, 31–40 (2005)
focused on Deep Learning technologies. These Egges, A., Kshirsagar, S. and Thalmann, N. M.: A model
results of the researches would be applied to for personality and emotion simulation: knowledge-
robotics fields. According to the advancement based intelligent information and engineering systems,
with these researches, an intelligent agent as a Lecture Notes in Computer Science Vol. 2773,
453–461, (2003)
user interface communicating with a human Ekman, P., Ekman, W.V. and Ellsworth, P.: What emotion
would be discussed with many research fields. categories or dimensions can observers judge from
Because the researches treat several study fields, facial behavior? In: Ekman, P. (Ed.): Emotion in the
e.g., Virtual Reality and Artificial Intelligence, in Human Face, pp. 39–55, (1982)
Gilleade, K. M. and Dix, A.: Affective videogames and
the future as well as current situations, the modes of affective gaming: assist me, challenge me,
researches will evolve with these related fields. emote me. Proceedings of the 2005 DiGRA Interna-
These challenging researches will make more tional Conference: Changing Views: Worlds in Play
friendly relationships between human beings and (2005)
Jirayucharoensak, S., Pan-Ngum, S. and Israsena, P.: EEG-
computers, and enable to build a coexisting based emotion recognition using deep learning
society of them. networkwith principal component based covariate
shift adaptation. Hindawi Publishing Corporation,
Scientific World Journal, Vol. 2014, Article ID
627892 (2014)
References Kaneko, K. and Okada, Y.: Facial expression system using
Japanese emotional linked data built from knowledge
Albornoz, E. M., Sánchez-Gutiérrez, M., Martinez- on the web. International Journal of Space-Based and
Licona, F., Rufiner, H. L. and Goddard, J.: Spoken Situated Computing, Vol. 4(3–4), 165–174, (2014)
emotion recognition using deep learning: progress in Kotsia, I., Zafeiriou, S. and Fotopoulos, S.: Affective
pattern recognition, image analysis, computer vision, gaming: a comprehensive survey. Computer Vision
and applications, Lecture Notes in Computer Science and Pattern Recognition Workshops (CVPRW),
Vol. 8827, 104–111, (2014) 663–670, (2013).
Aylett, R. and Cavazza, M.: Intelligent virtual environ- Li, L., Liu, G., Zhang, M., Pan, Z. and Song, E.: BAAP:
ments – a state-of-the-art report. In Proceedings of a behavioral animation authoring platform for emotion
Eurographics 2001 STARs (2001) driven 3D virtual characters. Entertainment
Engaging Dogs with Computer Screens: Animal-Computer Interaction 653

Computing – ICEC 2010. Lecture Notes in Computer based on MPEG-4. EURASIP Journal on Applied
Science, Vol. 6243, pp. 350–357, (2010) Signal Processing Archive 10(1), 1021–1038, (2002)
LISA lab, University of Montreal: Deep Learning Tutorial. Russell, S. and Norvig, P.: Artificial intelligence: a modern
http://deeplearning.net/tutorial/deeplearning.pdf(2015) approach. Prentice Hall 31–52, (1995)
Liu, Z. and Pan, Z. G. : An emotion model of 3D Smith, C. A. P., Kisiel, K. W. and Morrison, J. G.:
virtual characters in intelligent virtual environ- Working through synthetic worlds. Ashgate,
ment: affective computing and intelligent interac- 226–227, (2009)
tion, Lecture Notes in Computer Science 3784, Sullivan, J. W. and Tyler, S. W.: Intelligent user interfaces.
629–636, (2005) Computational Linguistics 18(3), 368–373, (1991)
Loyall, A., Reilly, W., Bates, J., Weyhrauch, P.: System for Wong, H. S. and Horace H. S.: Human computer interac-
authoring highly interactive, personality-rich interac- tion. Springer Encyclopedia of Multimedia, 289–293,
tive characters. Proceedings of the 2004 ACM (2008)
SIGGRAPH/Eurographics Symposium on Computer E
Animation, 59–68 (2004)
Martínez, H. P., Bengio, Y. and Yannakakis, G. N.: Learn-
ing deep physiological models of affect. Computational
Intelligence Magazine, IEEE, 8(2), 20–33, (2013)
Min W., Ha, E. Y., Rowe, J., Mott, B. and Lester, J.: Deep Engagement
learning-based goal recognition in open-ended digital
games. AAAI Publications, Tenth Artificial Intelli- ▶ Game Design and Emotions: Analysis Models
gence and Interactive Digital Entertainment Confer-
ence, (2014)
▶ Virtual Reality as New Media
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., ▶ Virtual Reality: A Model for Understanding
Antonoglou, I., Wierstra, D. and Riedmiller, M.: Immersive Computing
Playing atari with deep reinforcement learning. NIPS
Deep Learning Workshop, (2013)
Neagoe, V. E., Barar, A. P., Sebe, N. and Robitu, P.: A deep
learning approach for subject independent emotion
recognition from facial expressions. Recent Advances Engaging Dogs with
in Image, Audio and Signal Processing, 93–98, (2013) Computer Screens:
Neverova, N., Wolf, C., Taylor, G. W. and Nebout, F.:
Multi-scale deep learning for gesture detection and
Animal-Computer Interaction
localization. ECCV ChaLearn Workshop on Looking
at People (2014). Joshua Dove-Henfrey and Hoshang Kolivand
Ortony, A., Clore, G.L., Collins, A.: The Cognitive Struc- Department of Computer Science, Faculty of
ture of Emotions. Cambridge: Cambridge University
Press, 15–33, (1998)
Engineering and Technology,
Pan, Z., Cheok, A. D., Yang, H., Zhu, J. and Shi, J.: Virtual Liverpool John Moores University,
reality and mixed reality for virtual learning environ- Liverpool, UK
ments. Journal Computers and Graphics archive 30(1),
20–28, (2006)
Parrott, W.: Emotions in Social Psychology, Psychology
Press, Philadelphia, (2001) Synonyms
Picard, R. W.: Affective computing. M.I.T Media Labora-
tory Perceptual Computing Section Technical Report ACI; Animal-computer interface; DCI; Dog tech-
No. 321, (1995)
Plutchik, R.: The nature of emotions. American Scientist,
nology; Dogs
89(4), (2001)
Popescu, A. Broekens, J. and Someren, M.V.:
GAMYGDALA: an emotion engine for games. Affec- Definition
tive Computing, IEEE Transactions on 5(1), 32–44,
(2014)
Posner, J., Russell, J. A. and Peterson, B. S.: The Animal-computer interaction (ACI) and dog-
circumplex model of affect: an integrative approach to computer interaction (DCI) are extensions of
affective neuroscience, cognitive development, and human-computer interaction (HCI), which refer
psychopathology. Dev Psychopathol. 17(3), 715–734,
(2005)
to the usage of technologies for animals and
Raouzaiou, A., Tsapatsoulis, N., Karpouzis, K. and dogs, respectively. However, HCI refers to usage
Kollias, S.: Parameterized facial expression synthesis of technologies for humans.
654 Engaging Dogs with Computer Screens: Animal-Computer Interaction

Introduction needed in detail of the various solutions which

researchers in the field have produced, to see
Animal-computer interaction (ACI) includes which ones can be generally applied to each
developing a video game prototype for canines. type of application, based on the nature of the
This entry aims to provide context for the problem hardware and software that are need for the dev-
of attracting dogs to watch screens; the solutions eloping stage.
that have been presented, with aims, objectives, To design and implement any interactive appli-
and functional/nonfunctional requirements; the cation for animals (Zeagler et al. 2014) in the form
perspective that this project makes to the research of a video game prototype that attracts the atten-
field of ACI; and the scope of the entire project tion of dogs and be useable by them (Nielsen
(Mancini 2016). 1995; Kjeldskov and Graham 2003), some main
ACI is a relatively young but expanding field of steps are required which will be discussed in this
research with a lot of progress made in recent years entry. This means designing the application using
all across the world. However, there are still a num- various UML diagrams and techniques; develop-
ber of questions that need answering, such as the ing a demo via chosen programming language;
following: How do we change the design of human and having a selection of test participants
applications to allow animals to interact with them (all canine) to interact with it, under ethical con-
(Mancini et al. 2012; Mancini 2013, 2017)? How ditions (Grillaert and Camenzind 2016), in order
can we have animals use technology without human to ensure that the application meets the specified
assistance (Hirskyj-Douglas and Read 2014)? What requirements. The most important aspect usually
kind of applications benefit animal welfare is to ensure that the video game is fit for purpose.
(Geurtsen et al. 2015; Broom 1996; Rehn and This is the most crucial part of the study, as cannot
Keeling 2011)? How can we analyze how dogs be solved the main problem of understanding
feel about the applications that we present them what attracts canines to screens without ensuring
with (Baskin et al. 2015; Mankoff et al. 2005). that it is certainly a video game with which dogs
All of these questions are being asked so that would be interested in interacting. Otherwise, it
researchers can understand how to properly would be pointless to have controlled experi-
design various hardware and software for future ments, as they are more than likely to ignore
ACI projects, particularly for dogs. However, anything that is not designed with them in mind.
although these questions are important in the This research references to what dogs are inter-
design of the research, the main question that ested in and looks to incorporate those interests in
authors are trying to solve with it is: What can the video game, in order to come up with the list
be done to attract dogs to a computer screen in of interests for dogs in the field of ACI.
order to use the application created for them?
One of the most essential steps in ACI design
is getting the dogs to initiate interaction with Methods
the application; however, in order to do that,
they need to be interested in interacting with the Giving the canine incentive to notice and pay
application in the first place. To achieve this, attention to the screen which your application is
developers of ACI applications need to establish being displayed on is one of the most essential
appropriate techniques that they should be steps in initiating interaction between dogs and
employing in the design of their applications. software. This is evident in the experiments that
The problem, however, is that there are no defin- has been researched which all used some tech-
itive ways of implementing this essential step in nique to attract dogs to their own screens (BBC
ACI design. This leads to extra time and resources 2018; Baskin et al. 2015; Geurtsen et al. 2015).
being spent by developers trying to understand The use of techniques like this can also provide
what exactly they can do in order to solve this additional benefits to familiarize dogs with the
problem themselves. A comprehensive review is hardware that researchers want them to use, such
Engaging Dogs with Computer Screens: Animal-Computer Interaction 655

as the experiment by Dr. Lisa Wallace whose use to sounds that they would expect to hear from
of screen paste taught dogs how to interact small animals, e.g., mice rustling through grass
with their own touchscreens using their tongues. (Singer 1995).
Various such techniques will be documented, and A case has been made for this by researchers
their effectiveness will be detailed within this Golbeck and Neustaedter (2012), who in their
entry, so that researchers can understand why project to design a pet monitoring device (for pet
they are important and possibly use them in owners and canines to call and interact with each
their own experiments (Burghardt 2005). other) used a soundboard (among other ideas) to
attempt to attract dogs to their prototype system.
Placing Confectionery Items on a Screen The soundboard consisted of sounds such as dog
E
One of the more unique techniques for attracting barks/howls, cat sounds, squeaky toys, and other
dogs to a screen that researchers and scientists sounds that they believed would interest dogs and
have been using as of late is to place confectionery could be activated remotely by the pet’s owner
items on the screen of the computer/tablet. to play the sounds through the monitoring system.
Dr. Lisa Wallace from the Eötvös Loránd Univer- In their experiment to have dogs interact with their
sity has been using this technique in order to try monitoring system, they would have human users
and get dogs to interact with their brain training use the soundboard to attract their dogs to a pre-
program on a tablet touchscreen. They would first placed laptop screen in order to start interacting
smear a flavored paste all over the touchscreen with their pet through the monitoring system. All
without turning on the tablet and have the canines but one of the dogs successfully came to the
lick it off. They would then start the application screen, the lone exception being a golden retriever
and smear some more paste on the screen to have that was too excited to pay attention to the sounds
the dogs make contact with the application using being played (Golbeck and Neustaedter 2012).
their tongues. Once the application detected them One of the best ways to recognize what sounds
touching the screen, they would receive a dry treat dogs would want from an application would be
from a dispenser below the tablet; after some time, the use of dog personas gathered from canine
paste was no longer required to convince the dogs participants (Hirskyj-Douglas et al. 2017).
to lick the touchscreen and interact with the appli-
cation, as they had now learned how to interact Laser Pointer and Visual Cues
with it as the scientists had intended (BBC 2018). The same experiment mentioned above also
attempted to make use of the dog’s visual sense
Sound Design by two different means. The first was a laser
One of the best senses that dogs possess is their pointer, and the second a virtual object in the
ability to hear, which is generally considered bet- form of a tadpole, both of which were displayed
ter than that of humans. Their hearing spectrums remotely on the screen.
range from 40 hz to 60 khz (Jensen 2007), and
moreover they can locate the source of distinctive
sounds (Geurtsen 2014). As stated earlier, we Discussion
want dogs to exhibit playful behavior and hunting
instincts with the prototype, and thanks to this This section contains discussions on the effective-
excellent hearing range, certain sounds can be ness of each technique and what have been con-
used to help initiate this behavior when it is sidered in existing methods to make them engaged
needed to interact with the video game. The as much as possible.
sounds that dogs will react to can vary on a case
by case basis: some may prefer the sounds gener- Placing Confectionery Items on a Screen
ated by squeaky toys if they particularly enjoy Placing confectionery items on a screen has
playing with them, or if a dog prefers hunting proven to be very useful for the particular exper-
small animals, then it would probably respond iment demonstrated as they managed to show how
656 Engaging Dogs with Computer Screens: Animal-Computer Interaction

useful it was for training dogs to use their devices Another reason for this being a viable tech-
with their tongues. Eventually they managed to nique is that it is a very easy method to use for
have their dogs use the tablet device without attracting dogs to the screen, since all you really
the need for confectionary pastes on the screen, need is a variety of sound files that would attract
proving that this is indeed a great technique for the attention of canines upon hearing them.
having dogs engage with a computer screen. Developers of dog-centered applications and dog
There is only one problem with this technique, technology should be able to make use of this
which is that it really requires specific equipment technique in some form by implementing sound
in order for anyone to make use of it properly. files within their projects.
In the experiment, it was specified that the con- The only problem that some people would
fectionery items they placed on their tablets was a have with this method is that developers would
flavored paste that was completely smothered all have to take time to research what kind of sounds
over the screen for the dogs to lick off. The prob- dogs would be interested in. This is to ensure that
lem with this is that most computers are not they use as many as possible within their pro-
designed to have any kind of confectionery item, jects, as some dogs may not have the same inter-
let alone flavored paste, smothered over their dis- est in certain sounds as others. Dog owners
plays. There should be a few exceptions such as would need to experiment themselves and see
the devices they used in their experiment. (Note: which sounds from the variety that developers
The devices used in the experiment were not have provided catches the attention of their par-
specified in the information source that have ticular canine.
been examined (BBC 2018).) However, it would In the experiments with this technique, the
require dog owners to go out of their way to majority of the participants expressed a moderate
find that equipment if they want to try this tech- to great amount of interest toward the computer
nique themselves. Most dog owners would be when sound effects are used in the video game
completely averse to trying this technique, so as prototype, particularly squeaky toy sounds. Some
not to risk causing any kind of damage to their of their behavior patterns indicated interest toward
own computers due to the use of paste on their the source of the sound as they would bark and
displays. This particular technique has not been jump toward the screen recognizing it as the
taken into account as there was no access to source of the sound. However, the only problem
the necessary technology, but there are rooms is that during the experiments, one of the canines
to acquiring the means necessary to try it in was confused by the sounds, as they would
the future. attempt to stick their nose and paws underneath
the stool that the host device was placed on. This
Sound Design indicated that the canine did not recognize the
The use of specific sounds in DCI seems to be the laptop as the source of the sound and instead
most viable and generally best method for was looking for a squeaky toy underneath the
attracting dogs to a screen. This is for several stool.
reasons, the first of which is that dogs very much
respond to recognizable sound effects and will Laser Pointers and Visual Cues
tend to locate the source of recognizable sounds Judging by the research, this technique may not
as soon as they hear them; therefore it should be as viable as the other methods described ear-
theoretically lead them to the computer, which lier, as proven by the experiment by Jenifer
they will investigate upon realizing that it is the Golbeck et al. In their own experiment, they
source of the sound. Again, this has been proven tried to attract the attention of dogs with both a
by the experiment by Jennifer Golbeck who had laser pointer and a tadpole game playing on their
almost all the participants in their experiment computer screen, but with little success of
attracted to the screen via their own pre-developed attracting the dogs successfully to the screen
soundboard (Golbeck and Neustaedter 2012). (Golbeck and Neustaedter 2012). This is
Engaging Dogs with Computer Screens: Animal-Computer Interaction 657

because these objects were simply not very useful factors for attracting dogs to a screen.
interesting to the canine participants, as other Factors such as teaching fogs on how to use a
experiments have managed to successfully touchscreen with their tongues, as well as to attract
attract dogs to screen via screen cues, such as them to the screen, and keeping them engaged with
the one done by Sofia Baskin et al. Their exper- the screen for a prolonged period of time when they
iment used two different kinds of visual cues, notice that something of interest is there have been
both a ball and mouse type object that would investigated by Baskin et al. (2015).
move around the screen. It was reported in this Overall, the viability of each method depends
entry that both of the canine participants used in largely on the kind of hardware and software that
this experiment had taken interest in the objects DCI developers are using for their particular
E
by exhibiting certain behavioral patterns that project. The viability also depends on the variety
indicated this (Baskin et al. 2015). That leads of assets used with each method and how well
me to conclude that the use of visual cues can they use them for getting the dogs’ attention. If the
only work on a case-by-case basis for canines as available options for each method are researched
they are only individually interested in certain well, chosen well, and implemented to the highest
objects. standard expected of them, then they should
If developers want to use visual cues, then it overall have success in attracting dogs to their
would be wise to implement a variety of objects in computer screen.
their projects to ensure every dog they work with
has something that will interest them. However,
depending on what kind of application they are
Cross-References
developing, they will either have to develop these
cues themselves, which take a lot of time and
▶ Computer Go
work, or get them from somewhere else, such as
▶ Game Development Leadership Tips
the Unity Asset Store. The latter option can be
▶ Interaction with Mobile Augmented Reality
considered very expensive, depending on what
Environments
kind of assets the developer needs, how many
they need, and whether they are made at the qual-
ity level that the developer needs them to be at in
References
order to use them properly.
Baskin, S., Anavi-Goffer, S., Zamansky, A.: Serious
games: is your user playing or hunting? In: Inter-
Conclusion national Conference on Entertainment Computing,
pp. 475–481. Springer, Cham (2015)
BBC, Cheddar Man; Millirobots in the body; Dog brain
When comparing all three options for attracting training BBC Inside Science, 15 Feb 2018. [Online].
canines to a screen, it is safe to say that the most https://www.bbc.co.uk/programmes/b09r3nwz.
viable method that developers can utilize is the Accessed 16 Jun 2018
Broom, D.K.: Animal welfare defined in terms of attempts
use of sound cues. This is due to various facts to cope with the environment (1996). [Online]
including that any sounds that developers require Burghardt, G.M.: The genesis of animal play: Testing the
are easy to access and, in most cases, do not have limits. Mit Press, Cambridge, MA (2005)
any drawbacks to downloading them, such as Geurtsen, A.: An experiment in animal welfare informat-
ics: effects of digital interactive gameplay on the psy-
fees; they are very easy to implement in most chological welfare of home alone dogs. Master
software applications aimed toward dogs; and of Science thesis, Media Technology program, Leiden
they have been proven to be the most successful University (2014)
method when tested with a large range of dogs Geurtsen, A., Lamers, M.H., Schaaf, M.J.: Interactive dig-
ital gameplay can lower stress hormone levels in home
(Golbeck and Neustaedter 2012). alone dogs – a case for animal welfare informatics.
However, this does not mean that the other two In: International Conference on Entertainment
methods are nonviable, as they each have their own Computing, pp. 238–251. Springer, Cham (2015)
658 Engine Architecture

Golbeck, J., Neustaedter, C.: Pet video chat: monitoring

and interacting with dogs over distance. In: CHI’12 Engine Architecture
Extended Abstracts on Human Factors in Computing
Systems, pp. 211–220. ACM (2012)
Grillaert, K., Camenzind, S.: Unleashed enthusiasm: ethi- ▶ Panda3D
cal reflections on harms, benefits, and animal-centered
aims of ACI. In: Proceedings of the Third International
Conference on Animal-Computer Interaction, p. 9.
ACM (2016)
Hirskyj-Douglas, I., Read, J.C.: Who is really in the center Enhanced Visualization by
of dog computer design? In: Proceedings of the 2014 Augmented Reality
Workshops on Advances in Computer Entertainment
Conference, p. 2. ACM (2014)
Ricardo Nakamura1, Fátima L. S. Nunes2 and
Hirskyj-Douglas, I., Read, J.C., Horton, M.: Animal
personas: representing dog stakeholders in interac- Romero Tori1
1
tion design. In: Proceedings of the 31st British Com- Polytechnic School, University of São Paulo,
puter Society Human Computer Interaction São Paulo, Brazil
Conference, p. 37. BCS Learning & Development 2
School of Arts, Sciences and Humanities,
Ltd (2017)
Jensen, P. (ed.): The Behavioural Biology of Dogs. Cabi
University of São Paulo, São Paulo, Brazil
(2007)
Kjeldskov, J., Graham, C.: A review of mobile HCI
research methods. In: International Conference on Synonyms
Mobile Human-Computer Interaction, pp. 317–335.
Springer, Berlin, Heidelberg (2003)
Mancini, C.: Animal-computer interaction (ACI): chang- Augmented reality displays; Visualization tech-
ing perspective on HCI, participation and sustainabil- niques in augmented reality
ity. In: CHI’13 Extended Abstracts on Human Factors
in Computing Systems, pp. 2227–2236. ACM (2013)
Mancini, C.: ACI’16 Proceedings of the Third Interna-
tional Conference on Animal-Computer Interaction,
Definition
16–17 November 2016. [Online]. https://dl.acm.org/
citation.cfm?id¼2995257. Accessed 18 June 2018 Augmented reality is a combination of technology
Mancini, C.: Towards an animal-centred ethics for Ani- and interaction techniques to superimpose virtual
mal–Computer Interaction. Int J Hum-Comput Stud.
98, 221–233 (2017)
objects onto the real world. Data or information
Mancini, C., Wingrave, C., van der Linden, J., Lawson, S., visualization aims to provide visual ways to help
Noz F.: ACI SIG at CHI’12: meeting report users to fastly comprehend and interpret data,
2012. [Online]. http://www.open.ac.uk/blogs/ACI/wp- mainly when the subject is abstract. As such,
content/uploads/2012/11/ACI-SIGCHI-report-final.
pdf. Accessed 6 June 2018
there are different approaches to enhance visuali-
Mankoff, D., Dey, A., Mankoff, J., Mankoff, K.: zation using augmented reality.
Supporting interspecies social awareness: using peri-
pheral displays for distributed pack awareness.
In: Proceedings of the 18th Annual ACM Symposium
on User Interface Software and Technology,
Introduction
pp. 253–258. ACM (2005)
Nielsen, J. (1995). 10 usability heuristics for user interface Augmented reality (AR) can be defined as a var-
design, vol. 1, no. 1. Nielsen Norman Group iation of virtual reality (VR) in which virtual
Rehn, T., Keeling, L.J.: The effect of time left alone at
home on dog welfare. Appl Anim Behav Sci. 129(2–4),
objects are superimposed onto the real world
129–135 (2011) (Azuma 1997). It can also be understood as one
Singer, P.: Animal Liberation. Random House (1995) class of displays in the reality-virtuality contin-
Zeagler, C., Gilliland, S., Freil, L., Starner, T., Jackson, M.: uum (Milgram et al. 1995).
Going to the dogs: towards an interactive touchscreen
interface for working dogs. In: Proceedings of the 27th
Importantly, augmented reality systems share
Annual ACM Symposium on User Interface Software three characteristics: combine virtual and real
and Technology, pp. 497–507. ACM (2014) objects, allow real-time interaction, and perform
Enhanced Visualization by Augmented Reality 659

3D registration (Azuma 1997). Each of these Facilitating the Process of Architectural and
requirements can be met by different technolo- Interior Design
gies, as well as metaphors, which are explained Applications of augmented reality have been
in this text. developed to provide visualization of furniture
Visualization can be defined as the use of for design and decoration (Viyanon et al. 2017)
visual representations, sometimes interactive, to (Nasir et al. 2018). The main motivation is to
enlarge cognition and, thus, help users to compre- provide a cost-effective solution to experiment
hend and interpret abstract data (Gershon and with different options and configurations of furni-
Eick 1997; Card et al. 1999; Liu et al. 2014). ture. For this purpose, marker-based and
Visualization enhanced by AR uses techniques markerless solutions are deployed, so that 3D
E
of AR to enlarge cognition in three-dimensional models of furniture in actual scale can be visual-
interactive environments joining virtual and real ized in a room and thus, space constraints, as well
objects. as aesthetic concerns, can be evaluated.

Examples of Enhanced Visualization by Improving the User Experience in Retail

Augmented Reality Applications

This section presents a few examples of There are augmented reality systems that improve
existing and potential systems in which infor- user experience by showing additional layers of
mation visualization is enhanced by augmented information that are relevant to the users. Two
reality. Those examples are useful to give con- examples are: displaying nutritional information
text to the technologies and techniques on images of actual food (Jiang et al. 2018), and
discussed later. allowing the user to try virtual shoes, visualizing
pressure points on the feet (Bizen et al. 2021).
Visualization of Invisible Phenomena
Augmented reality can be used to provide visual
information about phenomena, such as electro- Augmented Reality Display
magnetic fields, which are not visible to the Technologies
human eye. Similarly, it can display consolidated
information that might be distributed or occluded; Augmented reality (AR) is a concept instead of a
the system proposed by Bhattarai et al. (2020) specific technology. Therefore it is possible to
aims at improving situational awareness of fire- implement such a concept using different solu-
fighters in a fire environment. tions. Following, it is presented the most common
technologies used in AR for displaying spatially
Enhancing the Study of Art Works and registered visual information onto the real world.
Cultural Heritage
Augmented reality systems might be used to visu- Handheld/Mobile
alize additional information over paintings, such Mobile applications are the most popular way of
as layers of pigment, construction lines, as well as enhancing visualization via AR. Almost all cur-
highlighting separate subjects. A number of rent smartphones and tablets incorporate all tech-
mobile augmented reality systems have also nologies necessary for capturing real-world
been developed to enhance the visitation of cul- scenes, tracking movements, and superimposing
tural heritage sites by presenting layers of addi- registered visual information over the real world.
tional information and virtual reconstructed Only a software layer is required to produce the
buildings and other objects (Vlahakis et al. AR effect. The most commonly used metaphors
2001) (Choudary et al. 2009). are “window” and “mirror.”
660 Enhanced Visualization by Augmented Reality

Desktop/Large Screens Visualization Metaphors

Before mobile technology has incorporated high-
quality video processing and tracking technolo- Metaphor can be understood as a main principle of
gies, the most common way of producing AR was mapping an application domain to a visual
via desktop displays or large screen projections universe (Averbukh 2001). In the design of
(via datashow). The “mirror” metaphor was most human-computer interfaces, metaphors are used
used in this case, as the user watched his/her to provide a conceptual reference that might aid
actions by looking at the display screen and not users in understanding abstract data. Thus, some
directly to where the action was taking place in the behaviors and properties of the reference entity
real world. In the beginning of AR popularization, are represented or instantiated on the interface
the most common way of registering was based on (Preece et al. 2015).
cheap fiducial markers (see the section on tracking It is interesting to note that visualization meta-
technologies). phors are found most often in AR systems, in
which the mediation of the display technology
Head-Mounted Display (HMD) cannot be hidden from the user, such as hand-
Goggles, technically known as HMDs (head- held/mobile, desktop/large displays, and
mounted displays) in VR and AR applications, projector-based AR.
are mostly used in professional or public instal-
lations, due to high costs and complexity
involved in setting up and use. It is expected Mirror
that in the future those goggles would be as
cheap and easy to use as sunglasses, which The mirror metaphor is found in desktop/large
would eventually surpass smartphones and tab- screens and handheld/mobile AR applications in
lets as the preferred mobile solution for which one or more cameras are pointed at the user
AR. There are two main approaches for produc- and a mirrored image, composited with virtual
ing AR effect via goggles: optical see-through objects is displayed. It leverages the familiarity
and video see-through. of interacting with and inspected one’s reflected
image in the plane of the mirror to promote the
Optical See-Through visualization of the virtual objects.
In this approach, a semitransparent screen is used,
allowing the direct visualization of the real world Window
while showing registered superimposed virtual The window, or sometimes lens, metaphor is also
images, digitally produced on that screen. encountered in handheld/mobile and desktop AR
applications. In this case, the display surface is used
Video See-Through as a framed viewport to the enhanced visualization.
It is basically a VR HMD, combined with a video As such, it is expected that the virtual objects can
camera that captures the real-world scene, which only be seen through the limited display area.
is augmented with registered superimposed digital
synthetic elements. What the viewer sees is totally X-Ray Vision
digital, but the sensation is of watching directly to The X-ray vision metaphor can be found in pro-
the real environment with AR effects. jective AR systems. The image with enhanced
virtual elements is understood as a view of the
Projector-Based (Spatial AR) inside of the objects onto which it is projected.
This solution avoids the use of HMD by pro-
jecting virtual images directly over physical Customized Metaphors
objects, altering their perceived texture or even In this type of metaphor, developers seek to
producing the illusion of movement. include visual objects (e.g., images, icons, 3D
Enhanced Visualization by Augmented Reality 661

surfaces) in their systems to represent abstract both cases, vision-based tracking is influenced
data in order to help users to comprehend the by factors such as lighting conditions, occlusion
dataset that is being visualized in an AR system. and the presence of unexpected objects in the
scene, which can disturb the pattern recognition
process.
Tracking Technologies
Inertial Tracking
In AR systems, tracking refers to the process of Inertial tracking makes use of inertial sensors
identifying reference objects or locations in order such as gyroscopes and accelerometers attached
to perform 3D registration of the virtual objects: to a tracked object, to determine its position and
E
the correct geometric alignment of those objects orientation. Those sensors are not susceptible to
onto the real world. There are a number of electromagnetic interference, but they are
approaches for tracking, which may be grouped affected by drifting, which reduces their
into four categories: magnetic, vision-based, accuracy.
inertial, and GPS-based (Billinghurst
et al. 2015). GPS-Based Tracking
The use of GPS tracking for augmented reality
Magnetic Tracking systems is feasible in outdoor environments, to
Magnetic tracking involves a transmitter and one determine the user’s position relative to a large
or more receivers of magnetic fields; the varia- area. Due to precision limitations, it is often com-
tions of that field’s properties allow for the deter- bined with other forms of tracking.
mination of each receiver’s position and
orientation in space. While magnetic tracking
allows for fast and accurate tracking, it is also
Cross-References
limited to small volumes in space, and it is sensi-
tive to electromagnetic interference in the
▶ History of Augmented Reality
environment.
▶ Mixed Reality
▶ Mixed Reality and Immersive Data
Vision-Based Tracking
Visualization
Vision-based tracking involves the detection of
features in an image captured by a camera, which
can work with visible or infrared light. The fea-
References
tures may have been added artificially, in the
form of fiducial markers; they may also be natu- Averbukh, V.L.: Visualization metaphors. Program.
rally occurring in the real world, such as patterns Comput. Softw. 27, 227–237 (2001)
of lines, shapes, or textures, which are detected Azuma, R.: A survey of augmented reality. Presence
Teleop. Virt. 6(4), 355–385 (1997)
by image processing algorithms. A fiducial
Bhattarai, M., Jensen-Curtis, A.R., Martínez-Ramón,
marker is a piece of any material containing a M. An embedded deep learning system for augmented
nonsymmetric pattern used to define the place reality in firefighting applications, In: 2020 19th IEEE
where a virtual object should be included in a International Conference on Machine Learning and
Applications (ICMLA), pp. 1224–1230. IEEE (2020).
real scene. The cheapest way of producing these
https://doi.org/10.1109/ICMLA51294.2020.00193.
markers is by printing an image in common Billinghurst, M., Clark, A., Lee, G.: A survey of aug-
paper. Fiducial tracking is comparatively sim- mented reality. Found. Trends Hum. Comput. Interact.
pler, but the requirement for markers added to 8(2–3), 73–272 (2015)
Bizen, H., Yoshida, M., Jimbu, M. Kawai, Y. Virtual shoe
the environment limits its applicability. Natural fitting system that uses augmented reality to measure
feature tracking is more flexible, but the related feet and try on shoes. In: 2021 IEEE 3rd Global Con-
algorithms require more processing power. In ference on Life Sciences and Technologies (LifeTech),
662 Enjoyable Informal Learning

pp. 217–219. IEEE (2021). https://doi.org/10.1109/

LifeTech52111.2021.9391966. Enjoyment
Card, S., Mackinlay, J., Shneiderman, B.: Readings in
Information Visualization Using Vision to Think,
1st edn. Morgan Kaufmann, San Francisco (1999) ▶ Videogame Engagement: Psychological
Choudary, O., Charvillat, V., Grigoras, R., Gurdjos, Frameworks
P. MARCH: mobile augmented reality for cultural her-
itage. In: Proceedings of the 17th ACM international
conference on Multimedia 2009 Oct
19, pp. 1023–1024.
Gershon, N., Eick, S.G.: Information visualization.
IEEE Comput. Graph. Appl., IEEE. 17(4), 29– Entertainment
31 (1997)
Jiang, H., Starkman, J., Liu, M., Huang, M.: Food nutrition ▶ Professional Call of Duty Player Matthew
visualization on Google glass: design tradeoff and field
“Nadeshot” Haag: An e-Sports Case Study
evaluation. IEEE Consum. Electron. Mag. 7(3), 21–31.
IEEE (2018). https://doi.org/10.1109/MCE.2018.
2797740
Liu, S., Cui, W., Wu, Y., Liu, M.: A Survey on Information
Visualization: Recent Advances and Challenges.
TheVisual Computer, Springer Berlin/Heidelberg Escape Room
(2014)
Milgram, P., Takemura, H., Utsumi, A., Kishino, F.:
Augmented reality: a class of displays on the
▶ ROP-Skill System: Model in Serious Games for
reality-virtuality continuum, Proc. SPIE 2351, Universities
Telemanipulator and Telepresence Technologies
(1995)
Nasir, S., Zahid, M.N., Khan, T. A., Kadir K., Khan, S.:
Augmented reality application for architects and inte-
rior designers: interno a cost effective solution. In: Pro- Esports
ceedings of the 2018 IEEE 5th International
Conference on Smart Instrumentation, Measurement ▶ Call of Duty Franchise, an Analysis
and Application (ICSIMA). IEEE (2018). https://doi.
org/10.1109/ICSIMA.2018.8688754.
▶ Diversity in Gaming and the Metaverse
Preece, J., Rogers, Y., Sharp, H.: Interaction Design, ▶ Counter-Strike Global Offensive, an Analysis
4th edn. Wiley, West Sussex, UK (2015) ▶ Fortnite: A Brief History
Viyanon, W., Songsuittipong, T., Piyapaisarn, P., Sudchid, ▶ Professional Call of Duty Player Matthew
S.: AR furniture: integrating augmented reality tech-
nology to enhance interior design using marker and
“Nadeshot” Haag: An e-Sports Case Study
markerless tracking. In: Proceedings of the 2nd Inter- ▶ Super Smash Bros. Ultimate and E-sports
national Conference on Intelligent Information Pro-
cessing (IIP’17). ACM Press, New York (2017).
https://doi.org/10.1145/3144789.3144825
Vlahakis V, Karigiannis J, Tsotros M, Gounaris M,
Almeida L, Stricker D, Gleue T, Christou IT, Essential Experience
Carlucci R, Ioannidis N. Archeoguide: First Results of
an Augmented Reality, Mobile Computing System in
Cultural Heritage Sites. Virtual Reality, Archeology, ▶ Dark Souls Through the Lens of Essential
and Cultural Heritage. (2001). Experience

Enjoyable Informal Learning

Ethical Issues in Gamification
▶ Conceptual Model of Mobile Augmented Real-
ity for Cultural Heritage ▶ Gamification Ethics
Everyday Virtual Reality 663

a novel technology – but one with great potential

Ethics in Gamification for ubiquity in the near future.

▶ Gamification Ethics
Introduction

First coined in 1987 by Jaron Lanier (see Slater and

Ethnic Inclusiveness Sanchez-Vives 2016), the underlying meaning of
VR is one that has broad similarities but also slight
▶ Diversity in Gaming and the Metaverse differences between its various definitions. VR def-
E
initions broadly fall into one of two categories, one
more technology focused and the other more cen-
tered upon user experience. The technological def-
Evaluation Model inition describes VR as multidimensional,
computer-generated content that is perceived by a
▶ MEEGA+, Systematic Model to Evaluate Edu- user as a continuous and distinct environment
cational Games (Craig et al. 2009; Seidel and Chatelier 2013). The
user experience definition by comparison explains
VR in terms of “presence” (feeling physically/cog-
nitively that you are in an environment). Steuer
Everyday Virtual Reality (1992), for instance, argues that to be considered
VR, a system should evoke a degree of presence in
Tom Alexander Garner, Wendy Powell and the virtual world that is approaching our sensation
Vaughan Powell of presence within the physical world. In further
VIA Research, University of Portsmouth, delineations, VR using head-mounted displays
Portsmouth, UK (HMDs) is normally termed “fully immersive
VR,” with “semi-immersive VR” referring to the
use of flat screen displays. Additionally, there is the
Synonyms issue of the reality-virtuality continuum (Milgram
and Kishino 1994) which encompasses VR, aug-
Immersive technologies; Mixed reality; Virtual mented reality (AR), and augmented virtuality
reality; Virtual systems (AV) – all broadly outlined in Fig. 1.
By popular definition, VR is again defined
technologically, but in more restrictive terms.
Definition Here, VR describes a system using a consumer-
grade HMD with head and/or body tracking. This
Everyday virtual reality (VR) can describe any article will focus primarily upon VR by the popu-
activity that the majority of us would typically lar definition, looking at the use of contempo-
engage in at least once per day, experienced rary PC and mobile VR devices in everyday
through the medium of VR. Its meaning is inter- purposes. The rationale for this focus is that, as
woven with user experience and the concept of the term would suggest, popular VR encompasses
technological acceptance, which describes users’ the technology most available to the general pub-
feelings toward a technology’s design and its lic, with its development largely dictated by user
intended purpose. In some contexts, VR already experience requirements in everyday use contexts.
functions as an accepted and almost ubiquitous Everyday VR emerged as a more formal con-
part of everyday life, while in others, it remains cept in 2015, with the first IEEE Workshop in
664 Everyday Virtual Reality

Everyday Virtual Reality, Fig. 1 Edited version of Milgram and Kishino’s “reality-virtuality continuum”

Everyday Virtual Reality (Simeone et al. 2015a). how diversity in VR technology is supportive of
Here the concept is described as the use of its everyday use.
consumer-grade VR technology in an everyday
setting, such as a domestic or office environment.
Everyday VR in the Home

Current Everyday VR Technology Recreational use is undeniably the primary way

in which everyday VR has infiltrated our homes.
Broadly speaking, consumer-grade VR technol- Though the function of VR in the home began
ogy can be separated into four main classes based with video games, the technology has quickly
on their platform; PC-based (also known as “teth- been developed to support our interactions
ered VR” – Oculus Rift, HTC VIVE), smartphone- with various additional forms of media. All
based (Samsung Gear VR, Google Cardboard), four VR platforms support streaming of televi-
console-based (PlayStation PSVR), and most sion and film, with all but console VR supporting
recently, standalone mobile platforms (Oculus web browsing – enabling users to interact with
Go, Lenovo Mirage). Each class of VR comes web-based content within an immersive VR
with a unique set of advantages and disadvan- environment.
tages, with each proving more capable in certain Central to the evaluation of technology used
contexts while less capable in others. Table 1 for accessing media is the concept of user experi-
below summarizes these pros and cons across ence (UX). As Albert and Tullis (2008) observe,
each platform class. the defining requirements of positive UX are that
Speaking again in more general terms, teth- the user feels involved, interested, and in control.
ered and smartphone systems are largely pre- Similarly, research concerning UX for web brows-
sented to consumers as antitheses of one ing indicates that the interface should essentially
another, with the majority of one platform’s strive for perceptual invisibility, becoming an
advantages mirroring the other’s disadvantages. extension of the user to enable the most efficient
Console systems are positioned somewhere in and direct interaction with internet content
between on most points of evaluation while the (Yu and Kong 2016). This presents a significant
newer standalone systems are more comparable challenge to VR which, as a novel consumer tech-
to smartphone VR, but largely differentiate nology with different means of interaction (e.g.,
themselves with improved optimization – being gaze selection in place of touch screen and mouse
built bespoke for VR. Throughout this article, we interfaces) is anything but invisible. As a result,
shall refer back to these pros and cons to show VR faces a significant challenge in becoming a
Everyday Virtual Reality 665

Everyday Virtual Reality, Table 1 Summary of key advantages and disadvantages of VR, separated by platform
VR platform
PC/Tethered Smartphone Console Standalone mobile
Advantages Visual fidelity Very low-cost headsets Very comfortable for Better mobile
Greater comfort Uses hardware many extended use optimization
(supports of us already have Integrated headphones Highly portable
extended use) Highly portable Average FOV (100 ) More reliable
Integrated Easy to setup and 6 DOF (positional More social
headphones operate tracking) Greater comfort
6 degrees of Wireless reduces trip Motion controllers as Wireless reduces trip
freedom hazards standard hazards
(positional More social Easy setup and use Easy setup/use E
tracking) More reliable More reliable 6 DOF (limited)
Motion Poor optimization Controllers as standard
controllers as (heat issues/battery Integrated headphones
standard intensive) FOV (100 )
Greater field of
view (110 )
Room-scale VR
Highly
adjustable HMD
(straps, lenses)
Disadvantages High cost Poorer visual quality Limited application (usage Poorer visual quality
Requires More susceptible to restricted to video games More susceptible to
powerful PC frame lag, causing and media) frame lag, causing
Requires large motion sickness High cost of both console motion sickness
space Lacks positional and VR headset More expensive than
Lacking tracking Lacking portability smartphone
portability Controllers not Lacking room-scale VR Room-scale VR is
Increased risk of standard limited
trips and falls Less comfortable for Less adjustable HMD
with wires extended use
More difficult to Poorer FOV (90 )
setup
More liable to
technical
problems
More isolated
experience

genuine alternative to PC and touchscreen tracking) are less applicable in this context. That
smartphones interfaces; both of which have been said, issues such as extensive power consumption
heavily refined in terms of both usability and UX and heat generation limit users to significantly
for many years (Lobo et al. 2011). Ongoing devel- shorter sessions when compared to using a non-
opment in smartphone and computer interfaces VR smartphone interface. Additional barriers to
also means that VR will likely be playing a per- its technological acceptance include social issues,
petual game of catch-up. relating to feelings of social awkwardness in
Of the four VR platforms, smartphone systems “wearing a computer” in public (Busel 2017).
arguably standout as the most appropriate for this More in the domain of academic research,
particular everyday context; its accessibility, ease “substitutional reality,” features as a particularly
of use, and wireless operation all support the interesting area of development. Simeone et al.
above UX requirements, while many of its limita- (2015b) define substitutional reality as “a VR
tions (poorer graphics, lacking positional experience that incorporates physical objects into
666 Everyday Virtual Reality

the virtual world by using them to represent A further review by Jensen and Konradsen (2017)
objects relevant to the virtual context” (p. 3307). observes that VR offers support to memory and
The general idea is that real-world objects are spatial awareness training and can also help with
tracked in virtual space and mapped to virtual the development of motor skills and emotion con-
objects that may be very different in nature, but trol. One of the most significant benefits of VR in
comparable in their weight and dimensions. In classrooms relates to the concept of the “extended
this instance, a bottle may become a vial of classroom” (see Loureiro and Bettencourt 2011),
magic potion, a broom a lightsabre, an umbrella with the technology enabling students to engage
may even become Mjolnir (Thor’s Hammer). with simulations of various locations and events
While not yet commonplace in our homes, substi- that would be expensive, impractical, or even
tutional reality is arguably an advocate for the physically impossible to experience otherwise.
“everydayness” of VR, as it utilizes a range of VR within an educational context has built
distinctly everyday objects about the home and upon the progress made by semi-immersive vir-
repurposes them to create multisensory and tual systems; serious games that present the user
deeply immersive mixed-realities. with a three-dimensional world from a first-person
Although VR is highly unlikely to replace tra- perspective. A prominent example of this would
ditional interfaces for 2D media content, it is in the be in the use of Minecraft (Persson et al. 2011)
third dimension where the technology has the in schools. As Nebel et al. (2016) point out,
most potential. 360 films and television pro- games such as this are deeply supportive of
grams can be viewed upon 2D screens but lack constructivist pedagogy (contextualized learning
the immersive quality of head-tracked HMDs, to through active construction of knowledge) and
the extent that such content is essentially exclu- encourage cooperative learning, self-management
sive to VR. As the majority of web content is two- and provide an accessible route to understanding
dimensional, however, there remain significant complex systems. Minecraft is currently being
limitations to the perceived usefulness of VR for used in numerous school systems all over the
web browsing. That said, research is pursuing this world to teach an impressively broad range of
issue by exploring various means of deploying academic subjects (Short 2012). VR arguably
immersive, 3D websites built upon existing web- provides key additional benefits beyond those
standard technologies (de Paiva Guimarães et al. offered by traditional serious games. These
2018). The future success of VR in this context include greater perceived realism through more
remains an optimistic but unknown quantity. immersive audiovisual content and more natural-
istic controls, and a deeper connection to the con-
tent through enhanced feelings of presence (see
In the Classroom Freina and Ott 2015).
Although its advantages are distinct, the likeli-
Virtual technologies are becoming increasingly hood of seeing VR as a ubiquitous technology
notable within education and skills training. within this everyday context is dependent upon
A recent meta-review by Gutierrez-Maldonado its disadvantages being overcome. Jensen and
et al. (2017) noted that $1.2 billion has been Konradsen (2017) note that the benefits of VR
invested globally in the last couple of years to over traditional computer interfaces are largely
increase access to virtual content, specifically by limited to those stated above and that the technol-
supporting the development of HMDs. This is not ogy also presents issues that can be counter-
at all surprising as some of the fundamental char- productive to learning, such as the prevalence
acteristics of VR, namely the potential to create of cybersickness and the potential of the im-
accurate and immersive simulations of real-world mersive experience being a distractor from the
phenomena, are ideally suited to contextualizing learning task. One of the primary current issues
and grounding complex and abstract information. with classroom VR is financial feasibility. Though
Everyday Virtual Reality 667

a far more significant issue 15 years ago (see consumer hype and inflated expectations that
Mantovani et al. 2003), the costs of purchasing, plagued the entertainment market. The result of
maintaining and supervising the operation of this is that VR has been steadily improving and
HMDs in the classroom remains prohibitive for integrating itself as an everyday technology
many schools (Merchant et al. 2014). The invest- within industry for decades. At present, some of
ment mentioned above, however, seeks to over- the most prominent applications of VR in the
come this issue by driving down the consumer workplace are telecommunications, data visuali-
costs of the hardware, making it more compatible zation, and rapid prototyping.
with cheaper computers and developing the Telecommunications is arguably an applica-
potential of smartphone-based VR (Gutierrez- tion of VR that could have also featured in the
E
Maldonado et al. 2017). above discussion on VR in the home. The grand
At the cutting edge, research into classroom ambition of telecommunications systems is to
VR is largely prioritizing developments in multi- evoke a sense of telepresence that is equal to
user cooperative systems (Greenwald et al. 2017), face-to-face conversation. Coined by Minsky in
integration of bio/neurofeedback (Blume et al. 1980, telepresence originally described a feeling
2017), and designing for students with special of “being there,” experienced by individuals
educational needs such as autism and attention- remotely operating robots. In a more recent defi-
deficit hyperactivity disorder (Negut‚ et al. 2017). nition by Ting et al. (2017), telepresence refers to
In our current classrooms, fully immersive VR “the degree of awareness of another person in an
may not be as “everyday” for most students as interaction and the consequent appreciation of an
semi-immersive systems, but in some schools, it is interpersonal relationship” (p. 382). The use of
being used with intent and as an integrated part of technology as a medium for communicating
the curriculum. across great physical distances is by no means a
In terms of technological acceptance in this new concept, with teleconferencing and video-
context, classroom VR studies have revealed per- conferencing systems possessing an extensive
ceived usefulness and openness to new learning history of development and a now global usage.
methods to be significant factors (Liou et al. VR aims to bring telecommunications closer to
2017). Prior experience of technology has also realizing its grand ambition, with recent research
been shown to influence acceptance (Neguţ et al. projects exploring the use of VR combined with
2016), suggesting that the use of VR in the class- facial and body capture technology (Thies et al.
room will continue to expand, in part due to the 2016) and “mixed-reality telecoms,” in which
simple fact that we as humans are increasingly webcam feeds of multiple speakers are presented
growing up with the technology. within a single, multiuser virtual environment
(Regenbrecht et al. 2015). The potential value
of VR communications is substantial across
In the Workplace every sector of industry, enabling users to conduct
presentations, seminars, board meetings, job
The extent of VR’s everydayness within work- interviews, and more, all with a quality of com-
place environments varies significantly between munication that is closer to in-person contact, but
specific industries, but overall this particular con- across global distances.
text is where VR has become the most normalized Early data visualizations began to appear in the
and the technology is at its most “mature, stable early nineteenth century with bar charts, scatter
[and] usable” state (Berg and Vance 2017: p.1). plots, and line graphs (Rimland et al. 2013). Tra-
The entertainment market for VR has endured a ditional (and much of contemporary) data visual-
volatile few decades. In contrast, VR develop- ization is graphics-only, two-dimensional, and
ments in industry continued unabated, as the moti- static, which significantly limits our ability to
vations for progress were not tied up in the comprehend complex or evolving information.
668 Everyday Virtual Reality

In such instances, VR has a powerful potential to Out in the World

overcome such limitations by way of representa-
tions that are dynamic, multidimensional, and Everyday VR is not confined to interior applica-
immersive (Laha and Bowman 2012; Reda et al. tions within homes, schools, and workplaces.
2013). As one would expect, the specific indus- Rather its meaning stretches out to exterior
tries heavily utilizing VR for data visualization are venues and even the great outdoors. In recent
those that deal in high volumes of complex data. years, the theatre has taken a significant interest
These include the finance, energy, and pharma- in the potential of VR technology to both
ceutical sectors (Marr 2017). enhance and explore new approaches to dramatic
VR’s ability to digitally represent architectural performance. As Moneta (2017) points out, asso-
designs in actual scale makes it ideally suited to ciations between theater and VR go back at least
cost-effective prototyping, with users able to 20 years, where the concepts of blending physi-
experience designs in a way that is far closer the cal and virtual content were initially
intended final construction than a 2D image or experimented with. Moneta also points to the
scale model (Wolf et al. 2017). Near-future ambi- use of VR as a means of reimagining existing
tions for architectural visualization largely con- works, both to allow the initiated to experience a
cern three main areas: wireless connectivity, performance in a distinctly different way, and to
multiuser systems, and the integration of Building make such works more accessible to newcomers.
Information Modelling (BIM). Together, multi- One cutting-edge example of VR-theater is
user functionality and the integration of BIM Fatherland VR (Council 2018), a VR play in
data within VR enables multiple stakeholders which the audience directly interact with multi-
(from architects and interior designers, to engi- ple virtual characters, some manipulated by a
neers and construction workers, not to mention live actor (through motion capture) with others
the clients themselves) to work much more controlled by the computer. On occasions, the
closely and collaboratively. Here, a single shared way in which a single virtual character is con-
environment can present multiple facets of the trolled switches between human and computer
design, from the aesthetics to the mathematical control, with the audience left unaware.
characteristics of the structure, all of which can Since 2015, The British Museum has been
be experienced within a single VR representation incorporating VR into their exhibitions as a
by everyone, all together and in real time. means of enhancing the experience and impro-
Architecture is most certainly not the sole ving educational potential (Rae and Edwards
industry currently utilizing VR as a means 2016). Here, smartphone VR is preferable
of rapid prototyping. The ability to evaluate largely due to its efficiency of use, specifically
a product concept as a multimodal, immersive, the speed at which the headset can be fitted and
and interactive virtual prototype has benefit to a removed – enabling a greater number of visitors
wide range of industries including the military, to experience the VR content throughout the day.
aerospace, automotive, and agriculture. In many Recent research has pointed to increased enter-
instances, the value of VR prototyping extends tainment value as a significant factor in the
beyond a representation of the product, to a appeal of virtual technology in museums but
simulation of the real-world environment in also points to the requirement of improved aes-
which that product would function. This enables thetics and better social presence for VR to
product development to improve various aspects acquire greater technological acceptance in this
of a design, from ergonomics and usability, context (Jung et al. 2016). As with other appli-
to construction method and aesthetics (Craig cations such as architectural visualization,
et al. 2009). For those working in such sectors museum VR reveals a need for further develop-
of industry, VR’s rapidly increasing usage is ments in multiuser systems that immerse users
making it a prominent element of their everyday within virtual worlds that can be shared with
working lives. others.
Everyday Virtual Reality 669

VR also takes things outside, presenting (for ease of use and accessibility for both its elderly
example) visitors of heritage sites the opportu- users and the practitioners delivering the ther-
nity to experience “smart tourism” (see Chung apy (Benoit et al. 2015). By comparison, phys-
et al. 2015). Here, VR software that can be ical rehabilitation favors tethered systems as it
downloaded directly to phones and mobile typically requires full-body tracking with
devices provides visitors with access to addi- 6 degrees of freedom to enable the system to
tional virtual information relevant to the imme- evaluate the user’s movements correctly.
diate physical environment. Mobile technology, Reviews of technological acceptance are gener-
GPS navigation, and object recognition sup- ally positive in this context, though studies have
ports location-based content generation and revealed that (in line with various findings in
E
enables visitors to hold their device towards an other contexts discussed earlier) improvement
historic ruin and observe a digital recreation of in ease of use and social presence (i.e., multi-
how it once looked many centuries ago. While user) are key to raising acceptance further
research and development in these technologies (Roberts et al. 2018).
is ongoing, many applications of these types are
already available to consumers across the
world. Closing Comments
Lastly, healthcare is one of the most signifi-
cant applications of VR technology and repre- To summarize, VR already presents us with new
sents a workplace environment in which means of engaging with numerous everyday
professionals are increasingly seeing VR activities, from browsing the web or using
become an everyday presence. Outside of this film-streaming services, to family days out to
context, however, everyday VR also extends to museums or to the theatre. The annual work-
healthcare for the wider population by way of its shop on Everyday Virtual Reality is now in its
application for exercise. Here, a range of fourth year (Simeone et al. 2018) and is con-
approaches are now available to consumers. tinuing to grow. Several challenges remain
These include using VR video-streaming ser- however. Motion sickness remains a persistent
vices as a distraction during endurance cardio issue for many users and contemporary headsets
training and using VR action games whilst wear- continue to cause neck soreness and eye strain
ing body weights to enable inherently engaging after prolonged use. The hardware is not dis-
activities such as playing games to also provide crete and, in social contexts, potential users
moderately intense workouts (Holly 2017). As remain uncomfortable with the idea of using
individuals get older, the likelihood that they VR technology in public. Costs remain prohib-
will engage with VR exercise in an everyday itive in many everyday contexts and in terms of
context becomes significantly greater. The tech- usability and user experience, VR interfaces
nology is now featuring in physical rehabilita- continue to lag behind more established tech-
tion for issues with balance, gait training, nologies. Despite these issues, VR is already
mobility, and muscular degeneration, to name a enhancing many everyday tasks and procedures
few (see Park et al. 2014; Park et al. 2015; that have themselves been commonplace for
McEwen et al. 2014). many decades, and in some cases, even centu-
Healthcare, in particular, reveals the benefits ries. Ultimately, it is very difficult to make pre-
of there being a range of VR platforms, each dictions for the future, particularly when
with different pros and cons. For example, considering that VR technology is becoming
smartphone VR is ideal for delivering reminis- increasingly interconnected to mixed reality, a
cence therapy (reminiscing on past experiences concept arguably more compatible with every-
as a means of preventing memory loss) as it day activities. Where VR prioritizes isolated
typically involves 360 video, requiring mini- experiences and immersion in virtual worlds,
mal processing power, but does require greater mixed reality favors the integration of virtual
670 Everyday Virtual Reality

worlds with our shared physical world. For Freina, L., Ott, M.: A literature review on immersive
everyday VR, the future is mixed. virtual reality in education: state of the art and per-
spectives. In: The International Scientific Conference
eLearning and Software for Education, vol. 1, p. 133.
Carol I National Defence University, Bucharest
Cross-References (2015)
Greenwald, S., Kulik, A., Kunert, A., Beck, S., Frohlich,
B., Cobb, S., Parsons, S., et al.: Technology and Appli-
▶ Cybersickness cations for Collaborative Learning in Virtual Reality,
▶ Mixed Reality pp. 719–726 (2017)
▶ Mixed Reality, Gamified Presence, and Gutierrez-Maldonado, J., Andres-Pueyo, A., Jarne, A.,
Storytelling for Virtual Museums Talarn, A., Ferrer, M., Achotegui, J.: Virtual reality
for training diagnostic skills in anorexia nervosa: a
▶ Substitutional Reality usability assessment. In: International Conference on
▶ Virtual Reality Applications in Education Virtual, Augmented and Mixed Reality, pp. 239–247.
▶ Virtual Reality Exercise and Rehabilitation Springer, Cham (2017)
Holly, R.: Best VR apps for exercise. VR Heads. https://
www.vrheads.com/best-vr-apps-exercise (2017).
Accessed 12 Apr 2018
References Jensen, L., Konradsen, F.: A review of the use of virtual
reality head-mounted displays in education and train-
Albert, W., Tullis, T.: Measuring the User Experience: ing. Educ. Inf. Technol. 23(4), 1515–1529 (2017)
Collecting, Analyzing, and Presenting Usability Met- Jung, T., tom Dieck, M.C., Lee, H., Chung, N.: Effects of
rics. Newnes Morgan Kaufman. New York, USA virtual reality and augmented reality on visitor experi-
(2008) ences in museum. In: Information and Communication
Benoit, M., Guerchouche, R., Petit, P.D., Chapoulie, E., Technologies in Tourism 2016, pp. 621–635. Springer,
Manera, V., Chaurasia, G., et al.: Is it possible to use Cham (2016)
highly realistic virtual reality in the elderly? Laha, B., & Bowman, D. A.: Identifying the benefits of
A feasibility study with image-based rendering. immersion in virtual reality for volume data visualiza-
Neuropsychiatr. Dis. Treat. 11, 557 (2015) tion. In: Immersive visualization revisited workshop of
Berg, L.P., Vance, J.M.: Industry use of virtual reality in the IEEE VR conference, pp. 1–2 (2012)
product design and manufacturing: a survey. Virtual Liou, H.H., Yang, S.J., Chen, S.Y., Tarng, W.: The influ-
Reality. 21(1), 1–17 (2017) ences of the 2D image-based augmented reality and
Blume, F., Hudak, J., Dresler, T., Ehlis, A.C., Kühnhausen, virtual reality on student learning. J. Educ. Technol.
J., Renner, T.J., Gawrilow, C.: NIRS-based Soc. 20(3), 110–121 (2017)
neurofeedback training in a virtual reality classroom Lobo, D., Kaskaloglu, K., Kim, C., Herbert, S.: Web
for children with attention-deficit/hyperactivity disor- usability guidelines for smartphones: a synergic
der: study protocol for a randomized controlled trial. approach. International journal of information and elec-
Trials. 18(1), 41 (2017) tronics engineering. 1(1), 33 (2011)
Busel, M.: The 6 biggest challenges facing augmented Loureiro, A., Bettencourt, T.: The extended classroom:
reality. Haptic.al. https://haptic.al/augmented-realitys- meeting students’ needs using a virtual environment.
biggest-threats-3f4726a3608 (2017). Accessed 06 Apr Procedia Soc. Behav. Sci. 15, 2667–2672 (2011)
2018 Mantovani, F., Castelnuovo, G., Gaggioli, A., Riva, G.:
Craig, A.B.., Sherman, W.R., Will, J.D.: Developing Vir- Virtual reality training for health-care professionals.
tual Reality Applications: Foundations of Effective Cyberpsychol. Behav. 6(4), 389–395 (2003)
Design. Morgan Kaufmann, Amsterdam (2009) Marr, B.: How VR and AR will change how we visualise
Chung, N., Han, H., Joun, Y.: Tourists’ intention to visit a data. Forbes.com. https://www.forbes.com/sites/
destination: the role of augmented reality bernardmarr/2017/08/31/how-vr-and-ar-will-change-
(AR) application for a heritage site. Comput. Hum. how-we-visualize-data (2017). Accessed 10 Apr 2018
Behav. 50, 588–599 (2015) McEwen, D., Taillon-Hobson, A., Bilodeau, M., Sveistrup,
Council, A.: Fatherland VR. Limbic. https://www. H., Finestone, H.: Virtual reality exercise improves
digitalcatapultcentre.org.uk/news-creativexr- mobility after stroke: an inpatient randomized con-
prototypes/ (2018). Accessed 12 Apr 2018 trolled trial. Stroke. 45(6), 1853–1855 (2014)
de Paiva Guimarães, M., Dias, D.R.C., Mota, J.H., Gnecco, Merchant, Z., Goetz, E.T., Cifuentes, L., Keeney-
B.B., Durelli, V.H.S., Trevelin, L.C.: Immersive and Kennicutt, W., Davis, T.J.: Effectiveness of virtual
interactive virtual reality applications based on 3D web reality-based instruction on students' learning out-
browsers. Multimedia Tools and Applications. 77(1), comes in K-12 and higher education: a meta-analysis.
347–361 (2018) Comput. Educ. 70, 29–40 (2014)
Evolutionary Algorithms 671

Milgram, P., Kishino, F.: A taxonomy of mixed reality Short, D.: Teaching scientific concepts using a virtual world—
visual displays. IEICE Trans. Inf. Syst. 77(12), Minecraft. Teaching Science-the Journal of the Australian
1321–1329 (1994) Science Teachers Association. 58(3), 55 (2012)
Minsky, M.: Telepresence. Omni. 2(9), 44–52 (1980) Simeone, A. L., Powell, W. & Powell, V.: In: 1st Workshop
Moneta, A.: How virtual reality is changing the on Everyday Virtual Reality. IEEEVR. 23–24 March.
way we experience stage shows. The Conversation. Arles, France. http://ieeevr.org/2015/indexb963.html?
http://theconversation.com/how-virtual-reality-is- q¼node/39#WS2 (2015a). Accessed 21 June 2018
changing-the-way-we-experience-stage-shows-81542 Simeone, A. L., Powell, W., Powell, V., Johnsen, K. &
(2017). Accessed 12 Apr 2018 Bialkova, S.: 4th Workshop on Everyday Virtual Real-
Nebel, S., Schneider, S., Rey, G.D.: Mining learning and ity. IEEEVR. 18 March. Reutlingen, Germany. http://
crafting scientific experiments: a literature review on w w w. i e e e v r. o r g / 2 0 1 8 / p r o g r a m / w o r k s h o p s .
the use of minecraft in education and research. html#WEVR (2018). Accessed 21 June 2018
J. Educ. Technol. Soc. 19(2), 355 (2016) Simeone, A.L., Velloso, E., Gellersen, H.: Substitutional E
Neguţ, A., Matu, S.A., Sava, F.A., David, D.: Task diffi- reality: using the physical environment to design virtual
culty of virtual reality-based assessment tools com- reality experiences. In: Proceedings of the 33rd Annual
pared to classical paper-and-pencil or computerized ACM Conference on Human Factors in Computing
measures: a meta-analytic approach. Comput. Hum. Systems, pp. 3307–3316. ACM, New York (2015b)
Behav. 54, 414–424 (2016) Slater, M., Sanchez-Vives, M.V.: Enhancing our lives with
Negut‚, A., Jurma, A.M., David, D.: Virtual-reality-based immersive virtual reality. Frontiers in Robotics and
attention assessment of ADHD: ClinicaVR: classroom- AI. 3, 74 (2016)
CPT versus a traditional continuous performance test. Steuer, J.: Defining virtual reality: dimensions determining
Child Neuropsychol. 23(6), 692–712 (2017) telepresence. J. Commun. 42(4), 73–93 (1992)
Park, E.C., Kim, S.G., Lee, C.W.: The effects of virtual Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., &
reality game exercise on balance and gait of the elderly. Nießner, M. (2016). FaceVR: Real-Time Facial Reen-
J. Phys. Ther. Sci. 27(4), 1157–1159 (2015) actment and Eye Gaze Control in Virtual Reality. arXiv
Park, J., Lee, D., Lee, S.: Effect of virtual reality exercise preprint arXiv:1610.03151
using the nintendo wii fit on muscle activities of the Ting, Y.L., Tai, Y., Chen, J.H.: Transformed telepresence
trunk and lower extremities of normal adults. J. Phys. and its association with learning in computer-supported
Ther. Sci. 26(2), 271–273 (2014) collaborative learning: a case study in English learning
Persson, M., et al.: Minecraft. Mojang/Microsoft Studios, and its evaluation. Interact. Learn. Environ. 25(3),
Stockholm (2011) 382–396 (2017)
Rae, J. & Edwards, L.: Virtual reality at the British Wolf, K., Funk, M., Khalil, R., & Knierim, P.: Using virtual
Museum: What is the value of virtual reality environ- reality for prototyping interactive architecture. In: Pro-
ments for learning by children and young people, ceedings of the 16th International Conference on
schools, and families? Museums and the Web 2016. Mobile and Ubiquitous Multimedia, pp. 457–464.
Los Angeles, CA, USA, 6–9 April 2016 ACM (2017, November)
Reda, K., Febretti, A., Knoll, A., Aurisano, J., Leigh, J., Yu, N., Kong, J.: User experience with web browsing on
Johnson, A., et al.: Visualizing large, heterogeneous small screens: experimental investigations of mobile-
data in hybrid-reality environments. IEEE Comput. page interface design and homepage design for news
Graph. Appl. 33(4), 38–48 (2013) websites. Inf. Sci. 330, 427–443 (2016)
Regenbrecht, H., Alghamdi, M., Hoermann, S., Langlotz,
T., Goodwin, M., & Aldridge, C.. Social presence with
virtual glass. In: Virtual Reality (VR), 2015 IEEE,
pp. 269–270. IEEE (2015, March)
Rimland, J., Ballora, M., Shumaker, W.: Beyond visuali-
zation of big data: a multi-stage data exploration Evolutionary Agent Design
approach using visualization, sonification, and
storification. SPIE Defense Secur. Sens. 8758, ▶ Constructing Game Agents Through Simulated
87580K (2013) Evolution
Roberts, A.R., Schutter, B.D., Franks, K., Radina, E.E.:
Older adults’ experiences with audiovisual virtual real-
ity: perceived usefulness and other factors influencing
technology acceptance. Clin. Gerontol. (2018). Just
accepted Evolutionary Algorithms
Seidel, R.J., Chatelier, P.R. (eds.): Virtual Reality, train-
ing’s Future?: Perspectives on Virtual Reality and
Related Emerging Technologies, vol. 6. Springer, ▶ Constructing Game Agents Through Simulated
New York (2013) Evolution
672 Evolutionary Computation

Evolutionary Computation Experiential Media: Using

Machine Vision and Sensor-
▶ Constructing Game Agents Through Simulated Input to Create Dynamic Real-
Evolution Time Generated Media

Tammuz Dubnov
Zuzor, Tel Aviv, Israel
Evolutionary Machine
Learning
Synonyms
▶ Constructing Game Agents Through Simulated
Experiential media; Interactive displays; Mixed
Evolution
reality; Personalized content; Reactive visuals;
Real-time generated content

Exercise Definitions

▶ Virtual Reality Exercise and Rehabilitation Experiential media (EM) refers to models of
media computing that incorporate contextual
understanding of human activity, at different
scales of time and space, to affect the human
experience of the content through such activity.
Exergames
It allows user engagement by utilizing live sensors
as input to a media computing unit that then out-
▶ Rehabilitation Games
puts multimedia to output sources, such as dis-
plays or speakers. Experiential media is dynamic
real-time generated media based on the live phys-
ical activity perceived with the goal of achieving
Exergaming enhanced and unified physical-digital experiences
(Sundaram and Rikakis 2008).
▶ Virtual Reality Exercise and Rehabilitation

Introduction

Experiential media (or EM) was used primarily by

Experience
the museum industry for specialized exhibits
using methods of experiential learning (Huang
▶ Virtual Reality as New Media
and Lin 2013), but has grown since then into a
general tool for experiential content delivery in
applications such as digital signage, architecture,
hospitality, entertainment, and more. EM is often
Experiential Media used for branding and marketing purposes among
many other uses. It is growing in popularity as it
▶ Experiential Media: Using Machine Vision and allows for interactivity and higher user engage-
Sensor-Input to Create Dynamic Real-Time Gen- ment with brands, prolonging the duration of
erated Media interaction with advertising and increasing
Experiential Media 673

impression counts. With the progression of intel- Uses and Frameworks of EM

ligent computing, using methods such as Artificial
Intelligence (AI), greater contextually appropriate EM has multiple uses, depending on the type of
EM is possible. content it displays, such as entertainment, educa-
EM utilizes live sensor inputs from devices tional, infotainment, wayfinding, advertising, or
such as wearables, microphones, proximity sen- others. Furthermore, as large format displays
sors, or cameras. This entry will primarily focus become more affordable and prolific, the Digital
on camera-based EM, using machine vision and Signage and ProAudioVisual industry are begin-
further narrow down to focus on single camera ning to offer EM as a tool that combines content
setups. EM goes beyond classic human computer generation with immersive experiences. As of
E
interface (Sutcliffe 1995) (HCI) as it is touchless, now, EM is known to be used in the following
responsive from a distance, and its use of AI industries: hospitality, nightlife, entertainment,
technology. It is the natural evolution of tradi- gaming, events, public speaking, retail, museums,
tional audiovisual content as it gradually becomes and experiential marketing.
more sophisticated and starts to become personal- At the time of writing, the mainstream
ized to the viewer/user. EM is defined as “experi- approaches to creating EM are using platforms
ential” as it integrates user action in the design, and libraries such as Openframeworks, Unity,
requiring the user to physically act and move in LumoPlay, and Zuzor. The different platforms
order to experience the full media. are targeted at creators with different levels of
EM has the tendency to shift the user from expertise: from full coding for developers, such
passive viewer role to an active participant role as Openframeworks, to creative suites that allow
as its elements will remain hidden or remain at for rapid customization for graphic designers,
their default behavior unless the user is actively such as Zuzor. There are more tools available
moving in the intended way, as can be seen in such as Touchdesigner, Quartz Composer, etc.,
Fig. 1. EM differs from mixed reality (Kasapakis which are beyond the scope of this entry. A brief
et al. 2018) (MR) as it does not necessarily incor- description of the mainstream approaches is pro-
porate real world objects but can act as its own vided below.
autonomous object. EM is presented through dis-
play mediums that are built into the environment Openframeworks: openFrameworks is an open-
in ways that are intended to augment or mix with source toolkit designed for creative coding
the visual experience coming from physical written in C++ and built on top of OpenGL
spaces. (openFrameworks 2020.)

Experiential Media: Using Machine Vision and Sensor-Input to Create Dynamic Real-Time Generated Media,
Fig. 1 Experiential Media projected on a wall
674 Experiential Media

Experiential Media: Using Machine Vision and Sensor-Input to Create Dynamic Real-Time Generated Media,
Fig. 2 Upright setup (left) and top-down setup (right)

Unity: Unity is a cross-platform game engine time capabilities bring EM technology to enter-
developed by Unity Technologies that is pop- tainment with a more robust stage performance
ular for Games and Cinematics around 3D (Dubnov 2014). Previously, in many entertain-
content (Unity 2020.) ment and staged environments, it was common
LumoPlay: Lumo Play is a software solution for to utilize offline approaches to create the illusion
Windows to create games for kids using of audiovisual content as EM. While these
motion, gesture, and touch experiences approaches may incorporate dynamic generated
(LUMOplay 2020.) media, it lacks the key real-time element that is
Zuzor: Zuzor is an Experiential-Media platform core to this entry. The general approach was often
that enables the graphic design industry to for performers to train to move in synchrony, both
easily create experiential media and the Pro- temporarily and spatially, with elements of the
AudioVisual industry to easily deploy experi- content to make it appear as though it is interactive
ential media while tracking engagement using and contextual to their movement. This approach
AI analytics (Zuzor 2020.) is prone to human error as it requires the per-
formers to move exactly the same each time, in
the exact same timing and the exact same spatial
Real-Time Sensing position, to create the effect of EM for the broader
viewers. This approach further limits the partici-
As EM establishes a feedback loop between the pants to individuals that are a part of the produc-
audiovisual content and the user, a real-time solu- tion, as they often require numerous rehearsals,
tion to capturing the movement and context of the and limits an untrained individual from being able
user is necessary. Machine vision broadly encom- to seemingly “interact” with the system.
passes all industrial and nonindustrial applications
that utilize photo detective sensors to perceive an
image and utilize software to extract information Machine Vision in EM
from the image (Beyerer et al. 2016). The infor-
mation extracted from the image is then utilized in The usage of Machine Vision in EM is optimized
the system’s overall function. When utilizing to be perceived as instantaneous, or close to real-
machine vision for EM, there is a further distinc- time, with the movement it detects. From an
tion between real-time and offline usage to sense implementation perspective, this often incorpo-
movement in a space. rates software optimized to run on Graphic Pro-
EM is used in live performances, where the cessing Units (GPUs) in order to reach peak
display medium is on stage responding to dancers Frames Per Second (FPS). FPS is the rate at
or performers on stage (Performances Using which the visual content is updated, for example,
Experiential Backdrops – YouTube 2020). Real 5 fps indicates that the system updates the visual
Experiential Media 675

content every 200 milliseconds. From a from the positioning of the display. As such, it is
psychovisual perspective, a human requires visual possible to create EM using upright activations
content with a minimum of 15 FPS (Li 2009), that project on a wall directly in front of the user or
translating to an update every 66 milliseconds, to activate a video wall in a different location
perceive the content as continuous and smooth. entirely. Similarly, it is possible to create EM
Machine vision techniques can be applied with top-down activations that project on the floor as
different hardware in different scenarios to offer a users move through the space or activate the ceil-
range of interactions with dynamic media. Further ing or a tabletop surface.
perspective considerations are required as parts of For different setup variations, different camera
a user’s body may become obscured or no longer types may be more appropriate. By large there are
E
directly visible to the camera. From a broader three main camera types that can be used for
framing consideration, the angle from which the different variants of interactions. They are as
camera perceives the user and the angle from follows:
which the user views the display medium need
to be accounted for. Other considerations include – IR cameras: utilized with infrared (IR) markers
broader viewers not interacting with the system that the user wears on whatever body points or
and their perspective to the user and the display objects would like to be tracked, providing the
medium showing the dynamic content. EM system with a set of (x,y) coordinates
Other than the media-generation procedures (Nawrat et al. 2013).
that are known to use GPU, machine vision – Depth cameras: typically utilizing projected
approaches are becoming increasingly optimized infrared mapped with structured light or stereo
for GPUs. Some of the common approaches cameras that gives a full RGB-D (red green
include: blue – depth) of the space, providing the EM
system a full pixel map (Smisek et al. 2013).
– RGB cameras: the standard camera used in
Gesture tracking and recognition: tracking
most laptops, security cameras, and phones,
markers to recognize specific movements
providing the EM system an rgb pixel map.
(Dubnov and Wang 2015; Wang and Dubnov,
2015)
The RGB cameras are better suited if the scene
Bodypix: a deep learning based method to extract
is only meant to react to human movement. If the
a body contour from an rgb image (Wang and
scene is meant to react to broader movement, such
Dubnov 2015)
as moving objects or those that require a sharp
Human Pose Estimation: a deep learning based
contour, then the depth camera is appropriate and
method to extract body joins from an rgb image
can be used with depth-thresholding. If the screen
(Su et al. 2020).
is only meant to react to the movement of specific
objects in the space then an IR camera may be
Setup and Hardware Options most appropriate. If the EM is designed to only
react to human movement then an RGB camera
EM constitutes multiple pieces: the sensing hard- with subsequent artificial intelligence
ware, the display medium, and the spatial area the (AI) methods, such as human pose estimation
user is detected in. EM falls into two main setup that can be abstracted into contours or a set of
variations as can be seen in Fig. 2, upright (left) body joint coordinates (Sigal 2014), may be
and top-down (right). (Eye Setup 2020.) Upright enough. With the top-down setup scenarios, a
configurations position the camera upright, paral- depth camera may be most appropriate as AI
lel to the floor. Top-down configurations position models are often not trained on birds-eye-view
the camera facing perpendicular to the floor, sim- of humans.
ilar to a birds-eye-view looking down. The posi- Some hardware solutions may support the dif-
tion and orientation of the camera is independent ferent variants simultaneously. One of the early
676 Experiential Media

breakthroughs introducing creative coding and the amount of time it takes a user to realize an EM
EM to the larger population came with the Micro- installation is interactive, in an unprompted envi-
soft Kinect that was released with the Xbox in ronment, and the overall length of interactions the
2010 (Smisek et al. 2013). At the time of this user will engage with EM depends on the users
writing, a prominent provider for depth cameras, demographic. As a reference, children have been
for EM and broader machine vision tasks, is seen to engage with a single EM experience sig-
Intel’s RealSense (Intel ® RealSense 2020) series nificantly longer than adults, whereas adults have
that offers the most high resolution and physically been seen to have increased engagement time if
compact solution in the market that can support the experience includes multiple triggers that
any of the three variants described. The other large allows them to transition between content quickly
non-US player is Orbbec Astra that offers a depth- to explore the entire available set of EM. Depth
camera solution over usb-2 (Orbbec 2020). Of cameras are seeing an increased demand after
course more providers exist, but they are outside privacy and hygiene awareness rose and as EM
the scope of this entry. offers a touchless solution, it is becoming increas-
ingly attractive. Additionally, 3D cameras have
the capability to work in very low lights and also
Analytics in no-light environments like theatres and
museums giving creators additional creative
As EM often utilizes a camera to capture RGB, flexibility.
broad demographic analytics can be aggregated
about the users. The measures can further be tal-
lied against the specific experiential media
displayed at the time that the metric was recorded. Cross-References
EM can have triggers inside the media, where a
specific movement by a user can trigger the tran- ▶ Augmented Reality Entertainment: Taking
sition to a different media. As such, the use of a Gaming Out of the Box
sequence of triggers by users can be recorded to ▶ Mixed Reality
quantify an overall map of the media a user has
experienced and the overall trend on trigger acti-
vations in scenarios where multiple triggers are References
present.
Beyerer, J., Puente, L.F., Frese, C.: Introduction. In:
Machine vision. Springer, Berlin/Heidelberg (2016).
https://doi.org/10.1007/978-3-662-47794-6_1
Discussion Dubnov, T.: Interactive projection for aerial dance using
depth sensing camera. SPIE 9012, The Engineering
When designing EM, the effectiveness and impact Reality of Virtual Reality 2014, 901202 (28 February
of the interaction depends on the content design 2014). https://doi.org/10.1117/12.2041905
Dubnov, T., Wang, C.: Free-body gesture tracking and
and is affected by context, such as location, the
augmented reality improvisation for floor and aerial
users demographics, and more. Broadly speaking, dance. (2015). https://doi.org/10.13140/2.1.
the memorability of the EM compared to tradi- 2863.1845.
tional media is enhanced due to its multisensory Ebert, A., Deller, M., Steffen, D., Heintz, M.: “Where did i
put that?” – effectiveness of kinesthetic memory in
nature as it involves visual, audio, and kinesthetic immersive virtual environments. In: Stephanidis,
senses – where their combination has been shown C. (eds.) Universal Access in Human-Computer Inter-
to improve the associated memory (Ebert et al. action. Applications and Services. UAHCI 2009. Lec-
2009). When interacting with the EM, there is a ture Notes in Computer Science, vol. 5616. Springer,
Berlin/Heidelberg (2009). https://doi.org/10.1007/978-
barrier to entry for users in that they need to 3-642-02713-0_19
understand the interaction each time so that they Eye Setup – Set the Camera and Content Window | Zuzor
can move appropriately. It has been observed that App. 20 Aug 2019, https://www.zuzorapp.com/forum/
Exploring Innovative Technology: 2D Image Based Animation with the iPad 677

tutorials/eye-setup-set-the-camera-and-content-
window. Accessed Sep 2020 Exploring Innovative
Huang, C., Lin, F.S.: Exploring Visitors’ Experiential
Experience in the Museum: A Case of the National Technology: 2D Image Based
Museum of Taiwan Literature. National Yunlin Univer- Animation with the iPad
sity of Science and Technology (2013)
Intel ® RealSense.: https://www.intelrealsense.com/. Jennifer Coleman Dowling
Accessed Sep 2020
Kasapakis, V., Gavalas, D., Dzardanova, E.: Mixed reality. Communication Arts Department, Framingham
In: Lee, N. (ed.) Encyclopedia of Computer Graphics State University, Framingham, MA, USA
and Games. Springer, Cham (2018). https://doi.org/10.
1007/978-3-319-08234-9_205-1
Li, Y.: Video representation. In: LIU, L., ÖZSU, E
M.T. (eds.) Encyclopedia of Database Systems. Synonyms
Springer, Boston (2009). https://doi.org/10.1007/978-
0-387-39940-9_1441 2-dimensional animation; Innovative technology;
LUMOplay.: https://www.lumoplay.com/. Accessed Sep iPad animation; Mobile devices
2020
Nawrat, A., Daniec, K., Warmuz, T.: Object detection
using IR camera. In: Nawrat, A., Simek, K., Świerniak,
A. (eds.) Advanced Technologies for Intelligent Sys- Definition
tems of National Border Security Studies in Computa-
tional Intelligence, vol. 440. Springer, Berlin/
Heidelberg (2013). https://doi.org/10.1007/978-3-642- 2D image based animation with the iPad is the
31665-4_11 process of developing new visualization and pro-
openFrameworks.: https://openframeworks.cc/. Accessed duction methods with the use of a mobile device,
Sep 2020 while simultaneously providing theoretical and
Orbbec – Intelligent computing for everyone everywhere.
https://orbbec3d.com/. Accessed Sep 2020 practical instruction of fundamental animation
Performances Using Experiential Backdrops – YouTube. techniques.
17 Nov 2019, https://www.youtube.com/watch?v=4t_
5QBsYi1w. Accessed 29 Dec 2020
Sigal, L.: Human pose estimation. In: Ikeuchi, K. (ed.)
Computer Vision. Springer, Boston (2014). https:// Introduction
doi.org/10.1007/978-0-387-31439-6_584
Smisek, J., Jancosek, M., Pajdla, T.: 3D with Kinect. In: Teaching computer animation techniques using
Fossati, A., Gall, J., Grabner, H., Ren, X., Konolige,
K. (eds.) Consumer Depth Cameras for Computer
innovative approaches was made possible for me
Vision. Advances in Computer Vision and Pattern Rec- with two consecutive “Teaching with Technol-
ognition. Springer, London (2013). https://doi.org/10. ogy” grants from Framingham State University.
1007/978-1-4471-4640-7_1 The goal of these grants is to enhance faculty
Su, Z., Xu, L., Zheng, Z., Yu, T., Liu, Y., Fang, L.:
RobustFusion: human volumetric capture with data-
competencies and improve student engagement.
driven visual cues using a RGBD camera. In: Vedaldi, iPads were procured for inventive ways to learn
A., Bischof, H., Brox, T., Frahm, J.M. (eds.) Computer digital animation and time-based media for artistic
Vision – ECCV 2020. ECCV 2020. Lecture Notes in and commercial purposes.
Computer Science, vol. 12349. Springer, Cham (2020).
https://doi.org/10.1007/978-3-030-58548-8_15
In this entry, I will share how the use of tech-
Sundaram, H., Rikakis, T.: Experiential media systems. In: nology has enriched and broadened the academic
Furht, B. (ed.) Encyclopedia of Multimedia. Springer, experience for students learning computer anima-
Boston (2008). https://doi.org/10.1007/978-0-387- tion. I will also cover the goals and outcomes of
78414-4_317
Sutcliffe, A.: Human-computer Interface Design. Macmil-
this research, including student survey results,
lan, UK (1995). https://doi.org/10.1007/978-1-349- assessments, and animation examples. While
13228-7 learning to animate with iPads, students sketched
Unity.: https://unity.com/. Accessed Sep 2020 ideas, brainstormed, planned narrative and story-
Wang, C., Dubnov, S.: The variable markov oracle: Algo-
rithms for human gesture applications. IEEE MultiMe-
telling structures, conducted research, and pre-
dia. 22(4), 52–67 (2015) sented and shared their work. In addition, they
Zuzor.: https://www.zuzor.co/. Accessed Sep 2020. had ongoing opportunities for collaborating with
678 Exploring Innovative Technology: 2D Image Based Animation with the iPad

one another on group projects, exchanging ideas, Overview of Creating 2D Animation

discussing work, and giving/receiving feedback. on the iPad

The technological benefits to students were that

State-of-the-Art Approaches the iPad provided them with tools to collaborate,
experiment, and expand their skill set. It allowed
Complementary tactics with iPads included the students to develop new visualization capabili-
studying of historical and contemporary figures ties and production methods while simulta-
in the animation field; sketching characters, neously providing the theoretical and practical
scenes, and storyboards; manipulating timeline instruction of fundamental animation techniques.
key frames and stage elements, and adjusting It also facilitated a more imaginative process for
camera views; digitizing and editing audio tracks; solving problems, discovering inspiration, creat-
and capturing and manipulating photography and ing concepts, and exchanging ideas so they could
video. They also digitized and edited audio tracks; more fully develop their knowledge of the sub-
and capturing and manipulating photography and ject while building more versatile computer ani-
video. The iPad is commonly used for consump- mation capabilities. The technology that was
tion, such as watching videos and reading articles. distributed and available to students consisted
It is not typically viewed as a tool for creating of: iPads with retina display, stylus pens, protec-
computer graphics and animation (Dowling tive cases with keyboards, and iPad stands. In the
2012). In my project, iPads in addition to iMacs computer lab they had access to iMacs, Wacom
were used as tools to supplement the teaching of digitizing tablets with stylus pens, a scanner,
animation. speakers, microphones, a large digital display,
and Apple TV. Some of the apps that were pro-
vided and used on the iPads were: Adobe Ideas
Summary of Work (vector drawing), Adobe PS Express (photo edi-
tor), Animation Creator (frame-by-frame anima-
The “Teaching with Technology” grant funding tion), Animation Desk (high end animation),
allowed me to explore new instructional CELTX Scripts (screenwriting), CELTX Shots
approaches and inventive ways for students to (storyboarding), DoInk (character animation),
learn digital animation and time-based media for Flip Boom Cartoon (digital flipbook),
artistic and commercial purposes. One objective GarageBand (audio creating, recording, and
was to minimize technical variables by utilizing editing), iMotion (stop motion animation) and
identical mobile devices, so as to eliminate such Stop Motion Studio (stop motion animation),
problems as computer lab restrictions and lack of and Storyboards (narrative storytelling).
available software on laptops, and to encourage There were advantages to teaching animation
synchronicity. The iPads allowed students to use with mobile devices, which included: the mini-
one primary device for learning new topics and mizing of technical variables by utilizing identical
completing projects in a timely manner, which mobile devices; allowing students to conduct
provided the opportunity to adapt my way of ongoing research, write, create, record ideas,
teaching existing design theories to a new digital take notes, access course content, present and
platform. They also exposed students to new con- share work, and collaborate using one device, so
cepts and helped to build their skill base with as to learn new topics and complete projects in a
innovative tools (Davis 2012). Students were timely manner.
able to garner capabilities that would not be pos- Assessment tools were used to monitor what
sible with the iMacs or laptops alone. The iPads students were learning. They were beneficial as a
also facilitated animation creation by giving stu- means of informing pedagogical decisions. The
dents unlimited access to the same technology. following assessment tools were used: surveys at
Exploring Innovative Technology: 2D Image Based Animation with the iPad 679

the beginning and end of the semester, rubrics for iPad for creating animation compare with the
grading project assignments, journal for under- Mac?
standing the learning process, and self-
evaluations and critiques. Students were also
asked to write in a journal every three weeks and Experiments and Examples
these entries were only shared with me. Questions
included: What are your initial thoughts about the Projects that my students worked on and topics
course with respect to the technology provided? they experimented with included: researching ani-
How has your project development evolved (i.e., mators, critical analysis presentation, storyboards
research, concepts, current progress, software and scripts for narrative storytelling, digital audio
E
covered)? What are you learning, how are you and video recording and editing, introductory
learning it, and what do you want more (or less) logo sequences, and web banner ads, kinetic
of at this point in the semester? typography, character animation, stop motion
Students were also given surveys to fill out at animation, social cause animation (public service
the beginning and end of the semester. Some of announcement), and photographic manipulation
the final exit survey questions were: Do you use with iPad apps and iMac software. Examples of
the iPad or a computer for viewing examples and some of the student animations can be found on
animation videos, and participating in discus- my YouTube channel: https://www.youtube.com/
sions? Has the iPad been an asset or a hindrance channel/UCepa8uuWVj5H0-19yuFUoEw (Figs. 1
in your learning of animation? How does using an and 2).

Exploring Innovative Technology: 2D Image Based Animation with the iPad, Fig. 1 Students working in
Animation Studio Fall 2013
680 Exploring Innovative Technology: 2D Image Based Animation with the iPad

Exploring Innovative Technology: 2D Image Based Animation with the iPad, Fig. 2 Examples of student work
from computer animation techniques class

Conclusion and Goals virtual office hours, FaceTime for one-on-one assis-
tance, eBook required in addition to web resources,
Success when teaching with iPads included: iPads voice dictation for my comments and for students
for sketching, storyboards, planning narrative posing questions, video capture for teaching how to
sequences, conducting research, demonstrating use software (uploaded to Blackboard), sharing
techniques, presenting work; stylus pens for draw- ideas on class blog so as to keep ongoing dialogue,
ing and general use; student surveys; Blackboard for group work and collaborative projects, YouTube
communication and posting course material; and/or Vimeo uploads for regular online presence,
Dropbox & Google Drive for project submission social media conversations/interactions/sharing
and file storage; student interaction: exchanging (Google Communities, Facebook, or Twitter).
ideas and assisting each other in class; apps on
iPad: FlipBook, DoInk, Animation Creator,
GarageBand, and iMotion. References
My future goals and approaches to technology
include more interaction outside of class using iPads Davis, A.M., et al.: Technology enhancement tools in an
and/or other social media approaches, inviting visit- undergraduate biology course. Educause Review.
http://er.educause.edu/articles/2012/12/technology-
ing lecturers (live or via Skype), mobile classroom
enhancement-tools-in-an-undergraduate-biology-course
both on and off campus, students demonstrate tech- (2012)
niques to classmates on large display and/or gather Dowling, J.C.: Multimedia Demystified. McGraw-Hill,
in small groups, Blackboard collaborate sessions for New York (2012)
Eye Tracking in Virtual Reality 681

Jennifer Coleman Dowling is an experienced new media Definitions

specialist, designer, educator, author, and artist. She holds
an M.F.A. in Visual Design from the University of Massa-
chusetts Dartmouth and a B.A. in Studio Art from the Eye tracking techniques have many applications
University of New Hampshire. Dowling is a Professor in of research in medicine, psychology, market-
the Communication Arts Department at Framingham State ing, and human factors. It is also used as a
University in MA focusing on Integrated Digital Media. human–computer interface for applications such
She has been dedicated to her teaching and professional
work for over 25 years and is the author of Multimedia as gaze-based typing and communication
Demystified published by McGraw-Hill. Her current line of (Majaranta 2012; Ten Kate et al. 1979), driving
research and practice is analog-digital approaches safety (Chen et al. 2018; Grace et al. 1998; Kutila
pertaining to media, fine art, and design. et al. 2007; Sinha et al. 2018), and gaming (Smith
E
and Graham 2006; Tobii Gaming n.d.). Besides
being a research tool and human–computer inter-
face in VR (virtual reality), gaze-based techniques
Extended Malossi Alphabet are also used to enhance the graphics quality
and performance of displays with methods of
▶ Data Gloves for Hand and Finger Motion gaze prioritized graphics, also known as foveated
Interactions rendering. Furthermore, statistical models of eye
tracking data are employed to provide eye move-
ments for computer-generated avatars (Gemmell
et al. 2000; Seele et al. 2017; Vinayagamoorthy
et al. 2004).
Extended Reality
The human sight is limited to 135 vertically
and 160 horizontally, but the detailed sense of
▶ 3D Puzzle Games in Extended Reality Environ-
vision occurs only within a 5 central circle which
ments
is projected to fovea centralis (or fovea), the reti-
▶ Immersive Technologies for Accessible User
nal region tightly packed with color cone recep-
Experiences
tors. By detecting the movements of the eye, it
▶ Virtual Reality and Robotics
is possible to distinguish its rapid movements
from longer fixations to detect a person’s point
of regard.
There are four techniques mainly used for mea-
Extended Reality Spectrum suring eye movements (Duchowski 2017: 49–56).
Electro-oculography (EOG) is based on sensors
▶ Artificial Reality Continuum around the eye that measures changes on the skin
conductance due to eye movements. Another
technique uses scleral contact lenses that usually
employ a search coil that produces electro-
Eye Tracking in Virtual Reality magnetic signals. Photo-oculography (POG) and
video-oculography (VOG) techniques are less
Mehmet Ilker Berkman invasive since they require investigation of state
Communication Design, Bahçeşehir University of the eye visually. These techniques are slightly
Faculty of Communication, Istanbul, Turkey different from the video-based combined pupil
and corneal reflection, as they do not only mea-
sure eye movements, but also measure the size of
Synonyms the pupil and position of limbus. Unlike the other
three techniques, the video-based combined pupil
Foveated rendering; Gaze prioritized graphics; and corneal reflection method does not require
Gaze tracking; Gaze-contingent displays users’ head to be fixed, since it captures the
682 Eye Tracking in Virtual Reality

corneal reflection of a light source relative to the manipulating the scene elements in response to
position of pupil. The relative positions of the gaze direction. The expressive use of gaze-based
corneal reflection and pupil indicate the user’s interactions may utilize real-time or prerecorded
gaze direction. For example, if the corneal reflec- eye tracking data besides algorithms for synthe-
tion is below the pupil, the gaze direction of the sizing eye movements of virtual avatars.
user is directed above the camera (Poole and Ball
2006). The corneal reflection is usually acquired
using infrared (IR) light sources. The video-based Eye Tracking in Context of Virtual Reality
combined pupil and corneal reflection technique
is broadly used in the contemporary eye tracking The studies considered here were selected with
devices, including the VR equipment with eye a focus on eye tracking applications combined
tracking capabilities. with VR systems, but eye tracking studies
Two primary measurements used in eye track- in other fields that contribute to VR were also
ing research are fixations and saccades. Fixations considered. As the VR mainly depends on 3-D
are stabilization of retina over a stationary object (3-dimensional) virtual worlds, the methods
of interest. Saccades are rapid eye movements which are integrating 3-D objects and eye tracking
used in repositioning the fovea to a new location are mentioned, followed by a brief review of
in the visual environment. These movements can gaze-based interaction work in VR according to
be voluntary in order to switch between objects of the taxonomy of Duchowski (2018). The scope of
interest. Otherwise, they can be smooth pursuits VR is limited to HMD (head mounted display)
which are invoked as a corrective optokinetic or and CAVE (cave automatic virtual environment)
vestibular measure. The vestibular smooth pur- systems.
suits occur due to the head movements of the
observer while optokinetic pursuits are inter- Estimation of Point of Regard in Three-
spersed with saccades invoked to compensate for Dimensional Spaces
the retinal movement of the target. In the context Rötting et al. (1999) used a scene camera along
of gaze-contingent system design, the identifica- with the eye tracking cameras to determine the
tion of fixations, saccades, and smooth pursuits is point of regard in the 3-D real world, using a two-
a primary requirement (Duchowski 2017: 39–46). step offline process. At the first step, at least two
The eye-movement based research has its his- views from scene camera are used to detect the
torical roots in cognitive research on reading contours of the object in the real world to create
(Rayner 1998). Although the characteristics of a geometry model that approximates object in the
eye movements differ across different tasks such space. At the second step, the fixations mapped on
as reading, scene perception, and visual search the image of the scene camera were classified
(Rayner 2009), the fixation of the gaze can be on each frame that the observed model was deter-
considered as “the top of the stack” of the cogni- mined. Methods used in VR reality follow a sim-
tive processes, the focus of attention (Just and ilar approach. Using HMDs, where the content is
Carpenter 1976). already being described by geometries, the sys-
Duchowski (2018) categorized gaze- tems could directly use the projection rendered for
based interactions into four: diagnostic, active, the particular eye from the framebuffer (Pfeiffer
passive, and expressive. The diagnostic methods 2012). As the eye position to the display is fixed,
use offline measurement of eye movements the detected position of the eye on the display can
for research purposes, training, or assessment of be used to cast a ray into the 3-D world to detect
expertise. Active methods use real-time eye track- collisions with 3-D object geometry (Duchowski
ing data as a human–computer interface for selec- et al. 2000, 2002). The methods using this
tion tasks or inputs based on gaze gestures. geometry-based approach assume that the first
Passive methods utilize eye tracking in gaze- object that the casted ray intersects is the point
contingent displays for foveated rendering, of regard. Monocular eye tracking is sufficient for
Eye Tracking in Virtual Reality 683

geometry-based approaches but binocular visualize eye tracking data on both real-world
eye tracking enables detecting the vergence of objects and virtual environments. His method pro-
the eyes to estimate fixation depth (Pfeiffer vides a volume-based modeling of visual atten-
et al. 2008). tion. Later, Pfeiffer and Memili (2016) developed
“attention textures” as a method of representing
Uses of Eye Tracking in VR gaze fixations on 3-D objects.

Diagnostic Use and Visualization of Gaze Data Active Use of Eye Tracking in VR as a Control
Duchowski et al. (2000) developed an HMD Interface
with eye tracking capabilities to track the users’ Mine (1995) described gaze-pointed selection
E
eye movements in a virtual aircraft cargo bay among the other interaction methods. A draw-
training system. Their work is an early application back in gaze-based interfaces is the Midas touch
of matching user’s gaze direction on a planar problem, i.e., the unintentionally activated com-
surface with the polygonal 3-D virtual environ- mands while the user looks at an interactive
ment (Duchowski et al. 2002). The diagnostic element (Jacob 1995). Tanriverdi and Jacob
use of eye tracking, such as in usability studies, developed a system (2000) to compare the
require the visualization of gaze movements gaze-based interaction to point-based interaction
in forms of attentional maps (or heatmaps) that and revealed that gaze-based interaction is effi-
represent the density of fixations or as scanpaths cient in a virtual environment explored via
that describe saccade-fixade-saccade sequences. HMDs, especially to interact with distant objects.
Blascheck et al. (2017) reviewed these methods Pfeiffer (2008) with Haffegee and Barrow (2009)
as point-based and area-of-interest based developed systems that employ gaze input to
methods. Point-based methods of analyzing 3-D interact with objects in CAVE-like environ-
data were proposed for nonstereoscopic 3-D ments, defining a ray-casting approach to trans-
games (Ramloll et al. 2004) or real-world envi- form users’ gaze into 3-D environment.
ronments mapped into 3-D computer models Greenwald et al. (2016) manufactured a card-
(Paletta et al. 2013). Several area-of-interest- board VR system with eye tracking capability
based methods are proposed for visualization of for user interactions.
real-world eye tracking data (Baldauf et al. 2010; Gaze can be used as a “lazy” method to
Itoh et al. 1998, 2000; Schulz et al. 2011; Tsang eliminate and minimize hand movements, as a
et al. 2010) with an exception of work by “helping hand” to extend existing hand-based
Duchowski et al. (2002), which is a pioneering interactions, or “hand down” methods in which
use of eye tracking in virtual reality. The studies gaze provide a context in VR for an additional
that include both point-based and area-of-interest- hand-held device such as a tablet computer
based methods provide techniques for real-world (Zeleznik et al. 2005). Novel interaction methods
settings such as flight simulator cockpits (Weibel can be evaluated within the context of these
et al. 2012) as well as 3-D virtual environments. three classifications, e.g., duo-reticles, radial-
Stellmach et al. (2010) proposed three methods pursuit, nod and roll (Piumsomboon et al. 2017)
for 3-D attentional maps that are applicable to as “lazy” methods, or gaze + pinch (Pfeuffer et al.
VEs (virtual environments). Projected attentional 2017) and VRpursuits (Khamis et al. 2018) as
maps are 2-D planar representations of 3-D fixa- “helping hand” approaches.
tion data. Object-based attentional maps represent Currently, there are several HMDs with eye
models’ attractiveness by assigning a color on its tracking capabilities on the market, which pro-
whole textural surface. Surface-based attentional vide software development kits and add-ons for
maps depict aggregated fixation data directly on different VE development platforms to transfer
the textural surface by attributing gaze positions user’s gaze direction for interacting with
on mesh triangles of models. Pfeiffer (2012) objects in the 3-D environment (Hollomon
established a holistic approach to collect and et al. 2017).
684 Eye Tracking in Virtual Reality

Passive Use of Eye Tracking in VR for Gaze depth (Watson et al. 1997). Recent research
Prioritized Graphics exploits this approach to vary the streaming
An early suggestion to use eye tracking for ren- video bit rate according to observer’s gaze
dering gaze prioritized 3-D graphics described (Arndt and Antons 2016).
several computational methods (Ohshima et al. A hybrid of both object-based and pixel-based
1996), but it is not clear that the proposed system methods have also been explored (Murphy
is a simple prototype at proof-of-concept level or a et al. 2009).
head-mounted display with working eye tracking The research on increasing user comfort of
capabilities. Former studies (Iwamoto et al. 1994) stereoscopic displays utilizes gaze-contingent
suggested using mechanical/optical solutions for rendering to simulate depth of field (Kellnhofer
foveated rendering on HMDs. Duchowski et al. et al. 2016; Duchowski et al. 2014). Typically,
(2000) applied several algorithms to match gaze the human eyes converge and accommodate
direction with polygonal 3-D objects. Those algo- at the same point; but in 3-D displays, conver-
rithms evolved into gaze prioritized rendering gence occurs in front of or behind the screen,
methods (Murphy and Duchowski 2001). The where the image forms while the accommodation
latency of eye trackers had been an issue in occurs on the surface of the flat screen. This
foveated rendering (Triesch et al. 2002), and accommodation-vergence conflict often leads to
research revealed that “foveated rendering is tol- a discomfort in viewers of stereoscopic displays
erant to eye-to-image latencies of about 50–70 ms, (Mon-Williams and Wann 1998) and it is one of
beyond which it is easily perceived and hence the reasons of cybersickness (Carnegie and Rhee
loses effectiveness” (Albert et al. 2017). Current 2015). To overcome this problem in VR, objects
research focuses on the comparison of different in the 3-D world that do not lie on the convergence
foveated rendering techniques in terms of hard- point of user’s eyes are blurred to simulate real-
ware performance and user perception (Albert world depth of field.
et al. 2017; Pohl et al. 2016; Swafford et al. The passive gaze-contingent techniques also
2016; Roth et al. 2016). employ methods that estimate the gaze move-
There are two objectives of gaze prioritized ments and fixations, to overcome the latency
methods: increasing the rendering performance which occurs between eye tracking and rendering
and increasing user comfort (Duchowski 2018). (Arabadzhiyska et al. 2017; Kulshreshth and
The speed of rendering is increased by freeing up Laviola 2016). Some research focuses on employ-
the computational resources through matching ing models of estimation without using real-time
user’s “retinal and visual acuity resolutions” eye tracking input, employing saliency maps
with the resolution of the area viewed by the (Advani et al. 2013; Swafford et al. 2016), or
user, leaving the nonfoveated areas rendered in a using machine learning methods to predict the
lower resolution. To achieve this, model- or pixel- important objects (Koulieris et al. 2015, 2016).
based approaches can be followed. Model-based Lavoué et al. (2018) compared 3-D eye tracking
approaches employ polygonal simplification results and several saliency mapping algorithms
(Levoy and Whitaker 1990; Luebke and Erikson and detected that algorithms still remain poor at
1997; Luebke and Hallen 2001; Zha et al. 1999) predicting fixation locations.
by controlling the level of detail in polygonal
objects. Current research focuses on enhancing Expressive Use of Eye Tracking in VR
the algorithms and optimizing the rendering field The real-time eye movements of user captured
to improve the image quality perceived by users via eye tracking, prerecorded eye tracking data,
(Patney et al. 2016; Pohl et al. 2016; Roth et al. or algorithms developed to synthesize the eye
2017; Weier et al. 2016). movements are used to increase realism of avatars
The pixel-based approach focuses on spatial or virtual characters, as well as to increase the
and temporal complexity of pixel data just prior perceived quality of communication in a shared
to rendering, e.g., decreasing resolution or color virtual environment. The visual deictic reference
Eye Tracking in Virtual Reality 685

enslaved to users’ gaze found to be enhancing Reality. ACM Transactions on Applied Perception.
the trainee performance in collaborative VEs 14, 1–13 (2017)
Arabadzhiyska, E., Tursun, O., Myszkowski, K., Seidel,
using HMDs (Duchowski et al. 2004; Sadasivan H., Didyk, P.: Saccade landing position prediction for
et al. 2005). In a multiparty object-focused puzzle gaze-contingent rendering. ACM Transactions on
scenario conducted on a CAVE-like environment Graphics. 36, pp. 1–12 (2017)
(Steptoe et al. 2009), the tracked gaze was obser- Arndt, S., Antons, J.N.: Enhancing video streaming
using real-time gaze tracking. In: Proceedings of the
ved as the highest performing condition. Murray 5th ISCA/DEGA Workshop on Perceptual Quality of
et al. (2009) implemented an avatar-based video Systems, pp. 6–9 (2016)
conferencing for CAVE-like VR systems that sup- Baldauf, M., Fröhlich, P., Hutter, S.: KIBITZER:
ports eye tracking and they evaluated the role of a wearable system for eye-gaze-based mobile
urban exploration. In: Proceedings of the 1st E
gaze in remote communication. Recently, Roberts Augmented Human International Conference,
et al. (2015) implemented an eye tracker in their pp. 9–13 (2010)
end-to-end telepresence system. Blascheck, T., Kurzhals, K., Raschke, M., Burch, M.,
Duchowski (2018) criticized the method of Weiskopf, D., Ertl, T.: Visualization of eye tracking
data: a taxonomy and survey. Comput. Graph. Forum.
mapping prerecorded eye movements to a virtual 36, 260–284 (2017)
character as it is costly since this method requires Carnegie, K., Rhee, T.: Reducing visual discomfort with
an actor for every recording while eye tracking HMDs using dynamic depth of field. IEEE Comput.
devices inject noise to eye movement data during Graph. Appl. 35, 34–41 (2015)
Chen, L.B., Chang, W.J., Hu, W.W., Wang, C.K., Lee, D.
recording. Alternatively, synthetic eye move- H., Chiou, Y.Z.: A band-pass IR light photodetector for
ments are employed in development of virtual wearable intelligent glasses in a drowsiness-fatigue-
characters and humanistic robots, which involve detection system. In: Consumer Electronics (ICCE),
modeling saccades, smooth pursuits, binocular 2018 IEEE International Conference on, pp. 1–2.
IEEE (2018)
rotations implicated in vergence, coupling of eye Duchowski, A.T.: Eye Tracking Methodology. Theory and
and head rotations, pupil dilations and constric- Practice. Springer International Publishing AG, Cham
tions, and microsaccadic jitters during fixations (2017)
(Duchowski and Jörg 2016). Duchowski, A.T.: Gaze-based interaction: a 30 year retro-
spective. Comput. Graph. 73, 59–69 (2018)
Duchowski, A.T., Jörg, S.: Eye animation. In: Müller, B.,
Wolf, S.I. (eds.) Handbook of Human Motion, Springer
Cross-References Nature, Cham, pp. 1–19 (2016)
Duchowski, A.T., Shivashankaraiah, V., Rawls, T.,
Gramopadhye, A.K., Melloy, B.J., Kanki, B.: Binoc-
▶ Foundations of Interaction in the Virtual Real- ular eye tracking in virtual reality for inspection
ity Medium training. In: Proceedings of the 2000 Symposium
▶ Perceptual Illusions and Distortions in Virtual on Eye Tracking Research & Applications,
pp. 89–96 (2000)
Reality Duchowski, A.T., Medlin, E., Cournia, N.,
▶ Spatial Perception in Virtual Environments Gramopadhye, A., Melloy, B., Nair, S.: 3D eye move-
▶ Virtual Hand Metaphor in Virtual Reality ment analysis for VR visual inspection training.
▶ Virtual Pointing Metaphor in Virtual Reality In: Proceedings of the Symposium on Eye Tracking
Research & Applications – ETRA 02 (2002)
▶ Virtual Reality Game Engines Duchowski, A.T., Cournia, N., Cumming, B.,
▶ Virtual Reality System Fidelity Mccallum, D., Gramopadhye, A., Greenstein, J.,
Sadasivan, S., Tyrrell, R.A.: Visual deictic reference
in a collaborative virtual environment. In: Proceedings
of the Eye Tracking Research & Applications Sympo-
References sium on Eye Tracking Research & Applications –
ETRA2004 (2004)
Advani, S., Sustersic, J., Irick, K., Narayanan, V.: A multi- Duchowski, A.T., House, D.H., Gestring, J., Wang, R.I.,
resolution saliency framework to drive foveation. Krejtz, K., Krejtz, I., Mantiuk, R., Bazyluk, B.: Reduc-
In: 2013 IEEE. International Conference on Acoustics, ing visual discomfort of 3D stereoscopic displays
Speech and Signal Processing (2013) with gaze-contingent depth-of-field. In: Proceedings
Albert, R., Patney, A., Luebke, D., Kim, J.: Latency of the ACM Symposium on Applied Perception –
Requirements for Foveated Rendering in Virtual SAP 14 (2014)
686 Eye Tracking in Virtual Reality

Gemmell, J., Toyama, K., Zitnick, C.L., Kang, T., Seitz, S.: Kulshreshth, A., Laviola, J.J.: Dynamic stereoscopic 3D
Gaze awareness for video-conferencing: a software parameter adjustment for enhanced depth discrimina-
approach. IEEE MultiMedia. 7(4), 26–35 (2000) tion. In: Proceedings of the 2016 CHI Conference on
Grace, R., Byrne, V., Bierman, D., Legrand, J.-M., Human Factors in Computing Systems – CHI 16 (2016)
Gricourt, D., Davis, B., Staszewski, J., Carnahan, B.: Kutila, M., Jokela, M., Markkula, G., Rue, M.R.: Driver
A drowsy driver detection system for heavy vehicles. distraction detection with a camera vision system.
In: 17th DASC. AIAA/IEEE/SAE. Digital Avionics In: 2007 IEEE International Conference on Image
Systems Conference (1998) Processing (2007)
Greenwald, S.W., Loreti, L., Funk, M., Zilberman, R., Lavoué, G., Cordier, F., Seo, H., Larabi, M.: Visual atten-
Maes, P.: Eye gaze tracking with google cardboard tion for rendered 3D shapes. Comput. Graph. Forum.
using purkinje images. In: Proceedings of the 22nd 37, 191–203 (2018)
ACM Conference on Virtual Reality Software and Levoy, M., Whitaker, R.: Gaze-directed volume rendering.
Technology, pp. 19–22 (2016) ACM SIGGRAPH Comput. Graph. 24, 217–223
Haffegee, A., Barrow, R.: Eye tracking and gaze based (1990)
interaction within immersive virtual environments. In: Luebke, D., Erikson, C.: View-dependent simplification of
International Conference on Computational Science, arbitrary polygonal environments. In: Proceedings of
pp. 729–736. Springer, Berlin/Heidelberg (2009) the 24th Annual Conference on Computer Graphics
Hollomon, M.J., Kratchounova, D., Newton, D.C., and Interactive Techniques – SIGGRAPH ‘97 (1997)
Gildea, K., Knecht, W.R.: Current status of gaze control Luebke, D., Hallen, B.: Perceptually Driven Simplification
research and technology literature. Technical report. for Interactive Rendering. Eurographics. 223–234
Federal Aviation Administration, Washington, DC (2001)
(2017) Majaranta, P.: Communication and text entry by gaze.
Itoh, K., Hansen, J.P., Nielsen, F.R.: Cognitive modelling In: Majaranta, P., et al. (eds.) Gaze Interaction and
of a ship navigator based on protocol and eye- Applications of Eye Tracking: Advances in Assistive
movement analysis. Trav. Hum. 61, 99–127 (1998) Technologies, pp. 63–77. IGI Global, Hershey (2012)
Itoh, K., Tanaka, H., Seki, M.: Eye-movement analysis Mine, M.: Virtual Environment Interaction Techniques.
of track monitoring patterns of night train operators: UNC Chapel Hill Computer Science technical report
effects of geographic knowledge and fatigue. Proc. TR95–018. University of North Carolina, Chapel Hill
Hum. Factors Ergon. Soc. Annu. Meet. 44(27), (1995)
360–363 (2000) Mon-Williams, M., Wann, J.P.: Binocular virtual reality
Iwamoto, K., Katsumata, S., Tanie, K.: An eye movement displays: when problems do and don’t occur. Hum.
tracking type head mounted display for virtual reality Factors. 40, 42–49 (1998)
system: evaluation experiments of a prototype system. Murphy, H., Duchowski, A.T.: Gaze-contingent level of
In: IEEE International Conference on Humans, Infor- detail rendering. In: EuroGraphics (2001)
mation and Technology, no. 1, pp. 13–18 (1994) Murphy, H., Duchowski, A., Tyrrell, R.: Hybrid
Jacob, R.J.: Eye tracking in advanced interface design. image/model-based gaze-contingent rendering. ACM
In: Barfield, W., Furness, T.A. (eds.) Virtual Environ- Trans. Appl. Percept. 5, 1–21 (2009)
ments and Advanced Interface Design, pp. 258–288. Murray, N., Roberts, D., Steed, A., Sharkey, P.,
Oxford University Press, New York (1995) Dickerson, P., Rae, J., Wolff, R.: Eye gaze in virtual
Just, M.A., Carpenter, P.A.: Eye fixations and cognitive environments: evaluating the need and initial work on
processes. Cogn. Psychol. 8, 441–480 (1976) implementation. Concurr. Comput. Pract. Exp. 21(11),
Kellnhofer, P., Didyk, P., Myszkowski, K., Hefeeda, M.M., 1437–1449 (2009)
Seidel, H.-P., Matusik, W.: GazeStereo3D: seamless Ohshima, T., Yamamoto, H., Tamura, H.: Gaze-directed
disparity manipulations. ACM Trans. Graph. 35, 1–13 adaptive rendering for interacting with virtual space.
(2016) In: Proceedings of the IEEE Virtual Reality Annual
Khamis, M., Oechsner, C., Alt, F., Bulling, A.: International Symposium, pp. 103–110 (1996)
VRpursuits: interaction in virtual reality using smooth Paletta, L., Santner, K., Fritz, G., Mayer, H.,
pursuit eye movement. Proceedings of the 2018 Inter- Schrammel, J.: 3D attention: measurement of visual
national Conference on Advanced Visual Interfaces - saliency using eye tracking glasses. In: CHI’13
AVI ’18 May 29-June 1, Castiglione della Pescaia, Extended Abstracts on Human Factors in Computing
Italy (2018) Systems, pp. 199–204. ACM (2013)
Koulieris, G., Drettakis, G., Cunningham, D., Mania, K.: Patney, A., Salvi, M., Kim, J., Kaplanyan, A., Wyman, C.,
An automated high-level saliency predictor for smart Benty, N., Luebke, D., Lefohn, A.: Towards foveated
game balancing. ACM Trans. Appl. Percept. 11, 1–21 rendering for gaze-tracked virtual reality. ACM Trans.
(2015) Graph. 35, 1–12 (2016)
Koulieris, G., Drettakis, G., Cunningham, D., Mania, K.: Pfeiffer, T.: Towards gaze interaction in immersive virtual
Gaze prediction using machine learning for dynamic reality: evaluation of a monocular eye tracking set-up.
stereo manipulation in games. In: 2016 IEEE Virtual In: Virtuelle und Erweiterte Realität-Fünfter Workshop
Reality (VR) (2016) der GI-Fachgruppe VR/AR (2008)
Eye Tracking in Virtual Reality 687

Pfeiffer, T.: Measuring and visualizing attention in Sadasivan, S., Rele, R., Greenstein, J.S., Duchowski, A.T.,
space with 3D attention volumes. In: Proceedings of Gramopadhye, A.K.: Simulating on-the-job training
the Symposium on Eye Tracking Research and Appli- using a collaborative virtual environment with head
cations, pp. 29–36 (2012) slaved visual deictic reference. In: Proceedings of
Pfeiffer, T., Memili, C.: Model-based real-time visualiza- HCI International Annual Conference, pp. 22–27
tion of realistic three-dimensional heat maps for mobile (2005)
eye tracking and eye tracking in virtual reality. In: Pro- Schulz, C.M., Schneider, E., Fritz, L., Vockeroth, J.,
ceedings of the Ninth Biennial ACM Symposium on Hapfelmeier, A., Brandt, T., Kochs, E.F.,
Eye Tracking Research & Applications, pp. 95–102 Schneider, G.: Visual attention of anaesthetists during
(2016) simulated critical incidents. Br. J. Anaesth. 106(6),
Pfeiffer, T., Latoschik, M.E., Wachsmuth, I.: Evaluation 807–813 (2011)
of binocular eye trackers and algorithms for 3D gaze Seele, S., Misztal, S., Buhler, H., Herpers, R., Schild, J.:
interaction in virtual reality environments. J. Virtual Here’s looking at you anyway!: how important is real- E
Real. Broadcast. 5(16), 1660 (2008) istic gaze behavior in co-located social virtual reality
Pfeuffer, K., Mayer, B., Mardanbegi, D., Gellersen, H.: games? In: Proceedings of the Annual Symposium on
Gaze + pinch interaction in virtual reality. In: Proceed- Computer-Human Interaction in Play, pp. 531–540.
ings of the 5th Symposium on Spatial User Interaction, ACM (2017)
pp. 99–108 (2017) Sinha, O., Singh, S., Mitra, A., Ghosh, S.K., Raha, S.:
Piumsomboon, T., Lee, G., Lindeman, R., Billinghurst, M.: Development of a drowsy driver detection system
Exploring natural eye-gaze-based interaction for based on EEG and IR-based eye blink detection analy-
immersive virtual reality. 2017 IEEE Symposium on sis. In: Bera, R., Kumar, S., Chakraborty, S.S. (eds.)
3D User Interfaces (3DUI). 18–19 March, Los Advances in Communication, Devices and Network-
Angeles, CA, USA. pp. 36–39 (2017) ing, Springer Nature Pte Ltd., Singapore pp. 313–319
Pohl, D., Zhang, X., Bulling, A.: Combining eye tracking (2018)
with optimizations for lens astigmatism in modern Smith, J.D., Graham, T.C.: Use of eye movements
wide-angle HMDs. In: 2016 IEEE Virtual Reality for video game control. In: Proceedings of the 2006
(VR) (2016) ACM SIGCHI International Conference on
Poole, A., Ball, L.J.: Eye tracking in HCI and usability Advances in Computer Entertainment Technology
research. In: Encyclopedia of human computer interac- (2006)
tion, vol. 1, Idea Group Reference, London, UK Stellmach, S., Nacke, L., Dachselt, R.: 3D attentional
pp. 211–219 (2006) maps: aggregated gaze visualizations in three-
Ramloll, R., Trepagnier, C., Sebrechts, M., Beedasy, J.: dimensional virtual environments. In: Proceedings
Gaze data visualization tools: opportunities and chal- of the International Conference on Advanced Visual
lenges. In: Proceedings of the 8th International Confer- Interfaces, pp. 345–348. ACM (2010)
ence on Information Visualisation, pp. 173–180 (2004) Steptoe, W., Oyekoya, O., Murgia, A., Wolff, R., Rae, J.,
Rayner, K.: Eye movements in reading and information Guimaraes, E., Roberts, D., Steed, A.: Eye tracking
processing: 20 years of research. Psychol. Bull. 124(3), for avatar eye gaze control during object-focused multi-
372–422 (1998) party interaction in immersive collaborative virtual
Rayner, K.: The 35th Sir Frederick Bartlett lecture: environments. In: 2009 IEEE Virtual Reality Confer-
eye movements and attention in reading, scene percep- ence (2009)
tion, and visual search. Q. J. Exp. Psychol. 62(8), Swafford, N., Iglesias-Guitian, J., Koniaris, C., Moon, B.,
1457–1506 (2009) Cosker, D., Mitchell, K.: User, metric, and computa-
Roberts, D.J., Fairchild, A.J., Campion, S.P., Ohare, J., tional evaluation of foveated rendering methods.
Moore, C.M., Aspin, R., Duckworth, T., In: Proceedings of the ACM Symposium on Applied
Gasparello, P., Tecchia, F.: withyou – an experimental Perception – SAP ‘16 (2016)
end-to-end telepresence system using video-based Tanriverdi, V., Jacob, R.J.: Interacting with eye movements
reconstruction. IEEE J. Sel. Top. Signal Process. 9, in virtual environments. In: Proceedings of the SIGCHI
562–574 (2015) Conference on Human Factors in Computing Systems,
Roth, T., Weier, M., Hinkenjann, A., Li, Y., Slusallek, P.: pp. 265–272. ACM (2000)
An analysis of eye-tracking data in foveated ray tracing. Ten Kate, J.H., Frietman, E.E., Willems, W., Romeny, B.T.
In: 2016 IEEE Second Workshop on Eye Tracking and H., Tenkink, E.: Eye-switch controlled communication
Visualization (ETVIS) (2016) aids. In: Proceedings of the 12th International Confer-
Roth, T., Weier, M., Hinkenjann, A., Li, Y., Slusallek, P.: A ence on Medical & Biological Engineering, pp. 19–20
quality-centered analysis of eye tracking data in (1979)
foveated rendering. J. Eye Mov. Res. 10(5) pp. 1–12 Tobii Gaming.: https://tobiigaming.com/
(2017) Triesch, J., Sullivan, B.T., Hayhoe, M.M., Ballard, D.H.:
Rötting, M., Göbel, M., Springer, J.: Automatic object Saccade contingent updating in virtual reality.
identification and analysis of eye movement record- In: Proceedings of the Symposium on Eye Tracking
ings. MMI Interakt. 1(2) pp. 1–7 (1999) Research & Applications – ETRA 02 (2002)
688 Eye Tracking in Virtual Reality

Tsang, H.Y., Tory, M., Swindells, C.: eSeeTrack – visual- flight deck. In: Proceedings of the Symposium on Eye
izing sequential fixation patterns. IEEE Trans. Vis. Tracking Research and Applications, pp. 107–114.
Comput. Graph. 16(6), 953–962 (2010) ACM (2012)
Vinayagamoorthy, V., Garau, M., Steed, A., Slater, M.: An Weier, M., Roth, T., Kruijff, E., Hinkenjann, A., Pérard-
eye gaze model for dyadic interaction in an immersive Gayot, A., Slusallek, P., Li, Y.: Foveated real-time ray
virtual environment: practice and experience. Comput. tracing for head-mounted displays. Comput. Graph.
Graph. Forum. 23(1), 1–11 (2004) Forum. 35, 289–298 (2016)
Watson, B., Walker, N., Hodges, L., Worden, A.: Zeleznik, R.C., Forsberg, A.S., Schulze, J.P.: Look-that-
Managing level of detail through peripheral degrada- there: exploiting gaze in virtual reality interactions.
tion: effects on search performance with a head- Technical report, Technical Report CS-05 (2005)
mounted display. ACM Trans. Comput.-Hum. Interact. Zha, H., Makimoto, Y., Hasegawa, T.: Dynamic gaze-
4, 323–346 (1997) controlled levels of detail of polygonal objects in 3-D
Weibel, N., Fouse, A., Emmenegger, C., Kimmich, S., environment modeling. In: Second International
Hutchins, E.: Let’s look at the cockpit: exploring Conference on 3-D Digital Imaging and Modeling,
mobile eye-tracking for observational research on the pp. 321–330 (1999)
F

Face Beautification in Antiage warped projections. The rise of the 3D print tech-
nology promotes the development of the beautifi-
Sun Ruoqi and Lizhaung Ma cation methods of 3D scans, most of which
Shanghai Jiao Tong University, Shanghai, China concentrate on face shape beautification. Due to
the great effect that it has in feature extraction and
expression, deep learning improves the results of
Synonyms beautification by matching the features of input
faces to the attractive ones based on the theory that
Face makeup; Face skin and shape beautification; faces combined with the good features of beauti-
Reversed face beautification ful faces have the perfect appearance. Currently,
numerous companies focus on accelerating the
research and development of human face beauti-
Definition fication applications.
There is a widespread knowledge that the
Face beautification in antiage is a technology of younger face with less wrinkles is more attractive.
making human face beautiful based on the Face makeup is one of the most effective methods
reversed aging revolution, which means that the to enhance a person’s appearance, which can also
algorithm produces a more beautiful face by make the face younger. Furthermore, medical
changing the texture and shape of the original technologies have developed rapidly; thus, an
human face. increasing number of people have cosmetic sur-
gery to achieve the younger effect. Although a
number of celebrities benefit from it, many people
Introduction failed due to the high risk of the surgery. It is
worth for some people to take the risk, while it is
Face beautification is an essential research topic in not necessary for most of other people. But we
the image process and computer graphics, which cannot deny that everyone would like to be more
contains dozens of algorithms based on 2D or 3D beautiful on photos. As technology progresses, it
human faces. The process of 2D image is gener- is convenient for people to increase the quality of
ally divided into two directions. Furthermore, the photos by using image post-processing technol-
main idea of beautification method is to smooth ogy. Face beautification in antiage attracts grow-
textures by using the special filter and to change ing attention in the image processing field, which
the geometry of face shapes by using designed generates numerous algorithms.

© Springer Nature Switzerland AG 2024

N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
690 Face Beautification in Antiage

In this entry, we do a survey and introduce the characteristics. Traditional methods, respectively,
present work in face aging, aesthetic plastic tech- change face textures and shape (Ramanathan and
nology, and reversed age face beautification. Chellappa 2006; Kemelmachershlizerman et al.
2014; Shu et al. 2015). In 2016, Wang proposed
recurrent face aging (RFA) framework based on
State-of-the-Art Work RNN to generate the aging face which produced
more natural results (Wang et al. 2016). However,
Many researchers have been exploring in the this method can only generate younger face stage
image processing field, and they constantly by stage, which cannot produce older faces across
improve face beautification algorithms based on the stage. Generally, the human face aging process
the face aging rule. Firstly, we introduce the pre- has close relationship to wrinkles and shapes
sent face aging algorithms, which are related to (Fig. 2).
the face beautification methods. Secondly, we
briefly survey the plastic technology to explore Aesthetic Plastic Technology
the mysteries of facial aesthetics. Finally, we dis- With the improvement of medical technology,
cuss the reversed age face beautification algo- people can change their face texture and shape to
rithms (Fig. 1). make a younger and more charming face by hav-
ing aesthetic plastic surgery. In 1962, Gonzalez-
Face Aging Ulloa M put forward the principle of aesthetic
Facial aging algorithm is mainly used in cross-age plastic (Gonzalez-Ulloa 1962). Therefore, cos-
face recognition, entertainment, and other fields. metic technology has been receiving increasing
The main idea of the algorithm is to add age features attention. A large amount of cosmetic methods
to the young faces without changing the identity have been proposed, such as antiaging plastic

Face Beautification in Antiage, Fig. 1 The face aging examples from teenagers to adults (Ramanathan and Chellappa
2006)
Face Beautification in Antiage 691

Face Beautification in
Antiage, Fig. 2 The skull
model (Todd et al. 1980)

+
Origin

(R0, )

(R1, )

Original Profile
Profile with k = 0.04

Pressure R0 (1– cos( ))

Profile with k = 0.08
Profile with k = 0.12
F
Profile with k = 0.16
Profile with k = 0.20

Face Beautification in
Antiage, Fig. 3 The
Marquardt mask (Stephen
1997)

surgery (Giampapa et al. 2003), double eyelid more symmetrical face. As mentioned above, Liu
surgery (Song and Song 1985), etc. What’s Fang did the research and showed that the sym-
more, Brennan proposed a method to beautify metrical face is more charming (Liu 2010). Thus,
the skin (Brennan 2015). There are dozens of morphing the face toward symmetry properly can
specific methods to make you look more beauti- improve the face’ attractiveness. But this is only
ful. Some of these patients would like to have a one general rule, while there are also some
692 Face Beautification in Antiage

asymmetric faces which are more beautiful. cosmetics, while there are corresponding algo-
Others who aim to change the shape of their face rithms in image processing area. There are lots
usually choose an attractive star as the target or of company proposing the applications to achieve
look for a generally accepted face. At this time, the demand of consumers, such as Microsoft,
most of the doctors would choose the Marquardt Facebook, Apple, Tencent, etc. Other algorithms
mask, which is the standard face of the aesthetics. are presented to learn the age features and apply
The Marquardt mask is also called the golden face the reversed aging features directly to beautify
for that it conforms to the golden ratio. When the human faces. In 2013, Chen use standard devia-
person who is older than 20 years old grows up, tion to extract the age feature and get younger face
the face shape would go far from the Marquardt by training SVM (Chen et al. 2013). In 2015, Sun
mask, which also means that the face aging has proposed a method to replace the aging skin with
great effect on the attractiveness of human faces neighbor skin with younger features, which pro-
(Fig. 3). duce younger faces without losing the identical
features (Sun et al. 2015). Deep learning devel-
Reversed Age Face Beautification oped rapidly which promotes the development of
In recent years, face beautification in antiage algo- the skin texture beautification. In 2015, Li learns
rithm has made unprecedented progress in the the features which are related to beautification by
image processing field. Face texture is affected using deep learning to generate beautified faces
by five characters, such as spots, wrinkles, puffi- (Li et al. 2015). In 2016, Xi Lu proposed a frame-
ness, dark circles, and shapes. In the real world, work focused on skin beautification based on
people change these characters by using layer dictionary, which is good at removing facial

Face Beautification in Antiage, Fig. 4 Face makeup examples (Liu et al. 2016)
Face Beautification in Antiage 693

Face Beautification in Antiage, Fig. 5 Face beautification examples (Sun et al. 2015)

wrinkles and spots (Xi Lu et al. 2016). These Glampapa, V.C., Fuente, d.C.A., Ramirez, O.M.: Anti-
algorithms focus on different aspects and have aging medicine and the aesthetic surgeon: a new per-
spective for our specialty[J]. Aesthetic Plast Surg.
achieved great results (Figs. 4 and 5). 27(6), 493–501(2003)
Gonzalez-Ulloa, M.: Quantitative principles in cosmetic
surgery of the face (profileplasty). Plast. Reconstr.
Conclusion and Discussion Surg. Transplant. Bull. 29(2), 186–198 (1962)
Kemelmachershlizerman, I., Suwajanakorn, S., Seitz, S.M.:
Illumination-aware age progression. IEEE Conf. Comp.
Facial beautification algorithm develops rapidly; Vision Pattern Recog. (CVPR), pp. 3334–3341 (2014)
there are many companies that invested a great Li, J., Xiong, C., Liu, L., et al. Deep face beautification.
deal of resources for in-depth study. A variety of ACM Int. Conf. Multimed, pp. 793–794 (2015)
face beautification algorithms are proposed Liu, F.: Face attractive research in plastic industry. Mod.
Bus. 33, 271–271 (2010)
recently. Increasingly importance has been Liu, S., Ou, X., Qian, R., et al.: Makeup like a superstar:
attached to the face beautification method in anti- deep localized makeup transfer network. International
age since it protects the personality of human Joint Conference on Artificial Intelligence. AAAI
faces in the beautification process. There are Press, 2568–2575 (2016)
Lu, X., Chang, X., Xie, X., et al.: Facial skin beautification
many applications that can beautify face in real via sparse representation over learned layer dictionary.
time, but these algorithms still have to be Int. Jt. Conf. Neural Netw, pp. 2534–2539 (2016)
improved, such as promoting the sense of reality Ramanathan, N., Chellappa, R.: Modeling age progression
and removing the age characteristics. Face beau- in young faces. IEEE Conf. Comp. Vision Pattern
Recog. (CVPR). 2, 387–394 (2006)
tification algorithm in antiage would have great Shu, X., Tang, J., Lai, H. et al.: Personalized age progres-
development in the future. sion with aging dictionary. IEEE Int. Conf. Comp.
Vision (ICCV), pp. 3970–3978 (2015)
Song, R.Y., Song Y.G.: Double eyelid operations. Aes-
References thetic Plast Surg. 9(3), 173–180 (1985)
Stephen, R.: Marquardt, method and apparatus for analyz-
Brennan, C.: “Skin facts” to 658 optimize aesthetic out- ing facial configurations and components, United
comes. Plast. Surg. Nurs. 35, 42–45 (2015) States Patent (1997)
Chen, Y., Ding, S.H., Gan-Le, H.U., et al.: Facial beautifi- Sun, R., Ma, Y., Fang, H., Ma, L.: Facial beautification
cation method based on age evolution. Comp. Aided algorithm based on reversed age evolution rule. Asian
Drafting Des. Manuf. 4, 7–13 (2013) Con Design Digital Eng. 1–6 (2015)
694 Face Detection

Todd, J.T., Mark, L.S., Shaw, R.E., Pittenger, J.B.: The

perception of human growth. Sci. Am. Sci. Am. 242(2), Facial Recognition
132–144 (1980)
Wang, W., Cui, Z., Yan, Y., et al.: Recurrent face aging. and Emotion Detection
IEEE Conf. Comp. Vision Pattern Recog. (CVPR) in Environmental Installation
(2016) and Social Media Applications

Pensyl William Russell1, Xiaoping Min2 and

Song Shuli Lily3
1
College of Arts, Media and Design, Northeastern
Face Detection University, Boston, MA, USA
2
College of Computer and Information Science,
▶ Digital Images Using Heuristic AdaBoost Haar Northeastern University, Boston, MA, USA
Cascade Classifier Model, Detection of Partially 3
School of Digital Art and Animation,
Occluded Faces Communications University of China, Beijing,
P.R. China

Synonyms
Face Makeup
Emotion detection; Facial recognition; Image pro-
▶ Face Beautification in Antiage
cessing; New media art work; Vision system

Definition
Face Recognition
Real-time detection and analysis of facial recog-
nition and emotion states is a technique that offers
▶ Digital Images Using Heuristic AdaBoost Haar
methods and processes for the control of media
Cascade Classifier Model, Detection of Partially
content, communication via interactive experi-
Occluded Faces
ences and social media.

Introduction
Face Skin and Shape
Beautification Facial recognition technology is a growing area of
interest, where researchers are using these new
▶ Face Beautification in Antiage applications for study in psychology, marketing
and product testing and other areas. There are also
applications where the use of facial image capture
and analysis can be used to create new methods
for control, mediation, and integration of person-
Facial Recognition alized information into web based, mobile apps,
and stand-alone systems for media content inter-
▶ Facial Recognition and Emotion Detection in action. Our work explores the application of facial
Environmental Installation and Social Media recognition with emotion detection, to create
Applications experiences within these domains. For mobile
Facial Recognition and Emotion Detection 695

media applications, personalized experiences can the face detection and facial recognition, which
be layered personal communication. Our current then can be used to control or mediate visual
software implementation can detect smiles, sad- information on surfaces in public spaces or to
ness, frowns, disgust, confusion, and anger allow mobile apps and web-based experiences
(Database FERET). In a mobile media environ- and through social media.
ment, content on a device can be altered to create
fun, interactive experiences, which are responsive
and intelligent. By intersecting via direct commu- Overview
nication between peer to peer mobile apps, moods
can be instantly conveyed to friends and family – Considering historical examples, artists have
when desired by the individual. This creates a explored the use of projected imagery or light
more personalized social media experience. Con- works as a primary medium. These works may F
nections can be created with varying levels of fall into one or more genre or may be in between
intimacy from family members to close friends different genres of art. Looking at examples of
out to acquaintances and further to broader groups installation, or environmental art works, the work
as well. This technique currently uses pattern rec- of Dan Flavin (https://chinati.org/collection/
ognition to identify shapes within an image field danflavin.php) is exemplary in the use of light as
using a Viola and Jones (2001) Haar-like feature a singular imaging medium. Flavin’s work, as he
detector, OpenCV (Bradski and Kaehler 2008), a has described it, is created and experienced in a
“Feret” database (Chang and Lin) of facial strict formalist approach. Formalism focuses on the
images, and a support vector machine library way objects are made and their purely visual
(LibSVM)(Burges 1998; Bolme 2003) to process aspects. Nevertheless, the works, such as Flavin’s,
video or images from a web camera and to identify though static light alter or inform audience spatial
if a face exists. The system processes the detected perception of spaces where they are installed. In our
faces using an elastic bunch graph matching study of the use of interactive elements, can the
(Hlang 2012) technique that is trained to deter- viewer’s perception be altered by the shifting of
mine facial expressions. These facial expressions color or imagery based on responses detected from
are graphed on a sliding scale to match the dis- the viewers themselves? Further, can we use the
tance from a target emotion graph, thus giving an detection of subtle emotional cues to alter the qual-
approximate determination of the user’s mood. ities of the imagery or installation? More recently,
the projection of video or animated imagery on
building facades or in public spaces has become a
State-of-the-Art Work common way to attract viewer engagement. In
these types of new media art work experiences,
Currently, many media artists are using vision such as the 2011 transformed façade of St. Patrick
systems, sensor-based systems, and other technol- Cathedral and the New Museum in New York
ogies to create interactive experiences and medi- (http://www.newmuseum.org/ideascity/view/flash-
ated arts works in public spaces. In many of these light-mulberry-street-installations, these altered
works, the images are projected onto building architectural and public spaces become a “canvas”
facades or use embedded LED arrays on building where images and media content can be viewed
surfaces. In Asia, it is common for newer build- outside of the special circumstance of the gallery
ings to use vast LED arrays on the façade of the or museum. Considering possible ways to allow for
building. These projections and LED arrays can audience interaction, we know that sensors and
use video playback, images changing over time or vision systems are being used to encourage audience
other ways to control the imagery. Our work participation. Can subtle emotional cues be used as
focuses on the possible use of vision systems for well?
696 Facial Recognition and Emotion Detection

Facial Recognition for Artistic, The viewer can change quality of the image by
Environmental Installation in public altering their facial expression Fig. 3.
Space

Detection of emotion states in a public art instal- Facial Recognition in Mobile Apps,
lation to change the environmental elements is Internet Webpage Detection, and
possible. Using webcams positioned in specific Stand-Alone Kiosk Systems
selected locations can capture facial information,
the emotion states can be detected. The detected In mobile apps, detected emotions of a viewer can
state can be used to alter projected imagery, audi- be shared via social media through simple #hashtag
tory ambiance and ambiance of lighting, intensity or Facebook posts. Using HTML5/CSS3 along with
and color. The location of the camera need not be Canvas, apps, and webpages can be used to capture
directly within the installation space. Indeed, the and submit an image to a back-end server applica-
control of the qualities of the imagery, lighting, or tion, which returns a detected emotion state. Apps
ambiance can be collected remotely in other and webpages submit an image to a cloud database.
building location, from the Internet and even by The server listener application listens for images
mobile apps (Fig. 1). arriving, tagged with random user IDs and time
In my work, “MoodModArt,” and the subse- stamps. The listener passes the image to a back-
quent system “MoodRing,” we use emotion end server application, which returns a detected
detection to change the quality of an image emotion state to the listener. The listener then returns
based on detected moods (Fig. 2). “In the result to the webpage or app (Fig. 4).
MoodModArt,” detection of the seven basic emo-
tions states (Ekman 1999) enables responses in
the imagery as a results of the emotion detected. Developmental Work in Facial
The seven basic emotion states as defined by Recognition: Gender and Age
Eckman are the states used for training and detec-
tion in “MoodRing.” Our work in facial recognition began with exper-
If the detected emotion of a viewer is positive, imentation with the detection of gender and age in
the streamed loop of video is vibrant and colorful. public spaces. In our earlier project “HiPOP,” we
If the detected emotion is negative, the streamed were successful in implementing a software tool
loop of video played to a drab and darker view. for facial recognition for use in public spaces. The

Facial Recognition
and Emotion Detection environment altered by
environment altered by detected emotion
in Environmental detected emotion
Installation and Social environment altered by
Media Applications, detected emotion
Fig. 1 A schematic view of
a museum with capture
location and installation
spaces

web camera

web camera
Facial Recognition and Emotion Detection 697

Facial Recognition and Emotion Detection in Environmental Installation and Social Media Applications,
Fig. 2 (a, b) Images from the looped media streams in MoodModArt
F
Facial Recognition POSITIVE
and Emotion Detection
in Environmental HAPPINESS
Installation and Social
Media Applications,
SURPRISE
Fig. 3 Graphing of
emotion states on a
NEUTRAL
continuum from negative to
DISGUST
positive
ANGER

SADNESS
NEGATIVE

Facial Recognition and Emotion Detection in Environmental Installation and Social Media Applications,
Fig. 4 Transfer of captured images to a server application and the return of a detected emotion

focus of this work revolved around the detection www.nist.gov/humanid/colorferet; Chang and
of gender and age. This implementation uses an Lin) of facial images. A support vector machine
image processing approach by identifying shapes (LibSVM) (Burges 1998) was used to classify the
within an image field using methods published by faces to glean attributes such as gender, age, and
Viola and Jones (2001). The technique employed other individual characteristics. The system seg-
a Haar-like features application (Viola and Jones mented the captured image to recognize face
2001; Burges 1998) and a “Feret” database (http:// rectangles. The detected face area is scaled to a
698 Facial Recognition and Emotion Detection

Facial Recognition and Emotion Detection in Environmental Installation and Social Media Applications,
Fig. 5 Detected genders invoke playback of targeted media

64
64 pixel grayscale image and an equalized each with an elastic bunch graph match (EBGM)
histogram is used to increase contrast. The (Wiskott et al. 1997; Hlang 2012). The
OpenCV (Bradski and Kaehler 2008) library was software implementation was designed with
used to detect and segment faces from video two modules to process the captured video
images through the following methods: images and give the resulting detected emotion.
The “ImageNormalizer” module detected the
1. Using a cascade of boosted classifiers working face from an image, cropped, resized to a stan-
with Haar-like features. dard size (90
100 pixels), and converted these
2. Training classifiers by a database of face and to grayscale. The normalized image was input to
nonface images. the EBGM program. Training for detection of
3. Scanning input images at different scales to emotion states in an individual was required for
find regions that are likely to contain faces. accuracy. The technique used a database of fil-
4. A SVM classifier method using data points as a tered images defined with an index set that were
p-dimensional vector was used to detect smiles identified as one of seven emotion states. The
in the captured images, where p is the number “EmotionRecognition” module integrated with
of feature pixels in the image. “ImageNormalizer” so that every captured
frame was normalized and the detected face was
Application of such a system is feasible in stored in normalized form on the fly.
environments where marketing messages can be “EmotionRecognition” used EBGM with the
targeted for individuals based on gender, age, or on the fly normalization program to output a
other cues that can be identified. The design of the detected emotion state from the captured image
system installation allows marketing or media (Fig. 6).
content to be played based on the detection of
certain demographic information detected from
consumers in a retail environment (Fig. 5). Development of Emotion Detection:
Emota v2.0 (Mood Ring)

Development of Emotion Detection: Additional work in the detection of emotion has

Emota v1.0 continued with the version 2.0, entitled
“MoodRing.” This implementation has four mod-
Work on emotion detection in the initial stages ules: MoodRing MFC, MoodRing Core, Weight
used a hybrid approach with a library of images, Trainer, and Database Compressor.
Facial Recognition and Emotion Detection 699

Facial Recognition and Emotion Detection in Environmental Installation and Social Media Applications,
Fig. 6 A screen capture of Emota v1.0 in action

Both MoodRing MFC and MoodRing Core are seven emotion state expressions, and the images
implementations of the project’s core part. are captured and accepted. Once the seven states
MoodRing Core is the interface version which are stored, the system is effectively trained for that
shows how to set up this project under different user (Fig. 8).
platforms. Weight Trainer is used to train weight User emotion detection allows real-time user
of each anchor point to calculate similarity among emotion detection. This can be used as a control
subgraphs. Once a model is trained, elastic for interaction, media, and environmental
bunching graphs (Wiskott et al. 1997; Hlang elements.
2012) can be stored and compared instead of
images. Database Compressor is used to compress
elastic bunching graphs by comparing, searching,
and combing similar graphs based on the distance Emota v2.0 Mood Ring Core and Data
among them. Processor
MoodRing MFC is a stand-alone MFC version
which supports emotion storage, batch training, Image Preprocessing
and emotion detection. There are two options for First, we apply some image standardizations to get
emotion storage: user emotion storage and batch a small size gray scale image. Second, a series of
emotion storage. Batch emotion storage allows image preprocessing operations are adopted,
user to parse batch amount of images to xml files including noise removal and image balance.
and add these files to dataset of certain user. The Noise Removal (Ester et al. 1996). For each
batch module is designed mainly to train large pixel, we calculate and accumulate the difference
amount of images in order to set up the default of all its neighbor points as the weight of this
dataset which belongs to the default user (Fig. 7). pixel:
User emotion storage allows user to capture,
extract, and store emotions to numeric values one K 2 jf ðpÞ  f ðxÞj
by one using a web camera. To use this system, the weightðxÞ ¼ K 1 e pP

user initiates the training and capture of emotion

state facial expressions by instantiating “user where P is the neighbor point set of pixel x, and f
emotion storage.” The system prompts for the (x) is the pixel value of x.
700 Facial Recognition and Emotion Detection

Facial Recognition
and Emotion Detection
in Environmental
Installation and Social
Media Applications,
Fig. 7 The interface
windows for operation of
MoodRing MFC

Facial Recognition and Emotion Detection in a periodic manner to a text file that is readable by other
in Environmental Installation and Social Media software, such as Max/MSP
Applications, Fig. 8 Detected emotion states are stored

Then, we traverse the image again with a Image Balance. We have noticed that vague
weighted average filter for each pixel. shadow will not heavily affect Haar classifier per-
formance, and hard shadow edges can heavily
weightðpÞ  f ðpÞ weaken performance. An image balance method
pP is adopted such that useful information like edges
gðxÞ ¼ a þ ð1  aÞ  f ðxÞ
weightðpÞ is strengthened while noise like shadows is weak-
pP ened. Thus, instead of complex shadow removal
Facial Recognition and Emotion Detection 701

algorithm, we adopt following operations to con- Gabor Bank

centrate effective image information so that Haar Gabor filters are implemented to derive orienta-
classifiers can find target more easily: tions of features in the captured image using pat-
tern analysis, directionality distribution of the
f ðxÞ features. Using Gabor filters increases accuracy
aK logðxÞ þ ð1  aÞx , if x < 127 of the anchor points derived in the elastic bunch
¼ graph matching. Gabor filters of all scales and
a½255  K logðxÞ þ ð1  aÞx , if x  127 orientations compose the Gabor Bank to detect
edges and textures. In the Gabor filters:

Face Detection k2 k ðx2sþy

2 Þ 2 2

After above operations, we adopt a set of pre- gðx, yÞ ¼ 2

e 2
s F
trained Haar classifiers (Wang et al. 2007) to locate x
only one pair of eyes and mouth (Alpers 2011; ik
s2
 e y  e2 , where k
Messom and Barczak 2009). For facial detection,
only eyes and mouth are used in this case. If
multiple rectangles are found for the same part, vþ2
we run a clustering method to estimate the target ¼ ½ kv cos ’kv sin ’, kv ¼ 2 2 p
rectangle based on the Euclidean distance. We run
a clustering algorithm based on Euclidean distance where s is the standard deviation of Gaussian
between locations of each possible candidate. envelope, ’ is the direction of Gabor stripes, and
For example, to find one’s left eye in all possi- n determines the wavelength of Gabor filters.
ble locations, say A, B, C, and D: We choose 18 Gabor filters with six directions
and three phases to compose a Gabor Bank.
1. First, these possible eyes are clustered based on Directions include:
their locations and distance to each other.
(A, B, and C are on the left side, while D is 1 1 1 2 5
0, p, p, p, p, p;
on the right side; A is near to B, while C is far 6 3 2 3 6
from them. Then, we have {A, B}, {C}, and
{D}.) phases include:
2. Then, clusters are selected based on their size
p p
and location with another eye. (If {A, B} are at Cp, C 2p, C 3p ;
similar vertical location as {D}, and symmetric
to centerline of face about {D}, we choose {A,
where C is a constant. Such a Gabor Bank will be
B} as left eye and {D} as right eye.)
initialized when the program starts, and used
3. Finally, location and size are calculated using
every time extracting features.
{A, B} and {D}.

Then, a set of anchor points can be delivered Elastic Bunch Graph

based on these rectangles. Before feature extrac- Operations of elastic bunching graph include
tion, lumen normalization is adopted to detected graph matching, graph pruning, and adding sub-
facial part of the image such that light conditions graphs from either an image or an xml file. Elastic
have less effect to the feature extraction process. bunching graphs apply convolutions of certain
areas of images using all filters in the Gabor
Feature Extraction Bank. This results in a list of anchor information
Numeric features are extracted through convolu- for all anchor points, where each anchor informa-
tions with a set of precalculated Gabor filters tion contains a list of all convolution results
called Gabor Bank. corresponding to filters in the Gabor Bank. If the
702 Facial Recognition and Emotion Detection

program is in a training mode, it will store the the default graph set, which only contains graph
hierarchical results as an xml file. Otherwise, of the default user. As mentioned above, dataset
emotion detection is followed after feature for the default user is usually trained in the Batch
extraction. Emotion Storage module. Since default user’s
Graph pruning is the core function of Database dataset contains large amount of samples from
Compressor. The pruning algorithm is basically a existing database like “Feret” (http://www.nist.
variety of DBSCAN (Ester et al. 1996), where the gov/humanid/colorferet, it can be used without
distance of subgraphs defined as sum of Euclidean user graph set. However, user graph set is still a
distance of all convolution results for all anchor better choice because it contains fewer but more
points. If one cluster contains at least the mini- informative graphs. Based on given user ID, the
mum number of neighbor point subgraphs, and program will load graphs of that user into user
distances of these subgraphs are at most eps, we graph set if program can find user emotion data of
combine all subgraphs in one cluster into one. this user. Otherwise, only the default graph set
Thus, very similar subgraphs are merged to reduce will be loaded.
storage space and comparing time. Emotion
Detection. Weight Trainer
The emotion detection is a similarity compar- Weight Trainer is the first step to set up the
ing process. Target graph is compared with all MoodRing system. Input of this module is a set
subgraphs in all seven emotions (Database of elastic bunching graphs with all seven emo-
FERET) in the FERET dataset. We categorize tions; output is a weight matrix stored as local file.
the target graph for the same emotion type as its Given a set of seven graphs, Gi (i ¼ 0, 1, 2, 3,
most similar subgraph. In comparison of two 4, 5, or 6), and each graph Gi has subgraphs gij, we
graphs, we can calculate a weighted average on first generate the dataset through a pairwise
the distance of all such convolution results of all comparison:
anchors in graphs. When the program is initial-
ized, a mathematical model determined by Weight x ¼ gij  gmn , y
Trainer is loaded, such that the weight of each
0, if i ¼ m ðsame emotionÞ
anchor can be used to measure graph similarity ¼
(Fig. 9). 1, if i 6¼ m ðdifferent emotionÞ
There are two types of preloaded dataset used
in the detection process: default graph set and user Then, because y is between 0 and 1, we apply the
graph set. When initialized, the program will load following logistic function on X:

Facial Recognition and Emotion Detection in Environmental Installation and Social Media Applications,
Fig. 9 Training if the EmotionDetector
Fall Risk Detection in Computer Vision 703

Input matrix : gðXÞ, where x  X, and gðxÞ References

1
¼ Alpers, G.W.: Happy mouth and sad eyes: scanning facial
1 þ et expressions. Am. Psychol. Assoc. Emot. (4):860–865
(2011). doi:10.1037/a0022758
Output matrix : Y, where y  Y; Bolme, D.: Elastic Bunch Mapping. cs.colostate.edu/
vision/publications/Bolme2003.pdf (2003)
Bradski, G., Kaehler, A.: Learning OpenCV. OReilly,
Now that we have transferred the dataset into this
Sebastopol (2008)
form, we adopt certain classification methods, like Burges, C.J.C.: A tutorial on support vector machines for
LibSVM (Bolme 2003) to train the weight matrix. pattern recognition. Data Min. Knowl. Disc. 2,
If size of X is small (e.g., for individual users), we 121–167 (1998)
Chang, C.C., Lin C.J.: https://www.csie.ntu.edu.tw/cjlin/
will use batch training; if size of X is large (e.g.,
libsvm/
for the default user), we will use mini-batch sto- Database FERET http://www.nist.gov/humanid/color F
chastic training instead. The boundary of these Feret FA|FB|QR|QL|HL|HR 2. Rate of accuracy FB
algorithms is a constant value. Boundary p00 in (dvd2): 246/268 ¼ 91.791%
Ekman, P.: Basic emotions. In: Dalgleish, T., Power,
this case means:
M. (eds.) Handbook of Cognition and Emotion.
For given testing sample gij and dataset sample gmn, Wiley, Sussex (1999)
we can calculate the estimated ŷ value using the Ester, M., Kriegel, H., Sander, J., Xu, X.: A density-based
trained model. If ŷ  p, we can conclude gij and algorithm for discovering clusters in large spatial data-
gmn are the same emotion type; otherwise, they are bases with noise. In: Proceedings of the 2nd Interna-
different. tional Conference on Knowledge Discovery and Data
Mining (KDD, 96), pp. 226–231 (1996)
Hlang, H.K.T.: Robust algorithm for face detection in color
images. Int. J. Mod. Educ. Comput. Sci. 2, 31–37 (2012)
Messom, C., Barczak, A.: Fast and efficient rotated Haar-
Future Work like features using rotated integral images. Int. J. Intell.
Syst. Technol. Appl. 7(1), 40–57 (2009)
• Application of the detection is feasible in Viola, P., Jones, M.: Robust real-time object detection.
installation and environments, and public Paper presented at the Second International Workshop
on Theories of Visual Modelling Learning, Computing,
spaces.
and Sampling (2001)
• Further experimentation is necessary to deter- Wang, M., Xuguang, Z., Guangliang, H., Yanjie, W.: Elim-
mine accuracy of the facial capture and emo- ination of impulse noise by auto-adapted weight filter.
tion detection. Opt. Precis. Eng. 15(5), 779–783 (2007)
Wiskott, L., Fellous, J.-M., Kuiger, N., von der Malsburg,
• Further work will include continued refine-
C.: Face recognition by elastic bunch graph matching.
ment of the image processing and normaliza- Pattern Anal. Mach. Intell. IEEE Trans. Mach. Intell.
tion to operate in varying lighting conditions. 19(7), 775–779 (1997)
• Mobile app and social media application
exploration will continue.
• Experimentation and comparison of image
library-based implementations and EBGM for Fall Risk Detection in
the further development of a “universal” Computer Vision
detector.
Yen-Hung Liu and Patrick C. K. Hung
Faculty of Business and IT, Ontario Tech
Cross-References University, Oshawa, ON, Canada

▶ Cognitive Psychology Applied to User Experi-

ence in Video Games Synonyms
▶ Emotion-based 3D CG Character Behaviors
▶ Modeling and Mesh Processing for Games Anomaly detection; Computer vision; Fall risks;
▶ Vector Graphics Machine learning; Pose estimation
704 Fall Risk Detection in Computer Vision

Definition implement advanced technologies to monitor an

individual’s balance and movement.
Anomaly Detection
Anomaly detection identifies unexpected items or
events in data sets from the norm. Related Works

Fall Risks Previous studies on fall detection can be classified

Fall risks are common threats affecting all indi- into two main groups: vision-based and sensor-
viduals, including the elderly or young based methods.
children. Sensor-based methods can be further divided
into wearable-based and ambient fusion-based
Computer Vision methods. Wearable-based methods mostly
Computer vision is an Artificial Intelligence employ accelerometers and gyroscopes, and
(AI) system incorporated with other scientific combining multiple sensors can improve their
fields, such as signal processing and neurobiol- accuracy (Wu et al. 2012). In contrast, ambient
ogy, to interpret and gain a high-level understand- fusion-based methods require various sensors,
ing of digital images or videos. such as vibration, acoustic, pressure, infrared,
Doppler, and a near-electric field, to detect falls.
Machine Learning To identify fall occurrences, Werner et al. (2011)
Machine learning employs software tools from studied the vibration generated by a fall event
advanced analytics that use statistical algorithms from an Activity of Daily Living (ADL) event.
to find patterns in datasets. Machine learning However, this approach’s high false alarm rate is a
algorithms have two categories: (1) Supervised significant drawback.
learning: The data is labeled and trained On the other hand, vision-based methods can
according to various classes, such as malicious be categorized into two types: Red, Green, and
and legitimate, to form mathematical models for Blue (RGB) images and depth images. Kinect is a
the algorithms. Next, (2) unsupervised learning: commonly used equipment for extracting depth
The data is not labeled or trained. However, the information in images; where used it to obtain a
algorithms determine the degree of data coher- human bounding box, and used it to extract a
ence to create classes according to the quality of human skeleton. Machine learning techniques
data coherence within the classes and data mod- such as Support Vector Machines (SVMs) were
ularity between them. applied to identify fall events based on these fea-
tures. However, Kinect has limitations when it
Pose Estimation comes to sunlight and outdoor environments.
Pose estimation is a computer vision technique Convolutional Neural Network (CNN) has
used for identifying human postures. emerged as a game changer for RGB images. Lu
and Chu (2018) used an object detection model
(YOLO V3) that considers the relationship with
Motivation and Background the surrounding object to design their fall detec-
tion system. However, the performance of this
Fall risk is a common occurrence threatening approach heavily relies on camera angles, leading
individuals of all ages, from young to older adults. to the development of the pose estimation
Typically, older adults are at a higher risk of approach. The pose estimation method,
falling. Poor eyesight, poor balance, and medica- OpenPose, can identify human skeletons and pos-
tion that causes drowsiness can cause severe harm tures, leading to better identification of human
and danger. Researchers continuously investigate motions used OpenPose to identify unstable cen-
evidence-based strategies to prevent falls, exam- ters of gravity during a fall event to detect fall
ine contributing factors like human behavior, and events more accurately.
Fantasy XVI Online, a Massively Multiplayer Online Role-Playing Game 705

Structure of the Learning System

Fantasy XVI Online, a
In general, older adults are at high risk of this Massively Multiplayer Online
injury due to changes in their body frame, physi- Role-Playing Game
cal and mental health, and cognitive alterations. In
addition, fall risk depends on the individual, their Nicholas Ries2 and Newton Lee1,2
1
physical environment, and their lifestyle. How- Institute for Education, Research, and
ever, children and other individuals can also be Scholarships, Los Angeles, CA, USA
2
prime victims. Falls are severe, but computer Vincennes University, Vincennes, IN, USA
vision technology can detect them. In addition,
fear of falling is an issue that individuals are
prone to feel after their fall incident (Araya and Synonyms F
Iriarte 2021). Many studies show different assess-
ment methods and evidence-based approaches to MMORPG; Multiplayer game; Action adventure
deliver accurate data regarding fall risk causes and game
prevention methods.

Definition
Cross-References
MMORPG ¼ Massive Multiplayer Online Role-
▶ Locomotion and Human Tracking in Playing Game is a multiplayer game designed to
Healthcare Robots be played online
Multiplayer Game ¼ a game that is designed
for multiplayer mode where two or more players
References are expected throughout the entire gameplay
Action Adventure Game ¼ a game that com-
Araya, A.X., Iriarte, E.: Fear of falling among community- bines core elements from both action game and
dwelling sedentary and active older people. Invest.
Educ. Enferm. 39(1), e13 (2021)
adventure game genres
De Miguel, K., Brunete, A., Hernando, M., Gambao, E.:
Home camera based fall detection system for the
elderly. Sensors. 17(12), 2864 (2017) Introduction
Lu, K.L., Chu, E.T.H.: An image-based fall detection
system for the elderly. Appl. Sci. 8(10), 1995
(2018) Final Fantasy 14 Online is an MMORPG devel-
Werner, F., Diermaier, J., Schmid, S., Panek, P.: Fall detec- oped by Square-Enix and published for the PC
tion with distributed floor-mounted accelerometers: An and PlayStation markets. It is popular to the point
overview of the development and evaluation of a fall
detection system within the project eHome. In: The
that they had to lock down the free-trial and
2011 5th International Conference on Pervasive Com- stopped selling digital copies for a few months
puting Technologies for Healthcare (PervasiveHealth), in order to keep the player countdown as to not
pp. 354–361 (2011) overload the servers with the expected launch of
Wu, W., Dasgupta, S., Ramirez, E.E., Peterson, C., Nor-
“Endwalker,” the latest expansion to come down
man, G.J.: Classification accuracies of physical activi-
ties using smartphone motion sensors. J. Med. Internet the pipeline.
Res. 14(5), e130 (2012)

Gameplay

Fall Risks Players create a Warrior of Light to play through

the game. With a focus on role-playing, each
▶ Fall Risk Detection in Computer Vision player takes the helm of one of the major Scions
706 Fantasy XVI Online, a Massively Multiplayer Online Role-Playing Game

or outside characters like Gaius. Featuring a vari- generating attacks and a “tank-stance” that
ety of classes and various activities, there are increases the amount gained per strike, along
many ways to approach the game including with damage mitigation and a few debuffs that
smithing, adventuring, buying houses, and can be applied to enemies. Each of the tanks has
raiding. a different playstyle: Paladin and Warrior are the
With 19 “combat” classes split between four safest whereas Dark Knight and Gunbreaker are
tanks, four healers, and 11 “DPS,” there is a wide the risky options. Gunbreaker is a favorite for
variety of playstyles to choose from. From close- some gamers due to the high DPS combos along
range physical DPS to long-range magical, with the extremely precise barrier abilities.
healing, or tanking, you can truly be the “Warrior Figure 1 is an image of a game character as a
of Light” that you have always dreamed about in a Gunbreaker.
huge fantasy game. Along with this, there are also DPS: Monk, Dragoon, Ninja, Samurai, Reaper,
11 classes dedicated to crafting and gathering, Bard, Machinist, Dancer, Black Mage, Sum-
allowing you to make your own gears and collect moner, Red Mage, and Blue Mage
your own materials. Not only can you run dun- Physical DPS, Ranged DPS, and Magical DPS
geons or take on fantastic bosses, you can spend classes play quite differently from one another,
hundreds of hours crafting your own gears and and they often have wildly different abilities. For
playing the in-game markets to make “Gil” and example, Summoner and Red Mage both possess
buy wondrous prizes. the ability to resurrect other players, which often
Tanks: Paladin, Warrior, Dark Knight, and comes in handy during raids and major fights.
Gunbreaker (tvtropes.org 2022) Ninja is famous for its high burst potential, but
The primary style of play for tanks includes many players prefer more consistent DPS output
gathering “aggro” through the use of enmity- from classes like Samurai or Reaper. The king of

Fantasy XVI Online, a Massively Multiplayer Online Role-Playing Game, Fig. 1 A game character as a Gunbreaker
wielding the newest Relic weapon
Fantasy XVI Online, a Massively Multiplayer Online Role-Playing Game 707

all of these is Black Mage, having been the Reception

undisputed lead of all DPS since the game’s incep-
tion at a cost of limited mobility. Another favorite Originally facing lukewarm reception, each expan-
DPS is Reaper due to its high damage output and sion of the game has received more rave reviews,
maneuverability, allowing it to zip around the culminating in being named one of the best MMOs
battlefield and top the aggro chart with ease (see of the generation. Many thought the hype sur-
Fig. 2). rounding the game had died with the culmination
Healers: White Mage, Scholar, Astrologian, of the ShadowBringers storyline, with no one
and Sage believing the following expansion would be able
Healers in this game are divided into two to top it. But the Endwalker expansion was not only
categories by the community: “Regen” Healers considered a worthy addition, but it is now also
such as White mage and Astrologian, and being upheld as the bar of excellence in the MMO F
“Shield” Healers like Sage and Scholar. Interest- world. The game has achieved more credibility
ingly, the Astrologian straddles a line between among even casual gamers, bringing another
the two categories given its abilities. While wave of new participants to the world of Eorzea.
Regen healers focus on getting everyone to max It was certainly helped by a generous free-trial offer
health quickly with strong heals, Shield healers’ that began around the initial outbreak of COVID
main mechanic is to place shields on teammates pandemic. The free trial allowed users to play the
in order to prevent major damage from taking original “A Realm Reborn” storyline as well as the
place. One of the favorite healers is Sage due to hugely popular “Heavensward” expansion with
the strong shield and its flexible cooldowns (see two additional races and three extra jobs (Bussel
Fig. 3). 2022).

Fantasy XVI Online, a Massively Multiplayer Online Role-Playing Game, Fig. 2 A Reaper holding one of the new
Major Relic Weapons
708 Fantasy XVI Online, a Massively Multiplayer Online Role-Playing Game

Fantasy XVI Online, a Massively Multiplayer Online Role-Playing Game, Fig. 3 A favorite Sage healer, not
wielding a Major Relic Weapon

Tournaments trial and temporarily restrict people from purchasing

the game in order to reduce the heavy load on their
The game features a robust raiding community servers (Teixeira 2021). It has a large community
dedicated to “savages” and “ultimate” content that that is generally positive and welcoming to out-
exist solely to challenge themselves to “races,” that siders. The community even offered a warm wel-
is, to be the first player to complete the new content come to gamers who fled World of Warcraft when
as soon as it drops (mogtalk.org 2022). In addition, the controversies surrounding Blizzard came out.
there was an official tournament in July 2022 for With continuously growing numbers, it cannot be
the game focused on the Crystalline Conflict mode denied that Final Fantasy XVI Online has become
that was region locked to Europe (Png 2022). In one of the biggest Massively Multiplayer Online
fact, one of the major game developers known as Games in the world.
“Yoshi-P” can be regularly found participating in
the PVP modes and fighting in Crystalline Conflict
in order to personally test the competitive modes
Cross-References
for the game and further improve it.
▶ Disney Toontown Online, A Massively Multi-
player Online Role-Playing Game
Conclusion

Final Fantasy XVI Online (FFXVI or FF-14) has

References
become one of the best-selling MMOs, even sur-
passing internal sales metrics. The game experi- Bussel, C.: Final Fantasy 14 free trial: how much can you
enced a huge boom of players during COVID-19, play for free? Retrieved from techradar: https://www.
to the point that they had to stop offering the free techradar.com/how-to/final-fantasy-14-free-trial#:~:
Fingerprint Verification Based on Combining Minutiae Extraction and Statistical Features 709

text¼As%20a%20free%20trial%20player,in%20the%
20in%2Dgame%20chat (21 July 2022) Fingerprint Verification Based
mogtalk.org. mogtalk. Retrieved from mogtalk: https://
mogtalk.org/the-ffxiv-world-race/ (2022) on Combining Minutiae
Png, K.: gamestart.asia. Retrieved from https://news. Extraction and Statistical
gamestart.asia/first-ever-official-final-fantasy-xiv- Features
esports-tournament-to-commence-july-2022/ (23 June
2022)
Teixeira, M.: nme.com. Retrieved from nme.com: https:// Anwar Yahya Ebrahim1 and Hoshang Kolivand2
1
www.nme.com/news/gaming-news/final-fantasy-xiv- University of Babylon, Babylon, Iraq
2
digital-copies-sold-out-after-surge-in-popularity- Department of Computer Science, Faculty of
2991295 (12 July 2021) Engineering and Technology, Liverpool John
tvtropes.org. tvtropes.org. Retrieved from tvtropes.org:
https://tvtropes.org/pmwiki/pmwiki.php/Characters/ Moores University (LJMU), Liverpool, UK
FinalFantasyXIVTankClasses (2022) F
Synonyms

Fingerprint image; Fingerprint ridge thinning;

Fighting Game Fingerprint verification; Minutiae extraction; Sin-
gular point; Statistical features
▶ King of Fighters, a Brief History

Definition

One of the most popular forms of biometrics used

File Format
for personal identification is fingerprints. The
extraction of multi-features illustrates the diver-
▶ Color: Pixels, Bits, and Bytes
sity of data represented from fingerprint samples,
allowing mitigation of the intrapersonal variable.
This study uses multiple feature extraction based
on statistical tests of co-occurrence matrices to
Fingerprint Image overcome the drawbacks of previous methods
and minutiae extraction to achieve high accuracy
▶ Fingerprint Verification Based on Combining toward an efficient fingerprint verification system.
Minutiae Extraction and Statistical Features

Introduction

The most important type of human biometrics is

Fingerprint Ridge Thinning fingerprints. Fingerprints have been used for per-
sonal recognition in forensic applications, such as
▶ Fingerprint Verification Based on Combining criminal investigation tools, national identity card
Minutiae Extraction and Statistical Features validation, and authentication processors. The
uniqueness and immutability of fingerprint pat-
terns as well as the low cost of associated biomet-
ric equipment make fingerprints more desirable
Fingerprint Verification than the other types of biometrics (Maltoni and
Cappelli 2009). A characteristic attribute of false
▶ Fingerprint Verification Based on Combining fingerprints in the large view of complexly refers
Minutiae Extraction and Statistical Features to forgery, combined with the actuality that
710 Fingerprint Verification Based on Combining Minutiae Extraction and Statistical Features

fingerprint samples are distinctive to each individ- Preprocessing Fingerprint Enhancement

ual. In reality, fingerprints present a distinguished Image improvement of fingerprint samples is very
basis of entropy that is used for security important for FV to function correctly. The effi-
implementations (Dodis et al. 2006; Dass and ciency of the image of the imprint sample is
Jain 2007; Khan 2009). There are two classes of influenced by noise in ridges produced by the
features utilized for the recognition system: global under-inked region, varying the fingerprint attri-
attributes and local attributes (Ebrahim 2017a). butes because of skin elasticity, where splits are
The global attribute pattern represents a distinc- from dry skin and wounds may cause ridge discon-
tive modality of a singular point as the center point tinuities. To preserve the high accuracy of the FV
that is utilized for fingerprint recognition. The system, two procedures utilized in the unique mark
local feature represented from fingerprint’s ridges recognition basis (STFT) analysis suggested by
data is called minutiae. The fingerprint verifica- Chikkerur et al. (2007) are utilized here for image
tion (FV) system includes three processes: the enhancement of fingerprints and procedures. The
fingerprint obtaining apparatus, extracted fea- system can be shortened as follows: The image of
tures, and verify features (Igaki et al. 1992). The the fingerprint is split into overlapping squares.
dataset of FVC2002 is utilized to obtain the finger Then STFT analysis is performed. The test pro-
imprint by an optical device, which has high capa- ductions are images of ridge orientation O(x,y)
bility (Maio and Maltoni 1997). The extracted and ridge hesitation F(x,y). The next stage O(x,y)
minutiae and minutiae verification are further image represents smoothing of the orientation. For
obtained in the next stage. improvement, each overlapping square B(x,y) in
A fingerprint authentication scheme is a the image has been rebuilt for the sample by creat-
model matching scheme that distinguishes the ing improved blocks B0(x,y).
individual based on their fingerprint attribute
(Maltoni et al. 2003). A number of various fin- Feature Extraction Methods
gerprint classification systems have been The fingerprint sample illustrates a scheme of
improved using myriad classification approaches oriented texture and contains important data.
and datasets. Each classification method has its This scheme utilizes an ensemble of dichotomizes
own specific characteristics that researchers cap- to combine features through several measures and
italize on to advance fingerprint classification feature extraction methods, commanding lower
research using a particular dataset (Jain et al. cost and accurate FV. The approach utilizes
1997; Jea and Govindaraju 2005). In order to multi-feature extraction at different measure tech-
design a more reliable automatic identification niques for classification of preexisting extracted
system, preprocessing of fingerprints has to take feature systems, creating a large set of features
place to enhance and extract fingerprint features (Ebrahim 2017b; Ebrahim and Sulong 2014).
(Wu et al. 2006; Rajkumar and Hemachandran The performance of fingerprint images is realized
2011). According to Maltoni et al. (2009), most by applying two methods based on singular point
current fingerprint classification approaches detection and fingerprint ridge thinning, as
depend on global attributes, including ridge ori- discussed in the next section.
entation areas and singularities. Bazen and Gerez
(2002) found that accurate classification of fin- Singular Point Detection Technique
gerprints is highly dependent on the orientation The image of a fingerprint is made up of the
fields’ estimation and singular points detection design of ridges and valleys that is a copy of the
algorithms. Later, Arivazhagan et al. (2007) individual imprints. The reference point is the
suggested a fingerprint verification system utiliz- point with extreme curvature on the convex
ing Gabor wavelets. Yazdi and Gheysari (2008) ridge, and to define the single reference point
utilized co-occurrence matrices to verify the fin- reliably for all varieties of fingerprint samples,
gerprint image. the orientation area, which characterizes the
Fingerprint Verification Based on Combining Minutiae Extraction and Statistical Features 711

The ridge orientation of each pixel (x,y) within

* Orientation Area
* Dividing the Sample
a W
W window at points [xi,yj] is calculated as
STAGE I into non-Overlap Blocks follows (Ratha et al. 1995):
* Compute the Gradient
* Smoothing and Noise
1 2Gxy
Removal
yðx:yÞ ¼ tan 1 : ð1Þ
2 GyyGyy

* Reliability Because of noise, an inclined ridge, valley

Compute the Orientation*
STAGE II Field Reliability structures, and poor gray value, a low-pass filter
can be applied to regulate the incorrect local
ridge orientation. However, to achieve the low-
pass filtering, the orientation image must be F
* Singular Point
Location transformed into a continuous vector area, and
* Segmentation
* Thinning
the Gaussian low-pass filter can be used as
STAGE III * Shrinking follows:
* Pixel removal
* Opening and Closing
1 1
F:y ðx:yÞ ¼ u¼1 v¼1 lðu:vÞ
ð2Þ

Fy ðx  ul:y  vl Þ

Fingerprint Verification Based on Combining Minu- where l is a 2-directional low-pass filter with a
tiae Extraction and Statistical Features, complete unit.
Fig. 1 Singular point block method diagram (Kaas and The orientation area can be representing in this
Witkin 1987) section.

local orientations along the prevalent orientation, Reliability

is utilized to distinguish the reference point. The Meanwhile the singular point has the maximum
reliability can also be calculated applying the curvature. It can be found by calculating the
method projected by (Ebrahim 2017a) and (Kaas strength of the reliability peak applying the fol-
and Witkin 1987). Figure 1 shows the block lowing equation:
design for detecting the singular point and the
implementation. Y min
Y ¼1 : ð3Þ
Y min
Orientation Area
The first stage of finding the singular point is Figure 2 displays the orientation area reli-
computing the orientation area. The orientation ability and the singular point value in the center.
field is critical for the computation of the reliabil- The reliability rate of the singular point is ¼ 0,
ity. The accuracy of the reliability robustly but the value of background is also zero. How-
depends on the accuracy of the orientation area ever, there is a contour around the singular point
values calculated. Below are the steps for orienta- of the reliability value of the contour
tion calculation: between < 0.5 and > 0, and this is the area of
The fingerprint sample is split into a non- interest.
overlapping block of size (W
W). In this
paper, W is set to 16. Singular Point Location
The horizontal and vertical gradients Gx(x, y) After calculating the orientation area reliability,
and Gy(x, y) at each pixel (x,y) respectively are the next stage is to detect the singular point
calculated applying simple gradient factors place. This can be achieved by the following
(Gonzalez and Woods 2008). stages:
712 Fingerprint Verification Based on Combining Minutiae Extraction and Statistical Features

Fingerprint Verification Based on Combining Minu- map to three singular points, (d) Region of interest after the
tiae Extraction and Statistical Features, Fig. 2 The segmentation, (e) Singular point contour after thinning, (f)
detecting of singular point by (a) Orientation field on Filled singular point contour, (g) Singular point location on
original fingerprint, (b) Both singular point and orientation the original fingerprint
field reliability within the contour, (c) Reliability image

(i) The orientation field reliability needs to be (v) Here, the singular point pixel can be found
segmented into two distinct regions. The region by performing the shrinking process for the
of interest contains the values > 0 and < 0.5. singular point contour. Figure 2 shows the
The result of the segmentation can be singular point pixel after applying the
seen in Fig. 2. shrinking method, the position of the singu-
(ii) Thinning is the process of adjusting the lar point on the original fingerprint, and Sin-
width of contents of the image to one pixel gular point pixel.
while preserving the extent and connectivity
of the original shape. Fingerprint Thinning Minutiae Marking
(iii) After thinning, all pixels will be removed so Thinning is a procedure by which the sizes of the
that the contour shrinks to a connected ring ridges are reduced. In each scan of the full imprint
halfway between each hole and the outer image of a fingerprint, every sample square rep-
boundary, and the rest will shrink to a pixel resents (16
16) pixels (Maio and Maltoni 1997).
that will be removed. Figure 2 shows the This image along with other data will be recorded
singular point contour without any noise. into the dataset (Jain et al. 1999). After obtaining
(iv) The singular point contour is well defined, this input file, it will undergo binarization. Usu-
and to determine the location of the singular ally, a vision of the distinctive mark will be
point, the contour is filled using the morpho- obtained.
logical hole filling equation (Gonzalez and
Woods 2008):
Normalization
1I ðx:yÞ
Normalization is a procedure for fingerprint veri-
F ðx:yÞ ¼ 0 , ð4Þ fication. Samples of fingerprints do not come in
the same dimensions. Because of this, the samples
where “(x, y)” represents the border of I; need to be aligned suitably to confirm an overlap
otherwise, H is equal to the input image I, of the common area in the two samples of imprint
with holes filled. by the orientation of the image to zero at the
Fingerprint Verification Based on Combining Minutiae Extraction and Statistical Features 713

reference point (Gonzalez and Woods 2008). Fingerprint Verification

Once all the features of each fragment have been
extracted, then the feature values are normalized Matching is an important part of fingerprint veri-
in the interval [0 1]. Normalization of features is fication. It compares two features and returns a
very important because if the values of different likeness score to indicate how comparable the two
features are in different ranges then the higher participating fingerprints are. The comparison
values dominate the lower values. Thus, the nor- depends on discovery of the Euclidean distance
malization technique makes the feature values in between the attribute of the conforming
the same scales and ranges. The image of the fingerprints.
fingerprint is split into a non-overlapping set of The EER for experiment was computed by the
size W
W (for each 16
16) and the orienta- FVC2002. In Table 1, the databases were split into
tion that matches the most controlling orientation four databases: DB1, DB2, DB3, and DB4. Each F
of the block (Ebrahim et al. 2018; Ebrahim and dataset contains 800 fingerprint samples collected
Ashoor 2018; Ebrahim 2018). After the process, from 100 persons, and each one is eight impres-
the features of ridges in the fingerprint are sions. Sets of tests were carried out for each data-
represented in black and furrows are represented base, and the protocol is shown in Table 1. Yang
in white. et al. (Yang and Park 2008) used the tessellated
invariant moment feature, Ross et al. (2003) used
Multi-feature Extraction minutiae and ridge map features, Jin et al. (2004)
There is a pressing need to develop a finger- used integrated wavelet and Fourier–Mellin
print verification system with software-based invariant framework with four multiple training
minutiae extraction and statistical features. WFMT features, Amornraksa et al. (Amornraksa
The proposed approach is to extract the most and Tachaphetpiboon 2006) used the DCT fea-
suitable multi-features for classification. In ture, Khalil et al. (2010) used statistical descrip-
this study, we combine a set of the strongest tors, and Flores et al. (2017) used Delaunay
features. The effectiveness of the projected triangulations.
method has been evaluated through a compar-
ison with several existing techniques for
multi-features. Conclusions
Tests have been executed depending on stan-
dard datasets. However, more accuracy lowers Low-accuracy fingerprint images require multi-
the popularization capabilities. In fact, the back- features to expand distinction. The process for
ground minutiae extraction and statistical fea-
tures have already managed to detect the Fingerprint Verification Based on Combining Minu-
tiae Extraction and Statistical Features, Table 1 EER
differences in fingerprints, and statistical attri- (%) evaluation of the current techniques
butes are utilized by other FV systems, for
Technique DB1 DB2 DB3 DB4 EER
example (Yazdi and Gheysari 2008). In this
Yang et al. 1.63 3.78 4.20 4.68 3.57
research, multi-features are extracted from dif- (2008)
ferent parts of the fingerprint, therefore increas- Ross et al. 1.87 3.98 4.64 6.21 4.17
ing its selective power. There are also many (2003)
outliers in each class, which are circled. The Jin et al. 2.43 4.41 5.18 6.62 4.66
outlier is an observation that carries an irregular (2004)
distance from its neighboring feature values that Amornraksa 2.9 5.4 6.7 7.5 5.68
et al. (2006)
will result in misclassifications. Hence, it is
Khalil et al. 0.31 0.26 0.34 0.20 0.28
observed that minutiae extraction or statistical (2010)
features alone inherit some weaknesses, such as Flores et al. 0.125 0.125 0.75 0.75 0.44
overlapping and outliers, which reduce their dis- (2017)
criminating abilities.
714 Fingerprint Verification Based on Combining Minutiae Extraction and Statistical Features

FV comprises five sections: singular point detec- Ebrahim, A.Y., Kolivand, H., Rehman, A., Rahim, M.S.
tions, fingerprint ridge thinning, normalization, M., Saba, T.: Features selection for offline handwritten
signature verification: state of the art. Int J Comput
multi-feature extraction, and fingerprint verifica- Vision Robot. 8(6), 606–622 (2018)
tion. Moreover, minutiae extraction and the four Flores, G.M., Torres, G., Garcia, M.L.: Fingerprint verifi-
statistical descriptors characterize the fingerprint cation methods using delaunay triangulations. Int. Arab
texture. Each process plays an important role in J. Inf. Technol. 14(3), 346–354 (2017)
Gonzalez, R.C., Woods, R.: Digital Image Processing.
fingerprint verification. Hence, to facilitate the Prentice Hall, Upper Saddle River (2008)
accomplishment of such ambitious goals in the Igaki, S., Shinzaki, T., Yamagishi, F., Ikeda, H. Yahagi, H.:
near future, researchers ought to perform statisti- Minutia Extraction in Fingerprint Identification. US
cal tests and minutiae extraction of fingerprint Patent No. US5109428 A (1992)
Jain, A., Hong, L., Bolle, R.: On-line fingerprint verifica-
textures. tion. Pattern Anal. Mach. Intell. 19, 302–313 (1997)
Jain, L.C., Halici, U., Hayashi, I., Lee, S.B., Tsutsui, S.:
Intelligent Biometric Techniques in Fingerprint and
References Face Recognition. The CRC Press, Boca Raton (1999)
Jea, T.-Y., Govindaraju, V.: A minutia-based partial finger-
print. Recognition system. Pattern Recogn. 38,
Amornraksa, T., Tachaphetpiboon, S.: Fingerprint recog- 1672–1684 (2005)
nition using DCT features. Electron. Lett. 42(9), 522– Jin, A.T.B., Ling, D.N.C., Song, O.T.: An efficient finger-
523 (2006) print verification system using integrated wavelet and
Arivazhagan, S., Arul Flora, T.G., Ganesan, L.: Fingerprint ourierMellin invariant transform. Image Vis. Comput.
verification using Gabor co-occurrence features. Int. 22, 503–513 (2004)
Conf. Intell. Multimedia. Appl. Proceedings of Kaas, M., Witkin, A.: Analyzing oriented patterns. Comp:
ICCIMA, 2, 281–285 (2007) Vision Graphics Image Process. 37, 362–385 (1987)
Bazen, A.M., Gerez, S.H.: Systematic methods for the Khalil, M.S., Mohamad, D., Khan, M.K., Al-Nuzaili, Q.:
computation of the directional fields and singular points Fingerprint verification using statistical descriptors. In:
of fingerprints. IEEE Trans. Pattern Anal. Mach. Intell. Journal Digital Signal Processing, Elsevier, Academic
24(7), 905–919 (2002) Press, Inc. Orlando, FL, USA 20(4), 1264–1273 (2010)
Chikkerur, S., Cartwright, A.N., Govindaraju, V.: Finger- Khan, M.K.: Fingerprint biometric-based self-
print Enhancement using STFT Analysis. Pattern authentication and deniable. Authentication schemes
Recog. 40, 198–211 (2007) for the electronic world. IETE Tech. Rev. 26,
Dass, S.C., Jain, A.K.: Fingerprint-based recognition. 191–195 (2009)
Technometrics. 49, 262–276 (2007) Maio, D., Maltoni, D.: Direct gray-scale minutiae detection
Dodis, Y., Osrovsky, R., Reyzin, L., Smith, A.: Fuzzy in finger- prints. IEEE Trans. Pattern Anal. Mach.
extractors: how to generate strong keys from biometrics Intell. 19, 27–40 (1997). https://doi.org/10.1109/34.
and other noisy data. LNCS. 3027, Springer, 523–540 566808
(2006) Maltoni, D., Cappelli, R.: Advances in fingerprint model-
Ebrahim, A.Y.: Classification of Arabic autograph as gen- ing. Image Vis. Comput. 27(3), 258–268 (2009)
uine and forged through a combination of new attribute Maltoni, D., Maio, D., Jain, A.K., Prabhaka, S.: Handbook
extraction techniques. J Univ Babylon. 25(5), of Fingerprint. Recognition., Springer-Verlag New
1873–1885 (2017a) York, Inc., Secaucus, NJ (2003)
Ebrahim, A.Y.: Detection of breast cancer in mammograms Rajkumar, R., Hemachandran, K.: A review on image
through a new features and decision tree based, classi- enhancement of fingerprint using directional filters.
fication framework. J. Theor. Appl. Inf. Technol. Assam Univ. J. Sci. Technol. 7(2), 52–57 (2011)
95(12), 6256–6267, ISSN: 1992-8645 (2017b) Ross, A., Jain, A., Reisman, J.: A hybrid fingerprint
Ebrahim, A.Y.: A new model of Arabic handwritten rec- matcher. Pattern Recogn. 36, 1661–1673 (2003)
ognition using combination between DWT with data Wu, C., Tulyakov, S., Govindaraju, V.: Image Quality
reduction method. J. Theor. Appl. Inf. Technol. 96, Measures for Fingerprint Image Enhancement. Multi-
6376–6387. (1992–8645)- (2018) media Content Representation, Classification and
Ebrahim, A.Y., Ashoor, A.S.: Tumor classification using Security, pp. 215–222. Springer, Berlin (2006)
enhanced hybrid classification methods and segmenta- Yang, J.C., Park, D.S.: A fingerprint verification algorithm
tion of MR brain images. ARPN J. Eng. Appl. Sci. using tessellated invariant moment features.
3(20), 8270–8278 (2018) Neurocomputing. 71, 1939–1946 (2008)
Ebrahim, A.Y., Sulong, G.: Offline handwritten signature Yazdi, M., Gheysari, K.: A new approach for the finger-
verification using back propagation artificial neural print classification based on gray-level co-occurrence
network matching technique. JATIT LLS. 65(3), matrix. World Acad. Sci. Eng. Technol. 47, 313–316
790–800 (2014) (2008)
Fire Emblem Fates, Conquest 715

previous campaigns that may have not previ-

Fire Emblem 14 ously (Yamoto n.d.).
Unlike many strategy games of the time, Fire
▶ Fire Emblem Fates, Conquest Emblem: Shadow Dragon and the Blade of Light
differentiated units by giving them unique dia-
logue and personalities (Harris 2009). While the
text was very barebones in Shadow Dragon and
the Blade of Light, further entries in the series
Fire Emblem Fates, Conquest added a support system where units can talk to
each other. After a certain point, units that support
Sam Romershausen2 and Newton Lee1,2 each other will assist each other in combat by
1
Institute for Education, Research, and increasing stats in combat. F
Scholarships, Los Angeles, CA, USA
2
Vincennes University, Vincennes, IN, USA
Fire Emblem Gameplay

Synonyms Fire Emblem gameplay consists of moving units

across a grid-based map to complete objectives,
Fire Emblem 14; Fire Emblem if engage in combat, and manage character stats.
Map objectives usually include seizing castles,
defeating bosses, or defeating all enemies. Units
Definitions in the player’s army are differentiated by starting
stats, the rate that stats grow (often called growth
Fire Emblem Fates released on June 25, 2015 in rates), how well they can wield a weapon, and
Japan and in 2016 in other regions for the class. A unit’s class determines their movement
Nintendo 3DS system. Fire Emblem is a tactical type, growth rate modifiers, and weapons they can
role-playing series that debuted on the Famicom wield. The four movement types are infantry,
with Fire Emblem: Shadow Dragon and the Blade armored, cavalry, and flying. Infantry units can
of Light in 1990. Fates has three routes to play: be thought of as the basic movement level.
Birthright, Conquest, and Revelations. This arti- Armor units have lower movement but have
cle will explain the background of Fates and what more defensive stats. Cavalry units can move
makes Conquest unique. more spaces but are slowed down by terrain,
such as forests and hills. Flying units can move
across any terrain with ease but are weak to arrows
Introduction among other things. After a class has collected
enough experience by performing actions, they
Fire Emblem Fates is the fourteenth installment can be promoted to a higher-ranking class, giving
in the Fire Emblem franchise. Fates is split up them more stats and weapons to wield. Common
into three different routes that have different unit stats that are in many Fire Emblem games are
game designs but share the same engine and Health Points (HP), Attack, Defense, Magic,
mechanics. Birthright is designed to be similar Resistance, Speed, Skill, and Luck. Once a unit’s
to Fire Emblem: Awakening, the previous entry HP reaches zero, they die. Attack is how much
in the franchise. Conquest is designed for series damage a unit can do in combat. Defense is used
veterans and those who seek a challenge. The to reduce damage from an attack. Magic is used to
Revelations route was added after launch with calculate damage with tomes and other magical
sandbox elements to allow players to have char- weapons. Resistance reduces damage from magi-
acters interact with other characters from the cal attacks. Speed affects a unit’s dodge rate and
716 Fire Emblem Fates, Conquest

gives units the ability to perform a follow-up

attack (often called doubling) if they reach a cer-
tain threshold of speed difference with an oppo-
nent. Skill is used to calculate how accurate
attacks are and can be used to activate skills.
Luck is used to manage the rate of critical hits,
evasion, hit, and skill activation. A common Fire Emblem Fates, Conquest, Fig. 1 The Fates
mechanic that exists in many Fire Emblem weapon triangle
games is the Weapon Triangle, where sword
units have an advantage against axe units, axe rods can reach allies that are farther away. In
units have an advantage against lance units, and addition, the weapon triangle has been updated
lance units have an advantage against sword units. to include all weapons, besides stones and mon-
Other typical weapons are magic tomes, staves, ster classes (Fig. 1).
and stones that allow their units to transform in In Awakening, a mechanic called Pair Up allo-
battle. In most Fire Emblem games, weapons have wed two units to occupy the same square with one
durability that the player needs to keep in mind as leading and the other standing behind. The unit in
to not be left defenseless. In most Fire Emblem the front will gain stats according to their partner’s
games, when a unit falls in combat they are con- stats, class, and support level with the lead unit.
sidered dead and will not participate in the story. They may also occasionally enter combat to per-
form extra hits or block enemy hits. This system
was criticized by series veterans for being too
Fire Emblem Fates Gameplay overpowered and unstrategic as the bonuses
were unreliable and unviewable. Enemies in
During the development of Fire Emblem: Awak- Awakening are unable to use Pair Up. Fates
ening, it was internally planned to be the entry in responded to these criticisms by splitting Pair Up
the series if it did not sell well (Kantopia 2021). into Attack Stance and Guard Stance (Ramey
After Awakening sold very well, Intelligent Sys- 2020).
tem hoped to recreate what made it so popular In Attack Stance, an adjacent ally will be able
while addressing mechanical criticisms. Fire to join combat after the main unit has attacked
Emblem Fates tells the story of Corrin, a noble with slightly less damage and the inability to
born to Eastern-inspired nation of Hoshido but perform a follow-up attack. In Fig. 2, Odin’s dam-
raised in the Western-inspired nation of Nohr. age, accuracy, and critical chance are in the left
Nohr and Hoshido are on the brink of war when column while Ophelia’s is in the middle column.
the game begins. Corrin must soon decide what The archer’s statistics are in the right column.
nation to fight for. Fates made several changes to In Guard Stance, two units can occupy two
the weapon systems. Weapons no longer break spaces. The lead unit will gain stats according to
and several have secondary effects, typically rais- their partner’s class, support ranking with the lead
ing or lowering stats. Keeping with the themes of unit, and stats that each unit innately gives their
Western versus Eastern influences, each kingdom support partners. In addition to stat gains, the pair
has slightly different weapons: swords/katanas, gets a Guard Gauge. The Guard Gauge starts at
tomes/scrolls, axes/clubs, bows/yumi, lances/ zero shields and is full at ten shields. For every hit
naginata, and daggers/shuriken. Hoshidan that the lead unit receives or deals, the shield
weapons are typically weaker when coming to raises by two. Once the Guard Gauge is full, the
raw strength while adding more stats in other backing unit will jump in to fully block a hit from
areas, such as katana granting more speed while the lead unit. Units in Guard Stance cannot be hit
dropping defense and resistance. In addition, by the foe’s Attack Stance partner. If the units
Nohrian staves have stranger healing capabilities unpair or switch with other partners, the average
than their Hoshidian counterparts, rods. However, Guard Gauge will average between the two
Fire Emblem Fates, Conquest 717

Fire Emblem Fates, Conquest, Fig. 2 Odin is attacking Fire Emblem Fates, Conquest, Fig. 3 Keaton is paired
an enemy archer with assistance from Ophelia up with Camilla in guard stance to attack an enemy
Faceless
pairings. Enemy units can use both Attack and F
Guard Stance. Seal. If a child’s chapter is done after chapter
In Fig. 3, note the full Guard Gauge at the 18 of the main story, the unit will come with an
below the blue HP bar; Camilla will block the Offspring Seal that allows them to gain access to a
foe’s next attack in combat. higher-level class with extra level and weapon
Fates makes several changes to the class sys- rank that is curved to the enemies of the main
tems. Once again keeping with West versus East chapter.
themes, many Nohrian and Hoshidan classes Units that have blood of the First Dragons can
share characteristics. For example, the Nohrian access Dragon Veins during battle. With them,
class Outlaw can be seen as equivalent to the players can change terrain, heal, or buff allies,
Hoshidan Ninja as they both are used to pick damage or debuff allies, etc.. While Dragon
locks and are rather quick. There are also ways Veins aren’t usually necessary to a map’s objec-
to reclass. Units can reclass into a stronger class tive, they often provide more dynamic ways to
starting at level 10. All units have a secondary strategize as each map has a unique effect.
class they can reclass to with a Heart Seal. Units At the start of the game, the player is able to
can reclass into their married partner’s class with a customize their avatar, who’s default name is
Marriage Seal. Units can reclass to the class that Corrin. The player can customize their gender,
their best friend (units are best friends if they reach voice, hair color, and accessorize them. The
the highest level support rank, A+) has. Units will player can also choose their secondary class with
gain skills by leveling up in classes. Skills are their boons and banes. The boons determine what
buffs that can range from raw stats, more move- stats Corrin excels in by modifying growth rates.
ment options, the ability to give stats to allies, Likewise, the banes will determine what stats
more powerful attacks, etc. All main characters Corrin struggles with.
have personal skills that are innate and are exclu- Between chapters, players will have access to
sive to them. My Castle. My Castle is a place where players can
If units gain a high enough support level, they view supports, buy weapons and items, talk to
can wed and have children. Children will be based other characters, forge weapons, and collect
on their father (Silas will always have Sophie as a resources to customize the castle. My Castle
child) while the mother will pass down their hair serves as a downtime between chapters that
color. The opposite is true for Azura’s child and allows players to plan out their next moves. Sup-
Corrin’s child, if Corrin is female. With the power ports are built between characters as lines of con-
of space-time travel, the children are able to be versation. Characters have up to three supports
recruited into the player’s army after beating their with their support partners, labeled C, B, and
chapter. Child growth rates will change based on A. A fourth level of support is available to mar-
averages of their parents. Parents will also pass riage candidates and best friends (S and A+,
down their last equipped skill and their classes. respectively). Forging weapons allow for
Children can access these classes with a Heart weapons to become more powerful and for
718 Fire Emblem Fates, Conquest

players to name their weapons. Using the Mess will not engage in combat if they can do zero
Hall gives the chance to give slight buffs to allies damage. In maps where the main objective is to
for the next battle. The units Niles and Orochi have the player defend a point, enemies will
have personal skills that allow them to capture sometimes refuse to fight the player and rush
certain enemy units. They can be recruited into their objective. Enemies will also use Dragon
the player’s army at the Prison. Occasionally, Veins to their advantage. Another design differ-
players will be challenged to battles by mysterious ence from Birthright and Revelations is the lack of
forces and have to defend their castle. Players can what the Fire Emblem community calls “ambush
also go online and visit castles that belong to other spawns.” Ambush spawns are reinforcements that
players and challenge them to battle. appear and take an action on the same turn. All of
The first five chapters of Fates are the same. In these factors work together to make Conquest
chapter six, the player can choose to send Corrin more challenging and rewarding to players who
down the Birthright, Conquest, or Revelations. can plan out complex strategies in advance.
When selecting Birthright, the game gives the
description of, “This path includes opportunities
to freely accumulate experience points and gold. Unhappy Reunion
This will allow you to enjoy watching your char-
acters grow faster.” When selecting Conquest, the The tenth chapter of Conquest, Unhappy
game gives the description of, “This path allows Reunion, is thought by many Fire Emblem fans
you to test your skills and devise complex strate- to be an example of a well-designed map. Many
gies while progressing with limited access to more believe that Unhappy Reunion is one of the
experience points and gold.” When selecting Rev- hardest maps in the series. In chapter 10, Corrin is
elations, the game gives the description of, “It is attempting to find a boat to transport their army to
not recommended to select this path for your first find the Rainbow Sage. To Corrin’s surprise, the
playthrough. This route is best experienced after Hoshidian prince Takumi is sending an army to
playing both the Defend Hoshido and Fight with Nohr. Refusing to let Hoshidans on Nohrian soil,
Nohr paths.” Corrin sets out to defend the port town.
In Fig. 4, blue tiles represent water tiles that only
flying units can traverse, the darker tiles represent
Embrace the Dark walls, and white tiles represent roofs. The northern
section of the map is a boardwalk that has two
Conquest’s “limited access to experience points houses. The middle of the boardwalk has a green
and gold” changes the way players play the game. section that Corrin must defend. If a unit visits them,
In the two other routes, players return to My the top left house will give ten thousand gold and the
Castle after every chapter. From there they can top right house gives a Master Seal. South of the
choose to scout on completed maps for extra gold house lies a breakable wall. To the left of the
enemies or choose side chapters provided they starting position is a Fire Orb, an item that mages
meet the conditions to unlock them. Conquest’s can use. Next to the house that gives a Master Seal is
limited maps force players to play tighter in the a ballista. Another ballista can be found below the
chapters they do have. Conquest players will have starting position. The house in the middle left gives
a harder time gaining supports between units, so a Dual Club, a weapon that reverses weapon triangle
experienced players will think about supports advantage and doubles its effects. The wall to the
beforehand and dedicate some turns to building right of the house is breakable. The middle right
supports. On top of this, Conquest’s maps are house gives a defensive stat booster. At the south-
more difficult, having more varied objectives west section of the map is Takumi holding an Elixir,
and side objectives. Enemy units have more skills, an item that restores 99 HP. Takumi is standing over
higher stats, and stronger artificial intelligence. a Dragon Vein, but the game will not show its
Under most circumstances in Conquest, enemies description. Behind the middle left section of units
Fire Emblem Fates, Conquest 719

Fire Emblem Fates, Conquest, Fig. 4 Conquest

Fire Emblem Fates, Conquest, Fig. 5 Chapter 10 map
Chapter 10 Map
with water drained

is Hinata, Takumi’s retainer. Behind the middle right help. Clearing out units near the middle houses
section of units is Oboro, Takumi’s other retainer. At early will give the players the ability to visit
the southern part of the map are two boats with a them, although it is not required. Due to the com-
Sky Knight flying between them. Ninja, Oni Sav- plexity of this map, Veteran players of Conquest
ages, Archers, and Spear Masters will appear as will plan their playthrough around this chapter.
reinforcements periodically on the left and right Camilla and Beruka are very helpful as units due
side of the map. More Sky Knights will appear to their flying mounts. Being flying units, they are
from the South. able to quickly traverse the map. Using heart seals,
The opening position provides an advanta- some players change their units’ classes to have
geous position to defend the point: walls to the greater movement for this chapter. Elise and Corrin
South and West, a Fire Orb, and Ballistae. On turn (if they picked Wyvern or Sky Knight as their
three, Camilla and her retainers appear, Beruka secondary class) can reclass into flying classes.
and Selena. Camilla is a Malig Knight, Beruka is a Jakob has the ability to reclass into a Paladin for
Wyvern Rider, and Selena is a Mercenary. Camilla cavalry movement. Buying weapons that have one
and Beruka are flying units, so players will find and two range attacks will give the ability to
the ability to cross water useful. On the fifth turn, counterattack the ninja. Breaking the wall next to
reinforcements will start appearing and moving the house containing the Dual Club will allow for
toward the green section (Fig. 5). the ability to exit the starting position and challenge
On turn seven, Takumi will activate his Dragon Hinata faster. Camilla is one of three units by
Vein, causing the water to drain from the map default that can wield clubs, so most players will
allowing all units to traverse them. All enemy give it to her if they chose to fight Hinata. Purchas-
units will then begin to rush their objective. Players ing the Nosferatu tome for Odin will give him the
who played defensively will find it hard or impos- ability to leech life from enemies. With this, Odin is
sible to defend the objective without casualties due able to challenge several units without requiring a
to the sheer number of units rushing at them. staff unit to be nearby. Once Takumi drains the
Players who know about the reinforcements and water, some players will try to congregate near
Takumi using the Dragon Vein will tend to realize the starting position to form a wall and challenge
that playing defensively as the objective suggests the remaining units. Camilla, Beruka, and any
isn’t the best strategy. Trying to get out of the other high movement units are able to chase
starting position to chip away at the enemy num- down the enemy flying units. If the fliers get too
bers before they rush toward the point will greatly close, bow units will make quick work of them.
720 Fire Emblem Fates, Conquest

Unhappy Reunion’s complexity, rewards, side

objectives, and the ability to challenge the map in
many different ways give players a sense of
accomplishment when beating the chapter.

Eternal Stairway

The 21st chapter of Conquest, Eternal Stairway, is

another map that is lauded by many that play as a
map that tests players’ knowledge by flipping
traditional mechanics on their head. In Eternal
Stairway, Corrin’s army is invading Hoshido and
is using an alternate route through the Wind Tribe Fire Emblem Fates, Conquest, Fig. 6 Chapter 21’s map
lands to sneak around the main Hoshidian army.
As they are traversing the Eternal Stairway, they
are suddenly surrounded by monsters called Face- When tasked with a chapter that has a different
less and Stoneborns. The monsters are too vast in objective, players will be forced to adapt their
numbers to fight them all so Corrin and their allies playstyle. One solution to clear the map quickly
are forced to find a way to escape. is to undeploy everyone except Corrin and a flying
Chapter 21’s objective is “Escape with all unit that can use Dragon Veins. They will then pair
units.” All units must make their way from the up and activate every Dragon Vein as they fly over
starting area located at the bottom of the map the steps toward the end. While this is a valid
through several large steps to the top of the map strategy, players will miss experience gained by
to escape. On the steps are Dragon Veins that will defeating enemies. Players have several options to
reduce all foe movement to zero. Scattered across maneuver up the stairway. In much the same way
the stairway are Faceless and Stoneborn. Faceless that chapter 10 forces players to defeat enemies
are one ranged enemies that come equipped with early before they can swarm, Eternal Stairway
the Savage Blow skill, which reads, “When user forces players to defeat foes as they climb the
triggers the battle, enemies within a 2 tile radius stairway or fight the endless waves of monsters.
have their HP reduced by 20% after the battle.” Flying units have movement advantages as they
On harder difficulties they have skills that reduce can fly over the steps and easily access the Dragon
accuracy for a specific weapon type. Stoneborn Veins that are needed to freeze the monsters to
are enemies that launch stones at their foes from keep the other units safe. Armored units are paired
up to five tiles away. They come equipped with up with other units with more movement or have
Wary Fighter, a skill that prevents enemies from their class changed entirely as to not lag behind.
making follow-up attacks. On harder difficulties, Infantry and other mounted units can clear the
they come equipped with Renewal, a skill that lets Faceless on the sides of the stairs to make a path
them regain 30% of their HP at the start of the for units that can’t take many hits from the Face-
turn, and Heartseaker, a skill that reduces foes less and Stoneborn. In order to dispatch units with
avoid. At the top of the staircase is a boss Faceless Wary Fighter, some players use blessed weapons,
that holds an Arms Scroll, an item used to increase a class of weapons that deals extra damage to
weapon rank. Additional enemies will endlessly monsters. Other players use brave weapons.
appear at the top and bottom of the map. They Brave weapons are a class of weapon that lets
come equipped with Wary Fighter, Grisly Wound their wielders attack twice consecutively when
(a skill that reduces enemy HP by 20% after initiating combat but makes them lose five
battle), and Void Curse (a skill that prevents expe- speed. Brave weapons are usable at A rank so
rience gain) (Fig. 6). only units that have invested lots of time into a
First-Person Shooter Games, a Brief History 721

specific weapon can use them. The Lightning

tome is a brave tome that can be used a C rank Fire Emblem If
instead of the usual A. Attacking the Stoneborn in
Attack Stance will allow players to hit them four ▶ Fire Emblem Fates, Conquest
times in a round of combat if both attackers are
using brave weapons.
Eternal Staircase is a map that flips tradi-
tional Fire Emblem game mechanics on their First Nations
head in ways that force players to design new
strategies using the units and resources they ▶ Indigenous Language Revitalization with
already have. Stories and Games
F
Conclusion and Discussion
First-Person Shooter
Fire Emblem Fates: Conquest is a game that was
designed for fans of strategy games, Fire ▶ Call of Duty Franchise, an Analysis
Emblem veterans, and those who seek a chal- ▶ Star Wars Battlefront (2004), an Analysis
lenge. Conquest rewards players who can ana-
lyze the problems thrown at them and plan out
their moves in advance. The new mechanics of
Attack Stance and Guard Stance allow for First-Person Shooter (FPS)
players to perform more complex strategies that Game
previous entries did not allow for. Conquest’s
experience is crafted in a way that allows for ▶ Overwatch: Team-Based Multiplayer First-Per-
player experimentation while also retaining its son Shooter Game
difficulty.

References First-Person Shooter Games, a

Brief History
Harris, J.: Game Design Essentials: 20 rpgs. Gamasutra.
Retrieved February 2, 2022, from https://www. Brandon Ford2 and Newton Lee1,2
gamasutra.com/view/feature/4066/game_design_essen 1
Institute for Education, Research, and
tials_20_rpgs.php?page¼14 (2 July 2009)
Kantopia: Fire emblem awakening: Nintendo Dream first Scholarships, Los Angeles, CA, USA
2
year anniversary developer interview (June 2013). Vincennes University, Vincennes, IN, USA
kantopia. Retrieved February 2, 2022, from https://
kantopia.wordpress.com/2017/05/28/fire-emblem-
awakening-nintendo-dream-first-year-anniversary-
developer-interview-june-2013/ (16 January 2021) Synonyms
Ramey, J.: How fire emblem has tried to balance pair-up
supports (and why it’s so hard). TheGamer. Retrieved First-Person Shooter, FPS
February 2, 2022, from https://www.thegamer.com/
fire-emblem-balance-pair-up-supports-difficulty/
(5 November 2020)
Yamoto, S.: The past and future of “Fire Emblem.” Ask Definition
the production team about the secret story of develop-
ment during the NES era to the latest work “Fire
Emblem if”. 4gamer.NET. Retrieved February 2,
First Person A type of game play mode that
2022, from https://www.4gamer.net/games/287/ Shooter lets the user (gamer) see the
G028791/20150427113/ (n.d.) character they are playing.
722 First-Person Shooter Games, a Brief History

FPS First Person Shooter. by GT Interactive, would become one of the first
Maze War A simple 2D maze type of FPS. e-sports game that offered the winner a prize.
Wolfenstein A 3D version of Wolfenstein that Some FPS games such as “Call of Duty” use
3D came out 1981. real world events. “Call of Duty,” published by
DOOM The first true 3D game that came Activision, would take the gamer to the beaches of
out in 1993, which launched what Normandy, and all the way to future warfare that
we now know of FPS. has yet to be seen. Activision came out with the
Quake A 3D game that took what game in 2003 for PC, adding different campaigns
DOOM made and made it better; for different countries and the way they fought in
it was also one of the first Esports each war. The game used iD tech engine.
games. The game “Duke Nukem” gave the playable
Duke An FPS game that gives the main character an icon for wisecrack speeches during
Nukem character more of the starlight gameplay. Scott Miller of 3D Realms created the
with lines than he would use game and sold the rights to Gearbox Software in
during game play. 2010. Gearbox later released “Duke Nukem For-
Unreal A 3D FPS game engine. ever” with higher quality graphics.
Unreal A multiplayer game with area-
Tournament based “deathmatches.”
Deathmatch A genre of multiplayer video Unreal Engine
game in which a player’s goal is to
kill or eliminate the opponent FPS would go even farther with better game
from the match. engines (Buckley 2022). The second game engine
was called “Unreal” which came from the com-
First Person Shooter (FPS) Games pany Epic Games. FPS games created using the
first version of the Unreal engine include “Star
The first First Person Shooter (FPS) game came Trek: Next Generation: Klingon Honor Guard”
out in 1973. It was known as “Maze War.” The and an adaptation of Robert Jordan’s “Wheel of
game was created by Steve Colley, Greg Thomp- Time” series. But the most memorable game was
son, and Howard Palmer; it was playable over “Unreal Tournament” by Epic Games in 1999.
ARPANET between multiple universities “Unreal Tournament” was a multiplayer game
(Jensen 2017). with area-based “deathmatches.” A deathmatch
The true FPS games arrived in the 1990s when is a genre of multiplayer video game in which a
3D graphics took off and 3D game engines player’s goal is to kill or eliminate the opponent
became available. Wolfenstein 3D (developed by from the match.
id Software and published by Apogee Software The game that took everything known for FPS
and FormGen) came out in 1992. It was the first and rewrote it was “Halo: Combat Evolved” in
stepping stone for all FPS to date. The game was 1999. Bungie made “Halo” into the largest game-
created by employees at iD software: John based FPS ever. The game used artificial intelli-
Carmack, John Romero, Tom Hall, and Adrian gence that would learn from the gamer and would
Carmack. These same people made the next FPS make the gameplay harder at each level. It had
called “DOOM” that are still being used today to different multiplayer modes, both online and
help create new FPS games (Coldewey 2013). offline. The game publisher Microsoft Game Stu-
In 1996, another FPS game took 3D graphics dios later gave “Halo” to 343 Industries, which
farther by letting the player choose the 3rd person continued to make the franchise game.
mode, as well as giving the character a voice. The FPS started out as a basic 2D type of maze
game was called “Quake” that uses iD tech game, and it evolved to a 3D Real Time-based
engine – the first game engine. Quake, published strategy game – games where hearing and
Five Nights at Freddy’s, a Point and Click Horror Game 723

speaking can either help you or hurt you in the

gameplay – which added a new dimension of First-Person Shooter, FPS
realism.
▶ First-Person Shooter Games, a Brief History

Virtual Reality

Virtual reality (VR) gave FPS yet another dimen- First-Person-Shooter

sion of realism. Oculus Rift – developed by Ocu-
lus VR, a division of Facebook – came out on ▶ Counter-Strike Global Offensive, an Analysis
March 28, 2016, and with it came a new way of
gaming (Robertson 2016). VR uses both regular F
and movement controllers. The player can still
play the game normally but with a 360 degree or Five Nights at Freddy’s, a
using their entire body in the gameplay. When you Point and Click Horror Game
need to reload a gun in FPS VR, instead of just
pressing a button, you have to move your hands to Brian Edington2 and Newton Lee1,2
1
make it seem like you are actually reloading in Institute for Education, Research, and
real life. Even the military uses VR and FPS Scholarships, Los Angeles, CA, USA
2
games to train new soldiers for combat. Vincennes University, Vincennes, IN, USA
First Person Shooter games have come a long
way since the 1970s. With more advanced
graphics, game engines, artificial intelligence, Synonyms
and virtual reality, FPS will continue to evolve
and improve for years to come. Horror game; Point and click game; Transmedia
storytelling

Cross-References
Definitions
▶ Call of Duty Franchise, an Analysis
Point and An adventure game in which the
▶ Destiny and Destiny 2, an Analysis of an FPS
click game: player interacts mainly with the
▶ Overwatch: Team-based Multiplayer First-Per-
mouse or a pointing device.
son Shooter Game
Horror A video game based on the horror
game: genre.
Transmedia A technique of telling a single
References
storytelling: story or story experience across
Buckley, D.: How to create a first-person shooter in the multiple platforms and formats.
unreal engine. https://gamedevacademy.org/unreal-
engine-fps-tutorial/. 3 June 2022 Introduction
Coldewey, D.: Knee deep in history: 20 years of “doom”.
https://www.nbcnews.com/technolog/knee-deep-
history-20-years-doom-2d11722313. 10 Dec 2013 Technology has advanced dramatically over the
Jensen, K.: The complete history of first-person shooters. past century, going from giant supercomputers
https://www.pcmag.com/news/the-complete-history- and the invention of the radio to the portable
of-first-person-shooters. 11 Oct 2017 computer better known as a laptop and a small
Robertson, A.: Oculus Rift review. The Verge. https://
www.theverge.com/2016/3/28/11284590/oculus-rift- device most people carry daily to do nearly any-
vr-review. 28 Mar 2016 thing we need. The same can be said about video
724 Five Nights at Freddy’s, a Point and Click Horror Game

games, having gone from a colossal arcade developer Scott Cawthon. The game’s setting
machine to a downloadable application on a varies from game to game, but the premise
phone, computer, or console. Some of these between them remains the same: The player,
games have gone as far as being a part of esports, looking through the eyes of the games’ protago-
though some would not and do not fall under that nists, must stay alive for five or six days against
category. Indie horror games are made by either a an onslaught of animatronics who want your
small team or an individual using horror ele- blood” (Oxford 2021). The player’s only
ments – whether they are terror or gore, or both. defenses are the security cameras in the building,
One such indie game has made an impact on the the office door, and the lights. The player has a
future of indie horror games, though it is debatable limited amount of electricity each night. There
if it is for the better or for the worst. That game is are no playable characters, battle systems, or skill
Five Nights at Freddy’s. trees. It is a horror game with heart-pounding
Five Nights at Freddy’s, also known as FNaF, jump scares, but without the blood, gore, and
is an indie horror game released in mid-to-late extreme violence typically associated with hor-
2014 by, at the time, a financially struggling ror films. The player’s task is to survive, as the
game developer known as Scott Cawthon. game title states, five nights at Freddy’s. The
According to Patricia Hernandez, a writer on challenges are Easter egg hunting and surviving
Kotaku, “It’s been called the ‘scariest game in each night without getting killed by the anima-
years.’ It’s at the top of iOS and Android tronic characters. Here are the basic steps to play
app stores, it has torn up the Steam charts on the survival horror game as a security guard
PC. It has racked up millions and millions of working the night shift:
views (and screams) on YouTube. It has hundreds
of earnest fan games, and dozens of rabid fan 1. Monitor the animatronics using security cam-
theories” (Hernandez 2015). With each new eras and lights.
installment or addition to the FNaF series, the 2. Manage limited resources wisely (e.g., Should
fanbase grew bigger, adding on fan-made songs, the player close the doors to be safe or leave
remixes, gameplay, original content based on the them open to conserve battery?)
series, and more. As of 2022, the Five Nights at 3. Protect yourself from the animatronics by
Freddy’s franchise has roughly nine games to its using a set of tools.
name, three book series based on the games, a
movie that has been in the works for quite a few Each version of Five Nights at Freddy’s
years now, and even an initiative to fund some offers a different set of tools. For example, in
game developers to work on their projects. Five Nights at Freddy’s, the player can control
Despite what negative or derogatory things some the two security doors connecting their office to
people have to say about Cawthon, his games, or the adjacent hallways as a barrier against anima-
the fanbase, there is no denying that the first tronics in the vicinity. However, in Five Nights
game – the game that started it all – is what at Freddy’s 2, there are no protective doors.
brought a new form of horror to light. Instead, the player must use an empty anima-
tronic head and flashlight to defend itself against
the animatronics. Five Nights at Freddy’s
Gameplay 3 offers a monitor panel, which contains audio,
camera, and ventilation. The common elements
According to Nadia Oxford, a staff writer for across the different versions are security cam-
USGamer, “Five Nights at Freddy’s (FNAF) is eras, lights, doors, vents, jump scares, mini
a series of point-and-click horror games for PC games, phone calls, Easter eggs, and location
and mobile made by independent game closings.
Fluid Simulation 725

Reception Cross-References

At first, FNaF did not receive a lot of attention, but ▶ Narrative in Video Games
Cawthon’s luck would turn around soon enough. ▶ Video Game Storytelling Fundamentals: Set-
A YouTuber/theorist who goes by FuhNaff cre- ting, Power Status, Tone, and Escalation
ated a video about the history of the FNaF fran-
chise, its humble beginnings, its rapid rise in
popularity, and its ever-increasing amount of con- References
tent. In the video, he mentions that the game’s first
demo was released on a site called IndieDB but FuhNaff: The Entire History of FNAF, YouTube. https://
www.youtube.com/watch?v¼fg6XBl2sEOM&t¼321s
hardly anyone played the demo, except for an
(2022, August 09). Accessed 26 Sept 2022
unknown YouTuber at the time, who was given Hernandez, P.: Why Five Nights at Freddy’s Is so Popular. F
the full game by Cawthon himself. Fast forward to Kotaku. https://kotaku.com/why-five-nights-at-
August of that year, Cawthon released the full freddys-is-so-popular-explained-1684275687 (2015,
February 09)
game on another site called Desura where another
Oxford, N.: Murder, Dysfunctional Families, and Purple
YouTuber, more well-known back then, did a Guys: The Larger Story behind the Five Nights at
“Let’s Play” on the game, and it started to get Freddy’s Games [Updated for Freddy Fazbear’s Pizze-
noticed. People began to adore the game. With a ria Simulator and UCN]. USgamer.net, USgamer.
https://www.usgamer.net/articles/murder-ghosts-and-
growing audience and the more popular revenge-the-larger-story-behind-the-five-nights-at-
YouTubers playing the game, FNaF keeps freddys-games-06-2018 (2021, March 26)
expanding the fanbase (FuhNaff 2022).

Conclusion Fluid Simulation

Five Nights at Freddy’s became a huge success Manuel Schmidt1 and Yann Savoye2
1
after its slow start, racking up millions of down- University of Innsbruck, Innsbruck, Austria
2
loads on various websites and mobile phones. Institut fur Informatik, Innsbruck University,
FNaF has become an icon in certain parts of the Room 3M11, Innsbruck, Austria
gaming community, being recognized as an old
game that some might consider it a game that
should have lost traction long ago, while others Synonyms
see it a continuous story to be solved by the
community as long as they find the right clues to Computed fluid flow
solve the mysteries of the old haunted pizzeria.
The game has spawned a novel trilogy, the
Definition
Fazbear Frights series, and the Tales from the
Pizzaplex series cowritten by Scott Cawthon as
Simulation of fluid flow based on real-time compu-
well as graphic novels adapted and illustrated by
tation on a workstation using different assumptions
other authors. A film adaptation was announced in
to approximate the overall behavior of the fluid.
August 2022 that the script had been written by
Cawthon, Emma Tammi, and Seth Cuddeback,
and the film will be produced by Blumhouse. Introduction
Five Nights at Freddy’s is a prime example of
transmedia storytelling across multiple platforms Digital simulation has been a well-explored
and formats. research topic over the last decade. In particular,
726 Fluid Simulation

the simulation of natural phenomena like fluid representation for fluid simulation and compari-
movement has attracted a lot of attention. There son to the traditional grid approaches. Later,
are mostly two different kinds of fluid simulation. numerous SPH techniques were presented follow-
One is for realistic movement of fluids in a high- ing the same principle. For instance, interested
performance computing area, and the other one is readers are referred to the survey of Ihmsen
a real-time fluid simulation with desktop worksta- et al. (2014). Then, we propose to detail various
tions. This literature review should give an over- key features of existing SPH techniques: surface
view on existing work in the field of real-time tension, surface meshing, bubbles and foam, level
fluid simulation and the differences between dif- of details and multi-scales, solver convergence,
ferent fluid representations. In the last decades, we multiple liquids, and elasticity material.
can observe three different classes for real-time Surface Tension. The surface tension comes
fluid simulations. The first one is dealing with by design while using a height field for fluid
fluid in a height-field environment that is very simulation. In contrary, surface tension should be
simple – but – yet efficient approach. This explicitly defined while relying on particles as
approach has drawbacks to show more sophisti- claimed by Yu et al. (2012) and Schechter and
cated effects like splashing and bubbles. To cal- Bridson (2012). Both papers present two different
culate the fluid flow in all dimensions makes the types of particle-based simulation that just not
simulation less efficient but enables to show track particles for fluid simulation but also employ
effects like particles. Last but not least, the particles for modeling airflow around the fluid
particle-based approach is the more recent one surface. Defining two kinds of particles and mak-
where each particle is a sample of the fluid flow, ing them interact with each other enable to simu-
and with this samples the overall fluid flow gets late surface tension. Furthermore, Yu et al. (2012)
approximated. present a surface tracker to map particles into
meshes, and Akinci et al. (2013) propose a
method to simulate large-scale surface tension.
Related Work However, Busaryev et al. (2012) extend the idea
to use different particles to represent droplets and
In this entry, we overview various classes of fluid bubbles. Finally, using different particle types for
representations currently used for commercial different specificities is a very promising way to
applications in computer graphics. Three tech- simulate natural phenomena in particles.
niques are most commonly employed: height Surface Meshing. A major difficulty with
field, cubic Eulerian grid, and particles. Each particle-based fluid simulation is to transform
family of fluid representation exhibits different free particles into a mesh for visualization and
characteristics that can fit different application then render them without flickering artifacts.
demands. For a more specific introduction about This problem is even more challenging to solve
existing techniques for fluid simulation, we refer in the presence of splashing fluid. Yuksel
the reader to one of the most comprehensive et al. (2007) realized wave simulation and convert
books about fluid simulation published by Robert particles into a height field for fast rendering. To
Bridson (2008). calculate collisions with the fluid particles, Akinci
et al. (2012) propose a two-way coupling method
Particle-Based Fluid Simulation for SPH fluids, using particles inside the solids.
The class of particle-based techniques uses a set of Also, a collection of previous works (Hoetzlein
independent particles that approximate the fluid and Höllerer 2009; Batty et al. 2010) generate
flow function by discrete values. The critical SPH meshes from particles. Still, the problem of gen-
drawback is the amount of computation time erating triangular surface meshes from particles
needed to process a large set of particles. The remains a challenging problem with active
seminal work for SPH is described by Premoze research. For instance, Wojtan et al. (2010) intro-
et al. (2003) introducing a particle-based duce a mesh-based surface tracking method for
Fluid Simulation 727

particle-fluid simulations. Finally, Ando Heterogeneous Liquids. A collection of

et al. (2013) introduce a tetrahedral discretization works allows the simulation of multiple liquids
for the pressure projection and a method to extract at different densities and viscoelastic properties
a smooth surface from particles. inside the same simulation (Losasso et al. 2006;
Bubbles and Foam. Thanks to the computa- Ren et al. 2014; Shin et al. 2010). For instance,
tional power available on standard workstations Lenaerts et al. (2008) present a SPH fluid tech-
nowadays, particle-based simulation offers more nique to flow through a porous deformable using
sophisticated natural phenomena and realistic nonlinear materials. Also, Batty et al. (2007) com-
effects like bubbles and foam. A collection of bine fluid and solid with irregular boundaries.
previous works (Hong et al. 2008; Cleary Recently, Robinson-Mosher et al. (2008) prefer
et al. 2007; Kim et al. 2007) simulate bubbles to merge the momentum together.
with SPH fluid simulation. The key idea is to F
mix bubble-labeled particles into the overall Eulerian Grid Fluid Simulation
particle-based fluid. Moreover, Busaryev Another well-known class of approaches for fluid
et al. (2012) extend the classical particle-based simulation is the 3D Eulerian grid. An important
liquid simulation by incorporating bubbles inside property of the Eulerian grid is its ability to rep-
the foam. To simulate a bubble effect with higher resent complex 3D effects like splashing. Unfor-
realism, the connectivity information of a Voronoi tunately, the quality of the fluid effects is limited
diagram should be built over the foam particles. by the grid resolution. An important work intro-
Level of Detail and Multi-scales. Levels of ducing 3D Eulerian grids is Chentanez and Müller
details are employed to speed up rendering while (2011) using cubic grid cells to demonstrate com-
increasing the realism as much as possible with plex 3D fluid effects. Lentine et al. (2010)
respect to available computer power. For improve the Eulerian fluid grid by dynamically
instance, Solenthaler and Gross (2011) improve adapting geometry from a given coarse grid.
standard SPH by varying particle size at different However, Brochu et al. (2010) prefer to rely on a
regions of the scene to be simulated. The key idea Voronoi mesh interpolation, starting from a
is to use smaller particles where more details are Eulerian liquid simulation. Also, Thürey
needed, for instance, at collision location with et al. (2010) extend the Eulerian techniques by
obstacles and larger particles far away from the incorporating surface tension to offer more con-
camera. In this direction, Yu et al. (2009) present trollability over the fluid. Moreover, Raveendran
view-dependent levels of detail technique to deal et al. (2012) employ a sparse mesh to control fluid
with large-scale scenes representing rivers with simulation. More recently, few works improve
particle simulation in real time. Recently, Zhu Eulerian liquid simulation by tracking fluid details
et al. (2013) rely on an extendable dynamic grid (Bojsen-Hansen and Wojtan 2013) finely and
to improve the efficiency. Finally, Edwards and enhancing the visual quality of the fluid simula-
Bridson (2014) introduce an adaptive fluid sim- tion without using a finer grid (Zhang and Ma
ulation enabling simulation at high resolution 2013). Finally, we notice that Eulerian-based
without the need of a fine discretization for the fluid simulation is not often employed in produc-
entire field. tion as a single representation, because of its
Solver Convergence. Some techniques focus inherent limitations. Thus, Eulerian grids are com-
on increasing time steps for particle-based fluid bined with other techniques to form hybrid solu-
simulation. For example, Macklin and Müller tions (as detailed in section “Hybrid Fluid
(2013) improve SPH fluid simulation with surface Simulation”).
tension toward a better convergence and a con-
stant density, allowing larger time steps using Hybrid Fluid Simulation
fewer particles. Finally, Solenthaler and Pajarola We observe the high demand of hybrid techniques
(2009) increase the time step of SPH fluid simu- combining different fluid representations to pro-
lation by incorporating the particle pressure. duce large-scale simulation with fast
728 Fluid Simulation

computational time and fewer memory require- close to a guided fluid simulation and to produce
ments. For instance, an efficient strategy is to a predictable fluid simulation. For instance, Fos-
start with a height field as input to allow rendering ter and Metaxas (1996) use height field for ren-
of large-scale scenes and then to switch to more dering effects like wave reflection, refraction,
sophisticated approaches in the case where more and diffraction. Also, Klein et al. (2003) describe
details are required. Outstandingly, it is feasible to how noise-based animation improves the appear-
combine height-field, Eulerian, and particle-based ance. Moreover, Müller-Fischer et al. (2008)
methods to obtain efficient, stable, and flexible reduce the computation time for a fluid simula-
fluid simulation for breaking waves, waterfalls, tion to be used in real-time applications like
and splash particles. Hybrid techniques enable games. The key idea is to employ 2D height
high details on large water simulation in real field with low resolution coupled with realistic
time by switching between the different fluid tech- shading effects, while freezing the simulation at
niques. One of the first hybrid fluid simulation non-visible regions. Finally, Yuksel and Keyser
techniques was proposed by O’Brien (2009) improve the visual appearance of height-
et al. (1995) combining height-field fluid simula- field fluid simulations with real-time caustics
tion with a particle-based approach to generate using a generated caustic map.
spray effects. Later, this approach was improved
by Chentanez and Müller (2014) by adding sta-
bility constraints and a combination with 3D Conclusions
Eulerian grid (Chentanez and Müller 2014).
Finally, we notice that hybrid strategies improve All fluid techniques have their advantages and
the quality and flexibility of fluid simulation. disadvantages. There is still a lot of ongoing
research in the field of fluid simulation. At all
Fluid-Oriented Height Fields techniques, we have observed that the time step
Height fields are often used to represent simple between two iterations is critical. A too long com-
surfaces thanks to its simple structure and fewer putation delay damages the simulation in the form
memory overheads. This representation is com- of stability or unpleasant visual results. Height-
monly used for game terrains and can be easily field fluid simulation is a simple and long-
adapted for LOD and tiling. Height fields are an standing fluid simulation technique developed
important class of fluid simulation motivated by for computer graphics but is still very often used.
an efficient use of resources. We refer interested The particle approach is newer with a lot of ongo-
readers to the excellent introduction of Cline ing research which tries to fix known drawbacks
et al. (2013) to the concepts of fluid simulation to make the simulation even more realistic. Very
on height field. Also, we highlight two seminal interesting is the idea to combine the different
works. Kass et al. (1991) approximate the shallow fluid approaches to a hybrid approach. This
water equation to simulate wave refraction with helps to avoid the drawbacks of the different tech-
depth, net transport of water, and boundary con- niques but introduces more logical complexity to
ditions with changing topology. Moreover, Stam the fluid simulation. Overall, fluid simulation is
et al. (1999) animate fluid in real time by splitting still an attractive research area where no perfect
and simplifying the Navier-Stokes equation. More solution exists.
recently, Nielsen et al. (2013) synthesize waves,
while resolving mesh self-intersection.
Mikls et al. (2004) introduce a computation- Cross-References
ally efficient layered water model to approximate
the upper and lower water surface, allowing 3D ▶ Lattice Boltzmann Method for Fluid
water effects, like water flowing into a jar. More Simulation
recently, Nielsen and Bridson (2011) introduce ▶ Lattice Gas Cellular Automata for Fluid
constraints on which a fluid simulation keeps Simulation
Fluid Simulation 729

References Hong, J.-M., Lee, H.-Y., Yoon, J.-C., Kim, C.-H.: Bubbles
alive. ACM Trans. Graph. 27, 48 (2008), ACM
Akinci, N., Ihmsen, M., Akinci, G., Solenthaler, B., Ihmsen, M., Orthmann, J., Solenthaler, B., Kolb, A.,
Teschner, M.: Versatile rigid-fluid coupling for incom- Teschner, M.: Sph fluids in computer graphics. In:
pressible sph. ACM Trans. Graph. 31(4), 62 (2012) Eurographics 2014State of the Art Reports,
Akinci, N., Akinci, G., Teschner, M.: Versatile surface pp. 21–42. (2014)
tension and adhesion for sph fluids. ACM Trans. Kass, M.: Height-field fluids for computer graphics. In:
Graph. 32(6), 182 (2013) Proceedings of the 23rd Conference on Winter Simu-
Ando, R., Thürey, N., Wojtan, C.: Highly adaptive liquid lation, IEEE Computer Society, pp. 1194–1198. Wash-
simulations on tetrahedral meshes. ACM Trans. Graph. ington, DC, WSC ‘91, IEEE Computer Society (1991)
32(4), 103 (2013) Kim, B., Liu, Y., Llamas, I., Jiao, X., Rossignac, J.: Sim-
Batty, C., Bertails, F., Bridson, R.: A fast variational frame- ulation of bubbles in foam with the volume control
work for accurate solid–fluid coupling. ACM Trans. method. ACM Trans. Graph. 26, 98 (2007), ACM
Graph. 26(3), 100 (2007) Klein, T., Eissele, M., Weiskopf, D., Ertl, T.: Simulation,
Batty, C., Xenos, S., Houston, B.: Tetrahedral embedded modelling and rendering of incompressible fluids in F
boundary methods for accurate and flexible adaptive real time. In: Proceedings of the Workshop on
fluids. Comput. Graph. Forum 29, 695–704 (2010), Vision, Modelling, and Visualization 2003
Wiley Online Library (VMV ‘03), pp. 365–373 (2003)
Bojsen-Hansen, M., Wojtan, C.: Liquid surface tracking Lenaerts, T., Adams, B., Dutré, P.: Porous flow in particle-
with error compensation. ACM Trans. Graph. 32(4), based fluid simulations. ACM Trans. Graph. 27,
68 (2013) 49 (2008), ACM
Bridson, R.: Fluid Simulation. A. K. Peters, Ltd., Natick Lentine, M., Zheng, W., Fedkiw, R.: A novel algorithm for
(2008) incompressible flow using only a coarse grid projec-
Brochu, T., Batty, C., Bridson, R.: Matching fluid simula- tion. ACM Trans. Graph. 29, 114 (2010), ACM
tion elements to surface geometry and topology. In: Losasso, F., Shinar, T., Selle, A., Fedkiw, R.: Multiple
ACM SIGGRAPH 2010 Papers, pp. 47:1–47:9. interacting liquids. ACM Trans. Graph. 25, 812–819
ACM, New York, SIGGRAPH ‘10 (2010) (2006), ACM
Busaryev, O., Dey, T.K., Wang, H., Ren, Z.: Animating Macklin, M., Müller, M.: Position based fluids. ACM
bubble interactions in a liquid foam. ACM Trans. Trans. Graph. 32(4), 104:1–104:12 (2013)
Graph. 31(4), 63 (2012) Mikls, B., Müller, A.D.M., Dr, P., Gross, M., Zrich, E.:.
Chentanez, N., Müller, M.: Real-time eulerian water sim- Real-time fluid simulation using height fields semester
ulation using a restricted tall cell grid. In: ACM thesis (2004)
SIGGRAPH 2011 Papers, pp. 82:1–82:10. ACM, Müller-Fischer, M.: Fast water simulation for games using
New York, SIGGRAPH ‘11 (2011) height fields. In: GDC2008, San Francisco America
Chentanez, N., Müller, M., Kim, T.-Y.: Coupling 3D (2008)
Eulerian, heightfield and particle methods for interac- Nielsen, M.B., Bridson, R.: Guide shapes for high resolu-
tive simulation of large scale liquid phenomena. In: tion naturalistic liquid simulation. ACM Trans. Graph.
Proceedings of the ACM SIGGRAPH/Eurographics 30, 83 (2011), ACM
Symposium on Computer Animation, SCA ’14, Nielsen, M.B., Söderström, A., Bridson, R.: Synthesizing
Copenhagen, pp. 1–10. Eurographics Association, waves from animated height fields. ACM Trans. Graph.
Aire-la-Ville (2014) 32(1), 2:1–2:9 (2013)
Cleary, P.W., Pyo, S.H., Prakash, M., Koo, B.K.: Bubbling O’Brien, J.F., Hodgins, J.K.: Dynamic simulation of
and frothing liquids. ACM Trans. Graph. 26, 97 (2007), splashing fluids. In Proceedings of the Computer Ani-
ACM mation, IEEE Computer Society, Washington, DC,
Cline, D., Cardon, D., Egbert, P. K.: Fluid flow for the rest USA, CA ‘95, IEEE Computer Society,
of us: Tutorial of the marker and cell method in com- pp. 198 (1995)
puter graphics. Technical report. Brigham Young Uni- Premžoe, S., Tasdizen, T., Bigler, J., Lefohn, A., Whitaker,
versity (2013) R.T.: Particle-based simulation of fluids. Comput.
Edwards, E., Bridson, R.: Detailed water with coarse grids: Graph. Forum 22, 401–410 (2003), Wiley Online
combining surface meshes and adaptive discontinuous Library
galerkin. ACM Trans. Graph. 33(4), 136:1–136:9 Raveendran, K., Thuerey, N., Wojtan, C., Turk, G.: Con-
(2014) trolling liquids using meshes. In: Proceedings of the
Foster, N., Metaxas, D.: Realistic animation of liquids. ACM SIGGRAPH/Eurographics Symposium on Com-
Graph. Model. Image Process. 58(5), 471–483 puter Animation, pp. 255–264. Eurographics Associa-
(1996) tion (2012)
Hoetzlein, R., Höllerer, T.: Interactive water streams with Ren, B., Li, C., Yan, X., Lin, M.C., Bonet, J., Hu, S.-M.:
sphere scan conversion. In Proceedings of the 2009 Multiple-fluid sph simulation using a mixture model.
Symposium on Interactive 3D Graphics and Games, ACM Trans. Graph. 33(5), 171 (2014)
pp. 107–114. ACM (2009) Robinson-Mosher, A., Shinar, T., Gretarsson, J., Su, J.,
Fedkiw, R.: Two-way coupling of fluids to rigid and
730 For Super Smash Bros.

deformable solids and shells. ACM Trans. Graph. 27,

46 (2008), ACM Formalize
Schechter, H., Bridson, R.: Ghost sph for animating water.
ACM Trans. Graph. 31(4), 61 (2012)
Shin, S.-H., Kam, H.R., Kim, C.-H.: Hybrid simulation of ▶ Abstraction and Stylized Design in 3D Ani-
miscible mixing with viscous fingering. Comput. mated Films: Extrapolation of 2D Animation
Graph. Forum 29, 675–683 (2010), Wiley Online Design
Library
Solenthaler, B., Gross, M.: Two-scale particle simulation.
ACM Trans. Graph. 30, 81 (2011), ACM
Solenthaler, B., Pajarola, R.: Predictive-corrective incom-
pressible sph. ACM Trans. Graph. 28, 40 (2009), Formative Game
ACM
Stam, J.: Stable fluids. In: Proceedings of the 26th Annual
Conference on Computer Graphics and Interactive ▶ Hypermedia Narrative as a Tool for Serious
Techniques, pp. 121–128. ACM Press/Addison- Games
Wesley Publishing Co (1999)
Thürey, N., Wojtan, C., Gross, M., Turk, G.: A multiscale
approach to mesh-based surface tension flows. ACM
Trans. Graph. 29, 48 (2010), ACM
Wojtan, C., Thürey, N., Gross, M., Turk, G.: Physics- Fortnite: A Brief History
inspired topology changes for thin fluid features.
ACM Trans. Graph. 29, 50 (2010), ACM
Yu, Q., Neyret, F., Bruneton, E., Holzschuch, N.: David M. Clark2 and Newton Lee1,2
1
Scalable real-time animation of rivers. Comput. Institute for Education, Research, and
Graph. Forum 28, 239–248 (2009), Wiley Online Scholarships, Los Angeles, CA, USA
Library 2
Vincennes University, Vincennes, IN, USA
Yu, J., Wojtan, C., Turk, G., Yap, C.: Explicit mesh sur-
faces for particle based fluids. Comp. Graph. Forum 31
(2pt4), 815–824 (2012)
Yuksel, C., Keyser, J.: Fast real-time caustics from height Synonyms
fields. Vis. Comput 25(5–7), 559–564 (2009)
Yuksel, C., House, D.H., Keyser, J.: Wave particles. In:
ACM SIGGRAPH 2007 Papers, ACM, New York, Battle; Competitive video games; Esports; Royal;
SIGGRAPH ‘07, ACM (2007) Shooter; Survival; Third-person
Zhang, Y., Ma, K.-L.: Spatio-temporal extrapolation for
fluid animation. ACM Trans. Graph. 32(6), 183
(2013)
Zhu, B., Lu, W., Cong, M., Kim, B., Fedkiw, R.: A new Definition
grid structure for domain extension. ACM Trans.
Graph. 32(4), 63:1–63:12 (2013) Fortnite ¼ a video game that has two game
modes: a co-op zombie survival mode and a Battle
Royale mode.
Fortnite Friday ¼ a weekly Fortnite tourna-
ment organized by Daniel “Keemstar” Keem and
For Super Smash Bros. Faze Clan founder Ricky Banks.
Fortnite World Cup ¼ an annual e-sports com-
▶ Super Smash Bros.: A Brief History petition based on the video game Fortnite.

Introduction

Formal Methods This entry covers the history of Fortnite with the
focus on the Battle Royale mode. It includes the
▶ Timed Automata for Video Games and story of live streamers who brought popularity to
Interaction the game, the advantage Fortnite had over other
Fortnite: A Brief History 731

Battle Royale games, the start of Fortnite Friday, free to play, making the game accessible to every
and the Fortnite World Cup. The entry concludes gamer in the world.
by discussing the effect Fortnite had on other
games in the same genre and different genres.
Fortnite Fridays

Gaining Popularity Fortnite Friday is a weekly Fortnite tournament

organized by Daniel “Keemstar” Keem and Faze
Fortnite was first released for early access in Clan founder Ricky Banks. (Faze Clan is a pro-
July 2017 with only one game mode: Save the fessional esports and entertainment organization
World, which is the co-op zombie survival founded in 2010 and headquartered in Los
mode. When it launched, it did not gain much Angeles, California.) In this tournament, two F
popularity with few people live streaming teams of two – comprised of either a Twitch or
it. That all changed on September 26, 2017, YouTube streamer or a big-name celebrity –
when they launched their new Battle Royale would compete in a best of three series where
game mode. It brought new players and they played in the same game and competed to
streamers to the game because this game mode be the duo with the most kills for that game. There
catered to a new audience of players who were ten tournaments over 10 weeks, each with a
enjoyed third-person competitive shooters. prize pool of $20,000 (Twin Galaxies 2021). This
After the release of the Battle Royale, more brought attention to the community’s desire for a
players and streamers started to take notice of competitive scene in Fortnite’s Battle Royale
Fortnite. One such streamer was Tyler “Ninja” mode. The tournaments even attracted the
Blevins with his flashy playstyle and skill that attention of the famous rapper Drake who gave a
seemed to be greater than everyone else. He shout-out on Instagram to Brett Squires who was
started to gain popularity not only for himself competing. Forbes and ESPN both took notice as
but also for Fortnite as well. Ninja went on to well writing stories related to the weekly tourna-
become the first streamer to get his own skin in ments (Erzberger 2019).
the game that featured his iconic spiked blue
hair and yellow headband (Rishabh 2020).
Fortnite World Cup

Fortnite’s Cross-Platform Play and Free- In July 2019, Fortnite held it biggest event to date
to-Play beginning on July 26 and lasting through July
28 with four tournaments and a whopping
Fortnite had several advantages over other Battle $30 million prize pool. The tournament venue
Royale games that helped propel its popularity. was Arthur Ashe Stadium where the US Open
When it was first released in 2017, Fortnite was for tennis is hosted. But for those 3 days, the
the first Battle Royale game available on both the stadium was home to thousands of Fortnite fans
PC and the consoles. This gave Fortnite a huge cheering on their favorite Fortnite competitors.
advantage because it attracts not only PC players Competitors had to go through a 10-week qualifi-
but also PlayStation 4 and Xbox One players. cation before they were able to compete in the
Fortnite was later released on mobile devices tournaments.
and Nintendo Switch as well, making it playable Outside the stadium was a miniature Fortnite
on almost any platform. Sony who owns amusement park featuring a zipline, mini-golf
PlayStation refused cross platform play at first, course, a giant pirate ship, and a giant ball to roll
but in 2018 Sony decided to allow their players around in. In between the tournaments, there was
to join the PC and Xbox One players in playing a Marshmello concert and teases of the tenth sea-
Fortnite (DigiZani 2021). Since day 1, Fortnite is son of the game.
732 Foundations of Interaction in the Virtual Reality Medium

Even with all those things going on, the main Erzberger, T.: Friday Fortnite bringing out famous faces
attraction of the event was still the competition. online. ESPN.Com, ESPN. https://www.espn.com/
esports/story/_/id/26928924/friday-fortnite-bringing-
The four tournaments played during the event famous-faces-online (8 June 2019)
were a creative mode showcase, celebrity Pro- Rishabh.: Fortnite: The story of Ninja, from a Classic
Am, duos championship, and a solo champion- Rager to the most-beloved streamer. Sportskeeda APP
ship featuring 100 of the world’s best Fortnite Is the No 1 personalised sports APP Available Today.
Just select your fav teams & players and you are done,
players. The duo and solo championships were Sportskeeda, https://www.sportskeeda.com/esports/
what most people came to watch as they were fortnite-the-story-ninja-from-classic-rager-beloved-
going to show the highest level of play that had streamer (4 July 2020)
ever been seen. A European duo of Emil Twin Galaxies: Who won Friday Fortnite – The winner of
every Friday Fortnite. https://www.twingalaxies.com/
“Nythrox” Bergquist Pederson and David “aqua” feed_details.php/2211/who-won-friday-fortnite-the-
Wang won the 3-million-dollar duo competition; winner-of-every-friday-fortnite. Accessed 4 May 2021
16-year-old Kyle “Bugha” Giersdorf won the Webster, A.: The Fortnite World Cup finals were a Victory
1-million dollar solo competition (Webster 2019). Lap for Epic Games – The Verge. The Verge, The
Verge, https://www.theverge.com/2019/7/29/8934329/
fortnite-world-cup-finals-epic-games-esports-ninja-
tfue-bugha-marshmello (29 July 2019)
Fortnite’s Influence

With Fortnite becoming one of the most popular

games in the world, it is natural for other games to
try to copy its success formula. One popular feature Foundations of Interaction in
in Fortnite is the battle pass system, which is essen- the Virtual Reality Medium
tially a system that rewards players for playing the
game and offering more rewards for players who Danielle Marie Olson1, Elisabeth Ainsley
Sutherland2, Cagri Hakan Zaman3 and
decide to upgrade to the paid version of the battle
D. Fox Harrell4
pass. The free to play model with microtransactions 1
to customize the game has also become a popular Computer Science Artificial Intelligence
Laboratory (CSAIL), Massachusetts Institute of
feature with the success of Fortnite.
Technology (MIT), Cambridge, MA, USA
2
Mediate VR, Cambridge, MA, USA
3
Conclusion MIT CSAIL, Cambridge, MA, USA
4
MIT Comparative Media Studies Program and
Fortnite had a slow start in July 2017 with the CSAIL, Cambridge, MA, USA
Save the World mode, but it quickly gained pop-
ularity with the launch of the Battle Royale mode
in September 2017. With the introduction of Synonyms
Fortnite Friday, the competitive scene was begin-
ning to develop, which led to Fortnite World Cup Artificial intelligence; Conceptual blending;
in July 2019 – the biggest event in Fortnite his- Immersive design; Immersive systems; Interac-
tory. Fortnite will go down as one of the most tive design; Spatial cognition; Virtual reality
influential games in history.

Definitions
References
The medium of virtual reality (VR) can be defined
DigiZani.: How Fortnite became so popular, Digizani, as the unique set of practices and novel
https://www.digizani.com/blogs/news/how-fortnite- affordances that emerge when using VR and is
became-so-popular#:~:text¼Over%20100%20million
%20players%20played,conceived%20a%20completely
distinct from work built for screen-based or other
%20different%20experience. Accessed 4 May 2021 media.
Foundations of Interaction in the Virtual Reality Medium 733

Introduction: VR, a Technology, and a tangible virtual environment, and of virtual pro-
Medium prioception, providing the sense of a tangible
virtual self. VR can include less interactive
VR systems bring together a family of technolo- work, such as 360- film, or highly interactive
gies to create a convincing computer-graphical work, which could include peripherals such as
space around the user. These spaces can range gloves equipped with sensors and treadmills to
from the photorealistic to the abstract and can simulate physical movement.
include a host of objects, interactions, and This section introduces definitions of these
effects. While many VR systems are primarily various technologies, discusses their key aspects,
visual (the focus here), others can involve other and demonstrates examples of how each – indi-
sensory modalities such as haptic or olfactory vidually and in combination – provide unique
feedback. Better understanding VR as a technol- impacts on creating VR systems. F
ogy and as a medium will aid in distinguishing
between media technologies, genres for using Real-Time Computer Graphics
such technologies, and particular works. Tech- Real-time computer graphics are familiar to most
nologies here refers to hardware and software users from media forms such as computer games
that are researched, developed, and used. and animated feature films. However, while ani-
A genre is a style of using some input/output mated films often use pre-rendered computer
capabilities, and a work is an instance of a graphic images (CGI), VR environments and
genre” (Goguen and Harrell 2014). The medium models are rendered as-needed in response to
of VR can then be defined as the unique set of user input such as gaze, position, movement, and
practices and novel affordances that emerge gesture while users explore and interact with the
when using VR. Genres of VR will continue to virtual environment. VR experiences can be built
emerge as conventions are innovated and from 3D models and environments implemented
established. Works created that uses the medium synthetically using CGI software. Alternately, VR
of VR will be distinct from work built for screen- imagery can be captured from the physical world
based or other media. This understanding of VR through techniques such as photogrammetry and
is especially important for understanding its role videogrammetry to build 3D models from photos
as a medium for works of computer-based art and video. Finally, VR environments can be cre-
(Bates 1992) such as videogaming and interac- ated using techniques integrating both synthetic
tive narratives (Jenkins 2004; Thiel 2009). modeling and physical world capture-based
The technology of VR can be defined as any modeling. For example, in Hospital with One
combination of devices and software that pro- Entrance, artist Deniz Tortum uses a laser scan-
duces a sense of virtual exteroception and propri- ning device to capture the physical dimensions of
oception in a user. Exteroception refers to a hospital’s operating room and then imports these
perception of an external environment. Proprio- data into VR software to program interactivity
ception refers to perception of one’s own body (Tortum 2016).
(Fotopoulou 2015).
These technologies most typically include: Key Graphics Technology Considerations
(1) real-time computer graphics, (2) dynamic Computer graphics used in VR require a frame
visual interaction including: a panoramic image rate of up to 75 frames-per-second (fps) in order to
space with positional and orientation tracking, ensure smooth movements and avoid noticeable
(3) sensory elements including stereoscopic judder to the user. Systems that fail to provide
vision, a head-mounted viewing device, and spa- high quality resolution and frame rate cause end
tial audio, and often (4) motor elements including users to perceive judder and may result in motion
peripheral devices such as handheld controllers. sickness. Given these requirements, VR computer
These elements combine to produce a sense of graphics systems must be equipped with powerful
virtual exteroception, providing a sense of a enough graphics processing unit (GPU)
734 Foundations of Interaction in the Virtual Reality Medium

capabilities and have low system latency, or time space and a wide field-of-view, create a visual
between user input and updates in displayed environment for the user that is closely connected
graphics. to natural perceptual activities. Moreover, in VR,
the CGI spatial environment also implies shape,
Key Graphics Design Considerations volume, location, and physics, encouraging users
Real-time graphics provide both constraints and to look under, around, or through elements in
affordances for designers. Latency and judder are space (Smith 2017).
examples of constraints.
System designers must negotiate these con- Key Tracking Technology Considerations
straints, for instance, tethering consumer VR Positional tracking in VR is generally achieved
headsets to powerful GPU-packed computers through sensors which may be internal or external
using cables. These challenges have inspired to the head-mounted display or controller. Some
research and development efforts focused on hardware also uses embedded sensors to track this
designing more efficient hardware to enable kind of motion. Orientation tracking is typically
higher-quality VR experiences on affordable tracked using a combination of accelerometers,
mobile devices with less processing power. gyroscopes, and magnetometers embedded into
Given that VR systems enable the design of the hardware. For positional tracking, laser- or
nuanced interfaces and experiences tailored to camera-based sensors can be used. Each of these
individuals based on their gaze, position, move- technologies has tradeoffs for design: for exam-
ment, and gesture over time, overcoming power ple, camera-based systems can be more accurate,
and latency issues enables more impactful uses of but require processing power and connections
the medium in educational, training, medical, and sufficient to avoid latency and have additional
therapeutic contexts (Chu and Cuervo 2016; Lai privacy considerations.
et al. 2017).
Key Tracking Design Considerations
Orientation and Positional Tracking Positional and orientation tracking activate the
Users can navigate their visual environments visual space outside the user’s immediate field of
more freely in VR than in other screen-based view, allowing users to glance and reach at things
media. This is due in part to the combination of in the periphery of their vision and to lean or
a panoramic virtual environment with orienta- reach towards and away from elements in the
tional and positional tracking. Positional tracking scene. Intentional looking and direct address
and orientation tracking refer to the ability of VR are two techniques for accounting for this in
systems to track user movement in the physical design. For example, consider a VR experience
world, and translate that movement to the virtual in which the spatial imagery around the user
environment. The degrees to which VR systems initially goes pitch black and then lights flicker
support translating user movement from the phys- on a little bit later. The user might react by
ical world to the virtual environment is referred to looking curiously around using and investigating
as the “degrees of freedom” (DOF) of the system. the full panorama image space to see what
Orientational tracking means that the system changed. The user’s investigation of the space
tracks the rotational movement of a viewer’s is motivated and intentional, more active than
head, thus allowing a user to look freely in any receptive. Direct address describes the “two-
direction in the virtual environment. All current way process of a user both seeing and being
VR systems enable orientational tracking. Posi- seen” by other characters within the environment
tional tracking follows the user’s translational and by the virtual environment itself (Sutherland
motion and positions the user in a 3-dimensional 2015). The tracking devices allow the system to
environment. Higher-end VR systems enable change in response to a user’s attention: for
positional tracking. These types of user tracking, example, objects appearing only when the user
combined with the complete panoramic image is gazing elsewhere.
Foundations of Interaction in the Virtual Reality Medium 735

Placement of objects within the environment However, active work is being done to develop
given the constraints of the positional tracking more efficient HMDs that can deliver higher-
system and the tethered display is of key concern quality VR experiences on more efficient and
when designing for HCI in VR. For example, wireless devices. HMDs can also deliver spatial
while activities such as “crouching” or “crawling” audio which simulates natural human localization
in VR games can provide an exciting sense of techniques using sound clues. The widespread use
immersion, they can easily result in user fatigue of consumer speech recognition systems, work in
if repeated too often during the experience. 3D binaural sound reproduction, and spherical
User tracking and spatial design also has microphones are advancing the quality of spatial
emergent effects in multiuser environments. For audio capture and playback (Jarrett 2017).
instance, nSpace is a project exploring collabora- While eye-tracking adds an additional compu-
tive aspects of VR for design tasks. This system tational load to systems, it can enable foveated F
uses a special sensor for hand tracking to visually rendering techniques, which are processes for
represent the user’s hands. This enables interac- reducing the workload on the system by high-
tion with user interface components that exploit quality rendering of the graphics the user is
the 360 visual representation. This enables users looking at and lower-quality rendering of the
to move through virtual environments and provide visuals in the user’s peripheral vision (Guenter
more specific and relevant feedback on objects et al. 2012; Padmanaban et al. 2017).
and instructions to other users using subjective –
in relation to a user’s body (Johnson 1987; Key Stereoscopy, HMD, and Spatial Audio Design
2007) – rather than objective language, such as Considerations
the absolute position of an element on a screen Stereoscopic depth enables VR to convincingly
(Zaman et al. 2015). situate users in CGI spatial environments,
resulting in a sense of a physical relationship to
Stereoscopic Vision, Head-Mounted Displays, objects and characters (Schröter 2014).
and Spatial Audio Tethered HMDs limit users’ range of transla-
The combination of stereoscopy and head- tional motion, but research and releases to the
mounted displays (HMDs) connects VR systems consumer VR market are trending towards
closely with natural visual perception. Stereo- improving phone-based, wireless, all-in-one
scopic vision in VR produces the illusion of HMDs. As barriers to entry for VR HMDs
depth and three-dimensional space and is become lower, HMD design must anticipate and
achieved by displaying parallax angles of images address the “brick in the face problem” which
through dual lenses. These dual images originate results from the opaque quality of the headset.
from different perspectives and slightly overlap, Given that the eyes provide a crucial means of
such that users fixating binocularly on a point will nonverbal communication in social contexts
perceive elements images on the same relative including gaming, eye tracking technology can
coordinates as a single object (Tam et al. 2011). be used as input to achieve better customization.
HMDs must also be equipped to deliver spatial Other VR experiences such as The Enemy, a jour-
audio that is synchronized with the visual experi- nalistic VR artwork by photographer Karim Ben
ence. Finally, many companies and researchers Khelifa addressing global conflict (Kennedy
are working towards enabling eye-tracking in 2016; Lacey 2016), allow for dynamic changes
consumer VR systems. to the experience’s narrative (e.g., events and dia-
logue) and staging (e.g., lighting and mise en
Key Stereoscopy, HMD, and Spatial Audio scène of the virtual environment) based on the
Technology Considerations users’ embodied input to the system using artifi-
As discussed above, most consumer VR headsets cial intelligence. For example, features including
are tethered to powerful GPU-packed computers users’ translational motion, head motion, direc-
with high power consumption and thermal output. tional orientation, and proximity to each other
736 Foundations of Interaction in the Virtual Reality Medium

and to virtual characters are used as proxies to human-computer interaction involving touch.
track users’ attention, nervousness, and biases These peripherals range from controllers or sim-
(e.g., asymmetries of attention and nervousness). ple touch screen devices with button-based con-
This real-time user tracking triggers feedback by troller paradigms, to a joystick, remote, or mouse,
the system including changes to behaviors of the as well as devices which enable gestural interac-
nonplayer characters, appearance of the users’ tion ranging from the 1990s “data glove”
avatar, voice-overs, and the virtual cloud cover (Premaratne 2014), treadmills, to Leap Motion,
(thereby impacting the lighting). Eye-tracking Microsoft Kinect, or the Myo armband. Periph-
capabilities can also enhance the evocative poten- erals can also include biometric devices such as
tial of this genre of VR experiences. In social VR, wireless wristbands that monitor real-time physi-
eye-tracking technology has been used in con- ological signals for affective computing inter-
junction with computer vision algorithms which faces, EEG-based biometrics for brain-computer
align and blend a 3D face model with a camera’s interfaces and more. Sensory-output-based
video stream of the user (Frueh et al. 2017). peripherals may also include vibrating floors and
These strides demonstrate how artificial intelli- mats, electrical muscle stimulation for muscle-
gence will help to enhance connection and inter- computer interfaces, and olfactory output devices.
action in multiple-user VR scenarios and third-
party gameplay viewers. Key Peripheral Device Design Considerations
Furthermore, spatial audio can be a powerful In combination with the technologies presented in
tool to present sounds from any direction, control the previous sections, peripheral devices present
user attention, give users cues on where to look, additional design opportunities and challenges.
and provide an immersive VR experience (Grau For example, while handheld peripherals and
2003). Spatial audio in VR has the potential to be gloves can enable users to manipulate virtual
applied as a powerful design tool for evaluating objects, developers must account for the effect of
planned architectural designs in combination with users seeing their own hands. Systems that enable
soundscapes prior to physical construction standing in combinations with tracked handheld
(Echevarria Sanchez et al. 2017). Tools such as peripherals afford the user the ability to be able to
Mediate VR which enable the evaluation of space reach into virtual environments and do things,
and soundscape designs through remote, asyn- requiring nuanced handling of embodied input.
chronous, voice-driven collaboration. Forward- By leveraging biometric devices that provide
looking designs of HMDs must be able to stream real-time physiological data or manipulate the
and play spatial audio in real-time, with tools such sensations of the user’s body, an additional layer
as TheWaveVR introducing the concept of social of immersion is added to the experience. For
platforms which host immersive VR music example, recent research has used muscle stimu-
concerts. lation through gentle electrical impulses as a new
approach to rendering the haptics which afford the
Peripheral Devices repulsion of a wall or gravity pulling down the
VR peripheral devices enable additional forms of weight of a heavy box (Lopes et al. 2017).
user input, output, and interaction in the
immersive environment. Although not the focus
of this article, it is important to consider key types Creating VR Experiences
of such devices.
Introduction
Key Peripheral Device Technologies This section presents two theoretical approaches
Peripheral device input can include the positional that can underpin and motivate the design of
tracking of hands and a variety of controller-like VR experiences. The first, constraints and
inputs. Peripheral device technologies includes affordances, is crucial to organize approaches to
haptic technologies, which describes a form of the many new interaction paradigms and sheer
Foundations of Interaction in the Virtual Reality Medium 737

variety of design choices. The second, concept- transmitted to the users’ fingertips through con-
ual metaphors and blends, suggests a way to bring trollers or point and click devices. Gestural
metaphorical thinking into the virtual environ- affordances provide cues which invite users for
ment in a way that can be used both for creative gestural interaction, such as cues to virtually touch
purposes and for efficient design. or pick up objects.

Constraints and Affordances in VR Conceptual Metaphors and Blends in VR

The technologies that enable the possibilities of VR experiences and tools can be thought of as a
immersive VR environments also present both sen- performance between users and their virtual envi-
sory constraints and physical/motor constraints. ronment (Laurel 2014). One potentially effective
Sensory constraints are limitations in being able way to design such mediation is to build interface
to provide visual information to the user. This may metaphors, a set of visuals which build on existing F
result in a lack of framing around the content of the notions, actions, and phantasms (Harrell 2013) to
experience, or a lack of self-representation of the facilitate meaningful and intuitive interactions in
user’s body in space. Many VR systems focus on VR. Interface metaphors are in turn grounded in
visual and auditory constraints. Physical/motor conceptual metaphors, which are mappings
constraints, which are limitations in the physical between ideas (including mental images) that are
capabilities of the hardware design, setup, of ability grounded in sensory-motor action (Lakoff and
for users to move in the physical world. This may Johnson 1980). For example, forefingers and closed
result in a lack of freedom in users’ movements and hands are typically associated with pointing and
interactions with their physical surroundings, grabbing; hence, selecting virtual objects may be
which may result in safety risks. achieved by virtually pointing and gathering virtual
The psychologist James Gibson provided a use- objects may be accomplished by virtually grabbing.
ful term when discussing what technologies enable They key to designing strong interface meta-
or constrain: he defines an affordance as what the phors is keeping them intuitive to the user (Gallo-
environment offers or furnishes to the user (Gibson way 2012). Users should be able to infer coherent
1977, 1979). In these terms, VR is a unique actions based on these metaphors unless they are
medium with the potential for novel interaction explicitly designed for creative effects (for
mechanisms given the particular constraints and instance, an interaction mechanism that is difficult
affordances of the technology. In VR, the designer to perform can be useful to represent a difficult
has a close hand in creating and varying the per- action in a VR game). This can be very challeng-
ceived affordances presented to individuals within ing in such an immersive environment, given that
the experience (Norman 1999). the user can do and perceive many things. Antic-
The design choices of developers, in conjunc- ipating the user’s inferences is important, and by
tion with the aforementioned technological ele- taking an environmental approach to design one
ments of the medium, also shape the sensory can begin to think this way. Likewise, for creative
affordances of VR systems. Especially of projects, anticipating users’ expectations can
note are their visual, tangible, and gestural allow for the creation of surprising and meaning-
affordances. Visual affordances are visual cues ful narratives.
which invite users for action in the form of visual
guides or floating user interface (UI) elements. Conclusion
For example, in any given VR experience, users Virtual reality is an important set of technologies
can have either no body, an object instead of a and media. This is because, as the name indicates,
body, a partial body, their own body, another its technologies seek to make use of many of the
person’s body, or even multiple bodies. Tangible modes of interaction that humans have in the
affordances are provided by peripherals which “reality” of the physical world while enabling
enable haptic interaction and cues in the environ- the new “synthetic” possibilities of the virtual. It
ment, such as different vibration patterns being has also been said that objects in the physical
738 Foundations of Interaction in the Virtual Reality Medium

“real” world can lose their reality due to layers of Jenkins, H.: Game design as narrative architecture.
mediation (Baudrillard 1995) and experiences in In: Wardrip-Fruin, N., Harrigan, P. (eds.) First Person:
New Media as Story, Performance, and Game. MIT
virtual environments can have real physical world Press, Cambridge, MA (2004)
impacts ranging from bullying to discrimination Johnson, M.: The Body in the Mind: The Bodily Basis of
(Harrell and Lim 2017). In light of these complex- Meaning, Imagination, and Reason. University of
ities, VR’s technologies and their constraints and Chicago Press, Chicago (1987)
Johnson, M.: The Meaning of the Body: Aesthetics of
affordances must be account for while creatively Human Understanding. University of Chicago Press,
making, subverting, and adapting them for Chicago (2007)
impactful and powerful experiences. Kennedy, R.: Meeting ‘the Other’ Face to Face.
The New York Times. https://www.nytimes.com/
2016/10/30/arts/design/meeting-the-enemy-face-to-
face-through-virtual-reality.html (2016)
Lacey, S.: Face to face with “The enemy”. http://news.mit.
References edu/2016/face-face-with-the-enemy-karim-ben-
khelifa-1202 (2016)
Bates, J.: Virtual reality, art, and entertainment. Teleop. Lai, Z., Hu, Y.C., Cui, Y., Sun, L., Dai, N.: Furion: engi-
Virt. Environ. 1(1), 133–138 (1992) neering high-quality immersive virtual reality on
Baudrillard, J.: Simulacra and Simulation, 1st edition, 17th today’s mobile devices. In: Proceedings of the 23rd
printing edition, Ann Arbor. Michigan Press (1995) Annual International Conference on Mobile Comput-
Chu, D., Cuervo, E.: FlashBack: immersive virtual reality ing and Networking – MobiCom ’17, pp. 409–421
on mobile devices via rendering memoization. Pro- (2017)
ceedings of the 14th Annual International Conference Lakoff, G., Johnson, M.: Conceptual metaphor in everyday
on Mobile Systems, Applications, and Services. 20(4), language. J. Philos. 77(8), 453–486 (1980)
291–303 (2016) Laurel, B.: Computers as Theatre. Pearson Education,
Echevarria Sanchez, G.M., Van Renterghem, T., Sun, K., Crawfordsville (2014)
De Coensel, B., Botteldooren, D.: Using virtual reality Lopes, P., You, S., Cheng, L.-P., Marwecki, S., Baudisch, P.:
for assessing the role of noise in the audio-visual design Providing haptics to walls & heavy objects in virtual
of an urban public space. Landsc. Urban Plan. 167, reality by means of electrical muscle stimulation. In: Pro-
98–107 (2017). https://doi.org/10.1016/j.landurbplan. ceedings of the 2017 CHI Conference on Human Factors
2017.05.018 in Computing Systems – CHI ’17, pp. 1471–1482 (2017)
Fotopoulou, A.: The virtual bodily self: mentalisation of Norman, D.A.: Affordance, conventions, and design. Mag.
the body as revealed in anosognosia for hemiplegia. Interact. 6(3), 38–43 (1999)
Conscious. Cogn. 33, 500–510 (2015) Padmanaban, N., Konrad, R., Stramer, T., Cooper, E.A.,
Frueh, C., Sud, A., Kwatra, V.: Headset removal for virtual Wetzstein, G.: Optimizing virtual reality for all users
and mixed reality. In: Proceedings of SIGGRAPH ’17 through gaze-contingent and adaptive focus displays.
(2017) Proc. Natl. Acad. Sci. 114(9), 2183–2188 (2017)
Galloway, A.R.: The Interface Effect. Polity Press, Premaratne, P.: Historical development of hand gesture
Cambridge (2012) recognition. In: Human Computer Interaction Using
Gibson, J.J.: The Theory of Affordances. Lawrence Hand Gestures Cognitive Science and Technology.
Erlbaum Associates, Hillsdale (1977) Springer, Singapore (2014). https://doi.org/10.1007/
Gibson, J.J.: The Ecological Approach to Visual Percep- 978-981-4585-69-9_2
tion. Houghton Mifflin, Boston (1979) Schröter, J.: 3D: History, Theory, and Aesthetics of the
Goguen, J.A., Harrell, D.F.: Foundations for active multi- Transplane Image International Texts in Critical
media narrative. Igarss 2014. 1(1), 1–5 (2014) Media Aesthetics. Bloomsbury, New York (2014)
Grau, O.: Virtual Art: From Illusion to Immersion. MIT Smith, L.: 360 photos vs. VR. Bevel. https://bevel.space/
Press, Cambridge (2003) news/2017/8/1/is-it-virtual-reality-360-photos-vs-vr
Guenter, B., Finch, M., Drucker, S., Tan, D., Snyder, J.: (2017)
Foveated 3D graphics. ACM Trans. Graph. (TOG). Sutherland, Elisabeth Ainsley.: Staged Empathy: Empathy
31(6), 164 (2012) and Visual Perception in Virtual Reality Systems.
Harrell, D.F.: Phantasmal Media: An Approach to Imagi- DSpace@MIT. Massachusetts Institute of Technology
nation, Computation, and Expression. MIT Press, (2015)
Cambridge (2013) Tam, W.J., Speranza, F., Yano, S., Shimono, K., Ono, H.:
Harrell, D.F., Lim, C.: Reimagining the avatar dream: Stereoscopic 3D-TV: visual comfort. IEEE Trans.
modeling social identity in digital media. Commun. Broadcast. 57(2 PART 2), 335–346 (2011)
ACM. 60(7), 50–61 (2017) Thiel, T.: Where Stones Can Speak: Dramatic Encounters
Jarrett, D.P.: Theory and Applications of Spherical Micro- in Interactive 3-D Virtual Reality. Third Person.
phone Array Processing. Springer, Cham (2017) Authoring and Exploring Vast Narratives. P. Harrigan
Fused Filament Fabrication (FFF) 739

and N. Wardrip-Fruin, (eds.). Massachusetts Institute of

Technology, Cambridge, Massachusetts (2007) Free-to-Play
Tortum, Halil Deniz.: Embodied montage: reconsidering
immediacy in virtual reality. DSpace@MIT. Massachu-
setts Institute of Technology (2016) ▶ Counter-Strike Global Offensive, an Analysis
Zaman, C.H., Yakhina, A., Casalegno, F.: nRoom:
An Immersive Virtual Environment for Collaborative
Spatial Design. CHIuXiD ’15, pp. 10–17 (2015)

Fun Learning

Four-Legged Robot ▶ Educational Game Abzû and the Lens of Fun

Learning
▶ Virtual Reality and Robotics F

Funware
Fourth Industrial Revolution
▶ Gamification
▶ Design Framework for Learning to Support
Industry 4.0

Fused Filament Fabrication

Foveated Rendering (FFF)

▶ Eye Tracking in Virtual Reality ▶ Open Source 3D Printing, History of

G

GA Genetic Algorithm Game Based Learning

▶ Genetic Algorithm (GA)-Based NPC Making ▶ ROP-Skill System: Model in Serious Games for
Universities

Gachas
Game Bot
▶ Loot Boxes: Gambling-Like Mechanics in
▶ Detecting and Preventing Online Game Bots in
Video Games
MMORPGs

Game AI Game Bot Detection on

Massive Multiplayer Online
▶ Machine Learning for Computer Games Role-Playing Games
▶ RTS AI Problems and Techniques (MMORPGs) Systems
▶ StarCraft Bots and Competitions
Andrea Lanzi
Università degli Studi di Milano, Milan, Italy

Game Assets
Definition
▶ 3D Game Asset Generation of Historical Archi-
Massive multiplayer online role-playing games
tecture Through Photogrammetry
(MMORPGs) clients (the players) connect to
online servers (the virtual worlds). The servers
constantly update the client software with the
sights, sounds, and happenings in proximity to
Game Balancing the player. Attack against MMORPGs is
performed by automatic program, bot, that plays
▶ Quality Assurance-Artificial Intelligence automatically and performs cheating actions
© Springer Nature Switzerland AG 2024
N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
742 Game Bot Detection on Massive Multiplayer Online Role-Playing Games (MMORPGs) Systems

against players’ moves. In such a context for bot game bot activities (Bethea et al. 2010). The
detection, we identify all types of detection algo- most fundamental point for solving the problem
rithms that are able to recognize the game players is to have a strong mechanism to identify game
that are not human users by using several statisti- bots. Several detection techniques have been pro-
cal features (e.g., machine learning, statistics on posed by game vendors. In the following, we
log analysis, etc.). For example one algorithm describe the most practical ones used in the real
can perform a comprehensive statistical analysis online games.
of user behaviors defined in game activity One of the proposed techniques is to focus on
logs and then set up several threshold levels that the detection based on repetitive activities of game
can be used to distinguish between game bots and bots, which are typically found in game log activ-
human users. ity. Such technique shows that game bots fre-
quently repeat certain activities that are different
from human user ones. Consequently, using this
Overview analysis, researchers proposed a new bot detection
framework that uses a metric called “self-similarity
A game bot is a program that plays games auto- measure.” Self-similarity is used to show the sim-
matically instead of human users, typically used ilarity of user actions as a function of the time lag.
for game cheating. Game cheating identifies “Any This method is designed for finding repetitive pat-
behavior that a player uses to gain an advantage terns, especially periodic patterns of the series of
over his peer players or achieve a target in an actions and their frequency. The method considers
on-line game is cheating if, according to the several actions such as moving pattern to provide a
game rules or at the discretion of the game oper- strong self-similarity that is able to resist to the
ator, the advantage or the target is one that he is changes of target games in their following updates.
not supposed to have achieved.” Cheating can be Another strategy is to construct a model of proper
performed at several levels. For example, it can be client behavior against which actual client behav-
done by exploiting a bug in the software or pro- iors are compared (e.g., system call model). More
tocol or by exploiting vulnerabilities of various precisely remote system calls are compared to a
people involved in operating or playing online control flow model generated from the binary code
games. When we talk about game bot, we con- during its training execution.
sider the cheating performed by an automatic pro- A different approach to protecting against cli-
gram that can play without break; consequently it ent misbehavior in client-server settings is to
can accumulate money, items, and score much ensure that clients manage no privilege state that
faster than normal human players. could affect the server; this is commonplace for
In the recent years, the use of game bots has games today. This approach is for the client to
become one of the most serious security threats to simply forward all unseen user inputs to the
MMORPGs. In fact a game results in significant server, where a trusted copy of the client-side
damages in terms of the economic game cost. computation monitors these inputs directly; this
For example, Castronova (2007) studied the mon- is implemented in the Ripley system. This system
etary damage caused by game bots in World replicates a copy of the client-side application on
of Warcraft, an MMORPG developed by Bliz- the trusted server, and any event is sent to the
zard, by considering several aspects including replica of the client for execution. The system
customer service cost, a technical cost for bot monitors results of the computation, both as com-
enforcement, etc. The indirect cost of game bots puted on the client side and on the server side
was approximately 18 million USD per year. To using the replica of the client-side code. Any
address these issues, game vendor invests signif- discrepancy is flagged as a potential violation of
icant efforts to design solutions for mitigating software integrity.
Game Bot Detection on Massive Multiplayer Online Role-Playing Games (MMORPGs) Systems 743

Another common approach to defeat a variety patterns between bots and human users. Kang
of cheats against the game bots which involves et al. (2013) also proposed a game bot detection
augmenting the client-side computer with moni- method based on the players’ network features.
toring functionality to perform cheat detection is Such methods can detect bots with high accuracy.
PunkBuster. Such approach requires consider- In addition to individual bot detection, researchers
ation of how to defend this functionality from also design mechanism to detect bot groups.
tampering, and some commercial examples have Chung et al. (2013) proposed a method that cluster
met with resistance from the user community features by behavioral similarities using the
(e.g., World of Warcraft’s Warden). More in K-means clustering algorithm and then detecting
details, PunkBuster is searching in the local mem- bots in each group by using support vector
ory of the client a certain pattern that can be a machine (SVM), and Mitterhofer et al. (2009),
symptom of game cheating. PunkBuster based its Chen et al. (2009), and Kesteren et al. (2009)
own efficacy on pre-built database that contains proposed similar methods with general features
several cheating behaviors in the form of patterns such as the moving paths of characters, respec- G
to search for in memory. tively. Those works use the characteristic of game
bots to move with fixed routes set up by bot pro-
grams. Such methods can be applied to most
History MMORPGs. More recently new detection tech-
niques have been proposed. For example, in the
Several works on game bot detection using sev- paper (Lee et al. 2011), the authors extend their
eral detection approaches have been published. works into a more generalized model; while their
Ahmad et al. (2009) presented a first study, in approaches (Mitterhofer et al. 2009; Chen et al.
2009, by evaluating the performance of various 2009; Kesteren et al. 2009) simply used the single
classification algorithms for bot detection in the feature of moving path, they build a generic
game called “EverQuest II.” They introduced the framework with several features by designing a
terms “gatherers,” “bankers,” “dealers,” and self-similarity algorithm to effectively measure
“marketers” for categorizing bots with several bots’ activity patterns, which was previously
characteristics such as demographic data, charac- used as a means of analyzing network traffic
ters’ sequential activities, statistical properties, (Crovella and Bestavros 1997) or developing
virtual item transactions, and network centrality intrusion detection systems (Kwon et al. 2011).
measures. At the same historical time, in 2008, Such method is significantly robust to changes in
Thawonmas et al. (2008) also presented an early the configuration settings of bot programs com-
study that tried to detect game bots using bots’ pared with existing approaches (e.g., (Mitterhofer
behaviors such as repeating multiple times the et al. 2009; Chen et al. 2009; Kesteren et al.
same activities than normal users. Their detection 2009)) because the method focuses on all activi-
rules were based on simple threshold values. ties and it represents the state of the art of game
Bethea et al. (2010) presented a defense technique bot detection techniques used nowadays.
based on symbolic execution, used for analyzing
the client output and determining whether that
output could have been produced by a valid Conclusion
game client. Their proposed technique cannot
detect cheats that are permitted by game clients Game bot detection has changed considerably by
that do not change their behaviors as seen at the the first introduction of the online game technol-
server level. At the same time on the network side, ogy. Several techniques for cheating have been
Kang et al. (2012) proposed a bot detection mech- addressed, and new attack techniques have been
anism based on the differences in communication invented. The business around the online game
744 Game Control

has grown a lot, and the attackers are really focus-

ing on such context since it is right now a new Game Control
business model that attracts cyber criminals that
are active in Internet context. ▶ Gaming Control Using BCI

References
Game Design
Ahmad, M.A., Keegan, B., Srivastava, J., Williams, D.,
Contractor, N.: Mining for Gold Farmers: automatic
detection of deviant players in MMOGS. In: Computa- ▶ Domain-Specific Choices Affecting Design
tional science and engineering international confer- Effort in Gamification
ence, vol. 4, pp. 340–345 (2009) ▶ Protection Korona: A Game Design on
Bethea, D., Cochran, R.A., Reiter, M.K.: Server-side ver-
ification of client behavior in online games. In: Pro-
Covid-19
ceedings of the 17th network and distributed system ▶ Underground Design of Kaizo Games
security symposium (2010)
Castronova: Effects of botting on world of warcraft.
http://virtuallyblind.com/files/mdy/blizzardmsjexhibit
7.pdf (2007)
Chen, K.-T., Liao, A., Pao, H.-K.K., Chu, H.-H.: Game
bot detection based on Avatar Trajectory. In: Entertainment Game Design and Emotions:
computing ICEC 2008, vol. 5309, pp. 94–105 (2009). [7] Analysis Models
Chung, Y., yong Park, C., ri Kim, N., Cho, H., Yoon, T.,
Lee, H., Lee, J.-H.: Game bot detection approach based
on behavior analysis and consideration of various play Roberto Dillon
styles. J ETRI. 35(6), 1058–1067 (2013) James Cook University, Singapore, Singapore
Crovella, M.E., Bestavros, A.: Self-similarity in world
wide web traffic: evidence and possible causes. IEEE/
ACM Trans Networking. 5(6), 835–846 (1997)
Kang, A.R., Kim, H.K., Woo, J.: Chatting pattern based Synonyms
game bot detection: do they talk like us. In: KSII
transactions on internet and information systems, Engagement; Immersion
vol. 6, no. 11, pp. 2866–2879 (2012)
Kang, A.R., Woo, J., Park, J., Kim, H.K.: Online game bot
detection based on party-play log analysis. In: Com-
puters and mathematics with applications, vol. 65, Definitions
no. 9, pp. 1384–1395 (2013)
Kwon, H., Kim, T., Yu, S.J., Kim, H.K.: Self-similarity
based lightweight intrusion detection method for cloud
Grinding The process of engaging in repetitive
computing. In: Intelligent information and database sys- tasks.
tems, pp. 353–362. Springer, Berlin/Heidelberg (2011) Farming Performing repetitive actions to gain
Lee, E. et al.: You are a game bot! Uncovering game bots in experience, points, or some other
MMORPGs via self-similarity in the wild. Published in
Network and distributed system security symposium
form of in-game currency.
(NDSS) (2016)
Mitterhofer, S., Kruegel, C., Kirda, E., Platzer, C.: Server- Introduction
side bot detection in massively multiplayer online
games. Secur Priv IEEE. 7(3), 29–36 (2009)
Thawonmas, R., Kashifuji, Y., Chen K.-T.: Detection
Evoking a complex emotional response in players
of MMORPG bots based on behavior analysis. In: is a characteristic trait of successful interactive
Advances in computer entertainment technology entertainment, and different models have been
conference, pp. 91–94 (2008) proposed to help designers in creating and analyz-
van Kesteren, M., Langevoort, J., Grootjen, F.: A step in
the right detecting: bot detection in MMORPGs using
ing compelling emotional experiences. This entry
movement analysis. In: The 21st Benelux conference introduces three well-known approaches: the
on artificial intelligence (2009) “Four Fun Keys,” the “MDA Framework”
Game Design and Emotions: Analysis Models 745

Game Design and

Emotions:
Analysis Models,
Fig. 1 Player experience
under the lenses of the Four
Fun Keys (Lazzaro 2004)

(including the “8 Types of Fun” model), and the • People Fun: relies on social interactions to
“AGE Framework” (including the “6–11 make players bond in and outside of the
Framework”). game. Here friendship and relatedness become
central to the playing experience.
The Four Fun Keys
Introduced by psychologist and player experience To be successful, games should then focus on
expert Nicole Lazzaro in the whitepaper “Why We one or more of these key types of fun in order to
Play Games” (Lazzaro 2004) and later in (Lazzaro deliver an emotionally rich experience and engage
2009), this model identifies four main types of fun players to the fullest.
with different characteristics which are able to The Four Fun Keys are summarized in Fig. 1.
deeply engage players by relying on different
sets of emotions. In particular: The MDA Framework and the 8 Types of Fun
Proposed by game designers and scholars Robin
• Hard Fun: relates to frustration and pride. It Hunicke, Marc Leblanc, and Robert Zubek in
involves the act of mastering increasingly dif- (Hunicke et al. 2004), the MDA (Mechanics,
ficult challenges. Dynamics, Aesthetics) was the first serious
• Easy Fun: relates to curiosity, surprise, and attempt to discuss games in a more formal and
awe. It engages players thanks to visually and rigorous approach. Central to the MDA is the idea
content rich environments able to stimulate of a game as an artifact whose consumption can
their imagination. be broken down into three separate components
• Serious Fun: relates to excitement and relax- (Fig. 2):
ation. It engages players by providing meaning These, in turn, have design counterparts as
and offering a purpose for the overall shown in Fig. 3 where each abstraction layer is
experience. defined as follows:
746 Game Design and Emotions: Analysis Models

Game Design and Emotions: Analysis Models,

Fig. 2 Consumption of games according to the MDA
framework

Game Design and Emotions: Analysis Models,

Fig. 3 Design counterparts for the elements identified in
Fig. 2

• Mechanics: the particular components of the

game, at the level of data representation and
algorithms
• Dynamics: the run-time behavior of the
mechanics acting on player inputs and each
other’s outputs over time
• Aesthetics: the desirable emotional responses Game Design and Emotions: Analysis Models,
evoked in players when they interact with the Fig. 4 Schematic representation of a generic game
game systems according to the AGE framework

Aesthetics, in particular, is discussed further in • Submission: game as pastime. It may involve

a dedicated taxonomy called the “8 Kinds of Fun.” grinding or farming and can be used to balance
These include: “Challenge” to change the overall game pacing.

• Sensation: game as sense pleasure. This can be For example, a game like “The Sims” (see
created by visuals, soundscape, and proper pacing. https://en.wikipedia.org/wiki/The_Sims) can be
• Fantasy: game as make-believe. This is about discussed in terms of discovery, fantasy, expres-
empowering players and offering new sion, and narrative, while a game based on the
experiences. “Final Fantasy” (see https://en.wikipedia.org/
• Narrative: game as drama. Storytelling helps wiki/Final_Fantasy) franchise would be most
giving a sense of purpose to the whole likely centered on fantasy, narrative, expression,
experience. discovery, challenge, and submission.
• Challenge: game as obstacle course. New
challenges, finely tuned with players’ own The AGE and 6–11 Frameworks
skills, will easily keep them engaged. The 6–11 Framework was first proposed by game
• Fellowship: game as social framework. design Professor Roberto Dillon in the book On
Playing with friends is often more engaging the Way to Fun (Dillon 2010) and then formally
than playing alone. integrated into a MDA-inspired model named
• Discovery: game as uncharted territory. This is AGE (Actions, Gameplay, Experience) in
a fundamental trait of adventure games. (Dillon 2014).
• Expression: game as self-discovery. Typical The model describes a game as three main sys-
of sandbox games where players are free to tems (see Fig. 4) interconnected to each other via
experiment as they please. in-game rules and goals and defined as follows:
Game Design and Emotions: Analysis Models 747

Game Design and Emotions: Analysis Models, Fig. 5 AGE Framework analysis for the 1981 Konami arcade game
Frogger (see https://en.wikipedia.org/wiki/Frogger)

• Actions: the core, atomic actions a player can revenge, competition, collecting, communication,
perform in a game, usually described in terms and color appreciation.
of verbs. Examples are moving, jumping, The underlying idea of the model is that a
kicking a ball, punching, shooting, taking subset from the emotional palette outlined
cover, etc. above can be triggered by the game and then
• Gameplay: the resulting play that players can interact with other elements as the game
achieve by using and combining the available progresses, engaging and motivating players
actions according to a predefined set of rules. throughout their playing sessions. Once the expe-
These can be either higher-level concepts or rience is outlined, one or more of its components
verbs, for example, fighting, race-to-an-end, can then be linked to the gameplay for analysis and
territorial acquisition, etc. discussion purposes, allowing designers to
• Experience: the emotional experience that formalize their ideas and overall vision (see
engages players during the game while trying Fig. 5, for an example).
to reach certain goals, overcoming obstacles, Different game genres may emphasize a differ-
or solving problems. ent set of emotions, and the model can be applied
to serious games as well (Dillon 2013).
The experience is then exemplified, thanks to
the 6–11 Framework, which comprises 6 basic
emotions and 11 instinctive behaviors such as Cross-References
Anger, happiness (or joy), fear, sadness, pride,
excitement, survival, curiosity, self-identification, ▶ Emotion-Based 3D CG Character Behaviors
protection (or caring for), greed, aggressiveness, ▶ Game Thinking X Game Design Thinking
748 Game Design Evaluating Using Machine Learning

▶ Psychology in Game Design

▶ Semiotics of Computer Games Game Development
▶ Videogame Engagement: Psychological Leadership Tips
Frameworks
Paulo Zaffari
Hoplon Infotainment S.A., Florianopolis, Brazil
References

Dillon, R.: On the Way to Fun. AK Peters, Natick (2010) Synonyms

Dillon, R.: Serious games and fun: An analysis. Int.
J. Innov. Res. Dev. 2(5), 1046–1063 (2013.) ISSN:
2278-0211 Administration; Conduction; Direction; Management
Dillon, R.: Towards the definition of a framework and
grammar for game analysis and design. Int J Comput
Info Technol. 3(2) (2014). Available online: https://
researchonline.jcu.edu.au/32010/1/Dillon_IJCIT_V3- Definition
2.pdf
Hunicke, R., Leblanc, M., Zubek, R.: MDA: A formal
approach to game design and game research. In: Leadership is the craft of inspiring and guiding
Proceedings of the Challenges in Game AI Workshop, people to achieve an objective. In game develop-
19th National Conference on Artificial Intelligence, ment, this objective is completing a game within
San Jose (2004). Available online: https://www. time using the allocated budget while presenting
researchgate.net/publication/228884866_MDA_A_
Formal_Approach_to_Game_Design_and_Game_ great quality.
Research
Lazzaro, N.: Why We Play Games, XEO Design. (2004).
Available online: http://xeodesign.com/xeodesign_
whyweplaygames.pdf Introduction
Lazzaro, N.: Understand emotions. In: Bateman, C. (ed.)
Beyond Game Design: Nine Steps toward Creating
Video game development has deeply technical
Better Videogames. Charles River Media, Boston
(2009) details inherently related to it. The most basic of
them is the hardware on which the game will run.
It will shape, in part, the software and often artistic
assets.
There are also more subjective points related to
Game Design Evaluating it like: how fun is it, how beautiful are its graphics,
Using Machine Learning how good is its music or sound, and how good is
the composition of all the above items. The game
▶ Automated Game Design Testing Using story and the many subtleties of the craft can also
Machine Learning be summed to these elements.
There is still one factor that is decisive for the
success of failure of the vast majority of games:
the people who make those games. While many
games had great ideas and teams behind them,
capable of producing all a great game needs,
Game Development many of them still failed to deliver as a good
product.
▶ Game Thinking X Game Design Thinking There are certainly many explanations for why
▶ Strategies for Design and Development of Seri- they failed. When hearing such teams, they will
ous Games: Indian Perspective often refer to the lack of a common vision, com-
▶ Underground Design of Kaizo Games munication problems, micromanagement, and
▶ Virtual Reality Game Engines other problems which are clearly not related to
Game Development Leadership Tips 749

the game itself (DeMarco and Lister 1999). This Loving the people they command is a value
category of problems is game production related. some successful leaders have. More than a prac-
Among the many things needed for a game tice, this value guides leaders into intuitively relat-
production process to succeed, leadership is a ing well to many people as well as shaping
major facilitator. In this text, a few leadership policies and actions that foster a healthy work
tips will be presented in order to give a head environment.
start to any leader working in the game industry Finally, it’s not without a reason that love is the
or person seeking education on the topic. Those central value within religions. It had deep appeal
tips are pieces and bits of information and ideas for the human psyche, and it’s a strong element
I came up with during my 10 years of experience that leaders should keep in mind.
in leadership positions in game development.

Tip 2: People
Tip 1: Love G
A very common mistake committed by new team
The love in question is much similar to the one leaders, especially those from an engineering
parents, more often than not, have for their chil- background, is not figuring out their work instru-
dren: They want their children to grow and to ments changed. Someone working in the game
achieve their dreams. They will provide whatever industry will often be a person whose working
subsidies on their reach. tools are programming languages, digital content
More reason for this the similarity with paren- generation tools, source control systems, and
tal love being a working technique and value for IDEs. Once promoted to a leadership position,
leadership is the psychological mechanism of this can change overnight: The working instru-
transference and countertransference. Leaders ments of a leader are the people.
will be in a position where their approval to their This change has often deep implications:
subordinates will effectively affect (Stone et al. Where once a person’s job was to implement
2000), if not their chances of survival (Stone et al. some feature or create content, the leader is now
2000) or quality of life, at least their work life. responsible for getting some other person to do
This has a strong similarity to the relationship so. This requires a fundamental change in the
between parents and their children. kinds of skill used and sometimes mentality.
The people working under the leaders will Instead of answering how something should be
often, therefore, try replicating the relationship done, a more appropriate question for a leader to
they had with their parents with their leaders, answer is: Who would be the best person or peo-
due to the transference mechanism. This means ple to do it at this time? Answering this question
their expectations, at least in a subconscious level, requires the leader to understand the talents of
will be similar to those they had with their parents. each individual on his or her team. The leader
Sometimes their relationships are or were trou- should be able to answer: “What is a person
bled, causing some degree of hostility toward the good at?”
leader. Often those relationships were based on Still, answering properly who is the best person
love and trust. to do a given task takes more than just understand-
Leaders acknowledging this should be pre- ing the skills of each individual. Sometimes even
pared to gracefully handle hostility and manage highly skilled team members will not perform
the expectations of their subordinates. The hostil- well on assigned tasks.
ity handling may have positive impacts on the One of the common causes for it is motivation,
person often going beyond the work environment, or rather the lack of it. Determining what moti-
similar to the effects of a therapy session. Once the vates a person is, therefore, also a key factor for
leader gets through with it, which may not be being a good leader. While motivation factors can
always possible, strong bonds are often formed. vary from person to person, there are some which
750 Game Development Leadership Tips

are commonly reported and, for this reason, can be look bad in front of the leader or even do it in
addressed proactively by the leader. attempt to become the boss (Stone et al. 2000).
One such motivation element exists when This is often a problem, as it may lead into grow-
someone doesn’t understand the goal of a given ing tension over an accumulated set of small
task; this person will often be unable to evaluate things.
how good a solution is or, at times, even if the Worse yet, what may feel like a small problem
solution is correct. While some individuals will to some of the team members can, in fact, be
ask for the information required to clarify what perceived as something big for the leader. Also,
they need, a good leader cannot assume this to be possibly even more worrying is the opposite case:
the case. Proactively ensuring people have when something small for the leader feels big for
enough context is a crucial part of the job. the team member. As there is an undeniable power
Another factor is that one has to work with relationship, this can make the subordinate
people he or she prefers not to. This might, at feel fear.
least, put strain on the shoulders of those doing No good relationship, personal or professional,
it. Understanding the affinities of a team and deal- comes from fear. So, it is an important part of
ing with them can be a vital part of distributing leadership to handle communication properly:
tasks and, thus, of being a leader in game both in the sense of content and form.
development. One possible point to start with, regarding com-
Also, even when the leader chooses the right munication, is the tone of voice used. Much of what
person for a task at a given time, circumstances humans communicate is nonverbal, and the tone of
may change during its execution (DeMarco and voice, for instance, is a very important factor. Care
Lister 1999). From unpredicted technical difficul- must be taken to pass the correct information.
ties to personal dramas, there is a whole universe Apart from the tone of voice, a leader must ask
around this chosen person which can change. for feedback and give feedback whenever possi-
A good leader should always be attentive to his ble (Acton 2014). It’s usual for a team member to
or her team members, being always ready to help be passive on giving or asking for feedback. The
them whenever possible. leader should be prepared to proactively work
As a reminder, dealing with people involves far around it.
more than logic, algorithms, task allocation, or A good instrument for feedback is having peri-
aesthetic sense. It involves questions like self- odic one-on-one meetings with each and every
esteem, sensitivity, sense of righteousness, pride, team member periodically Stone et al. 2000;
and many others to be listed. The tool to deal with DeMarco and Lister 1999). Their objective should
all of them is communication. be having a human exchange, to prevent possible
latent tensions from rising.
Performance feedback should, ideally, be
Tip 3: Communication given in a task basis, or even in a subtask basis
for long tasks, especially if this means for the
When working with people, the most important leader to demonstrate multiple times a day
tool to use is communication (DeMarco and Lister approval for a well-done job. Showing disap-
1999; Acton 2014). It should be done far more proval is just as important: The earlier the leader
often than what would feel natural to a leader: shows it, the earlier the team member can correct
Many things which are obvious to the leader the cause of it. A good leader should always keep
may be even counterintuitive to some team in mind that when he or she does not communicate
members. criticism, it is denying a chance to the person
Other times, people will simply not tell the receiving it to improve.
leader when they disagree or have problems with Finally, if a leader gives enough context to the
something. Considering the issues tolerable, they team members, passing information about the pro-
will often prefer to avoid conflict in order not to jects and about the company, they will be far more
Game Development Leadership Tips 751

likely to understand their contribution to the big More than just noticing, people will first try to
objectives. This practice provides great help in understand why the leader is being insincere. And
motivating people: Humans, as shown by reli- often they will pay back behaving the same way:
gions, feel the need to be part of greater plans being insincere as well. This is not the trust envi-
and have higher purposes. ronment expected from the successful teams.
Some new leaders sometimes will consider
they have good reasons to be less than sincere
Tip 4: Responsibility with their teams. Sometimes they actually want
to protect them from criticism from other people
Another sudden change for those who become or teams; other times there are bad news such as
leaders is the scope of their responsibility. Before possible layoffs ahead and the leaders don’t want
being in a leadership position, each individual is the team to lose focus. Still, a leader should under-
accountable for his or her work alone. This is stand that if he or she can cope with it, so do the
obviously different for leaders. Still, there are team members. G
not so obvious changes worth mentioning. As intelligent adults, they most likely faced
A typical situation which exposes these dif- bad news before (Stone et al. 2000). Most of
ferences is the case when the newly appointed them had a broken heart; most of them heard of
leader is confronted by his or her boss regarding people dear to them dying: This certainly dwarfs
some issue on the project. Any good leader will any bad news a job in the game industry can bring.
understand who was responsible for the part Going further, when presented with difficulties
which had issues. The sometimes the counterin- those people may help in finding good solutions
tuitive part is deciding on how to proceed from to it or, when not possible, either start coping with
this point. the issue or accepting the loss.
Many beginner leaders will, sometimes in the Still, a caring leader will have concerns about
human act of self-preservation, explain to their how a person will feel after criticism is delivered.
bosses that the issues are coming from a specific A caring leader won’t want to hurt people’s feelings
team member (Stone et al. 2000). This is an inap- or self-esteem. He or she will deliver the criticism
propriate approach. making it clear that the problem is not the person,
The leader is responsible for the results, regard- but the action, behavior, or result achieved.
less of the circumstances. The leader should take
responsibility for the current state of the project at
all times. If a team member is underperforming, it Tip 6: Trust
is the responsibility of the leader to deal with it
before it becomes a problem to the project. Trust is an absolute key element when working
Moreover, in the cases where the leader doesn’t with people (Sinek 2009). It is one of the founda-
have the appropriate people or resources to handle tions of human society: When humanity was
a task, it is his or her responsibility to inform it to young, threat was all around. There were no cities
the boss as soon as this information becomes where you could be safely away from predators.
available. Carnivores such as wolves, bears, and big felines
were a common threat. Sleeping was particularly
dangerous for humans in the wild.
Tip 5: Sincerity Survival required cooperation (Sinek 2009).
There had to be people awake during the night to
One of the most important traits of a leader is alert the others of possible attacks, and the lives of
sincerity. The people working with a leader are the entire group would depend on these people
intelligent adults. More often than not, the team doing their jobs. The group had to trust them
members will notice when the leader is not being completely and reciprocate, or else this equilib-
sincere. rium would eventually fail.
752 Game Development Leadership Tips

This was the reality of the human race for a problem will pass a message that there are no
period far longer than recorded history up until consequences to breaking the trust, which can
when this text was written for the first time. This is potentially undermine the trust of the entire team.
not just a fad, a modern behavior; this is some-
thing so old that it is likely even recorded in
human genes: Humans are programmed to coop- Tip 7: Vulnerability
erate with each other, but this cooperation requires
trust. A common issue new leaders have, especially
Let’s now take these ideas what most would those who were excellent doers, is dealing with
consider a somewhat safer environment: the office their own vulnerabilities. In the game industry,
of a game studio. The central idea remains: often a doer is evaluated by his or her individual
A good leader should understand that if he or capacity to generate code or assets. Leaders, on
she wants to be trusted, the first thing to be done the other hand, are not evaluated only by their
is to trust the people with whom he or she works. individuality: for this reason, even the best of the
And what does trusting someone mean? It professionals or the most self-demanding one will
means primarily that the leader will delegate inevitably have vulnerabilities, even if they come
work as one of his or her main activities. More- from one of the team members.
over, this should be done trusting that the people Still, it’s far more usual for leaders to have their
receiving the tasks will be able to accomplish own vulnerabilities (Brown 2010). A common
them well. Also, just like a loving parent, a leader, one for the newly hired or promoted is not being
especially if he or she is a good engineer, should acknowledged as a leader by the team or some of
understand that it’s time for the team members to the team members, usually the most senior ones.
shine and his or her to support it. Another usual vulnerability is being simply less
A natural consequence of this line of technically knowledgeable, if at all, than the team
reasoning is the inherent absurdity of microman- members.
agement. It not only simply doesn’t work with It is of vital importance for the leader to cor-
engineers but also increases turnover. Microman- rectly access the situation he or she is in and
agement screams out loud: “I don’t trust you can do understand his or her vulnerabilities. More specif-
your work well and on time.” Any leader seeking ically, a leader should understand what he or she
trust must absolutely forget micromanagement. cannot do.
Returning to the concept of reciprocity, for a Either facing a limitation or struggling to avoid
leader to have team members who will answer mistakes, the key element to take away is asking
honestly, the leader should first answer honestly. for help. The leader is not expected to know it all;
If a leader wants the team members to take risks, he or she is not expected to be a super programmer
the leader should be the first to take those risks and or super artist. The leader is not expected to be
take them for the sake of the team. able to solve all problems from a team. The leader
For example, if the leader disagrees with some is expected to care for the team and for the goals,
decision, he or she should state so while and having help from others is the best way to
reminding that still his or her role is to uphold do it.
this decision. The team members will reciprocate A good way for a leader to access if he or she is
up to the point they believe a leader will go for doing properly on this topic is the feeling of being
them, and they will just believe by seeing the a fraud. A leader who feels he or she is a fraud is
leader to it first. much more likely to access his or her vulnerabil-
There are, also, situations when a team member ities correctly. As the philosopher Socrates said:
loses the trust of the leader. If this happens, action “All I know is that I know nothing.”
should be taken by the leader. Often involving the Feeling as a fraud, of course, is not a necessary
team on deciding which action to take is a good condition. But it’s easy to feel like one when doing
option. A leader trying to dodge this kind of right. A good leader should not only care for his or
Game Development Leadership Tips 753

her team members to achieve their goals and, just leader is talking to, especially if they are engi-
as importantly, strive to grow the team members neers, will have poor communication skills mean-
into becoming independent of him or her. ing that they won’t be able to properly articulate
Having a team independent of the leader leads the answer to the “whys” leading to an incorrect
into his or her mind the natural questioning of how analysis. Apart from this, sometimes, people, for
much he or she is needed by the company. Having many reasons, will simply prefer not to verbalize
this question in mind, while being well regarded some arguments.
by his or her peers or being able to repeatedly One way to help in avoiding some of the risks of
grow independent teams (just like being able to challenging arguments is, when possible, give peo-
raise independent children), is a great sign for a ple time to think about the questioning avoiding
leader. taking decisions during the challenging of an argu-
ment. This allows people to rethink about the
answers they gave and improve their arguments.
Tip 8: Challenge The bottom line is: a good leader should chal- G
lenge tasks and arguments but should understand
While a leader must inherently trust his or her that winning a discussion doesn’t make the win-
team members, it is critical for the job to ascertain ner’s ideas correct.
the right problems are being solved (Acton 2014).
A common tool for this task is periodically
checking what a team member is doing and chal- Tip 9: Commitment
lenging tasks when planning for them.
In order to maximize the chance of the priori- Commitment is a key element in successful game
ties and plans being right, the leader can use a very projects (Acton 2014). Committed people tend to
important tool: asking questions (aka challenge). deliver work on time and with better quality. More
There are two main questions: What problem do often than not, therefore, successful teams are also
we want to solve with a given task and why are we committed teams: a good leader should foster this
solving this problem? trait.
While both questions are important, the “why” A key element on this context is having people
question is by far more relevant: a bad answer to a agree on what should be done and, in many cases,
“why” might indicate that performing a given task is how it should be done. An effective means to
plain senseless. Going further, it happens quite often achieve it is letting the team decide by itself
that questions in game development have several those two points: it’s far easier to convince some-
layers of complexity: to reach the primary reason one of his or her own ideas than of ideas coming
why some tasks are required may need a series of from anyone else.
“whys” asked on top of each other’s contexts. Obviously there are requirements for this idea
The idea behind this analysis is similar to the to work: the team should be well informed of the
Socratic philosophy. Asking sufficient “whys” is a sought objectives, it should not be formed only by
good way to understand where one knowledge juniors, and the leader should be ready to chal-
ends and beliefs begin. lenge the proposed plans. All of those elements, in
There is a risk into this approach, though: most environments, are under control of the
reaching a wrong conclusion. A reason for this leader. A positive side effect of it is making them
to happen is that proving a decision is based on a feel useful.
belief makes it easy for someone to discredit an Once the agreement on what and how a plan
argument as most people want to make decisions should be done is reached, the proper scheduling
seemingly based on reason. of the tasks is another critical element to achieve
Still, it’s likely that most people will not apply commitment. If the team members consider a
the same technique on possible counterarguments. schedule unreasonable, they are likely to be right
Also there is a good chance that the people the and unlikely to commit to it.
754 Game Development Leadership Tips

To support this argument, the people working individual contribution for it is a great start. Peo-
in the game industry, no matter who is their leader, ple will understand they have a future in the com-
are often professionals. Sometimes they are even pany doing so and are likely to stay longer
specialists on their craft. They are most likely (DeMarco and Lister 1999; Acton 2014).
better aware of their own capacity than anyone There are some cases, though, where there is
else. Going against it often justifies lack of com- no grand strategy plan behind a company. Some-
mitment, as itself is often a lack of common sense. times the company is looking for a new identity. It
Still, just planning with the team what should is conceivable to see this happening with one-hit
be done, how it should be done, when on time it studios, which often have a lot of money but don’t
should be done, and by whom it should be done is know what to do with it. When this happens, an
not enough to achieve proper commitment. unusual opportunity takes place: the leader and his
A leader should also be ready to face a dire reality: or her team have the chance to create a grand
intermediate steps of plans fail a lot. This often strategy for the company.
happens quite early in projects. A common element of winning visions inside
Delegating responsibilities over a plan’s area to companies is market disruption. This can happen
an individual or a group is the next step a leader through new business or monetization models,
can take into fostering commitment, especially on new technology, inventing a new gameplay
what regards fail contingency measures. Doing genre, finding an undersupplied niche, or reaching
this, even when something unpredicted happens, out for people who weren’t previously gamers.
people will understand who should solve it and A good leader, therefore, should be ready to
the impact it will have on others: their primitive provide his or her team with the company vision.
trust mechanism is more likely to trigger, gener- If none exists, he or she should create one.
ating commitment.
On the other hand, if plans are given to the
team instead, they are much more likely to dis- Tip 11: Self-Improvement
agree upon its feasibility. People will feel forced
instead of committed. Reality is simple: no one is perfect. Accept it.
The consequence of this affirmation is that all
leaders can improve. A good one, therefore,
Tip 10: Vision should always work on improving himself or her-
self (Acton 2014).
One of the important parts of the human mind is There are several means to achieve self-
the desire to be part of something big: finding a improvement. A good leader will often try a few
meaning to a set of actions, something more than of them at a time. Still, among all the possibilities,
just surviving. It’s no coincidence that religions one of them is mandatory: firsthand experience.
can be very appealing: it’s a human need. Rarely anything will yield a deeper result than
A good leader should understand this aspect of actually facing a challenge. Luckily this usually
human nature and help people fulfill their needs. comes naturally as part of a leader’s job; when it
Care should be taken, though, not to abuse it: most doesn’t, though, this should be regarded as a warn-
game companies won’t want their employees ing sign and could be a reason to change jobs.
blowing themselves, or at least ruining their per- Most of the other means to achieve self-
sonal lives, for the sake of their game. improvement gravitate around the sharing of
The way for a good leader to work this aspect experiences. Talking to other leaders and reading
out is providing his or her team with the grand leadership-related texts can give the leader ideas
strategic vision behind the companies. In the about what to try when facing the challenges of
game industry, the term “world domination” is this job.
often heard. Explaining how the company intends Another category of actions is presenting one’s
to achieve it and showing each team member’s experience. Once a leader finds him or her in a role
Game Development Leadership Tips 755

where he or she is expected to present his or her decisions. They tend to be less tolerant, be irri-
accumulated knowledge about leadership, this can tated over small things, and often overreacting. In
be turned into an exceptional opportunity: change the work environment, due to the power relations
empiric information into systematic solidified between the leader and the team, this can be
knowledge. disastrous. A good leader must always control
Apart from that, a good leader should always his or her emotions. Emotional stability inspires
be ready to listen. Sometimes all that is needed to trust on the people you lead.
solve a problem is a person in a different mind-set: It’s, therefore, much easier to be a good leader
a good leader will understand this and be happy to when in a good state of mind. This requires the
expand his or her own mind-set whenever leader to take care of himself or herself. A happy
confronted with a different one. Listening is a leader is much more likely to be a good leader, and
key element for it. having a life outside work can be of great benefit
A good leader should also understand that he for it.
or she is often dealing with incomplete informa- Moreover, a great leader should go beyond G
tion, especially when it regards people. The result merely the work/life balance. Having good rela-
of it is that decisions which may seem obviously tionships outside work is of immense help. This
right can be the very opposite. Listening to peo- means that many of the things applied to be the
ple, especially the team members, is, therefore, a leader of a team should apply to the people outside
crucial element for improving a leader’s capacity the office.
when regarding specific teams. Above all, love should guide the relationships
Finally, even more important for a leader than outside work with other people. Conflicts should
his or her own self-improvement is fostering this be faced and resolved through communication
value for the team he or she leads. A good leader (Stone et al. 2000), instead of endured.
should not only be able to create a self-sufficient There should be no fear in sincerely showing
team but also a team capable of improving itself vulnerability (Brown 2010): trust people, but be
over time. ready to challenge any and all beliefs. Commit to
goals and to other people, and be accountable for
it (Acton 2014). Do so with a vision of yourself.
Tip 12: Yourself And always seek self-improvement.
In the end life is complicated because we are
Being a leader is hard. A leader is expected to always dealing with people: the same thing
always put the needs of the team above his or hers. leaders have to deal with every day in their work.
This can be very taxing, as there is a single person
having to care for many, having to worry about the
results of many, having to worry even if their life
outside work is going well. References
A common mistake novice leaders do is
Acton, M.: Lead Quick Start Guide. Gamasutra (2014).
succumbing to the sheer amount of things to care http://gamasutra.com/blogs/MikeActon/20141112/22
for and start having an unbalanced life. This is 9942/Lead_Quick_Start_Guide.php
particularly true for the work/life balance. Brown, B.: The Power of Vulnerability. TED (2010). http://
The problem of this approach is that the less a www.ted.com/talks/brene_brown_on_vulnerability
DeMarco, T., Lister, T.: Peopleware: Productive Projects
person has a life outside work, the less this person and Teams, 2nd edn. Dorset House Publishing, New
will come refreshed to the office. This is a usual York (1999)
way through which stress builds up, and in the Sinek, S.: How Great Leaders Inspire Action. TED (2009).
game industry this can be often perceived on http://www.ted.com/talks/simon_sinek_how_great_leade
rs_inspire_action
crunch periods, particularly the longer ones. Stone, D., Patton, B., Heen, S.: Difficult Conversations:
When under stress, especially over longer How to Discuss What Matters Most. Penguin Books,
periods of time, people tend to make poorer New York (2000)
756 Game Editor

dramatically improves the efficiency of the devel-

Game Editor opment process, enabling the developer to focus on
the manifest content of the game, such as, for
▶ Principle Structure to Create a 2D Game Level example, its user interface, level design and physics
Editor of the game world, game logic and game mechan-
ics, control scheme, etc. (Bishop et al. 1998).
A single game engine can also be reused in an
indefinite number of games, which further stream-
Game Engine lines the development process. Additionally,
many modern game engines support multiple plat-
Mikhail Fiadotau forms, making it possible, for example, to use the
Centre of Excellence in Media Innovation and same codebase to create a product for the personal
Digital Culture (MEDIT), Tallinn University, computer, a game console, and mobile devices
Tallinn, Estonia with minimal modifications.
Commercial-grade game engines are often
geared toward a specific type of game. For example,
Synonyms id Software’s idTech series of engines are specifi-
cally designed for use in first-person shooter games
2D Game Engines; 3D Game Engines; Particle such as Doom (2016) and Rage (2011), while Tell-
system; Physics engine tale Games’ eponymous Telltale Tool is intended for
choice-driven adventure games with quick-time
events. Engine specialization helps optimize
Definitions resources and perfect the implementation of the
particular features relevant for a given type of
A game engine is a software package designed to game: for example, skill trees and inventory man-
simplify and accelerate game development. Game agement for role-playing games or graphics–audio
engines provide low-level functionality such as synchronization for rhythm games. It also means
graphics rendering, game physics, and communi- that the choice of a game engine is a crucial deci-
cation with input devices. This allows developers sion: while the engine’s workings may not be visible
to focus on the actual content of the game: game to the player, game engines can, through their func-
logic, levels, and assets. tionality, “regulate individual videogames’ artistic,
cultural, and narrative expression”.
Recent years, however, have seen a rise to
Overview popularity of more universal game engines such
as Unity and Unreal Engine, whose functional
Game engines are based on the idea that a video range has gradually increased to the point where
game can be separated into two discrete layers: they can power most types of games. Even so,
low-level “core” functionality and the “content” each game engine has its own set of characteristics
of the game proper, which, although built on top and may be more or less suitable for particular
of the core, is essentially independent (Gregory tasks. For example, of the two engines mentioned
2009: 11–13). The “core” comprises basic opera- above, Unreal Engine has more advanced graph-
tions which are indispensable to the game but do ical capabilities, while Unity is less computation-
not directly constitute it, such as graphics rendering ally expensive and supports more platforms
and sound playback, collision checking and (Dickson et al. 2017).
in-game physics calculations, memory manage- While the term “game engine” in a narrow
ment and file operations, and so on. Delegating sense only designates a software infrastructure
these stock routines to the game engine that powers a video game, many popular engines
Game Engine 757

Game Engine, Fig. 1 Like many modern game engines, Unreal Engine includes a scene editor and an integrated
development environment

bundle with an integrated development environ- were created to deal with low-level operations
ment, allowing developers to code the game in the like graphics rendering.
same software ecosystem where it is tested. It is The desire to further optimize the development
also common for a game engine to come with a process eventually led to the emergence of the
level editor and sometimes a user interface editor, game engine. One of the earliest examples is
as is the case for Unreal Engine and Torque 3D. Infocom’s Z-machine, a virtual computer created
Some game engines, such as Unity and in 1979 which executed commands in its own
GameMaker, also support add-ons and plugins, language, specifically tailored for interactive fic-
making it possible to extend their out-of-the-box tion games such as Zork. The use of a virtual
functionality with third-party solutions (Fig. 1). machine helped Infocom avoid developing sepa-
rately for the multitude of home computer archi-
tectures that were in use at the time (Bartholomew
2008).
History In 1984, Sierra On-Line released King’s Quest:
Quest for the Crown, the first game based on
In the early days of gaming, video games were Sierra’s own Adventure Game Interpreter (AGI).
often programmed from scratch (Bishop et al. Initially, AGI was created because Sierra felt that
1998). However, the increasing complexity of an easier, tailor-made engine would allow writers
the video game medium, the evolution of software and designers to work more independently from
and hardware, and the advent of the multi-million- programmers, thus benefiting the workflow
dollar game industry with its pressure on cost and (Loguidice and Barton 2012: 150). The company
efficiency have made this approach unsustainable ended up using AGI in 14 different adventure
in commercial game development. Code was games before switching to the more advanced
increasingly recycled and dedicated libraries Sierra Creative Interpreter in 1988. By that time,
758 Game Engine

Game Engine, Fig. 2 Ultima Underworld (1992) was one of the first games based on a 3D game engine

Sierra’s competitor LucasArts was using its own Software in particular continued licensing the
custom engine SCUMM to create point-and-click engines for its subsequent titles such as Doom
adventure games (Black 2012). (1993) and Quake (1996) to other developers. In
The early 1990s oversaw the emergence of 3D addition, first-person shooters’ growing popular-
game engines. A notable early example is the ity demanded increasing technological innova-
engine used in Ultima Underworld (1992), tion, with each major title expected to be more
which was partially based on the basic 3D render- advanced than the previous one, thus necessitat-
ing engine of Space Rogue (1989), but was much ing a more sophisticated engine. As a result, first-
more advanced, adding the use of texture mapping person shooters “played a fundamental role in
(Paul et al. 2012). In the same year, id Software founding the industry of game engines” in the
released Wolfenstein 3D, a pioneering first-person 1990s (Fig. 2).
shooter, also powered by an engine from an earlier Another significant development was the
game (Hovertank 3D) with the addition of texture emergence of consumer-grade game engines,
mapping and ray casting. Following the success of aimed at hobbyist game creators with little exper-
Wolfenstein 3D, id Software set an important pre- tise in programming. Clickteam’s Klik & Play,
cedent by licensing the game engine to other released in 1994, was an important early game
companies who then produced such titles as engine which was easy to use, had an integrated
Blake Stone: Aliens of Gold (1993) and Super event and level editor, and relied on solely visual
3D Noah’s Ark (1994). This heralded the era of coding (Djaouti et al. 2010). It was followed by
third-party engine licensing, where the use of an other engines, such as GameMaker and Con-
in-house game engine was no longer inevitable. Id struct, as well as technologies such as
Game Engine 759

Game Engine, Fig. 3 Klik & Play (1994) was a pioneering consumer-grade game engine

Macromedia Flash, which have empowered universal, the choice of an engine is still a crucial
enthusiasts with little technical knowledge to decision which can have much influence on the
make their own games (Fig. 3). development process and its outcome.

Conclusion Cross-References

Game engines have changed the logic of video ▶ Game Physics Engine, Overview
game development, introducing a more efficient, ▶ Panda3D
content-centered approach and simplifying the ▶ Unity, a 2D and 3D Game Engine
production of multiplatform content. The appear- ▶ Unreal Engine, a 3D Game Engine
ance of consumer-grade engines and the easy
availability of professional solutions such as
Unity and Unreal Engine, coupled with the advent References
of Web 2.0, has also significantly lowered the
Bartholomew, D.: A tale of two languages. Linux
entry barrier to game development. Despite J. 2008(174) (2008) Available from: https://dl.acm.
some popular engines positioning themselves as org/citation.cfm?id¼1434965
760 Game Engine Loop

Bishop, L., Eberly, D., Whitted, T., Finch, M., Shantz, M.:
Designing a PC game engine. IEEE Comput. Graph. Appl. Game Integrity Validation
18, 46–53 (1998). https://doi.org/10.1109/38.637270
Black, M.L.: Narrative and spatial form in digital media: a
platform study of the SCUMM engine and Ron Gilbert’s ▶ Secure Gaming: Cheat-Resistant Protocols and
the secret of Monkey Island. Games Culture. 7(3), Game History Validation
209–237 (2012). https://doi.org/10.1177/155541201244
0317
Dickson, P.E., Block, J.E., Echevarria, G.N., Keenan,
K.C.: An experience-based comparison of Unity and
Unreal for a stand-alone 3D game development course. Game Interface: Influence of
In: Proceedings of the 2017 ACM Conference on Inno- Diegese Theory on the User
vation and Technology in Computer Science Educa-
tion, pp. 70–75. ACM (2017). https://doi.org/10.1145/ Experience
3059009.3059013
Djaouti, D., Alvarez, J., Jessel, J.-P.: Can gaming 2.0 help Isabel Cristina Siqueira da Silva and Felipe
design serious games?: a comparative study. In: Pro- Oviedo Frosi
ceedings of the 5th ACM SIGGRAPH Symposium on
Video Games, pp. 11–18. ACM (2010). https://doi.org/ UniRitter Laureate International Universities,
10.1145/1836135.1836137 Porto Alegre, Brazil
Gregory, J.: Game Engine Architecture. CRC Press, Boca
Raton (2009)
Loguidice, B., Barton, M.: Vintage Games: An Insider
Look at the History of Grand Theft Auto, Super Synonyms
Mario, and the Most Influential Games of All Time.
Focal Press, Oxford (2012) Diegetic interfaces; Graphical interface; HUD
Paul, P.S., Goon, S., Bhattacharya, A.: History and compar- (heads-up display); Human-computer interaction
ative study of modern game engines. Int. J. Adv. Comput.
Math. Sci. 3, 245–249 (2012) Available from: https://
pdfs.semanticscholar.org/d910/7fad25a54767701eea41
bbc1ac37266182a3.pdf Definitions

User experience (UX) is the set of elements and

factors related to the interaction of the user with a
Game Engine Loop particular product, system, or service whose result
generates a positive or negative perception.
▶ Game Loop and Typical Implementation
The game interface is fundamental for the
player experience once transmits information
such as life and/or power bars, stopwatches,
Game Engine Main Loop inventory, punctuation, etc. This information is
referenced by heads-up display (HUDs).
▶ Game Loop and Typical Implementation Diegese refers to the world in which the events
of a story occur, defining what makes or does not
part of the virtual scenario and what is visible to
the characters that inhabit this alternative
Game Engine Update Loop universe.

▶ Game Loop and Typical Implementation

Introduction

The user experience (UX) is the set of elements

Game Engines and factors related to the user interaction with a
particular product, system, or service whose result
▶ Virtual Reality Game Engines generates a positive or negative perception. When
Game Interface: Influence of Diegese Theory on the User Experience 761

interacting with a product, people seek new expe- the transposition of the fourth wall as a strategy to
riences through perceptions that involve practical achieve the projected user experience.
and subjective aspects such as usability, effi- The text is organized as follows.
ciency, and satisfaction. Beyond subjective, UX Section “Games and Diegese Theory” addresses
is dynamic since changes over time due to the the main issues related to diegese applied to the
presentation of new challenges and rewards game interfaces. Section “Applying Diegetic and
(Hassenzahl and Tractinsky 2006; Schell 2008; Nondiegetic Interfaces in the Game Design” pre-
Garrett 2010; ISO 9241-210, 2010; Costa and sents a discussion of the concepts presented in
Nakamura 2015). section “Games and Diegese Theory,” and finally,
According to Russell (2011), the interface section “Conclusions” presents the final
design is often one of the most challenging considerations.
aspects of game design. This fact occurs
because there is a great amount of information
to transmit to the player in relation to the screen Games and Diegese Theory G
space available. If this relationship is not bal-
anced correctly, the user experience with the Considering the user experience in games, the
game can be frustrating. Other elements that graphical interface should be constructed in
also directly influence the perception of the order to provide the communication between
interface by the user are color and composition gameplay objects and the player. Among different
concepts (Ware 2008). studies involving games interaction, Fagerholt
Game interfaces are usually composed of two and Lorentzon (2009) suggest the use of diegese
main elements: (1) controls common to other theory, adapted from the areas of literature,
computer systems, such as buttons and menus cinema, and theater.
and (2) heads-up display (HUD), which refer to In game design area, the diegese refers to the
graphic elements present on the game interface game universe, defining what is or is not part of
that transmit information to the user. The integra- the virtual world, and is based on two main prin-
tion of the HUD components and the game world ciples: the narrative and the fourth wall. The nar-
influence the immersion of the user experience rative is related to the game story (fictitious
and interaction. world). Besides, the fourth wall deals with the
In this sense, the HUDs can be displayed in a imaginary division between the player and the
traditional way or incorporated into the universe game world. For the player to immerse himself
of the game, in order to offer immersion to the in the game world, he must pass through the
player and, consequently, a more concrete experi- fourth wall.
ence. In this sense, a rereading of the diegese The player’s ability to move between the real
theory can be done for the definition of HUDs, world and the game world depends on how the
breaking with traditional paradigms of graphical interface designer provides information for him.
interfaces in games. Two main aspects are part of In this sense, the HUD elements are considered
the theory of diegese: the narrative and the fourth diegetic if they are part of the universe of the game
wall (Genette 1980; Rohden et al. 2011). and the characters are aware of these elements.
This entry discusses concepts of diegese theory Thus, the HUDs must communicate something
in the scope of design and development of graph- understandable to the characters according to
ical interfaces for games. We relate the elements events and rules defined in the universe. In some
of the narrative and the fourth wall to the experi- cases, the characters will interact directly with
ence provided to the player, comparing the differ- these elements and, eventually, these can exist
ent forms of HUD presentation during the exclusively to communicate something, not
evolution of digital games. Diegetic and non- being functional.
diegetic elements, especially in HUD, are The game Dead Space (Electronic Arts
discussed in order to clarify how these relate to Inc. 2018a) is an example that employs the
762 Game Interface: Influence of Diegese Theory on the User Experience

Game Interface:
Influence of Diegese
Theory on the User
Experience, Fig. 1 Dead
Space game (Electronic
Arts Inc. 2018a)

concept of diegetic interface. As shown in Fig. 1, industry maturity, diegetic interfaces began to
attached to the character’s armor is a luminous gain space.
marker that represents his life bar. This element Besides diegetic and nondiegetic HUDs,
visibly is part of the universe of the game, being Fagerholt and Lorentzon (2009) propose two
present along with the main character, that is, it is other types: spatial and meta. The concept of
not an element that only the player can visualize, spatial HUDs is applied when a certain element
but any character of the game universe. is present in the 3D space of the game, but it is not
The integration of diegetic elements into the part of the universe, nor can the characters see
game universe is considerably relevant to creating it. In general, these interface elements are used to
the projected experience – not only in relation to indicate avatars selected or other indicators. In the
the interface but also to the game in general. If a game The Sims 4, for example, in addition to
diegetic element escapes the context of the uni- nondiegetic elements, some elements with the
verse, this can generate a noise (Schell, 2008) in spatial approach are employed. Figure 3 shows a
the player’s experience, which potentiates the character, positioned to the right in the image,
opposite effect in relation to the goal of diegese which is accompanied by a spatial interface ele-
in games, which is to increase the immersion and ment (the balloon).
quality of the experience. Thus, diegetic interfaces On the other hand, meta elements are part of
require a work of harmonization with the elements the game universe but not necessarily in the 3D
of the game, in order to transpose the fourth wall plane. In general, meta elements are used as
in the proper way (Rohden 2011; Russel applied effects on the game camera, such as
et al. 2011). blood dripping from being hit by a shot
Unlike diegese theory, nondiegetic interfaces (as well as other types of damage), or effects
are defined as being outside the game universe. of rain particles, solar reflection among others.
The use of nondiegetic HUDs was initially recur- The Watch Dogs game (Ubisoft Entertainment
rent in the gaming industry and this fact led in 2017a) is an example of using HUDs meta with
some cases to an increase in the insertion of the transposition of the elements of the game
graphic elements, polluting the game interface universe, which causes the fourth wall to be
(Fig. 2). This practice contradicting the idea that crossed by the player (Fig. 4). Therefore, dur-
only the essential elements should be displayed in ing the use of the smartphone of the character,
an interface (Hassenzahl and Tractinsky 2006; the screen is presented to the player as a non-
ISO 9241–210 2010; Garrett, 2010). With game diegetic element.
Game Interface: Influence of Diegese Theory on the User Experience 763

Game Interface: Influence of Diegese Theory on the User Experience, Fig. 2 NDI HUDs of World of Warcraft
game (Blizzard Entertainment, Inc., 2018)

Game Interface: Influence of Diegese Theory on the User Experience, Fig. 3 The Sims game (Electronic Arts
Inc. 2018b)
764 Game Interface: Influence of Diegese Theory on the User Experience

Game Interface: Influence of Diegese Theory on the User Experience, Fig. 4 Watch Dogs game (Ubisoft
Entertainment 2017a)

These four types of HUD discussed in this 7. Does the game depend on immersion?
section can be combined to allow an effective 8. What is the relationship between aesthetics,
immersion of the user in the game universe, complexity and gameplay in games?
enhancing their experience with the game.
Regarding question 1, it is noted that some
game genres can improve their immersion with
Applying Diegetic and Nondiegetic the use of DIs, which help keep the player’s atten-
Interfaces in the Game Design tion within the game world. Some games, by their
simple nature, do not even need an interface;
This section focuses on the discussion of the main others, that are not dependent on a large number
aspects related to the design of HUDs in game of mechanics, can use some part of the avatar itself
interfaces based on diegese theory. Thus, some to give feedback to the player, such as the Journey
questions that can help the HUD project are listed, avatars (Sony Computer Entertainment America
seeking to perform a critical analysis situated in LLC 2014a; thatgamecompany 2018a, Fig. 5) and
the current scenario of games and based on the the Dead Space avatar armor (Fig. 1), which mask
research carried out in this work: use of a status bar.
Concerning question 2, with the beginning of
1. What are the main advantages of DIs? the popularity of independent games (indie
2. What is the motivation for using DIs? games) in the middle of the seventh generation
3. Is there any kind of idea that only works in DI? of consoles, some games present the intention to
4. Are NDIs still required to achieve the required stand out in a market long structured and domi-
level of gameplay? nated by large companies that focused on HUDs
5. Can DIs make use of non-diegetic elements in based on NDIs. However, the use of DIs
their design? represented an innovation, since the use of crea-
6. What is the tolerance for excess HUDs in NDIs? tivity was the only resource for small gaming
Is it possible to improve this issue with the free companies to gain prominence in this market.
customization of HUDs over the interface? Thus, new games, especially so-called art games,
Game Interface: Influence of Diegese Theory on the User Experience 765

Game Interface:
Influence of Diegese
Theory on the User
Experience,
Fig. 5 Journey game
(Sony Computer
Entertainment America
LLC 2014a;
thatgamecompany 2018a)

Game Interface:
Influence of Diegese
Theory on the User
Experience, Fig. 6 Flowe
game (Sony Computer
Entertainment America
LLC 2014b;
thatgamecompany 2018b)

explore DI integrated in a natural and functional (Campo Santo, 2018) is another example where
way to game design. The game Flower (Sony the DI is fundamental, because, instead of an NDI
Computer Entertainment America LLC 2014b) with a map containing arrows that aim their objec-
(thatgamecompany 2018b) (Fig. 6), for example, tives, the player has only a map and a compass and
was one of the first indie games to become popular must be guided by its own reasoning (Fig. 7).
in the market and to present an innovative inter- Regarding question 4, in game genres where
face concept and user experience. the player needs accurate information, usually in
The Journey and Flower games proposals are numerical form or through bars, NDIs are still the
examples of experiments based on DIs (question best form of HUD. In the King of Fighters series
3) and probably would not reach their level of (SNK Corporation, 2018) (Fig. 8), for example,
immersion with the use of NDIs. The concepts each millimeter of the bar makes a significant
of these games can only be demonstrated with difference for the player to plan attacks on the
elements of HUD inserted in the universe, in enemy. In fighting games, hardly an element of
order to give the player the information and the identification would be as precise and with com-
notion he needs. In this sense, the game Firewatch munication as clear as the bars of life. One
766 Game Interface: Influence of Diegese Theory on the User Experience

Game Interface:
Influence of Diegese
Theory on the User
Experience,
Fig. 7 Firewatch game
(Campo Santo, 2018)

Game Interface: Influence of Diegese Theory on the User Experience, Fig. 8 King of Fighter game (SNK
Corporation, 2018)

possibility in this type of game would be to dem- In relation to question 5, the game Dead Space
onstrate how close the character is to being (Fig. 2) can be considered an example of ID that
defeated by his body wounds. In fact, several makes use of the style of NDI, incorporating the
games have used this feature, but it does not latter to the universe of the game. For example,
meet the need of the life bar. This is an example status bars and menus are displayed as screens
of how NDI is still needed and cannot always be within the game universe in a futuristic way. The
replaced. Far Cry 2 game (Ubisoft Entertainment 2017b)
Game Interface: Influence of Diegese Theory on the User Experience 767

Game Interface: Influence of Diegese Theory on the User Experience, Fig. 9 DI presenting information in the
form of NDI in Far Cry game (Ubisoft Entertainment 2017b)

(Fig. 9) is another example that mixes concepts of game is inseparable from the gameplay. Some
NDIs and DIs on the same screen. Although most indie games exploit this relationship, breaking
HUDs follow the concepts of DIs, information conventional paradigms of games through an
such as ammunition, opportunities for interaction, abstract experience related to exploration and dis-
and health are displayed as a message in the for- covery as in the Hohokum game (Sony Computer
mat of NDI on the screen. In the same context, Entertainment America LLC 2014c; Honeyslug
Team Fortress 2 game (Valve Corporation, 2018) Ltd., 2011) (Fig. 11).
presents spatial elements in addition to diegetic Finally, regarding question 8, it should be con-
and nondiegetic elements (Fig. 10). sidered that it is possible to play a fun game with an
Considering question 6, it is observed that the underdeveloped narrative and an unpleasant aes-
amount of HUDs in NDIs is directly proportional thetic, but it is difficult to play a game with good
to the game complexity. In some cases, a large aesthetics if the experience it offers is not satisfac-
amount of HUDs is unavoidable for the desired tory. The concept of gameplay, however, is exten-
gameplay, such as the World of Warcraft game sive and even a game that offers little complexity
(Fig. 2). On the other hand, the possibility of does not necessarily become an inferior game.
interface customization helps the player to create Such games can abdicate the complexity for a
an identity with the game, allowing the adaptation game that focuses on aesthetics, as long as the
of the HUD positions according to their needs and gameplay does not suffer any kind of commitment.
preferences (Preece et al., 2005). This feature is Sound Shapes game (Sony Computer Entertain-
especially important when the game interface has ment America LLC 2012, Fig. 12) is an example
many types of HUDs. that had significant success in the gaming industry.
The question 7 addresses gameplay versus After analyzing the eight questions raised in this
immersion. In this sense, it must be considered section, is noticed the type of HUD to be employed
that the focus of the player is on the gameplay. in a game depends on some key elements:
However, gameplay and immersion can go hand
in hand in order to enhance the player experience. • Player/user experience
Aragão (2016), for example, states that the issue • Immersion level
of immersion in the fictional environment of the • Narrative
768 Game Interface: Influence of Diegese Theory on the User Experience

Game Interface:
Influence of Diegese
Theory on the User
Experience,
Fig. 10 Team Fortress
2 game (Valve Corporation,
2018): (a) Spatial elements;
(b) diegetic and
nondiegetic HUDs

• Aesthetics the user experience also depends on the player’s

• Style level of expertise and game genre.
• Target audience
• Complexity of mechanics
• Feedback Conclusions

Issues related to the definition of DI or NDI The evolution of the gameplay changes the way
should take into account these listed aspects, since the player interacts with the game, having differ-
the gameplay and usability depends on the level of ent forms of HUDs and feedbacks. Most games
accuracy of the information related to the mechan- with diegetic interface end up being natural
ics of the game. Moreover, some studies have enough that the player does not realize that he
been carried out in order to analyze the preference crossed the whole adventure without using a con-
of users for diegetic or nondiegetic interfaces as ventional nondiegetic interface. On the other
well as to assess the accuracy of diegese design for hand, the increasing complexity of the games is
games (Fagerholt and Lorentzon 2009; Iacovides directly related to the demand for nondiegetic
et al. 2015; Peacocke et al. 2015). Is noticed that interfaces, due to their greater precision in the
diegetic interfaces positively impact the user display of information.
experience, increasing immersion, cognitive The understanding of the diegetic and non-
involvement, and a sense of control. However, diegetic interfaces presented in this work can
Game Interface: Influence of Diegese Theory on the User Experience 769

Game Interface: Influence of Diegese Theory on the User Experience, Fig. 11 Honokum game (Sony Computer
Entertainment America LLC 2014c; Honeyslug Ltd., 2011)

Game Interface:
Influence of Diegese
Theory on the User
Experience,
Fig. 12 Sound Shapes
game (Sony Computer
Entertainment America
LLC 2012)

collaborate for the best quality in interfaces design. Cross-References

However, there is no interface approach that will
always be the most appropriate, just as in some ▶ Cognitive Psychology Applied to User Experi-
cases the combination of different approaches can ence in Video Games
foster the construction of a more consistent HUD. ▶ Pervasive Games
770 Game Level Design

References Sony Computer Entertainment America LLC.: Available in

http://www.soundshapesgame.com/pt/home/public.
Aragão, O.: Todas as vidas são reais: narratividade, html (2012)
jogabilidade e imersão. ECO Digital, UFRJ 1(1) (2016) Sony Computer Entertainment America LLC.: Available in
https://www.playstation.com/pt-br/games/journey-ps4/
Blizzard Entertainment, Inc.: Available in http://us.battle.
(2014a)
net/wow/pt/legion/#bottom (2018)
Sony Computer Entertainment America LLC.: Available in
Campo Santo.: Available in http://www.firewatchgame.
https://www.playstation.com/pt-br/games/flower-ps3/
com/ (2018)
(2014b)
Costa, A.F., Nakamura, R.: Experiência de usuário e Sony Computer Entertainment America LLC.: Available in
experiência de jogador: discussão sobre os conceitos e https://www.playstation.com/pt-br/games/hohokum-
sua avaliação no projeto de jogos digitais. In: Proceed- ps4/ (2014c)
ings of SBGames 2015. SB, Teresina, PI (2015) thatgamecompany.: Available in http://thatgamecompany.
Electronic Arts Inc.: Available in http://www.ea.com/ com/journey/ (2018a)
deadspace (2018a) thatgamecompany.: Available in http://thatgamecompany.
Electronic Arts Inc.: Available in https://www.thesims. com/flower/ (2018b)
com/pt_BR/ (2018b) Ubisoft Entertainment.: Available in https://www.ubisoft.
Fagerholt, E., Lorentzon, M.: Beyond the HUD: user inter- com/pt-br/game/watch-dogs/ (2017a)
faces for increased player immersion in FPS games. Ubisoft Entertainment.: Available in https://far-cry.ubisoft.
Master of Science Thesis, Division of Interaction com/game/pt-br/home/ (2017b)
Design, Chalmers University of Technology Göteborg Valve Corporation.: Available in http://www.teamfortress.
(2009) com/ (2018)
Garrett, J.J.: The Elements of User Experience: User- Ware, C.: Visual Thinking: for Design (Morgan Kaufmann
Centered Design for the Web and Beyond, 2nd edn. Series in Interactive Technologies), first edition Edi-
Thousand New Riders Publishing, Thousand Oaks (2010) tion, Morgan Kaufmann (2008)
Genette, G.: Narrative discourse. Basil Blackwell, Oxford
(1980)
Hassenzahl, M., Tractinsky, N.: User experience – a
research agenda. Behav. Inform. Technol. 25(2),
91–97 (2006) Game Level Design
Honeyslug Ltd.: Available in http://www.honeyslug.com/
(2011) ▶ Principle Structure to Create a 2D Game Level
Iacovides, I., Cox, A., Kennedy, R., Cairns, P., Jennett, C.:
Removing the HUD: the impact of non-diegetic game
Editor
elements and expertise on player involvement. Pro-
ceedings of the 2015 Annual Symposium on Computer
Human Interaction in Play. ACM New York (2015)
ISO 9241-210.: Ergonomics of human-system interaction – Game Level Editor
Part 210: Human-centred design for interactive sys-
tems. International Organization for Standardization
(ISO) (2010) ▶ Decoupling Game Tool GUIs from Core
Peacocke, M., Teather, R.J., Carette, J., MacKenzie, I.S.: Eval- Editing Operations
uating the Effectiveness of HUDs and Diegetic Ammo
Displays in First-Person Shooter Games. Games Entertain-
ment Media Conference. IEEE, Piscataway (2015)
Preece, J., Rogers, Y., Sharp, H.: Design de interação: além Game Loop and Typical
da interação homem-computador. Bookman, Porto
Alegre (2005)
Implementation
Rohden, L., Kussler, L., Silveira, D.: O Jogo enquanto
dialética da recepção fílmica. Journal of Aesthetic and Aaron Hitchcock and Kelvin Sung
Philosophy of Art. Graduation Program on Aesthetics Computing and Software Systems, University of
and Philosophy of Art – UFOP, 11 (2011)
Russell, D.: Video game user interface design: Diegesis
Washington Bothell, Bothell, WA, USA
theory. (2011). http://devmag.org.za/2011/02/02/
video-game-user-interface-design-diegesis-theory/.
Accessed 12 Apr 2018 Synonyms
Schell, J.: The Art of Game Design: A Book of Lenses.
Taylor & Francis. (2008)
SNK Corporation.: Available in https://www.snk-corp.co. Game engine loop; Game engine main loop;
jp/us/games/kof-xiv/ (2018) Game engine update loop
Game Loop and Typical Implementation 771

Definitions Typical Game Loop Implementation

At the innermost core of every game engine is a A game loop is the mechanism through which the
compact infinite loop. This loop continuously in-game logic and drawing of visible elements are
iterates through the steps of receiving and pro- continuously executed and presented. A simple
cessing player input, updating the entire game game loop consists of processing the input,
state, and rendering and displaying the game updating the state of game objects, and drawing
objects. This loop is executed at a short but those objects, as illustrated in the following
discretised time interval supporting seemingly pseudocode:
instantaneous interactions between the players’
input and the graphical gaming elements. This initialize();
// Initialize the game state
constantly running loop is referred to as the
while(game running) {
game loop. input();
// Receive input from the user G
update();
// Iterate through and update
Introduction every game object
draw();
To convey a lifelike sense of instantaneity, each // Render and display all visual
cycle of the game loop must be completed within elements
}
a normal human’s reaction time. This is often
referred to as real time: the amount of time that
is too short for humans to perceive both visually As discussed, an FPS of 60 is required to
and cognitively. Typically, real-time can be maintain the sense of real-time interactivity.
achieved when the game loop is running at a rate When the game complexity increases, one prob-
of at least 40–60 cycles in a second. The speed of a lem that may arise is that sometimes a single loop
standard game loop, with one drawing operation can take longer than 1/60th of a second to com-
per cycle, can be quantified as the number of plete. When this happens, a game would have to
drawn frames per second (FPS), commonly called run at a reduced frame rate and the entire game
the frame rate. will appear to slow down. A common solution is
An FPS of 60 is a good target for performance. to prioritize which operations to emphasize and
At such a frame rate, update of the screen will which to skip. Because correct input and updates
coincide with a game update. This synchronicity are required for a game to function as designed,
increases the fluidity of motion and reduces the the draw operation is often the one that is skipped
perception of the graphics lagging or appearing when necessary. This is known as frame skipping,
jittery. This is to say, your game must receive and the following pseudocode illustrates one such
player input, update the game world, and then implementation:
draw the visual elements of the game world all
elapsedTime ¼ now;
within 1/60th of a second. // Time for one loop cycle
The game loop itself, including the implemen- previousLoop ¼ now;
tation details, is one of the most fundamental // Begin of previous cycle
control structures of a video game. With the while(game running) {
elapsedTime +¼ now - previousLoop;
main goal of maintaining real-time performance, // Time for previous cycle
the details of a game loop’s operation are of no previousLoop ¼ now;
concern to the rest of the game. For this reason,
the implementation of a game loop should be input();
tightly encapsulated in the core of the video // Receive player input
while( elapsedTime >¼
game with its details hidden from other operations UPDATE_INTERVAL ) {
and gaming elements.
772 Game Mechanics

// if previous cycle took too

much time Game Mechanics
update();
// Ensure game is up to date
elapsedTime -¼ UPDATE_INTERVAL; ▶ Gamification of Modern Society: Digital
} Media’s Influence on Current Social Practices
draw(); // Draw the updated game
}

In the pseudocode above: UPDATE_INTER-

Game Performance
VAL is the maximum time allowed for one game
loop cycle. When the elapsed time between com-
▶ Online Gaming Scalability
pleted game loop cycles is greater than the
UPDATE_INTERVAL, the update() function
will be continuously called until the actual game
state is caught up. This means that the input() and
Game Physics
draw() operations are essentially skipped when
the game loop is running too slowly. When this
▶ Game Physics Engine, Overview
happens, the entire game will appear to run
slowly, with lagging player input responses and
frames skipped. However, the game logic and
state will continue to be correct.
Game Physics Engine
Notice that the while loop that encompasses
the update() function call simulates a fixed update
▶ Unity, a 2D and 3D Game Engine
time step of UPDATE_INTERVAL. This fixed
▶ Unreal Engine, a 3D Game Engine
time step update allows for a straightforward
implementation in maintaining a deterministic
game state.
Game Physics Engine,
Overview
Cross-References
Aaron Hitchcock and Kelvin Sung
▶ Character Animation Scripting Environment Computing and Software Systems, University of
▶ Decoupling Game Tool GUIs from Core Washington Bothell, Bothell, WA, USA
Editing Operations
▶ Game Engine
▶ Game Physics Engine, Overview Synonyms
▶ Physical, Virtual, and Game World Persistence
Game physics system; Game physics; Physics
engine; Rigid body physics
References

Gregory, J.: Game Engine Architecture, 2nd edn. CRC

Press, Boca Raton FL (2014). ISBN: 978-1-46-
Definitions
656001-7
Nystrom, R.: Game Programming Patterns. Genever A game physics engine is a self-contained sub-
Benning (2014). ISBN: 978-0-99-058290-8. www. system of a game engine which defines the phys-
gameprogrammingpatterns.com
Sung, K., Pavleas, J., Arnez, F., Pace, J.: Build Your Own
ical characteristics of objects in the game world.
2D Game Engine and Create Great Web Games. The subsystem is typically utilized to simulate and
APress, Berkeley (2015). ISBN: 978-1-48-420953-0 approximate lifelike interactions between game
Game Physics Engine, Overview 773

objects. A game physics engine may not be pre- 2. Detect collisions between the shapes, if present
sent in all game engines, but when one is defined, 3. Resolve potential interpenetrations between
it operates on simple geometric shapes and is only the colliding shapes
invoked when necessary. This is because the sim- 4. Compute proper responses for the colliding
ulation of physical interactions is computationally shapes
costly and typically only a selective subset of
objects in the game world would define physics The proper implementation of these simulation
components and participate in the simulation. steps enables believable scenarios when objects
physically interact with each other in the game
world.
Introduction

Game object interactions that mimic real life have Efficiency Considerations
become a key element of many modern PC and G
console games as well as, more recently, browser The computation involved in detecting and
and smartphone games. For example, when shoot- resolving collisions between arbitrary rigid shapes
ing a basketball in a video game, the player would can be algorithmically complicated and computa-
expect the ball’s trajectory and interactions with tionally costly. Most rigid body game physics
the backboard and hoop to resemble the physical engine implements two optimizations to address
world. An effective way of conveying real world these challenges: simple shape approximation and
object behaviors is by approximating and simu- selective computation.
lating the underlying physics based on a game Simple shape approximation. Rigid body sim-
physics engine subsystem. ulations are typically based on a limited set of
simple geometric shapes. For example: rigid
spheres, rectangular boxes, and cylinders. In
A Rigid Body Game Physics Engine most game engines, these simple rigid shapes
can be attached to geometrically complex game
The range of topics within physics for games is objects for approximating their physical behav-
broad and includes, but is not limited to, areas iors. For example, attaching rigid spheres on the
such as rigid body, soft body, fluid, and particles. hands of a basketball shooter and using the rigid
Rigid bodies are objects that do not change shape. body physics simulation of the rigid spheres to
For example: a Lego block, your desk, or the approximate the physical interactions between the
hardwood floor. The interactions between rigid hands and the basketball.
bodies, for example, a falling Lego block bounc- Selective computation. To avoid excessive
ing off your desk and landing on the hardwood runtime complexity and resource demands, phys-
floor, are best understood scientifically and thus ics simulation are only carried out for essential
most straightforward to simulate computationally. game objects. For the basketball game example,
Rigid body simulation approximates many while it may be important to model and simulate
types of object interactions in video games and the physical interactions between the basketball
is the core of most game physics engines. A rigid and the hands of the in-game characters, such
body game physics engine defines rigid shapes to computations would be unnecessary between
serve as the components of objects in the game other nonessential objects like the basketball
world. At runtime, during each game loop update and the heads of the characters. To facilitate the
cycle, the game physics engine would iterate selective invocation of computation on only a
through all defined rigid shapes and: designated subset of game objects, physics repre-
sentation and computation simulation are usually
1. Calculate the motion of the shapes and move abstracted and structured as an independent sub-
them system within the game engine.
774 Game Physics System

The Context of Entity-Component-

System Pattern Game Player Modeling

Referencing the entity-component-system (ECS) Sehar Shahzad Farooq and Kyung-Joong Kim
pattern, all objects in the game world are compo- Department of Computer Science and
nents with only a selective subset having a Engineering, Sejong University, Seoul,
corresponding physics component defined. The South Korea
game physics engine simulation only involves
the physics components. It is important to recog-
nize that a physics component only attempts to Synonyms
approximate the physical properties of a game
object for interaction purposes and that the com- Player modeling; Preference modeling
ponent typically cannot be used to represent the
game object in general as it lacks fine detail. For
example, a rigid sphere may be defined as the Definition
physics component of a basketball game object.
A more striking example would be the in-game Game player modeling is the study of computa-
character’s hands which are likely to be tional models to gain an abstracted description of
represented by detailed 3D models while the players in games. This description helps to detect,
physics components of these objects may be a predict, and express the behavior and feelings of
collection of simple rigid spheres and cylinders. players and personalizes games to their
preferences.

Cross-References
Introduction
▶ Game Engine
▶ Game Loop and Typical Implementation Game player modeling is the study of computa-
tional models to gain an abstracted description of
players in games. This description helps to detect,
References predict, and express the behavior and feelings of
players and personalizes games to their prefer-
Entity-component-System. Available: https://en. ences. These models can be automatically created
wikipedia.org/wiki/Entity-component-system. using computational and artificial intelligence
Accessed 28 June 2018
Tanaya, M., Chen, H.M., Pavleas, J., Sung, K.: Building a
techniques which are often enhanced based on
2D game physics engine using HTML5 and JavaScript. the theories derived from human interaction with
Apress (2017). ISBN: 978-1-4842-2582-0 the games (Yannakakis et al. 2013). It offers two
major benefits. First, it helps in content customi-
zation to cover broader range of players with
different skill levels and adapt challenges on the
Game Physics System fly in response to the player’s actions (Bakkes
et al. 2012). Second, it works as a form of feed-
▶ Game Physics Engine, Overview back for the game developers and designers so
that they may add new innovative features to the
games as well as develop new games that advance
knowledge, synthesize experience, and escalate
Game Platforms the interest of the player (Yannakakis et al. 2013).
The very first instance of research on player
▶ Game Venues and Platforms modeling was reported in the 1970s where Slagle
Game Player Modeling 775

and Dixon attempted to model the behavior of for high-level player modeling using limited
opponent players in the domain of classical amount of data.
games by assuming the elementary fallibility of Based on the type of the input data, several
the opponent (Slagle and Dixon 1970). Later on, a learning and data mining approaches are used for
search method based on knowledge about oppo- player modeling as can be seen in Table 1. The
nent players (i.e., strengths/weaknesses) was effectiveness of the modeling technique based on
invented in 1993 (Carmel et al. 1993). In 2000, user data is calculated using demographic/stereo-
Donkers improved opponent modeling by taking type approaches (Butler et al. 2010). The major
into account the computer player’s uncertainty challenge in such models is that they are limited to
(Donkers 2003). Afterward, an increasing interest deal with situations where individuals greatly
developed in the player modeling of modern deviate from the average. The sensory data is
video games to raise the entertainment factor correlated to the player’s behavior, emotions,
(Charles and Black 2004). Recently, player preferences, cognitive, and affective states
modeling has extrapolated its perspective from (Drachen et al. 2009). Physiological signals are G
opponent modeling to a number of other research correlated to arousal and valance using Plutchik’s
topics including player satisfaction (Yannakakis emotion wheel and the valence-arousal scale by
2008), modeling player’s preferences (Spronck Russell (1980), facial expressions using continu-
and Teuling 2010), runtime challenge adaptation ous, categorical, and active appearance models,
(Yannakakis et al. 2013), playing style, and learn- speech or psycholinguistic narrations using PER-
ing effective game strategies (Lockett et al. 2007). SONAGE, and psychological factors using Big
A comprehensive history of player modeling is Five model (Lankveld 2013). In-game data fea-
given in (Bakkes et al. 2012). tures collected during the game play are used to
A player model can have three types of inputs: identify or predict the type of the players which
user data, sensory data, and in-game data can then be further used for personalized compo-
(Yannakakis et al. 2013; Martinez and Shichuan nent generation or level modifications (Drachen
2012). User data includes personal assessment et al. 2009). An overview of input data gathering,
and third-person observation. The negligible lim- modeling approaches, computational analysis,
itations of user data are non-relevant data assess- and applications of game player modeling is
ments, short-time memory, and player’s self- shown in Fig. 1.
deception (Yannakakis 2012). Sensory data Although a lot of work has been done on the
includes data collected from the sensors mounted player modeling, several remaining issues need to
on the player’s body or in the player’s surround- be addressed. For instance, sensory data-based
ings. The most common sensor data includes models lack non-obtrusive data assessment, data
biometrical (Gunes and Piccardi 2006), physio- reliability, data validity, vigilance recognition, and
logical (Drachen et al. 2010; Martinez quick reactivity. User data-based models exhibit
et al. 2013), peripheral (Omar and Ali 2011), low correlation with the data collection time and
and nonverbal natural user interface with the the particular situation. In-game data-based
games (Amelynck et al. 2012). However, the models are restricted to particular players’ per-
sensor’s interface with the player faces chal- sonal interests in game, expert level, mood, enthu-
lenges when it comes to accuracy and perfor- siasm, and surrounding environment, making it
mance. In-game data is based on the player’s difficult to generalize for all players. However
actions taken within the game to infer perfor- the generalization problem is resolved by contin-
mance, skills, strategies, behavior, and game uously comparing and adjusting procedural per-
contexts including level completion time, mis- sonal behavior with human behavior and active
sion failure counts, resource utilization, situation player modeling (Holmgard et al. 2014; Togelius
handling, and target achievements (Nachbar et al. 2014). Furthermore, hybrid approaches are
2013; Kim et al. 2012; Weber et al. 2011). The used to overcome the issues of individual data-
big challenge is to interpret the raw data correctly based player models (Arapakis et al. 2009;
776 Game Player Modeling

Game Player Modeling, Table 1 Techniques used for game player modeling based on the input data types
Data
type Techniques
Supervised learning Unsupervised learning Other
User Supervised learning (shaker et al. 2010) Probabilistic learning Rating-based approach
data Neural network (Schmidhuber 2006) (Togelius et al. 2014) (Mandryk et al. 2006)
Committee selection strategy (Togelius Clustering (Yannakakis Active learning (Togelius
et al. 2014) et al. 2013) et al. 2014)
Classification and regression
(Yannakakis et al. 2013)
Sensory Neuroevolution (Pedersen et al. 2009) Cognitive appraisal theory
data (Frome 2007)
Usability theory (Isbister and
Schaffer 2008)
Neural network and clustering (Charles Belief-desire intention (Ortega
and Black 2004) et al. 2013)
Facial action coding system
(Ekman and Friesen 1978)
In-game Neural network (Charles and Black Clustering (Drachen
data 2004; Pedersen et al. 2009) et al. 2009)
Supervised learning with labels
(Togelius et al. 2014)
Multilayer perceptron (Togelius
et al. 2006)
Sequential minimal optimization
(Spronck and Teuling 2010)

Game Player Modeling, Fig. 1 Inputtypes, modeling approaches, analysis, and applications of game player modeling
Game Player Modeling 777

Kivikangas et al. 2011; Nogueira et al. 2013a, b). Donkers, H.H.L.M.: Searching with opponent models.
Game player modeling is also experimented in PhD Thesis, Faculty of Humanities and Sciences,
Maastricht University, Maastricht (2003)
some commercial games (e.g., Tomb Raider, Civ- Drachen, A., Canossa, A., Yannakakis, G. N.: Player
ilization IV, and Left 4 Dead), but there are still modeling using self-organization in Tomb Raider:
some problems of generalization (Drachen underworld. In: Proceedings of the IEEE Symposium
et al. 2009; Spronck and Teuling 2010; Ambinder on Computational Intelligence and Games (CIG),
pp. 1–8. IEEE, Milano (2009)
2011). Even though player modeling can be gen- Drachen, A. Nacke, E. L., Yannakakis, G., Pedersen, L.A.;
eralized, there is still a gap between the player’s Psychophysiological correlations with gameplay expe-
characteristics within a game and the real world rience dimensions. In: Brain, Body and Bytes Work-
which needs to be bridged in the future research shop, CHI 2010, Boston (2010)
Ekman, P., Friesen, W. V.: Facial action coding system: a
(Holmgard et al. 2014). technique for the measurement of facial movement. In:
From Appraisal to Emotion: Differences among
Unpleasant Feelings, Motivation and Emotion, vol.
Cross-References 183 12, pp. 271–302. Consulting Psychologist Press, G
Palo Alto (1978)
Ekman, P., Friesen, W.V.: Facial action coding system: a
▶ Constructing Game Agents Through Simulated technique for the measurement of facial movement. In:
Evolution From Appraisal to Emotion: Differences among
Unpleasant Feelings, Motivation and Emotion, vol.
12, pp. 271–302 (1978)
Frome, J.: Eight ways videogames generate emotion. In:
References Proceedings of Digital Game Research Association
(DiGRA), pp. 831–835. DIGRA, Tokyo (2007)
Ambinder, M.: Biofeedback in gameplay: how valve mea- Gunes, H., Piccardi, M.: A bimodal face and body gesture
sures physiology to enhance gaming experience. In: database for automatic analysis of human nonverbal
Proceedings of the Game Developers Conference affective behavior. In: Proceedings of the Eighteenth
(2011) International Conference on Pattern Recognition, vol.
Amelynck, D., Grachten, M., Noorden, L.V., Leman, M.: 1, pp. 1148–1153 (2006)
Toward e-motion-based music retrieval a study of Holmgard, C., Liapis, A., Togelius, J., Yannakakis, G. N.:
affective gesture recognition. IEEE Trans. Affect. Evolving personas for player decision modeling. In:
Comput. 3(2), 250–259 (2012) Proceedings of the IEEE Conference on Computational
Arapakis, I., Konstas, I., Joemon, M. J.: Using facial Intelligence and Games (CIG), pp. 1–8. IEEE Dort-
expressions and peripheral physiological signals as mund (2014)
implicit indicators of topical relevance. In: Proceedings Isbister, K., Schaffer, N.: Game usability: advancing the
of the seventeenth ACM International Conference on player experience. A theory of fun for game design.
Multimedia, pp. 461–470. ACM Press, New York CRC Press, Boca Raton (2008)
(2009) Kim, K.-J., Seo, J.-H., Park, J.-G., Na, J.-C.: Generaliza-
Bakkes, S.C., Spronck, P.H., Lankveld, G.V.: Player tion of TORCS car racing controllers with artificial
behavioural modelling for video games. Entertain. neural networks and linear regression analysis.
Comput. 3(3), 71–79 (2012) Neurocomputing 88, 87–99 (2012)
Butler, S., Demiris, Y.: Using a cognitive architecture for Kivikangas, J.M., Ekman, I., Chanel, G., Jarvela, S.,
opponent target prediction. In: Proceedings of the Third Salminen, M., Cowley, B., Henttonen, P., Ravaja, N.:
International Symposium on AI and Games, pp. 55–62. A review of the use of psychophysiological methods in
AISB, Leicester (2010) game research. J. Gaming Virtual Worlds 3(3),
Carmel, D., Markovitch, S.: Learning models of oppo- 181–199 (2011)
nent's strategy in game playing. In: Proceedings of Lankveld, G.V.: Quantifying individual player differences.
AAAI Fall Symposium on Games Planning and Learn- PhD thesis, Tilburg University (2013)
ing, pp. 140–147, Technion-Israel Institute of Technol- Lockett, A.J., Chen, C.L., Miikkulainen, R.: Evolving
ogy, Israel (1993) explicit opponent models in game playing. In: Proceed-
Charles, D., Black, M..: Dynamic player modeling: a ings of the Ninth Annual Conference on Genetic and
framework for player-centered digital games. In: Evolutionary Computation (GECCO), pp. 2106–2113.
Proceedings of the International Conference on ACM, New York (2007)
Computer Games, Artificial Intelligence, Design and Mandryk, R.L., Inkpen, K.M., Calvert, T.W.: Using psy-
Education, pp. 29–35. Ulster University, Reading chophysiological techniques to measure user experi-
(2004) ence with entertainment technologies. Behav. Inf.
778 Game Production Effort

Technol. Spec. Issue User Experience 25(2), 141–158 Togelius, J., Shaker, N., Yannakakis, G.N.: Active player
(2006) modelling. In: Proceedings of the Ninth International
Martinez, A., Shichuan, D.: A model of the perception of Conference on Foundations of Digital Games (FDG)
facial expressions of emotion by humans: Research (2014)
overview and perspectives. J. Mach. Learn. Res. 13 Weber, B.G., John, M., Mateas, M., Jhala, A.: Modeling
(1):1589–1608 (2012) player retention in Madden NFL 11. In: Proceedings of
Martinez, H.P., Bengio, Y., Yannakakis, G.N.: Learning the Twenty-Third Innovative Applications of Artificial
deep physiological models of affect. IEEE Comput. Intelligence Conference (IAAI) AAAI Press, San
Intell. Mag. 8(2), 20–33 (2013) Francisco (2011)
Nachbar J.: Learning in games. In: Meyers R. (ed.) Ency- Yannakakis, G.N.: How to model and augment player
clopedia of Complexity and Systems Science: satisfaction: a review. In: Proceedings of the First
SpringerReference (www.springerreference.com). Workshop on Child, Computer and Interaction
Springer, Berlin (2013). 2013-04-30 11:57:51 UTC (WOCCI) (2008)
Nogueira, P.A., Rodrigues, R., Oliveira, E., Nacke, L. E.: A Yannakakis, G.N.: Game AI revisited. In: Proceedings of
hybrid approach at emotional state detection: merging the 9th Conference on Computing Frontiers. ACM
theoretical models of emotion with data-driven statisti- (2012)
cal classifiers. In: Proceedings of the IEEE/WIC/ACM Yannakakis, G.N., Spronck, P.H., Loiacono, D., Andre, E.,
International Joint Conference on Web Intelligence Playermodeling, In: Dagstuhl Seminar on Artificial and
(WI) and Intelligent Agent Technologies (IAT), pp. Computational Intelligence in Games, pp. 45–55.
253–260. IEEE, Atlanta (2013a) Schloss Dagstuhl, Germany (2013)
Nogueira, P.A., Rodrigues, R., Oliveira, E.: Real-time psy-
chophysiological emotional state estimation in digital
gameplay scenarios. In: Engineering Applications of
Neural Networks, pp. 243–252. Springer, Berlin/Hei-
delberg/New York (2013b)
Omar, A., Ali, N.M.; Measuring flow in gaming platforms. Game Production Effort
In: Proceedings of the International Conference on
semantic Technology and Information Retrieval ▶ Domain-Specific Choices Affecting Design
(STAIR), pp. 302–305. IEEE, Putrajaya (2011) Effort in Gamification
Ortega, J., Shaker, N., Togelius, J., Yannakakis, G.N.:
Imitating human playing styles in Super Mario Bros.
Entertain. Comput. 4(2), 93–104 (2013)
Pedersen, C., Togelius, J., Yannakakis, G.N.: Modeling
player experience in super mario bros. In: Proceedings
of IEEE Symposium on Computational Intelligence Game Project
and Games (CIG), pp. 132–139. IEEE, Milano (2009)
Russell, J.A.: A circumplex model of affect. J. Pers.
Soc. Psychol. 39(6), 1161–1178 (1980) ▶ Game Thinking X Game Design Thinking
Schmidhuber, J.: Developmental robotics, optimal artifi-
cial curiosity, creativity, music, and the fine arts. Con-
nect. Sci. 18, 173–187 (2006)
Shaker, N., Yannakakis, G.N., Togelius, J.: Towards auto-
matic personalized content generation for platform
games. In: Proceedings of Artificial Intelligence and Game Prosumption
Interactive Digital Entertainment (AIIDE), pp. 63–68.
AAAI Press, California (2010)
Slagle, J.R., Dixon, J.K.: Experiments with the M & N tree-
Selcen Ozturkcan
searching program. Commun. ACM 13(3), 147–154 Jönköping International Business School,
(1970) Jönköping University, Jönköping, Sweden
Spronck, P.H., den Teuling, F.: Player modeling in Civili- Faculty of Communication, Bahcesehir
zation IV. In: Proceedings of the Sixth Artificial Intel-
ligence and Interactive Digital Entertainment
University, Istanbul, Turkey
Conference (AIIDE), pp. 180–185. AAAI Press, Cali-
fornia (2010)
Togelius, J., Nardi, R.D., Lucas, S.M.: Making racing fun Synonyms
through player modeling and track evolution. In: Work-
shop on Adaptive Approaches for Optimizing Player
Satisfaction in Computer and Physical Games, pp. 61– Playbor; Gamers; Multiplayers; Producers; Con-
71. CogPrints (2006) sumers; Co-creation; Network economy
Game Prosumption 779

Definitions by contributing their valuable time. Indeed, some

argued that vast majority were happy to be only
Game prosumption refers to the process of both passive consumers at the lack of incentives
production and consumption of computer games (Haque 2010). However, games provided the
rather than focusing on either one (production) or much-needed incentive.
the other (consumption).

Introduction Contemporary Computer Game

Prosumption
Concept of prosumption was first introduced by
Toffler (1980) as part of his futuristic outlook to Most computer gaming experience involves the
the post-industrial age. Kotler (1986) found the gamer’s engagement in the production of an epi-
idea of involving individuals in the production of sode of game play, which is then simultaneously G
the goods and services they consume rather pro- consumed (Harwood 2011). Gaming itself is a
vocative. Yet, it took a while for consumers not to prosumption activity. Furthermore, gamers con-
be considered as passive responders but active suming multi-player e-games not only buy and
creative actors contributing to the value creating play but also observe other players actions. This
process (Xie et al. 2008). collective action produced is then called a virtual
game. The integrated experience of online play
containing labor during virtual gaming is also
Key Principles and Concepts referred as playbor (Kücklich 2005; Ritzer 2014;
Schott 2010).
In its contemporary use, prosumption implies that
consumer participate in the production of the
goods that they consume (Ritzer and Jurgenson
Examples
2010). Accordingly, presumption involves more
than just a single act. It is an integration of partic-
There are some players who choose to dedicate an
ipatory physical, mental, and/or social actions in
extra effort to modify the games they play, such as
the process of creating desired products. These
the creators of Counter-Strike, Minh Le, and Jess
roles of consumer and producer blended and
Cliffe (Kücklich 2005). With the advent of highly
blurred even more with the introduction of digital
profitable sandbox games, more players are intro-
media (Pathak-Shelat 2014).
duced to prosumption of their own story as they
interact with the game world, decide for them-
selves what they want to play.
Digital Prosumption

In the digital and online world, prosumption

actions are not necessarily solely physical, but Cross-References
may also be virtual. Digital technologies pro-
vided more opportunity and visibility to con- ▶ Area of Interest Management in Massively
sumer’s input in various design, manufacturing, Multiplayer Online Games
and distribution dimensions of prosumption ▶ Immersive Technologies for Medical Education
(Rayna and Striukova 2016). The emerging net- ▶ Online Players: Engagement, Immersion, and
work economy enabled further prosumption pos- Absorption Across Secondary Worlds
sibilities for market participants and consumers ▶ RTS AI Problems and Techniques
alike (Grün et al. 2011). Individuals, on the other ▶ Videogame Engagement: Psychological
hand, not always wanted to engage in co-creation Frameworks
780 Game State Validation

References
Game Strategies
Grün, C., Huemer, C., Liegl, P., Mayrhofer, D., Motal, T.,
Schuster, R., Werthner, H., Zapletal, M.: eBusiness. In:
▶ Strategies for Design and Development of Seri-
Domingue, J., Fensel, D., Hendler, J.A. (eds.) Hand-
book of Semantic Web Technologies, pp. 787–847. ous Games: Indian Perspective
Springer, Berlin/Heidelberg (2011)
Haque, N.: Playbor: when work and fun coincide.
Wikinomics. Available http://www.wikinomics.com/
blog/index.php/2010/02/25/playbor-when-work-and-
fun-coincide/ (2010). Accessed 29 Jan 2018
Harwood, T.: Convergence of online gaming and Game Studies
e-commerce. In: Virtual Worlds and E-Commerce:
Technologies and Applications for Building,
▶ Cross-cultural Game Studies
pp. 60–89. Business Science Reference, Hershey
(2011)
Kotler, P.: The prosumer movement : a new challenge for
marketers. In: Lutz, R.J., Provo, U. (eds.) Advances in
Consumer Research, pp. 510–513. Association for
Consumer Research, Provo (1986)
Kücklich, J.: Precarious playbour: modders and the digital Game Thinking X Game
games industry. In: Brett Neilson and Ned Rossiter Design Thinking
(eds.) Fibreculture J. (5), Open Humanities Press.
(2005). http://five.fibreculturejournal.org/fcj-025-pre Isabel Cristina Siqueira da Silva1 and
carious-playbour-modders-and-the-digital-games-
industry/ João Ricardo Bittencourt2
1
Pathak-Shelat, M.: Media literacy and well-being of young UniRitter Laureate International Universities,
people. In: Ben-Arieh, A., Casas, F., Frønes, I., Porto Alegre, Brazil
Korbin, J.E. (eds.) Handbook of Child Well-Being: 2
Universidade do Vale do Rio dos Sinos
Theories, Methods and Policies in Global Perspective,
pp. 2057–2092. Springer, Dordrecht (2014) (UNISINOS), Porto Alegre, Brazil
Rayna, T., Striukova, L.: Involving consumers: the role of
digital technologies in promoting ‘prosumption’ and
user innovation. J. Knowl. Econ. 1, 1–20. Springer, Synonyms
US. (2016). https://link.springer.com/article/10.1007/
s13132-016-0390-8
Ritzer, G.: e-Games and Prosumption. Available https:// Game development; Game project; Gamification
georgeritzer.wordpress.com/2014/09/10/e-games-and-
prosumption/ (2014). Accessed 29 Jan 2018
Ritzer, G., Jurgenson, N.: Production, consumption, pro-
sumption: the nature of capitalism in the age of the Definitions
digital ‘prosumer’. J. Consum. Cult. 10, 13–36 (2010)
Schott, B.: Playbor. The New York Times. Available Game thinking (or gamification) is related to
https://schott.blogs.nytimes.com/2010/03/12/playbor/ game-oriented thinking, in processes that are not
(2010). Accessed 29 Jan 2018
Toffler, A.: The Third Wave. Bantam Books, New York games, aiming to reach a specific objective and
(1980) using engagement mechanisms.
Xie, C., Bagozzi, R.P., Troye, S.V.: Trying to prosume: Game design thinking is focused on the design
toward a theory of consumers as co-creators of value. and development of games based on design think-
J. Acad. Mark. Sci. 36, 109–122 (2008)
ing methodology.

Introduction
Game State Validation
The concepts of game thinking (or gamification)
▶ Secure Gaming: Cheat-Resistant Protocols and and game design thinking are commonly mixed,
Game History Validation although both have different proposals and
Game Thinking X Game Design Thinking 781

Game Thinking X Game

Design Thinking,
Fig. 1 Gamification or
game thinking process
(Reproduced from
Manrique 2013)

applications (Deterding et al. 2011; Castro 2013; Thinking and Game Design Thinking Method-
Jewell 2016; Sailer et al. 2017). Marczewski ologies” addresses the main issues related to G
(2014) performed an analysis involving the main game thinking and game design thinking. sec-
differences between game design and game think- tion “Game Design Thinking” presents the
ing and points out that, while the game design is game design methodology proposal, and,
related to something that entertains people in a fun finally, section “Conclusion” presents the final
way, the game thinking proposal is to do some- considerations.
thing in order to achieve a specific goal.
The term game thinking relates to the use of
game elements in contexts that do not necessarily Game Thinking and Game Design
consist of games and/or digital media resources Thinking Methodologies
(Currier 2008; Alves 2008). Paffrath and Cassol
(2014) argue that game thinking is related to Both game thinking and game design thinking
understand different aspects of human psychol- have their own goals. The game thinking focuses
ogy, such as the mechanisms of personal motiva- on pushing the participant toward their business
tion and the concept of fun. goal, while the game design thinking explores
Game design, on the other hand, explores mechanics and gameplay, to make the game
mechanics and gameplay, among other things, to enjoyable, employing design thinking as a game
make the game enjoyable (Silva and Bittencourt development methodology.
2016). In this sense, game design thinking is a For game thinking, there are different applica-
methodology focused on the design and develop- tion proposals in the literature (Liu et al. 2011;
ment of games based on adapted concepts of Werbach and Hunter 2012; Guin et al. 2012;
design thinking and that can be developed in a Landers and Landers 2014). These proposals
significant way to the process of design and game apply some common steps to the game thinking
development (Gestwicki and McNely 2012). methodology: definition of challenge and goals,
Considering these aspects, this article dis- identification of target behaviors, players’ under-
cusses the main differences between game think- standing, and specification of the fun activity.
ing and game design thinking and presents a Figure 1, proposed by Manrique (2013), summa-
methodology proposal for game design based on rizes these steps.
design thinking and agile and lean project man- The first step is the definition of the challenge
agement concepts. We also discuss the results and goals that will drive the game thinking pro-
obtained from the application of such methodol- cess. In addition, it is necessary to understand the
ogy in classrooms of undergraduate courses profile of the players in order to plan activities that
related to the game design. motivate them. In this context, the definition of
The text is organized as follows: In addition fun activities is the core of the gamification
to this introductory section, section “Game process.
782 Game Thinking X Game Design Thinking

On the other hand, in the game design process, common constraints involving design thinking:
formalisms and methodologies are needed in desirability, feasibility, and feasibility.
order to optimize, streamline, and professionalize This section presents an evolution of our pre-
the game development (Hunicke et al. 2004; Bem vious methodology for game design based on
et al. 2014; Kristiansen and Rasmussen 2014; design thinking (Silva and Bittencourt 2016).
Jewell 2016). It is noted that the game design The original proposal was adapted in order to
still requires specific methodologies, capable of following the iterative cycle of refining designs
giving developers the understanding of the game and getting user feedback as proposed by Wagner
development process in a complete way. and Piccoli (2007).
Gestwicki and McNely (2012) argue that the
adoption of design thinking as a methodology for The Methodology
the game development is adequate because it pro- This methodology is divided into four main
vides immersive process based on research, bring- stages: team definition, conception, prototyping,
ing academic objectives closer to business and validation. As can be seen in the Fig. 2, after
environments. In addition to immersion, the gen- the team definition starts, an iterative cycle of
eration of ideas, the possibilities prototyping and conception, prototyping, and tests is realized.
the selection of solutions are characteristics of The team definition is a fundamental step to
design thinking that can be adapted to the game reach the objectives of the game design, due to its
design. According to Vianna et al. (2012), design interdisciplinary character related to the different
thinking is originated from the need to seek new stages of conception and prototyping. Team mem-
paths to innovation from the human-centered bers should interact with each other, identifying
approach, where multidisciplinary, collaboration opportunities and developing creative and inno-
and the tangibilization of thoughts and processes vative solutions for the game development pro-
lead to innovative solutions. cess. Thus, the team must know each other,
However, while the game thinking area pre- focusing on the personality of the members as
sents methodologies that have a common and well as their professional aptitude. For this first
well-founded basis, the area of game development stage, two strategies are proposed: influence map
lacks proven methodologies, which help game and T-shaped profile (Glushko 2008).
designer to think about the creation process of In addition to the game objectives, the game
the game. In this sense, the next section presents conception depends on four basic premises: inspi-
a proposal of a game design thinking ration, creativity, innovation, and identification of
methodology. tendencies. Such premises must be present in all
phases of conception: divergent ideation, immer-
sion, analysis and synthesis, and convergent ide-
Game Design Thinking ation. It is noted, therefore, that the conception
begins and ends based on the creative process and
The game design is characterized by the constant its two types of thinking: divergent and
need for innovation and reinvention in order to convergent.
meet new audiences, new static, new experiences, Divergent ideation seeks to create options,
and new technologies. The professional who from a significant quantity and diversity of
works in the game design must be able to integrate ideas, in order to promote different possibilities
different aspects necessary to the proposal of a of game design for the immersion phase. Thus,
game, which has a multidisciplinary nature. In different brainstorming techniques can be
this sense, the area of design thinking has added applied, as the method 635 (Rohrbach 1969),
quality to the game design, since it is an active and the mental map (Buzan and Buzan 2006), the
user-centered methodology in addition to pre- heuristic ideation technique (Gray et al. 2010),
aching the culture of innovation. The latter can and game idea generator tool (Riftpoint
be considered as the intersection between three Entertainment 2017).
Game Thinking X Game Design Thinking 783

Game Thinking X Game Design Thinking, Fig. 2 Game design thinking (Adapted from Silva and Bittencourt 2016)

After the divergent ideation, the immersion between them. Positioning and impact matrices
stage identifies the needs and opportunities that also can be applied at this stage to the defini-
will guide the generation of solutions in the next tion of criteria such as time, complexity, inno-
phase of the project. For the game development, vation, costs, team members’ abilities, and
four techniques are suggested at this stage: explor- motivation among others.
atory and desk researches, moodboard, and per- The convergent ideation closes the game con-
sonas (Vianna et al. 2012; Gray et al. 2010). These ception phase and is characterized by making
tools help in the understanding of the game sce- choices based on existing alternatives related to
nario, through information about the theme of the the game. In this step, the ideas generated in the
project from diverse sources (websites, books, previous steps are critically analyzed and judged
journals, blogs, papers among others), and the so as to select them based on previously defined
identification of market niches, with little explo- criteria, expanding the original ideas. In this stage,
ration and potential for expansion. it is proposed to use two main tools for the con-
In the analysis and synthesis step, definitions vergent ideation of the game: game model canvas
related to the game must be made. In addition (Jimenéz 2013) and game design canvas (Sousa
to the data gathered in the exploratory research 2014). Another tool that can help the generation of
and the moodboard panel, cards created in the new ideas is the game genesis virtual deck
desk search can be arranged in an affinity dia- (Gehling 2016). These tools allow the analysis
gram to identify similarities and patterns of the game in a systemic, integrated and fast
784 Game Thinking X Game Design Thinking

way, providing insights on how the team should definition of the game concept in an agile and
act in order to compose the main idea. It thus helps efficient way, summarizing the game design in
game developers build, differentiate, and innovate panels which facilitate the identification of trends,
processes, improving their business model to win creativity, inspiration, and innovation among
over the target audience and gain profits. team members.
During prototype and tests, four major actions Some groups that started the game design
should be considered: rapidly prototype, publi- thinking process with a pre-defined concept of
cize/publish the game for feedback and evalua- their game felt motivated to rethink this. During
tion, feedback analyze, and make improvements if the prototype stage, the adoption of visual think-
necessary. However, before the prototyping, it is ing and/or narrative allowed a complete visuali-
important to align the development process with zation of the flow and mechanics of the game.
the team members, and two resources can be used Later, the construction of the game based on
for that: visual thinking (Ware 2008) and story- rapid prototype promoted the initial understand-
telling (Rouse 2000). Once the process is clear to ing about mechanics and interaction proposed
team members, the game’s implementation begins before the development of the complete
by choosing technologies that are easily inte- prototype.
grated and make it possible to obtain the final In some cases, the game design thinking results
designed product. motivated groups to abandon the idea of digital
Finally, the game developed must be published gaming and invest in a board game, given that the
in order to get feedbacks and evaluation of player board proposal exceeded the digital proposal in
users. For this, remote and face-to-face usability terms of gameplay.
tests supported by the Likert scale (Likert 1932)
are recommended. According to Nielsen (1994),
usability comprises five dimensions: learning, Conclusions
memorization, errors, efficiency, and satisfac-
tion/acceptance. Based on the results of testing, This article discusses the main differences
the most recent iteration of a game design, between the terms game thinking (gamification)
changes, and refinements are made, and a new and game design thinking. While for gamification
cycle starts. there are a significant number of proposed and
proven methodologies, for game design there are
Case Studies and Results few methodologies that really point to design and
This methodology was applied in 15 undergrad- development stages in an agile and lean way,
uate courses of different institutions of higher stimulating inspiration, creativity, innovation,
education since 2014. Through an empirical and identification of trends.
analysis, it can be observed that the methodol- Then, a methodology based on design thinking
ogy of game design thinking allowed groups of for the game design is presented and discussed in
students to work in a “free” , focusing on high practice, with the report and analysis of case stud-
level concepts. ies results. The results obtained from case studies
Thus, as the flow of the methodology pro- show the generation of more interesting games
gressed, the students had more creative ideas, when compared to games developed without the
connecting points not so obvious from the initial use of a methodology that stimulates creativity
premises. The phase of divergent ideation allows and innovation such as design thinking
the teams an enriching brainstorming, “opening methodology.
their minds” for the generation of game concepts. This study intends to contribute to the area of
In the immersion stage, there is an evolution in the digital game design and development, either pro-
students’ original ideas, while the analysis, syn- fessionally or in activities that involve the learning
thesis, and convergent ideation stages allow the process.
Game Thinking X Game Design Thinking 785

Cross-References Jewell, D.: Game-Design Thinking in Education and

Beyond. Online & Blended Learning, Pearson. http://
www.pearsoned.com/education-blog/game-design-think
▶ Challenge-Based Learning in a Serious Global ing-in-education-and-beyond (2016). Accessed 12 April
Game 2018
▶ Gamification Jimenéz, S.: Gamification Model Canvas. Game On! Lab.
▶ Game Development Leadership Tips http://www.gameonlab.com/canvas/ (2013). Accessed
12 April 2018
▶ Gamification and Serious Games
Kristiansen, P., Rasmussen, R.: Building a Better Business
Using the Lego Serious Play Method. Wiley, New
Jersey (2014)
References Landers, R.N., Landers, A.K.: An empirical test of the
theory of gamified learning: the effect of leaderboards
Alves, L.R.G.: Games e educação – a construção de novos on time-on-task and academic performance. Simulat.
significados. Revista Portuguesa de Pedagogia. 42(2), Gaming. 45(6), 769–785 (2014)
225–236 (2008) Likert, R.: A technique for the measurement of attitudes.
Bem, R.F.S., Alquete, T., Martins, V.F.: Game design – Archiv. Psychol. 22, 1–55 (1932) G
Geração de Alternativas, Técnicas Criativas e Suas Liu, Y., Alexandrova, T., Nakajima, T.: Gamifying intelli-
Ferramentas. In: Proceedings of XIII Brazilian Sympo- gent environments. In: Proceedings of International
sium on Computer Games & Digital Entertainment ACM Workshop on Ubiquitous Meta User Interfaces,
(SBGames). SBC, Porto Alegre (2014) Scottsdale (2011)
Buzan, T., Buzan, B.: The Mind Map Book. BBC Active, Manrique, V. A Complete Guide to Gamification Design.
Harlow, Essex (2006) TechnologyAdvice. https://technologyadvice.com/
Castro, D.: Gamificação da pedagogia: entenda como os blog/information-technology/a-complete-guide-to-ga
jogos podem auxiliar no processo de aprendizagem. mification-design/ (2013). Accessed 12 April 2018
Empresa Brasil de Comunicação (EBC). http://www. Marczewski, A.: Gamification Design vs Game Design.
ebc.com.br/tecnologia/2013/01/gamificacao-da-pedag Gamasutra, The Art & Business of Making Games.
ogia-como-os-jogos-podem-auxiliar-no-processo-de-a http://www.gamasutra.com/blogs/AndrzejMarczewski/
prendizagem (2013). Accessed 12 April 2018 20140422/215939/Gamification_Design_vs_Game_
Currier, J.: Gamification: Game Mechanics is the New Design.php (2014). Accessed 12 April 2018
Marketing. Ooga Labs. http://bit.ly/1hCETKp (2008). Nielsen, J.: Usability Engineering. Academic Press,
Accessed 12 April 2018 Londres (1994)
Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From Paffrath, R.E., Cassol, V.J.: Gaming abroad: o uso de
game design elements to gamefulness: defining gamificação no projeto de um sistema para Apoio a
“gamification”. In: Proceedings of the 15th Interna- Turistas. In: Proceedings of XIII Brazilian Symposium
tional Academic MindTrek Conference: Envisioning on Computer Games & Digital Entertainment
Future Media Environments (MindTrek ‘11), (SBGames). SBC, Porto Alegre (2014)
pp. 9–15. ACM, New York (2011) Riftpoint Entertainment: The Amazing Game Idea Gener-
Gehling, M.: Game genesis virtual deck: uma ferramenta ator. https://riftpoint.itch.io/the-amazing-game-idea-
para criar ideias de jogos. In: Proceedings of XV Bra- generator (2017)
zilian Symposium on Computer Games & Digital Rohrbach, B.: Kreativ nach Regeln – Methode 635, eine
Entertainment (SBGames). SBC, São Paulo, SP (2016) neue Technik zum Lösen von Problemen: creative by
Gestwicki, P., McNely, B.: A case study of a five-step rules – method 635, a new technique for solving prob-
design thinking process in educational museum game lems. Absatzwirtschaft. 12, 73–53 (1969)
design. In: Proceedings of Meaningful Play, East Lan- Rouse, R.: Game Design Theory and Practice, 2nd edn.
sing, MI, USA (2012) Wordware Publishing, Plano (2000)
Glushko, R.J.: Designing a service science discipline with Sailer, M., Hense, J.U., Mayr, S.K., Mandl, H.: How
discipline. IBM Syst. J. 47(1), 15–38 (2008) gamification motivates: an experimental study of the
Gray, D., Brown, S., Macanufo, J.: Gamestorming. effects of specific game design elements on psycholog-
A Playbook for Innovators, Rulebreakers, and ical need satisfaction. Comput. Hum. Behav.
Changemakers. O’Reilly Media, Massachusetts (2010) 69, 371–380 (2017)
Guin, T.D., Baker, R., Mechling, J., Ruyle, E.: Myths and Silva, I.C.S., Bittencourt, J.R.: Game thinking is not game
realities of respondent engagement in online surveys. design thinking! Uma proposta de metodologia para o
Int. J. Mark. Res. 54(5), 1–21 (2012) projeto de jogos digitais. In: Proceedings of XV Bra-
Hunicke, R., Leblanc, M., Zubek, R.: MDA: A formal zilian Symposium on Computer Games & Digital
approach to game design and game research. In: Pro- Entertainment (SBGames). SBC, São Paulo, SP (2016)
ceedings of the AAAI Workshop on Challenges in Sousa, T.C.: Game Design Canvas. http://abxygames.
Game, AAAI Press, (2004) wixsite.com/gdcanvas (2014). Accessed 12 April 2018
786 Game Usability

Vianna, M., Vianna, Y.R., Adler, I.K., Lucena, B., Russo, the living room in front of the TV? The location of
B.: Design $Thinking: Inovação Em negócios. MJV the “venue” of a game can be a direct result of how
Press, Rio de Janeiro (2012)
Wagner, E.L., Piccoli, G.: Moving beyond user participa- successful the game can become. If we were to
tion to achieve successful is design. Commun. ACM. design a game, the decision of what platform –
50(12), 51–55 (2007) console, PC, tabletop, mobile, or tablet – is an
Ware, C.: Visual Thinking: For Design. Morgan important decision to make. For centuries,
Kaufmann, Burlington (2008)
Werbach, K., Hunter, D.: For the Win: How Game Think- humankind has revolved around many different
ing can Revolutionize your Business. Wharton Digital aspects when it comes to the social aspects of
Press, Philadelphia (2012) entertainment.
Studying history we can see how our choices
for entertainment and comfort have changed. Just
look at the evolution of the TV going from some-
Game Usability thing that we see in stores and restaurants for
public entertainment to be a staple inside of the
▶ Player Experience, Design and Research poorest of homes in America. The same can be
said about games and the platforms they are run-
ning on. In the 1980s, a game console was rarely
found in a household, but today, more than 89 mil-
Game Venues lion American households own at least one type of
game console or a PC. Over 40% of them have
▶ Game Venues and Platforms multiple types of gaming consoles or platforms
(Statista 2018).
The type of games can affect the kind of venues.
For a game that tells a grand story with videos and
Game Venues and Platforms cinematic scenes, a console connected to a big
screen in the living room is a good choice.
Michael McMillan2 and Newton Lee1,2 A game that brings people together to spend quality
1
Institute for Education, Research, and time playing would call for a tabletop game, such
Scholarships, Los Angeles, CA, USA as cards or board games. For a massively multi-
2
Vincennes University, Vincennes, IN, USA player game connecting hundreds of people around
the world, a PC would be the best platform.
The main issue with consoles is that it can be
Synonyms limiting and expansive at the same time. Cur-
rently, there are three companies who are at con-
Game platforms; Game venues stant battle with one another to dominate the
console war: Microsoft with their Xbox One
X the console which has the fastest graphics and
Definitions processing speed, Sony with the PlayStation 4 Pro
that is a serious contender to Microsoft with their
Game venue ¼ a physical place where one or console exclusive games, and Nintendo Switch
more persons can play a video game. which has the ability to seamlessly swap from
Game platform ¼ a device suitable for playing console on the big screen to mobile handheld
video games, e.g., a console, a smartphone, or console in an instant without compromising the
a PC. current gaming activity.
What venues do we think of when we think It is ideal for a game to run on all platforms.
about playing games? The dining room table with However, if, say Sony PlayStation decides to make
a board game, the computer desk in our office, or a game its console exclusive, it would take away
Game Venues and Platforms 787

the three-company bliss and cut it down to one, that are built for gaming have a high-quality
literally reducing the game market size by 66%. graphics card and processing power to be able to
Another consideration is input devices for play games with better graphics quality and reac-
game platforms. Consoles offer a more limited tion time. Nevertheless, Microsoft is closing the
control through a controller whereas a PC key- gap on PCs by adding keyboard and mouse sup-
board allows a wider range of controls. The Xbox port on Xbox One in 2018.
and the PlayStation controllers only has 12 indi- While mobile and tablet game platforms are
vidual buttons with 2 joysticks, which provide a less powerful than consoles and PCs, they have
range of combinations but are less than an average a built-in touchscreen and other features that
PC with a keyboard of 62 buttons plus a mouse make them unique. The large amount of people
(Rigg 2018). Depending on basic versus gaming that own a smartphone means that mobile
equipment, the mouse itself may have extra but- games have a large potential reach. The ability
tons on it, making it the best contender for button to play games anywhere enables the creation of
combos and linking actions to buttons. The games that integrate GPS technology. Another G
PlayStation Controller does have one advantage application that can be used by game devel-
against its competitors, a large Touchpad at the top opers is the multiple cameras smartphones pos-
of the controller. It is designed to allow the player sess to run augmented reality, where graphics
to swipe and click on information very much like generated by software is overlaid from pictures
a laptop touch pad. So far it seems to only be from the camera. Games such as Pokémon Go
useful for menu navigation, but with the redesign has used all of these technologies to allow
Sony was planning on a future with new innova- players to travel to a place in real life and
tive games which will include the Touchpad into then catch a Pokémon using the touchscreen
their game. Nintendo Switch comes with two con- and augmented reality. For games that would
trollers, Joy-Con L and Joy-Con R, each of which require a proper controller, games can utilize
contains an accelerometer and gyroscope for the smartphone’s bluetooth technology to allow
motion control support. All three companies a user to buy a third-party controller, pair it
have designed motion control systems before, with the smartphone, and then affix the phone
Microsoft with the Kinect from Xbox 360 and to the top or middle of the controller. Likewise,
Xbox One, Sony with the Playstation Move for iOS and Android operating systems allow their
the Playstation 4, and Nintendo who has been users to project their screen onto another device
doing it the longest with the Wii, Wii-U, and for a better viewing experience.
now finally the Switch. The Switch Joy-Con’s
are possibly the most accurate out of any compa-
nies attempt at motion control. The actions with
the controller are spot-on: if you overexaggerate Cross-References
your actions, the results appear on screen as over-
reacted. It is a finely tuned gyroscope inside of the ▶ Video Games
controller. Not to mention that 9/10 games pro-
duced for the Nintendo Switch make use of this
motion control, unlike many predecessors which References
only had a few games made to use the motion
Rigg J.: With keyboard and mouse support on Xbox,
control. Microsoft closes the gap on PCs. (2018). https://www.
There are also arguments between gamers engadget.com/2018/11/15/microsoft-xbox-keyboard-
about which one offers a better reaction time: the mouse-pc/
Statista: Monthly number of game console users in the
joystick on a controller versus a PC’s mouse. Most
United States from 2nd quarter 2012 to 2nd quarter
gamers agree that the PC’s mouse outperforms the 2017 (in millions) (2018). https://www.statista.com/
console’s controllers. In addition, most computers statistics/320315/number-users-game-consoles-usa/
788 Game with a Purpose

Overview
Game with a Purpose
A writer does not have an enviable role in video
▶ Hypermedia Narrative as a Tool for Serious games. Sure, game writing beats writing copy for
Games a hardware catalogue. And it is more fun than
writing dry text for, say, an academic publication.
(Irony acknowledged.) But in the world of video
games, writing is often a misunderstood craft and,
as such, is often shunted to the side until the very
Game Writer’s Dilemma: last minute of a game’s development.
Context vs. Story There are many reasons for this, including (1)
people think, because they have the ability to
Ross Berger write sentences, that they have the ability to
Vistance Consulting, Los Angeles, CA, USA write a narrative; (2) writing does not involve
lines of code, thus words are easier to adjust at
the last minute to fit features that are perceived to
Synonyms be cool at the time; and (3) gameplay is king and
thus story must be subservient to it.
Context; Game writing; Narrative; Narrative The last point is unassailable. Games are a
design; Storytelling gameplay-driven medium. If one wants to work
in this field as a writer, the first thing he/she must
understand is that narrative will need to serve
Definitions gameplay, not vice versa.
But the other two points are unnecessary yet
Narrative is a fictional, cohesive universe made up frequent obstacles that often hinder a writer. Fake
a mythology of characters and settings bolstered understanding of the craft is common and dis-
by a story (but often a series of stories), theme, and misses its burden. A nonwriter’s attitude is simply,
tone. Story depicts a character’s journey that, “I write emails and texts all the time. It’s so easy.
through a series of escalating challenges and Anyone can write.” Obviously, this facile com-
accomplishments, results in that character’s self- ment ignores thousands of years of story innova-
discovery. It is part of a narrative, but, by itself, is tion including compelling characters, plot,
not. Context is descriptive, light information that structure, and theme. More importantly: crafting
provides a player with justification as to why words to communicate these requirements of sto-
he/she is about to take on a certain task or pursue rytelling is not the same as crafting an email.
a certain goal. It is also filler or casual stimuli, like (If so, why not then abnegate the works of
ambient dialogue, that provide authenticity to a Homer and Shakespeare?) But what is dangerous
world. about this attitude is that it lowers quality expec-
tations, encourages amateurs to assume the role of
writer when companies are crunched for budget,
and does not allow for producers to allocate
Introduction enough time or personnel for story development.
Writing story is also viewed as a lesser craft
There is an industry-wide confusion over the role than writing lines of code. Accordingly, writers
of narrative in games. Often times, it is confused are often pushed to make dangerous compromises
with context. This entry will define “narrative,” on story in order to avoid upending a feature that
“story,” and “context” and provide how their is hard-coded into the game. It is foolish to say
applications in the game experience are distinct that storytelling is harder than programming.
from one another. Because it is not. But it is also foolish to say that
Game Writer’s Dilemma: Context vs. Story 789

storytelling is easily flexible, so moldable to As they should. But “narrative” is not universally
whim. Because it is not. Yet, it is often “strong- understood. Each game studio, in fact, might have
armed” by whim to change ad infinitum. Such their own definition. To be clear:
arbitrary demands are myopic and cavalier, and
often lead to the compromise of quality. Story Narrative is not text on screen.
beats will be abbreviated and then combined into Narrative is not writing 27 different ways of say-
a later one, at which time a player will be deluged ing, “Take cover!”
with too much information. Even worse, casually Narrative is not mission descriptions.
thought-out rescoping can lead to serious coher- Narrative is not tutorials.
ence issues.
All of these pitfalls are a result of one common What narrative is, instead, is a fictional uni-
problem: a profound misunderstanding of what verse made up a mythology of characters and
narrative is. settings bolstered by a story (but often a series of
This entry will focus on a particular type of stories), theme, and tone. G
misunderstanding (yes, there are many) that does It does not stop there. Narrative is also respon-
not necessarily over-tax the writer, but nor does it sible for the granular details that buttress these
empower him/her to maximize the craft to its pillars and provide necessary information, authen-
fullest capacity. It is the misunderstanding of the ticity, and orientation for a player’s journey.
term “context.” So prevalent is its confusion with It is the overemphasis on the granular
narrative that it is the intention of this entry to details, however, which the majority of game
correct the record once and for all. companies incorrectly define narrative to be:
the minutiae.
What Is Narrative and How Does It Get So yes, narrative does involve: text on screen;
Misused 27 different ways of saying, “Take cover”; mis-
In the recent history of game development, sion descriptions; and tutorials. But without a
narrative has become a “nice-to-have” feature central, cohesive fictional universe, these discrete
in an industry where gameplay reigns supreme. parts do not add up to narrative.
Of course, there are a multitude of games that Individually, they are, instead, context.
are narrative-centric or, at least, that push story
as a major feature. (Telltale Games, for What Is Context?
instance.) Context is descriptive information that provides a
But narrative still remains, for a significant player with justification as to why he/she is about
majority of games, an after-thought to game to take on a certain task or pursue a certain goal. It
design due to popular expectations from is also filler or casual stimuli, like ambient dia-
players. Disruption of the gameplay experience logue, that provide authenticity to a world. (Think
is a high crime, and narrative is often seen that of combat chatter in Call of Duty.) Context is light
way. Even if the vehicles that deliver narrative information and does not provide enough connec-
are interactive (like cinematics) and offer direct tive tissue from one moment to the next to be
incentives that enhance gameplay (boosts, for defined as narrative.
example), players often do not look forward to Context is necessary in the sense that, without
these mini departures. Writers do not welcome it, players will not understand why they are about
this approach either. Narrative gets diluted to embark upon a certain part of a journey or what
when it offers incentives to a player just to the immediate goal will be.
validate its existence. Context also has an important role in
For the most part, games are not a narrative- reinforcing the fictional universe, be it through
driven medium. communicating the personalities of characters of
Nonetheless, this does not stop game studios the game, backstory of the world, or game rules.
from wanting to include narrative in their games. Tone is also a critical driver of context and vice
790 Game Writer’s Dilemma: Context vs. Story

versa. For instance, Rolf and Jenny land their ship connective tissue that binds those elements
on a desolate planet, hoping to find a lost ship of together. Where story needs a protagonist, nar-
fellow explorers. Rolf and Jenny are a married rative needs a world or universe of which that
couple and are ambitious scientists with a pen- protagonist is a part. Where story moves for-
chant for puns. When they come across a volcano ward through plot, narrative moves forward
on the verge of eruption, on-screen text tells them through mythology and world building. Where
they must cross the volcano to reach the other side story is the single journey of a single hero,
of the mountain, where a light beacon assures that narrative is potentially an endless series of jour-
they will reunite with one of the lost crew mem- neys and incorporates many heroes.
bers. Here we have an excellent opportunity to
infuse the on-screen text with some of Jenny and Why Is Not Context Story?
Rolf’s humor. “They’ve studied the topology of The light information that context provides is
this planet before. Therefore, the volcano’s vola- critical to orienting a player on their path forward.
tility is no surprise. The only test here is their lava But it is not an ideal vehicle for story as it can
for one another.” hamper a player’s momentum. Stopping to read
This type of context informs a player of the long tomes of story will engender resentment
following (1): Rolf and Jenny have an extensive from most players. And bite-sized, frequent
knowledge of the planet; (2) Rolf and Jenny knew appearances of on-screen text are met with equal
of the planet’s volcanic activity prior to landing; disapproval. These instances happen more fre-
and (3) Rolf and Jenny have a terrible sense of quently than one can imagine.
humor. Context should never be expected to do the
As long as it reinforces what this world is about heavy lifting for story. A few reasons why (1) a
and remains consistent, tone can go a long way in mission could be added after the development of
adding more flavor to context and, in turn, the story has been “baked” and is therefore inele-
reinforcing the narrative even if through second- gantly shoe-horned into the core progression;
ary impact. (2) one-off activities (e.g., sports drills, tutorials,
missions, live raids, etc.) may not be thematically
What Is Story and How Does It Differ from or structurally related, and therefore imposing a
Narrative? faint connection that ties them together will, most
Story depicts a character’s journey that, through a likely, dilute or neuter story; (3) the game is com-
series of escalating challenges and accomplish- prised of an endless stream of missions or grind-
ments, results in that character’s self-discovery. ing activities in which ongoing story would fail in
Upon that self-discovery (also known as anagno- quality or logic to serve as an effective
risis from Aristotle’s Poetics), a character will complement.
face a big decision: Will they change for the However, context would still be appropriate to
better? Will they continue on their path but with give a player an understanding of basic stakes and
larger challenges ahead? Will they do something consistency to the world that was already set up in
selfless for the betterment of those around them? the narrative (assuming a narrative was
Story is a progression of a hero. Everything he established in the first place).
or she encounters along the way should enhance But due to its nature, context does not move the
the stakes of that progression and provide more story forward. It provides ancillary, and some-
insight into the hero’s psyche and past. times inconsequential, information that keeps a
Story is a critical part (often times, the player aware of stakes. But there is nothing in
majority) of a game’s narrative. But story, by that information that contributes anything emo-
itself, is not narrative. Narrative is the bible that tional or meaningful to the hero’s psychological
is comprised of a series of related stories, char- journey, the main force behind story.
acters, and environments. Narrative is the fic- Context is narrative in its lowest, most
tional universe, the elements inside, and the unambitious form. It is one step above stimulus.
Game-Based Interventions in Public Health: Exploiting the Engaging Factor of Gameplay 791

Cause for Confusion References

The most common use of context in games is
through on-screen text and informational voice- Books
over, which are very attractive for game devel- Aristotle, et al.: Aristotle Poetics. Brill, Leiden, (2012)
opers as they provide low cost solutions for
tutorialization and player orientation. Games
Call of Duty: WW2. Activision Publishing, Inc., Raven
But to label words on screen or informational
Software, Sledgehammer Games, 2017
voice-over, without having any other elements of
narrative, as narrative is to diminish the craft of
writing. This oversight or lack of understanding
by video game professionals is neither malicious
nor negligent. One can attribute the prevalence of Game Writing
the misuse of these terms to the rise of social
media. ▶ Game Writer’s Dilemma: Context vs. Story G
Due to the increased volume and accessibility ▶ Video Game Storytelling Fundamentals: Set-
of content creation tools on mobile, terms such as ting, Power Status, Tone, and Escalation
“story” and “narrative” have become less clear
now that social media companies have co-opted
those terms to mislabel new digital products.
Snapchat My Story, for instance, is often a collec- Game-Based Approach
tion of disconnected videos that do not build to a
cohesive message. Instagram Story is no different ▶ Game-Based Interventions in Public Health:
in result (other than it features photos only). It is Exploiting the Engaging Factor of Gameplay
the culture – not the game developer – that has
obfuscated the meaning of story.

Game-Based Interventions
Conclusion in Public Health: Exploiting
the Engaging Factor
Narrative, story, and context are often used syn- of Gameplay
onymously in game development; however, as
described above, they are not the same. As Sylvester Arnab
media changes, so does the craft of writing. Keep- Disruptive Media Learning Lab, Coventry
ing up with how stories are told year after year University, Coventry, West Midlands, UK
remains a challenge for writers. So, too, does the
ability to communicate the distinctions between
narrative, story, and context. Synonyms
Video game story professionals need to tackle
that task effectively. Doing so will result in real- Game-based approach; Games for health;
istic expectations from engineers, designers, and Gamification; Health games; Serious games
producers, and will empower writers to advocate
for narrative in ways that it deserves.
It is the hope of this author that nonwriting Definition
professionals in game development will soon
understand that pushing narrative to its artistic Game-based intervention in public health:
maximum (including its original intention) will exploiting the engaging factor of gameplay is
result in higher quality games, no matter the the application of game science, techniques, and
genre. technologies for supporting public health
792 Game-Based Interventions in Public Health: Exploiting the Engaging Factor of Gameplay

interventions, specifically focusing on serious others have continued this approach, leading
games and gamification approaches. Serious games for change projects that raise awareness
games refer to digital game technologies that and support good causes or try to change behavior
serve a purpose other than pure entertainment, for social purposes. Self-regulation and health
and gamification is a technique that exploits coaching, for instance, have a big potential to
game concepts and mechanics in non-game con- empower individuals, and combined with mobile
texts in order to motivate engagement and sustain and social platforms, game-based approaches
participation in serious activities. could facilitate community building and peer sup-
port system around healthy lifestyles in a more
pervasive and active way.
Introduction The need for efficient and effective educa-
tion of healthcare professionals has also seen
The increase of health impediments due to game-based approaches employed in a diverse
unhealthy lifestyle has put a lot of pressure on range of forms to address training needs, while
public health spending. Even though public health in a clinical setting, games have been used to
programs are important in raising awareness improve therapeutic outcomes for patients.
toward lowering the prevalence of physical and Fundamental to the success of game-based
psychological health complications, the general intervention across these areas is the ability of
public should be made accountable for their own designers to realize the power of interactive and
health and well-being. The approaches by which immersive environments to engage and
awareness is raised, attitudes and behaviors are immerse the users while conveying learning
transformed, and positive habits are nurtured outcomes in a demonstrably effective fashion.
should be improved to be more effective, which Research, therefore, must play a key role in
could potentially ease pressure on public health identifying the strengths, weaknesses, and best
services in the long run. practices in the use of game technologies and
The level of receptiveness to public health techniques in the healthcare sector, providing
awareness programs is highly subjective to decision-makers with the evidence they need to
whether positive engagement, persistent involve- consider their value as a solution.
ment, and discourse can be fostered. There is a With these perspectives, this chapter provides
growing interest in improving and sustaining an overview of the implications of using game
engagement with such programs across the concepts and technologies for supporting health
healthcare sector using technologies, such as dig- interventions and discusses key development
ital games. Games such as America’s Army, for trends and challenges. The next two sections,
instance, are able to reach a large number of respectively, introduce serious games and
players and engage them for long periods of gamification initiatives, discuss their contribu-
time, which has encouraged gaming to be used tions to and potential in health interventions, and
to achieve serious outcomes, such as cognitive conclude with lesson learned and trends in the
gain, awareness raising, and change of attitude domain. The conclusion section summarizes and
and behavior. highlights key takeaways from this chapter.
Games as positive technology capitalize on its
engaging and fun characteristics toward fostering
positive emotions, nurturing positive habits, Serious Games and Health
encouraging positive attitude and behavior, and
promoting optimal human functioning. Game By definition, serious games refer to applications
approaches can be used as an enabling tool developed using computer game technologies that
and/or concept to promote qualities that could serve purposes other than pure entertainment. The
enable individuals and communities to strive for term has been used to describe a variety of game
and build the best in life. McGonigal (2011) and types, particularly those associated with
Game-Based Interventions in Public Health: Exploiting the Engaging Factor of Gameplay 793

Game-Based
Interventions in Public
Health: Exploiting
the Engaging Factor
of Gameplay,
Fig. 1 Screenshot from
“Privates” game (www.
sizefivegames.com)

G
e-learning, military simulation, and medical train- Recognizing the increasing popularity of digi-
ing. Serious games capitalize on their ability to tal games in health applications, there is a need for
reach target audiences who are already engaging empirical studies to be carried out that can serve as
with interactive, rich graphic games benchmarks for establishing scientific validity for
recreationally, allowing them to convey instruc- the efficacy of such an approach. This is a critical
tional outcomes to audiences resistant to more trajectory for the application of games within the
formal methods. healthcare contexts as to encourage uptake within
The application of games within the health formal deployment; existing initiatives should
sector ranges from tackling sexual and mental move “away from the evangelistic early stage
health to promoting healthy living and raising work to the practicalities of implementing and
awareness on pharmaceutical processes. The Pri- testing game technologies in real contexts of
vates game (Fig. 1), for instance, has been use” (Arnab et al. 2012). For example, several
commissioned by UK’s Channel 4 TV Company important studies have been undertaken that
to engage and educate young people on sexual have shown the efficacy of game-based
health issues. Other entities such as the Parliamen- approaches over traditional approaches (e.g.,
tary Education Group, DEFRA, and the US gov- Hainey et al. 2011; Kato et al. 2008; Knight
ernment (who held a competition around games et al. 2010; Marchiori et al 2012; Brown
for health) are also increasingly commissioning et al. 2012; Arnab et al. 2013). The first controlled
games for learning purposes (Ulicsac 2010). trials for game-based intervention (Kato
Physical and mental rehabilitation has been pro- et al. 2008) showed how game-based approaches
moted via the PlayMancer game, which was in the Re-Mission game (Fig. 2) fostered medica-
awarded the Best European Health Serious tion adherence in children with cancer.
Game in 2011 at the Fun & Serious Games Festi- Other examples include a game-based inter-
val in Spain. Most recently, the EU-funded vention to support the delivery of relationships
PEGASO project exploits game technologies to and sex education (RSE) program (Fig. 3), a
support an ecosystem of beneficiaries in game called PR:EPARe (Positive Relationships:
addressing challenges related to obesity, a world- Eliminating Coercion and Pressure in Adoles-
wide public health problem (Panese et al. 2014). cent Relationships) developed by the Serious
Inspired by the success of FarmVille, a pharma- Games Institute and the Studies in Adolescent
ceutical brand Boehringer Ingelheim releases Sexual Health (SASH) research group at Cov-
their own game – Syrum – aimed at demonstrating entry University, UK (Brown et al. 2012; Arnab
the brand’s continuous commitment in research et al. 2013). A cluster-randomized controlled
and innovation and to educate the public on their trial in local schools (n ¼ 505) demonstrates
product development process. positive outcomes in favor of the game-based
794 Game-Based Interventions in Public Health: Exploiting the Engaging Factor of Gameplay

approach when compared to existing methods highlights existing research and development
based on surveys of self-reported measures of trends, methodologies, and experimental work
psychosocial preparedness for avoiding coer- in the use of game technologies to address
cion or coercive behavior. The Serious Games health-related topics, which aimed at pro-
for Healthcare book (Arnab et al. 2012) gressing the understanding of serious games,
the methodological rigor, and the implications.
This volume explores the issues including
ethics, modern game engines, design consider-
ations, and research methodologies underpin-
ning the use, evaluation, and validation of
games applications.
For a game to be considered “serious,” its
efficacy should be proven rather than simply
intended, and games should not be afforded
exemptions from the rigor applied to assessment
of other approaches to education and training
across the sector. Only then can relevant
decision-makers be provided with the evidence
needed to make informed selections of game-
based approaches as ideal solutions to specific
intervention, learning, or training needs.
The study on the long-term behavioral impact
is however longitudinal. Games’ ability to reach
and engage a large number of players for long
periods of time provides an opportunity for vital
user data to be recorded, monitored, and analyzed
continuously. The challenge is how to best collect,
record, and analyze the potential wealth of data
and utilize the analysis to provide appropriate
feedback and support to the individuals, which
Game-Based Interventions in Public Health: could potentially promote self-management and
Exploiting the Engaging Factor of Gameplay, health coaching.
Fig. 2 The Re-Mission game (Kato et al. 2008)

Game-Based Interventions in Public Health: Exploiting the Engaging Factor of Gameplay, Fig. 3 Screenshots
of the PR:EPARe game (Arnab et al. 2013)
Game-Based Interventions in Public Health: Exploiting the Engaging Factor of Gameplay 795

Gamification and Health knowledge of their employees, which will subse-

quently help increase productivity and reduce
The need for a more long-term regulation of healthy health insurance premiums.
behavior signals a move toward greater By tapping into our natural tendency to react
gamification, commonly defined as the use of positively to entertainment and the competitive
game design elements in non-game contexts nature of most gameplay, actionable steps to over-
(Deterding et al. 2011). It will take the essence of come personal challenges can be designed. This
what it is that makes games so appealing, decode can potentially help initiate healthier activities in
the mechanics that makes them work, and then any number of areas: losing weight, sleeping
apply these mechanics in activities that activate more, making healthier food choices, improving
psychological motivators, which in turn drive fitness, monitoring health metrics, and medication
desired behaviors. A strong body of research work compliance. Most of the activities will however
has already been generated and the number of new require individuals to embrace delayed satisfac-
publications on gamification is growing daily, as tion, where the reward may be as elusive as the G
underlined in a recent study (Hamari et al. 2014). prevention of a chronic condition. With this per-
The pervasiveness of game (play) mechanics spective, gamification allows rewards and incen-
and dynamics employed in day-to-day activities tives to be used to sustain positive engagement.
to support serious outcomes is the current trend The fundamental fact of motivation is that we
within the context of gamification. Gartner cannot be forced to change our behaviors. Behav-
(2015) expects gamification to reach the “plateau ioral change may be initiated by extrinsic sources
of productivity” within 5–10 years; that is, main- of motivation or external factors that influence
stream adoption begins to take off and more rigid how we behave (Seifert et al. 2012).
assessment criteria are evident. At this stage, the Intrinsic motivation and positive habit may be
technology’s general market applications and rel- nurtured through sustained engagement, where
evance are accepted (Gartner 2015). The biggest personal incentives and rewards for healthy behav-
players exploiting this trend include the major ior could be discovered. For example, the Monster
health insurers such as Aetna, Cigna Health, Manor gamification program (Fig. 4) involves par-
UnitedHealth, and WellPoint. The main objective ents and clinicians in the “playful” and “incentiv-
is to improve the health and health-related ized” ecosystem aiming to motivate children with

Game-Based Interventions in Public Health: Monster Manor, checks their blood glucose level, and
Exploiting the Engaging Factor of Gameplay, receives a reward for positive efforts to be used within
Fig. 4 A support and rewarding system – a child plays the game (www.ayogo.com)
796 Game-Based Interventions in Public Health: Exploiting the Engaging Factor of Gameplay

Game-Based
Interventions in Public
Health: Exploiting
the Engaging Factor
of Gameplay, Fig. 5 The
Walk commissioned by the
UK’s Department of Health
and National Health
Services (NHS), exploiting
the success of Zombies Run
(www.thewalkgame.com)

type 1 diabetes to check their blood sugar regularly. Taking into account the behavioral challenges
The testing schedule was paired with a reward restricting the reach and effectiveness of health
schedule in a virtual world of “pet monsters.” Par- interventions, gamification could help to revolu-
ents and clinicians can also intervene in this “Mon- tionize the existing intervention for the general
ster Manor” virtual economy to reward children for population into incentivized, future-looking, pre-
consistent checking. Other examples include an ventative, and personalized healthcare. Personali-
online and social community facilitated by the zation provides individuals with a sense of control
HealthSeeker program utilizing on competitions over their own healthcare. The benefit of
and recognition, where adults issue health chal- gamification within the health sector is longitudi-
lenges to each other through Facebook. Success- nal and will require rigor in the evaluation of long-
fully completing shared “missions” will result in term efficacy. With the advancement in data track-
points that allow players to progress through the ing and analytics, qualitative assessment can be
levels of the game. Analysis of the resulting data paired with data from the gamified activities to
indicates that these peer-to-peer challenges are sub- better understand the individuals and continu-
stantially more effective at encouraging change ously provide personalized feedback within the
than simple software-originated challenges. engagement loop.
The pervasiveness of play spaces has seen
gamification exploiting actual gameplay in both
the physical and digital spaces. Games such as Conclusions
Zombies Run and The Walk (Fig. 5) exploit
location-based mobile gaming to advocate run- The use of game technologies and techniques in
ning and walking, respectively. These initiatives the form of serious games and gamification pre-
intrinsically encourage “players” to run or walk as sents an opportunity for the engaging mechanics
part of the mechanics for the location-based and dynamics of gameplay to be exploited in
adventure game instead of explicitly campaigning order to promote receptiveness to the serious mes-
about the benefits of running or walking as a sage of public health campaigns. Longitudinal
fitness regime. Pervasive gamification, when engagement with gamified platforms allows the
coupled with wearable technologies, opens up facilitation of the recording and reasoning of
exciting opportunities for individuals, who are large-scale health and well-being data. By better
not normally engaging with digital games to par- understanding knowledge, attitude, and behavior
ticipate in positive playful activities anywhere and of the “players” and assessing their progress con-
at anytime. Applications such as FitBit and Nike+ tinuously, personalized and actionable feedback
capitalize on mobile and wearable technologies can be provided to nurture healthier habits. The
and the common features of gamification (points, academic labeling and debate by semantics and
badges, and leaderboards). taxonomy quantize the differences between
Game-Based Interventions in Public Health: Exploiting the Engaging Factor of Gameplay 797

gamification and serious games. However, the Self-reinforcement of personal healthcare will be
aims of any of these terms are that they are all more enhanced and engaging, relevant, and ubiq-
trying to solve a problem and motivate using uitous. Anytime, anywhere healthcare when
game-based thinking and techniques gamified will introduce playfulness, competition
(Kapp 2012). and collaboration-playful healthy “regime,” com-
The generations who grew up playing elec- petition to drive self-improvement, and collabora-
tronic games increasingly represent and include tion to foster a community of health-conscious
both professionals and decision-makers in the citizens.
healthcare sector, which makes increased recep- Research plays a key role in experimenting
tiveness to this form of instruction becoming more and providing evidence in the use of game tech-
apparent. If this receptiveness is coupled with the nologies and concepts in the healthcare sector.
research required to validate and refine the use of The prospect of a gamified and pervasive health
game-based approaches across a wide range of and well-being ecosystem can potentially affect
applications, this may lead to exciting opportuni- the design and deployment of health strategy G
ties to address challenges existing interventions and policy in the future. Despite significant chal-
have so far failed to satisfactorily overcome. As lenges for researchers in this domain in terms of
the public health sector increasingly pressed to the lack of standard methodologies or formulaic
tackle chronic diseases and behaviors among the frameworks that guarantee success and efficacy,
general population, the introduction and discus- there are some empirical studies that can serve
sion put forward by this chapter, alongside the as benchmarks for establishing the scientific
past and current projects in the area it highlights, validity. There is thus a need to tap into best
suggest game-based interventions may form a practices of such a multidisciplinary domain and
critical part of a long-term strategy to address infuse knowledge from relevant disciplines
these challenges. within the application domain toward develop-
The diminishing boundaries between physical ing an infused and transdisciplinary methodo-
and digital spaces provide great opportunities for logical framework that may act as a validated
game-based approaches to be applied in everyday guide to inform the development process of a
contexts. Game mechanics are becoming more game-based approach.
pervasive as real and virtual interactions and
events are merged within the context of gameplay.
The application of gamification and pervasive Cross-References
gaming, such as The Walk and Zombies Run
demonstrates the potential for gameplay to be a ▶ Cloud for Gaming
key catalyst for the nurturing of long-term healthy ▶ Cognitive Psychology Applied to User
behavior. As the sophistication of mobile and Experience in Video Games
wearable technologies is advancing, for instance, ▶ Game Player Modeling
the potential for a more connected and seamless ▶ Gamification in Crowdsourcing Applications
gameplay experience within a hybrid space will ▶ Games and the Magic Circle
be possible. ▶ Interaction with Mobile Augmented Reality
The future trend in games and gamification Environments
will thus exploit such a hybrid space, which will
see the crossings between pervasive gaming and
gamification that will inject gameplay into their References
surroundings and community. The advancement
of the Internet of Things (IoTs), mobile technol- Arnab, S., Dunwell, I., Debattista, K. (ed.): Serious Games
for Healthcare: Applications and Implications.
ogy, and data analytics will allow everyday spaces
Hershey, PA: IGI Global (2012)
to be transformed into a personalized playground Arnab, S., Brown, K., Clarke, S., Dunwell, I., Lim, T.,
enriched with contextual resources and activities. Suttie, N., Louchart, S., Hendrix, M., de Freitas, S.:
798 Game-Based Learning (GBL)

The development approach of a pedagogically-driven

serious game to support relationship and sex education Game-Based Learning (GBL)
(RSE) within a classroom setting. Comput.
Educ. 69, 15–30 (2013). Elsevier
Brown, K., Arnab, S., Bayley, J., Newby, K., Joshi, P., ▶ Computer Games in Education
Judd, B., Baxter, A., Clarke, S.: Tackling sensitive ▶ Gamification and Serious Games
issues using a game-based environment: Serious game ▶ Immersive Technologies for Medical Education
for relationships and sex education (RSE). The 17th
Annual CyberPsychology and CyberTherapy Confer-
▶ Serious Online Games for Engaged Learning
ence (CYBER17), September 25th-28th, Brussels Through Flow
(2012) ▶ Transformational Games
Deterding, S., Khaled, R., Nacke, L., Dixon, D.:
Gamification: Toward a definition. CHI 2011. Pre-
sented at the Computer Human Interaction, ACM, Van-
couver (2011)
Gartner: Gartner Hype cycle, http://www.gartner.com/ Gameplay Preference
newsroom/id/2819918. (2015). Accessed 1 July
2015 Categories
Hainey, T.H., Connolly, T.M., Stansfield, M., Boyle, E.
A.: Evaluation of a game to teach requirements ▶ Player Personas and Game Choice
collection and analysis in software engineering at
tertiary education level. Comput. Educ. 56, 21–35
(2011)
Hamari, J., Koivisto, J., Sarsa, H.: Does gamification
work? – a literature review of empirical studies on
gamification. Proceedings of the 47th Hawaii
Gamers
International Conference on System Sciences. Hawaii
(2014) ▶ Game Prosumption
Kapp, K.: The Gamification of Learning and Instruction: ▶ Online Players: Engagement, Immersion, and
Game-Based Methods and Strategies for Training and
Absorption Across Secondary Worlds
Education. San Francisco: Pfeiffer (2012)
Kato, P.M., Cole, S.W., et al.: A video game improves
behavioral outcomes in adolescents and young
adults with cancer: a randomized trial. Pediatrics
122(2), 305–317 (2008)
Knight, J., Carly, S., Tregunna, B., Jarvis, S., Smithies, R., Games
de Freitas, S., Mackway-Jones, K., Dunwell, I.: Serious
gaming technology in major incident triage training: a ▶ Accessibility of Virtual Reality for Persons
pragmatic controlled trial. Resuscitation J. 81(9),
1174–1179 (2010) with Disabilities
Marchiori, E.J., Ferrer, G., Fernández-Manjón, B., Povar-
Marco, J., Giménez-Valverde, J.F.-S.A.: Education in
basic life support maneuvers using video games.
Emergencias 24, 433–437 (2012)
McGonigal, J.: Reality Is Broken: Why Games Make Us Games and Active Aging
Better and How They Can Change the World. Jonathan
Cape, London (2011)
Panese, L., Morosini, D., Lameras, P., Arnab, S., Dunwell, Inês Amaral1,2 and Frederico Fonseca1
1
I., Becker, T.: Pegaso: A Serious Game to Prevent Instituto Superior Miguel Torga, Coimbra,
Obesity HCI International Conference (HCII 2014), Portugal
25–27 June, Crete, LCNS, 427-435 (2014) 2
University of Minho, Braga, Portugal
Seifert, C.M., Chapman, L.S., Hart, J.K., Perez, P.:
Enhancing intrinsic motivation in health promotion
and wellness. Am. J. Health Prom. 26(3), 1–12
(2012) Synonyms
Ulicsak, M.: Games in Education: Serious Games,
Futurelab. http://media.futurelab.org.uk/resources/doc
uments/lit_reviews/Serious-Games_Review.pdf (2010). Active learning; Active videogames; Cognitive
Accessed 4 Jan 2015 games
Games and Active Aging 799

Definition wellbeing of the elderly and includes the

dimensions of health, participation, security,
Games are an important toll of active aging as they employment, independence, autonomy, and inte-
enable knowledge acquisition processes, attribu- gration (Parra 2013).
tion of meaning to information, and enhance qual- Quality of life of older people may be affected
ity of life and psychological wellbeing to the older within three macro-areas such as physical
people. capacity, sensory capacity, and cognitive capacity
(Parra 2013). Therefore, there is a need to create
“innovative ways to re-invent strategies for sus-
Introduction tainable active ageing and healthier lifestyles”
(Costa and Veloso 2015).
The aging of the population is one of the main
challenges facing contemporary societies. As a
result of a shared social construction, there is a Games and Active Aging G
discursive “requalification” of aging, associating
it with a terminological plurality that deconstructs Games’ approaches focused on healthy and
the prevalence of negative stereotypes that asso- positive aspects of individuals are transverse to
ciate old age with dependence, lack of autonomy, the theoretical and empirical scientific contribu-
disease, institutionalization, and a disregard of tions in this field. The individualization stands in
their heterogeneity. the mainstream of political agendas to value the
Social exclusion of the elderly is a conse- individual as a builder of his life course. The
quence of globalization as well as the condition Declaration of Alma-Ata (1978) appeals to health
of second-class citizenship (Amaral and Daniel for all and the need for health promotion models
2016) for those who do not have access to digital that avoid the traditional top-down logic (WHO
capital. Games can enhance quality of life of the 2002). It is the recognition of people’s participa-
elderly in three perspectives: cognitive, sensorial, tion in promoting their citizenship as “the right
and physical (Costa and Veloso 2015). and duty of the people to participate individually
and collectively in the planning and delivery of
health care” (WHO 2002). Giving digital compe-
Active Aging tence to older people is a way to empower them as
this concept may be assumed as a transversal key
The requalification of aging has emerged with competence that enables the acquisition of other
organizations such as the United Nations and the skills, as the European Commission has broadly
European Union that, since the 1980s and 1990s, defined it. Games can be a way to promote a
have launched the concept of “Active Aging.” critical and creative use of the ICT to achieve
New approaches and political solutions have lev- goals that may be related to health, work, leisure,
eraged this notion as a social representation for learning, inclusion, and participation in
political and media discourses, seeking to combat public life.
the stereotypy that produces negative aging. The The games may enable self-learning by provid-
term “active aging” is the successor of concepts ing meaningful active learning experiences (Costa
such as “healthy aging” or “successful aging.” and Veloso 2015), as well as embody mental
In 1997 the World Health Organization, inspired and emotional representations. Digital games
by the United Nations Principles for Older can improve the quality of life of the elderly in
Persons, presented the concept of “active aging” particular with regard to reaction time, visual
as “the process of optimizing opportunities perception, and eye-hand coordination (Green
for health, participation and security in order and Bavelier 2007; Bialystok 2006). Previous
to enhance quality of life as people age” (2002). research demonstrated that these benefits are
Active aging is a concept that promotes the related to increased dopamine levels in the brain,
800 Games and the Magic Circle

which decline with age, and are elicited by digital References

games (Green and Bavelier 2007; Parra 2013).
Exergames can be used to motivate older Amaral, I., Daniel, F.: Ageism and IT: social representa-
tions, exclusion and citizenship in the digital age. In:
adults to engage in healthy activities in a
International Conference on Human Aspects of IT for
positive way. These games are designed in order the Aged Population, volume 9755 of the series Lecture
to prevent physical decline and preserve and Notes in Computer Science, pp. 159–166. Switzerland:
even physical abilities such as muscle strength Springer International Publishing (2016)
Bialystok, E.: Effect of bilingualism and computer video
and balance (Parra 2013). Serious games can stim-
game experience on the Simon task. Can. J. Exp.
ulate cognitive functioning skills that positively Psychol. 60(1), 68–79 (2006)
affecting psychological wellbeing (Parra 2013). Costa, L., Veloso, A.: Games for triggering active ageing
Monitoring applications can also be used to and healthier lifestyles. TechDays Aveiro. https://www.
researchgate.net/profile/Ana_Veloso8/publication/297
analyze user’s behaviors such as reaction time,
758626_Games_for_triggering_active_ageing_and_
short-term memory, and discernment. These healthier_lifestyles/links/56e2e25708aee84447bf3714/
applications inform the doctor of the evolution Games-for-triggering-active-ageing-and-healthier-lifes
of the elderly’s performance over time, allowing tyles.pdf (2015). Accessed 20 Nov 2017
Green, C., Bavelier, D.: Action-video-game experience
planning or adjusting therapies that may improve
alters the spacial resolution of vision. Psychol. Sci.
the psychological wellbeing. 18(1), 88–94 (2007)
Parra, C.: Information technology for active ageing:
a review of theory and practice. Found. Trends
R Human-Comput. Interact. 7(4), 351–447 (2013)
WHO: Active Ageing: A Policy Framework. World Health
Conclusion and Discussion Organization, Geneva (2002)

Games have the capacity to improve quality of

life of elders and contribute to their active aging.
The dimensions of health, participation, security,
employment, independence, autonomy, and inte- Games and the Magic Circle
gration are promoted by the use of games, foster-
ing lifelong learning through digital inclusion. Douglas Brown
Stimulation of cognitive, sensory, and physical Games Academy, Falmouth University,
abilities can be achieved with tangible and non- Cornwall, UK
tangible interfaces. The learning curve is also an
important factor that supports the evaluation of
the elderly. Gaming can be an integration tool Synonyms
and not necessarily a therapy, as it can also be
integrated in the context of health. Playground; Play space

Cross-References Definition

▶ Game-Based Approach In the context of games, the magic circle is the

▶ Game-Based Interventions in Public Health: area within which the rules of the game apply, a
Exploiting the Engaging Factor of Gameplay special space, ideally but not necessarily demar-
▶ Games for Health cated by the rules within which play occurs. It
▶ Health Games need not be a physical space, but can instead be
▶ Serious Games virtual or a frame of mind.
Games and the Magic Circle 801

Introduction Calleja’s discussion of the concept makes clear

(Calleja 2012). This is why the magic circle
The magic circle is a concept still widely used and concept most usually used in games studies in
referenced by games studies scholars and games particular is just as much rooted in Katie Salen
designers and is both helpful shorthand and a and Eric Zimmerman’s definition and retooling
problematic theory. This entry will look over the of Huizinga’s concept in their 2004 book Rules
history of the concept and explore the issues and of Play, one of the first major games studies
controversies it raised, with a particular eye textbooks and a substantive games design
toward games studies’ frequent criticism of the resource. To Salen and Zimmerman, the magic
concept. Despite its contested and controversial circle is a potent metaphor of real use for game
nature, the magic circle borders on other central designers. They first define the concept in a way
concepts of games studies and games design similar to Huizinga but assert that the physical
including play, immersion, and suspension of dis- element of the spatial magic circle is not
belief, and these linkages are also something this mandatory: G
entry will explore. In a very basic sense, the magic circle of a game is
where the game takes place. To play a game means
entering into a magic circle, or perhaps creating one
as a game begins. (Salen and Zimmerman 2004: 95)
History
Later in the book, they also discuss the magic
The magic circle was a term first used as part of circle as a form of boundary but also a kind of
Johan Huizinga’s seminal study of play, Homo special space or context for the game created by
Ludens. While it is only mentioned only a few the player:
times in the English translation of this text, it is Beginning a game means entering into the magic
regarded as a primary metaphor for how play circle. Players cross over this boundary to adopt the
artificial behaviors and rituals of a game. During the
occurs, as explained when the term is first used: game, the magic circle persists until the game con-
cludes. Then the magic circle dissolves and players
All play moves and has its being within a play-
return to the ordinary world. (ibid: 333)
ground marked off beforehand either materially or
ideally, deliberately or as a matter of course. Just as Salen and Zimmerman’s magic circle rein-
there is no formal difference between play and
ritual, so the “consecrated spot” cannot be formally forces the view that games create their own con-
distinguished from the play-ground. The arena, the texts, so that the differences between a person’s
card-table, the magic circle, the temple, the stage, playing with a physical game piece while not
the screen, the tennis court, the court of justice, etc., playing a game with it and doing so during a
are all in form and function play-grounds,
i.e. forbidden spots, isolated, hedged round, hal- game session is brought into stark relief, since
lowed, within which special rules obtain. All are there is now a magic circle operative.
temporary worlds within the ordinary world, dedi-
cated to the performance of an act apart. (Huizinga
1955: 10)
State of the Concept
It is important to note here that Huizinga sees
play as behavior distinct from the everyday norms The magic circle concept sits in a useful place for
of society, but does not ascribe triviality to play – introducing games as objects of study and is an
hence why lawcourts are as valid play spaces as extremely accessible gateway to more complex
card tables or magic circles. concepts such as liminality, presence, the suspen-
Huizinga’s seeking out of play spaces was sion of disbelief, and the lusory attitude (Suits
also grounded in the discourse of cultural stud- 1978). It also functions as a useful metaphor for
ies, far from actual game design practice, as game designers to discuss games and their
802 Games and the Magic Circle

potential while enshrining a degree of ambiguity no accident that Huizinga chose the magic circle,
about the space where rules, play, and gamers rather than the other playgrounds he lists, as
collide. emblematic, since “magic” implies a tangible
Ever since Salen and Zimmerman’s definition link to ritual spaces) which centers around the
of the magic circle, the term has not been off the self-reflexivity of play is where Klabbers (2009)
radar of games studies academics, who frequently situates arguments for renovation of the concept
question, dismiss, or attempt to redefine it. As best articulated and discussed in a critique by
such, using it in an unqualified fashion is not Myers (2012).
advisable to games studies students new to the Other criticisms run the gamut from the
concept. explicit link to ritual which persists in the
“magic” element of the phrase, its growing irrel-
evance as an increasingly networked society
Issues and Controversies spends time often in parallel virtual and
non-virtual spaces, or its implied dichotomy
Attacking and interrogating the magic circle con- between work and play through to the value of
cept was so prevalent for a time in games studies the pragmatic approach to the concept taken by
that there was even a conference in 2005 con- Salen and Zimmerman in their redefinition. The
vened around “breaking the magic circle.” existence of magic circles around digital games in
Zimmerman quips when looking back over the particular, where possible play actions are often
history of games studies since Rules of Play was unalterably authored into the system, is often
published that: challenged (Calleja 2012).
It seems to have become a rite of passage for game While there are many perceived and well-
studies scholars: somewhere between a Bachelor’s articulated problems with the concept, a majority
Degree and a Master’s thesis, everyone has to write of the papers written on it fall into the trap of
the paper where the magic circle finally gets what it attempting to replace the magic circle with a system
deserves. (Zimmerman 2012).
of their own design, while more considered schol-
Scholars generally have two principal prob- arship (Consalvo 2009) suggests the adoption of
lems with the magic circle concept – either that sociologist Erving Goffman’s broader approach of
the circle’s boundaries are permeable and things frame analysis in its place. Many other critics try, if
can travel through them both ways or that the not to replace the concept wholesale, then to repair
magic circle concept is reductive, outmoded, or it in order to make up for perceived deficiencies.
unhelpful and would be better left behind. The Castronova takes up the concept’s permeability as a
idea of a privileged space where gameplay occurs virtue, seeing markets, politics, and law pass in and
also makes the figure of the critic awkward. Can a out through what he redefines as a “membrane”
critic stand outside the magic circle and meaning- thrown around the text by the magic circle. He still
fully comment on what goes on on the other side eventually declares that, at least in the case of the
of the boundary? MMO-RPGs which he is discussing:
For some critics coming from particular areas of What we have is an almost-magic circle, which
games studies such as gamification or educational seems to have the objective of retaining all that is
games, the very concept of a magic circle is unac- good about the fantasy atmosphere of the synthetic
ceptable, as the texts with which they are world, while giving users the maximum amount of
freedom to manipulate their involvement with
concerned constantly strive to handle the transition them. (Castronova 2005: 159–160)
between in-game and out-of-game information, be
it advertising material or learning curricula. In the retrospective cited above, Zimmerman
The magic circle’s ambiguity and its imperfec- himself takes responsibility from Huizinga for the
tion as a game system model drives other critics controversial redefinition of the magic circle, but
who would see it removed or replaced. The awk- defends the concept as a worthwhile game design
wardly inexplicable “magic” of the circle (and it is metaphor, rejecting the way it has been read by
Games and the Magic Circle 803

many as a hard boundary hiving games off from certainly be weaker without both it and the debate
the outside world. He does this while pointing out which its prominence has engendered. The idea of
the exaggerated way many of the critics of the magic circles, be they bounded off playgrounds or
magic circle style their offerings to attack straw chalk lines on the ground generating a special
man arguments instead of actually focusing on the place in time and space where gameplay happens,
perceived deficiencies of the magic circle. While is enticing, but also extremely broad and open to
it is not an ideal metaphor, he contends, the kind interpretation. Drawing attention to the border
of reductive formalism which it has been linked to regions of games can be worthwhile, as shown
does not really exist in scholarship, the concept is in the context of social, material, and cultural
generally accepted as what it is – an imperfect elements by Stenros (2014) in a fruitful paper
metaphor. also trying to rehabilitate both Huizinga and
Other games studies academics are more Salen and Zimmerman’s magic circles and show
embracing of the core ideas represented by the that the concept(s), ambiguities, and all hold a
magic circle and suggest not their own interpreta- place in the modern games studies lexicon. G
tions of the concept nor its replacement, but rather
actively seek to repair it. In his response to various
magic circle controversies, Jesper Juul defends the
Cross-References
idea of the magic circle as a boundary, but sees it as
potentially an awkward metaphor. His suggested
▶ Immersion
replacement is the idea of a puzzle piece:
▶ Telepresence
Perhaps the problem with the magic circle as a ▶ Virtual Worlds
metaphor is that it suggests a uniform interface
between the game and that which is around the
game. We could alternatively describe a game as a
puzzle piece. This makes it easier to talk about some References and Further Reading
of details surrounding games: a puzzle piece has
different interfaces on its sides. Seen as a puzzle Calleja, G.: Erasing the magic circle. In: Sageng,
piece, a game may or may not fit in a given context. Fossheim, Larsen (eds.) The Philosophy of Computer
We can then analyze how a game fits into a context, Games, pp. 77–91. Springer, New York (2012)
no longer arguing whether games are separate or Castronova, E.: Synthetic Worlds. University of Chicago
not. (Juul 2009: 8) Press, Chicago (2005)
Consalvo, M.: There is no magic circle. Games Cult. 4,
Like Zimmerman, Juul also sees the controver- 4 (2009)
sies around the concept as overblown and points Huizinga, J.: Homo Ludens: A Study of the Play Element
toward academics’ tendency to seek out and chal- in Culture. Beacon, Boston (1955)
lenge binary dichotomies, often for short-term Juul, J.: The magic circle and the puzzle piece. In: Philos-
ophy of Computer Games Conference. http://opus.
political ends as partly culpable for the sheer kobv.de/ubp/volltexte/2008/2455/digarec01_03.pdf
amount of criticism and debate the concept has (2009). Accessed 12 Jan 2015
received. His is possibly the most accepted reha- Klabbers, J.: The Magic Circle: Principles of Gaming and
bilitation of the magic circle metaphor into a tool Simulation. Sense Publishers, Rotterdam (2009)
Myers, D.: Circles tend to return. Game Stud. 12, 2 (2012)
useful for modern games studies, even if the orig- Salen, K., Zimmerman, E.: Rules of Play: Game Design
inal concept is of use to games designers and those Fundamentals. MIT Press, Boston (2004)
outside of the scholarly arena. Stenros, J.: In defence of a magic circle: the social, mental
and cultural boundaries of play. Trans. Digit. Games
Res. Assoc. 1, 2 (2014)
Suits, B.: The Grasshopper. Broadview Press, Toronto
Conclusion (1978)
Zimmerman, E.: Jerked Around by the Magic Circle –
Clearing the Air Ten Years Later. Gamasutra. http://
While the concept of the magic circle is undoubt- www.gamasutra.com/view/feature/135063/jerked_
edly problematic, it also feels core in many ways around_by_the_magic_circle_.php (2012). Accessed
to games and game studies, and the field would 12 Jan 2015
804 Games for Change

transfer a problem into an entertaining game struc-

Games for Change ture, it could possibly be crowdsourced to a
worldwide gaming community. Motivated by
▶ Hypermedia Narrative as a Tool for Serious game mechanics and the higher purpose of help-
Games ing science, many players have invested a tremen-
dous amount of time in these games.
One of the most popular examples of this strat-
egy is Foldit, an experimental computer game in
Games for Health which the player folds protein structures as per-
fectly as possible. The game has been developed
▶ Game-Based Interventions in Public Health: in 2008 at the University of Washington. The best
Exploiting the Engaging Factor of Gameplay scores are analyzed by experts. In 2011, Foldit
▶ Transformational Games players helped to decipher the structure of the
Mason-Pfizer monkey virus in only 10 days. The
problem had remained unsolved for 15 years
before (http://fold.it/; Cooper et al. 2010). Science
Games in Science Games like these are not only simulations but
actively include game elements like points,
Christian Stein and Thomas Lilge badges, and leaderboards to engage players. Sci-
gamelab.berlin, Humboldt University Berlin, ence games like these are useful in every field in
Berlin, Germany which humans outperform algorithms (Hand
2010).
Citizen science games are created equally to
Synonyms allow gamers to actively participate in research as
to raise interest and awareness about science-
Citizen science; Crowdsourcing, scientific games; related topics. An example of a citizen science
Educational games; Games with a purpose project in which citizens are helping researchers
is Galaxy Zoo, created by a university collabora-
tion between Oxford, Portsmouth, and Johns
Definition Hopkins. Over 150,000 players have helped to
classify unknown galaxies within only a year.
Games in Science are used for scientific purposes, Researchers could not have done this work in
like crowdsourcing scientific problems, collecting years that players achieved in under a month. By
data from gamers, educating the gamers, or now, there are many projects within Galaxy Zoo
gamifying science itself. following different astronomical cartography
goals (https://www.galaxyzoo.org/).
Another area of Games in Science is covering
Introduction games that are built to generate data about the
players while being played (Raykar et al. 2010).
Games are not only used for entertainment but A well-known example is NeuroRacer, which was
also for several serious purposes. Academia has developed in 2013 by the University of California.
used games for research and knowledge genera- The player has to multitask, as he is steering a car
tion as well as for teaching and knowledge trans- while he has to react correctly to some color codes
fer. It has been proven that some complicated shown. The graphics of the game were very sim-
scientific problems can be solved much better by plistic, as the game was relying fully on the mechan-
human players than algorithms. If it is possible to ics. The game was used to measure the effect of
Games with a Purpose 805

game-based multitasking training for elderly test ▶ Gamification and Serious Games
subjects with an EEG (https://neuroscape.ucsf.edu/ ▶ Gamification in Crowdsourcing Applications
technology/#neuroracer). Although the results of ▶ MEEGA+, Systematic Model to Evaluate Edu-
the study were questioned later, it became a cational Games
well-known example of how to create experi-
ments with video games and measure the training
effect of gaming to the brain (Abbott 2013; References
Anguera et al. 2013).
A growing field that is called Science Games is Abbott, A.: Gaming improves multitasking skills. Nature.
501(7465), 18–18 (2013)
covering games for teaching, experimenting, and
Anguera, J.A., Boccanfuso, J., Rintoul, J.L., Al-Hashimi,
education. A well-known example is A Slower O., Faraji, F., Janowich, J., et al.: Video game training
Speed of Light, developed in 2012 at the MIT enhances cognitive control in older adults. Nature.
Game Lab. It aims to teach the player about the 501(7465), 97 (2013)
relativity of space time, as the game setting alters Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J.,
Beenen, M., et al.: Predicting protein structures with a
G
the speed of light. The player is supposed to multiplayer online game. Nature. 466(7307),
collect a number of objects in a three-dimensional 756 (2010)
setting, and with every object the speed of light is Hand, E.: People power. Nature. 466(7307), 685 (2010)
Kortemeyer, G., Tan, P., Schirra, S.: A Slower Speed of
lowered. This way a very complex topic of phys-
Light: developing intuition about special relativity with
ics is made tangible and experienceable for a games. In: FDG, Chania, pp. 400–402, May 2013.
broader audience (Kortemeyer et al. 2013). http://www.fdg2013.org/program/papers.html
But also existing games that were built mostly Raykar, V.C., Yu, S., Zhao, L.H., Valadez, G.H., Florin, C.,
Bogoni, L., Moy, L.: Learning from crowds. J Mach
for entertainment are used for scientific purposes.
Learn Res. 11, 1297–1322 (2010)
In the very popular game Minecraft, players can
create their own architectures, machines, and
structures by crafting and building blocks
(https://education.minecraft.net/). This game has
been used as a basis for project Malmo, which is Games Industry
an artificial intelligence experimentation plat-
form. With this open-source platform, researchers ▶ Social-, Mobile-, and Multi-Player-Games and
can set experimental AIs into the world and Their Impact on Today’s Online Entertainment
observe how they learn to collaborate and interact Industry
with each other or human players. This way solu-
tions can be tested and compared and allow for the
integration of a huge base of human players.
Minecraft has also been used for teaching in an
engaging way. In a special educational edition, Games User Research
topics like math, physics, history, humanities, or
computer science are integrated in a playable way ▶ Player Experience, Design and Research
(https://www.microsoft.com/en-us/research/project/
project-malmo).

Cross-References Games with a Purpose

▶ Augmented Learning Experience for School ▶ Games in Science

Education ▶ Gamification in Crowdsourcing Applications
806 Gamification

Additionally, the player can collect virtual sup-

Gamification plies for the game’s home base while jogging.
Another example from city design is musical
Antti Knutas stairs, which combine fun and curiousness to
Lappeenranta University of Technology, encourage people to use the stairs instead of the
Lappeenranta, Finland escalator.
The use of gamification often involves
the use of common game design elements, such
Synonyms as stories, rules, goals, feedback, and tiers. How-
ever, in successful gamified design, user experi-
Applied gaming; Funware; Ludification; Playful ence is more important rather than the visual
design manifestation of certain elements (Deterding
2014). Just like in game design, in gamified
design the system has to be considered in its
Definition entirety from the perspective of the user, instead
of adding a set of elements, such as points or
Gamification is the intentional application of badges.
game elements to nongame contexts, with the Gamified design and gamification has also
intention of creating playful experiences or faced critique, especially in cases where a stock
gameful interaction. It is often used to motivate approach of gamification has been added on top
and increase user activity or user retention. of an existing system in so-called
“pointsification” (Kapp 2012). However,
gamification has potential for successful results
Introduction if the system is selectively designed, allowing for
personalization and customization, in order to
A synthesis by Seaborn and Fels (2015, p. 17) accommodate individual users, and if the design
that considers seminal works on gamification is informed by end users’ intrinsic motivators
(Deterding et al. 2011; Huotari and Hamari (Seaborn and Fels 2015).
2012) defines gamification as the “intentional
use of game elements for a gameful experience
of non-game tasks and contexts.” In this Theoretical Foundations
gamification is different from serious games,
which involve the use of full games in serious, The dominant theoretical framework for
nongame contexts. gamification is currently Self-Determination The-
Gamification has been used to increase ory (SDT), as developed by Deci and Ryan
engagement in education (Auvinen et al. 2015), (2000), and the most common design theory is
compliance and satisfaction in healthcare (Stinson user-centered design (Seaborn and Fels 2015).
et al. 2013), user activity in sharing economy In user-centered gamification design, the main
(Hamari 2017), and quality and performance priority is providing an experience for the user
in crowdsourcing (Liu et al. 2011; Massung and designing with the user’s needs and desires
et al. 2013). Currently most common fields in mind (Nicholson 2012). Theoretical frame-
for applying gamification are e-learning, sustain- works for gamification that follow self-
ability, and motivational software (Kasurinen determination theory (e.g., Aparicio et al. 2012)
and Knutas 2018). Successfully applied example posit that effective gamification is about using
from the mobile application industry is the game elements to support users’ innate need to
“Run, Zombies!” exercise game, which uses the seek out novelty and challenges, or intrinsic moti-
storyline of running away from monsters to help vation (Ryan and Deci 2000). These three princi-
the player regulate their pacing when running. ples according to SDT are:
Gamification 807

• Relatedness: the universal need to interact and Cross-References

be connected with others.
• Competence: the universal need to be effective ▶ Augmented and Gamified Lives
and master a problem in a given environment. ▶ Augmented Learning Experience for School
• Autonomy: the universal need to control one’s Education
own life. ▶ Computer Graphics, Video Games, and
Gamification Impacting (Re)Habilitation,
Healthcare, and Inclusive Well-Being
Design Frameworks for Gamification
▶ Domain-Specific Choices Affecting Design
Effort in Gamification
There have been several gamification frameworks
▶ Game Thinking X Game Design Thinking
published, of which Deterding’s Method for
▶ Game-Based Interventions in Public Health:
Gameful Design (Deterding 2015) is one of
Exploiting the Engaging Factor of Gameplay
the most recent and most comprehensive. It is
▶ Gamification and Serious Games G
based on the Player Experience of Need Satisfac-
▶ Gamification in Crowdsourcing Applications
tion (PENS) model, which is based on self-
▶ Gamification of Modern Society: Digital
determination theory. First the method uses
Media’s Influence on Current Social Practices
PENS to analyze what players find motivating
▶ Mixed Reality, Gamified Presence, and Story-
and enjoyable qualities in gameplay. Then, it pro-
telling for Virtual Museums
vides a framework to examine gameful design
through elements of interconnected challenges,
in which the player is taking actions in pursuit
References
of goals, within the system’s rules. It posits
that gameful design should use the skill-based Aparicio, A.F., Vela, F.L.G., Sánchez, J.L.G., Montes, J.L.
challenges that already are inherent in the activity I.: Analysis and application of gamification.
that the gameful design supports, with a data- In: Proceedings of the 13th International Conference
driven design process guided by iterative on Interacción Persona-Ordenador, p. 17. ACM,
New York (2012)
prototyping. Auvinen, T., Hakulinen, L., Malmi, L.: Increasing students’
Two other often used gamification design awareness of their behavior in online learning environ-
frameworks are for education (Kapp 2012) and ments with visualizations and achievement badges.
business (Werbach and Hunter 2012). Kapp’s IEEE Trans. Learn. Technol. 8, 261–273 (2015)
approach for education is partly inspired by seri- Deci, E.L., Ryan, R.M.: The “what” and “why” of goal
pursuits: human needs and the self-determination of
ous games (Deterding 2015), and Werbach and behavior. Psychol. Inq. 11, 227–268 (2000)
Hunter’s pattern-based approach is intended Deterding, S.: Eudaimonic Design, or: Six Invitations to
for motivating users of business platforms. Rethink Gamification. Social Science Research Net-
work, Rochester (2014)
Deterding, S.: The lens of intrinsic skill atoms: a method
for gameful design. Hum. Comput. Interact. 30,
294–335 (2015)
Conclusion Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From
game design elements to gamefulness: defining
Gamification is a powerful technique for engag- “gamification”. In: Proceedings of the 15th Interna-
ing users through gameful and playful design, tional Academic MindTrek Conference: Envisioning
Future Media Environments, pp. 9–15. ACM,
supported by a thriving multidisciplinary field New York (2011)
and an increasing number of design theories and Hamari, J.: Do badges increase user activity? A field exper-
frameworks (Nacke and Deterding 2017; Seaborn iment on the effects of gamification. Comput. Hum.
and Fels 2015). However, it is not an easy Behav. 71, 469–478 (2017)
Huotari, K., Hamari, J.: Defining gamification: a service
approach or a one size fits all approach. Care marketing perspective. In: Proceeding of the 16th Inter-
should be taken when choosing design elements national Academic MindTrek Conference, pp. 17–22.
and designing for user experience. ACM, New York (2012)
808 Gamification and Serious Games

Kapp, K.M.: The Gamification of Learning and Instruc- Definition

tion: Game-Based Methods and Strategies for Training
and Education. Wiley, Hoboken (2012)
Kasurinen, J., Knutas, A.: Publication trends in A video game is a mental contest, played with a
gamification: a systematic mapping study. Comput. computer according to certain rules for amuse-
Sci. Rev. 27, 33–44 (2018) ment, recreation, or winning a stake (Zyda 2005).
Liu, Y., Alexandrova, T., Nakajima, T.: Gamifying A digital game refers to a multitude of types
intelligent environments. In: Proceedings of the
2011 International ACM Workshop on Ubiquitous and genres of games, played on different plat-
Meta User Interfaces, pp. 7–12. ACM, New York forms using digital technologies such as com-
(2011) puters, consoles, handheld, and mobile devices
Massung, E., Coyle, D., Cater, K.F., Jay, M., Preist, C.: (DGEI 2013). The concept of digital games
Using crowdsourcing to support pro-environmental
community activism. In: Proceedings of the SIGCHI embraces this technological diversity. In contrast
Conference on Human Factors in Computing Systems, with terms such as “video games” or “computer
pp. 371–380. ACM, New York (2013) games,” it does not refer to a particular device on
Nacke, L.E., Deterding, S.: The maturing of which a digital game can be played. The common
gamification research. Comput. Hum. Behav. 71,
450–454 (2017) factor is that digital games are fundamentally pro-
Nicholson, S.: A user-centered theoretical framework for duced, distributed, and exhibited using digital
meaningful gamification. Games Learn. Soc. 8, technologies.
223–230 (2012) Gamification has been defined as the use of
Ryan, R.M., Deci, E.L.: Intrinsic and extrinsic motivations:
classic definitions and new directions. Contemp. game design elements in nongame contexts and
Educ. Psychol. 25, 54–67 (2000) activities (Deterding et al. 2011), which often aim
Seaborn, K., Fels, D.I.: Gamification in theory and to change attitudes and behaviors (Prandi et al.
action: a survey. Int. J. Hum. Comput. Stud. 74, 2015). Using game-based mechanics, aesthetics
14–31 (2015)
Stinson, J.N., Jibb, L.A., Nguyen, C., Nathan, P.C., and game thinking to engage people, motivate
Maloney, A.M., Dupuis, L.L., Gerstle, J.T., Alman, action, solve problems, and promote learning
B., Hopyan, S., Strahlendorf, C., Portwine, C., John- (Kapp et al. 2013; Kapp 2015), i.e., employing
ston, D.L., Orr, M.: Development and testing of a awards, ranks during missions, or leaderboards to
multidimensional iPhone pain assessment application
for adolescents with cancer. J. Med. Internet Res. 15, encourage active engagement during an activity,
e51 (2013) e.g., health fitness tracking or e-learning during an
Werbach, K., Hunter, D.: For the Win: How Game Think- online course. Thus, gamification uses parts of
ing Can Revolutionize Your Business. Wharton Digital games but is not a complete game.
Press, Philadelphia (2012)
Serious games are full-fledged games created
for transferring knowledge (Ritterfeld et al. 2009),
teaching skills, and raising awareness concerning
Gamification and Serious certain topics for nonentertainment purposes
Games (Deterding et al. 2011). Essentially is a mental
contest, played with a computer in accordance
George Papagiannakis with specific rules, that uses entertainment to fur-
Computer Science Department, University of ther government or corporate training, education,
Crete, Heraklion, Greece health, public policy, and strategic communica-
Foundation for Research and Technology Hellas, tion objectives (Zyda 2005).
Heraklion, Greece

Introduction
Synonyms
The appeal of mixed reality (MR) digital games
Game-Based Learning (GBL); Interactive learn- arouses interest among researchers and education
ing events; Mixed reality serious games; specialists who since their recent proliferation
Simulations have been trying to introduce their motivating
Gamification and Serious Games 809

potential in learning contexts. Previous work in purposes used in specific contexts so that they
this field has been focusing on whether digital can empower individuals and communities in
games can, via novel presence (feeling of “being ways that lead to social inclusion (Misuraca
and doing there” in a virtual or augmented world) 2012).
and MR gamification (dynamics, mechanics, Interactive Learning Events (ILEs) is a term
components), support and foster future learning used to include games, gamification, and simula-
and teaching, to address a wide variety and vari- tion (Kapp et al. 2013).
ation of educational contexts. The final aim is to Informal learning: Learning without the inten-
provide informal, nonformal, as well as formal tion to learn and without actual planning of learn-
learning for their end users. ing activities. Sometimes also referred to as
This entry aims to highlight their main concep- experiential or accidental learning.
tual differences and key indicative applications Formal learning: Learning as an intended and
and challenges across the mixed reality and planned activity taking place in an organizational
learning aims. context. G
Nonformal learning: Learning as a result of
planned general activities in which participants
Related Ecosystem Definitions and Key can learn both intentionally and unintentionally
Application Areas (Misuraca 2012; Centeno 2013).
Mixed Reality Serious Games and
The concept of “presence” refers to the phenom- Gamification (MRSG) is used about any kind of
enon of behaving and feeling as if we are in the serious game, simulation, or gamified process for
virtual world created by computer displays learning in mixed reality featuring presence, nat-
(Sanchez-Vives 2005). “Presence is an incredibly ural interaction, and the suite of the novel MR
powerful sensation, and it’s unique to VR; there’s technologies, MR gesture-based and game-based
no way to create it in any other medium. Most learning (Zikas et al. 2016).
people find it to be kind of magical” It is not the Indicative nonentertainment themes of serious
same as “immersion,” where the user is simply games and gamification include multitude
surrounded by digital screens (Abrash 2014). areas from general education and training
Presence is a key term that next-generation of (Magnenat-Thalmann et al. 2009) to cultural
serious-games will need to take into careful con- heritage (Anderson et al. 2009; Kateros et al.
sideration to be successful. 2015; Ioannides et al. 2017), health and
Mixed reality (MR) has been defined as a con- surgical training (de Ribaupierre et al. 2014;
tinuum of technologies that include both virtual Papagiannakis et al. 2018), and inclusive well-
reality (VR) (fully substitute reality with a virtual being (Brooks et al. 2014).
3D world) as well as augmented reality
(AR) (supplements reality by blending virtual
and real elements with the use of special displays) Challenges
(Azuma et al. 2001).
Simulation is a realistic, controlled-risk envi- One of the main challenges for the next-
ronment, where learners can practice behaviors generation of gamified simulations and serious
and experience the impacts of decisions. Simula- games involves answering the key research ques-
tions are designed to be realistic representations of tion on how allow the learners and teachers to
real-world environments, events, and processes, experience the feeling of “presence” under a
whereas games on the other hand may or may not novel MR educational learning framework, in
reflect the reality. Hence, Simulation + both virtual reality (VR) as well as augmented
Gamification ¼ Game (Kapp et al. 2013). reality (AR) formal, nonformal, and informal
Digital Games for Empowerment and Inclu- learning environments. The former (VR) allows
sion (DGEI) are Digital games for nonleisure for the unique feeling of “being there” and “doing
810 Gamification and Serious Games

there” in the virtual world, that will be trans- Anderson, E.F., McLoughlin, L., Liarokapis, F., Peters, C.,
forming the overall game-based learning experi- Petridis, P., Freitas, S.: Serious games in cultural heri-
tage. In: The 10th VAST Int’l Symposium on Virtual
ence, via latest innovations as well as recent Reality, pp. 29–48. Malta (2009)
progress in low-cost h/w head mounted displays Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S.,
(HMDs). The latter (AR) blends real and virtual Macintyre, B.: Recent advances in augmented reality.
elements so that the 3D virtual element is regis- IEEE Comput. Graph. Appl. 21(6), 34–47 (2001)
tered accurately in the real world and interacted Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., Berta,
R.: Assessment in and of serious games: an overview.
freely by the learner via various mobile displays, Adv. Hum. Comput. Interact. 2013(2), 1–11 (2013).
including smart glasses, natural, gesture-based https://doi.org/10.1155/2013/136864
interaction (mobile RGB and RGB-D), MR vir- Brooks, A.L., Brahnam, S., Jain, L.C.: Technologies of
tual characters (Vaccheti et al 2004, Magnenat- inclusive well-being at the intersection of serious
games, alternative realities, and play therapy. In: Tech-
Thalmann et al. 2009; Jung et al. 2011), and nologies of Inclusive Well-Being, vol. 536, pp. 1–10.
gamified learning processes (Sawyer 2002; Springer, Berlin/Heidelberg (2014). https://doi.org/10.
Misuraca 2012; Centeno 2013). 1007/978-3-642-45432-5_1
Another main challenge for serious-games and Centeno, C.: The potential of digital games for empower-
ment and social inclusion. JRC scientific and technical
gamification involves user-performance metrics, report, pp. 1–172 (2013)
characterization of the player’s activity, and better de Ribaupierre, S., Kapralos, B., Haji, F., Stroulia, E.,
integration of assessment and user analytics in Dubrowski, A., Eagleson, R.: Healthcare training
games (Bellotti et al. 2013). “[Serious games] enhancement through virtual reality and serious
games. In: Virtual, Augmented Reality and Serious
will not grow as an industry unless the learning Games for Healthcare 1, vol. 68, pp. 9–27. Springer,
experience is definable, quantifiable and measur- Berlin/Heidelberg (2014). https://doi.org/10.1007/978-
able. Assessment is the future of serious games” 3-642-54816-1_2
(Ritterfeld et al. 2009). In MR, this challenge Deterding S., Sicart M., Nacke L., O’Hara K., Dixon D.:
Gamification using game-design elements in non-
requires significant future research, but it can be gaming contexts. Paper presented at the CHI ‘11
aided by the fact that the end-user position, orien- extended abstracts on human factors in computing sys-
tation, gaze, gestures, and actions can be fully tems, Vancouver (2011)
tracked and recorded in VR/AR. Ioannides, M., Magnenat-Thalmann, N., Papagiannakis,
G. (eds.): Mixed Reality and Gamification for Cultural
Heritage. Springer, Cham (2017). https://doi.org/10.
1007/978-3-319-49607-8
Jung, Y., Kuijper, A., Fellner, D., Kipp, M., Miksatko, J.,
Conclusion Gratch, J., & Thalmann, D.: Believable virtual charac-
ters in human-computer dialogs. Eurographics 2011 –
In this entry, we have provided clear definitions state of the art report, pp. 75–100 (2011)
and latest bibliographical references for the terms Kapp, K.M.: What is gamification? and why it matters to
L&D professionals. http://learningcircuits.blogspot.gr/,
serious games, gamification, simulations, digital 1–4 (2015)
games, and related terminology suitable for mixed Kapp, K.M., Blair, L., And Mesch, R.: The Gamification of
reality continuum. Moreover, we have provided Learning and Instruction Fieldbook. Wiley, San
key future challenges that their application in the Francisco (2013)
Kateros, S., Georgiou, S., Papaefthymiou, M.,
mixed reality continuum poses. Papagiannakis, G., Tsioumas, M.: A comparison of
gamified, immersive VR curation methods for
enhanced presence and human-computer interaction
in digital humanities. Int. J. Herit. Digit. Era. 4(2),
References 221–234 (2015). https://doi.org/10.1260/2047-4970.4.
2.221
Abrash, M.: What VR could, should and almost Magnenat-Thalmann, N., & Kasap, Z.: Virtual humans in
certainly will be within two years. http://media. serious games. Presented at the 2009 international con-
steampowered.com/apps/abrashblog/Abrash%20Dev ference on cyberworlds, pp. 71–79. IEEE. https://doi.
%20Days%202014.pdf (2014) org/10.1109/CW.2009.17 (2009)
Gamification and Social Robots in Education 811

Misuraca, G.: Digital games for empowerment and inclu- Definition

sion (DGEI)D3 final vision and roadmap, pp. 1–20
(2012)
Prandi, C., Salomoni, P., Mirri, S.: Gamification in The term gamification originated in 2008 and
crowdsourcing applications. In: Lee, N. (ed.) Encyclo- became a part of the game design and digital
pedia of Computer Graphics and Games, pp. 1–6. media design industry lexicon in late 2010.
Springer International Publishing, Cham (2015). https:// Gamification involves using elements found in
doi.org/10.1007/978-3-319-08234-9_46-1
Papagiannakis, G., Trahanias, P., Kenanidis, E., and game design and applying them to contexts out-
Tsiridis, E., Psychomotor Surgical Training in Virtual side game design. A social robot has two parts: a
Reality. In: The Adult Hip - Master Case Series and physical presence that interacts with people and
Techniques. Springer, Cham, Cham, 827–830, 2018 a cyberphysical system that is connected to
Sanchez-Vives, M. V., & Slater, M. From presence to
consciousness through virtual reality. Nature Reviews cloud services. Social robots can be designed
Neuroscience, 6(4), 332–339. http://doi.org/10.1038/ to behave like humans in limited ways. By
nrn1651 (2005) using sensors, actuators, and internal systems
Sawyer, B.: Serious games: improving public policy to speak, make gestures, move, make eye con- G
through game-based learning and simulation.
Whitepaper for the Woodrow Wilson International tact, change color, and generate sounds, social
Center for Scholars (2002) robots can demonstrate emotions like joy, sad-
Ritterfeld, U., Cody, M., Vorderer, P. (eds.): Serious ness, and boredom.
Games: Mechanisms and Effects. Routledge,
New York (2009)
Vacchetti L. et al. A Stable Real-time AR Framework for
Training and Planning in Industrial Environments. In:
Ong S.K., Nee A.Y.C. (eds) Virtual and Augmented
Introduction
Reality Applications in Manufacturing. Springer,
London (2004) Gamification and social robots describe the com-
Zikas, P., Bachlitzanakis, V., Papaefthymiou, M., Kateros, bination of these two elements to increase levels
S., Georgiou, S., Lydatakis, N., Papagiannakis, G.:
of motivation and engagement in the first
Mixed reality serious games and gamification for
smart education. In: Proceedings of the European Con- instance, and increase adoption intentions and
ference on Game-Based Learning 2016, ECGBL’16, positive perceptions in the second. Since both
Paisley, vol. 1, pp. 1–9 (2016) gamification and robot design elements – as sep-
Zyda, M.: From visual simulation to virtual reality to
arate paradigms – have been shown to affect
games. IEEE Comput. 38, 25–32 (2005)
human attitude, researchers believe that the com-
bination of the two paradigms will have a stronger
effect on human behavior and attitude.
Gamification and Social
Robots in Education
Social Robot Gamification
Curtis Gittens1 and Patrick C. K. Hung2
1
Department of Computer Science, Mathematics The combination of gamification and social robots
and Physics, University of the West Indies, Cave was attempted in an educational context to
Hill Campus, Bridgetown, Barbados increase engagement, motivation, and enthusiasm
2
Faculty of Business and IT, Ontario Tech in self-learning and teacher-led environments
University, Oshawa, ON, Canada (Alsebayel and Berri 2019; Donnermann et al.
2021). Education is a major area used to study
the effects of gamification, and research has dem-
Synonyms onstrated that gamification has a positive influ-
ence on students and the learning experience
Embodiment; Interactive learning; Social robots (Hakulinen et al. 2015).
812 Gamification and Social Robots in Education

Gamification Elements and voice recognition capabilities enable the

In educational environments, four game elements robot to read prescripted text, ask questions, and
are particularly important in the gamification con- understand the learner’s response. These types of
text: (1) points, (2) badges, (3) levels, and robots can also announce achievements and
(4) leaderboards. learner points, which is more interactive and
Points are the foundation of all games. In the engaging than simply having such information
gamification educational context, these elements displayed in text form on the computer screen.
provide feedback for good conduct and report Additionally, before entering the market, com-
student progress (Sailer et al. 2017). mercially available social robots undergo multiple
Badges are realized achievements. It consists HRI studies in settings that reflect target use cases
of a name, an icon (e.g., an image of a gold cup, like elder care, education, and healthcare. This
medal, or other award), and an achievement level of testing assures researchers that the social
description (Hamari and Eranti 2011). Badges robot can reliably function in similar settings. This
give positive feedback and mark work accom- allows researchers to focus on higher-level social
plishments – just like points. They also can interactions in their studies including the robot’s
increase engagement by motivating players to effect on motivation, engagement, and enthusi-
develop game and/or level objectives. asm in learners instead of lower-level interactions
Levels. The three important level types in such as interface interaction, perception, and user
gamified systems are the following: (1) game response.
levels, which gradually rise in difficulty; Custom-built social robots. In the study by
(2) playing levels, which is a level a player may Alsebayel and Berri (2019), the custom-built
select; and (3) player levels, which can be attained social robot was integrated with the gamification
after the player gains enough experience points. system. The robot had three emotional expres-
Achieving a new level in education gamification sions: happy, sad, and resting. It could also blink
systems is used as a reward for learners that per- its eyes and move its head. Given the reduced
severed with assignments and tasks (Nah features and capabilities typical of custom-built
et al. 2014). robots, their use should be constrained in
Leaderboards show basic, similar achieve- gamification and social robot studies. Examples
ments where users can compare their perfor- of when to use a custom-built social robot may
mances, points, and badges. include the following: the need for local or
regional accents, parochial appearances, or tradi-
Social Robot Options tions that are integral to the learner’s experience.
The suggested role a social robot should play in
gamification systems is that of a tutor. This is Social Robot Gamification Configurations
based on the ideas found in self-determination There are three possible gamification and social
theory (Donnermann et al. 2021). Social robots robot configurations. Two of these configurations
that are suitable for this role can be purchased have been used in the literature by Donnermann
commercially or custom designed. The type of et al. (2021) and Alsebayel and Berri (2019).
robot used in the gamification system depends However, no work has been done to date using
on the rationale for its use. the third configuration. Details on the three con-
Commercially available social robots may figurations follow.
have a significantly larger feature set than a Robot not integrated with the gamification sys-
custom-built equivalent. One example is the type tem. In this configuration, the social robot is used
of expressions a commercially built robot can with the gamification system, but there is no inte-
express and detect in humans. Depending on the gration of the robot with the gamification system.
make, robots such as Reeti, Nao, and Pepper may Such a configuration allows researchers to inves-
have faces, heads, eyes, hands, or legs that possess tigate the effect the social robot and the
high degrees of motion. Built-in text-to-speech gamification system have on enthusiasm,
Gamification and Social Robots in Education 813

motivation, and attention as separate, and then as done using two of the three gamification configu-
combined systems. However, a potential short- rations: (i) robot combined but not integrated with
coming of this configuration is that because the the gamification system and (ii) robot connected
robot and the gamification system are not tightly to the game in the gamification system. There are
integrated, navigating the systems may become many opportunities for research to be done in the
challenging for the learner when both are in use. third configuration option where the social robot
Donnerman et al. (2021) reported the unexpected embodies the gamified system.
result of increased distraction when both systems
were in use.
Robot integrated with the game in the
Cross-References
gamification system. In this configuration, the
robot is integrated with the game that runs within
▶ Augmented and Gamified Lives
the gamification system. The degree of integration
▶ Challenge-Based Learning in a Serious Global
is limited to data sharing, where the robot acted as G
Game
an embodied user interface that indicated whether
▶ Gamification
the user’s response was right or wrong. The robot
▶ Gamification and Serious Games
possessed no ability to detect human expression,
▶ Gamification Ethics
and no other social factors were captured or used
▶ Gamification in Crowdsourcing Applications
to affect the game. The game itself kept track of
▶ Incremental Games
the correct responses and provided opportunities
▶ Serious Online Games for Engaged Learning
for the learner to correct their mistakes. There
Through Flow
were no other gamification elements in the system
beyond score keeping and game levels.
Robot embodies the gamification system. To
date, no research has been done using this config-
References
uration. As a result, there are many research ques- Alsebayel, G., Berri, J.: Robot based interactive game for
tions, including: teaching Arabic spelling. Int. J. Artif. Intell. Appl.
10(6), 15–32 (2019)
(i) To what degree should the robot embody the Donnermann, M., Lein, M., Messingschlager, T., Riedmann,
A., Schaper, P., Steinhaeusser, S., Lugrin, B.: Social
gamification system? robots and gamification for technology supported learn-
(ii) What gamification elements should be ing: an empirical study on engagement and motivation.
implemented? Comput. Hum. Behav. 121, 106792 (2021). https://doi.
(iii) How should the robot’s appearance and org/10.1016/j.chb.2021.106792
Hakulinen, L., Auvinen, T., Korhonen, A.: The effect of
behavior reflect the gamification elements? achievement badges on students’ behavior: an empiri-
(iv) How do these factors affect learner motiva- cal study in a university-level computer science course.
tion, enthusiasm, and engagement? Int. J. Emerg. Technol. Learn. 10(1), 18–29 (2015)
Hamari, J., Eranti, V.: Framework for designing and eval-
uating game achievements. In: 5th International Con-
ference on Digital Research Association: Think Design
Conclusion and Discussion Play, DiGRA. (2011)
Nah, F.F.-H., Zeng, Q., Telaprolu, V.R., Ayyappa, A.P.,
Research in gamification and social robots as a Eschenbrenner, B.: Gamification of education: a review
of literature. In: Nah, F.F.-H. (ed.) HCI in Business,
method for increasing motivation, engagement, pp. 401–409. Springer International Publishing (2014).
and enthusiasm of learners in an educational set- https://doi.org/10.1007/978-3-319-07293-7_39
ting has so far produced inconclusive results. This Sailer, M., Hense, J.U., Mayr, S.K., Mandl, H.: How
is because there are few studies done in the area to gamification motivates: an experimental study of the
effects of specific game design elements on psycholog-
date. Besides replicating, or repeating, experi- ical need satisfaction. Comput. Hum. Behav. 69,
ments were done in Alsebayel and Berri (2019) 371–380 (2017). https://doi.org/10.1016/j.chb.2016.
and Donnermann et al. (2021), whose work was 12.033
814 Gamification Ethics

Einstein according to which if he would have

Gamification Ethics known how atomic power was to be utilized, he
would have preferred to become a watchmaker
Sami Hyrynsalmi1, Kai K. Kimppa2 and instead of a scientist.
Jouni Smed3 As gamification touches the very basics of the
1
Pervasive Computing, Tampere University of playful nature of humanity, the designers and
Technology, Pori, Finland developers of gamified solutions can either
2
Turku School of Economics, University of through pure mistakes or with evil intentions cre-
Turku, Turku, Finland ate products and services which either endanger or
3
Department of Future Technologies, University worsen the condition of the players, the environ-
of Turku, Turku, Finland ment, or the society. Whereas it is hard to
enlighten developers with bad intentions, it is
crucial for the developers of gamified solutions
Synonyms with good intentions to understand the ethical
challenges inherent in the used techniques.
Dark side of gamification; Ethical issues in The main things for a developer to keep in
gamification; Ethics in gamification mind regarding ethics, on top of their own inten-
tion – which of course ought to be the good of the
stakeholders of the application – are that the appli-
Definitions cations are built in a just manner and that the
consequences benefit the client, the users, and
Gamification ethics refers to the study and under- the targets of the systems being designed (Moor
standing of right and wrong conducts by or with 1999). Even though most developers consider
gamified solutions. As gamification taps into the themselves to be good people, as Don Gotterbarn
natural playfulness of human beings, ethical always reminds us, they should also be aware that
issues are prevalent and must be considered by if they do not actively look after their character
the developers. traits related to their work in a virtuous manner,
they do not always act as they ought.
Based on the theoretical ethical principles,
Introduction applied ethics aims at tackling a certain area.
The ethical questions raised by gamification tech-
The term “gamification” usually refers to applying niques have only recently gained interest with the
game design elements into nongame contexts works of Bui et al. (2015), Hyrynsalmi et al.
(Deterding et al. 2011). Typically, it is used to (2017a, b), Kim and Werbach (2016), and Sicart
improve the motivation and performance of the (2015). The field remains still largely unexplored,
players to tasks like learning, well-being, rehabil- yet further studies are published with an
itation, or work efficiency. For example, Hamari increasing pace.
et al. (2014) show how gamification can improve While our aim here is to provide a broad over-
the players’ motivation in possibly arduous and view of ethical problems of gamification, this is
boring tasks. not a comprehensive list of all possible issues. As
The road to gamified solutions has often been both the field and the techniques are still evolving,
paved with good intentions where designers, new ethical questions are expected to pop up and
developers, and funders are aiming at improving some of the older topics will become outdated
the players’ quality of life. However, even pro- with the new systems.
jects that have been developed with good Our intention in this entry is to give an intro-
intentions may end up in creating ethically ques- duction to the ethical problems present in
tionable or even clearly morally wrong solutions. gamification. In the subsequent sections, we
This reminds of the statement attributed to Albert divide gamification ethics into three broad groups:
Gamification Ethics 815

ethical problems related to the design of exercise, and educational solutions? Thus,
gamification, ethical problems related to the tech- designers should also ask whether gamification
nology used in implementing gamified systems, brings long-lasting value in the particular context
and ethical problems related to the data utilized by or whether it would turn against its objective due
gamified systems. to potential overload.
Individual players cannot be handled as a
homogeneous group. For instance, let us consider
Issues on Gamification Design the case of an individual who is a game addict.
Should he or she be exposed at his or her work-
The design phase includes activities typically car- place to a gamified system? If not, would he or she
ried out before and during the implementation of a be in a different, possibly weaker position than
gamified system. While gamification aims at others? A similar kind of though experiment could
improving the players’ interests on virtuous issues be carried out with underaged pupils (e.g., educa-
and tasks, there are examples of using it for tional gamification), elderly, or cognitively chal- G
malevolent purposes such as stealing or damaging lenged users.
CCTV cameras or even prompting players to
commit suicide (Hyrynsalmi et al. 2017b). Omit- Societal
ting such extreme examples, there are, however, Employees can deliberately fake information on
solutions that are either legal but questionable or gamified system, for example, to use the lead-
that have been developed with good intentions erboard to advance their position in salary
whereas their consequences are ethically ques- negotiations (Callan et al. 2015). Cheating in
tionable. For an overly simplified example, a general is a likely problem, if the gamified
gamified solution for a nurse, paramedic, or fire- system has real-world benefits that can be
fighter could, in theory, improve their job satis- gained. Moreover, tapping into the competitive
faction; however, every second spent on drive of the players of a gamified system can
secondary purposes, such as gaining points in a have destructive consequences on the work
gamified environment could, literally, endanger environment as the competition leaks from
someone’s life or property in these kinds of con- gamification into the real world.
texts. The basic design question one should Technology-savvy younger players might have
always ask first is: does gamification work in an advantage in using gamification because of
this particular context? their familiarity of game mechanics from enter-
From the perspective of gamification design, tainment games.
we can recognize two general groups considering Putting the participants in an equal starting
the implications on an individual person and on position is a hard problem to solve, which is
the impacts on a society. tackled by game balancing in game design
(Adams 2014, pp. 403–405). A balanced game is
Personal fair, meaning that all players have an equal chance
Gamification overloading is a rarely addressed of winning at the start, and it should be appropri-
topic in design. An average player is likely ately challenging (i.e., not too hard nor too easy)
advance only in few different games at the same for the players. The skill in the actual task of the
time. The average player does not play several player, rather than in the game created on top of
massively multiplayer online games simulta- the tasks solved through gamification, should be
neously due to the cognitive burden caused by the most important factor in determining the
keeping up several different tasks, stories, and player’s success.
game mechanisms. The same cognitive limits Furthermore, it is possible that the setup or the
apply also to gamified solutions. Would an aver- story in a gamified system favors a majority of the
age gamification player be able to simultaneously players, ignoring the (gender or ethnic) minori-
keep up with gamified electricity saving, physical ties’ interests or values. In a workplace situation,
816 Gamification Ethics

this might even enforce the existing and possibly players to give their passwords (social engineer-
hidden attitudes and prejudices. ing attack).
The motivation behind gamification can be, in The technical attacks can be directed to the
some cases, hidden from its players. For example, clients, the servers, or the network connecting
the design of the game Ingress – developed by them (Smed and Hakonen 2017, pp. 290–291).
Niantic, a company spun off from Google – is An attack can take place over (e.g., reading pixel
assumed to have originally been motivated to values from the user interface), under (e.g.,
gather location information to improve Google’s hacking a driver), or in the client (e.g., altering
map services. From this commercially motivated the code in the memory). Apart from physical
example, we can draw a parallel to Sesame attacks (e.g., theft), servers are vulnerable to net-
Credit – developed (via affiliates) by the Chinese work attacks (e.g., IP spoofing or denial-of-
on-line marketing conglomerate Alibaba and the service attacks). Network communication can be
Chinese government – where the design motiva- compromised by tampering the packets (e.g.,
tion is, at the same time, both commercial and intercepting or replicating them or forging their
political. payload data).
If gamification is used for political purposes, it The social engineering attacks can include, but
opens the possibility for using it as a tool for are not limited, to blackmail, using the gullibility
propaganda and surveillance. The ethical implica- of the other users, gaining access through pre-
tions of this are manifold: conventional values tending to be a superuser – for some reason with-
such as “harmony” in the society are typically out superuser access – or bribing others either
enforced, a lack of revolutionary and thus with joint sharing of results (cooperating against
society-enhancing ideas can follow. And, of others unfairly) or paying smaller amounts for
course, there are clear risks for the privacy of the greater gains (Mitnick and Simon 2003).
players. Just think of Stasi (Staatssicher- The motivation behind the attacks on games
heitsdienst) or similar organization gamifying can stem from different sources (Consalvo 2007)
their surveillance of the citizens, or gamifying but broadly speaking, we can recognize three
catching illegal immigrants entering the country – areas:
one can be of the opinion that illegal immigrants
ought not to enter the country, but would it really • Enhancing the gameplay motivated by, for
be a good idea to make this kinds of consequences example, lack of skill or time or by
to actual living, breathing, and feeling human boredom
beings into a game? • Playing with the game system to explore and
experiment, extend the lifespan of the game, or
creating new ways to play
• Extra-game factors such as money, fame, van-
Compromising the Underlying dalism, or nonconformity
Technology
Although any breach of the information secu-
The technology used in gamification should pro- rity can have severe repercussions to the player,
tect the players’ sensitive information and allow the motivation plays an important role in discern-
them to decide how the information is used ing the possible ethical consequences. Players
(cf. Lahtiranta et al. 2017). Moreover, it should wanting to enhance they gameplay will, naturally,
provide a fair playing field for players and prevent increase inequality among the other players.
any kind of cheating. These attributes can be A similar situation may ensue even if the players
compromised by attacks utilizing either technical are playing with the gamified system, although
or social weaknesses. For example, passwords can their motivation is not directed against the other
be stolen by cracking them (technical attack) or participants. The biggest threat comes from the
pretending to be administrator and asking the last group. When extra-game factors are included,
Gamification Ethics 817

the other participants will be become expandable experience of health, however, is not a biomedical
and just means-to-an-end. experience but rather an existential experience;
what could be called homelike-being-in-the-
world (Svenaeus 2001). Hence, what people expe-
Use and Scope of the Gamified Data rience as good health varies from person to person
and depends on their personal goals and desires
Gamified solutions generate personal data from in life.
the players’ personal interests, actions, and habits.
Thus, the environment where the gamified solu- Work Life
tions are used affects the ethical questions of The current work-life environment – riddled with
gamification. In the following, we have identified financial crises, work automation, competition in
five environments where intentions to gamify as employment – leaves many employees with no
well as ethical questions differ: healthcare, work possibility to change their employer. Conse-
life, government, school, and leisure systems. quently, we are more and more attached to our G
current employment, and if the environment is
Healthcare gamified, a gamified system in the workplace
The healthcare sector is actively looking for ways could force us in an ever-increasing competition
to improve people’s health behavior using tech- against one another. This would turn us into an
nology, and gamification is seen as a promising exploitable “standing reserve” for corporate pur-
opportunity. It is possible to imagine a gamified poses that would take away considerable parts of
healthcare system provided by public healthcare our power over our own lives, creating new
that drives for a lifestyle change (e.g., to get rid of “rules” and endangering the authentic (self-
intoxicants, to get more exercise, or to lead a owned) being in the context of working life
generally medically reasonable life). We are not (Heidegger 1977).
criticizing gamification itself as it could be a good Nevertheless, there are many employers who
tool for many people to achieve these goals, but willingly take new technologies into work envi-
there are some risks we want to point out that are ronments with the aim of helping employees to be
involved. empowered at work. Such solutions can, for
The primary concern is that health is an area of example, be linked to job satisfaction, feedback,
life where people can be highly vulnerable, or suggestions for improvement. In such data-
because for many, it is not possible to choose the driven gamified solutions, particular care should
services they would want, for example, due to be taken to keep hidden the unique characteristics
financial or geographical limitations. The second- that would help to identify individuals.
ary concerns relate to the data produced with these
kinds of solutions: personal health records of any Government
kind are extremely private. Governmental information systems are the tools
There is a risk that the user could lose the that are used by government and citizens forming
control over the information gathered by the a part of how our society is working and commu-
gamified system, if it is also used for larger nicating. When thinking about the gamification of
healthcare purposes. There is a drive to collect government systems, one has to understand that
medical information for research purposes, systems can be such that citizens are obligated to
which is usually done in good faith. However, use them. Since the idea of gamification is to
we know from examples that the genetic informa- change people’s behavior through information
tion of entire countries has become tradeable collected through gamification, there is a risk
goods, in which individuals have lost control that individuality will be lost and the demand for
over their data. being an “average” citizen will increase. This
Healthcare gamification drives towards a should be avoided as it causes citizens to lose
biomedically desirable lifestyle. The personal their individual life goals and only become
818 Gamification Ethics

statistics in a government plan; after all, we do Leisure

have our own desires, hopes, and fears that should Gamified leisure is a different matter altogether.
be valuable in and of themselves. Although gamified systems may have an impact
In addition, citizens are unlikely to have the on who and what we are in our free time – unless
ability to know – and even less to control – who there is a monopoly (or oligopoly) of systems to
uses their information and for what purposes. use – we can always opt out as long as we under-
Another example of problems in gamification is stand the changes to us the system can make.
that it could be used to “activate” unemployed Nevertheless, we must at least be able to remove
citizens. It is often claimed that unemployed peo- our data from the system we have used or tried
ple should perform some activities to get their out, which is a minimum of control requirement.
unemployment benefits. However, gamification For instance, we could use sports applications
does not create new jobs but easily becomes just such as heart rate monitors, which these days
one more duty for those weakest in our society; provided a wealth of additional applications
this does not help them but highlights the lack of from GPS to following our sleep and beyond.
power of the unemployed. Since our governmen- These applications use our data as well as the
tal systems are a vital part of our modern society, it data of others and gamify the exercise experience.
is important to ensure that the privacy and liberties The “100 percent” is likely to be defined, at least
of citizens are secured by the government; other- in part, by utilizing aggregated user data. The user
wise, we risk the foundations and justification of can, if they want, stop using the system. However,
democratic society. they still have no control over the data already
collected, and it can later be used by the applica-
tion developers as they please.
School
For younger people, there is a risk that they may
not have the capacity to claim or the will to Summary
demand different solutions. When thinking of
gamification, pupils lack the power to choose This entry highlighted ethical issues embodied in
what is used for teaching. This underlines the the use of gamification tools and techniques. As
need to protect their privacy and other related gamification touches on the very basic nature of
rights, as they have no judicial or practical humans, it is important for the designers and
means to control how gamification and the infor- developers of gamified solutions to understand
mation collected from them affect them now or the ethical ramifications of the decisions on
later in life. design, technology, and data.
When we add gamification in education, there While our aim was to bring gamification ethics
is a risk that the division between “good” and into public discussion and extent the awareness of
“bad” pupils will be emphasized and, as a conse- possible pitfalls, we are not advocating avoiding
quence, the result may be that inequality between gamified solutions altogether. Rather, we encour-
children grows. Since pupils do not yet have full age designers, developers, and funders to actively
rights or responsibilities as adults do, it is our utilize the best of gamification techniques for the
responsibility to safeguard their right and govern best of humankind while still minding the ethical
their rights as long as they are considered to be considerations.
equal members of the society. In the case of young
people, this means that in adulthood they can
decide that they do not want their personal Cross-References
and/or identifiable information to be used; they
need to have the right to prevent the use of it and ▶ Games for Change
even to destroy information considering them, if ▶ Games for Health
they so decide. ▶ Gamification
Gamification in Crowdsourcing Applications 819

▶ Rehabilitation Games 157–173 (2016). https://doi.org/10.1007/s10676-016-

▶ Serious Games 9401-5
Lahtiranta, J., Hyrynsalmi, S., Koskinen, J.: The false
Prometheus: customer choice, smart devices, and
trust. SIGCAS Comput. Soc. 47(3), 86–97 (2017).
References https://doi.org/10.1145/3144592.3144601
Mitnick, K.D., Simon, W.L.: The Art of Deception: Con-
Adams, E.: Fundamentals of Game Design, 3rd edn. New trolling the Human Element of Security. Wiley,
Riders, Berkeley (2014) New York (2003)
Moor, J.H.: Just consequentialism and computing. Ethics
Bui, A., Veit, D., Webster, J.: Gamification – a novel
Inf. Technol. 1(1), 61–65 (1999). https://doi.org/10.
phenomenon or a new wrapping for existing concepts?
1023/A:1010078828842
In: Carte, T., Heinzl, A., Urquhart, C. (eds.) Proceed-
Sicart, M.: Playing the good life: gamification and ethics.
ings of the International Conference on Information
In: Walz, S.P., Deterding, S. (eds.) Gameful World:
Systems – Exploring the Information Frontier, ICIS
Approaches, Issues, Applications, pp. 225–244. The
2015, Association for Information Systems. (2015).
MIT Press, Cambridge, MA (2015)
URL: http://aisel.aisnet.org/icis2015/proceedings/
ITimplementation/23
Smed, J., Hakonen, H.: Algorithms and Networking for G
Computer Games, 2nd edn. Wiley, Chichester (2017)
Callan, R.C., Bauer, K.N., Landers, R.N.: How to avoid the
Svenaeus, F.: The Hermeneutics of Medicine and the Phe-
dark side of gamification: ten business scenarios and
nomenology of Health: Steps Towards a Philosophy of
their unintended consequences. In: Reiners, T., Wood,
Medical Practice, 2nd edn. Kluwer, Dordrecht (2001)
L.C. (eds.) Gamification in Education and Business,
pp. 553–568. Springer International Publishing,
New York (2015). https://doi.org/10.1007/978-3-319-
10208-528
Consalvo, M.: Cheating: Gaining Advantage in Video- Gamification
games. The MIT Press, Cambridge, MA (2007)
Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From
in Crowdsourcing
game design elements to gamefulness: defining Applications
“gamification”. In: Proceedings of the 15th Interna-
tional Academic MindTrek Conference: Envisioning Catia Prandi, Paola Salomoni and Silvia Mirri
Future Media Environments, pp. 9–15. ACM,
New York (2011). https://doi.org/10.1145/2181037.
University of Bologna, Bologna, Italy
2181040
Hamari, J., Koivisto, J., Sarsa, H.: Does gamification
work? – a literature review of empirical studies on Synonyms
gamification. In: 47th Hawaii International Conference
on System Sciences, IEEE, pp. 3025–3034. (2014).
https://doi.org/10.1109/HICSS.2014.377 Collective intelligence; Crowdsourcing: user-
Heidegger, M.: The Question Concerning Technology and generated content; Games with a purpose;
Other Essays. Harper & Row, New York, translated by Gamification: ludification; Serious games; Social
Lovitt, W. (1977)
Hyrynsalmi, S., Kimppa, K.K., Koskinen, J., Smed, J.,
collaboration; Virtual volunteering.
Hyrynsalmi, S.: The shades of grey: Datenherrschaft
in data-driven gamification. In: Meder, M., Rapp, A.,
Plumbaum, T., Hopfgartner, F. (eds.) Proceedings of Definition
the Data-Driven Gamification Design Workshop,
CEUR-WS, CEUR Workshop Proceedings, vol. 1978,
pp. 4–11. (2017a). URL: http://ceur-ws.org/Vol-1978/ Gamification is generally defined as the use of
paper1.pdf game design elements in non-gaming contexts.
Hyrynsalmi, S., Smed, J., Kimppa, K.K.: The dark side Crowdsourcing is understood as the process of
of gamification: how we should stop worrying and
study also the negative impacts of bringing game
obtaining content and/or services by involving
design elements to everywhere. In: Tuomi, P., contributions from a large group of people, in
Perttula, A. (eds.) Proceedings of the 1st International particular from an online community. The use of
GamiFIN Conference, CEURWS, CEUR Workshop gamification techniques in crowdsourcing appli-
Proceedings, vol. 1857, pp. 96–109. (2017b). URL:
http://ceur-ws.org/Vol-1857/gamifin17_p13.pdf
cations is mainly due to the need of involving and
Kim, T.W., Werbach, K.: More than just a game: ethical engaging more users and of improving their
issues in gamification. Ethics Inf. Technol. 18(2), experience.
820 Gamification in Crowdsourcing Applications

Introduction Crowdsourcing Systems and Social Media” pre-

sents some examples of crowdsourcing applica-
In recent years, new forms of game and game tions which benefit from the use of gamification.
technologies have emerged in the fields of indus- Finally “Conclusion” closes the paper.
try and academia research. In particular, it is evi-
dent in the growth of serious and pervasive games.
In this context, a new trend, called “gamification,” Gamification
has reached and won many sectors, including the
business and the marketing domains (Seaborn and The word “gamification” has been used for the
Fels 2015). Such a new trend essentially uses first time in 2010, when Jesse Schell gave a talk
game design and elements with the aim of improv- entitled “Design Outside the Box” Schell (2010):
ing users’ experience and increasing users’ he foresaw the use of game mechanisms with the
involvement in services and applications which aim of increasing users’ engagement in
are not games (Deterding et al. 2011). Its goal is non-gaming contexts (Bouça 2012). After that,
explicitly different from the merely users’ “gamification” has become a buzz and trendy
entertainment. word (Deterding et al. 2011), and its techniques
It is worth noting that gamification is not a new have been applied in different contexts and appli-
issue, but it can go back to marketing activities cations: from education (Foster et al. 2012) to
and techniques (i.e., points cards and rewards wellness and health (Cafazzo et al. 2012), from
memberships), usually exploited to engage cli- marketing (Downes-LeGuin et al. 2012) to sus-
ents, by creating or enforcing loyalty in a product tainability (Liu et al. 2011), etc.
or in a brand (Zichermann and Linder 2010). Actually, the use of game design, elements, and
Current technologies, together with the wide- mechanisms in non-gaming contexts is an old
spread and massive use of social media and topic: in human-computer interaction, the idea of
mobile devices, can be identified as joining causes exploiting enjoyable interfaces from games went
which are facilitating the diffusion and adoption back to the 1980s (Malone 1982). More recently,
of gamification techniques in so many and differ- several works have been inspired by game design
ent contexts (Seaborn and Fels 2015). in reaching the goal of identifying principles, pat-
The first successful example of gamified ser- terns, and elements that might provide joy of use
vice has been Foursquare. Starting from such an and enhancing the motivational affordances of
experience, other several gamified applications computer-supported collaborative work (Jung
were born, exploiting game elements together 2010), on the basis of researches on the motiva-
with interactive design and digital marketing tional psychology of video games (Ryan and Deci
issues (Zichermann and Linder 2010). At the 2000).
same time, gamification has been recognized as Several works propose some alternative terms
a key issue to support and incentive massive col- for gamification (Seaborn and Fels 2015;
laboration from and among users (McGonigal Deterding et al. 2011), including “ludification”
2011). (Bouça 2012), “behavioral games,” “serious
The goal of this work is to define gamification, games,” and of similar concepts, such as “games
by summarizing game design elements which with a purpose” (von Ahn 2006), “pervasive
characterize it and by describing how such ele- games,” and “augmented reality games.”
ments can be exploited in crowdsourcing systems, Even if the gamification concepts and tech-
improving crowd’s experience and involvement. niques, as well as the psychological and sociolog-
The remainder of the work is organized as ical users’ motivation, have been studied and
follows. The “Gamification” section describes applied in several contexts (Deterding
the introduction of the term, the game elements et al. 2011), a lot of questions are still open, such
which characterize it, and players’ motivations. as: “how many game elements does it take until a
The section entitled “Gamification in gamified system becomes a game?” (Seaborn and
Gamification in Crowdsourcing Applications 821

Fels 2015). In this sense, the term “gamification” – Game design principles and heuristics: they
can also be used to describe the transformation of can be identified with evaluative guidelines to
an existing system into a game: gaming elements approach a design problem or analyze a given
and concepts could be inserted into a system design solution. Some examples are enduring
(enhancing existing elements) or the system play, clear goals, and varieties of game styles.
could be converted into a game (Mekler – Game models: they can be understood as con-
et al. 2013). ceptual models of the components of games or
game experiences. Some examples are chal-
Game Design Elements lenges, fantasy, curiosity, game design atoms,
Seaborn and Fels (2015) and Mekler et al. (2013) and core elements of the gaming experience.
have analyzed several works, identifying main
game elements which are exploited in gamified Players’ Motivations
systems. A list of such elements follows: Game design mechanics and dynamics are mainly
based on users’ motivations (Blohm and G
– Points (experience points, score): they are Leimeister 2013), which can be driven by intrinsic
numerical units which indicate users’ progress. and/or extrinsic motivators. These psychological
– Badges (trophies): they are visual icons which terms describe different ways in which people can
indicate users’ achievements. “be moved to do something” (Ryan and Deci
– Leaderboards (rankings, scoreboards): they 2000). On the one hand, intrinsic motivation is
display ranks for comparison among users. defined as “the doing of an activity for its inherent
– Progression (leveling, level up): they are mile- satisfactions rather than for some separable con-
stones which indicate users’ progress. sequence.” When intrinsically motivated, a per-
– Status (title, ranks): they are textual names son acts for the fun or challenge entailed rather
which indicate users’ progress. than because of external prods, pressures, or
– Levels (stages, areas, worlds): they indicate rewards (Blohm and Leimeister 2013). On the
increasingly difficult environments. other hand, extrinsic motivation “is a construct
– Rewards (incentives, prizes, gifts): they indi- that pertains whenever an activity is done in
cate tangible, desirable items the users aim to order to attain some separable outcome,” just
obtain. like a reward (Mekler et al. 2013). In this sense,
– Roles (class, character): they indicate role- gamification is based on the extrinsic motivation,
playing elements of character. which can be effective in changing behavior and
creating loyalty in users (Bouça 2012).
Game design issues in systems applying In Blohm and Leimeister (2013), motivations
gamification have been analyzed by Seaborn and are associated to game design mechanics and
Fels (2015) and Deterding et al. (2011) and can be dynamics as follows:
listed as follows:
– Intellectual curiosity: it has been associated
– Game interface design patterns: they can be with documentation of behavior (as game
identified with common, successful interaction mechanics) and exploration (as game
design components and design solutions for a dynamics).
known problem in a context, including proto- – Achievement: it has been associated with scor-
typical implementations. Some examples of ing systems, badges, and trophies (as game
game elements based on this issue are badges, mechanics) and collection (as game
leaderboards, and levels. dynamics).
– Game design patterns and mechanisms: they – Social recognition: it has been associated with
are commonly derived from traditional game rankings, levels, reputations (as game mechan-
design, as, for example, time constraints, lim- ics), and competition and acquisition of status
ited resources, and turns. (as game dynamics).
822 Gamification in Crowdsourcing Applications

– Social exchange: it has been associated with the gamification techniques reached the main
group tasks (as game mechanics) and collabo- goals of engaging users and increasing the user-
ration (as game dynamics). generated content.
– Cognitive stimulation: it has been associated Labeling images is the aim of Mekler
with time pressure, tasks, and quests (as game et al. (2013) too. The authors applied different
mechanics) and challenge (as game dynamics). gamification strategies to an already existing
– Self-determination: it has been associated with image tagging platform and involved participants,
avatars, virtual worlds, and virtual trade asking them to enter labels related to a shown
(as game mechanics) and development and image. The authors developed different versions
organization (as game dynamics). of the same system, as many as the game elements
they applied (points, leaderboard, and levels).
Then, they have compared such versions together
Gamification in Crowdsourcing Systems with a non-gamified one. The tests show that those
and Social Media gamification elements concur in enhancing users’
performance, promoting a specific user’s
Nowadays, crowdsourcing is commonly adopted behavior.
by several systems, with the aim of performing People with special needs are taken into
distributed and collaborative tasks (Estellés- account by Kacorri et al. (2014), where the
Arolas and González-Ladrón-de-Guevara 2012). authors proposed the adoption of game elements
Some of these projects exploit human abilities in crowdsourced video captioning, so as to
when they can overcome and solve problems increase video accessibility for deaf and hard-of-
impossible to be completed by a computer, such hearing users, providing benefits also to second-
as the annotation and/or tagging of images, language learners. The authors exploited an
videos, or web and social media content. As an already existing collaborative caption editing sys-
example, the reCAPTCHA project (von Ahn tem, which has been equipped with points and
et al. 2008) takes advantage of the crowd in solv- time constraints. A pilot experiment showed the
ing CAPTCHAs to help to digitalize books and feasibility and the effectiveness of the proposed
newspaper, while the Dotsub platform (http:// gamification.
dotsub.com) aims to collect video captions from
the crowd. Gamification in Learning Language
Several crowdsourcing systems apply different and Translation Systems
gamification techniques with different aims and A very famous project which exploits
contexts. Some examples are presented in the crowdsourcing and gamification in the same
following. translation system is Duolingo (https://www.
duolingo.com/). Duolingo is a free language-
Gamification in Multimedia Tagging learning and crowdsourced text translation plat-
and Captioning Systems form. Engaged users learn a language, pro-
A famous project based on random web image gressing through the lessons, and help to
labeling is the ESP game (von Ahn and Dabbish translate any web content (including images,
2004), an online two-player game with the goal of video, tweets, and so on) into their native lan-
guessing what label the other player would give to guage, at the same time. Game elements are
the displayed image. This is one of the first exam- applied in the learning part of Duolingo: from
ples of game with a purpose (GWAP) performing points to time constraints and from levels to
useful tasks by means of crowdsourcing. The leaderboard.
gaming elements introduced are points, time con- The aim of AlRouqi and Al-Khalifa (2014) is
straints, comparison between the players the translation (also in the image-to-text form) of
(as leaderboard), and intermediate results Arabic documents, so as to make them more
(as progress). ESP game evaluations show that accessible, even to those users with low vision.
Gamification in Crowdsourcing Applications 823

The authors proposed a mobile crowdsourcing nearby. Swarm exploits badges, points, and com-
system, where they have applied gamification petition against all other users of the service to
mechanisms (in particular points, leaderboard, improve their level, becoming the “mayor” of a
and time constraints). location.
Liu et al. (2011) have designed and developed Gamification has also been applied in location-
UbiAsk, a mobile crowdsourcing application for based systems with specific purposes, such as
image-to-text translation. UbiAsk exploits points, supporting citizens with special needs in urban
status, badges, and leaderboard so as to encourage environments (Palazzi et al. 2011; Prandi
participants to tag and translate images provided et al. 2015; Salomoni et al. 2015). While the
by foreign travelers, so as to support their lan- former exploits points and rewards, in the latter
guage comprehension. Experiments were the game scope is totally different from collecting
conducted and confirm an increasing involvement data, but data gathering permits to gain weapons
of the participants when gamification strategies which can be used in zombie hunting.
are applied. G

Gamification in Location-Based Systems Conclusion

Foursquare (http://www.foursquare.com/) repre-
sents the most popular example of gamification In this work we have defined the term
in crowdsourcing systems and social media “gamification,” which identifies a still emerging
(Bouça 2012). It is a location-based mobile app, but significant trend related to serious games and
which lets the users provide information about games with a purpose. We have identified main
their location, about suggestions related to a cer- game design elements and users’ motivations at
tain location, and so on. It applies most of the basis of gamification. Moreover, we have
common game elements typically involved in described how gamification techniques can be
gamified apps: points, badges, leaderboards, and applied in crowdsourcing systems, by presenting
incentives (Zichermann and Cunningham 2011). some examples where gamification reaches the
Points are used to track game status and feedback, goals of engaging users, involving them in pro-
badges to set goals, leaderboards to add a com- viding user-generated content. In many contexts
petitive layer, and incentives represent the reward. and systems, gamification is still an ongoing
Other well-known examples are Waze and process.
Swarm. Waze is a GPS-based geographical navi-
gation application for smartphones with GPS sup-
port, which provides information and user- Cross-References
generated content about travel times and route
details, downloading location-dependent infor- ▶ Augmented Reality
mation over mobile networks. Waze uses ▶ Game-Based Interventions in Public Health:
gamification (points) to engage users and encour- Exploiting the Engaging Factor of Gameplay
age them to provide more information. Waze also
offers points for traffic or road hazard reports,
which can be used to change the users’ avatar References
and to increase users’ status in the community
AlRouqi, H., Al-Khalifa, H.S.: Making Arabic PDF books
(https://www.waze.com/). accessible using gamification. In: Proceedings of the
Swarm is a mobile app that lets users share 11th Web for All Conference (W4A’14). ACM Press,
their locations within their social network New York (2014). https://doi.org/10.1145/2596695.
(https://www.swarmapp.com/). It is a spin-off of 2596712
Blohm, I., Leimeister, J.M.: Gamification: design of
the older Foursquare, which supports users in IT-based enhancing services for motivational support
checking in to a given location, as well as making and behavioral change. Bus. Inf. Syst. Eng. 5, 275–278
future plans with friends and seeing who is (2013). https://doi.org/10.1007/s12599-013-0273-5
824 Gamification of Cooking

Bouça, M.: Mobile communication, gamification and Applications, pp. 66–73. ACM Press, New York
ludification. In: Proceedings of the 16th International (2013). https://doi.org/10.1145/2583008.2583017
Academic MindTrek Conference (MindTrek’12), Palazzi, C.E., Marfia, G., Roccetti, M.: Combining web
pp. 295–301. ACM Press, New York (2012). https:// squared and serious games for crossroad accessibility.
doi.org/10.1145/2393132.2393197 In: Proceedings of the 1st IEEE International Confer-
Cafazzo, J.A., Casselman, M., Hamming, N., Katzman, D. ence on Serious Games and Applications for Health
K., Palmert, M.R.: Design of an mHealth app for the (SEGAH2011). https://doi.org/10.1109/SeGAH.2011.
self-management of adolescent type 1 diabetes: A pilot 6165451
study. J. Med. Internet Res. 14, 13 (2012). https://doi. Prandi, C., Salomoni, P., Nisi, V., Nunes, N.J.: From
org/10.2196/jmir.2058 gamification to pervasive game in mapping urban
Deterding, S., Dixon, D., Khaled, R., Nacke, L.: From accessibility. In: Proceedings of the 11th Biannual Con-
game design elements to gamefulness: Defining ference on Italian SIGCHI Chapter (CHItaly’15), pp.
“gamification”. In: Proceedings of the 15th Interna- 126–129. ACM Press, New York (2015). https://doi.
tional Academic Mind Trek Conference: Envisioning org/10.1145/2808435.2808449
Future Media Environments, pp. 9–15. ACM Press, Ryan, R.M., Deci, E.L.: Intrinsic and extrinsic motivations:
New York (2011) classic definitions and new directions. Contemp.
Downes-LeGuin, T., Baker, R., Mechling, J., Ruyle, E.: Educ. Psychol. 25, 54–67 (2000). https://doi.org/10.
Myths and realities of respondent engagement in online 1006/ceps.1999.1020
surveys. Int. J. Mark. Res. 54, 613–633 (2012). https:// Salomoni, P., Prandi, C., Roccetti, M., Nisi, V.,
doi.org/10.2501/IJMR-54-5-613-633 Nunes, N.J.: Crowdsourcing urban accessibility: some
Estellés-Arolas, E., González-Ladrón-de-Guevara, F.: preliminary experiences with results. In: Proceedings
Towards an integrated Crowdsourcing definition. of the 11th Biannual Conference on Italian SIGCHI
J. Inf. Sci. 38(2), 189–200 (2012). https://doi.org/10. Chapter (CHItaly’15), pp. 130–133. ACM Press, New
1177/0165551512437638 York (2015). https://doi.org/10.1145/2808435.
2808443
Foster, J.A., Sheridan, P.K., Irish, R., Frost, G.S.:
Schell, J.: Design outside the box. Design, innovate, com-
Gamification as a strategy for promoting deeper
municate, entertain summit (DICE2010). http://www.
investigation in a reverse engineering activity. In:
dicesummit.org/dice_summits/2010-dice-archive.asp.
Proceedings of the 2012 American Society for Engi-
Accessed Jan 2015
neering Education Conference, pp. AC2012–AC5456
Seaborn, K., Fels, D.I.: Gamification in theory and action: a
(2012).
survey. Int. J. Hum.Comput. Stud. 74, 14–31 (2015)
Jung, J.H., Schneider, C., Valacich, J.: Enhancing the moti- von Ahn, L.: Games with a purpose. IEEE Comput. Mag.
vational affordance of information systems: the effects 39(6), 92–94 (2006). https://doi.org/10.1109/MC.
of real-time performance feedback and goal setting in 2006.196. IEEE Computer Society Press
Group Collaboration Environments. Manag. Sci. 56(4), von Ahn, L., Dabbish, L.: Labeling images with a com-
724–742 (2010) puter game. In: Proceeding of the SIGCHI Conference
Kacorri, H., Shinkawa, K., Saito, S.: Introducing game on Human Factors in Computing Systems (CHI’04),
elements in crowdsourced video captioning by pp. 319–326. ACM Press, New York (2004). https://
non-experts. In: Proceedings of the 11th Web for All doi.org/10.1145/985692.985733
Conference (W4A’14). ACM Press, New York (2014). von Ahn, L., Maurer, B., McMillen, C., Abraham, D.,
https://doi.org/10.1145/2596695.2596713 Blum, M.: reCAPTCHA: human-based character rec-
Liu, Y., Alexandrova, T., Nakajima, T.: Gamifying intelli- ognition via web security measures. Science
gent environments. In: Proceedings of the 2011 Inter- 321(5895), 1465–1468 (2008). https://doi.org/10.
national ACM Workshop on Ubiquitous Meta User 1126/science.1160379
Interfaces (Ubi-MUI’11), pp. 7–12. ACM Press, Zichermann, G., Cunningham, C.: Gamification by
New York (2011). Design: Implementing Game Mechanics in Web and
Malone, T.: Heuristics for designing enjoyable user inter- Mobile Apps. O’Reilly, Sebastopol (2011)
faces: Lessons from computer games. In: Proceedings Zichermann, G., Linder, J.: Game-Based Marketing:
of the 1982 Conference on Human Factors in Inspire Customer Loyalty Through Rewards, Chal-
Computing Systems, pp. 63–68. ACM Press, New lenges, and Contests. Wiley, Hoboken (2010)
York (1982)
McGonigal, J.: Reality Is Broken: Why Games Make Us
Better and How They Can Change the World. Penguin,
London (2011)
Mekler, E.D., Brühlmann, F., Opwis, K., Tuch, A.N.: Do
points, levels and leaderboards harm intrinsic motiva-
tion?: an empirical analysis of common gamification
Gamification of Cooking
elements. In: Proceedings of the 1st International Con-
ference on Gameful Design, Research, and ▶ On Computer Games About Cooking
Gamification of Modern Society: Digital Media’s Influence on Current Social Practices 825

through the mode of fun and gamification (Kim

Gamification of Modern 2015a). The scope of gamification has been
Society: Digital Media’s implemented in businesses from health and fitness
Influence on Current Social industries, to retail marketing, and to the education
Practices and learning industries. In 2013, the gamification
market was worth an estimated $421.3 million and
Matt Dombrowski1 and Jaime Lochner2 is expected to grow to $5.502 billion by 2018
1
University of Central Florida, College of Arts & (Gamification Market Worth nd). This would give
Humanities, School of Visual Arts & Design, the industry a compound annual growth rate of
Orlando, FL, USA 67.1 %! It would appear the use of gamification is
2
Licensed Mental Health Counselor, Assistant lucrative and will only continue to grow based on
Team Lead, CAT Team, Aspire Health Partners, the projected increase in worth.
Orlando, FL, USA Over recent years, society has witnessed the
ever-growing influence and acceptance of tech- G
nology and digital game concepts being incorpo-
Synonyms rated in our day-to-day lives. The use of these
“gamification concepts” include various psycho-
Game mechanics; Gamified experiences logical approaches regarding the use of technol-
ogy to aid in evoking, motivating, influencing
behavior and even changing the habits of the
Definition user (Kim 2015a).
Using today’s technology, users have begun to
Gamification is the implementation of game incorporate game-like point-based methods to
mechanics for use in areas typically not associated affect everything from shopping habits, education
with gaming (Kadison 2015). Gamification patterns, and even their physical and mental
engages users and solves problems through use personal health. With the ever-growing availabil-
of game-thinking and game mechanics processes ity of technologies such as wrist pedometers and
(Zichermann and Cunningham 2011). The differ- smart watches, to the language learning applica-
ence between gamification and game play (i.e., tions, we as a society are seemingly thriving more
physical fitness games versus candy clearing and more on technology and gamification to influ-
games) is that with gamification the creator is ence our everyday lives.
looking to solve a real-world problem (Kim What drives us as a society to explore and
2015a). For example, applications such as wrist accept these “seemingly empty” point-based
pedometers encourage individuals to get in shape applications that influence our actions so
through gamification and works to solve the real- strongly? What internal rewards do we receive?
world problem of weight gain. Applications What constructs are affected by us using
aimed at learning a second language encourage gamification in our everyday life? This paper
individuals to become bilingual and solves the will explore the gamified experience, current
real-world problem of limited bilingual speakers research, knowing the player, and implications
and services available. of gamification on society.

Introduction Where Do We Find Gamified

Experiences?
In today’s society, gaming culture and use of mod-
ern digital media devices has influenced the casual Though instituted in the early 2000s, the popular-
user by assisting in solving real-world problems ity of gamification has seen a relatively rapid rise
826 Gamification of Modern Society: Digital Media’s Influence on Current Social Practices

since 2013 (Gamification Market Worth nd). Its badges and encouraging statements. The individ-
initial development and implementation was ual can also interact with their peers and engage in
intended to leverage crowds, process data, and competitions to track their fitness for the day.
help influence consumer behavior through the Gamification has also impacted the educational
implementation of game mechanics (8 Reasons and business field. Deloitte (a small company)
2014). So where do we find the use of developed a gamified online training program
gamification? Many companies over the past few (Deloitte Leadership Academy) (Kim 2015b).
years have implemented the use of gamification to Use of the gamified training program resulted in
leverage their consumer interaction. The goal of an increase in amount of time people spent on
these games is to intrinsically motivate the user program and an increase in number of programs
and offer increased satisfaction (as the user is able completed. This training also showed a 37 %
to obtain immediate feedback and set achievable increase in users returning to the site (Kim 2015b).
goals), feel optimistic, (encourages self- Colleges are also using the concept of
determination and gives the user a sense of per- gamification in their orientation to increase
sonal achievement), encourage social interaction engagement. A professor at the University of
(through social exchange), and give the user Michigan went so far as to create his undergradu-
meaning (as they are working toward solving ate class in game form. This gave students the
problems) (Blohm and Leimeister 2013). ability to choose their own options to meet the
Gamification is also aimed at increasing mental learning goals of the class while getting feedback
health. It introduces the idea that the user can be and being able to join “guilds” in the classroom
autonomous while being in control and having fun (Kim 2015b). Bilingual applications have also
(Blohm and Leimeister 2013). These ideas are increased education by creating a fun way for an
based off of the idea of the “fun theory.” The fun individual to become bilingual. A study
theory asserts that fun is the easiest way to change conducted by the City University of New York
behavior for the better (The Fun Theory nd). and University of South Carolina found that 34 h
Gamification can also spark behavioral change. spent on a bilingual application is equivalent to a
As the individual is engaging in gamification, semester (roughly 11 weeks) of a language at
they are getting positive emotional feedback. university (DuoLingo 2015). This makes learning
This can spark a change in habits or create a free and accessible to anyone with access to this
new, healthier habit. For example, an application application.
that encourages mindfulness and meditation Even the retail industry has seen a rise in
encourages individuals to increase positive cop- gamification experiences. Many credit cards now
ing skills and is aimed at decreasing symptoms of have offer point systems in which they use
anxiety. One such application is advertised as a rewards such as, travel and consumer products,
gym membership for the mind and uses game to to entice the consumer to sign up for their credit
change an individual’s daily habit and routine in card (Olenski 2014). In this way, gamification is
order to improve mental health. Consequently, beneficial for both the industry and the consumer.
there has been an increasing interest in use of By participating in the loyalty reward programs,
games in child therapy and mental health over consumers feel that they are getting a “deal” by
the past 10 years (Schaefer and Reid 2000). cashing in points for simply spending their every-
Gamification has also broken through into the day dollar (this gives them the control previously
fitness industry. For example, Nike released an discussed). This gives the consumer multiple rea-
application that sparked two million individuals sons to spend money and motivates them in the
to burn 68 bn calories (Blohm and Leimeister market place (8 Reasons 2014). This also benefits
2013). There are also other popular fitness appli- the credit card companies as they can partner with
cations, which helps the individual track their various retail sponsors so that point users are
exercise, sleep, and calorie intake. As the individ- required to cash in their loyalty points with those
ual increases their steps they are rewarded through vendors. This free advertising is beneficial for the
Gamification of Modern Society: Digital Media’s Influence on Current Social Practices 827

retail companies and they in turn help sponsor and their world and a therapeutic release. Gaming
support use of the specific credit card. allows player to not only engage in the game
play but also become a bigger part of the games
community. In turn subcommunities are formed
Research and Future Direction within these gamified experiences to develop not
only a player’s social network, but also the
In 2014, 500 Thai consumers were surveyed about player’s reputation.
the influence of gamification in their everyday Gamification implements many common
lives (Zepeda 2014). The highest age of interest game mechanics into non-typical game activities.
was between 24 and 32 year olds, and 88 % of the Badges or rewards, points, level upgrades, leader-
500 consumers surveyed said they would choose boards, and challenges are just a few of the ways
to buy brands with a reward point system. In fact companies draw users into utilizing their gamified
85 % said they would pay more for the product if experience. Using these milestones and badges
there was a gamified mechanic involved in the creates support as individuals use gamification to G
product (Zepeda 2014). With this type of interest problem solve ways to accomplish the game’s
it is no surprise that businesses are jumping at the goal (Blohm and Leimeister 2013).
chance to implement game mechanics into their The badging effect promotes player participa-
products. tion by the reward of a graphic icon with any level
Other research show that child gamers are “let- or point upgrade in the game. By doing so,
ting off steam” in response to problems with badging helps members feel more involved in
friends or parents, with feeling of guilt or frustra- their community (i.e., social support as discussed
tion in their gameplay (Colwell 2007). There has earlier). Many gamified experiences start their
been an increasing interest and use of games in players off with a welcome badge just for down-
child therapy over the past 10 years (Schaefer and loading their applications. This immediately gives
Reid 2000). Child therapists often times have to the player a sense of community from the get-go.
carry multiple board games and toys into therapy In turn all badges are typically displayed on the
sessions to engage interactively with their clients. members profile so they may compete against one
Therapists use play therapy techniques to help another.
children gain emotional trust with them and heal Point systems are used to help obtain badges.
through use of fantasy in which they are able to When an application needs to provide the player
explore previous trauma or current life stressors in with a measurement of their accomplishments,
a safe and nonjudgmental environment (Schaefer they typically are rewarded points. This is typi-
and Reid 2000). The idea of taking preexisting cally done by the developers assigning points to
games and changing the rules is not uncommon. common actions such as checking in, miles run,
The future of gamification will be incorporating and even comments made on a particular forum.
these elements of games for noncommercial use For example, Texas Instruments gives points to
and providing them to doctors and health care their employees who ask and answer questions on
professionals in order to better diagnose and their internal forum. The points are, in turn, used
treat conditions. to boost an employee’s reputation and eventually
lead to social and economical promotion within
the company.
Why Do We Game? The driving force behind all gamified experi-
ences is, as mentioned, giving the player a sense
Before we understand the elements of of community, autonomy, and control (Blohm and
gamification, we must first understand why peo- Leimeister 2013). Players are influenced by
ple play games. The root of most game play is mechanics like points and badges but other ele-
interaction and engagement of the player. Players ments need to occur within the game in order to
are often times drawn to gaming as an escape from retain players for increased periods of time of the
828 Gamification of Modern Society: Digital Media’s Influence on Current Social Practices

game experience. Instant feedback and clear goals an arcade machine to play game has begun to
are two of the most important elements of keeping become a thing of the past. The elements of gam-
the user involved within the application. ing have gone beyond entertainment and found
Gamification should be composed of a variety their way into the consumer world. The
challenging but achievable tasks with compelling gamification of modern society really begins to
outcomes. By retaining players within the blur the lines of what is considered a game. Most
gamified applications, users will not only stay aspects of everyday consumer life are influenced
involved in the social community created but by elements found in games, such as, competition,
this will also influence them to bring new players reputation, and our social network.
into the social network. What is the future of gamified experiences?
As mentioned, the gamified experience has gone
beyond the arcade and entered our everyday
Knowing Your Player lives. When does the term “game” become obso-
lete in our society? Will there ever be a time a
Bartle describes four common types of gamers: place where these elements are simply inte-
killers, socialites, achievers, and explorers (Bartle grated to all aspects of our everyday routines?
2008). Killers focus on winning and direct com- In addition, will there be other uses for
petition, while socialites focus on developing a gamification beyond entertainment and
network of friends and contacts. Achievers focus consumerism?
on attaining status and achieving goals, while The future of gamification, in my opinion, is
explorers focus on discovering the unknown. to help invoke social change. Current research
Knowing these four types can help industries shows positive mental health benefits in playing
appeal to various players. commercial-based interactive games. Could
Businesses that wish to bring a gamification these same concepts be spun into gamified expe-
experience into their consumer relationship need riences to better serve treatment of patients in
to be aware of these player types. While focusing healthcare? With the increase in interest and use
on one player type might seem like the most of games in mental health, therapists are able to
logical step in developing a gamified experience, incorporate gamification into their session to
an experience with a combination of the four incorporate play therapy techniques to promote
player types might be the most impactful way in healing and behavioral changes. Future implica-
development and implementation. In doing so, tions could include gamification incorporating
businesses will be able to customize their games these elements of games for noncommercial use
and gaming experience to their consumers in very and providing them to doctors and health care
specific ways. It is important that the developing professionals in order to better diagnose and treat
business understand the competitive spectrum of conditions. As bilingual applications have done
the player when inciting players to interact with with their applications, the use of gamified expe-
their gamified experience. Players do not have to riences within the education industry is just as
fall into one of Bartle’s singular player types but, important. Providing instant, accessible, and effi-
put simply, their competitiveness must be mea- cient motivators of learning will indeed shape the
sured. Caring, collaborative, cordial, competitive, industry for years to come. Physical well being is
and combative are a few of the player motivation also increased with use of gamified physical fit-
scales (Bartle 2008). ness applications. Gamification seems to present
a holistic approach as it can incorporate mind and
body into its approach. Consumerism will
Conclusion and Discussion always be a part of this industry and practice,
but the more accessible we can make these appli-
In conclusion, the idea of sitting in front of the cations the more beneficial they can become to
board game, television with a console system, or society.
Gaming Control Using BCI 829

Cross-References
Gamified Cooking
▶ Gamification in Crowdsourcing Applications
▶ On Computer Games About Cooking

References

Blohm, I., Leimeister, J.M.: Gamification. Bus Inf Syst Gamified Experiences
Eng 5(4), 275–278 (2013). https://doi.org/10.1007/
s12599-013-0273-5
Colwell, J.: Needs met through computer game play among ▶ Gamification of Modern Society: Digital
adolescents. Personality and Individual Differences, 43 Media’s Influence on Current Social Practices
(8), 2072–2082 (2007). ISSN 0191–8869
DuoLingo: DuoLingo (Version 4.5.1) [Mobile appli-
cation software]. Retrieved from http://duolingo.com
(2015) G
Gamification Market worth $5.5 Billion By 2018. Retrieved Gaming
25 Sept 2015, from http://www.marketsandmarkets.
com/PressReleases/gamification.asp (n.d.)
Kadison, L. S.: Using gamification to increase adherence to
▶ Virtual Reality Game Engines
daily living routines (Order No. 1586088). Available
from ProQuest Dissertations & Theses Global.
(1674538550). Retrieved from http://search.proquest.
com.library.capella.edu/docview/1674538550?
accountid¼27965 (2015)
Gaming as a Service (GaaS)
Kim, B.: Gamification. Libr. Technol. Rep. 51(2), 10-0_3.
Retrieved from http://search.proquest.com.library.capella. ▶ Cloud for Gaming
edu/docview/1658221602?accountid¼27965 (2015a)
Kim, B.: Gamification in education and libraries. Libr.
Technol. Rep. 51(2), 20-0_3. Retrieved from http://
search.proquest.com.library.capella.edu/docview/1658
221615?accountid¼27965 (2015b) Gaming Control Using BCI
Schaefer, C. E., Reid, S. E.: Game play : therapeutic use of
childhood games. Wiley, New York (2000)
The Fun Theory. Retrieved 25 Sept 2015 (n.d.)
Faris Abuhashish1,2 and Ismahafezi Ismail3
1
Olenski, S.: 8 Reasons why loyalty programs are impera- Animation & Multimedia, University of Petra,
tive for marketers. Marketing Land. (2014). Web. 18 Amman, Jordan
Jan. 2016. http://marketingland.com/8-reasons-loyalty- 2
Arab Open University, Amman, Jordan
programs-imperative-marketers-109077 3
Bartle, R.: 8 Types. Retrieved 25 Sept 2015, from http://
Universiti Sultan Zainal Abidin, Gong Badak,
www.youhaventlived.com/qblog/2008/QBlog251108B. Malaysia
html (2008)
Zepeda, R.: Thai consumer sentiment towards
#Gamification. Retrieved 25 Sept 2015, from https://
www.linkedin.com/pulse/20140710100745-16088539-
Synonyms
thai-consumer-sentiment-towards-gamification (2014)
Zichermann, G., Cunningham, C.: Gamification by design: Brain control interface; Game control; Human-
Implementing game mechanics in web and mobile computer interaction
apps. O’Reilly Media, Sebastopol, CA (2011)

Definition

A Brain-Computer Interface (BCI) is a device that

Gamification: Ludification can obtain all operations of the brain that has
become a promising technology in human-
▶ Gamification in Crowdsourcing Applications computer interaction (HCI). HCI has a clear
830 Gaming Control Using BCI

primacy in the world of gaming as gamers are one frameworks (Future BNCI 2014). The number of
of the largest HCI app subscribers. The human potential users of BCI applications is largest (Ahn
thoughts are transmuted by the brain over brain et al. 2014), and the financial viability of BCI
signals and expressed as an attitude. This process technologies is high too (Van Erp et al. 2012).
is mainly carried out over brain signals, which are Furthermore, appropriate BCI devices with sen-
the key component in the electroencephalogram sors that do not need gel are launched on the
(EEG). market (Ahn et al. 2014). Taking into account
these advantages perceptions of BCI-based
games and the production for the public of EEG
Introduction instruments, it does seem apparent that the first
successful use of the platform would soon be BCI-
The gaming system is the main research trend of based games, whereas BCI’s medical technology
HCI, which comes from the widest implementa- is often presented as a groundbreaking technolog-
tion of games within various fields in this domain. ical software.
The BCI technique becomes a leading control for This means that playing instructions should be
the new virtual systems as it provides a direct operated at the exact real time with less delay and
pathway of communication between the external never changing user state. The BCI EPOC device
environment and the users’ brains. In 2017 and should be calibrated to control the game in a
2014, Ismahafezi et al. (2017) and Abuhashish proper way. Moreover, there are not many games
et al. (2014) agreed that EEG signals research is designed and developed to be used with a BCI
a current approach in BCI due to its usability in device. A BCI game needs to be designed and
the HCI field. Therefore, the rapid revolution developed in an accurate way to utilize BCI
growth of HCI in the field of gaming using BCI device capabilities completely.
considered a future pursuit. A cheap and easy way
to use the BCI device has been developed by
Emotiv, which is called EPOC. This device is Related Work
used without the intervention of tangible senses
including hands to control the game; it depends on Virtual technologies can be strong BCI advocates
brain activity that expresses the game player mov- and video games. Researchers showed that BCIs
ing control, facial expressions, and emotions. provide the correct interface instruments for both
Therefore, there is no need to use sticks, buttons, video games (Ismahafezi et al. 2018) and VR
motion sensors or gyroscopes as in traditional (Abuhashish et al. 2015a; Abuhashish 2020)
game controllers. To perform as a game controller, applications. On the other side, the researchers
the obtained EEG signals need to be interpreted widely agreed that VR is a promising and produc-
into meaningful instructions in a course that tive platform to create more research and improve
enables easy game playing. BCI systems. The process of interaction with the
Computer games always have a large market, virtual environment can be subdivided into spe-
according to Global Entertainment and Media. cific tasks (Wang et al. 2011), for instance, to
BCI Games could be a very interesting field in select and manipulate or change viewport of any
the near future to adapt this innovation. Allison objects in the virtual environment.
et al. (2007) stated that the gamer is listed as the The latest BCI technologies in the virtual
first BCI user, given that BCIs give a useful and world will allow users to modify the position of
reliable feature. Van Erp et al. (2012) have also camera within a video on the left or the right using
predicted that games and entertainment will be the two different brain stimuli, including left- and
first mainstream use for unmedical BCIs. The right-hand motor imagery (MI), or two visual-
fBNCI project reported that BCI is the second of evoked lasting potentials (SSVEPs). Also,
the top five most successful BCI support software MI-based BCIs were useful for controlling the
Gaming Control Using BCI 831

direction of a virtual car (Wang et al. 2011), nav- to the fixation of information contributes to a
igating a virtual bar (Ismahafezi et al. 2018) or better visual reaction.
moving along a virtual environment or via a vir-
tual plane (Abuhashish et al. 2015a; Abuhashish
2020). Problem Statement
Many medical and research games were
designed to study the effectiveness of brain- The approach of controlling games using BCI
controlled interfaces. At The University College has been eagerly awaited by game players since
Dublin and MediaLabEurope, the researchers the game players did not feel that they reflect the
developed a BCI-based video game called reality of their situation while playing games
MindBalance (Wang et al. 2011) for engaging using the traditional way. Furthermore, occasion-
with virtual environments. In response to phase- ally, some issues like slow response time in tra-
reversing checkerboard patterns, the formed BCI ditional game controllers appear, thus causing
uses the SSVEP created. This SSVEP signifi- disappointment. In addition, spending a long G
cantly simplifies the techniques of signal pro- period playing with traditional game controllers
cessing so that clients have little or no causes issues regarding health. It was believed
instruction (Yisi Liu et al. 2010; Wang et al. 2011). that using a brain-computer interface as a new
Neurofeedback studies have been carried out game controller may solve the issues as men-
based on existing video games with different tioned earlier. Although this research area
game consoles, such as Nintendo and PlayStation appeared relatively recently, it can grant us a
(e.g., Spyro the Dragon). In second life glimpse of the evolution of game controllers in
(Pineda 2003), and the MindBalance simulation the future. Nevertheless, the existing BCI tech-
(Lalor et al. 2005), movement visualization- nology like EPOC device does not focus on the
testing techniques have been used. In Second game controller’s application. Furthermore, until
Life, First-Person Shooter game (Pineda 2003), today, there are not many games that are
MindBalance game (Lalor et al. 2005), and designed using BCI technology. For the men-
Pacman (Wang et al. 2011), the researchers used tioned reasons, BCI-based games need to be
motor imagery applications. Low-cost designed to make the best use of brain
EEG-developers have created neurofeedback’s- computer-interface technology by providing a
modulated, exciting, and difficult games (e.g., better level of immersion to look and feel real.
Neurosky, Nintendo, Microsoft, and Arena by Many approaches are used in the five main
Emotiv) that are frequent-play-motivating for phases that are stated in Fig. 1.
the user.
With regard to the literature study (Abuhashish
et al. 2015b), it is believed that the developed Significance of the Study
EEG-based ADAPTIVE Game will sustain the
user involved in the game as the complexity Virtual reality has been utilized in several areas
level is changed by cognitive assessments of the including gaming (Abuhashish et al. 2015a). In all
participant. The results of the BCI proved high in previous studies, the main focus was on the world
avoiding disruptive visual stimulation and were of virtual reality in many fields. Moreover, con-
relatively consistent over six topics in the visually trolling a game using users’ brain signals has a
rich environment of the match, with 41 of strong effect on enhancing the interactivity due to
48 players being successfully achieved. Eighty- the expanding range of utilizing brain signals to
nine percent of respondents had acceptable preci- control the virtual environment, which still needs
sion in real time. Many subjects also increased more definition with efforts. Therefore, this sce-
progress in the completion of the game. It indi- nario depicts a significant research area to be
cates that a more focused attitude and experience investigated. The proposed approach attempts to
832 Gaming Control Using BCI

controlling recognition methods that have been

proposed and implemented within the last few
years. All of these works involved the extraction
of electroencephalogram (EEG) signals from
brain activities (Fig. 2). Then, the signals are
classified into several parameters suitable for con-
trolling the game accurately.

HIGUCHI Feature Extraction

In 2010 and 2011, Yisi Liu et al. (2010) and Wang
et al. (2011) introduced fractal dimension FD
based on the algorithm of quantification of pri-
mary controllers and described its implementation
as feedback in virtual environments. Players’ con-
troller is recognized and visualized in real time on
their virtual game by adding a parameter called
“controlling dimension” to human-computer
interfaces.
In this study, Higuchi algorithm (Higuchi
1988), based on fractal dimension computation
values, was used. Higuchi algorithm provided
improved accuracy, unlike other fractal dimension
algorithms as stated in the study of (Russell 1980;
Yisi Liu et al. 2010). At this point, the Higuchi
algorithm was evaluated using Brownian and
Weierstrass functions, where theoretical FD
values are known.
To summarize, when comparing the HIGUCHI
fractal dimension calculated values with the
Mean, they are identified as a form of control
state by correlating them with the representation
of control in the arousal-valence model. Any
change in the result of HIGUCHI fractal dimen-
sion is mapped alongside the axis of arousal
(Fig. 3), where the high value of HIGUCHI fractal
Gaming Control Using BCI, Fig. 1 Used approach dimension result is positively related to high
arousal level.
The Higuchi algorithm determines the fractal
combine users’ brains with the gaming system dimension (FD) quality of regression details.
using BCI (Abuhashish et al. 2015b). Wang et al. (2011) say that the Higuchi algorithm
(Higuchi 1988) can be relied on in order to calcu-
Brain Signal Classification late high accuracy in contrast to other fractal
In order to produce an accurate classified control- dimensional algorithms. A research assessment
ling system to be ready for mapping and synchro- on Brownian and Weiestrass methods (Wang
nizing with the gaming system, the acquired et al. 2011) has verified the utility of Higuchi
mental activity obtained by brain signals must be algorithm. However, the high value of the findings
subjected to preprocess by means of features rec- of Higuchi Fractal Dimension (HFD) is likely to
ognition. Therefore, there are lots of BCI-based generate an improved anticipation. Eqs. 1 and
Gaming Control Using BCI 833

Gaming Control
Using BCI,
Fig. 2 Controlling brain
signals

Gaming Control
Using BCI, Fig. 3
Valence_arousal model for
movement control
recognition

2 have illustrated the main parameters used as the Xm k ¼ xðmÞ, xðm þ kÞ, ⋯, xðm þ ½ðn  mÞ=k  kÞ
main parameters in HFD algorithm.
ð1Þ
Given a time series of one dimension x(1),x(2),
⋯,x(n)s, the HFD calculation algorithm can be where m ¼ 1,2,⋯,k and [ ] denotes Gaussian
defined as below: notation that rounds a number in the square
brackets to its maximum integer equal to or less
Step 1: Select one value of k than themselves, m the initial time, and k the time
Step 2: Construct the subseries Xmk from the interval. For example, when k ¼ 3 and n ¼ 100
time series as following having three subseries as follows:
834 Gaming Control Using BCI

X1 3 : xð1Þ, xð4Þ, xð7Þ, ⋯, xð100Þ X3 3 : xð3Þ, xð6Þ, xð9Þ, ⋯, xð99Þ

X2 3 : xð2Þ, xð5Þ, xð8Þ, ⋯, xð98Þ Then each subseries length Xmk is calculated.
Length Lm (k) of Xmk is equal to

int ðNm
k Þ
jx½m ¼ ik   x½m þ ði  1Þ  kj ½ðN  1Þ  ðint ððN  mÞ  kÞ  kÞ
i¼1
ð2Þ
k

Step 3: Calculate the average length L(k) of all frequencies of alpha-band power increases when
Lm(k) the human is subjected to movement control
Step 4: Repeat step 1–3 for several values of k order.
Step 5: Slope of the curve of ln(L(k)) versus
ln(k) is approximated. FD value is the Control Dataset
slope multiplied by 1. Since the recognition of control movements is about
a new domain, a dataset of EEG signals benchmark
In a previous study of the algorithm Higuchi, needs to be set up that could be used for further
this algorithm demands that the principal EEG EEG-based control recognition studies. In this
signal be divided into many signals to boost the study, by depending on the emotive brain controller
readings of human motor imagery and to create device, the dataset was collected after applying a
new parameters. This improved outcome results specific stimulus scenario in the course of analyzed
in higher reliability and allows further motor movement control to come across the inner move-
imagery to be explored. For these purposes, the ment control imaginary patterns (Fig. 4).
fractal dimension of Higuchi is ideal for most
human motor imagery.

Control Classification Based on the

Circumplex Model
It has been considered that each controller relates
to an approximation interval within the arousal-
valence model based on the Circumplex model
(Davidson et al. 1990) as in Fig. 4. To analyze
the EEG signals in terms of mind controller, two
frequency bands are usually considered: alpha
(8–13 Hz) and beta (13–30 Hz). It has been
shown that the power of these sub-bands carries
useful information related to controller states. In
1988, Higuchi (1988) showed that the left hemi-
sphere controls the right movement, which causes
a minimal amount of alpha-band power, whereas
the right hemisphere causes minimal alpha power
and, in turn, controls the left movement. More- Gaming Control Using BCI, Fig. 4 Collecting move-
over, Higuchi (1988) found that the peak in the ment imaginary brain EEG data signals
Gaming Control Using BCI 835

In addition, the research strongly depended on systems, aiming to achieve high levels of quality.
building questionnaires that can furnish us with The devices mentioned in the introduction are
the right subject regarding stimuli. In the subject considered popular even without players and do
matter, an individually random person was exam- not reflect user sentiments.
ined by giving certain stimuli to acquire imagi-
nary movement control such as go forward, go Game Controller
left, go right, and jump. Real human control generated from mental activ-
ities means that the main controller of any human
movement is the human mind based on certain
Control Mapping reactions. In this study, the focus was given on
human movement control stimulated by their
After the classification process, the real human reactions. Each movement control has an
imaginary movement control to the Circumplex approximation degree of angle that represents
model was mapped. This process involved all real the changing direction of the controlling process G
imaginary movement control that represents all that expresses the imaginary movement control
categories of the control stage. such as go forward, go left, go right, and jump
(Fig. 5).

Game Animation Behavior Controlling

Mental role control works by its action/reaction.
External effects and user interaction control the Real human behavior and characteristics simula-
game. Controlling a game based on movement tion in the gaming system has been growing in the
control imaginary is considered one of the prom- field of computer graphics and multimedia. How-
ising research domains in the animation field ever, there are many scientific studies regarding
(Abuhashish et al. 2015b) since there is a lack of real human behavior and characteristics that
interaction between game players and games in involve their recognition with their application
the course of mind controller (Abuhashish et al. (Ismahafezi et al. 2018).
2015b; Ismahafezi et al. 2018). Based on previous methods, this research come
Some ideas and concepts have been explored up with parameters to control the action/reaction
and applied to bring more significant game con- of the gaming system in terms of imaginary move-
trollers and devices in the last few years (Wang ment control in which each is represented by a
et al. 2011). Furthermore, in the majority of prior particular parameter to come up with its related
research, a wide range of devices has been behavior. By using the mind controller, the mental
employed to create immersive games and virtual activities can be sync and control. In this study,
reality experiences associated with control one of the famous low-cost devices for the mind

Gaming Control
Using BCI, Fig. 5 Go
forward transition
836 Gaming Control Using BCI

Gaming Control
Using BCI,
Fig. 6 Operational
framework for game control
using BCI

controller, namely, emotive, was used, which con- ▶ Character Animation Scripting Environment
sists of several sensors. ▶ Constructing Game Agents Through Simulated
In this proposed framework (Fig. 6), the walk- Evolution
ing movement was represented based on game ▶ Game Design and Emotions: Analysis Models
players’ action/reaction. For instance, if the player ▶ Game Development Leadership Tips
wants to move forward, his imaginary movement ▶ Game-Based Interventions in Public Health:
will be mapped based on the mapping method and Exploiting the Engaging Factor of Gameplay
will be synchronized with the gaming system. ▶ Redesigning Games for New Interfaces and
Platforms
▶ Videogame Engagement: Psychological
Conclusion Frameworks

The utilization of the brain interface can provide

strong credibility and impression to the users in References
any field. In this entry, an innovative framework
has been studied to show how to synchronize and Abuhashish, F.: Way-finding guidance using AR technol-
ogy. Int. J. Sci. Technol. Res. 9(02), 2277–8616 (2020)
control the gaming system based on imaginary
http://www.ijstr.org/final-print/feb2020/Way-finding-
movement recognition from EEG signals using Guidance-Using-Ar-Technology.pdf
BCI. Higuchi FD algorithm with a gaming system Abuhashish, F.A., Sunar, M.S., Kolivand, H., Mohamed,
was implemented since it can provide real-time F., Mohamad, D.B.: Feature extracted classifiers based
on EEG signals: a survey. Life Sci. J. 11(4) (2014)
analysis and sufficient accuracy. The study reveals
http://www.lifesciencesite.com/lsj/life1104/050_
that BCI can be used in many fields, including 19623life110414_364_375.pdf
augmented reality, virtual reality, virtual environ- Abuhashish, F.A.M., Kolivand, H., Shahrizal, M.: Frame-
ments, and games. work of controlling 3d virtual human emotional walk-
ing using Bci. Environment. 2, 4 (2015a) http://
researchonline.ljmu.ac.uk/id/eprint/8073/
Abuhashish, F.A., Zraqou, J., Alkhodour, W., Sunar, M.S.,
Cross-References Kolivand, H.: Emotion interaction with virtual reality
using hybrid emotion classification technique toward
brain signals. Int. J. Comput. Sci. Inf. Technol. 7(2),
▶ Augmented Reality Entertainment: Taking 159 (2015b) https://www.researchgate.net/publication/
Gaming Out of the Box 276457426_Emotion_Interaction_with_Virtual_
Gardenscapes and Homescapes, Casual Mobile Games 837

Reality_Using_Hybrid_Emotion_Classification_Tech Wang, Q., Sourina, O., Nguyen, M.K.: Fractal dimension

nique_toward_Brain_Signals based neurofeedback in serious games. Vis. Comput.
Ahn, M., Lee, M., Choi, J., Jun, S.C.: A review of brain- 27(4), 299–309 (2011) https://link.springer.com/arti
computer interface games and an opinion survey from cle/10.1007/s00371-011-0551-5
researchers, developers and users. Sensors. 14(8),
14601–14633 (2014) https://www.ncbi.nlm.nih.gov/
pmc/articles/PMC4178978/
Allison, B., Graimann, B., Gräser, A.: Why use a BCI if
you are healthy. In: Proceedings of the BRAINPLAY
07 Brain-Computer Interfaces and Games Workshop at Gaming Controller
ACE (Advances in Computer Entertainment), Salz-
burg, Austria. 13–15 June 2007 (2007). https://www. ▶ Query-by-Gaming
ncbi.nlm.nih.gov/pmc/articles/PMC4178978/
Davidson, R.J., Ekman, P., Saron, C.D., Senulis, J.A.,
Friesen, W.V.: Approach-withdrawal and cerebral
asymmetry: emotional expression and brain physiology
I. J. Pers. Soc. Psychol. 58(2), 330–341 (1990) https://
Gaming Infrastructures G
www.ncbi.nlm.nih.gov/pubmed/2319445
Future BNCI: A Roadmap for Future Directions in Brain/
Neuronal Computer Interaction. http://bnci-horizon-2020. ▶ Online Gaming Architectures
eu/images/bncih2020/FBNCI_Roadmap.pdf. Accessed
30 July 2014
Higuchi, T.: Approach to an irregular time series on the
basis of the fractal theory. Physica D. 31(2),
277–283 (1988) https://www.sciencedirect.com/sci
ence/article/pii/0167278988900814 Ganking
Ismahafezi, I., Sunar, M.S., Shamsuddin, S.N.W., Amin,
M.M., Arsad, M.A.M., Abuhashish, F.A.: A study on
motion capture data editing in virtual environment.
▶ Griefing in MMORPGs
World Appl. Sci. J. 35(8), 1241–1245 (2017) https://
www.idosi.org/wasj/wasj35(8)17/3.pdf
Ismahafezi, I., Abuhashish, F.A., Shamsuddin, S.N.W.,
Amin, M.M., Mohd, M.K.: Joint controller method:
application in 3D humanoid style deformation. Int. Gap
J. Eng. Technol. 7(2.14), 17–20 (2018) https://www.
researchgate.net/publication/324690538_Joint_Con
troller_Method_Application_in_3D_Humanoid_ ▶ Academic and Video Game Industry “Divide”
Style_Deformation
Lalor, E.C., et al.: Steady-state VEP-based brain-computer
interface control in an immersive 3D gaming environ-
ment. EURASIP J. Adv. Signal Process. 2005(19),
706906 (2005) https://link.springer.com/article/10.
1155/ASP.2005.3156 Gardenscapes and
Yisi Liu, Sourina, O., Nguyen, M.K.: Real-time Homescapes, Casual Mobile
EEG-based human emotion recognition and visualiza- Games
tion. In: Proceedings of the 2010 International Confer-
ence on Cyberworlds (CW ’10). IEEE Computer
Society, Washington, DC, USA, pp. 262–269 (2010). Nancy Johnson2, Sam Romershausen2 and
https://dl.acm.org/citation.cfm?id¼1918475 Newton Lee1,2
Pineda, A. Is there a linear potential at short distances?. 1
Institute for Education, Research, and
arXiv preprint hep-ph/0310135 (2003). https://arxiv.
org/abs/hep-ph/0310135
Scholarships, Los Angeles, CA, USA
2
Russell, J.A.: A circumplex model of affect. J. Pers. Vincennes University, Vincennes, IN, USA
Soc. Psychol. 39, 1161–1178 (1980) https://psycnet.
apa.org/record/1981-25062-001
Van Erp, J., Lotte, F., Tangermann, M.: Brain-computer
interfaces: beyond medical applications. Computer. 45,
Synonyms
26–34 (2012) https://ieeexplore.ieee.org/document/
6165246 Casual game; Mobile game; Puzzle game
838 Gardenscapes and Homescapes, Casual Mobile Games

Definitions themselves in the game. By completing various

side challenges, they can unlock new icons to
Casual A game that is designed to be played represent their new achievements. The player
game: occasionally for a relatively short also receives new icons by completing renova-
period of time without losing points or tions in specific areas. These icons can be cus-
competitive advantages. tomized to best fit each player. By solving
Puzzle A game that requires solving puzzles in puzzles, players earn stars that can be used to
game: order to advance in the game. clean and renovate the property as well as to
Mobile A game that runs on mobile phones. unlock new areas. Each puzzle board has a goal
game: that can be completed by matching three col-
lected pieces or removing bubbles/bushes.
Introduction Figure 1 shows a puzzle board which is the
main form of gameplay in both Gardenscapes and
Gardenscapes: New Acres and Homescapes are Homescapes. This specific puzzle board has cook-
puzzle matching mobile games created by Playrix. ies which can be broken by making matches next
Gardenscapes came out in 2016, and Homescapes to them. The side panel shows that there are
was released a year later in 2017. They became 15 moves, meaning that a player must collect the
extremely popular with over 1 billion downloads required items in 15 moves or less. It also shows
(Chapple 2020). Being casual games, they both what is required for this puzzle board: donuts and
require very little skill and are easily accessible on boosters. For example, the player must make the
smartphones as well as Facebook. donuts drop to the bottom of the board. Players
can use boosters bought before the board starts to
assist them in puzzle solving.
Gameplay Although the specific mechanics and objec-
tives of Gardenscapes may vary across different
The main story in Gardenscapes and Homescapes versions, here is the basic gameplay loop in
follows Austin the butler as he is renovating Gardenscapes:
either his family garden in Gardenscapes or his
family mansion in Homescapes. The player 1. Match three or more same objects to clear them
names themselves and picks an icon to represent and earn points.

Gardenscapes and Homescapes, Casual Mobile Games, Fig. 1 A puzzle board in Homescapes
Gardenscapes and Homescapes, Casual Mobile Games 839

2. Complete tasks assigned by the characters to buy in-game coins that can be used to give a
progress in the game and earn points. player additional moves as well as additional
3. Use points to buy and place items in the garden lives.
to decorate it.
4. Discover hidden secrets and interact with char-
acters to expand the garden further. Reception
5. Repeat the process to complete the game and
restore the garden to its former glory. While the games had seen much popularity with a
large number of downloads, they ended up getting
Similarly, Homescapes offers varying mechan- a rather negative reception when they created ads
ics and objectives in different versions, but here is that did not truly represent what the games were
the gameplay loop in Homescapes: about. The advertisements for the two games would
show game footage that was nothing like the actual
1. Match three or more same objects to clear them gameplay. Many players first downloaded the G
and earn points. games expecting a very different experience than
2. Complete tasks assigned by the characters to what the games provided. There had been talk of
progress in the game and earn points. banning the ads as many people accused the game
3. Use points to buy and place items in the house publisher of false advertising (BBC 2020).
to decorate it.
4. Discover hidden secrets and interact with char-
acters to expand the house. Conclusion
5. Repeat the process to complete the game and
renovate the house to your desired design. Mobile games are a great way for anyone to get
into playing video games. Gardenscapes and
Since the puzzle games are mostly luck based, Homescapes are examples of easy-to-play casual
there is very little skill required to win the games; games because they are easy to get into and they
it is out of the player’s control where the extra require little to no skill to play. Gardenscapes and
pieces fall. Players have to think on the fly and Homescapes’ main gameplay loop appeals toward
adapt quickly to sudden changes. With that being casual gamers (Hörgstetter 2020). With each
said, the longer a player plays the game the better level, a player earns stars that can be used to
they will get at recognizing moves that will renovate different areas. Players feel rewarded
increase their chances of winning. By teaming by completing levels as well as being able to
up with friends and other players, they can help complete renovations on a mansion or garden.
each other out by tips for the levels or gifting each
other extra lives.
Cross-References

In-App Purchases ▶ Animal Crossing: A Causal Game

Although there are in-game coins that give players

additional moves or lives, the games offer both References
free and payable prizes. For this specific micro-
transaction, players can pay $4.99 for a golden BBC: Homescapes and Gardenscapes ads banned as mis-
ticket to unlock many rewards that the player can leading. BBC News. Retrieved October 1, 2022, from
https://www.bbc.com/news/technology-54509970
obtain by playing the game and collecting specific
(2020, October 12)
pieces for that golden ticket event. There are other Chapple, Craig: Gardenscapes and Homescapes Power
microtransactions such as spending real money to Playrix Past 1 billion downloads. Sensor Tower –
840 Gatchas

Market-Leading Digital & Mobile Intelligence.

Retrieved November 18, 2022, from https:// Genetic Algorithm (GA)-Based
sensortower.com/blog/playrix-downloads-one-billion
(2020, May) NPC Making
Hörgstetter, R.: Homescapes vs. Gardenscapes: Which Is
Better? GamerForFun. Retrieved October 1, 2022, Umair Azfar Khan1 and Yoshihiro Okada2,3
from https://gamerforfun.com/is-homescapes-or- 1
School of Science & Engineering, Habib
gardenscapes-better/ (2020, November 14)
University, Karachi, Sindh, Pakistan
2
Department of Informatics, ISEE, Graduate
School of Information Science and Electrical
Engineering, Kyushu University Library, Kyushu
Gatchas University, Nishi-ku, Fukuoka, Japan
3
Innovation Center for Educational Resource,
▶ Loot Boxes: Gambling-Like Mechanics in Kyushu University, Nishi-ku, Fukuoka, Japan
Video Games

Synonyms

Artificial intelligence; GA genetic algorithm; NPC

Gaze Prioritized Graphics non-playable characters; RPG role-playing game

▶ Eye Tracking in Virtual Reality

Definition

Using genetic algorithm for creating non-playable

characters is the process of creating similar-
Gaze Tracking looking characters from a wide variety of parent
characters. After the production of subsequent
▶ Eye Tracking in Virtual Reality populations, the characters that exhibit the best
traits are chosen as the final selection.

Introduction
Gaze-Contingent Displays
Non-playable characters have always been a spe-
▶ Eye Tracking in Virtual Reality
cial part of video games. Their appearance is more
pronounced in role-playing games like the Elder
Scrolls ® series and the Fallout ® series or in the
open-world games like the Grand Theft Auto ®
Generative Music series or the Assassins Creed® series. The pres-
ence of these characters is either to provide the
▶ Adaptive Music player with objectives to complete or give a sense
of a living world with people going on about their
business. The variety of these characters is how-
ever lacking which takes the player away from the
illusion of a believable world. This lack of variety
Genetic Algorithm is attributed to the amount of man-hours required
to craft an individual character. With a tight bud-
▶ Classical Learning Method in Digital Games get, the number of these individuals remains
Genetic Algorithm (GA)-Based NPC Making 841

small; hence, the same characters are repeated functions and also for using a single parent to
throughout the game. generate variations of a creature through mutation
In this article, the utility of using genetic algo- (Hudson 2013).
rithm for creating non-playable characters in The usage of GA has been prevalent in vari-
games will be discussed. The previous approaches ous areas of character development from improv-
regarding the use of genetic algorithm for creating ing the AI, creating the morphology of the
characters will be explained first, and then the character from scratch to creating new breed of
usage of this algorithm in modern games will be similar-looking creatures by using already
analyzed. Finally the advantages of this technique defined parents through mutation and crossover.
will be explained as opposed to the normally used Video games have however not made use of this
method of character creation. algorithm for defining the NPCs due to several
reasons. One of the main reasons has been the
amount of characters that can be viewed on
Usage of GA screen at a time. Games are always trying differ- G
ent tips and tricks to give an illusion of huge
GA has been used in many research applications crowds with the least amount of processing
and character creation has not been any different. power and memory usage. The most common
Improving AI has been a major concern as GA methodology for showing a large collection of
was used to improve the bot behavior in Unreal NPCs is to use sprites as shown in Fig. 1 where
Tournament 2004 (Bullen and Katchabaw 2008). several sprites are repeated to give an illusion of
It has also been used to design decision trees to be a large collection of people.
used as bot’s AI in the game Planet Wars This technique is commonly used in modern
(Fernández-Ares et al.). games where the NPCs only behave as bystanders
GA has been used for defining the mental and do not play an active role in the gameplay.
attributes of an NPC (Khan and Okada 2013). These can be treated as graphical artifacts used to
The mental attributes of characters were com- enhance the immersion of the player when the
posed of actions which were distributed into focus is more toward the other features of the
good and bad actions. The good actions were game than the player-NPC interaction. In older
allotted low numerical values, while the bad games like Doom ®, which was a First-Person
actions were given high numerical values. The Shooter game, all the enemies were also made
concept here was that if the sum of the actions up of sprites as it helped to generate a large quan-
was small, then the character’s mental attributes tity of NPCs on screen even with low processing
contained mostly good attributes. If the sum of power as shown in Fig. 2.
the actions was large, then the character mostly The problem with sprites, however, is that, if
contained bad actions. By desiring a certain sum we want to have variety, we will need to generate
by the administrator, the GA tried to fit the art for every individual character which will
actions such that their sum came close to the increase the art assets, causing a huge demand
required value thus ensuring that an appropriate on memory to store those assets, thus degrading
character is selected with a random combination performance. As a result, the NPCs are identical
of actions. copies of each other and do not exhibit random-
Other uses have been to create creatures which ness in appearance.
move and behave in a simulated 3D world. The With the shift in displaying a character in 3D
morphologies of creatures and the neural systems rather than in 2D, the possibility of defining
for controlling their muscle forces were both gen- randomly generated characters has been made
erated automatically using genetic algorithms possible. With games like Assassin’s Creed
(Sims 1994). Genetic algorithm has been used Unity ®, the current hardware is able to produce
for breeding of multiple selected parents to pro- about 30,000 characters on screen if required as
duce offspring via crossover and mutation shown in Fig. 3.
842 Genetic Algorithm (GA)-Based NPC Making

Genetic Algorithm (GA)-Based NPC Making, Fig. 1 Crowd made up of sprites in Forza Motorsport 5 ®

Genetic Algorithm (GA)-

Based NPC Making,
Fig. 2 Multiple sprites on
screen in Doom ® (n.d.)

With that many characters on screen, a method father of the character played by the user resem-
is required that can cause the characters to adhere bled the character in facial appearance and color.
to a certain criteria and yet appear randomly. At If GA is used for creating NPCs, it is possible to
the same time if a persistent simulation is needed, create a vast number of random characters which
where new characters are introduced to replace the adhere to a certain criteria and can even be used to
old ones, the new characters should appear similar make the characters appear like people in a certain
to their older counterparts by acquiring their var- demographic. This will help in personalizing the
ious traits. Something similar was achieved on a game to the people playing in that part of the
very minor scale in the game Fallout 3® where the world.
Genetic Algorithm (GA)-Based NPC Making 843

Genetic Algorithm (GA)-Based NPC Making, Fig. 3 Thousands of NPCs in Assassin’s Creed Unity ® (n.d.)

Physical Appearance individual part of the body is modeled separately.

Current games, especially RPGs, provide a host of There is a skeleton of every character which keeps
features in order to let the players create a charac- these modules in place. The modules are swapped
ter according to their liking. Players spend count- to produce a random-looking character which
less hours just creating their own character using means that every module should have a wide
the tools provided before they even start to play variety of types to choose from (Khan and
the actual game. Elder Scrolls V: Skyrim is one Okada 2013). The disadvantage of this technique
such game where the player can define his own is the need to create 3D models of each body part,
character with a plethora of options available. An and this number can increase exponentially
example of the different types of characters that depending on how fragmented the character
can be created is best shown in Fig. 4. model is desired. The advantage however is in
The character creation tools are normally very defining various clothing articles that can be put
intuitive and can be used to create a vast majority on the character to make it appear different.
of characters, but creating such characters takes Mesh deformation allows a single mesh to be
time, and if we want to localize the look of char- changed by changing different numerical values.
acters according to a certain demographic, the task This means that the amount of effort required to
becomes overly complicated. An automated sys- make a 3D mesh is quite minimal. However
tem in such a scenario can play a very good role in careful calculations and algorithms need to be
quickly producing characters, and this is where developed so that the created mesh looks realistic
the usage of GA can be most beneficial. and allows changes within the believable body
There are two ways by which a character’s 3D shape and size. It also requires that the users are
model can be created, the first being modular provided with some predefined settings as a
character while the second being mesh deforma- starting point for their character generation and
tion. Both techniques have their advantages and they build upon that while defining their own
disadvantages. In a modular character, each characters.
844 Genetic Algorithm (GA)-Based NPC Making

Genetic Algorithm (GA)-Based NPC Making, Fig. 4 The many character options in Elder Scrolls V: Skyrim ® (n.d.)

Elitism and Sigma Scaling population to create new individuals. The benefit
The problem with NPC creation with GA is two- of this approach is that the algorithm converges to
fold. We want to create characters that first adhere a solution quickly. The disadvantage of this
to a certain set of values over a wide range of approach is that the algorithm might get stuck in
values. We then also want the characters to exhibit local maxima/minima from the beginning and
randomness within those characters so that they fit might not find out the best solution. The disad-
the selection criteria and yet remain different from vantage however favors its use because an ideal
each other. One way to solve these issues is to use solution is not a requirement for NPC creation.
elitism with sigma scaling. Elitism ensures that There needs to be a certain degree of randomness
the best individuals within a population are which elitism will provide in this case.
retained and they are used to perform crossover Sigma scaling will however ensure that the
and mutation with other members of the baseline of the fitness values remains near the
Genetic Algorithm (GA)-Based NPC Making 845

G
Genetic Algorithm (GA)-Based NPC Making, Fig. 5 Different body types (n.d.)

average (Hancock and Fogarty 1994). Standard approximating the child populations based on
deviation ensures that only those values are the interactions between the parent populations.
selected that are closer to the mean. It also If there is a persistent simulation, it will mean that
requires that there is some order in values that characters belonging to different demographics
represent the different mesh shapes or modules are going to breed together through migration
when defining a character. This is necessary as in which is normally a function used in distributed
real life similar-looking people live in different genetic algorithm implementation.
areas around the world. The differences in
height, color, and shape of people vary a little
within a local community, but it starts to vary as Conclusion and Discussion
you move further away. As a result, parent char-
acters that represent a community will need to be Genetic algorithm has been used in creating char-
defined, and their subsequent generations will acters in unique ways from defining the mental
produce characters which are random and yet attributes of a character to physical definition. GA
adhere to a defined criterion. works by mimicking the nature’s way of natural
From a fitness point of view, a human body has selection to find the best possible solution to a
three body types, namely, ectomorph, meso- problem. The usage so far has been to optimizing
morph, and endomorph as shown in Fig. 5. We the AI, for deciding the mental attributes of char-
can clearly see the body structure between the acters in an RPG setting, for defining the physical
three types which means that their body shape appearance of the characters, and for creating
attributes will be quite different from each other. optimized morphologies of creatures to improve
These attributes do not take into account the facial their movement in a 3D space. By combining
structure and skin color which will also increase these researches together, it can be hoped that
the number of attributes required to define a char- GA will be quite useful in the creation and opti-
acter physically. mization of NPCs in games and also in creating
As a result, the fitness values for the characters lifelike simulations where characters will improve
belonging to different demographics are going to with time and we are able to see the changes in the
be different. GA can use these values to approxi- population due to breeding, crossover, and muta-
mate the desired character as requested by the tion and how it affects the overall environment
users. This can even play a major role in created within a game.
846 Geoinformation System

References and Further Reading

Gesture Motion
Bullen, T., Katchabaw, M.: Using genetic algorithms to
evolve character behaviours in modern video games.
▶ Computer Graphics, Video Games, and
In: Proceedings of the GAMEON-NA 2008, McGill
University, Montreal, 13–15 Aug 2008 (2008) Gamification Impacting (Re)Habilitation,
Crowd made up of sprites in Forza Motorsport 5®: Viewed Healthcare, and Inclusive Well-Being
4 Jan 2015. http://www.hypebeyond.com/showthread.
php?tid¼1497 (n.d.)
Different body types: Viewed 4 Jan 2015. http://teemajor.com/
teemajorsblog/3-male-female-body-types-explained (n.d.)
Image courtesy of Govt. of Western Aust. Dept. of Health
(n.d.)
Fernández-Ares, A., Garcıa-Sánchez, P., Mora, A.M., Gesture Recognition
Castillo, P.A., Merelo, J.J.: Designing competitive
bots for a real time strategy game using genetic pro- ▶ Object Manipulation Using Real Hand Gesture
gramming. In: Camacho, D., Gomez-Martin, M.A.,
Gonzalez-Calero, P.A. (eds.) Proceedings 1st Congreso
for Augmented Reality Interior Design
de la Sociedad Espanola paralas Ciencias del Video-
juego, CoSECivi 2014, Barcelona. CEUR Workshop
Proceedings, vol. 1196, pp. 159–172 (2014)
Hancock, P.J.B., Fogarty, T.C.: An empirical comparison
of selection methods in evolutionary algorithms. Evol.
Comput. AISB Workshop 865, 80–94 (1994). Springer, Gesture-Based Interactions
Berlin/Heidelberg. https://doi.org/10.1007/3-540-
58483-8_7
Hudson, J.: Creature generation using genetic algorithms ▶ Data Gloves for Hand and Finger Motion
and auto-rigging. Masters Thesis. National Center for Interactions
Computer Animation at Bournemouth University, Pool
(2013)
Khan, U.A., Okada, Y.: Character Generation Using Inter-
active Genetic Algorithm. Proceedings of GameOn,
Brussels (2013)
Multiple sprites on screen in Doom®: Viewed 4 Jan 2015.
http://www.sinjinsolves.com/reviews/360/xbladoom/ Global Illumination
body.htm (n.d.)
Sims, K.: Evolving virtual creatures. In: Proceedings of the ▶ Rendering Equation
21st Annual Conference on Computer Graphics and
Interactive Techniques (SIGGRAPH ‘94). ACM, New
York, pp. 15–22 (1994). https://doi.org/10.1145/
192161.192167
The many character options in Elder Scrolls V: Skyrim®:
Viewed 4 Jan 2015. http://kupsikrecka.webnode.cz/
news/the-elder-scrolls-v-skyrim-news-23/ (n.d.) Global South
Thousands of NPCs in assassin’s creed unity ®: Viewed
4 Jan 2015. http://pcgamesnewsreviews.blogspot.jp/
▶ Video Game Culture in Cape Town, South
2014/11/gamespots-pc-reviews_11.html (n.d.)
Africa

Geoinformation System
Globalization
▶ 2-Simplex Prism as a Cognitive Graphics Tool
for Decision-Making ▶ Cross-cultural Game Studies
God of War (2018), an Action-Adventure Game 847

Kratos’ wife’s (Atreus’ mother’s) ashes on the

God of War (2018), an highest peak of all the nine realms of Norse
Action-Adventure Game Mythology. Kratos must come to terms with the
loss of his wife while also learn how to transition
Brayden Rexing2 and Newton Lee1,2 from the finely tuned instrument of war he once
1
Institute for Education, Research, and was to a caring father for his son. Kratos struggles
Scholarships, Los Angeles, CA, USA in deciding whether or not to tell Atreus of Kratos’
2
Vincennes University, Vincennes, IN, USA goodlihood and what that means for Atreus. Even-
tually, Kratos will be forced to make this difficult
decision if he wants to mend Atreus and make him
Synonyms whole.
The game consists mainly of dialogue, actions
Single-player game; Action adventure game (fighting), and puzzles. The main weapons in the
game consist of Kratos’ Leviathan Axe—an axe G
from Norse mythology that is primarily a cold or
Definitions ice weapon, and Kratos’ Blades of Chaos—twin
Greek blades both linked to chains connecting to
Single-Player Game ¼ a game that is designed for both of Kratos’ arms as a fire weapon. Both the ice
single-player mode where only one player is and fire weapons are essential for some of the
expected throughout the entire gameplay puzzles as well as the fighting because enemies
Action Adventure Game ¼ a game that com- often have a certain damage type: they are either
bines core elements from both action game and resistant or not. Kratos also has a guardian shield
adventure game genres given to him by his wife before her passing. The
shield can be used for both offense and defense.
Finally, Atreus wields a Talon Bow that can shoot
Introduction regular arrows, light elemental arrows, or electric
elemental arrows. This weapon is important for
God of War is an action-adventure game franchise both combat and puzzle solving.
developed by Santa Monica Studios and
published by Sony Interactive Entertainment that
owns the game studios. This article is about the Gameplay Mechanics
2018 God of War in the franchise that was
released for PlayStation on April 20th, 2018 God of War (2018) introduces many new mechan-
(Goldfarb 2018). Following the life of Kratos ics including new combat mechanics, puzzles, and
and how he went from being just a demigod, the a system for equipping gear, equipping abilities,
son of Zeus, to the god of war in Norse mythol- and upgrading the gear and abilities (Hornshaw
ogy. It is a solo, story-focused game that follows a 2021). The combat mechanics involve Kratos
father and his son on a journey across the nine using his axe, blades, and shield to fight various
realms. enemies. Combat also involves Kratos command-
ing Atreus to fire his bow at enemies using spe-
cialized ammo. The types of weapons the player
Gameplay and Story uses mostly depends on the player’s preference as
well as the types of enemies. For example, some
God of War (2018) is a single-player third-person enemies are immune to Kratos’ axe because it is
action adventure game. A god named Kratos and an icy weapon, so the player must switch to either
his son Atreus embark on a mission to spread the shield or the blades to defeat the enemy. If an
848 God of War (2018), an Action-Adventure Game

enemy is immune to the axe, it will usually be indirectly controlled by the player. Kratos wields
more effective to use the blades to defeat the two main weapons, the Leviathan Axe and the
enemy since the blades have the fire damage type. Blades of Chaos. Atreus utilizes a bow. The player
Unlike previous games in the God of War is not able to force Atreus to move to certain
franchise, Kratos has an armor that can be col- places, but the player can make Atreus fire an
lected, crafted, and upgraded. These upgrades arrow at a target or an area by looking at the target
allow the player to increase Kratos’ stats and or area and pressing the button or key
make him more effective in combat. The upgrades corresponding with Atreus’ shoot action. Atreus
can also be used on Kratos’ main weapons to will then fire an arrow at the target.
make them stronger while also changing their
appearance as player upgrades the weapons. In
addition to selecting the type of armor Kratos Reception
wears, higher-quality armor pieces provide addi-
tional bonuses for the player to exploit. The type God of War (2018) has sold about 19.5 million
of armor the player equips also influences six stats copies as of August 2021. This figure comes from
that apply to Kratos. Kratos’ stats are strength sales for PlayStation consoles alone, as this was
(damage from standard attacks), runic (elemental before the game was released for PC (Stockdale
and special attacks), defense (reduces damage 2021).
taken), vitality (increases maximum health), luck
(increases chance perks, experience, and money),
and cooldown (reduces recharge time of special Conclusion
attacks).
The player also has access to several abilities The main enjoyment players receive from God of
that may be used along with the weapons. Kratos’ War is the amazing and thrilling story, combat and
axe and blades both can be equipped with up to leveling systems, the mythology lessons, and the
two abilities each that can be described as a “spe- experience of conversing and/or fighting the many
cial move.” These abilities can be used as a pow- Norse mythological creatures and gods (Moore
erful attack that has a much more significant 2022). This title is particularly appealing to those
cooldown than Kratos’ basic attacks. In addition who enjoy a good story and mythology.
to these abilities, Kratos can also have another
ability not linked to his weapons but can provide
a buff to Kratos. For example, one of these abili-
Cross-References
ties resets the cooldowns of all his other abilities.
The player can also give Atreus an ability to use as
▶ God of War, An Analysis
well where Atreus will summon an elemental
animal or animals to attack enemies. The type of
animal and the number of animals depend on the
References
ability selected for Atreus (IGN 2018).
Goldfarb, A.: God of war PS4 release date announced
(23 Jan 2018). IGN. Retrieved 5 Oct 2022, from
Playable Characters https://www.ign.com/articles/2018/01/23/god-of-war-
ps4-release-date-announced
Hornshaw, P.: God of war combat guide: How to crush
The only playable character in this game is Kratos. enemies with Kratos. Digital Trends (11 Feb 2021).
Kratos is a former demigod/spartan warrior who Retrieved 4 Dec 2022, from https://www.digitaltrends.
eventually became a god by killing the former com/gaming/god-of-war-combat-guide/
IGN. Combat Mechanics – God of war (2018) wiki guide.
Greek god of war, Ares. While Kratos is the only
(19 March 2018). Retrieved 4 Dec 2022, from https://
character the player can directly control (such as www.ign.com/wikis/god-of-war-2018/Combat_
movement, attacks, combos, etc.), Atreus can be Mechanics
God of War, an Analysis 849

Moore, B.: What developer made god of war? Sports news Adventure title due to its extremely gory battles
(20 Jan 2022). Retrieved 18 Sept 2022, from https:// and in-depth storytelling. As a result, the game
www.sportskeeda.com/esports/what-developer-made-
god-war carries an M for mature rating for every title.
Stockdale, H.: God of war (2018) has sold just under The game begins with no real explanation or
20 million copies. Gfinity Esports (20 Oct 2021). character introduction. The main character Kratos
Retrieved 18 Sept 2022, from https://www. simply says, “The Gods of Olympus have aban-
gfinityesports.com/god-of-war/2018-20-million-
copies-sold/ doned me,” and he stepped off the edge of a cliff to
lead the player into thinking that Kratos has fallen.
The entire storyline for the first game is based
around Kratos trying to vanquish the God of
War, Ares. Throughout the game, the player con-
God of War, an Analysis trols Kratos and gathers many weapons and mag-
ical items and spells for use in their quests. The
Devon Myers2 and Newton Lee1,2 monsters consist of the undead and other ancient G
1
Institute for Education, Research, and Greek creatures, living, undead, or holy. Due to
Scholarships, Los Angeles, CA, USA the naturally bloody storyline and combat fea-
2
Vincennes University, Vincennes, IN, USA tures, the target audience for this game are teen-
agers and adults.
Gameplay can be controversial as there are
Synonyms some players who find that dealing with the
same combat mechanic repeatedly can be quite
Hack-and-Slash; Action-Adventure boring, but others think that the new stream of
weapons that can be collected keeps things inter-
esting. The mechanics include a simple series of
Definition light and heavy attacks, including specialized fin-
ishers that are completely different for every
Hack-and-Slash ¼ a subgenre of action games enemy the players face. The developers also
that challenges the player to fight hordes of ene- threw in other special attacks that are unlocked
mies that seem weak individually compared to the as players progress through the story, such as
powerful player character. They typically ask a different magic attacks. Other than these few sim-
player to complete a level filled with minor ene- ple gameplay mechanics, there is not much that
mies with a more powerful enemy at the end. God of War brings to the table.
Action-Adventure ¼ A mix of the action and Other than the latest installment in the series
adventure genres. Action-adventure games typi- that takes place in Nordic history, all the previous
cally have some fast-paced fighting elements of installments take place in Ancient Greece. The
action games with a story and world of an story revolves around the death of Kratos’ family,
adventure game. who were killed by Kratos himself under the
God of War, a single player game created by influence of Ares. Kratos swears revenge for his
Santa Monica Studios on the Bluepoint Engine, fallen loved ones and sets out to find the ancient
was published and released by Sony Computer Greek artifact called Pandora’s Box. Legend says
Entertainment on March 25, 2005. The game is that this box, when opened, could give any mere
part of an eight-game series spanning across mul- mortal the power to kill a god. Kratos plans to use
tiple Sony consoles. As of August 2021, God of this box to slay Ares for vengeance. Throughout
War 4 has sold over 19.5 million copies, making it this journey, players meet actual Greek mytholog-
the best-selling PlayStation 4 game as well as the ical characters, such as Athena, otherwise known
best-selling game in the series. as the Goddess of Wisdom.
Throughout the entire series, God of War has Levels in this game are not meant to be more
always been called a Hack-and-Slash or Action- than just a quick change of environments, but
850 God of War, an Analysis

rather a smooth flowing of events from one area to God of War was one of the major titles in the
another in order to strengthen the effect of the Playstation 2’s game lineup, and it is conceived as
storytelling. To give an example, after the players one of the most popular Playstation 2 games, sell-
find out that they need to travel to the Desert of ing 4.6 million copies. Its success has led to seven
Lost Souls to retrieve Pandora’s Box, they are met sequels (Harradence 2019).
with a cutscene or backstory that explains why However, God of War has one major contro-
Kratos needs the box in the first place. This versy on the subject of nudity in the game (Cooper
cutscene shows Kratos working for Ares by 2017). While most players thought the mature
destroying villages who do not follow the ways rating was due to the blood, gore, and violence,
of God. Only when it is too late does he realize some of the player base was met with a surprise
that he is being manipulated by Ares into targeting when they are given the opportunity to interact
the village in which his family resides. This with a woman on bed. Although players do not see
cutscene gives the players the reasons for all anything obscene, they can certainly tell what is
these events in the game. happening from the sounds and right joystick
The entire game is played in a third-person movements. There are also creatures in the game
mode with some cutscenes going into first-person who lack clothing, and the player is forced to see
mode. The interface consists of a health and mana them naked when fighting them, thus unnerving
bar at the top left corner. Instead of an enemy some players who do not want or intend to
health bar, players get a finisher button for when see this.
an enemy is weak enough to be killed. If there is a Looking at the God of War series, every game
boss in the vicinity, a new bar that contains the in the series so far has received good reviews from
boss’ or bosses' health will appear down at the GameInformer, IGN, and others. In fact, the latest
bottom of the screen that is both larger and longer installment got a perfect 10/10 rating from IGN.
than the health bar. For the very first installment, many people opined

God of War, an Analysis, Fig. 1 God of War 1

God of War, an Analysis 851

that God of War was the highlight of Playstation Here is a brief history of the God of War series:
2. Alex Navarro of GameSpot wrote, “God of War The first God of War was released in March
is one of the best action adventure games on the 2005. It has wonderful graphics for that era of
PlayStation 2, and it should not be missed” gaming. The game is often called the highlight
(Navarro 2005). of Playstation 2’s life and is generally highly

God of War, an Analysis, Fig. 2 God of War 2

God of War, an Analysis, Fig. 3 God of War 3

852 Graphical Interface

God of War, an Analysis, Fig. 4 God of War 4

recommended. The music consisted of loud choirs God of War 4 Playstation 4 – Valkyries: https://
followed by other instruments, such as gongs, to youtu.be/JbpD7Zn8EPU?t¼47m25s
give a sense that the main character is powerful
(see Fig.1).
The rest of the games throughout the series Cross-References
all share the same concept, but the visuals and
sounds have improved. New weapons and abil- ▶ Bayonetta 2, an Analysis
ities were thrown in as the series progressed.
The storyline has also changed throughout the
series. Kratos goes from killing one god to kill- References
ing all gods which throws the world into chaos.
Here are a couple of images to show the boss Cooper, D. God of War May Remove Controversial Series
Tradition. Game Rant. December 23, 2017. https://
fight progressions through the games (see Figs.2
gamerant.com/god-of-war-ps4-nudity-sex-mini-game-
and 3). esrb/
The real changes to the game occurred in God Harradence, M.: God Of War Dev Almost Gave Kratos
of War 4, with the introduction of a completely A Different Name. PSU (PlayStation Universe). July 8,
2019. https://www.psu.com/news/god-of-war-dev-
different soundtrack as well as a new style of
kratos-different-name/
combat and extremely detailed graphics. There is Navarro, A.: God of War Review. Gamespot. March
also a shift from Greek mythology to Norse 21, 2005. https://www.gamespot.com/reviews/god-of-
mythology that introduces new gods, enemies, war-review/1900-6120758/
and landscapes. Krato’s appearance was also
changed in the new addition:
The following YouTube videos illustrate how
music has changed from the earliest edition (God
of War 1) to the new installment (see Fig. 4) Graphical Interface

God of War 1 Playstation 2 – The Vengeful Spar- ▶ Game Interface: Influence of Diegese Theory
tan: https://youtu.be/yPPU0gcWk38?t¼6s on the User Experience
Griefing in MMORPGs 853

Definitions
Graphics Applications
MMORPGs refer to massively multiplayer online
▶ Teaching Computer Graphics by Application role-playing games, in which many players play
cooperatively or competitively within a persistent
world. Griefing is generally defined as a type of
online bullying within a gaming context and is
Graphics for Branding quite common in MMORPGs. Griefing involves
one player intentionally disrupting another
▶ Teaching Computer Graphics by Application
player’s game experience, through some form of
harassment, as they enjoy doing so.

Gratification
Introduction G
▶ Videogame Engagement: Psychological
Frameworks Massively multiplayer online role-playing games
(MMORPG) are a genre of game that is played
online with hundreds to thousands of people
playing simultaneously. Each player controls
Gray Zone an avatar and can play cooperatively or competi-
tively in the persistent virtual world (Achterbosch
▶ Toxic Behaviors in Online Gaming
et al. 2008; Wolf 2012). In MMORPGs and other
online virtual worlds, players may encounter what
is known as griefing. This is when one player
Griefers deliberately disrupts another player’s game expe-
rience for their own personal enjoyment, with
▶ Griefing in MMORPGs potential in-game gain (Bartle 2007; Foo 2008;
Foo and Koivisto 2004). The following is an
example scenario that outlines this phenomenon:
You’ve had a hard day at work. As soon as you
Griefing arrive home, you log into the latest MMORPG to
relax and wind down for the day. Your in-game
▶ Griefing in MMORPGs avatar is inexperienced, so you approach a virtual
farmer and accept their seemingly simple quest that
will increase your character’s experience points, as
well as reward you with a small amount of gold.
You proceed through the forest towards the
Griefing in MMORPGs location of the computer-programmed werewolves
that have been terrorizing the farm. Exiting into a
clearing you spot the alpha werewolf and engage in
Leigh Achterbosch and Peter Vamplew combat. You are winning the battle as you outrank
Faculty of Science and Technology, Federation the werewolf in melee combat. Just as you are about
University Australia, Mt Helen, Ballarat, to strike the finishing blow, out of nowhere a vet-
eran real-life player appears behind you and stabs
VIC, Australia you in the back, killing you in a single hit. They
proceed to finish off the werewolf and collect the
winnings for themselves, although to them the loot
Synonyms was secondary to the satisfaction of killing another
player and halting their progress.

Anti-Social Behavior; Ganking; Griefers; This scenario illustrates how a highly experi-
Griefing; Harassment enced player may use unfair advantage of
854 Griefing in MMORPGs

outmatched character progression to surprise et al. 2013), while 9.0% of all players that are
attack a wounded and inexperienced player. This subjected to griefing indicated ganking happened
type of griefing action is known as ganking, and to them multiple times per day (Achterbosch
players have acknowledged that it and many other 2015).
types of griefing occur regularly in MMORPGs Verbal Harassment is where chat and voice
(Achterbosch 2015). communications are misused to offend, harass,
insult, threaten, or humiliate another player. This
was commonly (just over 80%) defined as
Defining Griefing and Griefers griefing, and only 19.0% of players that are
subjected to griefing had never been griefed in
Bartle (2007) defined a griefer as someone who this method. However, the most pervasive type
deliberately did something for the pleasure in of griefing is spamming, which refers to intention-
knowing it caused others pain. Foo and Koivisto ally filling chat channels repeatedly with mes-
(2004) described griefers as players that engage in sages of low relevance or that are against the
play styles that specifically disrupt other players game rules. Just over 20% of the sample audience
experiences, while Lin and Sun’s (2005) defini- in Achterbosch (2015) indicated this occurred
tion was similar with the addition that they derive multiple times a day. Other highly pervasive
enjoyment from their behavior. Barnett and types include spawn camping, ninja looting, and
Coulson (2010) relate a griefer to a bully, stating kill stealing. In contrast, a deceptive griefing type
that these players enjoy participating in anti-social known as scamming is not very widespread with
behaviors that disrupt other players’ enjoyment of 64% having never been exposed to it.
the game.
It is apparent that researchers agree on the
overall terminology of the words griefing and Motivations for Griefing
griefer, with the perception that three effects
need to be present for a player to be considered a Achterbosch et al. (2017a) discovered that
griefer: griefers are motivated by a few factors of potential
gain. Pleasure is a common factor among all
1. The action is intentional. griefers as prior research suggested, but power,
2. The action disrupts another player’s challenge, and/or control are also strong motivat-
enjoyment. ing factors for some griefers. For example, the
3. The instigator gains pleasure from the action. player that retaliates against a griefer with some
griefing of their own is challenging the griefer,
If these three are present, then the player that and through defeating them, feels pleasure. The
causes the action can be called a griefer. Some- player that manipulates and deceives does so as
times, the griefer is looking for more than just they find it pleasurable to control situations and
pleasure and using griefing as an instrument to the potential lucrative gains of monetary value
gain something valuable to them within the game. serve to enhance their power.
Griefing behaviors can stem directly from
the “Online Disinhibition Effect,” which refers
Pervasive Griefing Actions to abandoning inhibitions and regular social
norms when interacting with other people online
Different types of in-game actions can be attrib- due to the anonymity and invisibility available
uted as griefing according to the definition above. and lack of repercussion (Suler 2004). With ano-
Ganking, as described in the introduction is one nymity, a player’s propensity towards griefing
type, and 78.6% of a representative sampled audi- increases (Chen et al. 2009). Griefing may be
ence defined this action as griefing (Achterbosch directed at players of either opposing or friendly
Griefing in MMORPGs 855

factions. It was documented that more griefing impact, and identifies with a few examples, dif-
was directed towards players of the same team ferent types of griefing and griefers.
within a participant observation of griefing
(Achterbosch et al. 2017b). The author joined
groups with total strangers on multiple occasions,
Cross-References
often known as a “pick-up-group” or PUG, in
which many cases of griefing was witnessed. ▶ Online Players: Engagement, Immersion, and
Absorption Across Secondary Worlds
With no fear from team disruption to stealing
▶ Sociality of Digital Games
items to verbal harassment.

Impact of Griefing References

Achterbosch, L.: Causes, magnitude and implications of

Griefing can hugely impact the players subjected to griefing in massively multiplayer online role-playing G
each action. A well-known case revolves around a games. PhD thesis, Faculty of Science and Technology,
perpetrator “Mr. Bungle” performing virtual rape. Federation University Australia (2015)
Achterbosch, L., Pierce, R., Simmons, G.: Massively
This griefer manipulated the game mechanics and
multiplayer online roleplaying games: the past, present,
program code to sexually assault multiple inhabi- and future. Comm-ent. 5(4), 9:1–9:33 (2008)
tants in an online virtual world (Dibbell 1999). Achterbosch, L., Miller, C., Vamplew, P.: Ganking, corpse
Many players were in shock and some admitted camping and ninja looting from the perception of
the MMORPG community: acceptable behavior or
to shedding tears in real-life. In further studies, it unacceptable griefing? In: Proceedings of the 9th
has been documented that many different types of Australasian Conference on Interactive Entertainment:
griefing affect individuals in vastly different ways Matters of Life and Death, Melbourne, Australia,
and levels of intensity (Achterbosch 2015). For pp. 19:1–19:8. (2013)
Achterbosch, L., Miller, C., Vamplew, P.: A taxonomy of
example, 20% of new players subjected to griefing griefer type by motivation in massively multiplayer
admitted that they played less, 7.1% stopped online role-playing games. Behav. Inform. Technol.
playing for a time, and 5.1% quit playing the 36, 846–860 (2017a)
game never to return. Players subjected to scam- Achterbosch, L., Miller, C., Vamplew, P.: Participant
observation of griefing in a journey through the World
ming and ganking indicated similar (but slightly of Warcraft. Loading. . .. J. Can. Game Stud.
lower) results. All three of these types of griefing Assoc. 10(17), 40–59 (2017b)
were rated as high to extreme in intensity by Barnett, J., Coulson, M.: Virtually real: a psychological
roughly one third of the players subjected to perspective on massively multiplayer online games.
Rev. Gen. Psychol. 14(2), 167–117 (2010)
them. The impact here represents a growing con- Bartle, R.A.: What to call a griefer? Terra Nova: Simula-
cern for not only players but also developers tion + Society + Play http://terranova.blogs.com/terra_
looking to keep a strong player base. nova/2007/10/what-to-call-a-.html (2007). Accessed
2 Dec 2017
Chen, V.H., Duh, H.B., Ng, C.W.: Players who play to
make others cry: the influence of anonymity and
Conclusion immersion. In: Proceedings of the International Con-
ference on Advances in Computer Entertainment
With the success of the MMORPG genre, griefing Technology, New York, USA, pp. 341–344. (2009)
Dibbell, J.: A rape in cyberspace. In: My Tiny Life, 1st edn.
has become an extremely pervasive problem with Holt Paperbacks, New York (1999)
differing levels of intensity for each griefing inci- Foo, C.Y.: Grief play management. VDM Verlag,
dent, and it needs to be addressed. In this article, Saarbrücken (2008)
definitions for MMORPGs, griefing, and the insti- Foo, C.Y., Koivisto, E.M.I.: Defining grief play in
MMORPGs: player and developer perceptions. In: Pro-
gators of griefing, called griefers, have been ceedings of the 2004 ACM SIGCHI International
provided. In addition, this article discusses moti- Conference on Advances in Computer Entertainment
vations for causing grief, pervasive griefing, the Technology, New York, USA, pp. 245–250. (2004)
856 Group Games Computer Games Industry

Lin, H., Sun, C.: The ‘White-eyed’ player culture: grief

play and construction of deviance in MMORPGs. In: Group Games Computer
Changing Views: Worlds in Play: Proceedings of the
2005 Digital Games Research Association Conference, Games Industry
University of Vancouver, Vancouver (2005)
Suler, J.: The online disinhibition effect. Cyberpsychol. ▶ Social-, Mobile-, and Multi-Player-Games and
Behav. 7(3), 321–326 (2004) Their Impact on Today’s Online Entertainment
Wolf, M.J.P.: Encyclopedia of Video Games: The Culture,
Technology, and Art of Gaming. ABC-CLIO, Santa Industry
Barbara (2012)
H

Haar Cascade Classifier Games, an independent studio in San Francisco

founded in 2009, and is best known for their
▶ Digital Images Using Heuristic AdaBoost Haar critically acclaimed games Bastion (2011), Tran-
Cascade Classifier Model, Detection of Partially sistor (2014), and Pyre (2017).
Occluded Faces

Development
Hack-and-Slash
Hades was first launched as an Early Access game
▶ Bayonetta 2, an Analysis originally in December 2018, exclusively on the
▶ God of War, an Analysis Epic Games Store, then on Steam in December
2019. In September 2020, the team completed
their Early Access development and launched
Hack-n-Slash v1.0 of the game, additionally adding the game
on the Nintendo Switch platform. This game was
▶ Kingdom Hearts (2002): An Analysis given a Teen rating by the ESRB. The game was
designed as an Early Access from ground up. This
allowed the community to play a creative and
Hades: An Analysis important part in aiding the developmental pro-
cess of the game, from design, worldbuilding, and
Rafael Gonzales1 and Sercan Şengün2,3 storytelling, and helped the game naturally evolve
1 (Schodt 2020).
Creative Technologies Program, Illinois State
University, Normal, IL, USA
2
Wonsook Kim School of Art, Illinois State
University, Normal, IL, USA Storyline
3
Massachusetts Institute of Technology CSAIL,
Cambridge, MA, USA Hades places the player in control of Zagreus, the
son of the God of the Underworld, Hades, as he
attempts to escape from the depths of the Under-
Definitions world and reach Mount Olympus. With every
attempt that Zagreus makes escaping from his
Hades is a rogue-like hack-n-slash dungeon- father, the Olympians on the over world aid in
crawler game that was developed by Supergiant his endeavor. There are 30 differently fully voiced
© Springer Nature Switzerland AG 2024
N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
858 Hades: An Analysis

characters, allowing players to experience the The player has six unique weapons to choose
story through thousands of unique events and from before beginning a run. The player will be able
interactions. The narrative slowly unfolds over to perform a primary attack, a special attack, and a
the progression of the game, creating different magic spell. Each chamber has the possibility of
opportunities and dialogue, about ten hours’ earning a reward to upgrade Zagreus’ weapons via
worth, between Zagreus, the Olympians, and Daedalus’ Hammer. These upgrades range from
other residents of the Underworld on every run. having more range or damage on a weapon or
For instance, Zagreus may meet Eurydice during a completely changing how the weapon attacks.
run, and when returning to the main house may Each chamber also has the possibility of granting
meet Orpheus, and because the player met Euryd- an aid from an Olympian, called a “boon.” These
ice prior, he may ask to have a message delivered boons improve the attacks of Zagreus based on the
to her, creating another chain of events that the theme that each Olympian has. For instance, grab-
player can seek to uncover. bing a boon from Zeus will grant Zagreus electrify-
Successfully escaping the first time will not ing attacks and abilities, or Aphrodite will grant the
end the game’s story, the player must go again to “Weak” status, causing enemies to do less damage
uncover what happens after that. to the player. Sometimes dialogue between two
gods will ensue after picking up their respective
boons, and the player will be able to select a com-
Game Mechanics bined boon from them. The player also may choose
a boon that enables Zagreus to call upon a god’s aid
The game is presented in an isometric point of when his “God Gauge” bar is filled to a certain
view. The game features four “biomes” that amount. Players can mix and match different
Zagreus must escape through. Notably, they are boons, upgrades, and weapons to optimize their run.
Tartarus, Asphodel, Elysium, and the Temple of Before starting a run, players can choose and
Styx. When a player starts a run, they begin in a upgrade which attributes Zagreus will have via the
room outside the main house. Afterwards, the Mirror of Night. These are upgraded through
player must fight their way through a series of darkness and are permanent through each run.
chambers and enemies. The layouts of each cham- For instance, players can choose to add more
ber are predetermined, though the order of the health, add an additional dash, or add more casts
chambers, as well as the number and types of for their magic spell. The player can choose to
enemies in the chambers, are different in every reroll their darkness to create different combina-
run. In some cases, there are chambers in which tions of these attributes. They are also able to
there are no encounters with enemies; instead the upgrade their weapons when they have progressed
player can shop, restore health, or meet a resident through the game enough and have the materials
of the Underworld and gain benefits from them. to upgrade them. The materials can be obtained
Each level will contain a certain number of cham- through defeating the bosses on each level.
bers as well as a boss. After clearing each cham- After successfully escaping the first time,
ber, the player will be presented with a reward, the player will have the opportunity to ramp up
which is shown previously above the chamber the games’ difficulty to earn more rewards via the
door. These rewards range from gifts from the Pact of Punishment. Through this, a player can
Olympians, improvements for those gifts, cur- turn up the “heat gauge” and add constraints to
rency for renovating the main house, upgrades themselves, make enemies more difficult, and
for Zagreus’s weapons, keys, and skill-point cur- other “punishments.” This allows for refreshing
rency called “darkness” to improve Zagreus’ attri- re-playable gameplay, and players can test and
butes. If Zagreus’ health points reach zero, he improve their skills. If players are more geared
“dies,” and emerges in the main house again. towards progressing the story, they can choose to
Any gifts granted on a run, besides currency, enable “God Mode” which makes them stronger
darkness, and keys, are lost upon death. on every run.
Headphones 859

Reception
Haptic
The game has been praised for its marriage of
“fantastic combat” and engaging story ▶ Tactile Visualization and 3D Printing for
(Macgregor 2020). Vazquez (2020) highlighted Education
that “Hades’ narrative is so entwined with its
combat is nothing new for the developers at
Supergiant Games, who’ve established them-
selves as masters of putting your actions in sync Harassment
with the stories they tell.” Supergiant Games’
previous game Bastion also garnered attention ▶ Griefing in MMORPGs
for its combination of gameplay mechanics and
branching storyline (Mitchell 2016). As of
November 2020, the game has a Metacritic score
of 92 out of 100. (https://www.metacritic.com/ HCI
game/pc/hades) H
▶ Cognitive Psychology Applied to User Experi-
ence in Video Games

Cross-References

▶ Video Games HD, High Definition

▶ The New Age of Procedural Texturing

References

Macgregor, J.: (September 18, 2020). Hades Review.

PCGamer.com. Retrieved from https://www.pcgamer.
com/hades-review/
Headphone
Mitchell, L.: The Political and Ethical Force of Bastion, or,
Gameplay and the Love of Fate. Loading. . .. ▶ Immersive Auralization Using Headphones
10(15) (2016) https://journals.sfu.ca/loading/index.
php/loading/article/view/189
Schodt, C.: (January 31, 2020). ‘Hades’ made me a
believer in early access games. Engadget.com.
Retrieved from https://www.engadget.com/2020-01- Headphone Impulse Response
31-hades-early-access-believer.html
Vazquez, S.: (September 18, 2020). Hades Review – The
Long, Hard Road Out of Hell. Gamespot. Retrieved
▶ Immersive Auralization Using Headphones
from https://www.gamespot.com/reviews/hades-
review-the-long-hard-road-out-of-hell/1900-641
7568/
Headphone Transfer Function

▶ Immersive Auralization Using Headphones

Hand and Finger Motion

Tracking
Headphones
▶ Data Gloves for Hand and Finger Motion
Interactions ▶ Immersive Auralization Using Headphones
860 Head-Related Impulse Response

Head-Related Impulse Healthcare Robots with

Response Islamic Practices

▶ User Acoustics with Head-Related Transfer Patrick C. K. Hung1, Farkhund Iqbal2 and
Functions Inon Wiratsin1
1
Faculty of Business and IT, Ontario Tech
University, Oshawa, ON, Canada
2
College of Technological Innovation,
Zayed University, Abu Dhabi,
Head-Related Transfer United Arab Emirates
Function

▶ User Acoustics with Head-Related Transfer Synonyms

Functions
Islam; Islamic calendar; Islamic prayers; Medical
robot; Muslim beliefs

Headset Definition

▶ Immersive Auralization Using Headphones Artificial Artificial Intelligence (AI) refers to

Intelligence the simulation of human
intelligence by machines,
especially computer systems.
Decision Decision Support System (DSS) is
Support the specific application of AI used
Health Games System to support decision-making
activities.
▶ Game-Based Interventions in Public Health:
Computer Computer vision is the Artificial
Exploiting the Engaging Factor of Gameplay Vision Intelligence (AI) system
incorporated with other
scientific fields, such as signal
processing and Neurobiology, to
interpret and gain high-level
Healthcare understanding from digital images
or videos.
▶ Computer Graphics, Video Games, and Healthcare Healthcare robot refers to a
Gamification Impacting (Re)Habilitation, Robot machine programmed by a
Healthcare, and Inclusive Well-Being computer capable of assisting
humans in the medical field. In
addition, it is able to provide care
and support to disabled patients
and the elderly.
Healthcare Robots Islamic Islamic practice refers to five
Practice pillars of Islam, which include the
▶ Locomotion and Human Tracking in statement of faith called
Healthcare Robots “shahadah,” daily prayers called
Healthcare Robots with Islamic Practices 861

“salah,” almsgiving called “zakat,” robot to recognize the patient and understand
fasting during the month of the current situation around the patient. Under-
Ramadan called “sawm,” and standing the current situation will decide the
at least once-in-a-lifetime trip appropriate action that the healthcare robot
to Mecca called “hajj” (Stefon should interact with patients (Ic et al. 2013).
2009). Healthcare robots are programmable to provide
medication service to patients at home. For
Introduction example, one can schedule a robot to deliver
medication pills to patients at home. In Muslim
People in today’s world suffer from various health society, prayer is the second pillar of Islamic
problems, and as a result, the demand for medica- belief. Praying is the most fundamental worship-
tion and healthcare services has increased. For ing practice that Muslims are obliged to take a
example, people suffer from chronic diseases certain number of times a day (e.g., five times).
and require continuous medication. There is an In general, Muslims can pray at any time of the
issue for a patient to forget and skip their medica- day for distinct reasons (e.g., busyness, travel,
tion routine, for example, an elderly person. etc.). Therefore, a healthcare robot with the abil- H
Therefore, an automated system for dispensing ity to detect prayers by posture recognition is
medicine is essential to avoid the issue. essential for Islamic families. Thus, the robot
A healthcare robot can autonomously deliver should recognize a person’s posture and
medicine to patients at a scheduled time. As a approach the patient with Islamic practices. An
result, the healthcare robot should also take care Islamic-cultural aware robot will significantly
of the patient and ensure that the medicine is taken improve the robot medication service while also
for the most effective treatment. demonstrating respect for Islamic traditions.

Motivation and Background Related Works

The healthcare industry plays a more critical role The first commercially assistive robot is Handy1
in providing effective healthcare treatment to (Topping and Smith 1999). Handy1 is controlled
patients at home. The idea of healthcare robots by a single switch input for selecting the appro-
has become a significant invention in this field. priate action. The Neater Eater assists patients
Muslims believe that illness and pain are tests with their meals, including scooping foods from
from Allah and perceive illness as a trail by plate to patient’s mouth (Song and Kim 2012).
which one’s sins are removed (Kemp 1996). Exact Dynamics’ iARM technology is applied to
Therefore, healthcare providers need to under- assist disabled patients who cannot produce arm
stand the Islamic culture for clinical practices in movements (Ghobreal et al. 2012). The robot arm
Islamic families. For example, a nurse of the is attached to the electric wheelchairs, allowing
same gender should be cared for, especially the patient to control the arm via the controller.
when the patient is a female (Alotaibi 2021). The RAVEN II is a laparoscopic device created
Many issues need to be considered in the process with objective clinical measures in mind to opti-
of delivering culturally competent care to Mus- mize surgical performance (Hannaford et al.
lim patients. These issues affect the design of the 2013). MiroSurge is being developed by the Ger-
Decision Support System (DSS) with computer man Aerospace Center (DLR) to be very adapt-
vision to decide the optimal action the robot able in terms of the number of surgical domains,
needs to perform on Muslim patients. Currently, arm-mounting sites, a number of robots, control
the standard robots are integrated with cameras modes, and capacity to integrate with other tech-
that can capture both image and video. Computer nologies (Hagn et al. 2010). The ARMAR III is
vision technology is applied in the healthcare developed to assist patients in a household
862 Healthcare Robots with Islamic Practices

environment, such as interacting with people and schedule. (3) Prayer Posing Detection System
manipulating household items (Asfour et al. (PPDS) is responsible for detecting and recogniz-
2006). Care-O-Bot 3 is a robot for assisting people ing the posing of the patient. It analyzes the image
in their daily lives (Reiser et al. 2009). A flexible and determines the posing of the patient at any
torso enables butler-like motions such as bowing time of the day. Object detection techniques are
and nodding, an arm and gripper for manipulating used to detect four main postures in Islamic
items, a tray for carrying and transferring goods, prayer, Qiyam, Ruku, Julus, and Sujud, but it
and a tray for holding and moving objects. As should also be able to distinguish the four main
mentioned above, most robots are trained to postures from traditional postures. For example,
respond to specific duties to assist patients. As a body centroid detection is used to identify
result, there has been considerable advancement whether the human object is moving and avoid
in the ability of healthcare robots to perform med- false detection on similar posture such as walking/
ical services while also exhibiting respect for standing as Qiyam and sitting as Julus. A machine
patients’ cultural and religious values, such as in learning model should also be implemented to
Islamic nations. detect and recognize some restricted posing set,
such as lying down, standing up, sitting down, and
walking.
Structure of Learning System The process of healthcare robots is described as
follows. When it is time to take medicine, the
The system architecture of healthcare robots is robot will continue searching for the patient in
described as follows. The healthcare robot usually the house. If the user is identified and located,
sends the data to the Robot Decision Services the robot will send the request to RDS to deter-
(RDS) for deciding which action the robot should mine the optimal action in real-time. The request
take. The decision system on RDS consists of data that is sent to the server consists of the image
three subsystems: the Face Recognition System, from the camera sensor and the timestamp. Tech-
the Prayer Time Decision System, and the Prayer nically, the images are converted into string for-
Posing Detection System. These three subsystems mat before sending through the network
will cooperate to determine the optimal action for connection. Then, the robot will act upon the
the robot. The optimal action will be sent back to decision response from the server.
the robot along with the action code. The descrip- The process of RDS is described as follows.
tion of subsystems is presented as follows: RDS can be considered a server-side web Appli-
(1) Face Recognition System (FRS) is responsible cation Programming Interface (API). RDS listens
for recognizing the patient’s face from the given for the request from the robot. After receiving the
image. The clear face image of users must be first data from the robot, RDS will check the validity of
submitted to the system. Technically, all face the request message. If the request message is not
images are encoded in high dimensional space matched, the server will send the error message
so that the distance between two vectors repre- back to the robot for collecting new data. Then,
sents the similarity of two different face images. the image is decoded and sent to FRS. If the face
All face vectors are recorded in the Face Database. in the image is not the targeted patient, RDS will
(2) The Prayer Time Decision System (PDS) is send the message to inform the robot that the
responsible for decision-making on the prayer following action is to continue searching for the
time. It annually updates the Prayer Time Data- patient. RDS will then check for the prayer time
base from the public Islamic calendar service. by considering the image’s timestamp, current
PDS is required in this system because the robot time zone, and praying schedule. If the timestamp
should not interfere with the user during prayer is between the praying time interval, the system
time. The system also supports any time zone and will go to a room to search the patient. However,
adds the predetermined offset time to each praying the prayer time may differ from the praying
Hearthstone: A Collectable Card Game Through the Lens of Problem Solving 863

schedule for many reasons. The system also deter- Song, W.K., Kim, J.: Novel Assistive Robot for Self-
mines the current action of the user to avoid inter- Feeding, Robotic Systems – Applications, Control
and Programming, the, 3rd edn. Ashish Dutta,
ference in between the praying processes. The IntechOpen (2012)
current action of the user can be determined Stefon, M.: Islamic Beliefs and Practices. Britannica Edu-
from the image. If the praying pose is detected, cational Publishing (2009)
the system will wait until the patient finishes Topping, M.J., Smith, J.K.: The development of Handy1.
A robotic system to assist the severely disabled.
Islamic praying. Technol. Disabil. 10(2), 95–105 (1999)

Cross-References
Healthcare Training
▶ Locomotion and Human Tracking in
▶ Nursing Education Through Virtual Reality:
Healthcare Robots
Bridging the Gap

H
References

Alotaibi, H.H.S.: A review on the development of Hearthstone: A Collectable

healthcare infrastructure through the history of Islamic Card Game Through the Lens
civilization. J. Healthc. Leadersh. 13, 139–145 (2021) of Problem Solving
Asfour, T., Regenstein, K., Azad, P., Schroder, J.,
Bierbaum, A., Vahrenkamp, N., Dillmann, R.:
ARMAR-III: an integrated humanoid platform for John Scott2 and Newton Lee1,2
1
sensory-motor control. In: IEEE RAS International Institute for Education, Research, and
Conference on Humanoid Robots, Genova, Italy, Scholarships, Los Angeles, CA, USA
pp. 169–175 (2006) 2
Vincennes University, Vincennes, IN, USA
Ghobreal, B., Giokas, A., Dort, F., Gandhi, A., Foulds, R.:
Telemanipulation using exact dynamics iARM. In: The
2012 38th Annual Northeast Bioengineering Confer-
ence (NEBEC), pp. 123–124 (2012) Synonyms
Hagn, U., Konietschke, R., Tobergte, A., Nickl, M., Jörg,
S., Kübler, B., Passig, G., Gröger, M., Fröhlich, F.,
Seibold, U., Le-Tien, L., Albu-Schäffer, A., Nothhelfer, Collectable card game
A., Hacker, F., Grebenstein, M., Hirzinger, G.: DLR
MiroSurge: a versatile system for research in endo-
scopic telesurgery. Int. J. Comput. Assist. Radiol. Definitions
Surg. 5(2), 183–193 (2010)
Hannaford, B., Rosen, J., Friedman, D.W., King, H., Roan,
P., Cheng, L., Glozman, D., Ma, J., Kosari, S.N., White, Collectable Card Game ¼ a type of strategy game
L.: Raven-II: an open platform for surgical robotics where players collect cards to form a custom deck
research. IEEE Trans. Biomed. Eng. 60(4), to overcome challenges presented by other types
954–959 (2013)
Ic, Y.T., Yurdakul, M., Dengiz, B.: Development of a of decks.
decision support system for robot selection. Robot.
Comput. Integr. Manuf. 29(4), 142–157 (2013)
Kemp, C.: Islamic cultures: health care beliefs and prac- Introduction
tices. Am. J. Health Behav. 20(3), 83–89 (1996)
Reiser, U., Connette, C., Fischer, J., Kubacki, J., Bubeck,
A., Weissh, F., Jacobs, T., Parlitz, C., Hägele, M., Verl, Hearthstone is a collectable card game that does a
A.: Care-O-bot ® 3 – creating a product vision for good job of showing the lens of problem solving,
service robot applications by integrating design and which is all about the problems that a game poses
technology. In: IEEE/RSJ International Conference on
Intelligent Robots and Systems, St. Louis, USA, to a player, and how the game continues to make
pp. 1992–1998 (2009) the player solve new problems. In Jesse Schell’s
864 Hearthstone: A Collectable Card Game Through the Lens of Problem Solving

book “The Art of Game Design: A Book of free. The solution is simple in concept but difficult
Lenses,” the Lens of Problem Solving states that in execution. Dealing 30 damage in a single turn.
“To use this lens, think about the problems your Some classes have it easier than others. Priests, for
players must solve to succeed at your game, for example, can just put a high health minion onto
every game has problems to solve.” (Schell, 2019) the field, increase its health to be at or above
The Lens of Problem Solving requires us to ask 30, then set its attack to be equal to its health,
the following questions: allowing it to just one shot the boss. While other
classes, like mages, have to set up intricate com-
1. What problems does my game ask the player to binations of cards, one example being having
solve? cards that reduce the mana cost of a spell and
2. Are there hidden problems to solve that arise as another card that allows you to cast that card
part of the gameplay? until you run out of mana. The real challenge
3. How can my game generate new problems so there being that you have to get quite a few cards
that players keep coming back? out at the same time or risk having important cards
being destroyed before you can use them.
Challenges start the player with a smaller deck
Gameplay to work with, but also place them against much
weaker opponents. After each opponent, you
In Hearthstone there are two main ways to play, choose between three sets of three new cards
multiplayer (which includes ranked and that you add into your deck. These cards ignore
unranked) and solo adventures (which include the normal limitations of the game, allowing you
raids and challenges), both of which have a max to have more than two of the same card. On top of
card limit of 30. In solo adventures, the developer this, after the first and fourth enemy defeated (out
directly interjects the problem solving, rather than of eight), you are given the option to choose a
allowing it to fall unto the players to create and buff. These buffs can be anything from increasing
solve problems for each other. The main content your health significantly, or giving you extra mana
in solo adventures is raids, which pits the player at the beginning of the game. With the way chal-
up against a variety of bosses with varying diffi- lenges are set up, the problem solving is less
culty, and challenges, where the player is put reliant on the cards that you personally own, and
against a group of weaker enemies while also more reliant on your own skill and knowledge of
having less choice on how they’re going to the game as a whole. However, it does introduce a
approach the enemies. With raids, players make luck element, where you might not get the most
their own decks around each individual boss, optimal options, and struggle because of it.
while with challenges, the player is given a deck,
and as they progress through enemies they are
given more cards and gameplay options to mix Multiplayer Mode
with their existing deck to make as strong a deck
as possible to defeat the later enemies. In the multiplayer mode, the game is constantly
Raids require the most direct problem solving. updated with new cards and interesting meta-
The game puts a boss with a unique ability and breaking combinations to find, so you get your
unique move set in front of you and you have to average competitive game where both players are
build a deck around that. This can lead to some set against each other with plenty of opportunities
insane decks being made. A good example is one to outplay the other. The ranked mode has
of the earlier bosses in the Icecrown raid; this boss 25 ranks that you have to go up before you can
has 30 health (a normal amount), and a free, reach the “legend” rank, and start playing to be
spammable ability that allows him to go back to placed as one of the best players in the world.
30 health every turn. So, the problem is how to In closing, Hearthstone has the entire problem
defeat an enemy that can heal so much damage for solving of a multiplayer game mixed with varied
High-Performance Many-Light Rendering 865

and difficult single-player content that can be very topics in photo-realistic image synthesis. In
challenging to figure out. It has a very large order to calculate a high dimensional integration
amount of cards to make new strategies and in the rendering equation proposed by Kajiya
there will always be something that requires a (1986), a huge amount of computational power
new deck to beat. is needed, which prevents the pure path-tracing
group of methods from implementing in an inter-
active to real-time systems. In the other hand, such
References tracing algorithms are often not regular, which
will cause workload balance problems or irregular
Schell, J.: The Art of Game Design: A Book of Lenses, memory access problems on parallel hardware
3rd edn. A K Peters/CRC Press Massachusetts, USA
architectures especially current GPUs, thus harm
(2019)
the scalability of the algorithm.
In this entry, we will introduce many-light
rendering, which is one of the simplest and
most elegant solutions in the global illumination
Hidden Markov Models field that can increase scalability while still being H
able to generate photo-realistic images. The main
▶ Machine Learning for Computer Games idea of many-light rendering is to split the ren-
dering into two steps: The first step is to generate
a massive amount of virtual-point-lights (VPLs)
in the scene according to the original rendering
High-Performance equation or path space integration framework
Many-Light Rendering proposed by Veach (1997). The second step is
that for each pixel, the irradiance will be col-
Tong Wang lected and accumulated from all VPLs to approx-
Cygames, Inc., Tokyo, Japan imate the final results. Among all the physically
based rendering techniques, many-light render-
ing methods are considered best suitable for fast
Synonyms preview and iteration situations, which are also
often suitable for GPU-based parallelization
Many-light rendering; Virtual point light scheme.
Many-light rendering algorithms can also be
applied in a scene with a large number of actual
Definitions light sources, which is becoming more and
more common recently. Scenes illuminated
Many-light rendering is a class of efficient image with numerous actual light sources can also
synthesis algorithms that try to render a scene with migrate algorithms from the VPL-based global
a large number of light sources. These light illumination algorithms. Some recent advances
sources can be real lights in the scene, or virtual can be found in literatures such as Dachsbacher
point lights (VPLs) approximate light transporta- et al. (2014), Bitterli et al. (2020), and
tion of current lighting conditions. Yuksel (2020).

Introduction Many-Light Rendering: Paths in VPL

Framework
Full global illumination is considered the crown
of rendering. Creating high fidelity global illumi- Keller (1997) is considered the beginning of
nation efficiently is one of the most important many-lights rendering. VPLs are generated by
866 High-Performance Many-Light Rendering

the scene. Trace the ray with a Russian roulette

termination condition.
• For each rendered point, collect illumination
from all VPLs. Iterate each VPL and conduct
a visibility test to determine if the VPL is
visible from the rendered point, then compute
and accumulate the contribution from
each VPL.

Figure 2 is a simple illustration of such two

steps.
Another situation that needs to calculate a huge
High-Performance Many-Light Rendering, amount of light is when there are actually many
Fig. 1 Illustration of the generation of VPL. A VPL x2 is light sources that exist in the scene. These two
generated from the light source point x1 during generating problems share a similar computing model, thus
step. It will then illuminate other surface points such as x3 are often classified in the same category. To dis-
and x3. It will be collected to the pixel plane point x4 on the
camera tinguish these two different end applications, we
refer to the VPL-based global illumination sce-
nario as many-light GI, and the second scenario as
light path random walk in the scene. Figure 1 is a
many-light DI (direct illumination from actual
simple illustration of such a VPL path. For the i-th
light sources).
VPL xi generated by vertex xi–1, among total VPL
The first step of many-light GI is to distribute
number of n, and an eye vertex xe, the radiance
VPLs in the scene. There is a lot of research
reflected at eye path intersection point (also
studying how to efficiently distribute VPLs in
referred to shading point) x is:
the scene. Georgiev and Slusallek (2010) intro-
n duced a simple reject-sampling method that will
L¼ f ðxi1 ! xi ! xÞ filter out unimportant VPLs. Hasan et al. (2009),
i¼1 ð1Þ on the other hand, are trying to generate virtual
axi1 Gðxi , xÞf ðxi ! x ! xe ÞÞ light sources that can avoid the singularity prob-
lem, which will cause bright spot artifact in the
where L means radiance coming to eye vertex final rendered results.
Lðx ! xe Þ, Gðxi , xÞ is the geometry term For the second step, our problem is more clear:
containing the visibility term V ðxi , xÞ. axi1 is the how to efficiently accumulate energy from a mas-
virtual light contribution of vertex xi  1, which is sive amount of virtual point lights distributed at
defined as a Monte Carlo style sampled path con- various regions in a complex scene geometry. The
tribution of a random walk path sequence gener- basic method is to calculate the color of
ated from the light source. N samples, from every light source (VPLs or
The two steps of a VPL-based many-light ren- actual light sources) of a total number M, which
dering are indicates at least M ⁎ N raycast and lighting cal-
culation. The O(N ⁎ M) algorithm is not accept-
• Generate a large amount of VPLs in the scene. able when there are massive samples on the pixel
Choose the light sources and starting position plane and a large number of light sources in the
randomly, then shoot a VPL ray from the point scene, especially for interactive and real-time
into the scene. If the intersection surface is applications. So we need to make approximation
diffuse, record the VPL with corresponding or calculate the accumulated energy in a clever
probability, BRDF (Bidirectional Reflectance way to reduce the cost and increase the scalability
Distribution Function), and geometry term on of the algorithm.
High-Performance Many-Light Rendering 867

High-Performance Many-Light Rendering, Fig. 2 The two steps of a general VPL-based illumination

High-Performance Many-Light less than a threshold solid angle will be calculated

Rendering as a whole to reduce computational cost. The
nearby lights that exceed the solid angle threshold
From the previous section, we know that to gather may need to be calculated separately to increase
the full lighting information for a specific point in the accuracy of the illumination.
many-light rendering, a huge number of raycasts Other works such as Ou and Pellacini (2011),
will be performed to calculate the visibility Walter et al. (2012), and Bus et al. (2015) dive into
between the shading point and light sources. this idea further by exploring theories and tech-
niques to cluster shading points as well. The scal-
Cluster-Based Method ability of many-light rendering methods is
An observation is that samples and light sources significantly improved with these hierarchical
(or VPLs) that are near to each other often share cluster-based methods.
similar visibility and lighting information. Another important group of techniques that
A natural idea that can help to solve the perfor- tries to cluster lights and shading points by
mance problem is to cluster nearby similar VPLs another formulation is the Matrix Row-Column
together to form stronger lights to decrease the Sampling (MRCS) method proposed by Hassan
necessary number of visibility raycasts (Fig. 3). et al. (2007). It formulates the many-light render-
Walter et al. (2005) is an early work on the ing problem into a matrix row-column sampling
approximation of VPL clustering. All lights or problem from the VPL evaluation matrix M. The
VPLs will be constructed into a tree structure row of M is all the shading points, and the col-
called a light tree. A light cut means all the umn of the matrix represents all the VPLs. Thus,
nodes that have been chosen for a specific shading the entry M(i, j) is the contribution from VPL j to
point. Figure 4 shows how a shading point is the shading point i. Hašan et al. (2007) observed
illuminated by a hierarchically constructed light that matrix M is often low rank, which means that
tree. From a certain shading point’s perspective, it can be compressed to a smaller matrix. Shading
the visibility and contribution of faraway nodes points and VPLs will be clustered through the
868 High-Performance Many-Light Rendering

High-Performance Many-Light Rendering, certain error-bound constraint. (b). Octree is a famous

Fig. 3 (a). VPLs can be clustered together with a certain spatial subdivision structure that can be used to construct
error-bound constraint, thus the similar project angles from such light tree
different distances. VPLs can be clustered together with a

High-Performance
Many-Light Rendering,
Fig. 4 Light cut of a light
tree. The shading point will
be shaded by the nodes and
leaves on a specific light cut
calculated with error bound

compressing algorithm. To do such a compress, heterogeneous media. Nabata et al. (2016) pro-
the author proposed to do a row-column sam- posed an error estimation framework for such a
pling algorithm that can effectively create the clustering scheme.
low-rank matrix which can be evaluated
efficiently. Sampling-Based Method
Huo et al. (2015) is a matrix-sampling-and- Clustering lights together is a biased approxima-
recovery scheme developed from MRCS that tion with a bounded error, which may not meet the
aims to efficiently accumulate the illumination demands of high-quality-image-synthesis tasks.
contribution by sampling and reconstructing the To generate unbiased synthesis with a huge
low-rank illumination matrix as well as predicting amount of light sources, an unbiased Monte-
visibility from spatially correlated shading points Carlo integrator is needed. There is a group of
and lights. Huo et al. (2016) explore the locally methods that try to find important lights through
coherent nature of scattered lighting in sampling in an unbiased way, and here we will
High-Performance Many-Light Rendering 869

introduce some of the important works developed hierarchically constructed light tree or shading
recently. point tree but will bring extra bias to the final
Moreau et al. (2019) proposed the many-light results, while the unbiased Monte-Carlo estimator
sampling method that maintains the hierarchical sampling the light source will be able to generate
light sampling data structures. Light sources are unbiased results but will bring noise due to the
organized in a bounding volume hierarchy variance which may not be suitable to use directly
(BVH). They will be stochastically traversed for interactive applications. In practice, these two
during the shading phase. Lights maintained in a groups of methods can be combined together to
two-level light acceleration structure will be sto- leverage accuracy and performance.
chastically selected by evaluating an important
function. They achieve two orders of magnitude
faster than the original off-line implementation. Cross-References
Yuksel (2020) presents a stochastic light cuts
technique that can replace the sampling correla- ▶ 3D-Rendered Images and Their Application in
tion of light cuts and replace it with noise to the Interior Design
resolve bias issues in light sampling. The basic ▶ Raycasting in Virtual Reality H
idea is that for each node in the light tree, choose ▶ Rendering Equation
the representative light in a stochastic way to
avoid extra bias. They will also be chosen on the
fly to avoid extra storage of light tree. Yuksel also References
developed a hierarchical importance-sampling
Bitterli, B., Wyman, C., Pharr, M., Shirley, P., Lefohn, A.,
scheme that can compute the light probabilities Jarosz, W.: Spatiotemporal reservoir resampling for
using the light tree during the lighting evaluation real-time ray tracing with dynamic direct lighting.
at a given point on the fly. ACM Trans. Graphics (TOG). 39(4), 148–141 (2020)
A recent important work is Bitterli et al. Bus, N., Mustafa, N.H., Biri, V.: Illuminationcut. In: Com-
puter Graphics Forum. Wiley Online Library, 34,
(2020). This work proposed a spatiotemporal 561–573 (2015)
reservoir-resampling method that can be applied Dachsbacher, C., Křivánek, J., Hašan, M., Arbree, A.,
in many-light condition. The main idea is based Walter, B., Novák, J.: Scalable realistic rendering with
on a resampled importance sampling (RIS) tech- many-light methods. In: Computer Graphics Forum,
Wiley Online Library, vol. 33, pp. 88–104 (2014)
nique that can effectively reuse statistics from Georgiev, I., Slusallek, P.: Simple and robust iterative
temporal and spatial neighbors for each pixel’s importance sampling of virtual point lights. In:
direct light sampling PDF. Combining with a Eurographics (Short Papers), The Eurographics Asso-
biased estimator, this work will be able to synthe- ciation, pp. 57–60 (2010)
Hašan, M., Pellacini, F., Bala, K.: Matrix row-column
sis photorealistic images for scenes containing sampling for the many-light problem. In: ACM
dynamic emissive triangles on GPU in real time. SIGGRAPH 2007 papers, Association for Computing
Machinery, pp. 26–es (2007)
Hašan, M., Křivánek, J., Walter, B., Bala, K.: Virtual
spherical lights for many-light rendering of glossy
Conclusion scenes. In: ACM SIGGRAPH Asia 2009 papers, Asso-
ciation for Computing Machinery, pp. 1–6 (2009)
In this entry, we introduced the basic theory and Huo, Y., Wang, R., Jin, S., Liu, X., Bao, H.: A matrix
techniques about an efficient set of algorithms for sampling-and-recovery approach for many-lights ren-
dering. ACM Trans Graphics (TOG). 34(6), 1–12
photorealistic global illumination: many-light (2015)
rendering problems. We summarize cluster- Huo, Y., Wang, R., Hu, T., Hua, W., Bao, H.: Adaptive
based and sample-based methods that can matrix column sampling and completion for rendering
improve the efficiency of this algorithm signifi- participating media. ACM Trans Graphics (TOG).
35(6), 1–11 (2016)
cantly. Both came with a price: The cluster-based Kajiya, J.T.: The rendering equation. In: Proceedings of the
methods generate an approximation of the global 13th annual conference on computer graphics and inter-
illumination with a bounded error by traversing active techniques, pp. 143–150 (1986)
870 History of Augmented Reality

Keller, A.: Instant radiosity. In: Proceedings of the 24th Introduction

annual conference on Computer graphics and interac-
tive techniques, pp. 49–56. ACM Press/Addison-
Wesley Publishing Co (1997) Augmented reality is a form of technology that
Kemppinen, P., et al.: Importance sampling in real-time falls under the category of computer-mediated
many-lights rendering (2019) reality. Like its fellow technologies, such as vir-
Moreau, P., Pharr, M., Clarberg, P., Steinberger, M., Foley, tual reality, this technology allows users to inter-
T.: Dynamic many-light sampling for real-time ray
tracing. In: High Performance Graphics (Short Papers), act in a computer environment. This technology,
Association for Computing Machinery, pp. 21–26 however, allows users to interact within their
(2019) existing world, instead of a fully immersive one.
Nabata, K., Iwasaki, K., Dobashi, Y., Nishita, T.: An error Only a few decades old, the technology still has
estimation framework for many-light rendering. In:
Computer Graphics Forum, Wiley Online Library, had an interesting path to its more mainstreamed
vol. 35, pp. 431–439 (2016) state that we enjoy today.
Ou, J., Pellacini, F.: Lightslice: matrix slice sampling for
the many-lights problem. ACM Trans Graph. 30(6),
179 (2011)
Veach, E.: Robust Monte Carlo methods for light transport History of Augmented Reality
simulation, vol. 1610. PhD thesis, Stanford University
(1997) The first reference to Augmented Reality technol-
Walter, B., Fernandez, S., Arbree, A., Bala, K., ogy is said to come from the author of The Won-
Donikian, M., Greenberg, D.P.: Light-cuts: a scalable
approach to illumination. In: ACM SIGGRAPH 2005 derful Wizard of Oz, Lyman (L.) Frank Baum in
Papers, Association for Computing Machinery, 1901. From another work of Baum’s called The
pp. 1098–1107 (2005) Master Key, a demon provides a character with the
Walter, B., Khungurn, P., Bala, K.: Bidirectional lightcuts. “Character Marker” – which is a pair of spectacles
ACM Trans Graph (TOG). 31(4), 1–11 (2012)
Yuksel, C.: Stochastic lightcuts for sampling many lights. that would place a “G” or “E” on the foreheads of
IEEE Trans Vis Comput Graph. (2020) each person that they would meet, which corre-
lated to whether they were good or evil. There was
also “W” for wise, “F” for foolish, “K” for kind,
and “C” for cruel. When asked who discovered
the device, the demon informs the reader that: “It
History of Augmented Reality is a fact that has always existed, but is now utilized
for the first time.” (Javornik 2016)
Emily Peed1 and Newton Lee1,2 Much like the virtual reality, some of the tech-
1
Institute for Education, Research, and nologies that brought augmented reality to life
Scholarships, Los Angeles, CA, USA overlap and have existed before the creation of
2
Vincennes University, Vincennes, IN, USA the specific technology itself. In fact, these tech-
nologies such as virtual reality, augmented reality,
and others are contained on a spectrum and can be
Synonyms referred to as mediated reality. Two less recog-
nized forms of this mediated-reality space are
Augmented reality; Computer-mediated reality; modulated reality and diminished reality (Grover
Mixed reality 2014). Overall, what this field of mediated reality
allows for is the ability of the computer to add or
subtract from our perceived reality – adding into
Definition the experience until it is completely immersive,
such as in virtual reality.
A form of computer-mediated reality that super- Similar technologies that fueled the develop-
imposes images into the view of a user. It uses the ment of virtual reality also lay behind the inspira-
existing environment, complementary imagery, tion for augmented reality. In 1929, Edward Link
and interactivity to bring an enhanced experience. created the “Link trainer,” which was an
History of Augmented Reality 871

electromechanical flight simulator that encapsu- showing further proof that virtual overlays helped
lated the individual into a replica cockpit. During the user interact in the real world with enhanced
WWII, 500,000 pilots improved their skills by capability. His moral underpinnings as an animal
logging hours of initial training on 10,000 “Blue rights supporter and vegan mostly likely aided in
Box” Link Trainer machines. Later, the his passion as he later develop the first VR surgi-
Sensorama was created in the mid-1950s by Mor- cal simulators that helped to reduced the use of
ton Heilig, a cinematographer that patented the animals for medical training (Rosenberg 2016).
device later in 1962. It had stereo speakers, a In 1994, Steve Mann, recognized more as the
stereoscopic 3D display, fans, smell generators, father of wearable computing, also cemented him-
and a vibrating chair. Using his background in self into the fathering of augmented reality with
cinematography, six short films were created by his invention of the Digital Eye Glass and medi-
Morton Heilig himself to accompany the device. ated reality. (Mann 2018) At one time during work
In the 1960s and 1970s, more advancements at MIT, he would wear 80 pounds worth of
were made on the technological front of these equipment to class as he worked on wearable
industries. These include a continuation of technology and interacting technology with the
Morton Heilig’s work within a device called the environment (Mann 2013). A believer that tech- H
Telesphere Mask in 1960. More attention was nology should be organic to humans, rather than
beginning to be paid as a pair of Engineers from humanity learning the ways of computing,
Philco Corporation, Charles Comeau and James inspired him to integrate these two worlds.
Bryan, created the Headsight in 1961. It was In 1996, Jun Rekimoto invents the 2D matrix
developed for immersive, remote viewing of markers for AR objects, also known as Cyber-
dangerous situations by the military. The 1960s Code. It is a technique that allows for augmented
also saw the writing of the Ultimate Display reality to identify real world objects and estimate
concept by Ivan Sutherland. His design housed their place in a coordinate system. The technology
a computer-generated virtual world that was uses barcodes to identify large numbers of objects.
maintained in real time. The 1965 paper laid the This additionally makes it so that objects can have
foundation for concepts that represent the core useful information associated with them when
blueprint for these types of mediated reality tech- interacted with. The technology kept on maturing
nologies today. However, it was not until recently as the decade ensued. One year before the millen-
that Augmented Reality began to take off. nium in 1999, Hirokazu Kato created ARToolKit
The 1990s is when Augmented Reality truly at HITLab, which started an open source com-
came into fruition. The term was first coined that puter tracking library that created augmented real-
year by Boeing Engineers’ Tom Caudell and ity applications that could be placed into the real
David Mizzel (Rauterberg 1990). This is one world.
year after virtual reality received its official title The first augmented reality game was created
from Jaron Lanier. From the coining of its term in by Bruce H. Thomas and was called ARQuake. It
1990 and every few years, major developments was demonstrated at the international Symposium
were made. It was in 1992 with Louis Rosenberg on Wearable Computers in 2000. Smaller devel-
that steps were taken towards a functional aug- opments were made in the technology that contin-
mented reality system. It was called Virtual Fix- ued to bring it to more market appeal, but it was
tures and was developed through the US Air Force not until 2013 with the announcement of Google
Research Laboratory. His research showed that an Glass that the technology began to gain more
AR system could enhance performance. mainstream hype. Even the Google Glass creation
Rosenburg is dyslexic and attributes some of his was met with some criticism, Steve Mann said
success in technology to this trait. In fact, his issue during the Augmented World Expo in Santa
with handwriting is what spurred him towards the Clara that the technology was more a generation
computer in the first place (Rosenberg 2016). He one device, similar to other prototypes of the past
continued his research at Standford University, (Hollister 2013).
872 History of Augmented Reality

What was the moment that sent AR over the come a plethora of new applications and interac-
edge? Well, one could say that it was the break- tions within the technology.
through phenomenon of the Pokemon Go game These mediated reality technologies are set to
that allowed the technology to spread its wings in make $162 billion by 2020 (Intelligence, BI
the full twenty-first century – at least in a house- 2016). This growth is poised over the distribution
hold use sense. We may begin to see more inclu- of the technology itself, as it does require hard-
sion of interesting advancements in augmented ware purchases. The wearable market, which
reality, as it has been dubbed the eighth medium – slightly overlaps in this technology, is estimated
coming on the heels of mobile technology, inter- to sell $34 billion dollars worth of devices by
net, TV, radio, cinema, recordings, and print. This 2020, compared to the $14 billion in 2016
technology will be used in multiple ways, as it is (Lamkin 2016). The industry is on the rise and
expected that one billion people will begin as we continue to integrate incredibly more pow-
interacting the technology by 2020. The Micro- erful watches, glasses, and cell phones into our
soft Hololens was finally delivered in 2016 after daily lives, then there is a way this technology
its initial introduction in 2015; the developer’s kit could deepen the experience already enjoyed. Per-
is now available and the technology is seeking to haps one day, built into every pair of glasses could
upend some of the ways that we have traditionally be a “Character Marker,” as first forecast in 1901
approached augmented reality. and already in use in some facial recognition
The technology places images and objects into smart glasses today.
the environment to interact with and is dubbed by
Microsoft as a mixed reality system. There are
exciting developments surrounding the technol-
ogy, for example, an individual could see through References
your device to guide you through a repair job
Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S.,
(Statt 2015). One could also Skype with another MacIntyre, B.: Recent advances in augmented reality.
person and walk around freely, and not worry IEEE Comput. Graph. Appl. 21(6), 34–47 (2001)
about having their full field of view obstructed. Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P.,
Damiani, E., Ivkovic, M.: Augmented reality technol-
The device also presents educational, gaming, and
ogies, systems and applications. Multimedia Tool.
a fundamental change in how computing is Appl. 51(1), 341–377 (2010)
expected to interact – as the system is completely Grover, D.: Augmented Reality History, Background and
self contained and independent from an underly- Philosophy. Macquarie University, 24 Feb 2014. wiki.
mq.edu.au/display/ar/Augmented reality history%2C
ing computer. This is unlike virtual reality systems
background and philosophy
like the Oculus Rift that tether you to a desktop Hollister, S.: In the shadow of Google Glass, an augmented
computer and corrals the users within the range of reality industry revs its engines. The Verge, 9 June
sensors. 2013. www.theverge.com/2013/6/9/4410406/in-the-
shadow-of-google-glass-at-augmented-world-expo-
What augmented reality will provide is an
2013
exciting link between traditional computing inter- Intelligence, BI.: The virtual and augmented reality market
actions and the real world. There is the ability to will reach $162 billion by 2020. Business Insider,
try on clothing or glasses before you purchase 22 Aug 2016. www.businessinsider.com/virtual-and-
augmented-reality-markets-will-reach-162-billion-by-
them from an online venue or store, or to try on
2020-2016-8
several outfits at the drop of a hat. There are Javornik, A.: The mainstreaming of augmented reality:
educational advantages and integration into our A brief history. Harvard Business Review, 4 Oct
self-driving car future. As more recent phones 2016. hbr.org/2016/10/the-mainstreaming-of-
augmented-reality-a-brief-history
from Google, Apple, and the like begin to inte-
Lamkin, P.: Wearable tech market to be worth $34 billion
grate more augmented reality into their latest By 2020. Forbes Magazine, 17 Feb 2016. www.forbes.
releases and software updates, then there will com/sites/paullamkin/2016/02/17/wearable-tech-
History of Virtual Reality 873

market-to-be-worth-34-billion-by-2020/#a123e41 available virtual reality equipment such as virtual

3cb55 reality glasses and data gloves.
Mann, S.: Electrical & Computer Engineering. University
of Toronto, Faculty of Applied Science & Engineering, Some of the artifacts mentioned in this entry
2018. www.ece.utoronto.ca/people/mann-s/ for an historical account of VR (virtual reality) are
Papagiannakis, G., Singh, G., Magnenat-Thalmann, N.: dated far earlier than 1987, since they were chosen
A survey of mobile and wireless technologies for aug- according to criterion that the artifact creates an
mented reality systems. Comput. Anim. Virtual
Worlds. 19(1), 3–22 (2008) immersion effect primarily based on sense of
Rauterberg, M.: AR at Boeing (1990). Technical Univer- vision. Using a technology-oriented approach in
sity Eindhoven. www.idemployee.id.tue.nl/g.w.m. relation with the arguments in contemporary
rauterberg/presentations/hci-history/tsld096.htm research, Garner (2018) suggests that stereoscopy,
Rosenberg, L.B.: InterPlanetary File System, 22 Nov 2016.
ipfs.io/ipfs/QmXoypizjW3WknFiJnKLwHCnL72 field of view, and synchronized multimodality are
vedxjQkDDP1mXWo6uco/wiki/Louis_B._Rosenberg. vital contributors to presence and immersion
html which are two of the most significant elements
Sandor, C., Fuchs, M., Cassinelli, A., et al.: Breaking the of VR experience. To enhance immersion and
Barriers to True Augmented Reality. arXiv.org cs.HC
presence, an artifact may also employ auditory,
(2015)
Statt, N.: Microsoft’s HoloLens explained: How it works olfactory, or somatosensory stimuli besides 2D H
and why it’s different. CNET, 24 Jan 2015. www.cnet. (two-dimensional) wide angle field of view or
com/news/microsoft-hololens-explained-how-it- 3D (three-dimensional) vision.
works-and-why-its-different/
Steve Mann.: Cyborg Anthropology, 27 Jan 2013. While the definitions given here were limited
cyborganthropology.com/Steve_Mann to the technical progress, devices, and products in
Wang, X., Kim, M.J., Love, P.E.D., Kang, S.-C.: Aug- relation to VR; some authors (Gigante 1993;
mented reality in built environment: Classification and Jerald 2015; Steinicke 2016) mentioned the fic-
implications for future research. Autom. Constr. 32,
1–13 (2013) tional representations of the virtual worlds in lit-
Zhou, F., Duh, H., Billinghurst, M.: Trends in augmented erature, cinema, and television as a part of VR
reality tracking, interaction and display: A review of ten history.
years of ISMAR. In: Proceedings of the 2008 7th IEEE/
ACM ISMAR (2008)

History

Mechanical Precursors
History of VR (virtual reality) can be taken as far
History of Virtual Reality back to 1793, when the Irish painter Robert
Barker exhibited his panoramic paintings at the
Mehmet Ilker Berkman rotunda in Leicester Square, which is the first
Communication Design, Bahcesehir University building purposefully constructed to view the
Faculty of Communication, Istanbul, Turkey paintings located on the inner facet of circular
walls, with internal staircases and platforms
(Benosman and Kang 2001). The building itself
Definitions is an immersive technology, patented in 1796 by
Barker. It is followed by a sequel of similar dis-
The term “virtual reality” was coined by Jaron play methods such as Franz Niklaus König’s
Lanier in 1987 during a period of intense research Diaphanorama, Daguerre’s Diorama, Charles
in immersive technologies (Virtual Reality A. Close’s Electronic Cyclorama, Thomas Bar-
Society 2017). Lanier owned a research company ber’s Electrorama, Lumière Brothers Photorama,
pioneered in 3D graphics and immersive interac- and Grimoin-Sanson’s Cinéorama (Uricchio
tions that produced the first commercially 2011). The Cinéorama, appeared in 1900 Paris
874 History of Virtual Reality

Exposition, consisted of ten synchronized 70 mm Zone 2014: 26). However, neither of these
motion picture projectors screening a 360 mov- attempts for a motion-picture stereoscope resulted
ing images on the walls of a circular space, where with a commercially successful product, due to
an audience of 200 viewers can be located at the the “contrast between the ease of inducing appar-
center in a gondola-shaped platform to experience ent motion and the difficulty of seeing depth in
a balloon ride over Paris. Some other immersive sequences of briefly presented stereoscopic
setups in Paris Exposition were Trans-Siberian images” (Wade 2012). Stereoscopes stayed in
Railway Panorama and the Mareorama, both use until early 2000s as an entertainment and
used moving panoramas painted on canvas to educational technology and finally a children’s
simulate motion within traveling experience. The toy, to view stereoscopic images printed on
nineteenth century panorama evolved into a mass cards, film, slides, and reels. The design of Brew-
media while people around Europe visited various ster evolved into phone-based VR products after
panoramic displays immersing themselves in 2014, which employ mobile phone screen to show
views of nature, landscape, and historic events stereoscopic images instead of printed images
(IJsselsteijn 2005). These circular display tech- (Jerald 2015: 16).
niques of panoramic vision evolved into wide- Stereoscopic projection artifacts should also be
screen cinema in twentieth century. mentioned within the early efforts that can be
Stereoscope was another early invention of related to virtual reality. Louis Ducos du Hauron,
nineteenth century that pioneers virtual reality the inventor of color photography, enhanced the
technology. In 1838, Charles Wheatstone, Profes- “anaglyph” methods that can also be used for
sor of Experimental Philosophy in King’s College forming three-dimensional visuals based on
(London) devised two mirrors located at 45 to the encoding each eye’s image using filters of differ-
viewer’s eyes, reflecting two slightly similar ent colors. The term “anaglyph” was first used by
images. Images were drawings, since it was a de Hauron in 1890s, but the principles were
year before Daguerre (who was also the director known since as early as seventeenth century
of Dioroma mentioned above) established a prac- (Zone 2014: 55). The first stereoscopic projection
tical photographic process called daguerreotype. based on anaglyph method was described by
Later in 1841, Wheatstone made experiments with Wilhelm Rollman in 1853, who could have been
photographs taken with a single lens camera from the first to project images with complementary
different angles (Wade 2012). In 1849, David colors but the first projector that is known to be
Brewster has developed a handheld stereoscope actually built was made in France by Charles
using prismatic lenses besides a binocular camera d’Almeida, in 1858. In nineteenth century, the
to take stereoscopic photographs. Brewster (1856) methods based on the polarization of light were
credits the first functional stereoscope to also employed to display three-dimensional
Mr. Elliot’s design in 1834, which is a stereoscope images. The principles of polarization had been
that does not employ any mirrors or lenses: the known since seventeenth century, but a British
ocular stereoscope. Referring to Wheatstone’s physicist named John Anderton is credited to be
device as reflecting stereoscope, Brewster named the first to use it for projection of three-
his own invention as lenticular stereoscope. The dimensional images with his mechanism patented
Brewster stereoscope had drawn a huge public in 1895.
attention. A London based company sold more
than half a million stereoscopic views from 1856 Immersive Movies
to 1858 and a million views in 1862 (Bendazzi During the first two decades of twentieth century,
2016: 15). The public attention in stereoscopes led motion pictures became a popular entertainment
to attempts to combine stereoscopic views and medium. However, until 1922, there was not a
motion, such as Claudet’s and Duboscq’s efforts successful product of 3D motion pictures that
in 1852, which was the earliest patent application has developed beyond prototype stage. In
for a motion-picture stereoscope (Wade 2012; September 1922, the first 3D feature film, Power
History of Virtual Reality 875

of Love, was screened to an invited audience at angle for the spectator. The curved screen
Ambassador Hotel Theater in Los Angeles. The consisted of vertical strips angled toward the audi-
film is taken by Harry K. Fairall’s camera which ence to prevent light reflected from one edge of
was later patented as “Binocular Nonstop Motion the screen to wash away the image on the other
Picture Camera” in 1930. Stereoscopic viewing edge. Originally, Waller developed an 11 projector
was achieved using anaglyph method, in which system called Vitarama for 1939 World’s Fair in
viewers use complementary colored glasses. New York and worked on five projector systems
Another anaglyph based 3D motion picture were for military training purposes during WWII.
Plasticons by William Van Doren Kelley’s stereo- Although the wide screen technologies revoked
scopic camera pair, followed by Jacob Leventhal the interest into movie theaters, systems like
and John Norling’s Plastigrams, which were ani- Cinerama, such as its successors Cinemiracle,
mated cartoons that obtained a large audience in Thrillerama, and Wonderama which were based
multiple theaters. Success of Plastigrams created a on multiple cameras and projectors, were not
demand for more novelty 3D film, which was widely accepted due to production and projection
filled with Stereoscopics created by Jacob costs. The Cinemascope technology became pop-
Leventhal in 1925, followed by Audioscopics in ular, which is based on single camera and projec- H
1935. Although the commercially successful tion that uses an anamorphic lens to create a wide
examples of 3D cinema was based on anaglyph screen picture (Reeves 1982; Patterson 1973).
method, there were attempts to use alternate frame Norton Heilig brought the immersive viewing
and polarized viewing technologies. The Tele- experience one step further, with his “Telesphere
view of Laurens Hammond was a public 3D Mask” and “Sensorama.” Patented in 1960 and
motion picture screening in New York, in 1962, these two inventions can be regarded as
December 1922. This 3D motion picture system multisensory theater with 3D images, stereo
installed at Selwynn Theater used a twin-strip 3D sound, wind, smells, and vibrations, but the inter-
camera with two lenses, dual projectors, and a activity was missing (Bown et al. 2017).
revolving electrical shutter affixed to the armrest Telesphere Mask was described as “Stereo-
of each spectator’s seat. The Zeiss Ikon company scopic Television Apparatus for Individual Use”
based in Dresden Germany used polarized 3D in patent documentation, with a pair of adjustable
methods. Working in cooperation with the State lenses, a pair of television tube units, a pair of
Establishment for Physics and Technology of earphones, and a pair of air discharge nozzles.
Braunschweig, Zeiss Ikon developed a high- Nozzles meant to provide air currents of varying
speed twin 16 mm stereo motion picture film velocities and temperature, with odor. Being a
system for use at the Berlin Olympics of 1936; wearable device, Telesphere Mask highly resem-
while in 1935, Otto Vierling developed a single- bles to the modern head mounted VR systems.
strip 35 mm stereo camera system using a prism in However, it does not provide any motion tracking
front of the lens for Zeiss Ikon, viewed using ability. Although Heilig actually build the device,
polarized glasses (Zone 2007). he focused on a more advanced system.
Wide screens were alternative to 3D stereo- Sensorama, patented in 1962, was a device
scopic vision in order to create an immersive roughly equivalent in size to a video arcade cab-
experience. In 1950s, the motion picture industry inet, with a housing that holds a hood to fit the
in USA was seeking for novelty in order to com- head of the observer, for viewing projected ste-
pete with television. Wide screen cinema was a reographic images through an optical setup. The
solution to attract viewers to movie theaters. hood also contains audio equipment and a breeze
Developed in 1952, Fred Waller’s Cinerama is directed toward to hood to enhance multisen-
used three simultaneously shot 35 mm films that sory experience. Heilig created five films for
were synchronized and interjoined into a single Sensorama, a bicycle ride, a ride of a dune
wide image projected on a huge curved screen by buggy, a helicopter ride over Century City, and a
three projectors, forming a 146 –55 viewing dance by a belly dancer, and a motorcycle ride in
876 History of Virtual Reality

New York city, which includes a vibrating seat, demonstrated his ideas with a prototype named
rush of air through observer’s head with odors “Sword of Damocles,” a head mounted 3D dis-
alongside the riding path. As this was the first play that is capable of presenting computer gen-
olfactory stimulus embedded within an immersive erated images to the user with a perspective image
technology, computer controlled olfactory inter- which changes as user moves. System employed
faces were not available until 1993 (Youngblut special spectacles containing two miniature cath-
et al. 1996). Heilig had the vision on the ode ray tubes for stereoscopic vision; two head
Sensorama’s potential in education, training, and position sensors, one mechanical and the other
marketing but it was a commercial failure and the ultrasonic, to measure the position of the user’s
only instance of it was employed as an arcade head. The name “Sword of Damocles” comes
console (Bown et al. 2017; Garner 2018). In the from the mechanical head positioning sensor, a
mid-1960s, he extended the idea to a multiviewer mechanical arm hanging from the ceiling. It is
theater concept patented as the Experience The- rather heavy and uncomfortable to use, and the
ater in 1969. ultrasonic solution was designed as an alternative.
The display system was an augmented reality
The Modern Age apparatus rather than a virtual reality device,
As there are numerous developments in VR dur- since the images on cathode ray tube displays
ing modern era, these are given as sections instead are reflected on half-silvered mirrors which
of chronological order. allow user to see real-world objects in the room
simultaneously. The images viewed by the user
Head Mounted Displays are transparent “wire frame” line drawings, due to
Built by Charles Comeau and James Bryan in the computational costs of rendering solid objects
1961, The Headsight was a remote surveillance in real time. Although the objects viewed by the
device that attaches a video camera to a head users are quite simple as a cubical room, Suther-
mounted display. It was not intended as an land (1968) reports favorable response of users to
immersive virtual reality technology. For its good stereographic vision.
motion tracking capabilities it is credited as a The origin of the contemporary VR HMD is
milestone in history of virtual reality, since it based on the design of Drs. Mike McGreevy and
uses magnetometers to track head movements of Jim Humphries at NASA Ames Research Center.
the user to control the attached camera (Jerald The system is called VIVED (Virtual Visual Envi-
2015; Bown et al. 2017; Garner 2018). ronment Display), which later evolved into VIEW
In 1967, a civil and military helicopter (Virtual Interactive Environment Workstation) as
manufacturing company tested a head-mounted a general-purpose, multisensory, personal simula-
display (HMD) that showed video from a servo- tor and telepresence device, configured with head
controlled infrared camera mounted beneath the and hand tracking, monochrome wide field-of-
helicopter. The camera motion is synchronized view stereo head-mounted displays, speech rec-
with pilot’s head, both augmenting his night ognition, 3D audio output, and a tracked and
vision and providing a level of immersion suffi- instrumented glove (Fisher et al. 1987) and has
cient for the pilot to equate his field of vision with encouraged several American companies to
the images from the camera (Fabri et al. 2008). develop related commercial products (Mcgreevy
A technical report of a project in Wright- 1991).
Patterson Air Force Base dated to 1969 is publicly An alternative to HMD’s were boom-mounted
available about the design of a helmet mounted displays, a stereoscopic binocular displays
display for US military pilots (Heard et al. 1969). attached to a multilink arm. While the arms pur-
Ivan E. Sutherland envisioned a computer con- pose was tracking the motion of user, it also helps
trolled virtual environment that emulates real-life to balance the display. Another advantage of this
physics in his classical article “The Ultimate Dis- kind of setup is that suitability for turn-taking use
play” (Sutherland 1965). In 1968, Sutherland for multiple people, i.e., when a user releases the
History of Virtual Reality 877

device, another person can take place and con- states that their efforts were also concentrated on
tinue to view the virtual environment from the design of virtual cockpits with very wide field of
same perspective. Sutherland’s Sword of Damo- vision. In September 1981, their work lead to a
cles was built around the same principle, while virtual cockpit system with 120 of view on the
several boom-mounted displays were available as horizontal (Carlson 2017).
commercial products in 1990s (Youngblut
et al. 1996). Input Devices: Data Gloves, 6DOF Input, and
Locomotion Systems
Flight Simulators In 1976, Thomas DeFanti and Daniel Sandin
The history of the mechanical flight simulators developed an inexpensive, lightweight glove to
can be dated back to 1910, but the foundations monitor hand movements. Based on an idea of
of modern day flight simulation were based on their colleague in Electronic Visualization Labo-
Edwin Link’s work (Allerton 2009). Based on the ratory, Richard Sayre, the glow is named as Sayre
simulator technologies developed by aviation Glove. Later in early 1980, MIT Architecture
industry, several end-user motion chairs, predom- Machine Group used a camera-based LED system
inantly intended for use in VE entertainment to track body and limb position for real-time com- H
applications, became available in 1990s puter graphics animation, termed “scripting-by-
(Youngblut et al. 1996). However, Link’s design enactment.” The method was intended as a body
was a motion simulator which did not include any motion tracking system, which can be also
external visual scene. Early visual systems were employed for hand tracking. In 1983, Gary
based on actual physical models, or pictures, such Grimes designed a glove, a specially tailored
as the German bombers were trained in WWII, in cloth for data entry using alphabet of hand signs
which a continuous picture of a ground scene was with numerous touch, bend, and inertial sensors
rolled under the trainee who looked through the sewn on the glove. In 1987, the team of Thomas
sight from a similar position to that in the actual Zimmerman developed the Data Glove that uses
bomber. Later, a miniature model of a scene fiber-optic wires and magnetic position tracking.
around an airport was used over which a video Commercialization of the product led to a wide-
camera traveled based on the aircraft’s position as spread use in research organizations, and inspired
calculated by the flight model (Lawn 1998). The a commercial gaming glove that is manufactured
first computer image generation systems for flight in 1989 (Sturman and Zeltzer 1994).
simulation is developed for space program (Rolfe As an alternative to relatively expensive
and Staples 1988). A version that is commercially gloves, 6DOF (six degrees of freedom) input
available for civil aviation companies, Vital devices were developed. In 1983, John Hilton
(Virtual Image Takeoff and Landing) was devel- sets out to develop a “3D force sensing joystick”
oped in 1969 as a laboratory prototype. The next for computer aided design applications while at
version, Vital II, was approved by FAA and the University of Sydney, Australia, and devel-
installed on a 737 simulator in 1972. Vital devices oped first prototype of Spaceball, which would
were only capable of visualizing lightpoints lead to a line of products named as spacemice
(Warwick 1987). Trainees see a night-like view (spacemice.org 2016). Spacemice devices have a
of the landing scene at simulator’s front windows, puck or ball that can be moved along X, Y, and
through a pair of cathode ray tube monitors. The Z axis as well as being twisted rotationally on each
VCASS (Visually Coupled Airborne Systems of those axis to roll, pitch, and yaw the 3D objects.
Simulator) was also an advanced flight simulator An alternative input device type is flying mouse,
training of fighter pilots (U.S. Congress, Office of or bat, originally developed by Ware and
Technology Assessment 1994). Trainees wore a Jessome (1988).
HMD that reflects the out of the window view. Another type of input devices are locomotion
Thomas A. Furness, who worked as a scientist in interfaces that allow users to walk in the virtual
the Wright-Patterson Air Force Base in 1970s, environment. These type of interfaces appeared as
878 History of Virtual Reality

treadmills, sliding shoes, and foot pads, as well as virtual environment through the head-mounted
some efforts to simulate walking with use of display (Medina et al. 2008). Circulafloor (Iwata
robotic tiles (Iwata 2013). et al. 2005) simulates an infinite surface by the
An early example of sliding shoes approach is circulation of a set of omnidirectional movable
Iwata’s “harness and roller skate” design in 1989, tiles. Each computer controlled tile move in the
in which the walker is fixed to the framework of opposite direction of the walker’s measured direc-
the system by a harness and a pair of roller skate tion, canceling the motion of the step and fixing
equipped with four casters which enables two- the walker’s position. Currently, the locomotion
dimensional motion as the motion of the feet devices have not become an affordable consumer
was detected by an ultrasonic range detector. product but there are several companies providing
The system evolved into “Virtual Perambulator,” turn-key solutions for industry, academia, and
which employs touch sensor equipped rubber san- business.
dals with low friction film located at the middle of
the sole (Iwata and Fuji 1996). Slater et al. (1995) Mirror Worlds
used a method that does not require an additional Mirror worlds or projected realities provide a
hardware, by employing a neural network algo- second-person experience in which the viewer is
rithm for analysis of the stream of coordinates represented by her image taken by a video camera
from the HMD, to determine whether or not the and merged into the virtual environment, and the
participant is walking on the spot. computer processes the users’ images to extract
One of the antecedent locomotion treadmills features such as their positions, movements, or the
were designed by James Lipscomb for the number of fingers raised (McLellan 2004). Com-
Walkthrough Project (Brooks 1987) in 1986, puter artist Myron Krueger’s efforts in combining
which uses a nonmotorized treadmill and bicycle interactive arts with VR were started in 1969, and
handles. Since directionality is acquired through led to Videoplace system in 1974 which is the
bicycle handles, the system was not an omni- earliest example of mirror worlds. The system
directional treadmill. Within US Army employs video-based motion tracking to analyze
Dismounted Infantry Training Program which the relationship between the user’s image and the
began in 1992 (Singer et al. 1998; Knerr 2000), computer-generated objects and combines both
a unicycle-like pedaling system was developed in on a projection display (Krueger and Wilson
1994, called Uniport. Uniport is the earliest exam- 1985). It can be used as a telepresence artifact
ple of foot pad approach in virtual locomotion, for two users interacting with each other through
followed by the OSIRIS in 1995, which utilizes a the computer generated graphics.
stair stepper device as same as that used in athletic Another early mirror world system dated back
gyms. Uniport is replaced with Treadport in 1995, to 1986 is the Mandala VR System developed by a
which is based on a unidirectional treadmill. group of Canadian performance artists and com-
Darken et al.’s (1997) omni-directional treadmill mercialized as a product. The system employs a
is a more advanced system, employing two layers video camera that implements the captured video
of belts controlled by servo motors. The top belt of the user into a computer generated environment
comprised of an array of freely rotating rollers lies or a previously taken video that is controlled by
atop a second, orthogonally oriented belt also computer. There were several applications of the
comprised of rollers. A tracking arm detects system including games and educational museum
user’s position. Another device within the tread- interactions (Wyshynski and Vincent 1993).
mill approach, Virtusphere, utilizes a human-sized
hamster ball. Built in 2006, Virtusphere is a ten- Virtual Projection and Virtual Spaces
foot hollow sphere, which is placed on a special In 1992, a research group in Electronic Visualiza-
platform that allows the sphere to rotate freely in tion Laboratory and the School of Art and Design
any direction according to the user’s steps. User is at the University of Illinois at Chicago introduced
able to walk and run inside the sphere, viewing the the concept of a room whose walls, ceiling, and
History of Virtual Reality 879

floor surround a viewer with projected images. an absence of case studies with cost-benefit ana-
The system they built had four screens, three lyses and a widespread absence of attention to the
walls, and a floor. It was named as CAVE requirements and limitations of the end users”
(CAVE Automatic Virtual Environment), with a were factors that affected the development con-
recursive acronym, in reminiscent of Plato’s alle- sumer level VR systems.
gory of the cave (Cruz-Neira et al. 1992). Several The first decade of the twenty-first century is
CAVE systems were built around the world, described as the “VR winter” (Furness 2014).
mostly in research institutions to be used as a There was little mainstream media attention
virtual environment, a virtual prototyping plat- given to VR from 2000 to 2012, while there
form, and for visualizing scientific 3D spatial were no consumer level VR products. By 2012,
datasets. ignited by a kickstarter project, companies rang-
Some examples are the CAVE’s in Ars Elec- ing from start-ups to the Fortune 500 began to see
tronica Center in Linz, Austria, which was the first the value of VR and started providing resources
to be installed outside the USA in 1996 (Kuka for VR development (Jerald 2015).
et al. 2009) and the world’s first six-screen CAVE
in Center for Parallel Computers at the Royal H
Institute of Technology in Stockholm, built in
Cross-References
1998 (Ihrén and Frisch 1999).
The advantages of CAVE environments over
▶ Interactive Virtual Reality Navigation Using
HMD displays is that multiple users can see each
Cave Automatic Virtual Environment
other in the CAVE at the same time, wearing
Technology
shutter glasses instead of heavy helmets. System
▶ Locomotion in Virtual Reality Video Games
usually runs on a cluster of networked computers.
▶ Natural Walking in Virtual Reality
Computers generate a pair of images following
▶ Presence and Immersion in Virtual Reality
each other, one for each eye of the user, synchro-
▶ Redirected Walking
nized with shutter glasses. As a result, images
▶ Virtual Hand Metaphor in Virtual Reality
seen by the user is three dimensional, as the
▶ Virtual Pointing Metaphor in Virtual Reality
objects are floating in the room. CAVE systems
▶ Virtual Reality: A Model for Understanding
employ a motion tracking technology to locate the
Immersive Computing
user in the room, besides interaction devices such
as data gloves, joysticks, or wands (Manjrekar
et al. 2014).
References
Consumer VR Allerton, D.: Principles of Flight Simulation. AIAA Edu-
In 1990s, VR systems became commercialized for cation Series, vol. 27. John Wiley and Sons, New York
consumers in entertainment industry. A leading (2009)
attempt was committed by Jonathan D. Waldern Bendazzi, G.: Animation: A World History. volume 1:
Foundations – the Golden Age. CRC Press (2016)
(1992). Some of these systems that aimed to be Benosman, R., Kang, S.B.: A brief historical perspective
commercialized as consumer products could not on panorama. In: Benosman, R., Kang, S.B. (eds.) Pan-
progress the prototype stage and failed to reach oramic Vision. Monographs in Computer Science.
consumers, while other were commercial failures Springer, New York (2001)
Bown, J., White, E., Boopalan, A.: Looking for the ulti-
due to the conflict in users’ high expectations and mate display: A brief history of virtual reality. In:
device’s functionality and reality (Garner 2018). Jayne, G., and Bown, J. (eds.) Boundaries of Self and
According to Stone (2008) “commercial naivety Reality Online: Implications of Digitally Constructed
on the part of VR companies, significant failures Realities, pp. 239–259 Academic Press (2017)
Brewster, D.: The Stereoscope; Its History, Theory
to deliver meaningful and usable intellectual prop- and Construction, with Its Application to the Fine and
erty on the part of so called academic ‘centers of Useful Arts and to Education. John Murray, London
excellence,’ expensive and unreliable hardware, (1856)
880 History of Virtual Reality

Brooks, F.P., Jr: Walkthrough – A dynamic graphics sys- Iwata, H., Yano, H., Fukushima, H., Noma, H.:
tem for simulating virtual buildings. In: Proceedings of CirculaFloor [locomotion interface]. IEEE Comput.
the 1986 Workshop on Interactive 3D Graphics, Graph. Appl. 25(1), 64–67 (2005)
pp. 9–21. ACM, New York (1987) Jerald, J.: The VR Book: Human-centered Design for Vir-
Carlson, W.: Computer graphics and computer animation: tual Reality. Morgan & Claypool, San Rafael (2015)
a retrospective overview. Open course materials from Knerr, B.W.: Interface issues in the use of virtual environ-
Ohio State Instructors, The Ohio State University ments for dismounted soldier training. Technical
Press books. Retrieved from www.osu.pb.unizin.org/ Report, Army Research Inst Field Unit, Orlando (2000)
graphicshistory, Accessed 10 Dec 2017 Krueger, M.W., Wilson, S.: Videoplace: a report from the
Cruz-Neira, C., Sandin, D.J., DeFanti, T.A., Kenyon, R.V., Artificial Reality Laboratory. Leonardo. 18(3),
Hart, J.C.: The CAVE: audio visual experience auto- 145–151 (1985)
matic virtual environment. Commun. ACM. 35(6), Kuka, D., Elias, O., Martins, R., Lindinger, C., Pramböck,
64–73 (1992) A., Jalsovec, A., Maresch, P., Hörtner, H., Brandl, P.:
Darken, R.P., Cockayne, W.R., Carmein, D.: The omni- DEEP SPACE: high resolution VR platform for multi-
directional treadmill: a locomotion device for virtual user interactive narratives. In: Proceedings of the 2nd
worlds. In: Proceedings of the 10th Annual ACM Sym- Joint International Conference on Interactive Digital
posium on User Interface Software and Technology, Storytelling: Interactive Storytelling, Guimarães,
pp. 213–221. ACM, New York (1997) pp. 185–196 (2009)
Fabri, D., Falsetti, C., Iezzi, A., Ramazzotti, S., Viola, S.R., Lawn, P.: The enhancement of a flight simulator system
Leo, T.: Virtual and augmented reality. In: Handbook with teaching and research applications. Doctoral dis-
on Information Technologies for Education and Train- sertation, Concordia University (1998)
ing, pp. 113–132 (2008) Manjrekar, S., Sandilya, S., Bhosale, D., Kanchi, S., Pitkar,
Fisher, S.S., McGreevy, M., Humphries, J., Robinett, W.: A., Gondhalekar, M.: CAVE: an emerging immersive
Virtual environment display system. In: Proceedings of technology – a review. In: 2014 UKSim-AMSS 16th
the 1986 Workshop on Interactive 3D Graphics, International Conference on Computer Modelling and
pp. 77–87. ACM, New York (1987) Simulation, pp. 131–136, IEEE, New York (2014)
Furness, T.A.: Introduction. In: Ben, D. (ed.) Sex Drugs Mcgreevy, M.W.: The virtual environment display system.
and Tessellation: The Truth About Virtual Reality as National Aeronautics and Space Administration, Tech-
Revealed in the Pages of Cyberedge Journal, pp. VII– nology 2000. 1:3–9 (1991)
X. CyberEdge Information Services, Oakland (2014) McLellan, H.: Virtual realities. In: Jonassen, D.H. (ed.)
Garner, T.A.: Echoes of other worlds: sound in virtual Handbook of research for educational communications
reality: past, present and future. Palgrave Macmillan and technology, Lawrence Erlbaum Associates, Lon-
(2018) don, pp. 461–498 (2004)
Gigante, M.A.: Virtual reality: definitions, history and Medina, E., Fruland, R., Weghorst, S.: Virtusphere: Walk-
applications. In: Earnshaw, R.A. (ed.) Virtual Reality ing in a human size VR “hamster ball.” In: Proceedings
Systems, pp. 3–14. Academic Press, London (1993) of the Human Factors and Ergonomics Society Annual
Heard, J.L., Hayes, D.O., Ferrer, J.J., Zilgalvis, A.: Design Meeting, vol. 52, no. 27, pp. 2102–2106. Sage Publi-
of an Airborne Helmet Mounted Display. Hughes Air- cations, Los Angeles (2008)
craft Company, Culver City (1969) Patterson, R.: Highlights from the history of motion picture
Ihrén, J., Frisch, K.J.: The fully immersive cave. In: formats. Am. Cinematogr. 54(1), 40–41 (1973)
Bullinger, H.J., Riedel, O. (eds.) 3. International Reeves, H.E.: The development of stereo magnetic record-
Immersive Projection Technology Workshop, ing for film (part I). SMPTE J. 91(10), 947–953 (1982)
10/11 May 1999, Center of the Fraunhofer Society Rolfe, J.M., Staples, K.J.: Flight Simulation Cambridge
Stuttgart IZS. Springer, Berlin (1999) Aerospace Series. Cambridge University Press, Cam-
IJsselsteijn, W.A.: History of telepresence. In: Schreer, O., bridge (1988)
Kauff, P., Sikora, T. (eds.) 3D Videocommunication: Singer, M.J., Ehrlich, J.A., Allen, R.C.: Effect of a body
Algorithms, Concepts and Real-time Systems in model on performance in a virtual environment search
Human Centered Communication. Wiley, Chichester task (no. ARI-TR-1087). Technical Report, Army
(2005) Research Inst for the Behavioral and Social Sciences,
Iwata, H.: Locomotion interfaces. In: Steinicke, F., Visell, Alexandria (1998)
Y., Campos, J., Lécuyer, A. (eds.) Human Walking in Slater, M., Steed, A., Usoh, M.: The virtual treadmill: a
Virtual Environments. Springer, New York (2013) naturalistic metaphor for navigation in immersive vir-
Iwata, H., Fujii, T.: Virtual perambulator: a novel interface tual environments. In: Göbel, M. (ed.) Virtual Environ-
device for locomotion in virtual environment. In: Pro- ments ’95. Eurographics. Springer, Vienna (1995)
ceedings of the IEEE 1996 Virtual Reality Annual Spacemice.org: Spaceball product line. spacemice.org/
International Symposium, pp. 60–65, IEEE, Santa index.php?title¼Spaceball (2016). Accessed 11 Dec
Clara (1996) 2017
Holography as an Architectural Decoration 881

Steinicke, F.: The science and fiction of the ultimate dis-

play. In: Being Really Virtual, pp. 19–32. Springer Hologram
International Publishing (2016)
Stone, R.J.: Serious games: virtual reality’s second com-
ing? Virtual Real. 13(1), 1–2 (2008) ▶ Holography as an Architectural Decoration
Sturman, D.J., Zeltzer, D.: A survey of glove-based input.
IEEE Comput. Graph. Appl. 14(1), 30–39 (1994)
Sutherland, I.E.: The ultimate display. Proc. IFIPS Con-
gress. 2, 506–508 (1965)
Sutherland, I.E.: A head-mounted three dimensional dis-
play. In: Proceedings of the AFIPS ’68 Fall Joint Com- Holographic Augmented
puter Conference, Part I San Francisco, California – Reality
December 09 – 11, pp. 757–764 (1968)
U.S. Congress, Office of Technology Assessment: Virtual ▶ Mixed Reality, Gamified Presence, and Story-
Reality and Technologies for Combat Simulation –
Background Paper, OTA-BP-ISS-136. U.S. Government telling for Virtual Museums
Printing Office, Washington, DC (1994)
Uricchio, W.: A ‘proper point of view’: the panorama and
some of its early media iterations. Early Pop. Vis. Cult.
9(3), 225–238 (2011)
H
Virtual Reality Society: Who coined the term “virtual real-
ity”? https://www.vrs.org.uk/virtual-reality/who-coined-
Holographic Filmmaking
the-term.html (2017). Accessed 8 Dec 2017
Wade, N.J.: Wheatstone and the origins of moving stereo- ▶ Tabletop Storytelling
scopic images. Perception. 41(8), 901–924 (2012) ▶ Volumetric Filmmaking
Waldern, J.: VR the applications and commercialisation.
In: Proceedings of AUUG 92 Conference on
Maintaining Control in an Open World, Melbourne,
8–11 Sept 1992
Ware, C., Jessome, D.R.: Using the bat: a six-dimensional
mouse for object placement. IEEE Comput. Graph. Holography
Appl. 8(6), 65–70 (1988)
Warwick, G.: Vital Revitalised. Flight International,
pp. 31–32. www.flightglobal.com/FlightPDFArchive/ ▶ Holography as an Architectural Decoration
1987/1987%20-%200369.PDF (1987) ▶ Mixed Reality and Immersive Data
Wyshynski, S., Vincent, J.V.: Full-body unencumbered Visualization
immersion in virtual worlds (the vivid approach and
the Mandala ® VR system). In: Wexelblat, A. (ed.) Vir-
tual Reality: Applications and Explorations, pp. 46–65.
Academic Press, New York (1993)
Youngblut, C., Johnston, R.E., Nash, S.H., Wienclaw, R.
A., Will, C.A.: Review of virtual environment Holography as an
interface technology (no. IDA-P-3186). Technical Architectural Decoration
Report, Institute for Defense Analyses, Alexandria
(1996)
Zone, R.: Stereoscopic Cinema and the Origins of 3-D Setsuko Ishii
Film, 1838–1952. University Press of Kentucky, Lex- Independent Artist, Bunkyo-ku, Tokyo, Japan
ington (2014)

Synonyms

Architectural decoration; Dichromate gelatin

HMD: Head-Mounted Display (DCG); Display holography; Hologram; Hologra-
phy; Multicolor rainbow hologram; Rainbow
▶ Virtual Reality Applications in Education hologram; Silver halide
882 Holography as an Architectural Decoration

Definition displays the colors of the light (Ishii and Tsujiuchi

2008).
Holography as an architectural decoration is an Various methods exist in which holograms are
application of holography to the everyday living used as raw materials. Holograms are used in
space of a building. sculptural decorations as three-dimensional
objects, atrium decorations, holography chande-
liers, wall decorations, and in the staging of indoor
prismatic displays of sunlight using holography
Introduction grating.
This entry describes various attempts to bring
Applications of holography have been developed the attractive, decorative applications of display
and implemented in many fields; display hologra- holography into architectural spaces (Ishii 2007).
phy is the most important and exciting application
(Ishii 2006b). Various types of holograms exist,
each of which has its own unique color or luster Shaw Cases
characteristics; some examples are reflection,
laser transmission, white light transmission Holography Chandeliers
(rainbow), silver halide, and dichromate gelatin The work (Fig. 1) was installed in the stairwell of
(DCG) holograms. In particular, the DCG reflec- a pavilion building at the Science Exposition,
tion hologram exhibits certain favorable charac- which was held in 1985 in Japan. DCG holograms
teristics such as a metallic luster or pearl color. and low-voltage miniature halogen lamps were
The white light transmission (rainbow) hologram assembled together, which formed a holography
is particularly impressive because of the way it chandelier.

Holography as an
Architectural
Decoration, Fig. 1 DCG
reflection type of hologram
Holography as an Architectural Decoration 883

Sculptural Object intersect. The use of the hologram in the design of

The case (Fig. 2) is installed in an entrance and the space has a significant influence on the sur-
consists of multiple DCG holograms, stainless rounding environment. Its influence is especially
steel poles, and a black granite base. Around the noticeable at night when passing pedestrians can
object’s circumference, beams of light which are enjoy the object’s brilliant effect in the entrance of
transmitted and/or are reflected from the hologram the office building.

Atrium Decoration in Public Space

Multiple DCG holograms and 10 large dichroic
mirrors were combined and installed in a skylight
window in 1997 at the public Health and Welfare
Center of Japan. This demonstrates the combina-
tion of vivid color interference and holographic
image (Ishii 2007). The appearance of the holo-
grams and colors change constantly, depending on
variance of ambient lighting conditions which is H
caused by factors such as weather and time
(Fig. 3).

Wall Decoration
The works (Figs. 4 and 5) were finished in the
shape of a mural painting. A silver halide reflec-
tion hologram with a diameter of 160 cm (Fig. 6)
is installed in the basement (Ishii 1993). The
reconstruction of the holographic image is
enabled by guiding sunlight through optical fibers
Holography as an Architectural Decoration, that stretch from the roof of the building to the
Fig. 2 Sculptural object in an entrance, 270 cm  200 cm basement (Ishii 2006a). Staff in the building are

Holography as an Architectural Decoration, Fig. 3 Atrium decoration in a public space

884 Holography as an Architectural Decoration

Holography as an Architectural Decoration,

Fig. 4 DCG reflection hologram, 150 cm  240 cm
Holography as an Architectural Decoration,
Fig. 7 On the ceiling of an entrance hall

Holography as an Architectural Decoration,

Fig. 5 Entrance of the residential building

Holography as an Architectural Decoration,

Fig. 8 At a cafeteria

aware of the weather outside without moving

from their seat.

Rainbow Production
When sunlight hits the grating installed in the
crosspiece of a glass window, the diffracted
light produces a rainbow color on a ceiling or
wall (Figs. 7, 8, and 9). The rainbow moves
across various surfaces with the movement of
the sun. The sunlight’s production of the rainbow
carries brightness and peacefulness into an
everyday living space. Holography is the easiest
Holography as an Architectural Decoration,
Fig. 6 Large format silver halide holograms reconstructed way to bring light of the sun inside a space (Ishii
from sunlight 2007).
Holography as an Architectural Decoration 885

Application for Large Format Multicolor construction space (Fig. 10a) in 2003. Film
Rainbow Holograms holograms are laminated between two pieces of
Three large format multicolor rainbow holo- glass. Behind the holograms, a flat vessel
grams (Fig. 10b) were installed in a large-scale containing water and with mirrors on its bottom
is situated on the floor. The device produces
ripples on the surface of the water that causes
water droplets to fall from the top. The lights
used for hologram reconstruction reflect from
the mirrors under the water, adding dynamic
movement and imparting a sense of vividness
to the virtual image.
The transmission hologram requires a large
area behind the hologram for lighting. At this
point, the type of reflection is easier to manipu-
late than the type of transmission. In the follow-
ing cases (Fig. 11a–c), multicolor rainbow H
holograms were installed and are dependent on
the quality of their reflection, which is produced
by mirrors. The works in Fig. 11b, c are installed
in Taiwan.

Conclusion

When a hologram is likened to a painting, a tile,

or a pottery plate, it can hold great potential as a
new material for wall surface ornamentation.
The wonderful and attractive effect created by
the holograms cannot be realized with any other
Holography as an Architectural Decoration,
Fig. 9 Dining room of a care home for the elderly materials currently used in everyday living

Holography as an Architectural Decoration, Fig. 10 (a) At Tokyo Institute of Technology. (b) Multicolor trans-
mission hologram with water ripple, 210 cm(h)  300 cm(w)  250 cm(d)
886 Holography as an Architectural Decoration

Holography as an Architectural Decoration, Fig. 11 Multicolor rainbow hologram with a mirror. (a)
110 cm  10 cm  4 cm. (b) 70 cm  140 cm. (c) 230 cm  110 cm

space. When a space is designed to accommo- References

date the use of a hologram, it has a significant
influence on its surrounding environment and Ishii, S.: A novel architectural application for art hologra-
phy. In: Holographic Imaging and Material, vol. 2043,
produces brilliant, delightful, and comfortable
pp. 101–103. SPIE, Quebec (1993)
spaces. Ishii, S.: Artistic representation with holography, Paper in
the Journal of the Society for Science on Form, Forma
21, 81–92 (2006a)
Ishii, S.: Holography Art, 3D (three dimensional) Image
Handbook, Part II, 4.1, pp. 257–265. Asakura Publish-
ing Co. Ltd, Shinjyuku-ku, Tokyo (2006b)
Cross-References Ishii, S.: Art, Interior and Decoration, Holography Material
& Application Handbook, Part II, Sec.1.1,
▶ Image Quality Evaluation of a Computer-Gen- pp. 205–213. NTS, Chiyoda-ku, Tokyo (2007)
Ishii, S., Tsujiuchi, J.: Chapter 6. Where are we going in art
erated Phase Hologram
holography? In: New Directions in Holography and
▶ Mixed Reality Speckle, pp. 95–113. American Scientific Publishers,
▶ Substitutional Reality Stevenson Ranch (2008)
Holography, History of 887

Advancement of Science in 1858 (Howard

Holography, History of 2015). Dircks tried to sell his effect to theaters,
but with no luck. It was a few years later in 1862
Emily Peed1 and Newton Lee1,2 that John Henry Pepper modified the setup and
1
Institute for Education, Research, and projected ghosts for Dickens’ The Haunted Man,
Scholarships, Los Angeles, CA, USA and while he tried to give proper credit to
2
Vincennes University, Vincennes, IN, USA Dircksian, it became renowned as “Pepper’s
Ghost.” This would turn up at fairgrounds,
haunted houses, magic shows, and any venue
Synonyms seeking to add flair until the technology modern-
ized (Howard 2015).
Diffraction microscopy; Holoscopy; Three- Holography is a now familiar technology from
dimensional picture; Wavefront reconstruction its once obscure background. Derived from the
microscopy Greek words holos, meaning “whole,” and
graphe, meaning “something written,” it was
coined in 1947 by British scientist Dennis Gabor H
Definition while working on improving the resolution of an
electron microscope. After learning of Gabriel
The use of illuminated light waves to create three- Lippmann’s work at the young age of 15, Gabor
dimensional photographic representations of became inspired to study physics. Lippmann
objects. Typically recorded on a photographic developed a theory of using light interference to
plate or film with a pattern of interference formed capture color photography decades prior in 1886,
by split laser beams that scatter light and create the eventually creating perfect color photographs to
appearance of a three-dimensional shape once the Academy of Sciences in 1893 and publishing
interacted with by light. his full theory in 1894. He won a Nobel Prize for
his work in 1908 and later became an adviser at
the Physics Department at the Sorbonne, where he
Introduction introduced Marie Sklodowska to her future part-
ner and scientific genius Pierre Curie (History of
The history of holographic technology begins Holography 2016).
hundreds of years ago in the inspiration of the Learning of this research, Dennis Gabor
past. Carried forward to each generation by stra- became inspired. But his story comes from a
tegic people, the total history of holographics is time of strife. He, as Lippmann, was also Jewish.
challenging to capture concisely. Below is an He relocated to England to escape religious per-
account of important people, pivotal moments, secution at the hands of the Nazis in 1933
and critical inventions that brought us to our holo- (History of Holography 2016). It was then that
graphic technology of today. he moved to Thomson-Houston Research Labo-
ratories in Britain. It was during his stay there
that he developed the theory of holography, but
History of Holographics due to technical infeasibility of the concept at the
time, it would be another 20 years before the
Holographics has a forebearer in the concept of technology that we know as holography was
“Pepper’s Ghost.” Inspired by the 1584 research able to come to fruition. He was attempting to
of Italian scientist Giambattista della Porta, the improve the resolution of the electron micro-
inventor of the camera obscura, Henry Dircks scope to be capable of seeing singular atoms,
experimented with bouncing images off of plate but came upon his own theory of wavefront
glass and created Dircksian Phantasmagoria, reconstruction, soon to be called holography.
which he presented to the Association for the What was required would be a powerful form of
888 Holography, History of

light, which came in the 1960s with the invention Upatnieks had fled the Soviet occupation of Lat-
of the laser. This allowed for the intense, coher- via with his parents, seeking asylum in Germany,
ent light necessary to construct holograms. and eventually emigrating to the United States
Holographics represents another trend for (Emmet Leith and Juris Upatnieks Co-Inventors of
humanity, our ability for many of us to stumble Holography). He attended high school in Ohio, and
upon a similar technology at the same time. While after studying Electrical Engineering at the Univer-
not knowing of Gabor’s work, a Russian in the sity of Akron, he received his Bachelor’s degree in
former Soviet Union by the name of Yuri 1960 and began conducting research at the Michi-
Denisyuk became inspired by the very man that gan facility. What this pair would create together
inspired Dennis Gabor, Gabriel Lippmann. After would cement their hands in the creation of mod-
reading of Lippmann’s work, he began to experi- ernized holographics, as they solved the twin image
ment in 1958 using a highly filtered mercury issue and greatly improved the technology. Even
discharge tube, as there were no lasers yet. His Gabor himself would mention Leith and Upatnieks
work was published in 1962 to lackluster support. in his 1971 Nobel Prize speech, stating that their
It wasn’t until a visiting delegation of American success “was due not only to the laser, but to the
scientist requested to meet with him that he saw long theoretical preparation of Emmett Leith, which
his own esteem and fortunes rise. started in 1955. . . This was in fact two-dimensional
Emmett Leith is another individual that holography with electro-magnetic waves. . . Their
followed a similar hunch like Gabor and results were brilliant” (Emmett Leith).
Denisyuk. At the relatively young age of 25, he Like other technologies, the 1960s and 1970s
began researching highly classified works at the were a time of true maturation. In 1965, Robert
University of Michigan’s Willow Run Laborato- Powell and Karl Stetson published a paper on
ries in 1952. He was set to work to study holographics interferometry, which proved useful
synthetic-aperture radar (SAR). The army desired for nondestructive testing of materials, fluid flow
high-quality imaging radar system, but without analysis, and quality control. Larry Siebert of the
further innovation in the field, then the antenna Conductron Corporation utilized a pulsed laser to
was projected to be so large that no airplane could create the first hologram of a person in 1967. This
carry it. They sought to create a synthetic antenna was instrumental in the early days of commercial
that had high-powered capability because it pro- display holography. Unfortunately, a recession in
cessed image data like a hologram – within small the early 1970s forced the company to close,
pieces over transmitted pulses. By 1957, Leith’s shutting the door on a potentially huge market.
new method was ready to be tested. After eight The late 1960s saw Stephen Benton’s inven-
flights yielded no images, the critics seemed val- tion of white-light transmission holography while
idated in their doubts of such a technology; how- researching holographic television for Polaroid
ever, on the ninth flight, the terrain was beautifully Research Development (Sergey). It was signifi-
captured and the SAR system became famous cant because it made the mass production of holo-
(Emmett Leith). grams possible by stamping the interference
It was in the 1960s that Leith turned toward the patterns onto plastic. These holograms could be
work of Dennis Gabor, realizing that there was duplicated millions of times over for a few cents
more interesting research to be conducted. apiece. This created embossed holograms that are
Gabor’s work produced fuzzy images that also now utilized by publishing, advertising, and many
contained twin images, and the doubling of other industries.
images was deemed unsolvable. Leith approached Lloyd Cross combined white-light transmis-
Juris Upatnieks, whom was not originally sion holography with more conventional cinema-
impressed by the work and recently joined the tography processes to create more realistic
facility but become convinced after reading illusions. One of the most famous is a series of
Gabor’s experiment in Principles of Optics by photographs called “Kiss II” in 1974, which was
Born and Wolf (1959). made from approximately 360 frames and shows
Holography, History of 889

an image of a woman named Pam Brazier blowing actually a twenty-first-century application of Pep-
a kiss and winking at the viewer as they walk by per’s Ghost. A custom-developed foil was utilized
(History). He would later create Multiplex Com- in the technique to create more realistic images.
pany that produced hundreds of images using his Tupac is not the only artist to be brought to life
technique. using this type of technology. The cartoon band
The field of holography was divided into sev- by the name of the Gorillaz has long experimented
eral camps. There were researches, artists, and with the concept, but due to technical issues,
artisans. In 1972, Tung Jeong began offering sum- hologram-based concerts were still a challenge
mer workshops for non-physicists at Lake Forest to tackle, and it took several more years before
College in Illinois. This introduced a new medium we were able to sit in a truly hologram-like
for expression for artists that many ran with. Art- experience.
ists like Salvador Dali utilized holographic tech- In 2014, Uwe Maass came into the limelight
nology in exhibitions, such as at the Knoedler again. Forming a venture with his peers, he cre-
Gallery in New York. Work like his and others ated Hologram USA and MDH Musion, both of
contributed toward the mainstreaming of this which are doing amazing things in the field of
technology, as more of the public was introduced holographics. One example of the impressive H
to it. sway this technology can hold comes from
Through the next few decades until the Narendra Modi, whom entered the race for
millennia, there was the development and appli- Prime Minister of India. With low polling num-
cation of the technology across different spaces. bers, he hired MDH Musion and began delivering
The 1980s saw more integration of the technology speeches simultaneously at hundreds of rallies
to familiar items. In 1983, MasterCard Interna- across 1400 locations – reaching an estimated
tional, Inc. was the first to use hologram technol- 14 million additional voters. His numbers went
ogy in their banking security (History). It was from a low 34% to 53% and won him the election
deemed the largest distribution of holographic (Howard 2015).
technology at that time. It was in March of 1984 The 2016 live performance by Callie and
that holographic technology made its way onto Marie at Niconico Tokaigi, Japan (Splatoon
the cover of the National Geographic 2016), and the 2018 Hatsune Miku Concert in
(volume 165, Number 3), with nearly 11 million Los Angeles (Hatsune Miku Concert 2018) have
holograms being carried throughout the world both demonstrated new advances in holography
(History). In 1997, German inventor Uwe Maass where concertgoers were able to enjoy a
created a roll up, transportable technology that hologram-like experience.
enabled a new type of holographs that he deemed The holography market is projected to hit
“eyeliner.” This technique was modernized and $3.57 billion by 2020 (Holographic Display Mar-
utilized by a company called Zebra Imaging. ket). With applications across entertainment, mil-
The army utilized the technology in 2006 to itary operations, financial markets, politics, work,
have field maps that were 2  3 ft and could and home use, there is an appetite for the technol-
have a light shined on them which reveal a holo- ogy. After many decades it took to bring it to full
gram of the terrain before them. It would show the actualization, the twenty-first-century evolution
steepness of the topography and where they might of the technology may finally bring what we
be vulnerable to ambush. The three- dimensional have been only able to speculate in science fiction
aspects of the new maps were a hit, and over novels to life.
14,000 holographic maps were utilized by Amer-
ican troops in Iraq and Afghanistan over the next
decade (Howard 2015). Cross-References
Some might remember the 2005 concert where
Tupac was presented on stage, while some may ▶ Hologram
consider this a hologram technology, it was ▶ Holography
890 Holoscopy

References
HRI AR
Emmet Leith, Juris Upatnieks Co-Inventors of Hologra-
phy: Emmett Leith and Juris Upatnieks, Millennium
▶ Augmented Reality for Human-Robot Interac-
Project, University of Michigan. http://um2017.org/
2017_Website/Emmett_Leith_and_Juris_Upatnieks. tion in Industry
html. Website information comes from: http://
um2017.org/ – a site dedicated to tracking the history
of University of Michigan for its 200 year
anniversary
Emmett Leith: Emmett Leith Inventor of Practical Holog-
raphy, University of Michigan. http://ece.umich.edu/ HRIR
bicentennial/stories/emmett-leith.html
Hatsune Miku Concert: Los Angeles HD 1080P 60FPS
Full Length. https://www.youtube.com/watch? ▶ User Acoustics with Head-Related Transfer
v¼0jrtOBM97X4 (2018) Functions
History: The History and Development of Holography,
hologram, holograms, holography, holography exhibi-
tions, holographic images, 3-D, 3-Dimensional images,
Hologram, Holograms, Holography, Holography Exhi-
bitions, Holographic Images, 3-D, 3-Dimensional
Images. HOLOPHILE, INC. www.holophile.com/
history.htm
HRTF
History of Holography: Holographic Studios, 22 Mar 2016.
www.holographer.com/history-of-holography/ ▶ User Acoustics with Head-Related Transfer
Holographic Display Market worth $3.57 billion by 2020. Functions
Markets and Markets. www.marketsandmarkets.com/
PressReleases/holographic.asp
Howard, D.: 400 Years of Holograms: The History of Illu-
sion. Popular Mechanics, Popular Mechanics, 24 June
2015. www.popularmechanics.com/technology/gadgets/
a16141/holograms-are- people-too/
Sergey, Z.: History of Holography. Holography – virtual HUD (Heads-Up Display)
gallery. www.holography.ru/histeng.htm
Splatoon: Squid Sisters – Live Concert at Niconico Tokaigi ▶ Game Interface: Influence of Diegese Theory
2016. https://www.youtube.com/watch?v¼wxkKUb
NnXKE
on the User Experience

Human Detection
Holoscopy
▶ Deep Learning Algorithms for 3D
▶ Holography, History of Reconstruction

Horror Game Human Factors

▶ Five Nights at Freddy’s, a Point and Click Hor- ▶ Cognitive Psychology Applied to User Experi-
ror Game ence in Video Games
Human Interaction in Machine Learning (ML) for Healthcare 891

What Is Machine Learning (ML)?

Human Interaction in
Machine Learning (ML) for Machine learning (ML) is a type of artificial intel-
Healthcare ligence (AI) technologies that allows software
applications to become more accurate at pre-
Sara Al Hajj Ibrahim and dicting outcomes. There are vast opportunities
Fatemeh Dehghani that ML gives, some being used right now, others
Faculty of Science, Ontario Tech University, yet to come. As ML becomes pervasive in our
Oshawa, ON, Canada day-to-day applications, most systems nowadays
involve ML. ML led to better outcomes and
improved efficiency in most scenarios. In this
Synonyms context, systems implement and utilize different
ML and AI algorithms. The nature of ML imple-
Artificial intelligence; Deep learning; Machine mentation involves systems having a more signif-
learning; Neural networks icant than a typical role in obtaining, utilizing, and
protecting data. ML models extract valuable fea- H
tures and insights from data to detect different
activities in systems. Systems are now responsible
Definitions for more advanced and expensive decisions.

The field of artificial intelligence (AI) is the devel-

opment of machines that can perform human-like Machine Learning (ML) in Healthcare
tasks, such as reasoning, perception, and decision-
making. Machine learning (ML) is a subfield of Due to the improvement in ML, the variety of
AI that allows machines to learn from data with- applications for healthcare information systems
out the need for explicit programming. Neural has increased dramatically. This advancement has
networks (NN) and deep learning (DL) take this opened the door to many medical operations based
a step further by enabling machines to learn from on ML models. ML systems provide medical situ-
complex and unstructured data like speech, ational insights, quick medical reminders, and accu-
images, and natural language, and use that data rate medical forecasts (Koh and Tan 2005).
to make predictions. NN is an ML algorithm that Utilizing these cutting-edge technologies has accel-
imitates the structure and function of the human erated the digitization of healthcare institutions,
brain, consisting of interconnected neurons that primarily driven by the need to improve medical
process information. DL is a specific subset of NN processes. ML systems are now a tool that can
that utilizes multiple layers of artificial neurons to supplement physicians’ decision-making process,
enable machines to learn and make predictions generate individualized and customized health
from vast amounts of data. Human-computer management plans, predict the next health crisis,
interaction (HCI) studies how people interact and develop personalized treatments that employ
with digital technologies and designs interfaces precision medicine (Chancellor et al. 2016).
and interactions that are user-friendly and intui- In recent years, healthcare systems got
tive. This is essential in creating effective and involved in many AI and ML contexts, from mon-
efficient AI and ML systems that enhance itoring things such as diabetes (Nasser et al. 2021)
human-computer interactions. Together, these and cancer care (Onasanya and Elshakankiri
technologies and approaches work in harmony to 2021; Pradhan and Chawla 2020) for identifying
create intelligent machines and improve the user different kinds of pathologies and chest x-rays or
experience. even in mental health as identifying depression
892 Human Interaction in Machine Learning (ML) for Healthcare

(Kumar et al. 2021). Healthcare systems now can many cases, humans outperform AI. To improve
predict and control any new infectious disease, the design of automated streetlights, a human–AI
e.g., COVID-19 (Mukherjee et al. 2021) or Mon- interface using both human and ML techniques is
keypox (Ahsan et al. 2022). As such, ML and AI presented. In a different case, the text is put into
in healthcare have shown to be very extensive in groups using ML, even though the initial data
achieving a higher quality of service and patient processing was done automatically (Yang et al.
health. 2019). Humans are added to the loop if the first
results are not good enough. Such issues show
some limitations of ML, and that human under-
Interactive ML (I-ML) standing is the primary remedy. ML can deliver
precise and accurate solutions to well-structured
Real-world uses of ML techniques have shown queries, but not too ambiguous ones while build-
weaknesses that result in poor results and the need ing complex new algorithms. When humans and
for improvement. ML algorithms require access to ML techniques are combined, systems become
good-quality, unbiased, and complete training more effective and exceptional, as humans are
data to work optimally. In most cases, a lack of the most informed and possess abstract thought.
high-quality data leads to poor results. While ML
techniques can provide exact and quite well solu-
tions for well-structured problems, they are I-ML in Healthcare
unsuccessful for nondeterministic polynomial
time (NP)-hard and ill-conditioned problems. Various research on human behavior for Al inter-
Humans are the only entities capable of actions has been published in the field of medical
explaining the limitations of ML approaches applications. Protein structure, genomic annota-
since humans are excellent at abstract thoughts tion, image analysis, and knowledge base popula-
and solving computationally complex problems. tion are healthcare problems still requiring human
As such, one way to improve the performance of involvement. In some instances, many humans are
ML systems is through human involvement in the necessary; in others, we need only a limited num-
process. The mechanism through which humans ber of highly trained professionals in specific
and ML systems work together is known as inter- fields. For example, in (Caruana et al. 2006), ML
active machine learning (I-ML). It is feasible to and biochemists work to categorize low-level pro-
construct systems more quickly and effectively by tein structures. They begin by clustering protein
merging human feedback with ML, and that leads structures. After then, biochemists would discuss
to more precise system design and predictions that the data. Biochemists review the discoveries and
are more accurate. For example, Fails and Olsen apply new constraints for the next iteration. Clus-
established I-ML, showing its importance and tering parameters are next changed to fit restric-
promise (Fails and Olsen 2003). Moreover, in tions. In another study, experiments evaluate the
(Zerilli et al. 2019), Zerilli et al. discuss including I-ML “human-in-the-loop” technique, especially
individuals in decision-making. Humans and ML when it comes to opening the “black box” and
will need to work together in the future for many letting a person alter and interact with an algo-
ML applications, no matter how well planned. rithm directly or indirectly (Holzinger et al. 2016).
Elish and Watkins observe that using an AI system The research involves a framework for the travel-
to assist in identifying sepsis patients disrupted ing salesman problem that solves practical diffi-
clinical operations and drove nurses to devise culties in health informatics, such as protein
novel solutions to the problem (Elish and Watkins analysis. In (Holzinger 2016), a study identifies a
2020). Nurses are responsible for integrating AI problem by finding globally optimal protein
findings with established clinical diagnostic pro- threading, which checks if threading with a score
cedures. In (Nascimento et al. 2018), ML experts less than or equal to K exists. This problem is
attempt to automate streetlights using ML. In categorized as NP-hard. Problems labeled as
Human Interaction in Machine Learning (ML) for Healthcare 893

NP-hard cannot be solved in polynomial time. annotations to be used for model training. More-
One of the primary advantages of human interac- over, experts fine-tune the model using the new
tion in ML is the ability to tackle NP-hard prob- recent annotated data until the ML algorithm is
lems. Another study in (Holzinger 2016) optimal. In the discipline of ML, “annotation”
discusses the k-anonymization problem of a refers to labeling data to indicate the outcome you
record publication that cannot be identified from want the model to predict. In this context, we train a
k other entities in the data. While the task of model with the new annotations, and the model
k-anonymization is similarly NP-hard, the sup- then uses the latest data to increase its prediction
pression or generalization of attributes can mea- accuracy. As such, we acquire more precise labels.
sure the effectiveness of the outcome until each This procedure, where experts feed data to the
database row is identical to at least k1 other row. computer to assist future decisions, is known as
Using generalization and suppression, an exten- supervised ML. The goal of training is to enable the
sion of the k-anonymity model maps any given algorithm to make correct decisions when pre-
record in the dataset to at least k others. A human sented with new data. Unlabeled datasets, on the
best performs this. other hand, are employed as unsupervised
ML. Under these conditions, the algorithm is pro- H
grammed to seek and define its own structure of
Experts Within Healthcare I-ML unlabeled data. This part is referred to as a deep
learning (DL) approach.
In I-ML applications, when the task is profes- – Testing phase: Testing and evaluation from
sional and complicated, we observe that humans experts help in fixing wrong results. There are two
with a higher domain expertise should be in the kinds of wrong decisions: those in which the
loop. With the help of domain experts, ML appli- algorithm is uncertain and those in which the
cations have coined a new term: HILML, or algorithm is certain, but the outcome is incorrect.
human-in-the-loop machine learning (Maadi To solve this problem, and after an ML model has
et al. 2021). Generally speaking, for humans been trained, experts evaluate model predictions
who collaborate with ML methods, the more and make changes to it to get the most accurate
expertise is, the better human–ML interaction out- results for data. When experts are involved, auto-
puts are achieved. Their assistance entails incor- mated predictions may need to be changed by
porating patient-specific information, treatment hand in order to meet those criteria. A model
outcomes, and any additional repercussions asso- must be capable of communicating with experts
ciated with previous decisions made by the semi- to provide meaningful interpretations of model
automated system. In addition to assisting with predictions. This lets users take the best action
pattern recognition and providing data from the when interacting with model outputs and
outside, their assistance also includes giving decreases human uncertainty.
external data. It is very beneficial to combine the
specialized knowledge of medical professionals,
such as physicians, and doctors, into intelligent I-ML and Serious Games
healthcare systems through interactive ML, which
is later strengthened by further data and expert Serious games are becoming increasingly impor-
knowledge. According to (Budd et al. 2019), tant in the field of I-ML in healthcare as they offer
there are two stages where medical professionals a unique and efficient way to solve medical prob-
can help. lems. By simulating real-world medical scenarios
– Training phase: The training phase consists through games, medical professionals can assess
of inputs and predicted output pairs manually their decision-making skills and improve their
labeled by experts. Expert engagement is helpful ability to diagnose and treat patients in a con-
in assisting ML when datasets are small or of poor trolled and safe environment. I-ML uses serious
quality. Hence, medical experts add new games to provide healthcare professionals with
894 Human Interaction in Machine Learning (ML) for Healthcare

hands-on learning opportunities and the opportu- For example, if a patient is told that an image led
nity to apply their knowledge in a hands-on envi- to a cancer diagnosis, they will undoubtedly want
ronment. In (Wang et al. 2016), a systematic to know why. Moreover, ML systems can deliver
review aims to assess the current state of research more accurate assessments combined with the
on the use of serious games in healthcare and expert knowledge of healthcare professionals,
provide an overview of such games’ history and particularly when using small or poor-quality
current usage. The authors further discuss poten- healthcare datasets. This is done using the previ-
tial benefits and challenges associated with using ous two roles mentioned, data labeling combined
serious games in healthcare. with consistent feedback on the algorithm’s
I-ML also uses computer graphics and data decisions.
visualization to create virtual environments that On the downside, however, experts need to
enable medical professionals to visualize and frequently annotate data during the learning
interact with complex medical structures and pro- phase and verify the decisions suggested by the
cesses. Additionally, serious games can provide ML model to ensure that it is the best decision
real-time feedback and analytics by monitoring regarding the risks imposed on patients. The pro-
player performance. This allows you to identify cess of data labeling and continuous feedback are
and address improvements. This kind of feedback time-consuming manual processes. Labeling
is extremely valuable to healthcare professionals requires experts to annotate and categorize com-
as it supports ongoing skill development. A study plex images such as X-rays, CT scans, etc. Also,
explores the role of visual analytics in healthcare whenever experts are added, the cost of bringing
and provides a comprehensive overview of state doctors or other experts into the loop makes it
of the art in the field (Preim and Lawonn 2020). It costly. Nevertheless, if there were errors, it
also covers various topics, including visualization would result in a considerable increase in the
techniques, interactive graphics, data-driven cost. In practice, and to save costs, it is feasible
models, and decision support systems. to determine what confidence level is acceptable
Games are a valuable tool in health education for the ML models involved. Confidence criteria
and can be used to educate patients about their can be lowered if wrong decisions do not have a
health, provide tools for effective health manage- negative impact, requiring less expert engagement
ment, and guide better health outcomes. They can and lowering the cost of interactive ML.
be used for patient education, physician training,
and medical scenario simulation. The benefits of
using serious games in healthcare include Discussion and Conclusion
improved knowledge retention, engagement, and
decision-making (Cain and Piascik 2015). It has Few studies have been completed on AI and
also been shown that it can educate patients about human collaboration in healthcare, and new
and improve health outcomes in the management research is being conducted in this field (Bossen
of chronic diseases such as diabetes (Talley and Pine 2022). This indicates that there are still
et al. 2019). important considerations when attempting to
incorporate ML into everyday life. More research
is required to identify the when, where, and why
Prons and Cons of I-ML on Healthcare of collaboration between humans and ML in
healthcare systems. Together, ML and humans in
In healthcare systems and society, the most critical healthcare may be incapable of making the best
tasks for I-ML are fostering trust and transpar- decisions. It has been proved that when ML incor-
ency. I-ML provides patients, clinics, hospitals, porates information from various sources, it can
specialists, and everyone interested with transpar- surpass humans in decision-making in some sce-
ency in its services. It is imperative that patients narios (Liu et al. 2019). In addition, most studies
establish reliable connections with ML systems. evaluating ML’s application in healthcare have
Human Interaction in Machine Learning (ML) for Healthcare 895

focused on improving the accuracy of medical Fails, J.A., Olsen, D.R. Jr.: Interactive machine learning.
workflows. ML is anticipated to impact healthcare In: Proceedings of the 8th International Conference on
Intelligent User Interfaces, pp. 39–45 (2003)
administration as well drastically (Bossen and Holzinger, A.: Interactive machine learning for health
Pine 2022). Given the recognized limitations of informatics: When do we need the human-in-the-
both human reasoning and ML, it is still being loop? Brain Informat. 3(2), 119–131 (2016)
determined whether doctors and ML systems can Holzinger, A., Plass, M., Holzinger, K., Cri ̧san, G.C.,
rely entirely on one another to make accurate Pintea, C.-M., Palade, V.: Towards interactive
machine learning (iml): Applying ant colony algo-
decisions. rithms to solve the traveling salesman problem with
Researchers believe that human connection is the human-in-the-loop approach. In: Buccafurri, F.,
essential for ML to succeed in gaining expert Holzinger, A., Kieseberg, P., Tjoa, A.M.,
confidence in a healthcare context. To put this in Weippl, E. (eds.) Availability, Reliability, and Security
in Information Systems, pp. 81–95. Springer, Cham
perspective, most researchers do not come from a (2016)
medical background, in striking contrast to med- Koh, H.C., Tan, G.: Data mining applications in healthcare.
ical practitioners. Interactions with medical pro- J. Healthc. Inf. Manage. 19(2), 64–72 (2005)
fessionals, whom each have their distinctive blend Kumar, P., Chauhan, R., Stephan, T., Shankar, A., Thakur,
of knowledge, experience, and expertise, have the S.: A machine learning implementation for mental H
health care. Application: Smart watch for depression
potential to improve ML systems. Following an detection. In: 2021 11th International Conference on
analysis of the relevant research, combining ML Cloud Computing, Data Science & Engineering
with human input has the potential to enhance the (Confluence), pp. 568–574. IEEE (2021)
delivery of healthcare. This entry highlights the Liu, Y., Kohlberger, T., Norouzi, M., Dahl, G.E., Smith,
J.L., Mohtashamian, A., Olson, N., Peng, L.H., Hipp,
potential reliability and precision of decisions J.D., Stumpe, M.C.: Artificial intelligence–based breast
generated by ML algorithms supported by the cancer nodal metastasis detection: Insights into the
knowledge of medical experts. black box for pathologists. Arch. Pathol. Lab. Med.
143(7), 859–868 (2019)
Maadi, M., Akbarzadeh Khorshidi, H., Aickelin, U.:
A review on human–ai interaction in machine learning
References and insights for medical applications. Int. J. Environ.
Res. Public Health. 18(4) (2021). https://doi.org/10.
Ahsan, M.M., Uddin, M.R., Farjana, M., Sakib, A.N., 3390/ijerph18042121
Momin, K.A., Luna, S.A.: Image data collection and Mukherjee, R., Kundu, A., Mukherjee, I., Gupta, D.,
implementation of deep learning based model in Tiwari, P., Khanna, A., Shorfuzzaman, M.: Iot-cloud
detecting monkeypox disease using modified vgg16. based healthcare model for covid-19 detection: An
arXivpreprint arXiv:2206.01862 (2022) enhanced k-nearest neighbour classifier based
Bossen, C., Pine, K.H.: Batman and robin in healthcare approach. Computing, 1–21 (2021)
knowledge work: Human-ai collaboration by clinical Nascimento, N., Alencar, P., Lucena, C., Cowan, D.:
documentation integrity specialists. ACM Trans. Toward human-in-the-loop collaboration between soft-
Comput.-Hum. Interact. (2022). https://doi.org/10. ware engineers and machine learning algorithms. In:
1145/3569892. Just Accepted 2018 IEEE International Conference on Big Data (Big
Budd, S., Robinson, E.C., Kainz, B.: A survey on active Data), pp. 3534–3540. IEEE (2018)
learning and human-in-the-loop deep learning for med- Nasser, A.R., Hasan, A.M., Humaidi, A.J., Alkhayyat, A.,
ical image analysis. CoRR abs/1910.02923 (2019) Alzubaidi, L., Fadhel, M.A., Santamaria, J., Duan, Y.:
Cain, J., Piascik, P.: Are serious games a good strategy for Iot and cloud computing in health-care: A new wear-
pharmacy education? Am. J. Pharm. able device and cloud-based deep learning algorithm
Educ. 79(4) (2015) for monitoring of diabetes. Electronics. 10(21),
Caruana, R., Elhawary, M., Nguyen, N., Smith, C.: Meta 2719 (2021)
clustering. In: Sixth International Conference on Data Onasanya, A., Elshakankiri, M.: Smart integrated IOT
Mining (ICDM’06), pp. 107–118. IEEE (2006) healthcare system for cancer care. Wirel. Netw. 27(6),
Chancellor, S., Lin, Z., Goodman, E., Choudhury, M.: 4297–4312 (2021)
Quantifying and predicting mental illness severity in Pradhan, K., Chawla, P.: Medical internet of things using
online pro-eating disorder communities, machine learning algorithms for lung cancer detection.
pp. 1169–1182 (2016). https://doi.org/10.1145/ J. Manage. Analytics. 7(4), 591–623 (2020)
2818048.2819973 Preim, B., Lawonn, K.: A survey of visual analytics for
Elish, M.C., Watkins, E.A.: Repairing innovation: A study public health. In: Computer Graphics Forum, vol.
of integrating AI in clinical care. Data & Society (2020) 39, pp. 543–580 (2020). Wiley Online Library
896 Human Tracking

Talley, M.H., Ogle, N., Wingo, N., Roche, C., Willig, J.:
Kaizen: Interactive gaming for diabetes patient educa- Humanness
tion. Games Health J. 8(6), 423–431 (2019)
Wang, R., DeMaria Jr., S., Goldberg, A., Katz, D.:
A systematic review of serious games in training health ▶ Uncanny Valley in Virtual Reality
care professionals. Simul. Healthc. 11(1), 41–45 (2016)
Yang, L., Li, M., Ren, J., Zuo, C., Ma, J., Kong, W.:
A human-in-the-loop method for developing machine
learning applications. In: 2019 6th International Con-
ference on Systems and Informatics (ICSAI),
pp. 492–498. IEEE (2019)
Humanoid Avatar
Zerilli, J., Knott, A., Maclaurin, J., Gavaghan, C.: Algo-
rithmic decision-making and the control problem. ▶ Deep Reinforcement Learning in Virtual
Mind. Mach. 29(4), 555–578 (2019) Environments

Human Tracking Humanoid Robot

▶ Locomotion and Human Tracking in ▶ Virtual Reality and Robotics

Healthcare Robots

Hybrid Reality
Human-Computer Interaction
▶ Substitutional Reality
▶ Cognitive Psychology Applied to User Experi-
ence in Video Games
▶ Emotion-Based 3D CG Character Behaviors
▶ Game Interface: Influence of Diegese Theory
on the User Experience Hypermedia Narrative as a
▶ Gaming Control Using BCI Tool for Serious Games
▶ Interactive Augmented Reality to Support Edu-
cation Andrés Adolfo Navarro-Newball1,
▶ Shadow Shooter: All-Around Game with e- Borja Barinaga López2 and
Yumi 3D Isidro Moreno Sánchez3
1
Electronics and Computer Science, Pontificia
Universidad Javeriana, Cali, Colombia
2
Universidad Francisco de Vitoria de Madrid,
Human–Computer Interaction Madrid, Spain
3
Universidad Complutense de Madrid,
▶ Biosensing in Interactive Art: A User-Centered Madrid, Spain
Taxonomy

Synonyms

Human-Computer Interface Interactive digital literature; Interactive multime-

dia narrative; (as a tool for) Applied game; For-
▶ Shadow Shooter: All-Around Game with e- mative game; Game with a purpose; Games for
Yumi 3D change
Hypermedia Narrative as a Tool for Serious Games 897

Definition Hypermedia narrative allows the author and

the reader to approach the creative process of a
Hypermedia refers to interactive multimedia in a new story promoting a partnership which
nonlinear medium. It may include video, plain empowers the reader as a reader-author. Within
text, audio, graphics, and animation brought this process, interactive stories, such as the ones
together using hyperlinks (links referring to used in games, are deconstructed and built again,
data). A narrative is a story or report connecting while the meaning attributed to them and the
events. It may include characters, time (duration degree of implication of the reader-authors are in
and frequency), and a space where characters continuous change. Space, time, personages, and
develop and perform actions. Hypermedia narra- actions conform the contents used by authors and
tive is a tool to develop ideas, documents, and reader-authors to construct an interactive speech
multimedia using digital tools. (Prakash and Rao 2015).
A game is an activity with specific rules related
to a physical or mental feeling of evasion of reality
and enjoyment performed by one or several Example
players who freely participate to solve a chal- H
lenge. A video game is an interactive computer- Hypermedia narrative can be used in contexts
based game using a user interface to provide feed- such as museography (museum methods of clas-
back, which includes automatic processes and sification and display). The Exploring Chimú
actions and hidden rules occurring in the com- exhibit at the Museo de América in Madrid,
puter. A serious game is a sort of game software Spain, was developed in four phases (Prakash
developed with intentions going beyond enter- and Rao 2015). The result was Exploring Chimú
tainment. Games may integrate a story, game (Fig. 1), a serious game where the player is an
mechanics (rules of the game) and dynamics, archaeologist who finds a Chimú pyramid (the
technology, and arts (visuals and sound) to create
a playability.
Hypermedia narrative as a tool for serious
games refers to using it to impact the story of a
game, affecting its mechanics and arts thorough
the use of interactive technology to produce non-
linear playability. The idea is motivating emotions
and using interactivity to make the player live
those emotions with a purpose that goes beyond
mere entertainment.

Introduction

Games favor knowledge acquisition and retention

and force players to focus on solving problems.
Their use brings benefits such as engagement,
motivation, skills development, and learning.
Serious games have been widely used and have
gained the attention of researchers because they
can capture players’ enthusiasm while raising
awareness of diverse subjects (Prakash and Rao Hypermedia Narrative as a Tool for Serious Games,
2015). Fig. 1 Exploring Chimú’s promo
898 Hypermedia Narrative as a Tool for Serious Games

Tantalluc pyramid) in the ancient Peru (Navarro- audio output and that the sensor had to be adjusted
Newball et al. 2016). The Chimú culture inhabited to support people of different heights. This phase
Northern Peru between 1000 and 1470 (Navarro- was oriented by the information and computer
Newball et al. 2016). scientists.

Phase 1: Writing the Narrative Script Phase 4: Final Implementation

Hypermedia narrative was used to identify, ana- Hypermedia narrative was used to integrate the
lyze, and validate important visual and textual serious game within the museum environment
elements, points of nonlinear interaction, and coherently. The designer developed a layout
potential technological solutions. An interdisci- including actual archaeological pieces related to
plinary team of artists, information scientists, the pieces shown within the game, thus, integrat-
archaeologists, designers, cultural managers, ing the museum’s real experience with the virtual
and computer scientists reunited to build a nar- one. The rest of the team validated the
rative script. Cultural managers coordinated the implementation.
interaction among professionals. Archaeologists
made sure that the script had historical rigor.
Artists created 3D visual models for the script Trends
and 2D models for the user interface. Designers
proposed the layout of the exhibit. Information Mobile technologies can provide portability that
scientists identified potential points of interac- goes beyond the place where the actual interactive
tion within the narrative. Computer scientists serious game has been deployed. Artificial intelli-
validated the technical feasibility of the proposal. gence can provide automated interactions, behav-
The interdisciplinary team served the interactive iors, contents, and evaluations adapted to the user
storyline in a way that the player (reader-author) or the context where the system is deployed.
feels the contents as his or her own Mixed and virtual realities can provide usable
(as coauthoring this story). and novel immersive interfaces that augment con-
tents. Technology usage guided by hypermedia
Phase 2: Developing the Interactive System narrative supports the development of better and
Hypermedia narrative was used to validate the more immersive interactive storylines,
user experience and the archaeological rigor of augmenting knowledge and making it accessible
the prototypes iteratively provided by the devel- in a noninvasive manner (Moreno Sánchez and
opers. The team found that user interaction had Navarro Newball 2016).
to be simplified. Artists developed 3D and 2D For example, Towards Tantalluc (Prakash and
models to be used in the video game, and com- Rao 2015) is a mobile application where the
puter scientists programmed it. The rest of the player is a Chimú native who builds the objects
team and likely users helped in the continuous that are going to be taken to the Tantalluc pyra-
validation of the interactivity provided by a ges- mid. Once an object is built, the player must find it
ture sensor and the interactions performed by the in the museum collection and have an augmented
players to achieve a good user experience reality experience with it, using a mobile device.
model. In this case, what mobile and mixed reality tech-
nologies do is to motivate the visitor to walk and
Phase 3: Production observe the museum exhibit.
Hypermedia narrative was used to validate the
original narrative script developed in the first
phase against the resulting serious game. It was Conclusion
an opportunity to perform the final user experi-
ence validation and to test the game’s accessibil- With hypermedia narrative, an interdisciplinary
ity. The team found that the system lacked some team where members must contribute in all phases
Hypermedia Narrative as a Tool for Serious Games 899

of the process actively considers technological ▶ User Experience

trends and advances coherently to develop serious ▶ Virtual Reality
games creating interactive storylines that augment
knowledge and make it accessible. Hypermedia
narrative usage in an educational game for a References
museum was shown. It can be used in other
kinds of serious games such as speech therapy Moreno Sánchez, I., Navarro Newball, A.A.: Mobile
hypermedia as new media and its role in transmedia
games (Navarro-Newball et al. 2014) and envi-
creation. Kepes. 13, 145–170 (2016)
ronmental consciousness (Prakash and Rao Navarro-Newball, A.A., Loaiza, D., Oviedo, C., Castillo,
2015), among others. A., Portilla, A., Linares, D., Álvarez, G.: Talking to
Teo: video game supported speech therapy. Entertain
Comput. 5(4), 401–412 (2014)
Navarro-Newball, A.A., Moreno Sánchez, I., Prakash, E.,
Cross-References Arya, A., Contreras Roldán, V.E., Quiceno Rios, V.A.,
Mejía Mena, J.D., Loaiza, D.F., Lozano, S.: Gesture
▶ 3D Modelling Through Photogrammetry in based human motion and game principles to aid under-
Cultural Heritage standing of science and cultural practices. Mult Tools H
Appl. 75, 11699–11722 (2016)
▶ Artificial Intelligence Prakash, E.C., Rao, M.: Transforming Learning and IT
▶ Augmented Reality Management through Gamification International Series
▶ Interaction on Computer Entertainment and Media Technology.
Springer, Cham (2015)
▶ Mixed Reality
I

Idle Game Image Quality Assessment

▶ Incremental Games ▶ Image Quality Evaluation of a Computer-Gen-

erated Phase Hologram

ILD
Image Quality Evaluation of a
▶ User Acoustics with Head-Related Transfer Computer-Generated Phase
Functions Hologram

Hiroshi Yoshikawa
Department Computer Engineering, College of
Science and Technology, Nihon University,
Image Captioning Funabashi, Chiba, Japan

▶ Automated Image Captioning for the Visually

Impaired
Synonyms

Image quality assessment

Image Processing

▶ Digital Images Using Heuristic AdaBoost Haar Definition

Cascade Classifier Model, Detection of Partially
Occluded Faces Quality of reconstructed image from computer-
▶ Facial Recognition and Emotion Detection in generated phase hologram is evaluated objec-
Environmental Installation and Social Media tively on its peak signal-to-noise ratio and
Applications brightness.

© Springer Nature Switzerland AG 2024

N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
902 Image Quality Evaluation of a Computer-Generated Phase Hologram

Introduction reconstructed image from the CGH. As one can

see from Fig. 1c, the reconstructed image includes
Hologram can record and reconstruct or play- the desired image appeared as same position of
back an optical wavefront on the hologram the original image and the conjugate image that
plane. It uses interference between two waves, appears as the point symmetry to the center. The
an object wave from the object to be recorded direct light (or non-diffracted light) is eliminated
and a reference wave. The interference intensity numerically in the figure, but it usually appears at
pattern is recorded on a photosensitive material. the center and should be taken into account to
Computer-generated hologram (CGH) simulates evaluate image quality. The original image should
this optical phenomenon in a computer be placed off-center not to overlap with the direct
(Lohmann and Paris 1967). CGH is widely used light and the conjugate image. Therefore, the
to show not only for 2D images but also complex original image is located center in vertical and
3D images. Image quality of the reconstructed right most side in horizontal. For the hologram
image from CGH is usually evaluated subjec- calculation, the pixel value other than the original
tively. For example, an observer compares 2D image is set to zero. The random phase is
two images and scores. Here shows basic multiplied to each pixel to make the reconstructed
research to evaluate reconstructed image quality image diffusing and bright. Then 2D Fourier
of phase-type CGH objectively on its peak transform is applied to the transmittance distribu-
signal-to-noise ration and brightness (Yoshikawa tion of o(x, y) on the input image plane, and the
and Yamaguchi 2015). result of O(X,Y) represents the complex amplitude
of the object beam on the hologram plane. If the
reference beam is collimated and its direction
Computer-Generated Hologram is perpendicular to the hologram, the complex
amplitude of the reference beam R(X,Y) can be
The Fourier hologram can be calculated with the represented as the real-valued constant r. The
Fourier transform of an original image. Figure 1a total complex amplitude on the hologram plane
shows the image location in the input image is the interference of the object and reference
plane for the Fourier transform, and Fig. 1b is a beam, represented as O(X,Y) + r. The total inten-
synthesized CGH. Figure 1c shows a numerically sity pattern

(a) 2D image location (b) Calculated CGH. (c) A simulated reconstruction

on an input image from the Fourier
image plane. hologram.

Image Quality Evaluation of a Computer-Generated Phase Hologram, Fig. 1 Image location and reconstructed
image of the Fourier transform hologram. 2D image size W ¼ H ¼ 120 and the hologram size N ¼ 256
Image Quality Evaluation of a Computer-Generated Phase Hologram 903

I ðX, Y Þ ¼ jOðX, YÞ þ r j2 Numerical Reconstruction of Phase Hologram

For reconstruction simulation, the complex ampli-
¼ jOðX, Y Þj2 þ r2 þ rOðX, Y Þ þ rO⁎ ðX, Y Þ
tude transmittance t(X,Y) of the transmission
¼ jOðX, Y Þj2 þ r2 þ 2rℜfOðX, Y Þg, phase CGH is assumed as:
ð1Þ
tðX, Y Þ ¼ exp½iDfI n ðX, Y Þ: ð4Þ
is a real physical light distribution on the hologram,
where ℜ{C} takes the real part of the complex Then t(X,Y) is inverse Fourier transformed to
number C and C⁎ means the conjugate of C. At obtain the reconstructed image. In the case of a
the right most hand of the Eq. 1, the first term sine-wave phase grating, the maximum
represents the object self-interference, and the sec- diffraction efficiency of 33.8% is obtained at
ond is the reference beam intensity. The third term Δ’ ¼ 0.59. Therefore, this value is used unless
is the interference of the object and the reference denoted.
beams and contains holographic information.
Diffraction Efficiency
Calculation Without the Object The diffraction efficiency (DE) is defined as the
Self-Interference ratio of the intensities of the reconstructed image
and the illumination light. It gives the brightness
I
In the CGH, it is quite easy to use only the inter-
ference term 2rℜ{O(X,Y)} of Eq. 1. This idea is of the reconstructed image. In the numerical
proposed at very early stage of CGH research experiments, the reconstructed image intensity
(Waters 1966). The interference part can be writ- is obtained by summing up all intensities in the
ten as: reconstructed image area as same size and posi-
tion of the original image in the input image
I b ðX, Y Þ ¼ ℜfOðX, Y Þg: ð2Þ plane.

The normalization defined in Eq. 3 is applied to Peak Signal-to-Noise Ratio

make final fringe intensity positive, The peak signal-to-noise ratio (PSNR) is defined
as the ratio of the maximum signal power and the
I b ðX, Y Þ  I min noise power. The reconstructed image is extracted
I n ðX, Y Þ ¼ , ð3Þ
I max  I min from numerically reconstructed image plane such
as Fig. 1c and normalized to 8-bit grayscale image
where Imax and Imin are the maximum and the that has same mean intensity of the original image,
minimum values of Ib(X,Y), respectively. and then the PSNR is calculated as:

Image Quality 10 35
Evaluation of a
Diffraction efficiency [%]

Computer-Generated 8 30
Phase Hologram, DE PSNR
Fig. 2 Diffraction
PSNR[dB]

6 25
efficiency and PSNR
against beam ratio for
transmission phase 4 20
hologram. Solid lines show
DE, and dashed lines 2 15
show PSNR
0 10
0 50 100 150 200
Beam ratio (|R|/|O|)2
904 Image-Based Modeling

PSNR References
2552 WH
¼ 10 log 10 W1 H1 2
½dB, Lohmann, A.W., Paris, D.P.: Binary Fraunhofer holo-
i¼0 j¼0 ½J ði, jÞ  K ði, jÞ grams, generated by computer. Appl. Opt. 6(10),
ð5Þ 1739–1748 (1967)
Waters, J.P.: Holographic image synthesis utilizing theo-
retical methods. Appl. Phys. Lett. 9(11), 405407 (1966)
where W and H are horizontal and vertical Yoshikawa, H.: Image Quality Evaluation of a Computer-
pixel numbers of the image and J and K are Generated Hologram, OSA topical meeting on Digital
intensities of the original and the Holography and 3D Imaging. Shanghai, OSA (2015)
Yoshikawa, H., Yamaguchi, T.: Image quality evaluation of
reconstructed image.
a computer-generated phase hologram. In: 10th Inter-
national Symposium on Display Holography, paper
4–4 (2015)
Numerical Experimental Results

Figure 2 shows DE (solid line) and PSNR (dashed

line) of the rigorous calculation against the beam Image-Based Modeling
ratio (BR, defined as |R|2/|O|2). The DE of the
phase hologram becomes over ten times larger ▶ 3D Game Asset Generation of Historical Archi-
than that of the amplitude hologram (Yoshikawa tecture Through Photogrammetry
2015). Since the object self-interference (OSI)
term of |O| in Eq. 1 causes noise on the
reconstructed image, it is known that larger
beam ratio gives better PSNR. However, DE Image-Guided Surgery
becomes smaller with larger beam ratio. The holo-
gram calculated without OSI as shown in Eq. 3 ▶ Augmented Reality in Image-Guided Surgery
gives PSNR of 25.0 dB with DE of 8.8%, which
achieves both low noise and bright image
simultaneously.
Imagineering Ceramic Pottery
Using Computer Graphics
Conclusion and Discussion
Sarah Dashti and Edmond Prakash
Image quality of phase CGH is evaluated objec- Cardiff School of Technologies, Cardiff
tively on the diffraction efficiency and the peak Metropolitan University, Cardiff, UK
signal-to-noise ratio. For the transmission
phase hologram, although it is obtained over
ten times of diffraction efficiency against Synonyms
amplitude hologram (DE ¼ 0.77%, PSNR ¼
38.9 dB), PSNR is not as good as that of the Computer-aided design (CAD); Computer-aided
amplitude hologram. Since the evaluated holo- industrial design (CAID); Three/Two dimensional
gram is very simple phase hologram, it is (3D/2D) graphics; Virtual pottery (VP); Virtual/
expected to evaluate other type of phase Augmented reality (VR/AR)
hologram.

Definition
Cross-References
Imagineering in Virtual Pottery is to realize the
▶ Holography as an Architectural Decoration artist’s dream of creating intricate shapes.
Imagineering Ceramic Pottery Using Computer Graphics 905

Imagineering at the next level seeks realistic inter- VR/AR painting and pottery making is the inno-
action with the shape digitally using devices that vative beginning of advanced 3D modeling in art
provide the touch and feel of real pottery in the practice, opening the imagination to conquer real-
virtual world. To push the boundary further in istic visual and physical real-world reality, to pro-
imagineering, the artist wants the virtual clay to duce, share, and enhance skills. Clay based
be malleable and behave like real clay in capturing modeling is an ongoing trend for developers of
intricate shapes, and also it is printable. Ideally, 3D/AR/VR modeling to explore, expand, and
imagineering in virtual pottery enables and develop. This method brings realistic deforma-
demands bidirectional fluidity. tion, representing real objects transformation, to
the human visual concept that augments a more
genuine and real visual appearance. Clay 3D
Introduction modeling is a method that integrates digital defor-
mation as simulation upon real geometric and
Traditional pottery relies on the hand skills of an physics data. It represents a 3D surface mesh
artist, the ceramic material and an understanding sculpting with tools to push, pull, twist, inflate,
of the kiln and making process. This is further surface relief, as well as a voxel-based geometry
refined by a range of decorating, glazing, and method to add/subtract.
I
firing techniques. The artist also uses techniques Imagineering using computer graphics has not
such as hand forming, throwing, and slip casting, only enabled artists to visually interact with
among others. This article highlights how all of the creation of artistic objects, but also provide
these techniques can be brought to the virtual the essential link for fabrication to physicalize the
world. Digital creative technologies are gaining artistic creations. This on the surface looks feasi-
maturity and now enable artists to reach greater ble but needs to grow in maturity to process and
heights. An example of imagineering that has transform the visual form to physical form. This
come to fruition and fundamental to this article requires a detailed understanding of the processes
is sound shape modeling. Sound creates several in manufacturing and even more detailed under-
detailed shapes when it interacts with clay and standing of the materials and their properties for
fluid. The artist would like to model those intricate ceramic fabrication.
shapes and patterns that are extremely difficult to
create using the artist’s hands.
This article focuses on virtual pottery model- Background
ing. The approach seamlessly captures the shape,
interaction, and making in the artistic pottery pro- Virtual Deformable Shape Modeling in
cess, albeit using digital creative tools. Computer Fine Art
graphics modeling over the past decades has Virtual reality modeling is one of the cutting-edge
shown a high demand for sophisticated realism creative technologies that support deformable
of geometric shapes. Graphical shape modeling shape modeling, using physical interaction data
has found pervasive use in various fields, e.g., to enhance objects by color, depth, and deforma-
Arts, Medicine, Engineering, and many other tion. One of the examples is volumetric modeling
fields. Digital technologies have also revealed a introduced by Kim et al. (2018). The authors
new path for contemporary art and creation. demonstrated a high-resolution method of volu-
Today arts, science, and technology are united metric painting in virtual reality with high depth
by a bridge of creativity, allowing cross- complexity. The technique consists of digital
disciplinary fields to meet. painting on 2D surfaces extended into 3D volu-
The impact of this bridge has created opportu- metric painting.
nities and challenges for a new innovative Ioana also examined another method using Tilt
approach, enhancing and promoting existing Brush. It is a VR painting application produced by
experiences and outputs (Krasteva 2016). Google as collaboration between artists and
906 Imagineering Ceramic Pottery Using Computer Graphics

scientists, enhancing creative practice opportunity because of the inevitable and expensive 3D defor-
for users and their 3D artwork created VR space mation computation. In their paper, the authors
with sharing creations online. More so, turning the proposed a technique that extends the conven-
3D VR object into a holographic format for the tional rigid approach using geometry images.
wider public (Pioaru 2017). Their approach flattens the geometry and also
An exploration on virtual clay modeling and helps to accomplish deformation effectively and
tools was presented in the literature (Sener et al. efficiently. They demonstrated that their method is
2002). The authors investigated attempts offered suitable for haptics computing to perform the
on using haptics for force-feedback that was used deformation on the geometry map to avoid expen-
by Gribnau (1999). They also suggested using sive 3D deformation computation. They presented
CAD tools to generate 3D objects using PHAN- the construction of the deformable geometry map
ToM haptic device’s human-computer interface representation and its application utilizing practi-
technology for users to touch and manipulate vir- cal methods for interactive surgery simulation and
tual objects, considering free-form for a better interactive textile simulation (Liu et al. 2007).
development process. Sener’s approach gave a This parametric approach has potential for the
solid ground to develop the concept of VR clay proposed Virtual Pottery application.
modeling and prototyping. This approach
supported the idea of integrating the system and Deformable 3D Shapes in Facial Animation
methods to establish a VP novel system by Dashti Zhang et al. introduced a method of exploring the
et al. (2020a). The integrated 3D graphics toolkit real natural deformation of a human face, using
software system showed new opportunities and animated representation. The system relies on a
challenges to enhance computer graphics and tra- physically based 3D facial model-based with ana-
ditional pottery-making fields. tomical knowledge. The approach involves a
dynamic, non-linear multi-layered skin model
Why Deformable Shape Modeling? where the skin is built as a mass-spring-damper
Deformable shape modeling is a reliable, func- (MSD) facial model (Zhang et al. 2004). In Chen
tional method for computer graphics, modeling a and Prakash’s face simulation (Chen and Prakash
realistic volumetric geometric texture. This 2006), the authors use an animation system for a
approach provides more accurate, pragmatic var- personalized human head. The deformation from
iable data of deformational behavior. The tech- the template model to the target head is through
nique produces volumetric deformation of, e.g., adaptation. Both general Radial Basis Function
height, depth, twist, and bend for medical scien- (RBF) and Compactly Supported Radial Basis
tists, artists, and engineers to enhance their per- Function (CSRBF) are applied to ensure the fidel-
formance and outputs. ity of the global shape and face features. Anima-
tion factor is also adapted so that the deformed
Challenges in 3D Graphics Shape Modeling model still can be considered as an animated head.
Computer graphics modeling is considered Situations with insufficient scanned data are also
expensive, as well as labor and compute- discussed in their paper. A related approach by
intensive. 3D modeling tools mainly creates Navarro-Newball et al. (2011) builds up a face
rigid object surface with texture deformation to using anatomy guided bottom up creature skin-
offer limited visual appearance of a detailed ning. These approaches have the potential for
representation. multilayered shape modeling for virtual pottery.

Deformable Shapes in Surgery Simulation Deformable 3D Shapes in Artistic Simulation

The approach for surgery simulation is an Dashti et al. presented a demonstration of a novel
advanced deformation system that uses a paramet- method using off-the-shelf tools to discover
ric model by Liu et al. for Surgery Simulation. unique texture for VP interactive modeling, a
Haptics on 3D deformable models is a challenge novel approach for virtual pottery, 3D modeling
Imagineering Ceramic Pottery Using Computer Graphics 907

Imagineering Ceramic Pottery Using Computer Graphics, Fig. 1 Imagineering the virtual pottery workflow

with augmented reality interaction technique for Figure 1 shows the different elements of
materializing deformable shapes sound-resonance imagineering for the virtual pottery system and
on 3D objects. The technical framework provided captures the relation between the various aspects.
a new method using simple processes to perform Figure 2 shows an example of an object that has
complex object transformations for virtual, 3D undergone the imagineering process and realized
modeling and augmented reality interaction. in a physical form.
Clay-based VR modeling uses the concept of This approach augments realistic shape and
deformable shape modeling to extend physical surface transformations, using 3D surface mesh
ceramics. It captures the visual and physical rep- sculpting with voxel-based geometry. The volu-
resentation deformation of actual ceramic making metric 3D Sound-Structure Texture Modeling
(Dashti et al. 2020b). Clay as material presents process first includes materializing sound reso-
some challenges of preserving real-world con- nance from fractal images, using Chladni plate
straints, such as gravity and evaporation, captur- software with bump and displacement mapping.
ing the artist’s creative physicalized imagination. Next, volumetric deformable shape maps are
908 Imagineering Ceramic Pottery Using Computer Graphics

Cross-References

▶ 3D Avatars in Virtual Reality Experience

▶ Artistic Data Visualization in the Making
▶ The New Age of Procedural Texturing
▶ UV Map Generation on Triangular Mesh

References

Chen, C., Prakash, E.: Adaptive processing of range

scanned head: synthesis of personalized animated
human face representation with multiple-level radial
basis function. EURASIP J. Adv. Signal Process.
2007, 1–16 (2006)
Dashti, S., Prakash, E., Hussain, F., Carroll, F.: Digital
pottery: novel information systems and workflow
process for virtual and physical artistic visualiza-
tions of sound on ceramics. Adv. Manag. Innov.
23 (2020a)
Dashti, S., Prakash, E., Hussain, F., Carroll, F.: Virtual
pottery: deformable sound shape modelling and fabri-
cation. In: 2020 International Conference on Cyber-
worlds (CW), pp. 133–136. IEEE (2020b)
Gribnau, M. W. (1999). Two-handed interaction in com-
Imagineering Ceramic Pottery Using Computer puter supported 3D conceptual modeling. PhD Thesis,
Graphics, Fig. 2 Physicalized output from the virtual TU Delft, ISBN 90-9013038-1, http://resolver.tudelft.
pottery imagineering system nl/uuid:179ba8d0-8384-49ce-ba06-7e4132e2d4bb
Kim, Y., Kim, B., Kim, Y.J.: Dynamic deep octree for high-
blended on the surface of the VR object to com- resolution volumetric painting in virtual reality. In:
bine intricate sound resonance patterns. The pro- Computer Graphics Forum, vol. 37, pp. 179–190.
Wiley Online Library (2018)
cedure is then extended to transform the output of Krasteva, M.: The impact of technology on the modern art.
complex 3D shape models from the above steps Digit. Present. Preserv. Cult. Sci. Herit. VI,
for rapid prototyping using appropriate pre-print 247–253 (2016)
tools by remeshing and physics manipulations. Liu, Q., Prakash, E., Srinivasan, M.A.: Interactive deform-
able geometry maps. Vis. Comput. 23(2),
119–131 (2007)
Newball, A.A.N., Botero, F.J.H., Buitrago, D.F.L.: Anat-
Conclusion omy guided bottom up creature skinning. Sist. Telemát.
9(17), 9–21 (2011)
Pioaru, I.: Visualizing virtual reality imagery through dig-
The time has come for Imagineering Ceramics ital holography. In: 2017 International Conference on
Pottery Using Computer Graphics. This work Cyberworlds (CW), pp. 241–244. IEEE (2017)
has highlighted the imagineering map with sev- Sener, B., Wormald, P., Campbell, I.: Towards’ virtual
eral elements seamlessly integrated. Challenges clay’ modelling-challenges and recommendations: a
brief summary of the literature. In: DS 30: Proceed-
still remain where every element can be expanded ings of DESIGN 2002, the 7th International Design
with more realistic features. One specific area of Conference, D. Marjanovic (Editor), ISBN: 953-
immediate focus in this work is Imagineeramics, 6313-45-6, Dubrovnik, The Design Society. pp.
to look at the full functionality of imagineering 545–550 (2002).
Zhang, Y., Prakash, E., Sung, E.: A new physical model
with ceramic material properties. It is all in the with multilayer architecture for facial expression ani-
imagination of the artist and computer scientist to mation using dynamic adaptive mesh. IEEE Trans. Vis.
push the boundary forward. Comput. Graph. 10(3), 339–352 (2004)
Immersive Auralization Using Headphones 909

and augmented reality scenarios where users

Immersion directly compare their real life sensory experience
with realistic multimodal synthetic stimuli and
▶ Game Design and Emotions: Analysis Models interactions.
▶ Mixed Reality and Immersive Data Visualiza-
tion
▶ Videogame Engagement: Psychological Introduction
Frameworks
▶ Virtual Reality System Fidelity In order to deliver natural auditory experiences
in virtual/augmented reality with headphones,
sound pressure level (SPL) of the synthesized
sound field has to be delivered at user eardrums
Immersive coherently with their natural listening experience
(see Fig. 1 for a general schematic view). In
▶ Virtual Reality as New Media particular, auralization algorithms should rely
on a well calibrated audio equipment and indi-
vidually equalized headphones (see “▶ Sound
I
Spatialization”).
Immersive Auralization Using Sound transmission from headphone to ear-
Headphones drum can be represented through an analogue
circuit model (Møller 1992) with the ideal goal
Michele Geronazzo to obtain:
Department of Architecture, Design, and Media
Technology, Aalborg University, København, Z headphone  Z radiation , ð1Þ
Denmark
where Zradiation denotes the equivalent acoustic
impedance outside the ear canal in free-field lis-
Synonyms tening conditions, and Zheadphone the equivalent
impedance outside the ear canal with head-
Earphones; Headphone; Headphone impulse phones. This equation can be interpreted as fol-
response; Headphone transfer function; Head- low: the opposition that headphone present to the
phones; Headset acoustic flow, i.e., playback audio signals inside
ear-canals, should be comparable to natural lis-
tening conditions without headphones. The
Definitions validity of such model is restricted to wave-
lengths greater than the ear canal’s width, i.e.,
Headphones are electro-acoustic transducers approximately under 10 kHz, leading us to define
able to convert two electric output channels into the so-called Pressure Division Ratio (PDR) as
two dichotic acoustic signals at user’s ears, pref- the ratio between the pressure divisions of two
erably binaural signals. The less distortion and situations:
flat response headphones hold the more control
of sound parameters and auralization accuracy • Open/blocked ear in a natural listening condi-
game developers and designers can obtain. tions with an external sound source in anechoic
Headphone-induced spectral coloration and space
distortion is a key factor in music listening but • Open/blocked ear when the sound source is the
even more critical in future immersive virtual headphone driver
910 Immersive Auralization Using Headphones

Immersive Auralization
Using Headphones, Binaural Room Impulse Response (BRIR)
Fig. 1 High-level acoustic
components for VR Spatial Room Impulse Head-related Impulse
auralization with focus on Response (SRIR) Response (HRIR)
headphone interaction

Room Acoustics Headphones Listener’s body

Headphone Impulse
Response (HpIR)

One can formally define the PDR as Sources of Variance in Binaural Signals
follow:
Audible artifacts and unnatural spectral coloration
Popen PHp
open
are likely to occur in the reproduction of binaural
¼ Hp , ð2Þ signals with headphones. In Brinkmann et al.
Pblocked Pblocked
(2017), a summary of main sources of error was
reported taking into consideration authenticity of
where Popen and Pblocked stand for the free field
audio rendering, i.e., the perceptual identity with a
sound pressure at the entrance of the open- and
real acoustic event (Blauert 1983). In Table 1,
blocked-ear canal, respectively, while PHp open and
Hp only key factors and average acoustic errors
Pblocked indicate the same sound pressure obser-
related to headphones are reported.
vation points when the sound source is a head- It has to be noted that the major sources of error
phone. Figure 2b depicts an example of PDR are headphone repositioning and presence. The
computation for 18 users. Headphones with first source is closely related to headphone form
PDR ≈ 1 satisfy the free-air equivalent coupling factors and robustness to movements, while the
(FEC) criterion (Møller 1992) where the acous- second mainly affects comparisons between nat-
tic load to the ear canal is equal to the radiation ural listening experience and binaural audio ren-
impedance of ears without headphones. Head- dering with headphones in tests on authenticity
phones act as an acoustic cavity that introduces where participants are usually asked to evaluate
a constant level variation at low frequencies, both conditions without removing and putting
i.e., ≈ 4 kHz, with few inter-subject variability. headphones on.
On the other hand, headphone position and The mix of standing waves that start to grow
user’s external ear anthropometry introduce inside headphone cups, with outer ear’s reso-
frequency notches in the higher spectrum. nances, results in an individual characterization
Headphone- and user-specific acoustic interfer- of headphone acoustics. Headphone form factor
ences are difficult to predict and compensate in (i.e., circum-aural, supra-aural, extra-aural,
order to have a robust flat headphone response earbuds, and inserted) heavily influences all
for auralization. these sources of variation. Headphone acoustic
Immersive Auralization Using Headphones 911

Immersive Auralization a 110

Using Headphones,
Fig. 2 Example of 100
headphone transfer function
for SENNHEISER
90
HDA200 supra-aural
headphones over a
20 human heads; data are 80
taken from BT-DEI HPIR
database, part of the 70

Magnitude [dB]
PHOnA archive (Boren
et al. 2014). (a) Average 60
magnitudes considering all
18 individual HpTF sets 50
with reference calibration
90 dbSPL at 1 Khz (dashed 40
blue line and  30 dbSPL
shifted dashed red line are
30
left and right channels,
respectively), (b) variability
20
on pressure division ratio
across human heads
Left
Right
I
10

0
0

00

00

1 0 00
0

0
10

00

00
10

50

80

15
Frequency [Hz]
b 40

30

20
Pressure divsion ratio

10

-10

-20

-30
0

00

00

0
10

00

00
10

50

10

15

Frequency [Hz]

contribution can be described by headphone transfer functions (HpTFs). HpIRs are typically
impulse response (HpIR) measurements or their measured on an (i) artificial ear (e.g., B&K 4153),
frequency domain counterpart, i.e., headphone (ii) a dummy head (e.g., KEMAR mannequin
912 Immersive Auralization Using Headphones

Immersive Auralization Using Headphones, sound-field; however, external sound sources eas-
Table 1 Sources of errors and variance due to headphones ily interfere with auralization, and head move-
for binaural reproduction
ments could cause large variations in placement
Typical Maximum of suspended headphones, thus leading their use
error (dB) error (dB)
for only research purposes. On the other hand, the
Headphone repositioning 5 20
scientific literature suggests that circum-aural
Acoustic headphone load 4 10
Sennheiser HD600 is usually adopted as a de
Headphone presence 10 25
Headphone 1 10
facto standard for a broad variety of psycho-
compensation acoustic studies on binaural reproduction. This
headphone model fulfills FEC criterion; however,
it do not allow isolation from external sound
sources. In order to overcome the isolation issue,
(Gardner and Martin 1995)), or (iii) human lis-
in-ear headphones seem to introduce smaller
teners. Since, the first two measurement methods
intra-subject variability (less inclusion of pinna
do not consider inter-subject and intra-subject
contribution) once the quality of the sealing is
variability, individual recordings should be pref-
high (Olive et al. 2017), at the cost of a nontrivial
erable in order to take into account multiple posi-
compensation for ear occlusion.
tionings of headphones and to fine tune
headphone compensation algorithms for a specific
user. However, this latter approach is not practi-
Conclusion
cable for a wide population of users; accordingly,
headphone equalization algorithms usually rely
Auralization relies on the amount of individuali-
on average HpIR responses for compensation
zation in the headphone correction of both mea-
which is based on available measurements for a
surement techniques (e.g., average, generic, and
specific headphones on a group of users thus
individual HpIRs) and equalization methods with
accounting only for headphone contribution (see
emphasis on high-frequency control in the inverse
Fig. 2a for an example); moreover, regularization
filtering problem (Boren et al. 2015).
methods act on high-frequency gain in order to
In-situ individual calibration for a transpar-
further level out user-dependent variations
ent headphone response is a challenging
(Schärer and Lindau 2009).
research issue with no straightforward proce-
It has to be noted that virtual and augmented
dures, especially for inserted earphones that do
reality applications require different headphone
not satisfy FEC criterion. Novel technological
levels of isolation or transparency making the
advances are contributing to the integration of
choice of a proper headphone design and equali-
headphones in even more smart headset, intro-
zation algorithm critical. Since noise-canceling
ducing a binaural earphone-plus-microphones
headphones can be considered an extreme exam-
system which will be able to extract the ear-
ple of acoustic isolation from the real world, hear-
canal transfer function (ECTF) in real-time and
trough devices require all the external acoustic
to perform an adaptive inverse filtering able to
information to be collected and processed
estimate sound pressure of an occluded ear canal
together with virtual sound sources (Valimaki
(Denk et al. 2017).
et al. 2015).
A high level of authenticity for a virtual audi-
tory display can be achieved using extra-aural
Cross-References
headphones where the acoustic coupling between
listener and headphones is minimized (Romigh
▶ Overview of Virtual Ambisonic Systems
et al. 2015), thus resulting in less influence of
▶ Sonic Interactions in Virtual Environments
the playback device on a recorded/synthesized
▶ Sound Spatialization
Immersive Technologies 913

▶ Spatial Perception in Virtual Environments Valimaki, V., Franck, A., Ramo, J., Gamper, H., Savioja,
▶ Training Spatial Skills with Virtual Reality and L.: Assisted listening using a headset: enhancing audio
perception in real, augmented, and virtual environ-
Augmented Reality ments. IEEE Signal Process. Mag. 32(2), 92–99
▶ User Acoustics with Head-Related Transfer (2015). https://doi.org/10.1109/MSP.2014.2369191
Functions

References
Immersive Design
Blauert, J.: Spatial Hearing: The Psychophysics of Human
Sound Localization. MIT Press, Cambridge, MA
(1983) ▶ Foundations of Interaction in the Virtual Real-
Boren, B.B., Geronazzo, M., Majdak, P., Choueiri, E.: ity Medium
PHOnA: a public dataset of measured headphone trans-
fer functions. In: Proceedings of 137th Convention
Audio Engineering Society. Audio Engineering Soci-
ety (2014). http://www.aes.org/e-lib/browse.cfm?
elib¼17449 Immersive Environments
Boren, B., Geronazzo, M., Brinkmann, F., Choueiri, E.:
Coloration metrics for headphone equalization.
I
In: Proceedings of the 21st International Conference ▶ Virtual Reality Systems, Tools, and Frameworks
on Auditory Display (ICAD 2015), pp. 29–34, Graz
(2015)
Brinkmann, F., Lindau, A., Weinzierl, S.: On the authen-
ticity of individual dynamic binaural synthesis.
J. Acoust. Soc. Am. 142(4), 1784–1795 (2017). Immersive Storytelling
https://doi.org/10.1121/1.5005606. http://asa.scitation.
org/doi/10.1121/1.5005606
Denk, F., Hiipakka, M., Kollmeier, B., Ernst, S.M.A.: An ▶ Storytelling in Virtual Reality
individualised acoustically transparent earpiece for
hearing devices. Int. J. Audiol. 1–9 (2017). https://doi.
org/10.1080/14992027.2017.1294768
Gardner, W.G., Martin, K.D.: HRTF measurements of a
KEMAR. J. Acoust. Soc. Am. 97(6), 3907–3908 Immersive Systems
(1995)
Møller, H.: Fundamentals of binaural technology. Appl. ▶ Foundations of Interaction in the Virtual Real-
Acoust. 36(3–4), 171–218 (1992). https://doi.org/10. ity Medium
1016/0003-682X(92)90046-U. http://www.science
direct.com/science/article/pii/0003682X9290046U
Olive, S., Welti, T., Khonsaripour, O.: A Statistical Model
that Predicts Listeners’ Preference Ratings of In-Ear
Headphones: Part 1 – Listening Test Results and Immersive Tech
Acoustic Measurements. Audio Engineering Society
(2017). http://www.aes.org/e-lib/browse.cfm?elib¼19
237 ▶ Immersive Technologies for Medical Education
Romigh, G.D., Brungart, D.S., Simpson, B.D.: Free-
field localization performance with a head-tracked
virtual auditory display. IEEE J. Selected Topics
Signal Process. 9(5), 943–954 (2015). https://doi.
org/10.1109/JSTSP.2015.2421874. http://ieeex Immersive Technologies
plore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber
¼7083725 ▶ 3D Avatars in Virtual Reality Experience
Schärer, Z., Lindau, A.: Evaluation of equalization
methods for binaural signals. In: Audio Engineering ▶ Everyday Virtual Reality
Society Convention 126 (2009). http://www.aes.org/e- ▶ Immersive Visualizations Using Augmented
lib/browse.cfm?elib¼14917 Reality and Virtual Reality
914 Immersive Technologies for Accessible User Experiences

Furthermore, there are several high-end and

Immersive Technologies for industry-grade immersive technologies, for exam-
Accessible User Experiences ple, Mixed Reality (MR) devices such as the
Microsoft HoloLens 2 offer holographic projec-
Alvaro Uribe-Quevedo tions blended into the real environment, which are
Software Informatics Research Centre, University currently used in medical, defense, and automo-
of Ontario Institute of Technology, Oshawa, tive applications (Speicher et al. 2019). Although
Canada VR, AR, and MR can provide highly immersive
and engaging user interactions, these technologies
typically offer isolated experiences, and recent
Synonyms trends favor social symmetric and asymmetric
experiences (i.e., those were users share the
Extended reality; Usability; User experience experience using the same technology or different
devices, respectively) (Jeong et al. 2020). The
interconnectivity of these immersive technologies
Definition is allowing cross-platform interactions, where VR
users can interact with those using AR or MR thus
Immersive technologies encompass hardware and producing eXtended Reality (XR) experiences
software that provide sensory feedback in the (Skarbez et al. 2021).
form of visual, auditory, haptic, or olfactory cues Immersive technologies interconnect human
employing diverse human interface devices. factors within interaction loops. The users will
Coupled with human-centered design considering process the perceptual information to make deci-
varying human factors associated with perception, sions and interact with the virtual environment
cognition, and physical ergonomics, these tech- while receiving sensory feedback as shown in
nologies aim to provide accessible user experi- Fig. 1. Depending on the experience (e.g., leisure,
ences. Although accessibility is typically entertainment, training, or education), actions or
associated with removing barriers for those information may be stored in either the short- or
experiencing disabilities, usability should be long-term memory. Within the interaction cycle,
used instead because its purpose is the ease of attention influences how the users gather spatio-
use regardless of the end-user. temporal information. Attention is required for the
orientation of sensory events and detection of
signals for processing, which has been predomi-
Introduction nantly achieved through visual stimuli (LaViola
et al. 2017).
Immersive technologies enable cross-sensory The perceived visual, auditory, and haptic stim-
experiences that blur the frontier between virtual uli allow users to act and execute tasks through a
and real worlds (Suh and Prophet 2018). The combination, substitution, or translation of sensory
cross-sensory stimuli are also referred to as mul- cues. This is required in order to compensate for the
timodal or cross-modal because of the use, com- limitations of immersive technologies with respect
bination, integration, or translation of sensory to providing realistic perceptual information. For
cues into diverse installations (i.e., artistic, educa- example, the lack of visual feedback for localiza-
tional, training, or health care, among others), tion tasks can be replaced by auditory cues
which affect the user experience (Kapralos et al. (Massiceti et al. 2019), and the combination of
2017). Currently, Virtual and Augmented Reality, haptic and auditory cues can further enrich locali-
VR and AR, respectively, are becoming ubiqui- zation skills for those with vision impairment
tous with off-the-shelf products including mobile (Reynal et al. 2019). The cross-sensory cues enable
devices, video game-handheld devices, and mental processes associated with thinking, remem-
headsets, which are becoming readily available. bering, and decision-making, also referred to as
Immersive Technologies for Accessible User Experiences 915

Immersive Technologies for Accessible User Experiences, Fig. 1 Interaction loop

cognition. The cognition process can be further through biconvex lenses. Figure 2a presents the
I
understood by analyzing in-experience metrics Oculus Quest 2, a VR headset featuring inside-out
such as time to completion, task completion, and tracking for seated, standing, and room-scale
information from physiological sensors including VR. The ability to customize the interactive vol-
eye tracking, heart rate, brain activity, muscle activ- ume allows using VR as a portable solution for
ity, and skin responses (Fralish et al. 2018). travel training for people with intellectual disabil-
A meta-analysis on AR in education highlighted ities (Checa et al. 2019). Figure 2b presents the
the importance of the acquisition of social, living, Aryzon AR headset, which uses an array of mir-
learning, and physical skills for users with cogni- rors that create the effect of holography, and
tive disabilities (Baragash et al. 2020). A recent unlike traditional AR, the Aryzon headset is
study into VR adoption in K-12 for students with hands-free. AR is the immersive technology
disabilities revealed the prevalent use of non- which is the most widely available to users as it
immersive screen-based interventions that do not can be used with most medium-range mobile
take advantage of the immersive and interactive devices, including phones and tablets. AR has
capabilities of immersive VR (Carreon et al. been used to facilitate wheelchair indoor naviga-
2020). The current availability of consumer-level tion by informing users about obstacles and haz-
XR technologies has sparked interest in developing ards in planning safe routes (De Oliveira et al.
accessible user experiences through customized 2017). Additionally, AR has been used to help
interactions employing speech recognition those with dyslexic impairment to decrease the
(Bryant et al. 2020), body tracking (Shao et al. overall time required for reading by enabling
2020), gaze tracking (Saha et al. 2019), and text customization adjustments, including back-
custom-made human interface devices (Mirzaei ground contrast among others (Gupta et al. 2019).
et al. 2020). Finally, Fig. 2c presents the Microsoft HoloLens,
a headset that overlays virtual content in the real
world by mapping the environment, thus creating
Visual Perception Technologies holographic visual feedback. User interactions
occur through remote control, hand tracking,
Predominantly, the visual domain is the most used voice recognition, and eye tracking.
in immersive technologies and has seen the most Within the field of visual immersion, holo-
advances. Immersive visual cues rely on the graphic displays are gaining momentum due to
capacity to perceive depth, which is achieved by their capacity of providing visualizations with
VR headsets employing stereoscopic views seen depth to multiple users without requiring a
916 Immersive Technologies for Accessible User Experiences

Immersive Technologies
for Accessible User
Experiences,
Fig. 2 Head-mounted
displays for immersive
technologies

(a) Oculus Quest (b) Aryzon AR

(c) Microsoft HoloLens
VR HMD headset

Immersive Technologies for Accessible User Experiences, Fig. 3 Looking Glass holographic display of a living
room for reminiscence therapy purposes (Tabafunda et al. 2020)

headset. Nonimmersive displays enable users who the cherry blossom when moving from left to right.
cannot use a headset to experience the virtual Similar motion parallax effects can be achieved
content. Figure 3 presents two views from a living employing depth sensors such as the Azure Kinect
room scene rendered on the Looking Glass holo- for providing more realistic telepresence human
graphic display for reminiscence therapy, an inter- interactions (Tölgyessy et al. 2021).
vention that helps individuals with dementia to
recollect memories from their past (Tabafunda
et al. 2020). Visual Tracking Technologies
The use of holographic displays and tethered
VR and MR HMDs requires a VR-ready computer, Measuring and driving user interactions
which makes this solution accessible to few users. employing eye tracking for understanding
A consumer-level solution to this problem is using human cognition in immersive technologies is a
nonimmersive VR where users navigate the envi- relatively novel field. Eye tracking allows captur-
ronments through a regular screen. The addition of ing information associated with gaze, regions of
motion parallax effects concerning the user’s interest, and attention span (Clay et al. 2019).
head position and orientation can be used with Recently, advances in eye tracking have led to
regular screens to provide depth perception. the use of foveated rendering for improving atten-
Figure 4 presents the nonimmersive VR motion tion and rendering optimization by increasing the
parallax implemented with a web camera, the visual fidelity of the areas of interest where the
FaceTrackNoir library, and the Unity game engine. gaze is attending (Matthews et al. 2020).
Figure 4 shows how the foreground tree occludes Concerning accessibility, Masnadi et al. (2020)
Immersive Technologies for Accessible User Experiences 917

developed an eye-tracking assistance tool that sound from different directions when navigating
projects corrections designed to help people with the virtual environment, which can be used as
visual impairments. travel aids for visually impaired users (Spagnol
Currently, off-the-shelf VR devices offering et al. 2018). Spatial audio can also be used to
eye-tracking capabilities include the HTC Vive increase empathy and awareness concerning
Pro Eye and Pico Neo 2, both featuring Tobii activities performed without visual cues
eye-tracking technology. Pupil Labs also offers (Guarese et al. 2021). For example, Cowan et al.
eye-tracking add-ons that can be attached to var- (2020) developed a framework for analyzing the
ious immersive technology hardware including effects of sound rendering in a virtual environ-
Microsoft HoloLens 1 and eyewear for AR. In ment, where participants navigated a virtual world
contrast to the Microsoft HoloLens 1, the Micro- through the localization of audio sources, con-
soft HoloLens 2 integrates eye-tracking technol- cluding that improvements in spatial audio ren-
ogy without requiring additional hardware. dering do increase task performance.
Figure 5 presents a view of the HTC Vive Pro Except for mobile AR, current VR, and MR,
Eye and Pico Neo 2. HMDs feature spatial audio through embedded
speakers either on the headband (e.g., Oculus
Quest, Oculus Rift S, and Microsoft HoloLens),
I
Auditory Perception Technologies or ear speakers hovering on top of the ear (e.g.,
Valve Index). As a result of advances in binaural
Second to vision, auditory cues provide percep- rendering and sound-based physics, spatial audio
tion of direction and distance from the audio provides more accurate auditory representations
source. Spatial audio applies directional audio than stereo or surround sound, which is limited to
filters by adjusting the frequencies and delays of environmental recordings on multiple channels.
what is heard by each ear (Tashev and Gamper Because of these properties, spatial audio can be
2017). The output causes the sensation of hearing used as a substitute for visual cues to help users

Immersive Technologies
for Accessible User
Experiences,
Fig. 4 Nonimmersive VR
motion parallax moving
from left to right

(a) Left (b) Center (c) Right

(a) HTC Vive Pro eye VR HMD (b) Pico Neo 2 VR HMD

Immersive Technologies for Accessible User Experiences, Fig. 5 Lens- and eye-tracking view for the HTC Vive
Pro Eye and Pico Neo 2
918 Immersive Technologies for Accessible User Experiences

with various degrees of sight loss to navigate role in immersion and presence as it enhances
virtual environments, such as a virtual conference perception to the point that it can be used to
venue (Robern et al. 2021). Research in this field replace visual and auditory feedback. Sorgini
has produced methods for mapping visual naviga- et al. (2018) conducted a literature review on
tion to auditory cues employing echolocation and haptic technologies for auditory and visual sen-
distance-dependent hum volume modulation sory disabilities, where it highlights the impor-
(Massiceti et al. 2019). tance of sensory substitution employing
consumer-level technologies to facilitate interac-
tions with nondisabled peers.
Speech Recognition Technologies Vibrotactile feedback is the most commonly
used technology due to the miniaturization of
Speech recognition plays an important role in actuators, its low-cost, and the seamless integra-
immersive interactions as it allows to create tion inside VR controllers, gamepads, clothing,
more natural user-avatar interactions by facilitat- and custom-made 3D-printed user interfaces. For
ing speech-to-text, text-to-speech, and enabling example, virtual environments can be coupled
automated dialogues driven by emotion recogni- with a walking cane to provide auditory and hap-
tion, physical interactions, and physiological tic cues to enable training and increase awareness
measures (Cinieri et al. 2020). A review when navigating unknown locations (Zhao et al.
conducted by Nassif et al. (2019) identified the 2018). Kim et al. (2021) developed a human-
prominent use of machine learning in English display interface with vibrotactile feedback for
settings determining efficiency through word- assistive applications such as devices relying on
error-rate, concluding the need for employing touch screens found in vehicles, wheelchairs, and
recurrent neural networks to yield better results. public locations such as airports, hotels, and shop-
For example, Teófilo et al. (2018) developed a ping malls. Vibrotactile feedback is limited to
system that employs language to assist deaf and cues associated with vibrations, taps, pressure,
hard of hearing users to improve speech under- and pinching feedback through skin deformation.
standing by supporting sentence prediction and This limitation of vibrotactile feedback impedes
spelling correction. Artificial Intelligence the capture of Kinesthetic cues that allow deter-
(AI) has been renewed in speech recognition mining the physical relationship with objects.
thanks to the advances in smart assistants in Force feedback haptic devices, on the other
mobile phones and smart assistants as it allows hand, require mechanisms that transfer mechani-
for creating natural user interactions to help users cal movement to the user through actuators that
with disabilities interact with the virtual environ- respond to the user’s inputs.
ment, whether for avatar interactions to help users Wearable haptic devices for visually impaired
with autism to better understand emotions through users provide AR installations featuring gesture
audiovisual cues (Yuan and Ip 2018) or facilitat- tracking and haptic gloves that enable computer
ing communication through the interpretation of interactions including writing emails, storing and
sign language (Cheok et al. 2019). retrieving files, making video calls, and accessing
email, among others (Kalra et al. 2021). Theil
et al. (2020) developed a tactile board for users
Somatosensory Perception who are both deaf and blind as a tool for translat-
Technologies ing text and speech into vibrotactile cues
displayed through a wearable haptic user inter-
Kinesthetic and proprioceptive cues, pertaining to face. Wang et al. (2018a) developed an intelligent
the somatosensory system, allow for the feeling of wearable controller for people with motor disabil-
textures, object properties, objects in motion, and ities employing machine learning and noninva-
a perception of self. Within this system, haptic sive sensors to drive computer interactions based
feedback or the sense of touch plays an important on eye muscle movement and facial expressions.
Immersive Technologies for Accessible User Experiences 919

Olfactory and Taste Perception Although body capture is used to facilitate the
Technologies interactions, users who experience limited mobil-
ity may require additional input devices to execute
Finally, olfaction and taste respond to chemical the virtual tasks. For example, brain-computer
cues that are underrepresented in immersive tech- interfaces allow capturing cerebral activity that
nologies as these mainly provide feedback that is can be used to interact with virtual elements.
difficult to register. While significant technology Coogan and He (2018) proposed a modular sys-
development is lacking in this area, olfactory feed- tem that allows users to customize the interactions
back in VR has been investigated as it can add based on their brain activity to empower users
immersion and presence to the user experience by with autonomy. Bobrova et al. (2020) captured
providing alerts, reinforce learning, and evoke imaginary lower limb movements through a
memories and emotion. Casillas-Figueroa et al. brain-computer interface as means to enable VR
(2020) conducted a study to assess the effective- locomotion. Further customization employing
ness of olfactory feedback in reminiscence therapy physiological sensors is becoming readily avail-
to provide additional feedback to the users. able in VR and MR headsets, as well as mobile
devices for AR.
Ergonomics
I
Ergonomics focuses on ensuring that the interac-
tions are usable and effective with respect to the Conclusion
user’s anatomy and physiology. Higher physical
levels of immersion require taking advantage of Immersive technologies have shown their disrup-
human interface devices that capture the user’s tive potential in changing how we engage with
body and their interactions within the virtual envi- education, entertainment, fitness, and training.
ronment. Currently, hand tracking is readily avail- While the benefits have been documented toward
able in consumer-level VR through image the average user, the landscape is rapidly chang-
processing (e.g., Oculus Quest and Microsoft ing as researchers, developers, and enthusiasts
HoloLens) as is finger tracking through proximity have started developing inclusive experiences.
sensors (e.g., Valve Index controllers). Shao et al. The affordability of XR devices has increased
(2020) developed a system for teaching American the number of users who install base and sparked
sign language in MR with noticeable improve- interest in sectors that were unable to use them due
ments when compared to computer desktop learn- to costs or infrastructure requirements.
ing. However, image processing with hand tracking The adoption of XR has allowed to identify
lacks accuracy in comparison to data gloves that several areas of research associated with usability,
allow capturing dexterity more accurately (Ahmed user experience, body tracking, locomotion,
et al. 2018). Furthermore, some gloves integrate motion sickness, user interactions, cognitive load,
diverse actuators to provide thermal, haptic, and security, and presence, among others. While the
force feedback (Wang et al. 2018b). hardware may come as one-size-fits-all, researchers
Locomotion is another field of interest as it are creating innovative solutions leading to novel
enables users with limited mobility to experience devices, techniques, retrofitting, reverse engineer-
virtual navigation. Virtual walking is typically ing, and customization of immersive technologies
achieved by means of head tracking, arm swing- for accessible user experiences.
ing detection, and custom-made foot-user inter-
faces in conjunction with 3D-user interactions
such as teleportation (Cherni et al. 2020). How- Cross-References
ever, it can also be accomplished by other input
mechanisms including eye tracking, muscle activ- ▶ Augmented Reality
ity, brain activity, speech recognition, or body ▶ Computer Vision
gestures. ▶ Interaction
920 Immersive Technologies for Accessible User Experiences

▶ Immersive Visualizations Using Augmented Cowan, B., Kapralos, B., Collins, K.: Does improved
Reality and Virtual Reality sound rendering increase player performance?
A graph-based spatial sound framework. IEEE Tran.
▶ Virtual Reality Games. (2020)
De Oliveira, L.C., Andrade, A.O., De Oliveira, E.C.,
Soares, A., Cardoso, A., Lamounier, E.: Indoor navi-
References gation with mobile augmented reality and beacon tech-
nology for wheelchair users. In: 2017 IEEE EMBS
International Conference on Biomedical & Health
Ahmed, S., Rehman, S. M., Ukkonen, L., & Björninen, T.: Informatics (BHI) IEEE, pp. 37–40 (2017)
Glove-integrated slotted patch antenna for wearable Fralish, B., Nickels, M., Garcia, J.: Improved physiology
UHF RFID reader. In 2018 IEEE International Confer- and psychosocial well-being for individuals with phys-
ence on RFID Technology & Application (RFID-TA) ical disabilities through virtual reality immersion: a
(pp. 1–5). IEEE. (2018) case study. J. Dev. Phys. Disabil. 30(6),
Baragash, R.S., Al-Samarraie, H., Alzahrani, A.I., Alfarraj, 771–782 (2018)
O.: Augmented reality in special education: a meta- Guarese, R., Bastidas, F., Becker, J., Giambastiani, M.,
analysis of single-subject design studies. Eur. Iquiapaza, Y., Macedo, L., Nedel, L., Maciel, A.,
J. Spec. Needs Educ. 35(3), 382–397 (2020) Zambetta, F., van Schyndel, R.: Cooking in the dark:
Bobrova, E.V., Reshetnikova, V.V., Frolov, A.A., a mixed reality empathy experience for the embodi-
Gerasimenko, Y.P.: Use of imaginary lower limb move- ment of blindness. In: Proceedings of the 1st XR in
ments to control brain–computer interface systems. Games Workshop. SBC (2021)
Neurosci. Behav. Physiol. 50(5), 585–592 (2020) Gupta, T., Sisodia, M., Fazulbhoy, S., Raju, M., Agrawal,
Bryant, L., Brunner, M., Hemsley, B.: A review of virtual S.: Improving accessibility for dyslexic impairments
reality technologies in the field of communication dis- using augmented reality. In: 2019 International Confer-
ability: implications for practice and research. Disabil. ence on Computer Communication and Informatics
Rehabil. Assist. Technol. 15(4), 365–372 (2020) (ICCCI) IEEE, pp. 1–4 (2019)
Carreon, A., Smith, S.J., Mosher, M., Rao, K., Rowland, Jeong, K., Kim, J., Kim, M., Lee, J., Kim, C.: Asymmetric
A.: A review of virtual reality intervention research for interface: user interface of asymmetric virtual reality
students with disabilities in K–12 settings. for new presence and experience. Symmetry. 12(1),
J. Spec. Educ. Technol., 0162643420962011 (2020) 53 (2020)
Casillas-Figueroa, R., Morán, A.L., Meza-Kubo, V., Kalra, S., Jain, S., Agarwal, A.: Gesture controlled tactile
Ramírez-Fernández, C., Acosta-Quiroz, C., Orihuela- augmented reality interface for the visually impaired.
Espina, F., Montero-Hernandez, S.: ReminiScentia: JITR. 14(2), 125–151 (2021)
shaping olfactory interaction in a personal space for Kapralos, B., Collins, K., Uribe-Quevedo, A.: The senses
multisensory stimulation therapy. Pers. Ubiquitous and virtual environments. Senses Soc. 12(1),
Comput., 1–17 (2020) 69–75 (2017)
Checa, D., Ramon, L., Bustillo, A.: Virtual reality travel Kim, K., Jeong, J.H., Cho, J.H., Kim, S., Kang, J., Ryu, J.,
training simulator for people with intellectual disabil- Lee, S.W.: Development of a human-display interface
ities. In: International Conference on Augmented Real- with vibrotactile feedback for real-world assistive
ity, Virtual Reality and Computer Graphics Springer, applications. Sensors. 21(2), 592 (2021)
Cham, pp. 385–393 (2019) LaViola Jr., J.J., Kruijff, E., McMahan, R.P., Bowman, D.,
Cheok, M.J., Omar, Z., Jaward, M.H.: A review of hand Poupyrev, I.P.: 3D user interfaces: theory and practice.
gesture and sign language recognition techniques. Int. Addison-Wesley Professional (2017)
J. Mach. Learn. Cybern. 10(1), 131–115 (2019) Masnadi, S., Williamson, B., González, A.N.V., LaViola, J.
Cherni, H., Métayer, N., Souliman, N.: Literature review of J.: Vriassist: an eye-tracked virtual reality low vision
locomotion techniques in virtual reality. Int. J. Virtual assistance tool. In: 2020 IEEE Conference on Virtual
Real. 20(1), 1–20 (2020) Reality and 3D User Interfaces Abstracts and Work-
Cinieri, S., Kapralos, B., Uribe-Quevedo, A., Lamberti, F.: shops (VRW) IEEE, pp. 808–809 (2020)
Eye tracking and speech driven human-avatar emotion- Massiceti, D., Hicks, S.L., van Rheede, J.J.: Stereosonic
based communication. In: 2020 IEEE 8th International vision: exploring visual-to-auditory sensory substitu-
Conference on Serious Games and Applications for tion mappings in an immersive virtual reality naviga-
Health (SeGAH) IEEE, pp. 1–5 (2020) tion paradigm. PLoS One. 13(7), e0199389 (2019)
Clay, V., König, P., Koenig, S.: Eye tracking in virtual Matthews, S., Uribe-Quevedo, A., Theodorou, A.: Render-
reality. J. Eye Mov. Res. 12(1) (2019) ing optimizations for virtual reality using eye-tracking.
Coogan, C.G., He, B.: Brain-computer interface control in In: 2020 22nd Symposium on Virtual and Augmented
a virtual reality environment and applications for the Reality (SVR) IEEE, pp. 398–405 (2020)
internet of things. IEEE Access. 6, Mirzaei, M., Kan, P., Kaufmann, H.: EarVR: using ear
10840–10849 (2018) haptics in virtual reality for deaf and hard-of-hearing
Immersive Technologies for Medical Education 921

people. IEEE Trans. Vis. Comput. Graph. 26(5), Computer Interaction Springer, Cham,
2084–2093 (2020) pp. 132–148 (2018)
Nassif, A.B.., Shahin, I., Attili, I., Azzeh, M., Shaalan, K.: Theil, A., Buchweitz, L., Gay, J., Lindell, E., Guo, L.,
Speech recognition using deep neural networks: a sys- Persson, N.K., Korn, O.: Tactile board: a multimodal
tematic review. IEEE Access. 7, 19143–19165 (2019) augmentative and alternative communication device
Reynal, M., Imbert, J.P., Aricò, P., Toupillier, J., Borghini, for individuals with deafblindness. In: 19th Interna-
G., Hurter, C.: Audio focus: interactive spatial sound tional Conference on Mobile and Ubiquitous Multime-
coupled with haptics to improve sound source location dia, pp. 223–228 (2020)
in poor visibility. Int. J. Hum.-Comput. Stud. 129, Tölgyessy, M., Dekan, M., Chovanec, Ľ., Hubinský, P.:
116–128 (2019) Evaluation of the azure Kinect and its comparison to
Robern, G., Uribe-Quevedo, A., Sukhai, M., Coppin, P., Kinect V1 and Kinect V2. Sensors. 21(2), 413
Lee, T., Ingino, R.: Work-in-progress-exploring VR (2021)
conference navigation employing audio cues. In: 2021 Wang, K.J., Liu, Q., Zhao, Y., Zheng, C.Y., Vhasure, S.,
7th International Conference of the Immersive Learn- Liu, Q., Thakur, P., Sun, M., Mao, Z.H.: Intelligent
ing Research Network (iLRN) IEEE, pp. 1–3 (2021) wearable virtual reality (VR) gaming controller for
Saha, D., Sayyed, A.S., Saif, A.F.M., Shahnaz, C., Fattah, people with motor disabilities. In: 2018 IEEE Interna-
S.A.: Eye gaze controlled immersive video navigation tional Conference on Artificial Intelligence and Virtual
system for disabled people. In: 2019 IEEE R10 Reality (AIVR) IEEE, pp. 161–164 (2018a)
Humanitarian Technology Conference (R10-HTC) Wang, D., Song, M., Naqash, A., Zheng, Y., Xu, W.,
(47129) IEEE, pp. 30–35 (2019) Zhang, Y.: Toward whole-hand kinesthetic feedback:
Shao, Q., Sniffen, A., Blanchet, J., Hillis, M.E., Shi, X., a survey of force feedback gloves. IEEE Trans. Haptics.
Haris, T.K., Liu, J., Lamberton, J., Malzkuhn, M., 12(2), 189–204 (2018b) I
Quandt, L.C., Mahoney, J.: Teaching American sign Yuan, S.N.V., Ip, H.H.S.: Using virtual reality to train
language in mixed reality. Proc. ACM Interact. Mob. emotional and social skills in children with autism
Wearable Ubiquitous Technol. 4(4), 1–27 (2020) spectrum disorder. London J. Prim. Care. 10(4),
Skarbez, R., Smith, M., Whitton, M.C.: Revisiting 110–112 (2018)
Milgram and Kishino’s reality-virtuality continuum. Zhao, Y., Bennett, C.L., Benko, H., Cutrell, E., Holz, C.,
Front. Virtual Real. 2, 27 (2021) Morris, M.R., Sinclair, M.: Enabling people with visual
Sorgini, F., Caliò, R., Carrozza, M.C., Oddo, C.M.: Haptic- impairments to navigate virtual reality with a haptic and
assistive technologies for audition and vision sensory auditory cane simulation. In: Proceedings of the 2018
disabilities. Disabil. Rehabil. Assist. Technol. 13(4), CHI Conference on Human Factors in Computing Sys-
394–421 (2018) tems, pp. 1–14 (2018)
Spagnol, S., Wersényi, G., Bujacz, M., Bălan, O., Herrera
Martínez, M., Moldoveanu, A., Unnthorsson, R.: Cur-
rent use and future perspectives of spatial audio tech-
nologies in electronic travel aids. Wirel. Commun.
Mob. Comput. (2018)
Speicher, M., Hall, B.D., Nebeling, M.: What is mixed Immersive Technologies for
reality? In: Proceedings of the 2019 CHI Conference Medical Education
on Human Factors in Computing Systems,
pp. 1–15 (2019)
Suh, A., Prophet, J.: The state of immersive technology Bill Kapralos1, Alvaro Uribe-Quevedo1 and
research: a literature analysis. Comput. Hum. Behav. Adam Dubrowski2
86, 77–90 (2018) 1
Software Informatics Research Centre,
Tabafunda, A., Matthews, S., Akhter, R., Uribe-Quevedo,
A., Sun, W., Horsburgh, S., LaFontaine, C.: Develop-
University of Ontario Institute of Technology,
ment of a non-immersive VR Reminiscence therapy Oshawa, Canada
2
experience for patients with dementia. In: International Disciplines of Emergency Medicine and
Conference on Human-Computer Interaction Springer, Pediatrics and the Marine Institute, Memorial
Cham, pp. 509–517 (2020)
Tashev, I.J., Gamper, H.: Head-related transfer function
University of Newfoundland, St. Johns, Canada
personalization for the needs of spatial audio in mixed
and virtual reality. J. Acoust. Soc. Am. 141(5),
3536–3536 (2017) Synonyms
Teófilo, M., Lourenço, A., Postal, J., Lucena, V.F.: Explor-
ing virtual reality to enable deaf or hard of hearing
accessibility in live theaters: a case study. In: Interna- Educational simulation; Game-based learning
tional Conference on Universal Access in Human- (GBL); Immersive tech; Sim
922 Immersive Technologies for Medical Education

Definition supervised clinical practice (Riener and Harders

2012). However, this approach is no longer viable
Computer A recreation of real-world and becoming less acceptable given the increasing
simulation phenomena employing focus on patient safety, care, and awareness, along
mathematical models that can be with budgetary constraints associated with teach-
visualized through a computer- ing in a clinical environment, particularly when
generated scene. considering invasive procedures that require high-
Immersion Sensation of being in a computer- risk care (Vozenilek et al. 2004). This increased
generated world created by awareness and focus on patient safety has been
surrounding hardware providing encouraged by studies that have examined medi-
sensory stimuli. Can be a purely cal errors and have suggested that as many as
mental state or can be 98,000 Americans and 23,000 Canadians may
accomplished through physical die each year due to medical errors. Although
means. the precise numbers have been debated, without
Presence The feeling of being immersed in a a doubt, medical errors affect patient outcomes
computer-generated world. and health-care costs (Brindley et al. 2007). A
Immersive Devices that provide sensory proposed solution is a shift from the traditional
technology stimuli to provide a sense of “see one, do one, teach one” apprenticeship model
realism and immersion to the of education to one where the transition from
interactions with the computer- theory to practice is augmented by carefully
generated world. designed simulated encounters with patients,
Virtual An interactive computer teams, and health systems. Termed simulation-
reality simulation which senses the based education (SBE), this approach enables a
user’s state and operation and replication of most clinical situations with various
replaces or augments sensory degrees of fidelity or realism, multimodality,
feedback information to one or immersion, and presence (Brindley et al. 2007).
more senses in a way that the user SBE can offer a viable alternative or complemen-
obtains a sense of being immersed tary mean to practice a range of skills, attitudes,
in the simulation (virtual and behaviors ranging from cognitive, to commu-
environment). nication, to crisis resource management, and to
Augmented The addition of computer- psychomotor to be developed by junior trainees or
reality generated objects to the real maintained by more experienced doctors
physical space to augment the (Brindley et al. 2007).
elements comprising it. Per Halamek et al. (2000), simulation involves
Mixed Integration of computer-generated immersing the trainee in a realistic situation
reality graphics and real objects (scenario) created within a physical or virtual
seamlessly. space (simulator) that replicates the real environ-
Serious A video game whose primary ment. In the context of medical education, simu-
game purpose is education, training, lation can be defined as an education technique
advertising, simulation, or that allows interactive and immersive activity by
education as opposed to recreating all or part of a clinical experience with-
entertainment. out exposing patients to the associated risks
(Perkins 2007). It has two components: the sce-
Introduction nario and the simulator. The scenario is a descrip-
tion of a simulation that includes the goals,
Traditionally, medical education has followed the objectives, feedback or debriefing points, narra-
time-honored concept of “see one, do one, teach tive description of the clinical simulation, staff
one,” where theoretical education is followed by requirements, simulation room setup, simulators,
Immersive Technologies for Medical Education 923

props, simulator operation, and instructions for the maintenance of the simulation facility, often
SPs (Alinier 2011). The simulator, on the contrary, referred to as simulation laboratory. Collectively,
is a setting, device, computer program, or system although SBE has been shown to be an effective
that performs simulation (Hancock et al. 2008), tool in providing new learning opportunities that
and can include manikins, cadavers, animals, lead to more skilled trainees and safer practice, it is
devices, technologies, computer programs and expansive. Therefore, the field of SBE is in search
virtual spaces, scenarios, standardized patients, of more cost-effective solutions which can aug-
and a host of other methods of imitating real- ment the current educational practices.
world systems (Curtis et al. 2012).
Simulation in medical education is a well-
established pedagogical practice (Reznick and Immersive Technologies
MacRae 2006). It provides a viable alternative to
practice with actual patients, providing medical The technologies of video games, virtual worlds,
trainees the opportunity to train until they reach and social networks have become collectively
a specific competency level. Simulation ranges known as immersive technologies because of
from decontextualized bench models and virtual their ability to engage users of all ages, driving
reality (VR-)-based environments, to high fidelity massive investment into technologies to attract,
I
recreations of actual operating rooms (Kneebone capture, and retain our attention (Wortly 2014).
2009). One of the prevailing arguments for using The increase in computational processing power
simulation in the learning process of trainees is the and accompanying decrease in the size of elec-
ability to engage the trainee in the active accumu- tronic components has led to the decreasing cost
lation of knowledge by doing with deliberate and rising availability of consumer-level
practice, while it also allows for careful matching immersive technologies which have helped
of the complexity of the learning encounter advance the adoption of virtual simulation in
to the trainees’ current level of advancement recent years. For example, hand and arm tracking
(Guadagnoli et al. 2012). technologies accomplished with controllers such
Although the economic evaluation in SBE is as the Leap Motion hand sensor, the Razer Hydra,
still in its infancy, recent reports suggest that the and the Thalmic Labs Myo interaction device are
costs associated with this approach are high (Lin allowing for the development of novel interaction
et al. 2017), yet if designed correctly they are cost- methods and techniques. Devices such as the
effective means of increasing the trainees’ skills, Microsoft Kinect V2 motion sensor can track the
knowledge, and attitudes (Isaranuwatchai et al. position of the user’s body and precisely track
2014). These costs are primarily related to the the movement of individual fingers in three-
“simulator” part of the simulation equation. The dimensional (3D) space. Haptic input devices
simulation equipment and material costs are high such as the Novint Falcon or the 3D Touch Stylus
due to low volume and costs of production. provide a sense of touch and feedback to motion
Finally, the depreciation, durability, and mainte- controls. Collectively, these devices provide more
nance cost of the equipment also affect these costs. natural and immersive interactions, which in
The purpose of this equipment is to allow the many applications help overcome the limitations
trainees to make errors and therefore the equip- associated with traditional keyboard and mouse-
ment wears out quickly. Another cost is that of based human-computer interactions where tasks
personnel, which includes faculty members’ time, are performed very differently to a real-life situa-
confederate and actor training (Adler et al. 2016), tion. The current trend on natural user interactions
administration staff and instructors, and a site is providing the designers and developers of vir-
champion to maintain interprofessional leader- tual simulations with tremendous freedom and
ship, team management, evaluation of the pro- opportunities to develop highly immersive
gram, and quality assurance of the training applications. The field of immersive/interactive
(Walsh and Jaye 2013). Finally, other costs include multimedia applications (including virtual
924 Immersive Technologies for Medical Education

environments (VEs) such as video games, virtual enhance, complement, and augment the
simulations, virtual reality, and serious games, immersive scenario. Moreover, in medical appli-
that is, video games whose primary purpose is cations AR has allowed visualizing medical data
education, training, advertising, simulation, or and the patient within the same physical space
education as opposed to entertainment), has seen (Sielhorst et al. 2008). The skills acquired during
significant advancement over the last couple of virtual reality-based simulation training have been
decades. The field of virtual reality alone is esti- proven to transfer over to subsequent performance
mated to grow from $1.37 billion (USD) in 2015 in operating rooms (e.g., see Seymour 2008).
to $33.90 billion (USD) by 2022 (Markets and During the last few years, there has been an
Markets 2016), while a recent estimate by explosive growth of interest in consumer-grade
Goldman Sachs suggests the fields of virtual and VR technologies, in part due to gaming and enter-
augmented reality are expected to grow into a $95 tainment applications, with several players such
billion market by 2025 (Hall and Takabashi as Facebook, Microsoft, and HTC investing in
2017). hardware and software to increase the VR install
Current VR installments are seeing imple- base. Since then, various competitors, including
mentations in different areas given the possibili- various technology company giants, have entered
ties to have users interact in safe, controlled, and the marketplace. For example, Samsung has
monitored environments. Mihelj et al. (2014) designed and developed the Gear VR, an inexpen-
define VR as “an interactive computer simulation, sive headset which uses the owner’s mobile
which senses the user’s state and operation and smartphone as the display and system processor.
replaces or augments sensory feedback informa- Sony developed the PlayStation VR which inte-
tion to one or more senses in a way that the user grates seamlessly into the Sony PlayStation 4 con-
obtains a sense of being immersed in the simula- sole platform. At the Apple Worldwide
tion (virtual environment).” Virtual reality has Developers Conference (WWDC) 2017, Apple
also been defined as “the use of computer model- introduced the ARKit, an augmented reality plat-
ing and simulation that enables a person to interact form that provides advanced augmented reality
with an artificial three-dimensional (3-D) visual or capabilities on iOS and will be supported by all
other sensory environment” (Riener and Harders phones that Apple releases. Facebook has
2012). At a minimum functionality, VR systems acquired 11 AR/VR companies, stressing the
typically utilize a head mounted display (HMD), company’s view that VR and AR will “form the
as the primary method for a user to view and next frontier” (Hall and Takabashi 2017). The
interact with a virtual world. A HMD is a display large investments and acquisitions by various
device worn on a user’s head which uses optical large technology firms indicate that these technol-
lenses and one or two small displays to show ogies will become increasingly integrated with the
computer-generated imagery. These headsets typ- platforms on which we consume content. The
ically use various sensors embedded in the device high-end consumer-grade VR option is currently
(such as accelerometers or gyroscopes) to trans- held by Taiwan-based HTC’s Vive device, a pack-
late real-world movement and rotation into age including an HMD, two hand controllers, and
corresponding changes of the view of the virtual two base stations used to track both the headset
world. A capability currently supported by mod- and controllers in a 3D volume of real-world
ern mobile devices, whose sensors and mobile space. In contrast to traditional VR (“seated”)
HMD provide consumer-level VR. Within the configuration where the user remains seated in a
taxonomy of VR, mixed reality (MR), augmented chair within the range of the motion tracking
reality (AR), and mixed reality providing various sensor, the HTC vive allows for room-scale VR
degrees of real and computer-generated images whereby the user can freely move within the real
integration (Milgram and Colquhoun 1999). This world (e.g., walk around, face any direction, bend
scenario has caused interest in training and edu- down to the floor), and have consistently tracked
cation due to visualization opportunities to hand controllers within a wide space.
Immersive Technologies for Medical Education 925

Traditionally, the video game industry has role particularly under the following six medical-
been the largest pushing influence of advancing based applications (Gardner 2016): (i) education
certain technologies such as computer graphics and training, (ii) surgical planning, (iii) tele-
and personal computer audio capabilities. This monitoring, (iv) patient experience, (v) treatment
trend follows suit with the modern VR device and therapies, and (vi) augmented reality-
companies specifically targeting video game enhanced surgery and patient care.
players, with one such example being usable
only with a video game console (the Sony
PlayStation VR). Although video game players Examples of Immersive Technologies in
are historically early adopters of new, exciting, Medical Education
and expensive technologies, VR device compa-
nies should not neglect the possibilities in the Wilcocks et al. (2017) developed a virtual simu-
space of serious games and virtual simulations lation of the angiogram procedure specifically to
for education and training. Modern consumer- educate patients about the commonly performed
grade VR devices are affordable, powerful, and angiogram procedure. Using an HTC Vive virtual
will soon be adopted by the mass market. reality headset, the patient is taken into a virtual
Although the strongest demand for immersive catheterization (cath) lab and introduced to the
I
technologies currently comes from industries in angiogram procedure in a highly immersive, inter-
the creative economy, and more specifically, gam- active, and engaging virtual environment. The
ing, live events, video entertainment, and retail, goal of their simulation is to educate the patient
immersive technologies will find wider applica- about the procedure they will undergo and thus
tions in industries as diverse as health care, help reduce the fear often associated with the
education, the military, and real estate over time procedure, while increasing the patient’s under-
(Hall and Takabashi 2017). According to Huff and standing and awareness, ultimately leading to
Saxberg (2009), immersive technologies – such as greater patient outcomes (Fig. 1).
multi-touch displays; telepresence (an immersive Shewaga et al. (2017) developed a room-scale
meeting experience that offers high video and epidural preparation serious game facilitated with
audio clarity); 3D environments; collaborative an HTC-Vive headset that allows a user (trainee)
filtering (which can produce recommendations to assume the role of a medical professional pre-
by comparing the similarity between your prefer- paring to perform an epidural procedure
ences and those of other people); natural language (an injection into the epidural space around the
processing; intelligent software; and simulations – spinal cord and spinal nerves). The serious game
will transform teaching and learning by 2025. focuses on the cognitive aspects of the epidural
Projections for the growth of the VR hardware procedure as opposed to the technical compo-
industry are staggering, with some research firms nents, and more specifically, the steps of the pro-
suggesting the market will grow to $50 billion by cedure and what needs to be done at each step
2021 (Sinclair 2016). This is driven largely in part (e.g., reading a detailed patient anesthetic record,
by video game early adopters although VR and washing their hands, wearing proper operating
AR are rapidly entering into the medical educa- room clothing, and gathering the various tools
tion and health-care fields. Within the medical needed for the procedure and placing them onto
field, VR and AR are not so much technologies a preparation tray). Using the epidural serious
of the future but rather, of the present where game, they conducted a quantitative and qualita-
researchers, doctors, and nurse educators are find- tive comparison between the usability, perfor-
ing innovative ways to leverage immersive tech- mance, and engagement of traditional desktop
nologies and transform both health-care teaching VR with a room-scale variation in order to
and practice (Craig and Georgieva 2017). In fact, develop a greater understanding of the differences
over time, it is anticipated that immersive technol- between the two configurations and determine
ogies and VR/AR specifically will play a large whether the additional requirements and resources
926 Immersive Technologies for Medical Education

a b
Hand manipulating
the tablet

Hand placing the

electrodes

First person view of the catheterizaƟon lab simulaƟon. Highlighted outline as visual cue upon interacƟon.

Immersive Technologies for Medical Education, Fig. 1 HTC Vive-based virtual angiogram simulation for patient
education

associated with a room-scale VR configuration is we go), VR is currently capable of providing a

warranted. Their quantitative results revealed lim- wide range of immersive experiences through
ited differences between the two configurations, numerous hardware that can provide various
although the room-scale VR configuration did forms of sensory stimuli to achieve presence.
lead to higher immersion, and was generally pre- This is of outmost importance in all VR install-
ferred more among the participants. Greater work ments as proper depiction of the tasks may affect
remains to determine whether one configuration is the engagement and successful results. Moreover,
superior, but their results highlight the importance in medical training the VR installments require the
of considering the limitations of computer hard- interactions to be tailored to match real-life sce-
ware when designing serious games and simula- narios to guarantee that the skills and competence
tions, particularly when utilizing VR devices. can be transferred to live settings. That being said,
Designers and developers of virtual simulations it is important to consider the limitations of com-
and serious games should consider the effects of puter hardware when designing serious games
the hardware being used and the requirements and simulations, particularly when utilizing VR
imposed upon it by the desired visual fidelity of devices. There are numerous open research
the simulation. Higher fidelity implies greater themes in VR, given the continuous hardware
computational requirements which may not be evolution and the high variability of users who
readily available to the average computer user may have motion sickness, 3D stereo blindness,
(a discussion regarding fidelity, multimodal inter- and disabilities that would require adjustment to
actions, and perceptual-based rendering as they current hardware or the creation of new one, while
pertain to serious games is available by Kapralos addressing usability issues.
et al. (2017)) (Fig. 2). The future of SBE will, without a doubt, focus
on introducing computer-based simulation with
immersive technologies as part of blended modal-
Conclusions ity approaches. That is, the different simulation
modalities such as task trainers, manikin-based,
As computing power expands, new human per- standardized/simulated patients, computer-based,
formance monitoring technologies evolve, and the virtual reality, and hybrid play different roles in
cost of simulation equipment falls, all medical the educational journey of the trainee. Based on
training programs will need to devote substantial research to date, one can speculate that the choices
curricular time to SBE (Reznick and MacRae and the use of various modalities in tandem with
2006). With respect to computer simulation other ones will relate to the level of trainees,
(or in-silico simulation, depending on which way learning objectives, availability of resources, and
Immersive Technologies for Medical Education 927

Immersive Technologies for Medical Education, Fig. 2 In-game first person view of the virtual operation room of
the room-scale VR epidural serious game (user is shown in the bottom-right inset)

physical location (urban vs. rural training sites) to Review. https://er.educause.edu/blogs/2017/8/vr-and-

name a few. ar-driving-a-revolution-in-medical-education-and-pati
ent-care (2017). Accessed 3 Nov 2017
Curtis, M.T., DiazGranados, D., Feldman, M.: Judicious
use of simulation technology in continuing medical
Cross-References education. J. Contin. Educ. Health Prof. 32, 255–260
(2012)
Gardner, J.A.: Augmented and virtual reality in medicine:
▶ Game-Based Interventions in Public Health: 6 applications were keeping our eye on. https://
Exploiting the Engaging Factor of Gameplay medtechboston.medstro.com/blog/2016/05/24/16045/
▶ Gamification and Serious Games (2016). Accessed 3 Nov 2017
▶ Interactive Augmented Reality to Support Guadagnoli, M., Morin, M.P., Dubrowski, A.: The appli-
cation of the challenge point framework in medical
Education education. Med. Educ. 46, 447–453 (2012)
▶ Physical, Virtual, and Game World Persistence Halamek, L.P., Kaegi, D.M., Gaba, D.M., Sowb, Y.A.,
Smith, B.C., Smith, B.E., Howard, S.K.: Time for a
new paradigm in pediatric medical education: teaching
neonatal resuscitation in a simulated delivery room
References environment. Pediatrics. 106, E45 (2000)
Hall, S., Takabashi, R.: Augmented and Virtual Reality:
Adler, M.D., Overly, F.L., Nadkarni, V.M., Davidson, J., The Promise and Peril of Immersive Technologies.
Gottesman, R., Bank, I., Marohn, K., Sudikoff, S., World Economic Forum. https://www.weforum.org/
Grant, V.J., Cheng, A.: An approach to confederate agenda/2017/09/augmented-and-virtual-reality-will-c
training within the context of simulation-based hange-how-we-create-and-consume-and-bring-new-r
research. Simul. Healthc. 11, 357–362 (2016) isks/ (2017). Accessed 3 Nov 2017
Alinier, G.: Developing high fidelity health care simulation Hancock, P.A., Vincenzi, D.A., Wise, J.A., Mouloua, M.
scenarios: a guide for educators and professionals. (eds.): Human Factors in Simulation and Training.
Simul. Gaming. 42, 9–26 (2011) CRC Press (2008) Boca Raton, FL, USA
Brindley, P.G., Suen, G.I., Drummond, J.: Medical simu- Huff, G., Saxberg, B.: Full immersion – how will 10-year-
lation: see one, do one, teach one. . . just not on my olds learn? Educ. Next. 9, 79–82 (2009)
Mom. Can. J. Respir. Ther. 43, 22–27 (2007) Isaranuwatchai, W., Brydges, R., Carnahan, H.,
Craig, E., Georgieva, M.: VR and AR: Driving a Revolu- Backstein, D., Dubrowski, A.: Comparing the cost
tion in Medical Education & Patient Care. Educause effectiveness of simulation modalities: a case study
928 Immersive Virtual Reality Serious Games

of peripheral intravenous catheterization training. Walsh, K., Jaye, P.: Simulation-based medical education:
Adv. Health Sci. Educ. Theory Pract. 19, 219–232 cost measurement must be comprehensive. Surgery.
(2014) 153, 302 (2013)
Kapralos, B., Moussa, F., Collins, K., Dubrowski, A.: Wilcocks, K., Halabi, N., Kartick, P., Uribe-Quevedo, A.,
Levels of fidelity and multimodal interactions. In: Chow, C., Kapralos, B.: A virtual cardiac catheteriza-
Wouters, P., van Oostendorp, H. (eds.) Techniques to tion laboratory for patient education: the angiogram
Improve the Effectiveness of Serious Games, Advances procedure. In: Proceedings of the 8th IEEE Interna-
in Game-based Learning Book Series, pp. 79–101. tional Conference on Information, Intelligence,
Springer (2017) Cham Switzerland Systems and Applications (IISA), 28–30 Aug 2017,
Kneebone, R.L.: Practice, rehearsal, and performance: Larnaca (to appear 2017)
an approach for simulation-based surgical and pro- Wortly, D.: The future of serious games and immersive
cedure training. J. Am. Med. Assoc. 302, 1336–1338 technologies and their impact on society. In: Baek, Y.,
(2009) Ko, R., Marsh, T. (eds.) Trends and applications
Lin, Y., Cheng, A., Hecker, K., Grant, V., Currie, G.R.: of serious gaming and social media. Springer
Implementing economic evaluation in simulation- Science+Business Media, Singapore (2014)
based medical education: challenges and opportunities.
Med. Educ. (2017). https://doi.org/10.1111/medu.
13411. [Epub ahead of print]
Markets and Markets: Virtual Reality Market by
Component (Hardware and Software), Technology Immersive Virtual Reality
(Non-Immersive, Semi- & Fully Immersive), Device Serious Games
Type (Head-Mounted Display, Gesture Control
Device), Application and Geography – Global Forecast
to 2022. MarketsandMarkets™ Research Private Ltd. Lal “Lila” Bozgeyikli and Evren Bozgeyikli
(2016). Accessed 3 Nov 2017 School of Information, University of Arizona,
Mihelj, M., Novak, D., Beguš, S.: Virtual Reality Technol- Tucson, AZ, USA
ogy and Applications. Springer Science & Business
Media, Dordrecht (2014)
Milgram, P., Colquhoun, H.: A taxonomy of real and virtual
world display integration. In: Ohta, Y., Hideyuki, T. Synonyms
(eds) Mixed Reality: Merging Real and Virtual Worlds,
Springer-Verlag, Berlin, Germany, pp. 1–26 (1999)
Perkins, G.D.: Simulation in resuscitation training. Resus- Learning; Rehabilitation; Serious games; Train-
citation. 73, 202–211 (2007) ing; Virtual reality systems
Reznick, R.K., MacRae, H.: Teaching surgical skills –
changes in the wind. N. Engl. J. Med. 355,
2664–2669 (2006)
Riener, R., Harders, M.: Virtual Reality in Medicine. Definitions
Springer Science & Business Media, London (2012)
Seymour, N.E.: VR to OR: a review of the evidence that Virtual A model of reality with which users
virtual reality simulation improves operating room per-
Reality can interact using senses such as
formance. World J. Surg. 32, 182–188 (2008)
Shewaga, R., Uribe-Quevedo, A., Kapralos, B., Alam, F.: sight, sound, or touch.
A comparison of seated and room-scale virtual reality Immersive A type of virtual reality in which
in a serious game for epidural preparation. IEEE Trans. Virtual the user’s complete view is
Emerg. Topics Comput. (to appear 2017) https://doi.
Reality surrounded by the synthetic
org/10.1109/TETC.2017.2746085. http://ieeexplore.
ieee.org/document/8017559/?reload=true environment, as if they had stepped
Sielhorst, T., Feurstein, M., Navab, N.: Advanced medical inside the virtual world. It is usually
displays: a literature review of augmented reality. distinguished by the high level of
J. Disp. Technol. 4, 451–467 (2008)
immersion and interactivity
Sinclair, B.: VR hardware will grow to $50 billion by
2021 – Juniper. http://www.gamesindustry.biz/articles/ offered.
2016-10-04-vr-hardware-will-grow-to-usd50-billion- CAVE The abbreviation of Cave
by-2021-juniper (2016). Accessed 8 Sep 2016 Automatic Virtual Environment,
Vozenilek, J., Huff, J.S., Reznek, M., Gordon, J.A.: See
which is a virtual environment
one, do one, teach one: advanced technology in med-
ical education. Acad. Emerg. Med. 11, 1149–1154 consisting of cube-shaped
(2004) projection screens on the walls,
Immersive Virtual Reality Serious Games 929

floors, and ceilings surrounding fidelity; high level of immersion; active participa-
the user. tion within virtual worlds; practicing in a safe
Video A game that is played by environment; real time alteration of several attri-
Game interacting with images in butes and parameters such as task complexity,
electronic form generated on a environmental properties, and speed; reinforce-
device such as computer monitor or ment through infinite repetition using an auto-
TV screen. mated system and software; easy variation and
Serious A video game with a purpose customization of content, tasks, and scenarios;
Game beyond entertainment such as highly structured automated training; high level
teaching users a new skill or of control on provided stimuli; increased focus
training users to improve their of interest by isolation from surroundings; high
existing skills on a subject. degree of visualization which especially helps
with abstract concepts; real-time prompts and
Introduction feedback; no severe consequences of mistakes
made during training; automated data collection
Virtual reality has become a prevalent tool in and the ability of replaying recorded sessions for
various areas because of the advancements seen reflection afterwards; automated assessment and
I
in recent years. Video games constitute the major reporting; being appealing to technology savvy
application area for virtual reality. Video games population.
can be categorized in themselves according On the other hand, some major disadvantages
to their aims, such as entertainment games and of virtual reality can be summarized as follows:
serious games. In this entry, advantages and dis- motion sickness; disorientation; nausea; isolation
advantages of virtual reality and scientific studies and low degree of social interactions; requirement
in the area of immersive virtual reality serious of wearing hardware which may be uncomfort-
games that has been published in the previous able; price of special hardware, even though the
10 years (between 2007 and 2017) are mentioned. technology is significantly affordable nowadays
The databases used were: Google Scholar, ACM as compared to the previous decades; latency and
Digital Library, IEEE Xplore, Springer Link, and frame rate problems; claustrophobia; risk of injury
Elsevier. The following keywords were sear- that may be caused by surrounding physical
ched in these databases: “Virtual Reality Seri- objects; hygienic concerns for multiple users shar-
ous Games,” “Virtual Reality Learning,” “Virtual ing the same headset.
Reality Education,” “Virtual Reality Training,”
and “Virtual Reality Rehabilitation.”
Immersive Virtual Reality Serious Games

Advantages and Disadvantages of This section includes recent immersive virtual

Virtual Reality reality serious game systems that focus on learn-
ing, training, or rehabilitation. There have been
Several advantages and disadvantages of virtual several studies that present virtual reality systems
reality have been identified in various previ- for learning. However, majority of these studies
ous studies (Dautenhahn 2000; Goldsmith and did not include immersive virtual reality systems
LeBlanc 2004; Parsons and Mitchell 2002; Put- but included desktop monitor displays or single
nam and Chong 2008; Rizzo and Kim 2005; screen projections. Many studies were published
Strickland 1997; Wang and Anagnostou 2014; earlier than the 10 years limit this entry imposed.
Sharp et al. 2007; Hale and Stanney 2014). Some recent works on immersive virtual real-
Major advantages of virtual reality can be sum- ity for education are mentioned next. Du has
marized as follows: higher level of interaction worked on an experimental immersive virtual
930 Immersive Virtual Reality Serious Games

reality system within a classroom learning context rehabilitation of individuals with cognitive and
aiming to replace the traditional single projected physical disabilities on several transferrable job
screens in classrooms for more effective learning skills (Bozgeyikli et al. 2017). Yu et al. have
(Du 2014). Angulo and Vasquez de Velasco have worked on an immersive virtual reality system
developed an immersive virtual reality simulation that aimed to train individuals with hearing
system as an aiding tool for architectural spatial impairments on CNC machine operation skills
experience design (Antonieta 2014). Izatt et al. (Yu et al. 2016). Sohn et al. have developed an
have constructed an immersive virtual reality sys- immersive virtual reality projection based system
tem which functioned as an application for visu- for vocational rehabilitation of individuals with
alization and data interaction with the main goal schizophrenia within the contexts of convenience
of introducing new physics students and mem- store employee and supermarket clerk (Sohn et al.
bers of public to physical concepts such as 2016). Webster has worked on an immersive vir-
Super-K, Hyper-K, the T2 K experiment, and tual reality system that aimed to teach basic cor-
water-Cherenkov detectors (Izatt et al. 2014). rosion prevention and control knowledge skills to
Another major area for immersive virtual real- the US Army soldiers (Webster 2014).
ity serious games is training. There have been Rehabilitation is another emerging area that
several studies in the area of improving social includes several virtual reality studies. Some
skills of challenged populations. Cheng et al. recent studies of immersive virtual reality rehabil-
have worked on using immersive virtual environ- itation systems are as follows: Maskey et al. have
ments to improve the following social skill developed an immersive virtual reality system and
aspects of children with Autism Spectrum Disor- utilized it along with cognitive behavior therapy
der: nonverbal communication, social initia- to reduce some forms of phobia and fears such as
tions, and social cognition (Cheng et al. 2015). crowded buses and pigeons in young people with
Matsentidou has developed an immersive virtual Autism Spectrum Disorder (Maskey et al. 2014).
reality system for improving social skills of chil- Rooij et al. have worked on an immersive virtual
dren with Autism Spectrum Disorder via social reality system for improving balance and/or gait in
stories presented in immersive virtual environ- rehabilitation of individuals after stroke (Rooij
ments (Matsentidou and Poullis 2014). Lorenzo et al. 2017).
et al. have worked on an immersive virtual reality
system that aimed to improve social and executive
decision-making skills of children with Limitations
Autism Spectrum Disorder with tasks that focus
on school- and home-based social activities This entry includes immersive virtual reality stud-
(Lorenzo et al. 2013). Ip et al. have worked on ies that were published in the last 10 years
an immersive CAVE-like virtual reality system for (2007–2017) and had a focus of learning, training,
training social adaptation of school-aged children or rehabilitation applications. This entry does not
with Autism Spectrum Disorder in inclusive edu- include the following: entertainment games; com-
cation settings (Ip et al. 2016). Park et al. have mercial games; medical, military, or exercise seri-
developed an immersive virtual reality system for ous games; and nonimmersive virtual reality
training individuals with schizophrenia on social studies that included desktop monitor, TV, or sin-
skills such as conversation, assertiveness, and gle projector displays.
emotion expression via role playing in immersive
virtual environments (Park et al. 2011).
Another emerging area for immersive virtual Conclusion
reality serious games is vocational training, espe-
cially catering for challenged populations. The recent works mentioned are intended to give
Bozgeyikli et al. have developed an immer- a snapshot of scientific studies in the area of
sive virtual reality system for vocational immersive virtual reality in learning, training,
Immersive Virtual Reality Serious Games 931

and rehabilitation. It was observed that the major- physics education. In: IEEE Virtual Reality (VR),
ity of the studies were catering for challen- pp. 83–84 (2014)
Lorenzo, G., Pomares, J., Lledó, A.: Inclusion of immersive
ged populations. The diversity of application virtual learning environments and visual control systems
areas even within the three selected subtopics to support the learning of students with Asperger syn-
(learning, training, and rehabilitation) is encour- drome. Comput. Educ. 62, 88–101 (2013)
aging for future studies in the area of immersive Maskey, M., Lowry, J., Rodgers, J., McConachie, H.,
Parr, J.R.: Reducing specific phobia/fear in young peo-
virtual reality serious games. ple with autism spectrum disorders (ASDs) through a
virtual reality environment intervention. PLoS One.
9(7), e100374 (2014)
Matsentidou, S., Poullis, C.: Immersive visualizations in a
Cross-References VR cave environment for the training and enhancement
of social skills for children with autism. In: Interna-
tional Conference on Computer Vision Theory and
▶ Accessibility of Virtual Reality for Persons
Applications (VISAPP), pp. 230–236 (2014)
with Disabilities Park, K.M., Ku, J., Choi, S.H., Jang, H.J., Park, J.Y.,
▶ Gamification and Serious Games Kim, S.I., Kim, J.J.: A virtual reality application in
role-plays of social skills training for schizophrenia: a
randomized, controlled trial. Psychiatry Res. 189(2),
166–172 (2011)
References Parsons, S., Mitchell, P.: The potential of virtual reality in I
social skills training for people with autistic spectrum
Antonieta, Â.: Immersive simulation of architectural spa- disorders. J. Intellect. Disabil. Res. 46(5), 430–443
tial experiences. Blucher Des. Proc. 1(7), 495–499 (2002)
(2014) Putnam, C., Chong, L.: Software and technologies
Bozgeyikli, L., Bozgeyikli, E., Raij, A., Alqasemi, R., designed for people with autism: what do users want?
Katkoori, S., Dubey, R.: Vocational rehabilitation of In: Proceedings of the 10th International ACM
individuals with autism spectrum disorder with virtual SIGACCESS Conference on Computers and Accessi-
reality. ACM Trans. Access. Comput. (TACCESS). bility, ACM, pp. 3–10 (2008)
10(2), 5 (2017) Rizzo, A.S., Kim, G.J.: A SWOT analysis of the field of
Cheng, Y., Huang, C.-L., Yang, C.-S.: Using a 3D virtual reality rehabilitation and therapy. Presence
immersive virtual environment system to enhance Teleop. Virt. 14(2), 119–146 (2005)
social understanding and social skills for children Rooij, I.J.M.D., van de Port, I.G.L., Meijer, J.W.G.:
with autism spectrum disorders. Focus Autism Other Feasibility and effectiveness of virtual reality training
Dev. Disabil. 30(4), 222–236 (2015) on balance and gait recovery early after stroke: a pilot
Dautenhahn, K.: Design issues on interactive environ- study. Int. J. Phys. Med. Rehabil. 5(417), 2 (2017)
ments for children with autism. In: Proceedings of Sharp, H., Rogers, Y., Preece, J.: Interaction Design:
ICDVRAT 2000, the 3rd International Conference on Beyond Human-Computer Interaction. Wiley, West
Disability, Virtual Reality and Associated Technologies Sussex, United Kingdom (2007)
(2000) Sohn, B.K., Hwang, J.Y., Park, S.M., Choi, J.S., Lee, J.Y.,
Du, X.: Design and evaluation of a learning assistant sys- Lee, J.Y., Jung, H.Y.: Developing a virtual reality-based
tem with optical head-mounted display (OHMD). vocational rehabilitation training program for patients
Doctoral dissertation, Carleton University Ottawa with schizophrenia. Cyberpsychol. Behav. Soc. Netw.
(2014) 19(11), 686–691 (2016)
Goldsmith, T.R., LeBlanc, L.A.: Use of technology in Strickland, D.: Virtual reality for the treatment of autism.
interventions for children with autism. J. Early Inten- Stud. Health Technol. Inform. 44, 81–86 (1997)
sive Behav. Interv. 1(2), 166 (2004) Wang, M., Anagnostou, E.: Virtual reality as treatment tool
Hale, K.S., Stanney, K.M.: Handbook of Virtual Environ- for children with autism. In: Comprehensive Guide to
ments: Design, Implementation, and Applications. Autism, pp. 2125–2141. Springer, New York (2014)
CRC Press, Boca Raton (2014) Webster, R.D.: Corrosion prevention and control training
Ip, H.H., Wong, S.W., Chan, D.F., Byrne, J., Li, C., in an immersive virtual learning environment. The
Yuan, V.S., Lau, K.S.Y., Wong, J.Y.: Virtual reality University of Alabama at Birmingham (2014)
enabled training for social adaptation in inclusive edu- Yu, S., Ryu, J., Han, I., Kim, M.: Developing a 3D virtual
cation settings for school-aged children with autism environment for hearing impaired learners’ learning of
spectrum disorder (ASD). In: International Conference CNC machine operation. In: Society for Information
on Blending Learning, Springer International Publish- Technology & Teacher Education International Confer-
ing, pp. 94–102 (2016) ence, vol. 2016, no. 1, pp. 2450–2454. Association for
Izatt, E., Scholberg, K., Kopper, R.: Neutrino-KAVE: an the Advancement of Computing in Education (AACE)
immersive visualization and fitting tool for neutrino (2016)
932 Immersive Visualizations Using Augmented Reality and Virtual Reality

different immersive technologies. The contin-

Immersive Visualizations uum was a continuous scale ranging between
Using Augmented Reality and the completely virtual and completely real envi-
Virtual Reality ronment with visualization and immersion as the
primary influencers (Milgram et al. 1995)
Madhusudan Rao1 and Manoj Dawarwadikar2 (Fig. 1).
1
NTT Data Services Pvt. Ltd, Bangalore, India In this context, augmented reality, typically
2
SP Jain School of Global Management, Sydney, called as AR, is a technology that superimposes
Bangalore, India digital content on a user’s view of the real physical
world. It provides a composite view through a
device such as a smartphone, a tablet, or a smart-
Synonyms glass. The AR has core capabilities of information
visualization, guiding the user and making the
Augmented reality; Data visualization; Immersive environment interactive through mediums such
technologies; Information visualization; Virtual as gestures and voice (Porter and Heppelmann
reality 2017). Latest developments in smartphone tech-
nologies such as sensor-based “simultaneous
localization and mapping” (SLAM), 3D render-
Definition ing, and camera capabilities have made aug-
mented reality accessible to most consumers
Immersive visualizations refer to a novel way of with better applications (Pangilinan et al. 2019).
representing data and information through spatial Virtual reality, typically called as VR, is a
computing technologies such as augmented and computerized simulation of a new environment
virtual reality. It aims to enhance the user perspec- with visuals and interactions through a head-
tive for enhanced insights from data and informa- mounted device and controls. Vision, audio, and
tion to assist in the various processes including but haptic feedback create experiences that feel like
not limited to decision making, enhanced learn- real. Virtual reality replaces physical reality with
ing, higher precision, and cost savings. computer-generated content and adds the core
capability of simulation in addition to AR
(Porter and Heppelmann 2017). VR spectrum pro-
Introduction vides several usage patterns based on immersion
and interactivity. Web browser-based non-
Immersive technologies such as augmented real- immersive, noninteractive simple applications
ity and virtual reality have a transformational such as consumption of 360-degree photos and
effect on how complex data and information is videos are at the lower end of the spectrum. At the
visualized. This trend has led to several applica- same time, head-mounted device (HMD) based
tions across industries to adopt immersive visu- experiences with controllers and 6 of freedom
alizations. The reality-virtuality continuum has provide complete immersion and real-time inter-
served as a reference framework for classifying active experiences. The hardware requirements

Immersive Visualizations Using Augmented Reality and Virtual Reality, Fig. 1 Reality-virtuality continuum as a
function of immersion (Milgram et al. 1995)
Immersive Visualizations Using Augmented Reality and Virtual Reality 933

for these experiences vary based on their com- such as AR and VR are being explored by
plexity and application (Pangilinan et al. 2019). researchers and businesses.
Mixed reality, typically known as MR, uses a
combination of augmented and virtual reality to Need for Immersive Visualization
create engaging experiences. The extended real- There are various reasons data visualization is
ity, typically known as XR, is an umbrella term increasingly becoming more critical in recent
that covers augmented, virtual, and mixed reality. times. With growing complexity and volume of
It also encompasses other supporting technologies data, aggregation and creating insightful or
such as AI (Artificial Intelligence), 5G, IoT delightful experiences is a significant challenge.
(Internet of Things) to create engaging and inter- The purpose of visualization varies based on
active applications (Pangilinan et al. 2019). applications, and some of the objectives are facil-
AR and VR have wide-ranging applications itating higher learning, speedier decision making,
across industries. Depending on the need for inter- and keeping the user at the center of data and
activity, immersion, and visualization, the appli- insights. This process must seamlessly work for
cations make use of AR or VR appropriately. general as well as professional users (Schintler
Enterprise sectors such as education, manufactur- and McNeely 2020).
ing, healthcare, military, real estate have seen Decision-making process analyses data to
I
early adoption of AR and VR for applications extract patterns and facilitate the discovery of
such as simulated walkthroughs, remote surgeries, knowledge or insights. Reducing the complexity
combat training, employee training, and more of data to discover trends and anomalies lead to
(Carmigniani et al. 2011; Sicaru et al. 2017). Con- accelerated and more accurate decision making.
sumer segments such as retail, e-commerce, edu- Visual data mining techniques use human visual
cation, gaming, entertainment, and tourism also perception to gain insights from patterns.
make use of AR and VR for applications such as Immersive technologies assist significantly in
virtual try-on for products (Bonetti et al. 2018, this entire process (Moloney et al. 2018).
interactive books, interactive advertisements, Scientific visualizations play an essential role
immersive videos, virtual tours of famous places in facilitating learning and information sharing
(Huang et al. 2016), and 3D interactive lessons among researchers, businesses, and government
Kosa and Uysal 2019). organizations. Applications such as stocks analy-
Since the advent of Internet and mobile sis or sports analytics are usually multi-
penetration, the amount of data and information dimensional and hence difficult to visualize on
processed in every enterprise and consumer appli- existing visualization technologies such as 2D or
cation have grown exponentially. This situation 3D. Entertainment and gaming applications
poses significant challenges for users to make use require greater interactivity to enjoy the overall
of data and information for activities such as deci- experience. Industrial applications, such as
sion making, gaining insights, learning effec- healthcare training or enterprise training, require
tively, and collaborating. One of the critical a sense of physical presence to be more productive
elements that makes data accessible and usable (Sicat et al. 2018). Due to these several complex
for users is the visualization of it. The primary situations, immersive visualizations become a
objective of data and information visualization is tool of preference for personal and enterprise use.
to facilitate learning and decision-making of users Augmented and virtual reality have been
across the personal and organizational spectrum extended from the original continuum to the
(Schintler and McNeely 2020). multidimensional realm. Now it covers a sense
As the complexity and volume of the informa- of presence, level of interactivity, and the mecha-
tion and data are increasing exponentially, the nism to achieve these through the embodiment of
traditional methods of 2D and 3D visualizations suitable devices (Flavián et al. 2019). In other
on flat screens of computers or phones are not words, the primary characteristics of immersive
sufficient enough. Hence, emerging technologies technologies such as fidelity with spatial, visual,
934 Immersive Visualizations Using Augmented Reality and Virtual Reality

and aural senses of humans make it perfectly improve product placements and, hence, increase
suitable for complex data and information visual- sales.
ization (Moloney et al. 2018).
Manufacturing/Production
The training and manufacturing process of assem-
Applications of AR and VR for Immersive bling desktop computers is enhanced with AR to
Visualizations convert instructions manual into a step-by-step
visual guide. The proposed AR app is a simplified
This section presents a summary of applications process to follow for new employees without los-
from various industries which use immersive ing the task’s context. The clarity of instructions
visualizations through AR and VR to facilitate leads to reduced errors and lesser training costs
decision making, learning, or experiences incurred by the organizations. Thus, productivity
effectively. is higher, and product quality is also enhanced
(Osborne and Mavers 2019).
Retail By combining the manufacturing industry with
Virtual Try-On of clothing or accessories through VR technology, enterprises can remain competi-
augmented reality is picking up among consumers tive by utilizing quality inputs in new product
as well as retailers. An in-store augmented mirror developments. In manufacturing enterprises, by
setup provides users with a digital catalogue to its visualization and immersive experience, VR
try-on for clothing and accessories. The proposed helps in aggregating relevant information and
AR application gives an enhanced shopping expe- enables faster decision-making during the product
rience compared to the physical model leading to development processes. A thorough analysis
higher customer satisfaction due to the speedier shows the increasing use of VR technologies and
decision-making process for shopping. As a further research being conducted to increase the
result, physical retail stores can provide greater practical use of VR technologies (Choi
choice to users similar to online methods and et al. 2015).
augment it with a sense of a physical presence
(Bonetti et al. 2018). Education
A study was conducted that ingested data from A virtual tour of heritage locations provides
neuro-scientific tools such as tracking eye move- immersive history and cultural experiences to stu-
ments, storing navigation, and selecting the dents. In one of the examples, a 360-degree virtual
brands in a virtual supermarket, performing data simulation is created at the historical center of the
analysis, and thereby gaining insights about con- city of Rethymno in Greece to teach ancient his-
sumer choices, customer experience, and shop- tory and culture. A virtual reality application gives
ping behavior in a store. Qualitative analysis was learners a higher attention span due to a virtual
also used to compare the choices of consumers environment free of distractions. The additional
and the subsequent outcomes and sales. The over- sense of physical presence helps to facilitate
all suggestions correlated high attention devoted greater understanding and engagement to the
to a brand and slow eye movements (between learners. The level of interactivity fostered
brands), to additional brand purchases within the through the VR application evokes higher interest
category. The outcome of the results meant that in learners than just consuming content over the
that the additional brand choices drive the time screen (Argyriou et al. 2020).
buyers spent on the first choice. Hence, less time
available for the first selection leads to additional Real Estate
purchases within the product category and 4D CAD models and immersive visualizations are
increased sales (Bigne et al. 2015). This is a used in structural steel framing operations to
good use-case of how a VR environment was showcase the design and construction process to
used to identify retail consumers’ behaviors to all stakeholders for approvals and updates.
Immersive Visualizations Using Augmented Reality and Virtual Reality 935

Through effective visualizations, construction Tourism

processes and operations are streamlined pro- Tourist destination marketing is using virtual real-
cesses. The outcome means the decision-making ity to spread interesting information among cus-
process is effective due to lesser ambiguity tomers. Creating interactive and informative
reduced errors and greater participation and virtual worlds to market tourist destinations is
engagement of all stakeholders (Kamat attracting customers. This method of using
et al. 2010). immersive visualizations provides higher con-
sumer engagement and more credible marketing
Healthcare efforts resulting in less ambiguity for customers.
Patient care and training in the poststroke phase Hence, marketers receive a higher satisfaction rate
are one of the healthcare use cases using from their consumers as the immersive visualiza-
immersive technologies. The rehabilitation of tion of destination gives a more realistic look and
body-part functioning (such as hands) is enhanced feel of what to expect. The outcome is that the
through AR assisted devices for common tasks customers make their decisions faster (Huang
such as reaching out and grasping objects. The et al. 2016; Huang and Liao 2015).
approach involving AR helps give haptic feed- Augmented reality applications are enhancing
back, and the interactivity helps in accelerating the overall tourism experience by providing more
I
the patients’ rehabilitation process. The process is context-specific information in a museum. The
highly scalable due to the re-use of the virtual tourists could use their smartphone camera and
environment resulting in cost saving for patient augmented reality application to know interesting
care. As AR systems are easy to set up, the manual information such as history and facts about each
operations effort is reduced (Xun et al. 2005) article presented in a museum. This experience
(Fig. 2). significantly adds value to the tourist’s knowledge
The surgical workplace is transforming with in a very engaging way without losing the context
the help of emerging technologies like VR and and language barriers in a foreign location (Yung
AR. VR applications play an important role in and Catheryn 2019).
healthcare by enhancing the medical use of data
such as anatomy, intraoperative surgery, or post- Wellness
operative rehabilitation. This process is acceler- Virtual reality and interactive, immersive games
ated by an increase in the availability and speed of are being used to increase mindfulness. Its appli-
VR hardware and a reduction in price (Khor et al. cations include therapy for pain management and
2016) (Fig. 3). mental wellbeing. The VR approach offers

Immersive Visualizations
Using Augmented
Reality and Virtual
Reality,
Fig. 2 Rehabilitation
process of the stroke
patients using AR apps.
(Xun et al. 2005)
936 Immersive Visualizations Using Augmented Reality and Virtual Reality

Immersive Visualizations
Using Augmented
Reality and Virtual
Reality, Fig. 3 Surgeons
using Google Glass in
Operating theatres. (Khor
et al. 2016)

contextual visualizations leading to lower mental account of data and information to an experience
stress and perceived pain. The immersive that heightened the learning as well as entertain-
approach avoids distractions from the real-world ment of the consumer. One of the limitations in
to assist in the healing process. This ultimately this approach is shorter usage span of virtual real-
leads to higher satisfaction among patients and ity due to headset issues, hence short, and bite-
speedier recovery (Kosa and Uysal 2018). sized documentaries are popularly used for such
experiences (Sirkkunen and Uskali 2016).
Journalism
Interactive prints and immersive storytelling are Defense
revolutionizing the journalism industry. Military training uses virtual reality to simulate
Augmenting the print media formats with digital battleground and teach strategic tactics through
content through AR and immersive storytelling interactive training for the forces. The VR simu-
makes news more effective and engaging. The lations enhance both the navigation and coordina-
immersive storytelling approach makes for more tion capabilities among soldiers. This method
effective storytelling (via interactive media). The provides a more accurate representation of war
digital augmentation enhances the value of scenarios and difficult situations than usual
existing media such as print to make it more mediums due to its immersive environment and
relevant for the users. It enables the users to get interactivity. The immersive simulations provide
an easier transition from the existing medium to a scalability and flexibility in training content
new medium without losing context. Some of the (Livingston et al. 2011) (Fig. 4).
limitations in this approach are lack of hardware Augmented reality is used as a handy tool to
availability, lack of tools for content creation, and disseminate real-time context-aware information
lack of awareness among consumers (Pavlik and in a military operation. This objective can be
Bridges 2013). achieved through a head-mounted AR glass or a
Immersive storytelling is also experimented handheld device such as a smartphone. The usage
through virtual reality by many leading broadcast- of AR also helps in getting remote assistance or
ing and publishing companies such as The recreation of a specific environment, such as a
New York Times to deliver a first-person account demolished structure. It helps understand minor
of refugees during displacement due to crisis. and vital details about the actual environment and
These short experiences put a consumer in the make rapid decisions in combat situations. Navi-
middle of the action through 360-degree videos gation information is also delivered on an AR
and evoke a strong sense of reality and empathy. headset to guide the personnel making the expe-
The use of VR transforms the news from an rience seamless (Dodevska et al. 2018).
Immersive Visualizations Using Augmented Reality and Virtual Reality 937

Immersive Visualizations Using Augmented Reality and Virtual Reality, Fig. 4 Military training simulation
example, using Virtual Reality
I
Limitations of suitable hardware, software, and content, is also
a challenge (Chandra and Kumar 2018;
There are various technical as well as business Leovaridis and Bahnă 2017; Porter and
challenges in implementing immersive visualiza- Heppelmann 2017).
tions using AR and VR. VR experiences are lim-
ited by the quality of the headset, quality of
software applications, and interactivity issues. Conclusion
Standalone wireless headset suffers due to pro-
cessing capability, weight, field-of-view, and bat- Various applications of AR and VR provide better
tery life. High-end PC based VR headsets are visualizations over traditional methods resulting
difficult to set up and expensive. If not designed in better decision making, cost savings for orga-
well, software applications can deliver a bad expe- nizations, enhanced experiences for consumers,
rience to users, including motion sickness. Inter- and many more. Immersive products provide an
activity is currently limited to the use of edge over the competition for businesses and
controllers and hand gestures, which are still enhance brand value, better stakeholder commu-
evolving. Augmented reality experiences on AR nication, and higher client satisfaction. At times it
glasses are limited by the design, size, weight, also facilitates cost reduction in specific scenar-
battery, and field of view of the headset. ios. Though AR and VR as technology are bene-
Smartphone-based AR experiences are widely ficial and advancing at a rapid pace, there are
used now. However, they do not provide rich, some short-term challenges in implementing
immersive experiences and interactivity. They immersive visualizations. To overcome these
suffer from issues such as dependency on a spe- challenges and make immersive visualizations
cific smartphone model, GPU capabilities of effective, solution providers need to design prod-
smartphone and smartphone hardware such as ucts and services which provide the appropriate
camera and display. Business limitations such as level of presence, interactivity, and immersion
awareness about the technology, cost of adopting based on application. The human-centered design
the technology, and consumer readiness are being needs to be adapted for more human senses as AR
researched for AR and VR implementation. Lack and VR can make use of visual, aural, and spatial
of maturity of the eco-system, such as availability dimensions.
938 Immersive Visualizations Using Augmented Reality and Virtual Reality

Cross-References Kosa, M., Uysal, A., et al. Acceptance of Virtual Reality

Games: A Multi-Theory Approach. International Jour-
nal of Gaming and Computer-Mediated Simulations
▶ Mindfulness, Virtual Reality, and Video Games (IJGCMS), 12(1), 43–70 (2020). https://doi.org/10.
4018/IJGCMS.2020010103
Leovaridis, C., Bahnă, M.: Aspects regarding virtual real-
References ity as innovation in creative industries. Rev Romana
Soc. 28, (2017)
Livingston, M., Rosenblum, L., Brown, D., Schmidt, G.,
Argyriou, L., Economou, D., Bouki, V.: Design methodol- Julier, S., Baillot, Y., Swan, J., Ai, Z., Maassel, P.:
ogy for 360 immersive video applications: the case Military Applications of Augmented Reality. Springer,
study of a cultural heritage virtual tour. Pers. Ubiquit. New York (2011). https://doi.org/10.1007/978-1-4614-
Comput. (2020). https://doi.org/10.1007/s00779-020- 0064-6_31
01373-8 Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Aug-
Bigne, E., Llinares, C., Torrecilla Moreno, C.: Elapsed mented reality: a class of displays on the reality-
time on first buying triggers brand choices within a virtuality continuum. Telemanipulator Telepresence
category: a virtual reality-based study. J. Bus. Res. 69, Technologies. 2351, 282–292 (1995). https://doi.org/
(2015). https://doi.org/10.1016/j.jbusres.2015.10.119 10.1117/12.197321
Bonetti, F., Warnaby, G., Quinn, L.: Augmented reality and Moloney, J., Spehar, B., Globa, A., et al.: The affordance of
virtual reality in physical and online retailing: a review, virtual reality to enable the sensory representation of
synthesis and research agenda. In: Jung, T., Claudia, multi-dimensional data for immersive analytics: from
M. (eds.) Augmented Reality and Virtual Reality, experience to insight. J Big Data. 5, 53 (2018). https://
pp. 119–132. Springer, Cham (2018) doi.org/10.1186/s40537-018-0158-z
Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P., Osborne, M., Mavers, S.: Integrating augmented reality in
Damiani, E., Ivkovic, M.: Augmented reality technol- training and industrial applications. In: 2019 Eighth
ogies, systems and applications. Multimed. Tools Appl. International Conference on Educational Innovation
51(1), 341–377 (2011) through Technology (EITT). IEEE, Biloxi (2019).
Chandra, S., Kumar, K.N.: Exploring factors influencing https://doi.org/10.1109/EITT.2019.00035
organizational adoption of augmented reality in Pangilinan, E., Lukas, S., Mohan, V.: Creating Augmented
e-commerce: empirical analysis using technology-- and Virtual Realities: Theory and Practice for Next-
organization-environment model. J. Electron. Commer. Generation Spatial Computing. O’Reilly Media, Inc,
Res. 19(3), (2018) Sebastopol (2019)
Choi, S., Jung, K., Do Noh, S.: Virtual reality applications Pavlik, J.V., Bridges, F.: The emergence of augmented
in manufacturing industries: past research, present find- reality (AR) as a storytelling medium in journalism.
ings, and future directions. Concurr. Eng. 23, (2015). J Comm Monog. 15(1), 4–59 (2013)
https://doi.org/10.1177/1063293X14568814 Porter, M.E., Heppelmann, J.E.: Why every organization
Dodevska, Z., Mihic, M., Manasijevic, S.: The role of needs an augmented reality strategy. Harv. Bus. Rev.
augmented reality in defensive activities, (2018) 95(6), 46–57 (2017)
Flavián, C., Ibáñez-Sánchez, S., Orús, C.: The impact of Schintler, A.L., McNeely, C.L.: Encyclopedia of Big Data.
virtual, augmented and mixed reality technologies on Springer, Cham (2020)
the customer experience. J. Bus. Res. 100, 547–560 Sicaru, I.A., Ciocianu, C.G., Boiangiu, C.A.: A survey on
(2019) augmented reality. J Inf Syst Oper Manag. 11(2), (2017)
Huang, T.L., Liao, S.: A model of acceptance of Sicat, R., Li, J., Choi, J., Cordeil, M., Jeong, W.K., Bach,
augmented-reality interactive technology: the moderat- B., Pfister, H.: Dxr: a toolkit for building immersive
ing role of cognitive innovativeness. Electron. data visualizations. IEEE Trans. Vis. Comput. Graph.
Commer. Res. 15(2), 269–295 (2015) 25(1), 715–725 (2018)
Huang, Y.C., Backman, K.F., Backman, S.J., Chang, L.L.: Sirkkunen, E., Uskali, T.: Journalism in virtual reality:
Exploring the implications of virtual reality technology opportunities and future research challenges. In: Pro-
in tourism marketing: an integrated research frame- ceedings of the 20th International Academic Mindtrek
work. Int. J. Tour. Res. 18(2), 116–128 (2016) Conference (2016)
Kamat, V., Golparvar-Fard, M., Martinez, J., Peña-Mora, Xun, L., Kline, T., Fischer, H.C., Stubblefield, K.A., Ken-
F., Fischer, M., Savarese, S.: CEC: research in visual- yon, R.V., Kamper, D.G.: Integration of augmented
ization techniques for field construction. J Const Eng reality and assistive devices for post-stroke hand open-
Manag. 137, (2010). https://doi.org/10.1061/(ASCE) ing rehabilitation. In: 2005 IEEE Engineering in Med-
CO.1943-7862.0000262 icine and Biology 27th Annual Conference, Shanghai
Khor, W., Baker, B., Amin, K., Chan, A., Patel, K., Wong, (2005)
J.: Augmented and virtual reality in surgery-the digital Yung, R., Catheryn, K.-L.: New realities: a systematic
surgical environment: applications, limitations and literature review on virtual reality and augmented real-
legal pitfalls. Ann Translat Med. 4, 454–454 (2016). ity in tourism research. Curr. Issue Tour. 22(17),
https://doi.org/10.21037/atm.2016.12.23 2056–2081 (2019)
Incremental Games 939

These examples have inspired numerous sub-

Incremental Games sequent games utilizing a similar dynamic, includ-
ing Clicker Heroes (2014), Plantera (2016), and
Mikhail Fiadotau the educational Rebuild the Universe (2015).
Centre of Excellence in Media Innovation and In the span of a few years, incremental games
Digital Culture (MEDIT), Tallinn University, have become an established genre, even as multi-
Tallinn, Estonia ple terms continue to be used to denote this genre,
focusing on different aspects of it: incremental
game, idle game, clicker game. Explaining the
Synonyms rising popularity of the genre, game critic Nathan
Grayson pointed out that due to not demanding
Clicker game; Idle game constant attention from the player, incremental
games were easy to integrate into a daily routine
and could be played even at one’s workplace
Definitions (Grayson 2015).

Incremental game (otherwise known as idle game

I
or clicker game) is a resource management game Origins
whose core mechanic consists of repeatedly
performing a simple action, such as clicking a The idea of a game playing itself goes back to at
button, in order to gain currency. A prominent least J.H. Conway’s Life (1970), where the player
feature of incremental games is that after an initial sets up the initial configuration of the game in
time investment by the player, automated cur- the beginning, but has no influence over the sub-
rency production is usually unlocked, allowing sequent progression of the game (Björk and Juul
the game to continue in the background with 2012). More recently, Godville (2010) was a nota-
little player interaction for extended periods ble game to have entirely optional player interac-
of time. tion, casting the player as a god overseeing a
world in which the nonplayer controlled hero
adventures.
Introduction In the same year, Cow Clicker, a game that
completely relied on repetitive clicking to pro-
The year 2013 saw the rise to prominence of gress was released by American researcher and
several early incremental games such as Cookie game designer Ian Bogost. Cow Clicker is a
Clicker, Candy Box!, and A Dark Room. Facebook game with a minimalist mechanic: the
While displaying significant variation in terms of player clicks on a cow in order to obtain currency
their aesthetics and storylines, all three relied on (clicks); however, they need to either wait for 6 h
the same core mechanic of continuously click- to click again or make a microtransaction to skip
ing to accumulate currency (cookies, candies, or the waiting time. Bogost explained Cow Clicker
wood in the respective examples) in order to as a satire of contemporary social games, which
exchange it for other game objects and resour- rely on the freemium distribution model and
ces. The speed of accumulation would gradually impose various limitations on the player in order
increase and after a few initial interactions with to encourage them to make in-game purchases
the game, the accumulation process would (Bogost 2012).
become automated, enabling the player to leave Despite its stripped-down gameplay and satir-
the game running in the background and return to ical nature, Cow Clicker gained significant popu-
it occasionally to click for additional currency or larity and inspired a number of similar games.
spend it on upgrades, artifacts, and access to new Most notably, it became the inspiration for
types of resources (Deterding 2016). aniwey’s Candy Box! and Julien Thiennot’s
940 Incremental Games

Cookie Clicker, which were among the earliest Unlocking new content requires increasing
incremental games proper. amounts of currency.

This logic of increasing complexity means

Structure that beyond the core dynamic of incremental
resource accumulation, idle games can rely on
Incremental games share the following key elements of different genres. For example, in
features: Candy Box! and its sequel Candy Box 2 the
main currencies (candy and lollipops) are mainly
1. Their core dynamic is based on accumulating used to unlock new locations and obtain items
currency. such as weapons and armor, which enables the
2. Initially, this is achieved by the player player character to go on adventures and com-
performing a certain repetitive action such as plete quests. This makes Candy Box! fundamen-
clicking. tally a role-playing game. By contrast, A Dark
3. After some player input, currency accumula- Room features a much bigger variety of resources
tion can be automated, enabling the game to (wood, fur, cloth, meat, steel, coal, medicine,
“play itself” in between interactions with the etc.) and mainly revolves around managing and
player. defending a growing village, making it a strategy
4. The game gradually increases in complexity, game (Fig. 1).
uncovering new features, resources, items, and Some idle games, such as Godville, require no
locations. input from the player whatsoever. These are
5. This new content usually has to be unlocked by known as zero-player games, although it needs
the player as it is not enabled automatically. to be pointed out that just as not all incremental

Incremental Games, Fig. 1 Resource management in A Dark Room

Indie Game 941

games are zero-player, neither are all zero-player https://www.jesperjuul.net/text/zeroplayergames

games incremental. (2012)
Bogost, I.: Cow Clicker: The Making of Obsession
[Online], http://bogost.com/writing/blog/cow_clicker_
1/ (2012)
Significance Deterding, S.: Progress wars: idle games and the demar-
cation of “real” games. In: Proceedings of 1st Interna-
tional Joint Conference of DiGRA and FDG.
Incremental games “break with conventions and Available from: https://library.med.utah.edu/e-
expectations regarding games” (Deterding 2016) channel/wp-content/uploads/2017/02/paper_267.pdf
and have led scholars to question existing defini- (2016)
tions of video games, as these often revolve Grayson, N.: Clicker Games Are Suddenly Everywhere On
Steam [Online], https://steamed.kotaku.com/clicker-
around player agency, which is limited in idle games-are-suddenly-everywhere-on-steam-
games (Khaliq and Purkiss 2015). Zero-player 1721131416 (2015)
games in particular are problematic due to the Khaliq, I., Purkiss, B.: A study of interaction in idle games
absence of an obvious player figure (Ruffino & perceptions on the definition of a game. In: IEEE
Games Entertainment Media Conference (GEM),
2016), leading Björk and Juul (2012) to argue Toronto, pp. 1–6. IEEE (2015). https://doi.org/10.
that the idea of a game as something actively 1109/GEM.2015.7377233
played by at least one human does not do justice Ruffino, P.: Games to live with: speculations regarding
I
to the complexity of both the notions of “game” NikeFuel. Digit. Cult. Soc. 2(1), 153–159 (2016).
https://doi.org/10.14361/dcs-2016-0111
and “player.” In Deterding’s (2016) words, while Toffler, A.: The Third Wave. Bantam Books, New York
incremental games “started as an artistic inversion (1981)
of game design conventions to demarcate the
boundary of ‘real’ games,” they ended up
“expanding rather than delimiting the category.”
The popularity of incremental games also Independent Game
reflects the evolution of the ways we use media.
By allowing players to attend to other tasks ▶ Indie Game
while the game plays itself, incremental games
accommodate the increasingly prevalent behav-
ior of media multitasking (Bardhi et al. 2010).
While incessant alternation between a variety of Indian Perspective
information sources was already conceptualized by
Toffler as “blip-culture” (Toffler 1981), the advent ▶ Strategies for Design and Development of Seri-
of broadband Internet and powerful computers has ous Games: Indian Perspective
furthered this process, leading to even shorter atten-
tion spans and more diverse media competing for
audience attention. Incremental games, due to not
demanding much player commitment while pro- Indie Game
viding a sense of progress, fit this paradigm well.
Mikhail Fiadotau
Centre of Excellence in Media Innovation and
References Digital Culture, Tallinn University, Tallinn,
Estonia
Bardhi, F., Rohm, A.J., Sultan, F.: Tuning in and tuning
out: media multitasking among young consumers.
J. Consum. Behav. 9, 316–332 (2010). https://doi.org/
10.1002/cb.320 Synonyms
Björk, S., Juul, J.: Zero-player games. In: Philosophy of
Computer Games Conference, Madrid. Available from: Independent game
942 Indie Game

Definition the grainy, black-and-white Limbo (2010); the

voxel-based 3D sandbox Minecraft (2011); and
An indie game is a game created by an indepen- the lo-fi, full-motion video-based Her Story
dent team of developers, typically with limited (2015). This approach to visual and sound design
resources, without a publisher or with no signifi- serves both a rhetorical and a practical function.
cant involvement in the artistic process on the On the one hand, it “is meant to invoke a type of
publisher’s part. Indie games are often experimen- authenticity <...> that marks it as distinct from
tal and/or employ a retro aesthetic. While the label the alleged realism” of mainstream games (Juul
is usually associated with digital games, it can also 2014); on the other hand, it enables indie devel-
apply to analog (e.g., tabletop) games. opers to deal creatively with their limited budgets
and skills (Lipkin 2012) (Fig. 1).
In addition to the purely aesthetic aspect, indie
Overview games also differ from mainstream big-budget
titles insofar as the former are more open to exper-
The term “indie game” became prominent in the imentation and “aim to push the envelope of game
mid-2000s (Garda and Grabarczyk 2016), design” (O’Donnell 2012). This experimentation
although the concept of “indie” itself dates back can manifest in using innovative mechanics and
to the 1980s (Hesmondhalgh 1999). “Indie” is a having unusual plotlines and settings. For example,
short form for “independent,” indicating creative Gods Will Be Watching (2014) features an unlikely
autonomy which indie producers are deemed to combination of point-and-click adventure with
possess. While the concept originated in the music resource management (the resources being vari-
industry (Hesmondhalgh 1999), it gradually ables relating to the physical and mental well-
expanded to other spheres such as cinema being of its characters). Braid enriches the tradi-
(Newman 2009), fashion (Hauge and Hracs tional puzzle platformer gameplay by adding the
2010), and, eventually, games. ability to rewind time; it is also revealed at the end
The diverse indie media form a loose contin- that the hero’s quest was a metaphor for the inven-
uum of “indie culture,” which “derives its identity tion of the atomic bomb (Jagoda 2013). Text-based
from challenging the mainstream” (Newman indie games by such authors as Anna Anthropy and
2009: 16) and generally embraces lo-fi aesthetics Porpentine have dealt with queer identities and
(Lipkin 2012). However, there is little systematic gender dysphoria, topics not commonly addressed
interaction between the different media within the by mainstream games. That Dragon, Cancer
spectrum of indie culture, with each of them hav- (2014) is an autobiographical indie game designed
ing its own conventions and audiences which do by Ryan and Amy Green to communicate their
not necessarily overlap. experience of caring for, and ultimately losing, a
In case of indie games, their rhetorical opposi- child diagnosed with cancer. These examples show
tion to mainstream gaming has resulted in a ten- the wide thematic range of indie gaming (Fig. 2).
dency to use pixelated retro graphics and chiptune
music evoking the look and feel of old video
games. Notable indie games such as Cave Story Evolution of the Concept
(2005), Fez (2012), Hotline Miami (2012), and
Stardew Valley (2016) can all be seen as homages When “indie games” first became an established
to “classic” video games of the 1980s and early term in the mid-2000s, it was often conceptualized
1990s, even as they subvert or enrich the conven- in terms of the parameters of production such as
tions of older games. Even the indie titles that do budget, team size, revenue, and distribution
not directly utilize retro aesthetics tend to have a model. Consider one of the early definitions of
distinct visual style which looks minimalist com- indie games in an academic paper: “Indie games
pared to the hyperrealistic graphics of most main- are video games that have a small or non-existent
stream games (at least in the West). This is the budget and are often primarily available online or
case for hand-painted platformer Braid (2008); through friend-to-friend sharing” (El-Sattar
Indie Game 943

Indie Game, Fig. 1 Fez (2012) utilizes the retro platformer aesthetic, but subverts it by adding a third dimension

the twenty-first century (Garda and Grabarczyk

2016).
The circumstances of production are thus less
central to defining indie games than the particular
temporal and cultural context of their origin. The
emergence of indie games was made possible by
the availability of consumer-grade game engines
such as Game Maker, RPG Maker, and Unity
(Garda and Grabarczyk 2016), as well as the rise
of digital distribution enabled by widespread
access to broadband Internet and the shift to
Web 2.0 (Parker 2013).
Moreover, as major corporate players, recog-
nizing the market potential of indie gaming,
started to publish games by indie studios and as
crowdfunding platforms made it possible for indie
Indie Game, Fig. 2 Anna Anthropy’s Dys4ia (2012)
developers to command large budgets, the under-
addresses an issue not commonly seen in digital games:
gender dysphoria standing of what constitutes “indie” has also
begun to shift. While it is still generally recog-
2008). These definitions overlooked the temporal nized that an indie game has to be developed by a
aspect of indie games: while “independent small studio (which may, however, operate on a
games” had existed for decades, designed by stu- large budget and work with a major publisher),
dents in computer labs and bedroom developers in much of the focus is now on a general “indie
their homes, the narrower label “indie” is over- sensibility” and indie aesthetic (Juul 2014). The
whelmingly used to refer to games developed in shift to digital distribution platforms such as
944 Indie Game Design

Steam, where indie games and big-budget main- at the Proceedings of the 9th International Conference on
stream titles coexist within a single ecosystem, the Foundations of Digital Games. Available at https://
www.jesperjuul.net/text/independentstyle/ (2014)
has rendered the circumstances of production Limbo: [video game] Playdead (2010)
nearly invisible, while highlighting the aesthetic Lipkin, N.: Examining indie’s independence: the meaning of
feel and gameplay features which set indie games “indie” games, the politics of production, and mainstream
apart. The notion of an indie game was nebulous cooptation. Loading... 7 (2012). http://journals.sfu.ca/
loading/index.php/loading/article/viewArticle/122
to begin with (Parker 2013), but now indie gaming Minecraft: [video game] Mojang (2011)
seems to have mirrored the development trajec- Newman, M.Z.: Indie culture: in pursuit of the authentic
tory of earlier indie media such as music and film: autonomous alternative. Cine. J. 48, 16–34 (2009)
a shift from a movement characterized by rela- O’Donnell, C.: The North American game industry. In:
Zackariasson, P., Wilson, T. (eds.) The Video Game
tively authentic opposition toward the mainstream Industry: Formation, Present State, and Future,
to an aestheticized label largely incorporated into pp. 99–115. Routledge, New York (2012)
the very mainstream industry it once set out to Parker, F.: Indie game studies year eleven. In: Proceedings
challenge (Lipkin 2012). of the 2013 DiGRA International Conference:
DeFragging Game Studies. Available at http://www.
digra.org/digital-library/publications/indie-game-studies-
year-eleven/ (2013)
Cross-References Stardew Valley: [video game] ConcernedApe, Chucklefish
(2016)
That Dragon, Cancer: [video game] Numinous Games (2016)
▶ Dōjin Game

References
Indie Game Design
Braid: [video game] Number None (2008)
Cave Story: [video game] Studio Pixel (2005)
Dys4ia: [video game] Anna Anthropy, Newgrounds (2012)
▶ Underground Design of Kaizo Games
El-Sattar, H.K.H.A.: A novel interactive computer-based
game framework: from design to implementation. In:
IEEE International Conference on Visualisation,
London, pp. 123–128 (2008)
Gods Will Be Watching: [video game] Deconstructeam,
Indie Game Developers
Devolver Digital (2014)
Garda, M.B., Grabarczyk, P.: Is every indie game indepen- ▶ Underground Design of Kaizo Games
dent? Towards the concept of independent game. Game
Stud. 16, (2016). http://gamestudies.org/1601/articles/
gardagrabarczyk
Fez: [video game] Polytron Corporation, Trapdoor (2009)
Hauge, A., Hracs, B.J.: See the sound, hear the style: Indigenous Knowledge for
collaborative linkages between indie musicians and Mental Health, Data
fashion designers in local scenes. Ind. Innov. 17,
11 3 – 1 2 9 ( 2 0 1 0 ) . h t t p s : / / d o i . o r g / 1 0 . 1 0 8 0 /
Visualization
13662710903573893
Hesmondhalgh, D.: Indie: the institutional politics and Hooria Hajiyan, Shawkh Ibne Rashid and
aesthetics of a popular music genre. Cult. Stud. 13, Mehran Ebrahimi
34–61 (1999)
Her Story: [video game] Sam Barlow (2015)
Faculty of Science, Ontario Tech University,
Hotline Miami: [video game] Dennaton Games, Devolver Oshawa, ON, Canada
Digital (2012)
Jagoda, P.: Fabulously procedural: Braid, historical pro-
cessing, and the videogame sensorium. Am. Lit. 85,
745–779 (2013). https://doi.org/10.1215/00029831-
Synonyms
2367346
Juul, J.: High-tech low-tech authenticity: the creation of inde- Data Visualization of Mental Health Issues;
pendent style at the independent games festival. Presented Mental Health of Indigenous Peoples
Indigenous Knowledge for Mental Health, Data Visualization 945

Definition Data Visualization

Indigenous is a common term used for First There have been various research surveys and
Nations, Inuit, and Métis in Canada. Indigenous studies conducted on the mental health of Indige-
peoples of Canada suffer from mental health nous people. These studies have provided a com-
issues more than their non-Indigenous counter- parison between Indigenous peoples’ mental
parts. According to history, the delocalization of health and their non-Indigenous counterparts’
these people to remote places made them live mental health. We can also see a gender-based
independently, which strained their mental comparison from some of the surveys. Figures 1
health (Kunitz 1996). Identifying the causes of and 2 (Nation n.d. 2010) show the gender-wise
mental health issues in Indigenous peoples and comparison in Indigenous peoples. In Fig.1, we
mitigating them is of dire need. One difficulty can observe that females have more suicidal
with understanding the main reasons for these thoughts than males and this is much smaller in
mental issues is the existence of multiple con- non-Indigenous people than Indigenous people.
tributing factors. These include different barriers We can also observe a higher suicide rate in Indig-
that Indigenous peoples face in accessing enous peoples in Canada, specifically among First
healthcare because of colonization, cultural dis- Nations, compared to the non-Indigenous peoples
I
continuity, and racism and residential schools. (see Fig. 2), where the rate is higher for males than
On the other hand, Indigenous knowledge is females.
still the main remedy for such kinds of mental Figure 3 (Nasir et al. 2018) represents the pres-
problems in Indigenous communities. ence of common mental disorders among Indige-
Representing the common mental problems nous Australians compared to the general
between different groups of Indigenous peoples population of Australia. We can see a similar
and the traditional way of life as a remedy for trend of high mental health issues in the Indige-
each specific group remains a challenge. Data nous population of Australia compared to the non-
visualization has evolved to create visual repre- Indigenous people. In this Figure, the superscripts
sentations of data to help people quickly assim- are indicated as the total Indigenous
ilate large amounts of information. Hence, data Australians % / total National Survey of Mental
visualization can help to identify and assess fac- Health and Wellbeing, (NSMHWB) %.
tors relating to mental issues in different groups
of Indigenous peoples.
Barriers in Accessing Healthcare

Introduction Barriers in accessing healthcare can be defined

into three different categories, Proximal, Interme-
The significant mismatch between their traditional diate, and Distal barriers. Proximal barriers
way of living and the current capitalism-based include geographical living places, lack of local
lifestyle plays a part in the increasing mental educational institutions, and lack of presence of
health cases among the Indigenous peoples. Sur- healthcare professionals that can negatively affect
veys conducted on this issue show the deteriorat- Indigenous peoples accessing primary healthcare
ing picture in terms of mental health in this (Nguyen et al. 2020). Intermediate barriers stem
community compared to their non-Indigenous from a lack of employment opportunities and
peers (Nation n.d. 2010). There are various causes inadequate income. The distal barrier comprises
which have been linked to the deteriorating men- colonialism, racism, and social exclusion. This
tal health condition of the Indigenous people. hampers the involvement of Indigenous peoples
Cultural discontinuity, colonization, and obstacles in policymaking and accessing the proper mental
in accessing healthcare all can contribute to this healthcare (Nguyen et al. 2020). The effects of
condition. Colonization, Cultural Discontinuity, and Racism
946 Indigenous Knowledge for Mental Health, Data Visualization

Indigenous Knowledge
for Mental Health, Data
Visualization,
Fig. 1 Comparing the
prevalence of suicidal
thoughts reported by First
Nations, Métis, Inuit, and
non-Indigenous people

Indigenous Knowledge
for Mental Health, Data
Visualization,
Fig. 2 Comparison of
suicide rate between First
Nations and non-
Indigenous people

Indigenous Knowledge for Mental Health, Data Visualization, Fig. 3 Standardized 30-day, 12-month, and lifetime
prevalence of frequent mental disorders among Indigenous Australians compared with the general Australian population
Indigenous Knowledge for Mental Health, Data Visualization 947

and Residential Schools are still the most signifi- do not recognize the system compatible with their
cant barriers that indigenous peoples are facing thinking (Firestone et al. 2015).
toward mental healthcare in Canada.
Racism and Residential Schools
Colonization The decentralization of Indigenous peoples from
The colonization of Indigenous peoples in Canada their natural habitats to remote locations and ran-
and its impact on their identities are significant dom social groupings forced them to adopt new
when discussing mental health issues of the Indig- ways of life. Relocating them was a part of the
enous community (Stout and Kipling 2003). The political and economic agenda. These policies
mental well-being and recovery of Indigenous served the non-Indigenous people with their inter-
peoples in Canada have always been tied to his- ests, and explicit and precise forms of racism
tory, mainly about dislocation from their tradi- sustained this. These suppression efforts and
tional lands and its consequences (Lavallee and harsh policies resulted from the thought that dif-
Poole 2010). Several studies show how coloniza- ferentiated Indigenous culture as primitive and
tion impacted all aspects of Indigenous peoples’ their people as uncivilized (Kirmayer et al.
lives, including health, tradition, access to ser- 2000). This prevented these people from practic-
vices, and equity, among others (Ristock et al. ing their religion and culture. They were even not
I
2019; MacDonald and Steenbeek 2015). The col- permitted to participate in the democratic govern-
onization of Indigenous peoples in Canada and ment. Therefore, as a means of getting rid of their
the historical impacts on their health, economic, culture and religion, residential schools were
and cultural experiences are well documented formed. Cultural assimilations were imposed on
(Ristock et al. 2019). The results show that the the Indigenous children by making it mandatory
source of low levels of mental well-being among for them to get admitted to these schools. These all
indigenous peoples in Canada stems from the loss played a negative role in the mental well-being of
of traditional practices, breakdown of the family Indigenous peoples and their children specifically
unit, and disconnection from the traditional (Nelson and Wilson 2017).
culture.

Cultural Discontinuity Common Mental Issues among

The cycles of family disruption, colonization, dis- Indigenous Peoples in Canada
location from traditional lands, and outlawing of
spiritual practices among Indigenous peoples First Nations adults experience a disproportionate
have led to many health and social inequities burden of mental well-being and addictions.
(Reading and Wien 2009; Halseth 2013). Rates A study conducted in Hamilton on an urban First
of suicide, depression, and substance abuse are Nations population shows that among the
significantly higher in many Indigenous commu- 554 First Nations adults who participated in the
nities than in the general population (Lafrance study, 42% had been told by a healthcare worker
et al. 2008). Colonization did not end with the that they had psychological and mental health
creation of the new nation-state. In the past, Cana- disorders. Furthermore, 39% of them were
dian government policies, including forced relo- suffering from high rates of depression, 34% had
cation of Indigenous peoples to remote regions, Post-Traumatic Stress Disorder (PTSD), 41% had
residential schools, and bureaucratic control, have suicide ideation, and 51% of attempts were
continued to destroy Indigenous cultures reported (Firestone et al. 2015). PTSD might be
(Kirmayer et al. 2007). This cultural discontinuity linked to residential school attendants, abuse and
has been linked to a high rate of depression, alco- family violence, family disruption by child pro-
holism, suicide, and violence in many communi- tection services, and transgenerational trauma
ties, with the most profound impact on youth. related to these, and other impacts of colonization
We see these traits in Indigenous peoples as they (Menzies 2010; Corrado and Cohen 2003).
948 Indigenous Knowledge for Mental Health, Data Visualization

Another study has pointed out the high rate of colonization has attacked and still affects them
mental disorders among Indigenous peoples (Danto and Walsh 2017). The assault on cultural
(Tait et al. 2017), which states that suicide is a identity has played a significant role in the ill
concern for many Indigenous communities in health of Indigenous peoples, and the spirit has
Canada. The rate of suicide among Indigenous been wounded. In that case, healing activities
peoples in Canada is higher than the national need to include rebuilding their individual and
average. According to 2000 reports from the collective identity. This healing activity consists
Canadian Institute of Health, suicides among of a spiritual understanding of the individual and
First Nations youth aged 15 to 24 were about collaborative Indigenous culture.
five to six times higher than non-Indigenous
youth in Canada (Hossain and Lamb 2019). Indig- Cultural Attachments
enous peoples face a range of mental health issues There is a relationship between cultural attach-
at a higher rate than their non-Indigenous commu- ment and mental health among Indigenous peo-
nities, and their life expectancy is also shorter ples in Canada. Cultural attachment is assessed by
(Frideres 1998; Waldram et al. 2006). The cause involvement in traditional activities and Indige-
of this can be traced to higher rates of death nous languages (Hossain and Lamb 2019). In
among accidental death in young people and sui- addition, culture includes notions of how people
cide. High levels of mental health problems have react to situations, family patterns, and social
been documented in Indigenous communities interactions. Therefore, a new generation of prac-
(Nation n.d. 2010; Waldram et al. 2006). Mental titioners is emerging who can combine local
health issues are directly affecting the social life knowledge about health and healing with the
and economic state of these people. The high rate most valuable aspects of psychology.
of suicide, alcoholism, and violence can be both
causes and effects of mental health problems. Language
Suicide is one of the most dramatic indicators of Language is a primary conveyor of culture and
distress in Indigenous populations. Epidemiolog- people are most readily connected to their emo-
ical surveys undertaken on different communities tions and thoughts in their first language. Behav-
of Indigenous peoples have pointed out the ele- ioral health scientists are assigning high
vated rates of suicide among First Nations, Inuit, importance to understanding a group’s culture
and Métis communities (Frideres 1998). These and traits in terms of treating mental health prob-
studies show that young people have a higher lems. On the other hand, language is one of the
tendency toward suicide and attempted suicide. most vital components of culture and thus can
The results prove that assault on cultural identity impose great possibilities in understanding the
besides low self-esteem is the main factor of this. psychology of a group of people. Various studies
have explored language through its relationship
to other demographic and cultural variables
Indigenous Knowledge for Mental (Gonzalez et al. 2017). Studying different com-
Health ponents of Indigenous language and focusing on
the impact that they can impose on the mental
Some remedies to these mental issues considering health of the Indigenous population can be a
the traditional Indigenous way of life include means of bolstering the mental stability of this
Identity, Cultural Attachments, Languages, Com- community.
munity, Demographics, and Physical Activities
and Games (Danto and Walsh 2017). Community
Despite the failures of the deinstitutionalization era
Identity in Canada, the community remains a central con-
Natural healing for Indigenous peoples must cept in mental health policy, clinical practice, and
include work around identity. The cultural identity research. The idea of “community” is present in
of these people is one of the primary aspects that mental health policy, research, and clinical practice
Indigenous Knowledge for Mental Health, Data Visualization 949

in Canada (Gonzalez et al. 2017). The concept of children and introduce their culture to non-
community in treating mental well-being problems Indigenous peoples (Dubnewick et al. 2018).
has a long history in the West, including thoughts Community-based participatory research was
about healing communities. The healing power of conducted in 2018 (Dubnewick et al. 2018) to
social relationships and the health impacts of the better understand how participation in traditional
social and physical environment were central to games can enhance the sport experiences, and
this perspective. The optimism behind this should further the mental well-being of Indigenous
turn us toward community-based mental health youth. Eight Indigenous youth (14–18 years) and
services (Kirmayer et al. 2000). 10 adults living in various communities in the
Northwest Territories, Canada, participated in
Demographics either a one-on-one interview or a group inter-
When we address such critical issues as mental view. Data were analyzed using an inductive con-
health, we should consider different needs and tent analysis approach, and findings suggest that
approaches for groups of people based on age, participating in traditional games can enhance the
race, etc. For example, based on the references, sport experiences of Indigenous youth by
there is a high rate of suicide among Indigenous (a) promoting cultural pride, (b) interacting with
youths, especially among First Nations, which is Elders, (c) supporting connection to the land,
I
related to unemployment and economic issues (d) developing personal characteristics, and
(Tait et al. 2017; Hossain and Lamb 2019). (e) developing a foundation for movement. Par-
ticipating in activities such as traditional games,
Physical Activities and Games which incorporate traditional ways and Indige-
The physical activity program is the best way to nous values, may provide a unique opportunity
understand Indigenous culture and identity. to enhance the sport experiences of Indigenous
A high sense of belonging to the community and youth. That being said, traditional games can pro-
physical activity is associated with improved vide Indigenous peoples with the opportunity to
mental health and spiritual growth, specifically engage in sport within an environment that
in older people (Bailey and McLaren 2005). Stud- reflects their cultures (Heritage 2005). Traditional
ies show that having the ability and motivation to games, including Inuit games (e.g., two-foot high
belong in groups and doing physical activities kick) and Dene games (e.g., stick pull), are
with others contributed to the mental well-being strongly influenced by life on the land whereby
of retirees (Bailey and McLaren 2005; Waldram such games optimize the endurance, strength, and
et al. 2006). The sense of belonging may need to agility that were, at one time, necessary for sur-
be facilitated to enhance mental well-being. The vival (Heine 2006, 2007). Traditional games were
results indicate that higher levels of sense of central to the lives of Indigenous peoples (Heine
belonging are related to lower depressive symp- 2013).
toms related to suicide ideation (Lavallée 2007). However, with European colonization and the
Traditional ways of hunting and gathering foods “broader assimilative agenda in Canada,” there
can help the Indigenous peoples improve their was an effort to eradicate the traditional sports
mental well-being. It is worth mentioning that and games of Indigenous peoples. More recently
some healing ceremonies can improve their men- a number of computer game development compa-
tal well-being, including Pipe Ceremony, Wed- nies have created new computer games while
ding Ceremony, Naming Ceremony, Sweat advocating for technology across the Indigenous
Lodge, Full Moon, Pow Wow, and Smudging. world. For example, Honour Water is an Anishi-
One of the appropriate ways to increase self- naabe singing game for healing water which is
esteem and happiness among children is by developed by Pinnguaq, a Nunavut and British
engaging them in games. It refers to common Columbia based corporation (see Fig. 4). At the
physical activities/games that Indigenous and root of their mission statement is the embracement
non-Indigenous children can play together, of technology as a means of unifying and enabling
which improves mental well-being in Indigenous both Indigenous and non-Indigenous cultures.
950 Indigenous Knowledge for Mental Health, Data Visualization

Indigenous Knowledge for Mental Health, Data Technology Inc. Songs by Anishinaabe elders and Sharon
Visualization, Fig. 4 Screenshots of the Honour Water Day sung by the Oshkii Giizhik Singers pass on teachings
game on App Store- Apple Inc. (Courtesy of Pinnguaq about water in Anishinaabemowin)

More research would be required to evaluate the aspects of their lives such as language, culture,
success of such computer games towards this traditional activities, and mental well-being. Sim-
mission. ilar to Indigenous populations of other parts of the
world, Indigenous peoples in Canada have suf-
fered adversaries leading to a very different way
Conclusion of living. Declining population due to epidemics
of infectious diseases, religious conversion,
Colonization and relocation of Indigenous peo- enforced colonization, and separation from family
ples in Canada to remote places affected different members have all played a part in the changed
Indigenous Knowledge for Mental Health, Data Visualization 951

lifestyle of Indigenous peoples. The traits and Close the Wellness Gap. National Collaborating Centre
nature of Indigenous groups are different and for Aboriginal Health, Prince George (2013)
Heine, M.: Dene Games: an Instruction and Resource
should not be generalized in terms of understand- Manual, 2nd edn. Sport North Foundation, Yellowknife
ing their culture to treat mental problems. This (2006)
will help us to have a better understanding of Heine, M.: Inuit Games: an Instruction and Resource Man-
their needs and thoughts. Equity of well-being ual, 3rd edn. Sport North Foundation, Yellowknife
(2007)
for the Indigenous population is vital for Heine, M.: Performance indicators: aboriginal games at the
constructing an equal and just society. To achieve Arctic winter games. In: Aboriginal Peoples and Sport
this goal, an analysis of the history and its impact in Canada: Historical Foundations and Contemporary
on the Indigenous mental well-being, such as the Issues, Vancouver, Canada: UBC Press,
pp. 160–181 (2013)
common mental issues and barriers to accessing Heritage, C.: Sport Canada’s Policy on Aboriginal Peo-
healthcare, should be studied and visualized based ples’ Participation in Sport, p. 1. Minister of Public
on the main factors, including Colonization, Cul- Works and Government Services Canada, Ottawa
tural Discontinuity, and Racism and Residential (2005)
Hossain, B., Lamb, L.: Cultural attachment and wellbeing
School. Indigenous knowledge is a way of under- among Canada’s Indigenous people: a rural urban
standing their needs and improving their mental divide. J. Happiness Stud., 1–22 (2019)
health around Identity, Cultural Attachments, Kirmayer, L.J., Brass, G.M., Tait, C.L.: The mental health
of aboriginal peoples: transformations of identity and I
Languages, Community, Demographics, and
community. Can. J. Psychiatr. 45(7), 607–616 (2000)
Physical Activities. Kirmayer, L.J., Brass, G.M., Holton, T., Paul, K., Simpson,
C., Tait, C.: Suicide Among Aboriginal People in
Canada. Aboriginal Healing Foundation, Ottawa
(2007)
References Kunitz, S.: Public health then and now. Am. J. Public
Health. 86(1464), 10 (1996)
Bailey, M., McLaren, S.: Physical activity alone and with Lafrance, J., Bodor, R., Bastien, B.: 12 synchronicity or
others as predictors of sense of belonging and mental serendipity? Aboriginal wisdom and childhood resil-
health in retirees. Aging Ment. Health. 9(1), ience. In: Resilience in Action. Toronto: University of
82–90 (2005) Toronto Press (2008)
Corrado, R.R., Cohen, I.M.: Mental Health Profiles for a Lavallée, L.: Physical activity and healing through the
Sample of British Columbia’s Aboriginal Survivors of medicine wheel. Pimatisiwin. 5(1), 127–153 (2007)
the Canadian Residential School System. Aboriginal Lavallee, L.F., Poole, J.M.: Beyond recovery: colonization,
Healing Foundation, Ottawa (2003) health and healing for Indigenous people in Canada.
Danto, D., Walsh, R.: Mental health perceptions and prac- Int. J. Ment. Heal. Addict. 8(2), 271–281 (2010).
tices of a Cree Community in Northern Ontario: a https://doi.org/10.1007/s11469-009-9239-8
qualitative study. Int. J. Ment. Heal. Addict. 15(4), MacDonald, C., Steenbeek, A.: The impact of colonization
725–737 (2017) and Western assimilation on health and wellbeing of
Dubnewick, M., Hopper, T., Spence, J.C., McHugh, T.L.F.: Canadian aboriginal people. Int. J. Region. Local Hist.
“There’s a cultural pride through our games”: enhanc- 10(1), 32–46 (2015)
ing the sport experiences of Indigenous youth in Menzies, P.: Intergenerational trauma from a mental health
Canada through participation in traditional games. perspective. Native Social Work Journal, 7, 63–85
J. Sport Soc. Issues. 42(4), 207–226 (2018) (2010)
Firestone, M., Smylie, J., Maracle, S., McKnight, C., Nasir, B.F., Toombs, M.R., Kondalsamy-Chennakesavan,
Spiller, M., O’Campo, P.: Mental health and substance S., Kisely, S., Gill, N.S., Black, E., Hayman, N., et al.:
use in an urban first nations population in Hamilton, Common mental disorders among Indigenous people
Ontario. Can. J. Public Health. 106(6), e375– living in regional, remote and metropolitan Australia:
e381 (2015) a cross-sectional study. BMJ Open. 8(6),
Frideres, J.S.: Aboriginal Peoples in Canada: Contempo- e020196 (2018)
rary Conflicts. Prince Hall, Scarborough (1998) Nation, D.: First Nations Regional Health Survey Report
Gonzalez, M.B., Aronson, B.D., Kellar, S., Walls, M.L., 2008–2010. 2012. Dene Nation, Yellowknife (n.d.)
Greenfield, B.L.: Language as a facilitator of cultural Nelson, S.E., Wilson, K.: The mental health of Indigenous
connection. Ab-Original J. Indigenous Stud. First peoples in Canada: a critical review of research.
Nations’ First Peoples’ Cult. 1(2), 176 (2017) Soc. Sci. Med. 176, 93–112 (2017)
Halseth, R.: Aboriginal Women in Canada: Gender, Socio- Nguyen, N.H., Subhan, F.B., Williams, K., Chan, C.B.:
Economic Determinants of Health, and Initiatives to Barriers and mitigating strategies to healthcare access
952 Indigenous Language Revitalization with Stories and Games

in Indigenous communities of Canada: a narrative Introduction

review. Healthcare (Switzerland). 8(2), 112 (2020)
Reading, C.L., Wien, F.: Health Inequalities and Social
Determinants of Aboriginal Peoples’ Health. National The technological revolution has transformed
Collaborating Centre for Aboriginal Health, Prince almost every aspect of life including novel educa-
George (2009) tional systems. Digital language learning technol-
Ristock, J., Zoccole, A., Passante, L., Potskin, J.: Impacts ogy has revolutionized language learning within
of colonization on Indigenous two-spirit/LGBTQ
Canadians’ experiences of migration, mobility and only a few decades and it leads to becoming a
relationship violence. Sexualities. 22(5–6), prominent mode to study and learn a new lan-
767–784 (2019) guage. It is no longer necessary to browse through
Stout, M.D., Kipling, G.D.: Aboriginal People, Resilience heavy bidirectional dictionaries and textbooks as
and the Residential School Legacy. Aboriginal Healing
Foundation, Ottawa (2003) a tool to learn a new language. The availability of
Tait, C.L., Butt, P., Henry, R., Bland, R.: ‘Our next gener- language data, the increasing demand from lan-
ation’: moving towards a surveillance and prevention guage learners, and greater access to native lan-
framework for youth suicide in Saskatchewan first guage speakers have enabled many innovations in
nations and Métis populations. Can. J. Commun.
Ment. Health. 36(1), 55–65 (2017) the fields of linguistics and language learning.
Waldram, J.B., Herring, A., Kue Young, T.: Aboriginal However, it is far more recent to apply these
Health in Canada: Historical, Cultural, and Epidemio- resources and technologies to the Indigenous lan-
logical Perspectives. University of Toronto Press, guages of people from around the world. Tradi-
Toronto (2006)
tionally, language learning resources and
technologies have been built mainly with the pur-
pose of teaching language aimed at tourism, busi-
ness, and employment. Indigenous language
Indigenous Language technologies, on the other hand, are primarily
Revitalization with Stories aimed at language documentation, revitalization,
and Games and reclamation. As a consequence, since Indige-
nous languages possess unique characteristics, the
Aref Abedjooy, Fatemeh Hirbodvash and current language technologies and methods may
Mehran Ebrahimi not be directly applicable in this context as a
Faculty of Science, Ontario Tech University, straightforward process. Indigenous language
Oshawa, ON, Canada technologies have received increased support in
Canada over recent years. However, greater
opportunities require more discussion about
Synonyms what these technologies can look like and how
they will be implemented in practice (Brinklow
First nations; Mobile applications; Revival of et al. 2019).
Indigenous languages

Background
Definition
Individuals use language as a means of commu-
According to Statistics Canada, numerous Indige- nication, education, social interaction, and devel-
nous languages are spoken in Canada. However, opment. That being said, language is also deeply
the number of speakers of these languages is related to identity, cultural heritage, tradition, and
declining due to historical and social factors. memory. Hence, when a language is lost, cultural
Efforts are underway to recognize the importance, heritage and identity are lost and may be lost
preservation, and revitalization of these languages. forever or not be recoverable at all (Oladimeji
A careful evaluation of the related challenges, et al. 2020). According to the United Nations
potential solutions, and benefits is essential. Educational, Scientific and Cultural Organization
Indigenous Language Revitalization with Stories and Games 953

(UNESCO), it is estimated that at least 40% of the Ojibwa Nation. Around 1700, the Mississaugas
7000 languages used worldwide are at risk of of Scugog Island moved from their previous home
extinction. The use of Indigenous languages in north of Lake Huron into southern Ontario. The
Canada has been suppressed for generations due Ojibwe language is the most prevalent Indigenous
to the history of restrictive colonial policies. Many language in the area, which serves as a focus of this
Indigenous languages are presently endangered study (O’Donnell and Anderson 2017).
since most of them are not taught or used in The authors of this entry are not Indigenous
schools, universities, industries, and business. peoples and acknowledge that they are not in a
The year 2019 was designated the international position to speak to the values or views of the
year of Indigenous languages by the United Indigenous communities or their culture. This
Nations, to emphasize the importance of revitali- entry aims to offer a view of possible options
zation of Indigenous languages at the regional, for designing a mobile app for learning Indige-
national, and international levels. In Canada, one nous languages. It is important to mention that
of the items listed in the Truth and Reconciliation elders can always provide more valuable guid-
Commission’s Call to Action was that the federal ance on how to meet the needs of their
government should acknowledge Indigenous lan- communities.
guages as an inherent right of Indigenous peoples
I
and support to “preserve, revitalize and
strengthen” these (Justice Laws 2019). In the Literature Review
2016 census in Canada, nearly 70 Indigenous lan-
guages were reported. There are tens of thousands In 2019, Morgan Cassels and Chloe Farr
of speakers for several of these languages, while published a review paper on mobile applications
others have a few tens or hundreds of mostly for Indigenous language (Cassels and Farr
elderly speakers (Anderson 2018). Thus, Indige- 2019). The authors discuss whether mobile
nous languages can be said to be endangered. apps can strengthen language revitalization
Based on 2016 statistics, 260,550 Indigenous peo- efforts and debate the advantages and disadvan-
ples can speak an Indigenous language well tages of using apps as a medium for Indigenous
enough to conduct a conversation (O’Donnell language learning. Furthermore, 32 apps are
and Anderson 2017). On the other hand, Indige- examined and concluded that some apps focus
nous peoples’ language and culture are dependent on a specific goal or strategy while others offer a
on each other since their cultures are rooted in oral wide variety of different activities, such as
traditions. By oral tradition, Indigenous peoples stories and games. According to the authors,
transmit their epic poems, prayers, speeches, spir- most Indigenous language apps provide users
itual teachings, songs, stories, and history. The with a mixture of dictionary features, common
oral tradition usually transfers culture from one phrases, vocabulary exercises, games, quizzes,
generation to the next through elders or older and cultural content.
people. The oral tradition makes the importance In another study, Oladosu Oladimeji et al.
of maintaining and revitalizing Indigenous lan- examine how technology can play a role in revi-
guages more apparent. talizing culture and language in acute areas
The study in this entry suggests a user-centered (Oladimeji et al. 2020). They notice that individ-
application for the Indigenous Education and Cul- uals must show interest in the language to learn
tural Services at Ontario Tech University that sup- it. A new way of learning the Yoruba language is
ports Indigenous languages. Ontario Tech presented by using mobile games to arouse the
University is on the Traditional Territory of the interest of students and thereby further promote
Mississaugas of Scugog Island and the territory is and revitalize the Yoruba language and culture.
covered by the Williams Treaties. The Mis- A survey questionnaire was used by developers
sissaugas is one of the branches of the largest to assess the quality of the mobile application
Indigenous groups in Canada which is known as that was installed by participants who explored
954 Indigenous Language Revitalization with Stories and Games

the application. The games are ranked based on shame to speak their language (Brinklow
some metrics including extensibility, security, et al. 2019).
ease of use, and whether the user finds it inter- English and French are the two official lan-
esting to learn Yoruba after playing it. According guages of Canada. According to the 2016 census,
to their result, 76% of respondents found the it reported that more than 95% of employees used
game easy to use, 70% rated it as sufficiently English or French at work regularly (Lepage and
extensible, and 90% found it gratifying to learn Corbeil 2017). Furthermore, English and French
Yoruba after playing the game. It will be an are used in schools, universities, business, etc.,
entertaining and interesting method of motivat- and can be considered the languages of success.
ing people to develop an interest in learning the By viewing this reality, it becomes clearer why
native Indigenous language individually or in some Indigenous peoples are reluctant to learn or
groups. speak their language.
In a more recent work, the National Research Finally, there is no doubt that an aging mother
Council of Canada (NRC) conducted a project tongue population contributes to the extinction of
known as “Indigenous Languages Technology Indigenous languages. If the Indigenous lan-
(ILT)” (Kuhn et al. 2020). Phase I of this project guages are not taught to the next generation,
allocated $6 million in funding by the Govern- they will disappear.
ment of Canada in March 2017 and Phase II of the
project is currently underway. The ILT’s project
aims to produce software that will facilitate Indig- Possible Solution
enous language preservation and revitalization
efforts throughout Canada. The authors mention Several efforts are underway to prevent Indige-
that several technologies were developed in nous languages from extinction. With the help of
response to community demands, and the project technology, Indigenous languages can be revived,
was split up into subprojects including text-based reclaimed, and supported (Oladimeji et al. 2020).
and speech-based components. The authors indi- In this paper, designing a mobile game language-
cate that a significant challenge in their project learning application based on Indigenous culture
was in building respectful relationships with with Indigenous community involvement, espe-
communities. cially among elders, is suggested.

The Importance of Connecting with the Elders

Issues Seeking support from the elders is crucial to
ensure language instruction is consistent with the
Many institutional and societal factors have con- standards and the dialect of the community. The
tributed to the decline of Indigenous languages in language and Indigenous peoples’ lives are
Canada, such as the development of residential connected (Desmoulins et al. 2019).
schools and the Indian Act. Assimilation of colo-
nial culture was the explicit purpose of Canada’s Advantages of Mobile Game Apps for
residential school system. This aim was achieved if Language Learning
Indigenous language and culture were not taught to A mobile device can be an excellent tool for learn-
generations of Indigenous children. These schools ing a language due to several reasons. First, the
housed children forcibly taken from their homes. If portability of mobile apps is one of the main advan-
they spoke their mother tongue there, they were tages. It allows users to use their free time more
punished. Thus, generations of Indigenous peoples productively. Second, a mobile app may be helpful
turned away from their heritage and language. In to those who are uncertain about how to pronounce
1996, after over 150 years, these schools were fully vowels correctly or are afraid of being mistaken.
closed. As a result of the residential school system, Third, a great advantage of mobile apps is that they
many Indigenous peoples may still feel a sense of give learners the choice of owning the learning
Indigenous Language Revitalization with Stories and Games 955

tool, rather than enrolling in expensive classes or • There are various dialects within Indigenous
using traditional study materials. Fourth, mobile languages. Still, there is a lack of references to
applications can provide a fun and entertaining whether the orthographic standards were
learning experience (Cassels and Farr 2019). developed for all dialects or if only one dialect
The popularity of video games and mobile is represented in the sources and articles.
games has grown considerably in recent years • There are several orthographies used to write
and their popularity among young people makes the Indigenous languages. Even though the
them a potentially useful tool in education orthographic union (unification) of the lan-
(Oladimeji et al. 2020). Hence, game apps can guage leads to a general improvement, there
be a valuable tool to revitalize Indigenous lan- are still significant differences between lan-
guages. Score points of players in a certain guages based on their location or age.
mobile application game could be a measure of
learning. It can enable the users to understand Challenges Regarding Mobile Game Apps
the reasoning behind obtaining a low score and In addition to many other uses, mobile game apps
utilize it to have a better learning experience can be useful tools for language revitalization
following repetition (Godoy 2020). Games that efforts. Indigenous language learning apps can
use a learning scenario-based approach involve be effective when language acquisition with cul-
I
the gamers in a problem-based setting to solve tural themes is combined. Hence, it is vital to
problems. Hence, the players will be required to consider how it can be designed pedagogical
make decisions about integrated questions, activities and tools in a manner to respect the
experiment with the options offered, interpret culture, community, worldview, protocols, and
feedback, and adapt to new learning methods physical environments of Indigenous peoples
for new insight and skill development (Godoy (Cassels and Farr 2019). A technology-based
2020). community initiative must deal with the rapid
pace of technological change. It is also common
for apps to be updated periodically, like other
General Challenges software products. Those who want to use the
app must have a device that is compatible with
Indigenous language technology development the current version. This compatibility issue may
will be faced with many challenges and problems. make it difficult for individuals to keep up-to-date
Here, there are two distinct types of challenges, devices since maintaining their devices may be
one associated with the Indigenous languages and expensive (Cassels and Farr 2019).
another one related to mobile game apps.

Challenges Regarding Indigenous Languages Games

There are several reasons why dealing with Indig-
enous languages can be challenging. Some of To revitalize Indigenous languages, many efforts
these challenges are listed below (Littell et al. are being made to use traditional games or mobile
2018). games. These games are helping to keep these
languages alive and provide a valuable resource
• The morphological complexity of Indigenous for those who would like to learn and preserve
languages is typically very high. these precious cultures. The following are some
• Often, a single word conveys the meaning of examples of mobile game apps for indigenous
what might otherwise be a complete clause in languages in Canada; see Fig. 1.
English or French.
• It is challenging to get a digitized text or sound • Anishinaabemowin Niwasa (Thornton Media,
available for training in most Indigenous Inc. 2019): This game is designed to teach
languages. players the Anishinaabe language. It features
956 Indigenous Language Revitalization with Stories and Games

Indigenous Language
Revitalization with
Stories and Games,
Fig. 1 Images from mobile
game app for indigenous
languages in Canada: Left:
Anishinaabemowin Niwasa
(Thornton Media,
Inc. 2019); right: Beginner
Ojibwe (shotgun.
experiments 2018)

a series of mini-games that help players learn teachings and stories to help players gain a
vocabulary and grammar in a fun and deeper understanding of the Cree people and
interactive way. their traditions.
• Beginner Ojibwe (shotgun.experiments 2018):
This app is designed for those new to the
Ojibwe language or those who are enthusiastic Such games provide a fun, engaging, and
about learning it. The app follows a “word of accessible way to learn indigenous languages in
the day” approach, allowing users to select up Canada, and they protect these precious languages
to 12 words per day to learn. It is meant to be a for future generations. They are, however, typi-
straightforward introduction to the language. cally only available for a single operating system
• Ojibway (Baxter 2017): This app teaches users or platform, and their static nature can lead to
the Ojibway language using professional- them being outdated.
quality audio, pictures, and syllabics. It covers
important aspects of the Ojibway culture,
including history, geography, famous people, Design Considerations
and treaties. The app also teaches common
phrases, pronunciation, and syllabic writing. Mobile game apps can help revitalize the Indige-
It provides comprehensive information about nous peoples’ languages if children and youth will
the Ojibway people, including their lifestyle, be taught to speak the language using these. How-
history, and notable figures. ever, to be accepted by its audience, the mobile
• Cree Language (GNWT Official Languages game application has to be entertaining and well-
2016): This game teaches players the Cree designed. The user interface, fast loading time,
language through a series of word puzzles high performance, and compatibility with differ-
and challenges. It also includes cultural ent mobile platforms are key factors here. If a
Inductive Learning 957

mobile game app did not meet these criteria, it Baxter, D.: Ojibway [iOS]. Apple Store. https://apps.apple.
would barely succeed. com/ca/app/ojibway/id477459816 (2017)
Brinklow, N.T., Littell, P., Lothian, D., Pine, A., Souter, H.:
The term cross-platform in mobile game appli- Indigenous language technologies & language recla-
cations refers to the fact that users of different mation in Canada. In: 1st International Conference on
hardware or operating systems can play the same Language Technologies for All, Paris. Proceedings of
game simultaneously. Therefore, it is very impor- the 1st International Conference on Language Technol-
ogies for All, pp. 402–406. European Language
tant to have a mobile game app that can run on Resources Association (ELRA) (December 2019)
Android, iOS, and other common operating sys- Cassels, M., Farr, C.: Mobile applications for indigenous
tems and hardware. language learning: Literature review and app survey.
One of the best ways to develop a cross- Working Pap. Linguist. Circle Univ. Victoria. 29(1),
1–24 (2019)
platform app is by using game engines. Game Desmoulins, L., Oskineegish, M., Jaggard, K.: Imagining
engines are the core software needed for a game university/community collaborations as third spaces to
to run properly. It is possible to have various support indigenous language revitalization. Lang. Lit-
versions of a game app for different platforms eracy. 21(4), 45–67 (2019)
GNWT Official Languages: Cree Language [Andriod].
after developing a game using a game engine. Google Play. https://play.google.com/store/apps/
Developing a cross-platform mobile game details?id¼com.languagepal.androidvancecreeversion
app using a game engine to teach the Ojibwe &hl¼en (2016)
Godoy Jr., C.H.: A review of game-based mobile I
language would be an excellent idea.
e-learning applications. Int. J. Comput. Sci. Res. 4(3),
Due to the need for an Ojibwe language- 340–350 (2020)
learning mobile app to be scenario-based, some Kuhn, R., et al.: The indigenous languages technology pro-
stories from this nation can be used in the process ject at NRC Canada: An empowerment-oriented
of app design and development. Concepts such as approach to developing language software. In: Proceed-
ings of the 28th International Conference on Computa-
treaty rights, tribal sovereignty, wild rice, and tional Linguistics, pp. 5866–5878 (December 2020)
spearfishing are a few examples that can help Lepage, J.F., Corbeil, J.P.: Languages used in the work-
mobile game apps to be better adjusted to the place in Canada. Statistics Canada Catalogue
community’s culture. no. 98-200-X2016031 (29 November 2017)
Littell, P., Kazantseva, A., Kuhn, R., Pine, A., Arppe, A., Cox,
C., Junker, M.O.: Indigenous language technologies in
Canada: Assessment, challenges, and successes. In: Pro-
Conclusion and Discussion ceedings of The 27th International Conference on Com-
putational Linguistics, pp. 2620–2632 (August 2018)
O’Donnell, V., Anderson, T.: The aboriginal languages of
In conclusion, scenario-based mobile game appli- first nations people, métis and inuit. Statistics Canada,
cations can play an important role to promote and Catalogue no. 98-200-X (25 October 2017)
preserve Indigenous languages and cultures by Oladimeji, O., Olorunfemi, T., Oladimeji, O.: Promoting
honoring the Indigenous way of life. It is also interest in learning Yorùbá language using mobile game.
J. Inf. Technol. Comput. Sci. 3(5), 293–301 (2020).
clear that interaction with the Indigenous commu- https://doi.org/10.25126/jitecs.202053232
nity, in particular with the elders, and receiving Shotgun.experiments: Beginner Ojibwe [Andriod]. Google
feedback from them can enable an application to Play. https://play.google.com/store/apps/details?id¼com.
be suited to their needs. Hopefully, this will help shex.beginnerojibwe&hl¼en_CA&gl¼US (2018)
Thornton Media, Inc.: Anishinaabemowin Niwasa
to promote the awareness of the Indigenous com- [Andriod]. Google Play. https://play.google.com/store/
munity, especially the new generation and youth, apps/details?id¼com.languagepal.androidniwasa&
about their language. hl¼en_CA&gl¼US (2019)

References
Inductive Learning
Anderson, T.: Results from the 2016 Census: Aboriginal
languages and the role of second-language acquisition.
Statistics Canada, Catalogue no. 75-008-X ▶ PBL-Based Industry-Academia Game Devel-
(7 December 2018) opment Education
958 Industrial Robot Augmented Reality

is usually distinguished by the high level of

Industrial Robot Augmented immersion and interactivity offered.
Reality Information Presentation Method: The way infor-
mation is conveyed to users in virtual world. It
▶ Augmented Reality for Human-Robot Interac- can be in various forms such as animation,
tion in Industry video, visuals, text, audio, and pictographs.
Serious Game: A video game with a purpose
beyond entertainment such as teaching users
a new skill or training users to improve their
Industry 4.0 existing skills on a subject.

▶ Design Framework for Learning to Support

Industry 4.0 Introduction

After early 2010s with the launch of more afford-

able and higher fidelity new generation headsets,
virtual reality (VR) has become a very popular
Information Presentation
medium for various areas such as games, training,
Methods
and rehabilitation. Despite its increasing preva-
lence and proven success in many topics, there
▶ Information Presentation Methods in Virtual
are still a lot of unknowns and unexplored areas
Reality
related to virtual reality. One of these areas that
has not been much explored yet is information
presentation methods in virtual reality. Informa-
tion presentation is important in the sense that it
Information Presentation informs users about what they need to do in the
Methods in Virtual Reality virtual world. Virtual reality offers a high degree
of visualization, but it is unknown whether three
Lal “Lila” Bozgeyikli dimensional instructions work better than two
School of Information, University of Arizona, dimensional, for example. If not prepared appro-
Tucson, AZ, USA priately, information presentation may affect user
experience negatively and cause frustration in
users. Thus, information presentation method is
Synonyms an important aspect of virtual reality. In this entry,
researches conducted in this area so far are
Information presentation methods; Serious presented along with future directions and
games; Tutorials; Video games; Virtual reality conclusions.

Definitions Dynamic and Static Information

Presentation Methods in VR
Virtual Reality: A model of reality with which
users can interact using senses such as sight, Several studies that had conducted in the early
sound, or touch. years of virtual reality stated that dynamic infor-
Immersive Virtual Reality: A type of virtual real- mation presentations such as animations were
ity in which the user’s complete view is more beneficial in terms of cognitive aspects,
surrounded by the synthetic environment, as than static methods such as text and pictures
if they had stepped inside the virtual world. It (Levie and Lentz 1982; Park and Hopkins 1992;
Information Presentation Methods in Virtual Reality 959

Rieber 1990). Some following studies games with a health focus (Ushaw et al. 2015).
contradicted with these prior findings, concluding The best practices that were found to be suitable
that static methods might have been as effective as for adopting were: brief in-game instructions that
dynamic methods (Hegarty 2004). The reasoning are on-screen (not on a manual or a printed out
behind this was that although the dynamic document) and readily available to the player for
methods might have had advantage over static the duration they like to view them; using pre-
ones in terms of visualization, they also had asso- recorded videos of required in-game actions and
ciated pitfalls such as demanding continuous using these recordings as an in-game tutorial.
attention of the user and presenting information Greyling and Calitz studied on developing a
only for the duration of the included motion. In computerized multimedia tutorial system they
this sense, the static information presentation named “ACCUTUT,” which aimed to train pro-
methods were available for viewing as long as spective higher education students in using
the user wanted to, since they were not associated computerized admission test user interfaces effec-
with any movement duration constraint. The tively (Greyling and Calitz 2003). In this system,
required cognitive load for viewing also was the following instruction methods were used:
lower, since they did not include any dynamic written, verbal, and simplistic icons (brief pic-
elements. A third realm of studies followed these tures). Researchers found out that using such
I
two contradicting views, which combined static assistive information presentation methods
and dynamic information presentation methods instead of requiring students to read large seg-
and examined the effects on user experience. ments of text improved user experience and elim-
The researchers found out that it resulted in dis- inated several interface problems while taking
jointed understanding, as some participants were these admission tests.
not able to figure out how to integrate the two or
when to focus on which one (Ainsworth et al.
2002; Anzai 1991). As a remedy, a following Spatial Information in Information
study suggested that having the user actively par- Presentation
ticipate in the integration of picture-based and
symbolic information presentations provided bet- To assist users with understanding information,
ter user experience and increased understanding some previous studies explored the use of spatial
(Bodemer et al. 2004). The importance of infor- information in VR systems. Bowman et al. looked
mation presentation method not only lies in the into whether using spatial information presenta-
level of understanding it provides the users but tions inside a virtual zoo would stimulate learning
also its effects on the production costs. Dynamic and comprehension in participants or not
methods such as animation and videos are costlier (Bowman et al. 1999). In the virtual zoo system,
and more time consuming to produce, whereas the following information presentation methods
static methods such as pictures or text are usually were used: verbal, text, a few accompanying
less costly and less time consuming. Hence, it is images (only for complex content), and audio
important to understand which information annotations that were triggered automatically if
method in virtual reality would provide the best the users were close to them. Although not eval-
user experience while being cost and time effec- uated in isolation, these assistive information pre-
tive at the same time. sentation methods were observed to increase
users’ understanding of the layout and design on
the virtual environment. The virtual zoo experi-
Tutorials in Serious Games ence enabled students not only to learn the
presented information directly but also to under-
Ushaw et al. looked at the video game industry stand the material better as compared to traditional
and studied which of the best practices could be teaching methods. Likewise, Ragan et al. also
adopted from commercial video games to serious explored effects of supplementary spatial
960 Information Presentation Methods in Virtual Reality

information on user experience (Ragan et al. for aviation safety training (Chittaro and Buttussi
2012). Written and symbolic information presen- 2015). Aggarwal et al. also utilized text-based
tation methods were used in the VR system and instructions and very brief pictures for presenting
participants’ performance on learning-based information in their VR laparoscopic surgery
activities were measured. In one version, informa- training curriculum development studies
tion was presented directly to the participants in (Aggarwal et al. 2009). Corato et al. chose to
the virtual environment, and in the other version, include real-time animation overlays along with
information was wrapped around the participant text-based on-screen instructions as information
on surrounding displays. Experiment results presentation methods in their VR system for
showed that spatial information presentation pro- training surgery staff on hand-washing proce-
vided improved memory scores whereas no dures (Corato et al. 2012).
improvements were observed for higher level
cognitive activities, such as critical thinking.
Conclusion
Commonly Used Information Although some previous studies have examined
Presentation Methods in VR Systems effects of information-presentation methods
on user experience in VR, there are still no well-
Unless they rely solely on human tutors who will
established design principles recommending
explain the users how to use the system and what
which method works best for different aspects
tasks to perform in it, almost all VR systems
such as entertainment, training, learning, memo-
include some form of instructions. In this subsec-
rization, or retention. More studies are needed
tion, we present a compilation of recent VR sys-
to identify effects of information presentation
tems and the information presentation methods
methods on user experience for improved VR
they utilized. These studies did not focus on
experiences.
exploring effects of instruction methods in VR
on user experience; however, observing the com-
mon choices of instruction methods chosen to be
Cross-References
used by the recent studies may shed light on
understanding information presentation methods
▶ Accessibility of Virtual Reality for Persons
better.
with Disabilities
Oliveira et al. utilized text-based instructions
▶ Cognitive Psychology Applied to User Experi-
in their VR system for industrial equipment
ence in Video Games
maintenance training (Oliveira et al. 2007).
▶ Gamification and Serious Games
Galvan-Bobadilla et al. used several different
instruction methods (animated, written, and ver-
bal) in the VR system they developed for training
References
users on maintenance of underground power dis-
tribution lines (Galvan-Bobadilla et al. 2013). Aggarwal, R., Crochet, P., Dias, A., Misra, A., Ziprin, P.,
Stinson and Bowman utilized text-based instruc- Darzi, A.: Development of a virtual reality training
tions in their VR system that aimed to train ath- curriculum for laparoscopic cholecystectomy.
Br. J. Surg. 96(9), 1086–1093 (2009)
letes on handling high-pressure situations Ainsworth, S., Bibby, P., Wood, D.: Examining the effects
(Stinson and Bowman 2014). Carlson et al. of different multiple representational systems in learn-
chose to include prerecorded video-based ing primary mathematics. J. Learn. Sci. 11(1), 25–61
instructions in their VR training system for (2002)
Anzai, Y.: Learning and use of representations for physics
assembly tasks (Carlson et al. 2015). Chittaro
expertise. In: Towards a General Theory of Expertise,
and Buttussi included text-based instructions vol. 30, pp. 64–92. Cambridge University Press,
and very brief pictures in their VR serious game New York (1991)
Integrating Virtual Reality and Augmented Reality into Advertising Campaigns 961

Bodemer, D., Ploetzner, R., Feuerlein, I., Spada, H.: The

active integration of information during learning with Information Visualization
dynamic and interactive visualisations. Learn. Instr.
14(3), 325–341 (2004)
Bowman, D.A., Hodges, L.F., Allison, D., Wineman, J.: ▶ Cognitive Processing of Information Visualiza-
The educational value of an information-rich virtual tion
environment. Presence Teleop. Virt. 8(3), 317–331 ▶ Immersive Visualizations Using Augmented
(1999)
Carlson, P., Peters, A., Gilbert, S.B., Vance, J.M., Luse, A.: Reality and Virtual Reality
Virtual training: learning transfer of assembly tasks. ▶ Mixed Reality and Immersive Data Visualiza-
IEEE Trans. Vis. Comput. Graph. 21(6), 770–782 (2015) tion
Chittaro, L., Buttussi, F.: Assessing knowledge retention of ▶ Scalable Techniques to Visualize Spatiotempo-
an immersive serious game vs. a traditional education
method in aviation safety. IEEE Trans. Vis. Comput. ral Data
Graph. 21(4), 529–538 (2015)
Corato, F., Frucci, M., Baja, G.S.D.: Virtual training of
surgery staff for hand washing procedure. In: Proceed-
ings of the International Working Conference on
Advanced Visual Interfaces, pp. 274–277. ACM, Innovative Technology
Capri Island (2012)
Galvan-Bobadilla, I., Ayala-Garcia, A., Rodriguez- ▶ Exploring Innovative Technology: 2D Image
Gallegos, E., Arroyo-Figueroa, G.: Virtual reality train- I
Based Animation with the iPad
ing system for the maintenance of underground lines in
power distribution system. In: Third International Con-
ference on Innovative Computing Technology
(INTECH), London, United Kingdom, pp. 199–204.
(2013)
Greyling, J.H., Calitz, A.P.: The development of
Integrating Virtual Reality and
a computerised multimedia tutorial system for a diverse Augmented Reality into
student population. In: Proceedings of the 2nd Interna- Advertising Campaigns:
tional Conference on Computer graphics, Virtual Real- History, Technology, and
ity, Visualisation and Interaction in Africa,
pp. 109–116. ACM, Cape Town (2003)
Future Trends
Hegarty, M.: Dynamic visualizations and learning: getting
to the difficult questions. Learn. Instr. 14(3), 343–351 Kenneth C. C. Yang1 and Yowei Kang2
(2004) 1
The University of Texas at El Paso,
Levie, W.H., Lentz, R.: Effects of text illustrations:
a review of research. Educ. Commun. Technol. 30(4),
El Paso, TX, USA
2
195–232 (1982) Kainan University, Taoyuan, Taiwan
Oliveira, D.M., Cao, S.C., Hermida, X.F., Rodriguez, F.
M.: Virtual reality system for industrial training. In:
IEEE International Symposium on Industrial Electron-
ics, Vigo, Spain, pp. 1715–1720. (2007)
Synonyms
Park, O.C., Hopkins, R.: Instructional conditions for using
dynamic visual displays: a review. Instr. Sci. 21(6), Advertising; Augmented reality; Campaign;
427–449 (1992) Virtual reality
Ragan, E.D., Bowman, D.A., Huber, K.J.: Supporting cog-
nitive processing with spatial information presentations
in virtual environments. Virtual Reality. 16(4),
301–314 (2012) Definitions
Rieber, L.P.: Animation in computer-based instruction.
Educ. Technol. Res. Dev. 38(1), 77–86 (1990)
Stinson, C., Bowman, D.A.: Feasibility of training athletes
Augmented reality (AR) is a simulated, but
for high-pressure situations using virtual reality. IEEE enhanced, reality that combines both computer-
Trans. Vis. Comput. Graph. 20(4), 606–615 (2014) generated virtual and real-world data to enable
Ushaw, G., Davison, R., Eyre, J., Morgan, G.: Adopting users to perform real-time interactions with dig-
best practices from the games industry in development
of serious games for health. In: Proceedings of the 5th
ital graphics, imagery, and objects, in a seamless
International Conference on Digital Health 2015, way and with an illusion of these layers of
pp. 1–8. ACM, Florence (2015) information coexisting in the same space.
962 Integrating Virtual Reality and Augmented Reality into Advertising Campaigns

Mixed reality (MR) refers to an umbrella concept allows consumers to constantly interact and
from the reality-virtuality continuum to allow engage with advertising campaigns (Persaud and
users to experience the key efficiencies of Azhar 2012). The social media population now
both AR and VR technologies. This technol- accounts for 37% of the world’s population or
ogy analyses users’ surrounding real-world 60% of Americans (Jorner 2017). The pervasive
environment before producing a digitally gen- presence of social media further prompts adver-
erated graphical reality. tisers to take advantage of the opportunities in
Virtual reality (VR) is an assembly of technolo- integrating both platforms (Swant 2017a). For
gies that offer immersive, interactive, and example, Facebook debuted its Facebook Spaces
information-rich experiences when users are AR platform to offer its users AR experiences
able to interact with objects and realities in a through an Oculus Rift headset (Swant 2017a).
computerized, simulated, real-time 3D virtual This entry will go over the definitions of AR
environment by employing their senses and and VR, discuss the technological developments
skills. relevant to the advertising industry, examine
major determinants in their adoption by advertis-
ing practitioners, and provide AR/VR campaign
Introduction examples to demonstrate their potential.

A recent research report by the International Data

Corporation (IDC) (2017) forecasts that the global Virtual Reality (VR)
revenue for both augmented reality (AR) and vir-
tual reality (VR) will reach $13.9 billion in 2017, VR is defined as a computer-generated and simu-
a dramatic increase from $6.1 billion in 2016. The lated environment in which users are immersed in a
compound annual growth rate of AR and VR will virtual environment to experience the presence of
reach $143.3 billion in 2020 (IDC 2017). By other objects or realities through interaction with
2020, growth in the online retailing and digital them (Heim 1998; Hsu 2017; Kerrebroeck et al.
game sectors will be the top two applications 2017). VR technologies have been used in military
of AR/VR technologies (IDC 2017). As a result, training as early as 1995 (Kerrebroeck et al. 2017).
AR/VR campaigns have been accepted into major Recent VR applications range from aviation, med-
advertising awards. For example, the Creative ical (Claudio and Maddalena 2014; Hsu 2017),
Arts Emmys granted Fox’s Sleepy Hollow Virtual advertising (Bazilian 2017a, b; Johnson 2017a, b,
Reality Experience (https://www.youtube.com/ c, d), marketing (Heine 2017; Javornik 2016), etc.
watch?v¼peFGMwVQXL0) an interactive As an emerging media platform for advertising
media award for its creative achievement practitioners, VR is characterized with three
(Giardina 2017). key attributes, or “Three I’s”: immersion, interac-
Digital advertising plays a particularly signifi- tivity, and information intensity (Claudio and
cant role in the growing electronic and mobile Maddalena 2014, pp. 2–3). First, VR users are
commerce markets. Its sheer size has prompted fully immersed in a computer-mediated environ-
advertisers to experiment with AR/VR technol- ment through VR headsets such as Sony
ogies (Metz 2015a, b). Past advertising literature PlayStation VR, HTC Vive, Oculus Rift, Google
has observed that AR/VR technologies are able to DayDream View. Scholars have found that
offer the benefits of immersive storytelling, prod- immersive VR experiences often strongly affect
uct demonstration, and emotional arousal to adults’ attitudes, behaviors, and thoughts (Bailey
engage consumers in advertising and marketing and Bailenson 2017). As a psychological state that
campaigns, according to Interactive Advertising is often experienced by VR users in the virtual
Bureau (IAB) (Klie 2016). Furthermore, the rapid environment, immersion is considered to be the
diffusion of smartphone devices also creates most important aspect of VR technologies
a ubiquitous communication environment that (Kerrebroeck et al. 2017).
Integrating Virtual Reality and Augmented Reality into Advertising Campaigns 963

Secondly, interactivity refers to the human- adults (lessonly.com 2016), this challenge may
computer interactions made possible by VR be resolved sooner.
gadgets such as VR helmets that offer high 3D Secondly, other VR researchers have also
sights and sounds, and high-resolution quality, observed other important determinants including
as well as motion-tracking hardware and soft- heightened consumer expectations, availability
ware to interact with the virtual world (Claudio of VR headsets, new ways of thinking in designing
and Maddalena 2014). As defined by Steuer more interactive contents, etc. (Klie 2016). Thirdly,
(1992), this concept refers to “the extent to a country’s telecommunications infrastructure will
which users can participate in modifying form also affect if the immersive potential of VR tech-
and content of a mediated environment in real nologies can be fully realized without data trans-
time” (p. 84). Interactivity of VR campaigns mission delays. Consumers of VR technologies
thus enables consumers to experience individu- will continue to expect higher frame rates and
alized and customized advertising contents. image quality. As a result, data demand of VR
Thirdly, VR also provides ample information devices are expected to exceed those of 4 K.
and functionality that involve all human senses According to Juniper Research, data consumption
(such as hearing, kinematic, proprioceptory, and is predicted to grow from 2,800 PB in 2017 to over
vision) to allow users to meet their entertain- 21,000 PB in 2021 (Laposky 2017). Lastly, the
I
ment and hedonic gratifications (Kerrebroeck future development of VR technologies also rely
et al. 2017). on how VR integrates with other popular platforms
A VR system is composed of several different such as Facebook or WeChat, both of which are
technologies that include (1) a graphic rendering currently developing their own VR platforms
system; (2) gloves, trackers, and user interface (Johnson 2017d; Jorner 2017; Laposky 2017).
to detect and input users’ movements; (3) output
devices to facilitate aural, haptic, and visual inter-
actions in the VR environment; (4) a software to Augmented Reality (AR)
model virtual objectives and to construct data-
bases; and (5) a system to deliver VR sensory Similar to VR, augmented reality (AR) also
stimuli such as visual display technology to offer allows users to interact with simulated digital
users interactive and immersive experiences graphics, imagery, and objects that combine both
(Claudio and Maddalena 2014). computer-generated virtual and real-world data to
Advertising practitioners who would like to make real-time interactions possible, with the illu-
take advantage of the potential of VR technolo- sion of co-existing in the same space (Rese et al.
gies are still faced with several challenges. 2017; Van Krevelen and Poelman 2010; Williams
First, the potential of VR technologies in adver- 2009). Researchers have claimed that AR is an
tising lies in their diffusion and the attractiveness important branch of VR because both integrate
of related applications. According to a survey virtual digital information into a 3D real environ-
of 811 American adults by YuMe, a majority of ment in real time (Chen et al. 2016). Therefore,
consumers have heard about (around 47% of sur- the definitions of AR center on its virtuality,
vey participants), but cost has been ranked as the enhanced telepresence and flow experience, and
most important determinant of VR adoption a sense of immersion. For example, Carmigniani
(Baumgartner 2016). Similarly, Bazilian (2017b) and Furth (2011) defined AR as “a real time direct
also concurred in another survey of 1,000 people or indirect view of a physical real-world environ-
between 18 and 64 years old that 37% of the ment that has been enhanced/augmented by
participants decided against buying VR devices adding virtual computer-generated information
after realizing its high cost, and 64% of them to it” (p. 342).
agreed that they would buy VR devices if they Figure 1 best captures how realities are created
were less expensive. Thanks to the popularity in the virtual world and where augmented
of smartphones adopted by 68% of the US reality stands in this popular Reality-Virtuality
964 Integrating Virtual Reality and Augmented Reality into Advertising Campaigns

Integrating Virtual Reality and Augmented Reality the virtual world. (Adapted with automatic permission
into Advertising Campaigns: History, Technology, from Kim et al. 2016, p. 162)
and Future Trends, Fig. 1 A continuum of realities in

(RV) continuum (Milgram et al. 1994). As seen in characteristics have made AR a promising
Fig. 1 (adapted from Kim et al. 2016, p. 162), platform for many advertisers.
mixed reality (MR), or Hybrid Reality, is often As early as 2010, HarperCollins Publishers
used as a stand-along application or can be used as has experimented with AR to promote Irish
an umbrella term that encompasses variations, yet author, Cecelia Athern’s book, The Book of
captures “the core efficiencies” of both AR and Tomorrow, in its campaign (everydayismagical.
VR technologies and applications (Milgram et al. com) (Shields 2010). With the popularity of social
1994; Reality Technologies n.d.; Pando 2017). media platforms, the agency Stickee also
MR include augmented reality, augmented virtu- employed Facebook in this multiplatform cam-
ality, and other mixed reality applications (Reality paign (Shields 2010). Similarly, Ford’s UK cam-
Technologies n.d.). The concept of mixed reality paign (https://www.youtube.com/watch?v¼bl8T
refers to an application by which real and virtual 9oYO5vY) targeted young demographics to
world objects and people are integrated with the encourage their purchase of Ka car model. The
virtual environment where digitally generated and technology allows mobile phone users to view
physical objects are juxtaposed to co-exist and to images superimposed virtually onto another pic-
allow users to interact in real time (Milgram et al. tures in a video kiosk (Clifford-Marsh 2009).
1994; Milgram and Kishino 1994). MR is there- Advertisers often integrated AR with other inte-
fore different from AR because MR analyzes grated marketing communications (IMC) plat-
users’ surrounding environment before projecting forms. In its 2017 AR campaign (http://www.
computer-generated contents and synthetic benjerry.com/flavors/special-stash), Ben & Jerry
objects that users are able to interact with virtually worked the advertising agency, 360i, through
(Pando 2017). In spite of the conceptual differ- Facebook’s latest augmented reality platform
ences, we will use AR and MR interchangeably, to promote its newly launched marshmallow-
following the current practices in the advertising flavored ice cream (Johnson 2017d; Loop 2017).
and marketing industry. This AR campaign also integrated a digital game
As an interactive and immersive media platform to allow players to use a smartphone’s
platform, AR offers a great potential to comple- rear-facing camera by allowing them to catch
ment what traditional advertising platforms are marshmallows falling from the sky. If consumers
not able to offer through its innovative ways to missed more than five marshmallows, the AR
interact with commercially relevant advertising game ended (Johnson 2017d). The ice cream
contents (Javornik 2016). Unlike these media, cones at the bottom of the campaign site tracked
AR is characterized with its “interactivity, virtual- how many marshmallows have been missed
ity (presence of elements of virtual reality), (Johnson 2017d; Loop 2017).
geolocation feature/location specificity, mobility Scholz and Smith (2016) proposed the
(in terms of portability and wearability) and “layer/world metaphor” (p. 149) to describe the
synchronization of virtual and physical/real applications of AR in advertising. Advertisers
(augmentation)” (Javornik 2016, p. 253). These developed digital information layers made up of
Integrating Virtual Reality and Augmented Reality into Advertising Campaigns 965

pictures, texts, and videos that later overlay with immersive mobile AR experiences in advertising,
physical objects (e.g., advertisements, landscapes, entertainment, gaming, retailing, sports, contexts
product packages, street scenes) in the real world to complete their consumer decision-making
(Scholz and Smith 2016). Through digital screens journeys (Heine 2017; Rese et al. 2017;
or holograms on digital out-of-home billboards, Shapiro 2017).
smartphones, or other video installations, uses are Table 1 below offers a good summary of major
able to experience their virtual presence and advantages of AR and VR as two emerging inter-
“hybridized realities” (Scholz and Smith 2016, active technologies. These media characteristics
pp. 149–150) that generate an illusion that they offer potentially promising applications for the
interact seamlessly and in real time with the advertising professionals. For example, interac-
computer-generated objects. tivity will allow advertisers to develop fully inter-
In the following 2014 Pepsi Max Unbelievable active digital ads that allow users to interact and
AR campaign in London (https://www.youtube. modify ad contents. The hypertextuality feature
com/watch?v¼Go9rf9GmYpM), a bus stop was will enable consumers to access product informa-
rigged with an digital display capable of produc- tion embedded in the digital ad and to connect
ing AR experience to daily commuters. The to e-commerce and mobile commerce site to
6-sheet digital screen displayed computer-
I
generated UFOs, giant hostile robot, and escaped
wild tiger images that overlay with live feed of Integrating Virtual Reality and Augmented Reality
the street (Dyakovskaya 2017). This campaign into Advertising Campaigns: History, Technology,
and Future Trends, Table 1 Technical characteristics
generated very positive results with 30% sale
of AR and VR technologies
volume increase, 120,000 mentions, likes, and
shares in major social media channels, and 87% Media characteristics of
interactive technologies Definition
earned view rate (Dyakovskaya 2017). The cam-
Interactivity Machine and personal
paign has demonstrated how a UFO attack sce- interactivity, feature-based
nario was overlaid with the street scenes in or perceived, composed of
London (Dyakovskaya 2017). The potential of control, responsiveness and
two-way communication
AR-enabled billboard can be maximized when
Hypertextuality Potentially high number of
interactivity is embedded in the campaign when linked sources
consumers are able to participate in the interac- Modality Diversity of content
tions with AR contents (Szymczyk 2009). Refer representation
to the campaign video site (https://www.youtube. Connectivity Technological capability of
com/watch?v¼Go9rf9GmYpM) to experience the expanding and sustaining a
role of AR in enhancing consumer engagement. model of network, where
many users can be connected
To conclude this section, Fortune magazine among themselves
predicts that AR will generate $120 billion in Location-Specificity Specificity with which a
revenue by 2020 (Gaudiosi 2015). Gartner technology and its user can
(2010) also predicted that AR could be considered be targeted based on the
precise geolocation
as one of the promising top ten technologies in the
Mobility Portability and wearability
information-communication technologies (ICT) that allow a mobile use
sector (cited in Kim et al. 2016). At the 2015 Virtuality Combination of virtual
Consumer Electronics Show, AR has the potential elements that causes
to “disrupt anything with a screen” (cited in immersion in an
Scholz and Smith 2016, p. 150). The advent environment constructed
with computer graphics and
of mobile AR content is likely to create the digital video
greatest influence on AR applications in the next
Reprinted from Journal of Retailing and Consumer Ser-
5–10 years (Heine 2017; Szymczyk 2009). AR vices, 30, Ana Javornik (2016), pp. 252–261, with permis-
users will be able to personalize their own sion from Elsevier
966 Integrating Virtual Reality and Augmented Reality into Advertising Campaigns

purchase advertised products. In addition, the 2017b). KFC has been proud of its products,
multimodality of both AR and VR technologies claiming “It’s made in the hard way” (KFC,
has blurred the traditional boundaries of print, https://www.kfc.com/about/how-kfc-makes-
electronic, out-of-home media, and emerging chicken). Cook trainees were challenged to
media. As a result, traditional print ads are able interact with the computer-generated objects
to offer videos through AR, VR, or holograms, through an Oculus Rift headset and controller
such as Porsche’s holographic magazine ad and learned the laborious five-step processes
(2016) (https://www.youtube.com/watch?v¼XT of inspecting, rinsing, breading, racking,
hnCB3s8y0). Connectivity has led to the creation and pressure-frying KFC’s chicken products
of social media advertising and electronic word of before placing these virtual objects in Colonel
mouth to facilitate the marketing communication Sanders’ mouth to complete the training
processes. Both location-specificity and mobility (Jardine 2017; Swant 2017b). KFC has
characteristics allow advertisers to send location- planned to use more VR e-learning and
sensitive mobile advertising to consumers’ mobile training contents to help train its cooks in its
devices and create a pervasive mobile environ- multistep and rigorous Chicken Mastery Certi-
ment. Finally, virtuality of the VR and AR plat- fication program (KFC Corporation 2017). VR
forms has generated a potential to integrate with technologies in this campaign have allowed
other media platforms to create fully immersive KFC to communicate its corporate mission
advertising experiences. statement with a fun and hand-on experience.
We used the following VR and AR campaigns As seen in KFC’s press release (2017), “What
to demonstrate the benefits of integrating these excites us is experimenting with new tools and
two emerging technologies into advertising cam- mediums to tell stories. VR became an obvious
paigns to generate positive results. choice to create an immersive experience that
teaches trainees how to make KFC’s Original
Recipe. The escape room concept builds on the
Integrating AR/VR into Advertising pure training and utility of the experience
Campaigns into something that’s also entertaining and
connected to KFC’s iconic founder.” Cam-
• Campaign Case #1: KFC’s “The Hard Way” paign Case #2: NHS Blood and Transplant
Virtual Training Escape Room VR Campaign AR Campaign in UK (2016) (https://www.
(2017) (https://www.youtube.com/watch? youtube.com/watch?v¼-zNWP4lzrJQ)
v¼GAlD0h9vCEc) (https://www.youtube.
com/watch?v¼JX5RmKcO_j8) This AR campaign was launched in Birmingham
• Kentucky Fried Chicken (KFC) launched and London though its interactive out-of-home
its 25-min VR video game campaign, “The digital billboard to change people’s stigma
Hard Way,” that required users to learn how about blood donation (Marketing Week 2016).
to fry chicken thighs and wings before they The campaign was created by agency, 23red
could escape from the virtual Colonel Sanders’ that prides itself to “change behavior for the
secret lodge (KFC Corporation 2017; Swant better” (Marketing Week 2016). This AR cam-
2017b). This VR campaign was developed by paign was an excellent demonstration of how an
Wieden + Kennedy’s emerging technologies AR app could make the best use of its interactive
branch, W + K Lodge (Swant 2017b). Though and multimodal capabilities to create “exper-
not intended for the public, the campaign iential activities” to increase the awareness
was designed to train KFC’s new “screen- of and actual behaviors of blood donations
obsessed” Generation Millennial cooks to (Benjamin 2016).
have a better understanding of the brand’s per- The interactive digital billboard showing a
sonality, according to Bob Das, KFC’s head sick patient includes a strong message, “I need
check in the USA (Jardine 2017; Swant your blood donation. Can you help? Show your
Integrating Virtual Reality and Augmented Reality into Advertising Campaigns 967

support by filing a virtual bag.” Spectators were technologies have been applied to a variety of
given a sticker that could be detected by visual brands, ranging from Elle Magazine (O’Shea
recognition on their smartphone (Benjamin 2016). 2016), social media giant, Twitter (Cohen
Afterwards, the AR app allowed them to donate 2016), Disney’s film promotion (Shaul 2014),
virtually their blood that will ultimately fill out Gatorade (Schultz 2015), etc.
an empty blood bag. Refer to the campaign video As demonstrated in the above AR and VR
link (https://www.youtube.com/watch?v¼-zNW campaigns, consumers are now able to interact
P4lzrJQ) that demonstrates the integration of and engage with advertisers’ messages (Giardina
mobile app with AR technologies in this campaign. 2017; Martin 2017), to avoid future ad blocking
As shown in the campaign video (https://www. (Martin 2017), to build better brand awareness
youtube.com/watch?v¼-zNWP4lzrJQ), the pale- (Martin 2017), to connect consumers to the prod-
looking patient’s complexion changed when the uct through a self-referencing effect (Baek
blood bag was filled to transfuse the blood, repli- et al. 2015), to make people better attached to a
cating the actual process of blood donation. The brand, and to measure campaign effectiveness
digital billboard showed “It is helping already. (Baumgartner 2016). As claimed by many mar-
Just an hour of your time can save or improve keting researchers, enhanced consumer engage-
three lives.” At the end of the virtual blood trans- ment is likely to build long-term customer
I
fusion, the real patient thanked the donors and relationship and brand loyalty and has become a
demonstrated “the power of a blood donation” strategic competitive advantage (Monllos 2017;
(Marketing Week 2016, p. 6). Volunteers were Scholz and Smith 2016).
also present on-site to hand out pamphlets about A sense of immersion in the advertising
blood donations while a blood donation truck was communication process has been found to create
parked nearby. In addition to its AR platform, the consumer engagement and better advertising
NHS Blood and Transparent campaign also effectiveness (Martin 2017). Vibrant’s test cam-
integrated sponsored posts on major social paigns used VR and 360 video to compare those
media and organic social activity (Benjamin using traditional 2D video and found better
2016). The campaign generated positive results effects were generated in terms of interaction
with 583 new donor sign-ups across the 5th day rates (600%), content recall (700%), brand recall
and 77,000 viewing of the support videos. The (2,700%), and product intent (200%) (Martin
success of this campaign lied on its abilities to 2017). VR ads delivered through consumers’
make the most of AR’s technical advantages to mobile devices continue to generate more positive
allow users to fully interact with AR contents effects in terms of interaction rates: 85% of
and subsequently generate the highest level of VR/360 video versus 2.5% of mobile devices
user-campaign/brand engagement (Scholz and (Martin 2017).
Smith 2016). According to a survey of agency clients by
Media Planners and Buyers Insperience, it was
reported that 67% of media buyers and planners
Conclusion were interested in integrating AR and VR into
their digital marketing campaigns. Twenty-nine
The integration of AR and VR technologies into percent of them in the survey had purchased AR
advertising has opened up many opportunities or VR ads for their clients (Martin 2017). In gen-
for advertisers and marketers (Javornik 2016). eral, VR (67%) was favored over AR (17%)
There is no available academic research on in their media buy (Martin 2017). In addition to
which product, service, and company are most their appeal to advertisers, marketing practitioners
suitable to apply AR, VR, or MR technologies in have echoed the same insights from the academic
their campaigns. However, quick keyword researchers. Felix Lajeunesse, Co-Director of
searches on the industry trade publications Felix & Paul Studios, concurred with the impor-
(AdWeek or AdAge) found that these emerging tance of these technologies by saying, “We
968 Integrating Virtual Reality and Augmented Reality into Advertising Campaigns

wanted to create the feeling that you are experien- Bailey, J.O., Bailenson, J.N.: Considering virtual reality in
cing this environment in a very personal way – children’s lives. J. Child. Media. 11, 107–113 (2017)
Baumgartner, J.: Top VR adoption challenge: cost. Broad-
feeling the emotion and connection between them casting & Cable, 12 (2016)
and the environment” (cited in Giardina 2017, Bazilian, E.: Infographic: what consumers really think
p. 70). about VR. AdWeek. http://www.adweek.com/digital/
Despite the hype, the durability of these infographic-what-consumers-really-think-about-vr/
(2017a). Accessed 15 Nov 2017
technologies remains to be a concern since their Bazilian, E.: Time Inc.’s VR guru is forging a new path
early introduction (Williams 2009). Information- through immersive storytelling. AdWeek. http://www.
intensive technologies such as AR and VR adweek.com/digital/time-inc-s-vr-guru-is-forging-a-
will require fast streaming speeds to transmit new-path-through-immersive-storytelling/ (2017b).
Accessed 15 Nov 2017
data to consumers’ devices (Laposky 2017). Benjamin, K.: NHS blood and transplant uses augmented
Therefore, whether AR and VR advertising cam- reality for donor appeal. https://www.campaignlive.co.
paigns can be globalized to other countries with uk/nhs-blood-transplant-uses-augmented-reality-dono
less-developed telecommunications infrastructure r-appeal/%7Bsubjects%7D/article/1395336#EkLCJW
KWI7ILs0yw.99 (2016). Accessed 20 Nov 2017
remains to be an issue. Advertising and marketing Carmigniani, J., Furth, B.: Augmented reality: an over-
research is scarce except for a few empirical view. In: Furth, B. (ed.) Handbook of Augmented
or theoretical studies (Baek et al. 2015; Hopp Reality, pp. 3–46. Springer, Heidelberg (2011)
and Gangadharbatla 2016; Javornik 2016; Chen, P., Peng, Z., Li, D., Yang, L.: An improved aug-
mented reality system based on AR. J. Vis. Commun.
O’Mahony 2015) examining factors to develop Image Represent. 37, 63–69 (2016)
more effective AR/VR campaigns. Programmatic Claudio, P., Maddalena, P.: Overview: virtual reality in
research should be conducted to understand medicine. J. Virtual Worlds Res. 7, 1–34 (2014)
better consumer-, system-, message-, and Clifford-Marsh, E.: Ford Ka’s augmented reality check.
Revolution, 19 (2009)
product-related factors to better integrate these Cohen, D.: Twitter Brings Augmented Reality to “The
technologies into future advertising campaigns. Voice”? AdWeek. http://www.adweek.com/digital/
Furthermore, ethical challenges also emerge twitter-augmented-reality-the-voice/ (2016). Accessed
with “immersive storytelling” about a brand and 18 Dec 2017
Dyakovskaya, A.: How augmented reality (AR) is shaping
manipulated realities by advertisers who use AR content marketing experiences. https://www.youtube.
and VR (Webb 2016). com/watch?v¼Go9rf9GmYpM (2017). Accessed
20 Nov 2017
Gaudiosi, J.: How augmented reality and virtual reality
will generate $150 billion in revenue by 2020. http://
Cross-References fortune.com/2015/04/25/augmented-reality-virtual-rea
lity/ (2015). Accessed 15 Nov 2017
▶ Accessibility of Virtual Reality for Persons Giardina, C.: Tux? Check. Virtual reality goggles? Got’em.
with Disabilities The Hollywood Reporter, 70 (2017)
Gartner’s.: Hype Cycle Special Report Evaluates Maturity
▶ Augmented Reality Entertainment: Taking of 1,800 Technologies. https://www.gartner.com/news
Gaming Out of the Box room/id/1447613 (2010). Accessed 12 Dec 2017
▶ Augmented Reality for Maintenance Heim, M.: Virtual Realism. Oxford University Press,
▶ Interaction with Mobile Augmented Reality New York (1998)
Heine, C.: Apple’s new animojis are the latest sign that
Environments brands need to embrace augmented reality. AdWeek.
▶ Interactive Augmented Reality to Support http://www.adweek.com/digital/apples-new-animojis-
Education are-the-latest-sign-that-brands-need-to-embrace-augm
ented-reality/ (2017). Accessed 15 Nov 2017
Hopp, T., Gangadharbatla, H.: Novelty effects in aug-
mented reality advertising environments: the influence
References of exposure time and self-efficacy. J. Curr. Issues Res.
Advert. 37, 113–130 (2016)
Baek, T.H., Yoo, C.Y., Yoon, S.: The impact of augmented Hsu, W.-Y.: Brain-computer interface connected to tele-
reality on self-brand connections and purchase inten- medicine and telecommunication in virtual reality
tions. In: Proceedings of Academy of American applications. Telematics Inform. 34, 224–238
Advertising, Chicago (2015) (2017)
Integrating Virtual Reality and Augmented Reality into Advertising Campaigns 969

International Data Corporation (IDC): Worldwide ows-with-ben-jerrys-new-facebook-ar-filter/ (2017).

spending on augmented and virtual reality forecast to Accessed 15 Nov 2017
reach $13.9 billion in 2017, according to IDC. https:// Marketing Week: NHS launches first ever augmented
www.idc.com/getdoc.jsp?containerId¼prUS42331217 reality billboard campaign to show power of blood
(2017). Accessed 15 Nov 2017 donations. Marketing Week, 6 https://www.marketing
Jardine, A.: KFC created a VR escape room to teach staff week.com/2016/05/18/nhs-launches-first-ever-augmen
how to fry chicken. AdAge. http://creativity-online. ted-reality-billboard-campaign-to-show-power-of-bloo
com/work/kfc-the-hard-way/52540 (2017). Accessed d-donations/ (2016). Accessed 15 Nov 2017
20 Nov 2017 Martin, E.J.: How virtual and augmented reality ads
Javornik, A.: Augmented reality: research agenda for improve consumer engagement. EContent. 5, 8 (2017)
studying the impact of its media characteristics on Metz, R.: Virtual Reality Advertisements Get in Your Face.
consumer behaviour. J. Retail. Consum. Serv. 30, MIT Technology Review, Retrieved on November 13,
252–261 (2016) 2017 from https://www.technologyreview.com/s/535556/
Johnson, L.: Why brands like L’Oréal and Acura are bet- virtual-reality-advertisements-get-in-your-face/ (2015a)
ting big on augmented reality. AdWeek. http://www. Metz, R.: Augmented Advertising. MIT Technology
adweek.com/digital/why-brands-like-loreal-and-acura- Review 118(3), 21 (2012b)
are-betting-big-on-augmented-reality/ (2017a). Milgram, P., Kishino, F.: A taxonomy of mixed reality
Accessed 15 Nov 2017 visual displays. IEICE Trans. Inf. Syst. E77-D(12),
Johnson, L.: How brands are using AR to engage con- 1321–1329 (1994)
sumers and measure results. AdWeek. http://www. Milgram, P., Takemura, H., Utsumi, A., Kishino, F.:
adweek.com/digital/how-brands-are-using-ar-to-engag Augmented reality: a class of displays on the reality-
e-consumers-and-measure-results/ (2017b). Accessed virtuality continuum. Telemanipulator and Tele- I
15 Nov 2017 presence Technologies, 282–292 (1994)
Johnson, L.: This horror movie’s campaign shows how VR Monllos, K.: Brands are doing more experiential
can affect your body. AdWeek. http://www.adweek. marketing. Here’s how they’re measuring whether it’s
com/digital/this-horror-movies-campaign-shows-how- working. AdWeek. http://www.adweek.com/brand-
vr-can-affect-your-body/ (2017c). Accessed 15 Nov marketing/experiential-can-create-more-meaningful-re
2017 lationships-with-consumers/ (2017). Accessed 15 Nov
Johnson, L.: Ben & Jerry’s created a Facebook AR 2017
filter that challenges you to catch marshmallows in O’Mahony, S.O.: A proposed model for the approach to
your mouth. AdWeek. http://www.adweek.com/digi augmented reality deployment in marketing communi-
tal/ben-jerrys-created-a-facebook-ar-filter-that-challen cations. Procedia. Soc. Behav. Sci. 175, 227–235 (2015)
ges-you-to-catch-marshmallows-in-your-mouth/ O’Shea, C.: Elle Cover Uses Augmented Reality. AdWeek.
(2017d). Accessed 15 Nov 2017 http://www.adweek.com/digital/elle-cover-uses-augme
Jorner, J.: How augmented reality is creating a new target nted-reality/ (2016). Accessed 18 Dec 2017
market on social media. AdWeek. http://www.adweek. Pando, A.: Mixed Reality Will Transform Perceptions.
com/digital/james-jorner-effective-inbound-marketin Forbes. https://www.forbes.com/sites/forbestechcounc
g-guest-post-augmented-reality/ (2017). Accessed il/2017/2012/2015/mixed-reality-will-transform-perce
15 Nov 2017 ptions/#7539aff2478af (2017). Accessed 18 Dec 2017
Kerrebroeck, H.V., Brengman, M., Willems, K.: Escaping Persaud, A., Azhar, I.: Innovative mobile marketing via
the crowd: an experimental study on the impact of a smartphones: are consumers ready? Mark. Intell. Plan.
virtual reality experience in a shopping mall. Comput. 30, 418–443 (2012)
Hum. Behav. 77, 437–450 (2017) Reality Technologies: Mixed Reality. http://www.realityte
KFC Corporation: KFC creates virtual world to train its chnologies.com/mixed-reality (n.d.). Accessed 18 Dec
real-world cooks “The Hard Way”. https://www.prnew 2017
swire.com/news-releases/kfc-creates-virtual-world-to- Rese, A., Baier, D., Geyer-Schulz, A., Schreiber, S.: How
train-its-real-world-cooks-the-hard-way-300509008. augmented reality apps are accepted by consumers: a
html (2017). Accessed 20 Nov 2017 comparative analysis using scales and opinions.
Kim, K., Hwang, J., Zo, H., Lee, H.: Understanding users’ Technol. Forecast. Soc. Chang. 124, 306–319 (2017)
continuance intention toward smartphone augmented Scholz, J., Smith, A.N.: Augmented reality: designing
reality applications. Inf. Dev. 32, 161–174 (2016) immersive experiences that maximize consumer
Klie, L.: Virtual reality to become a true reality. Customer engagement. Bus. Horiz. 59, 149–161 (2016)
Relationship Management, 14 (2016) Schultz, E.J.: Gatorade Lets Viewers Step Up to Bat With
Laposky, J.: A virtual reality data crunch is coming. Twice, Virtual Reality Campaign. AdAge. http://adage.com/
8 (2017) article/digital/gatorade-puts-viewers-bat-virtual-realit
lessonly.com: 2016–2017 Learning Trends Report. https:// y-campaign/300384/ (2015). Accessed 18 Dec 2017
www.lessonly.com/blog/2016-2017-learning-trends- Shapiro, A.: 6 ways marketers can get augmented
report/ (2016). Accessed 15 Nov 2017 reality right. AdWeek. http://www.adweek.com/digi
Loop, E.: Catch marshmallows with Ben & Jerry’s new tal/6-ways-marketers-can-get-augmented-reality-right/
Facebook AR filter! http://redtri.com/catch-marshmall (2017). Accessed 15 Nov 2017
970 Intellectual Disability

Shaul, B.: Disneynature Explore: interact with animals

in this augmented reality app. AdWeek. http://www. Intelligent Dynamic System
adweek.com/digital/disneynature-explore-interact-wit
h-animals-in-this-augmented-reality-app/ (2014).
Accessed 18 Dec 2017 ▶ 2-Simplex Prism as a Cognitive Graphics Tool
Shields, R.: HarperCollins uses augmented reality to pro- for Decision-Making
mote authors. New Media Age, 6 (2010)
Steuer, J.: Defining virtual reality: dimensions determining
telepresence. J. Commun. 42, 73–93 (1992)
Swant, M.: Developers see endless possibilities and pitfalls
with social VR after testing Facebook spaces. AdWeek. Intelligent Transportation
http://www.adweek.com/digital/developers-see-endles Systems
s-possibilities-and-pitfalls-with-social-vr-after-testing-
facebook-spaces/ (2017a). Accessed 15 Nov 2017
Swant, M.: KFC created a wacky, creepy virtual-reality ▶ Potential of Augmented Reality for Intelligent
escape game to train new cooks. AdWeek. http:// Transportation Systems
www.adweek.com/digital/kfc-created-a-wacky-creep ▶ User Interface (UI) in Semiautonomous
y-virtual-reality-escape-game-to-train-new-cooks/
(2017b). Accessed 15 Nov 2017 Vehicles
Szymczyk, M.: 2010: The year of augmented reality?
AdWeek. http://www.adweek.com/brand-marketing/
2010-year-augmented-reality-101138/ (2009).
Accessed 15 Nov 2017
Van Krevelen, R., Poelman, R.: A survey of augmented Intelligent User Interface
reality technologies, applications and limitations. Int.
J. Virtual Real. 9, 1–20 (2010) ▶ Emotion-Based 3D CG Character Behaviors
Webb, A.: Virtually convincing. INC. 72 (2016)
Williams, M.: Advertisers test alimented reality’s
durability. Campaign, 9 (2009)

Intelligent Virtual
Environment
Intellectual Disability
▶ Emotion-Based 3D CG Character Behaviors
▶ Computer Games for People with Disability

Interacting with a Fully

Intellectual Property Simulated Self-Balancing
Bipedal Character in
▶ Computer Games and the Evolution of Digital Augmented and Virtual
Rights Reality

Dominik Borer1, Simone Guggiari1, Robert W.

Intellectual Property Rights Sumner2 and Martin Guay2
1
ETH Zurich, Zürich, Switzerland
2
▶ Computer Games and the Evolution of Digital Disney Research, Zürich, Switzerland
Rights

Synonyms

Intelligent Argent Character - Avatar; SIMBICON - Simple Biped

Control; Embodied Agent - Autonomous Digital
▶ Emotion-Based 3D CG Character Behaviors Character; Augmented Reality; Virtual Reality
Interacting with a Fully Simulated Self-Balancing Bipedal Character 971

Definitions

Simulated characters offer rich and realistic

interactions with users and dynamic environ-
ments. They can be thought of as compliant
robots in the real world. Hence, for every
unique push or perturbation, the character
responds equally in a unique and realistic fash-
ion. As the result, the immersion for the user is
greatly increased and all the more powerful.
This article provides insights on how to make
compelling interactions with a a self-balancing Interacting with a Fully Simulated Self-Balancing
bipedal character in Virtual and Augmented Bipedal Character in Augmented and Virtual Reality,
Reality settings. It also describes a general Fig. 1 Because kinematic skeletons (right) are designed
by different digital artists, the relative rotations qi in the
method to interface a simulated character with parent frames are not consistent across characters and
a traditional skeleton, used in modern game typically do not correspond to the same relative rotations
engines such as Unity and Unreal Engine – qi in an articulated rigid body system (left). In conse-
quence, relative rotations from a simulated character can- I
thereby making simulated characters more
not be applied directly to those of a kinematic skeleton,
accessible. typically used in game engines

Introduction Fig. 1. Hence it is required to transfer the joint

transformations from the ARBS into relative
High-level game engines such as Unity and transforms, expressed in the coordinate system
Unreal Engine make easily accessible state-of- of the KS. This can be formulated as an optimi-
the-art graphics and rendering. However, their zation problem, where a set of features are to
character animation systems are based on the match in the world frame, and the free variables
state machine/blend tree kinematic controller, are the joint rotations. However, this requires
which blends between motion clips. This quickly iterative numerical optimization, which is com-
leads to repetitive animations and reactions, and putationally costly. This article describes a fast
increasing the realism requires exponential analytical algorithm that solves this problem
amounts of motion clips. In the past decades, sequentially, joint-by-joint. With this algorithm,
progress has been made in the area of simulated it is possible to use an external simulation and
characters, which allow to constantly generate control library directly in Unity and have a casual
realistic and novel motions on the fly. Because user experiment with a standard bipedal control-
the representation of an articulated rigid body ler, in particular a type of controller called
system (ARBS) is different from a kinematic SIMBICON (Simple Biped Control, Coros et al.
skeleton (KS) used in games, it is not possible 2010) which combines PD tracking of locomotion
to directly use simulated characters to drive the poses with virtual forces for balancing the center-
character’s motion in the game engine. Kine- of-mass. This article describes two different inter-
matic skeletons are used to deform the charac- actions with the environment usable both in AR
ter’s mesh through linear blend skinning (LBS). and VR: perturbations and terrain adaptation. This
Typically it is possible to set relative rotations to allows to have an AR character walk over real
the skeleton joints (but not the world transforms world terrain or have objects thrown at the charac-
or access the full LBS equations). The problem is ter while having him react in a unique way each
that the articulated rigid body systems and the time. Making these interactions compelling is actu-
kinematic skeletons are often different, and hold ally not trivial, and this article provides insights on
different local coordinate systems, as shown in overcoming these hindrances.
972 Interacting with a Fully Simulated Self-Balancing Bipedal Character

Related Work orientation) on the target character are to match

the source character’s features (Chai and Hodgins
The standard animation system used in game 2005; Choi and Ko 1999; Gleicher 1997; Seol
engines today is the state machine/blend tree et al. 2013).
model which transitions between motion clips Gleicher et al. applied this to whole motions in
(i.e., states) similar to motion graphs (Kovar order to transfer motions between characters of
et al. 2002) and blends between well-aligned different proportions (Gleicher 1997). Solving for
motion clips (i.e., blend trees), such as in Verbs individual poses (one at a time, together with
and Adverbs (Rose et al. 1998). Putting this type smoothness) is less prone to local minima. Also
of kinematic controller together requires exten- many applications do not have access to whole
sive manual labor in order to loop motion clips motion, as is the case with performance animation
and align them properly. Also, for every new (Chai and Hodgins 2005; Seol et al. 2013). The
motion and interaction, a new motion clip must methods above use iterative numerical algorithms
be created and inserted into the graph structure, to solve the optimization problem, while in con-
resulting in exponential growth of the graph. One trast, this article describes an analytical solution –
way to reduce the amount of hand-crafted clips is assuming the same topology and access to the full
to retarget the clips using layered inverse kine- source motion.
matics (IK), as to adjust the position of end- Others have used corresponding motion clips
effectors to new grabbing positions, or contact to compute an explicit transfer function from
points (e.g., NaturalMotion, Ikenema, FinalIK, pose-to-pose (Dontcheva et al. 2003; Yamane
and more). et al. 2010; Rhodin et al. 2014), typically through
While this approach is popular, it lacks fast a regression (Dontcheva et al. 2003) or through a
response to user controls and requires a large shared sub-space (Yamane et al. 2010). However,
manual effort to build and maintain. One idea to an explicit transfer function is never completely
increase the responsiveness is to stich small accurate and leads to drift in visually important
motion clips together online using some esti- features such as foot contacts, hence requiring
mate of a state-dependent cumulative reward additional optimization iterations to fix.
(Arikan et al. 2005; Lee et al. 2010; Levine Simulated characters. Another vision to
et al. 2012). This approach has been used in increase the realism of characters is to embed
practice in the game For Honour (Clavet them in a simulation, as if in the real world. This
2016), along a large data-set of human motion is a long and lasting effort in computer animation,
constructed with motion capture. Recently, as seminal works started with manually crafted
Holden et al. have learned the control parame- controllers for human athletics in 1995 (Hodgins
ters directly with a deep neural network (Holden et al. 1995). Since then, controllers have been
et al. 2017). While motion capture is still mostly specifically engineered for locomotion using sim-
available for human subjects and requires con- plified balance models such as the inverted pen-
siderable time to capture and clean the motion dulum (Yin et al. 2007; Coros et al. 2010; Lee
clips, this approach is gaining traction in games et al. 2010; Coros et al. 2011). While these con-
to increase the realism of character motions. trollers are very robust, they can only track loco-
Unfortunately, it cannot be used directly onto a motion and do not extend easily to other types of
different character if the coordinate systems of motions.
the joints are different. Many have tried to use inverse dynamics to
Motion transfer. Transferring motion from track motion capture, assuming knowledge of
one character or source of motion to another char- the full equations of motion. Given the dynamics
acter has been used and studied in many applica- of the system in generalized coordinates, it is
tions. This is often formulated as an optimization possible to estimate the torques required to
problem where a set of features (position, match acceleration-based targets (e.g., mocap
Interacting with a Fully Simulated Self-Balancing Bipedal Character 973

Interacting with a Fully

Simulated Self-Balancing
Bipedal Character in
Augmented and Virtual
Reality, Fig. 2 Using the
naive joint-to-joint structure
leads to more freedom of
motion for the simulated
ARBS compared to the
KS. In consequence, some
poses in the simulation
cannot be reached by the
kinematic skeleton in the
game, as shown by the
example with the upper
body at the bottom of this
figure

targets) by solving a quadratic program kinematic skeleton (KS) typically used in game
constrained to match dynamics and ground reac- engines and motion authoring software such as
tion forces (Abe et al. 2007; Da Silva et al. 2008; Maya. It is first required to define which type of
Macchietto et al. 2009; Mordatch et al. 2010; de ARBS our transfer algorithm supports. The naive
Lasa et al. 2010; Levine et al. 2012; Rabbani et al. approach to building an ARBS from a KS is to
2014). In practice, this controller requires param- associate joints together. However, this leads to an
eter fine-tuning and feasible objectives. Recently, excess freedom problem. In other words, the
there has been a renewed interest in using rein- ARBS can reach configurations the KS cannot,
forcement learning to learn robust feedback poli- thus preventing from directly transferring the
cies for a great class of motions (Tan et al. 2014; transforms from the former to the later, as shown
Peng et al. 2016; Peng et al. 2017) or guided in Fig. 2. The solution to this problem is to asso-
learning by alternating between open-loop control ciate the KS joints to a rigid body, as shown below
optimization and feedback fitting through regres- in Fig. 3.
sion (Liu et al. 2016). With a suitable ARBS, it is possible to tackle
While physics-based controllers are an excit- the problem of transferring the motion from the
ing approach to character animation, they remain ARBS to the KS. Since animation systems in
out of reach to casual users and interaction game engines typically allow to only set relative
designers that operate in high-level game engines. rotations to the joints, it is required to recover
By providing a familiar interface – the kinematic relative transforms. Note this cannot be simply
skeleton – around these controllers, the possibility computed directly from one and applied to the
to use them in many applications is greatly other, as the parent coordinate systems differ in
increased. each model, as shown in Fig. 1. The main insight
behind this solution is to compute the absolute
world transform, relative to the initial transform
Fast Pose Transfer and transfer this quantity to the KS, and then
recover the local relative joint transforms.
This section describes how to transfer the motion More formally, the root position for the ARBS
from an articulated rigid system (ARBS) to a is denoted as u, the local rotation as q, and the
974 Interacting with a Fully Simulated Self-Balancing Bipedal Character

Interacting with a Fully

Simulated Self-Balancing
Bipedal Character in
Augmented and Virtual
Reality, Fig. 3 Our
solution to avoiding excess
freedom of the ARBS: the
joints on the kinematic
skeleton (KS) correspond to
rigid bodies on the ARBS –
the ARBS should be the
dual of the KS

world rotation as Q. Similarly for the KS, the root

Finally, we recover the KS transform relative to
position is denoted as u, the local and world rota-
the parent:
tion as q and Q, respectively. Time is denoted ⁎(t)
and the index i is a notation abuse to traverse the
qi ðtÞ ¼ Q1
i1 ðtÞQi ðtÞ
joints in a hierarchy with i  1 being the parent
of i.
where Qi1 represents the orientation of the parent
At the beginning, the initial root offset between
joint. Note the root is a special case:
the ARBS and KS is stored, as well as all the
world orientations of the ARBS bodies and KS
joints: Q0 ðtÞ ¼ Q0 ðtÞ ð3Þ

Root off set : Dxð0Þ ¼ uð0Þ  uð0Þ ð1Þ uðtÞ ¼ uðtÞ  Dxð0Þ ð4Þ

ð2Þ This transfer is applied at each frame of the

World orientations : Qi ð0Þ, Qi ð0Þ
animation. The next section describes experi-
ments with transferring the motion from a
Then the process starts from the root joint of
SIMBICON-type of simulated character in VR.
the KS, and sequentially traverses down each
chain of joints/bodies to transfer the rotations
from the ARBS bodies in world, to relative joint
rotations in the KS, as follows:
AR VR Interactions
First, we compute the ARBS rotation relative
Typically simulation and control engines are
to its initial pose in world frame:
implemented in fast native code (C++). How-
1 ever, high-level game engines use high-level
Qi ðtÞ ¼ Qi ð0ÞQi ðtÞ managed or scripting languages such as C# or
Lua. The experiments described in this section
Then we compose the ARBS rotation with the were done with the Unity game engine, which
initial KS world rotation to obtain the current KS offers the possibility to load native libraries as
world transform: plugins.
Objects in the game engine that need to interact
Qi ðtÞ ¼ Qi ðtÞQi ð0Þ with the character require rigid body properties
Interacting with a Fully Simulated Self-Balancing Bipedal Character 975

Interacting with a Fully

Simulated Self-Balancing
Bipedal Character in
Augmented and Virtual
Reality, Fig. 4 We can
augment a real world
environment with the
virtual simulated character.
The simulated character can
interact with real and virtual
objects in the scene. Here
we see virtual cannon balls
shot at the character

and colliders to be given to the simulated. This

works as follows: the user can tag an object as a
simulated object, whose geometry is used to ini-
tialize the mass, moment of inertia, and collider
shape. Given a character mesh and a kinematic
I
skeleton, an ARBS is created and then fixed.
Applications can still use different character
meshes with the same bone topology, ideally
with the same proportions, but note that propor-
tions have no effect on the simulation. For exam-
ple, if the legs are longer, the feet would simply
penetrate the ground visually, but not affect the
behavior of the simulator (Fig. 4).
Designers conducted two types of interac-
tions with a SIMBICON character in AR and
VR. The simulated character reacts to perturba- Brief SIMBICON Description
tions in the environment. However, the control-
ler is designed around a simplified rigid body This controller relies on a simplified analytic bal-
system with simple collision primitives (cubes ance model to perform locomotion. A few inter-
and ellipses), resulting in repetitive collisions polated keyframes define the target poses for a
and reactions. Described next is a solution to walk cycle. To maintain balance, the foot place-
generate more variations and increase the ment position is computed with an inverted pen-
immersion. dulum model, and the corresponding leg shape in
Secondly, the controller seeks to maintain bal- the target pose is determined through an analytical
ance at every step. One benefit of this is the ability inverse kinematics (IK). The torques driving the
to cope with various terrains. It is possible to simulation are computed through PD-control
detect objects in the environment and have the (given the target pose), together with virtual
character walk over different terrain profiles. forces – using the Jacobian transpose – to better
Also, the designers created a new type of game control the velocity and root orientation of the
where the user tries to steer the character towards a character.
goal, by manipulating the ground underneath
the character. But first, before delving into the Perturbations
meat of things, the SIMBICON controller is Users experimented with throwing objects at the
briefly described. simulated character in both AR and VR. While
976 Interacting with a Fully Simulated Self-Balancing Bipedal Character

Interacting with a Fully Simulated Self-Balancing on the left here with cardboard in the shape of a bridge. On
Bipedal Character in Augmented and Virtual Reality, the right, we see a balancing game in VR where the user
Fig. 5 Character can walk over real world terrain, as show controls a ground with two vive controllers

seeing the character’s ability to self-balance is fun, cone, which would then be applied as the new force.
the reactions to the perturbations became visually This resulted in more variate behaviors.
repetitive even though they are being generated
online through the simulator. It could be due to the
simple collision primitives used to model the char- Balancing the Character
acter’s mechanics. To have more variations in the A self-balancing mechanism can be quite an
reactions, without increasing the complexity of amusing feature to watch and interact with.
the model, it is possible to detect the collisions A small game was devised where the user controls
with objects, recover the contact force, and the plane below the character. The character
slightly deviate its direction within a cone radius. moves towards the steepest point on the plane.
This allows the user to guide the character towards
objectives (milestones) by tilting the plane. The
player has to carefully tilt the plane such that the
inclination does not exceed the characters capa-
bilities, which would make the character fall. Fig-
ure 5 shows a screenshot of the balance game.

Conclusion

This entry described a solution to using ad-

vanced simulated characters in high-level game
engines such as Unity. This allows casual users to
create interactive AR and VR games centered
Note that several alternatives were tried, such as around realistic reactions to perturbations and
adding multiple collision detection primitives to self-balancing, which would be laborious to
have more diversity in the contact configurations, setup with the mainstream blend tree type of con-
as well as simply adding random forces at contact troller currently offered in game engines.
events. In the first case, more primitives did not
improve much the drama and caused the simulator
to slow down. The second idea was hard to fine tune Cross-References
properly and led to this solution: generate a cone
based on the collision point and direction, as well as ▶ Animation and Neurocinematics: Visible Lan-
magnitude, and sample a random vector inside this guage of E-motion-S and Its Magical Science
Interacting with a Fully Simulated Self-Balancing Bipedal Character 977

▶ Augmented Learning Experience for School Annual Conference on Computer Graphics and Inter-
Education active Techniques, SIGGRAPH ’95, pp. 71–78 (1995)
Holden, D., Komura, T., Saito, J.: Phase-functioned neural
▶ Augmented Reality Entertainment: Taking networks for character control. ACM Trans. Graph.
Gaming Out of the Box 36(4), 1–42 (2017)
▶ Augmented Reality for Maintenance Kovar, L., Gleicher, M., Pighin, F.: Motion graphs. In
▶ Character Animation Scripting Environment Proceedings of the 29th Annual Conference on Com-
puter Graphics and Interactive Techniques,
▶ Crowd Simulation SIGGRAPH ’02, pp. 473–482. ACM (2002)
▶ Emotion-Based 3D CG Character Behaviors Lee, Y., Wampler, K., Bernstein, G., Popović, J., Popović,
▶ Interactive Augmented Reality to Support Z.: Motion fields for interactive character locomotion.
Education In ACM SIGGRAPH Asia 2010 Papers, SIGGRAPH
ASIA ’10, pp. 138:1–138:8. ACM (2010)
▶ Sketch-Based Posing for 3D Animation Levine, S., Wang, J.M., Haraux, A., Popović, Z., Koltun,
V.: Continuous character control with low-
dimensional embeddings. ACM Trans. Graph. 31(4),
References 1–28 (2012)
Liu, L., Van De Panne, M., Yin, K.: Guided learning of
control graphs for physics-based characters. ACM
Abe, Y., da Silva, M., Popović, J.: Multiobjective control Trans. Graph. 35(3), 1–29 (2016)
with frictional contacts. In Proceedings of the 2007 Macchietto, A., Zordan, V., Shelton, C.R.: Momentum
ACM SIGGRAPH/Eurographics Symposium on Com- control for balance. In ACM SIGGRAPH 2009 Papers, I
puter Animation, SCA ’07, pp. 249–258. Eurographics SIGGRAPH ’09, pp. 80:1–80:8. ACM (2009)
Association (2007) Mordatch, I., de Lasa, M., Hertzmann, A.: Robust physics-
Arikan, O., Forsyth, D.A., O’Brien, J.F.: Pushing people based locomotion using low-dimensional planning.
around. In Proceedings of the 2005 ACM SIGGRAPH/ In ACM SIGGRAPH 2010 Papers, SIGGRAPH ’10,
Eurographics Symposium on Computer Animation, pp. 71:1–71:8. ACM (2010)
SCA ’05, pp. 59–66. ACM (2005) Peng, X.B., Berseth, G., van de Panne, M.: Terrain-
Chai, J., Hodgins, J.K.: Performance animation from adaptive locomotion skills using deep reinforcement
low-dimensional control signals. In ACM SIGGRAPH learning. ACM Trans. Graph. 35(4), 1–81 (2016)
2005 Papers, SIGGRAPH ’05, pp. 686–696. ACM (2005) Peng, X.B., Berseth, G., Yin, K., van de Panne, M.:
Choi, K.-J., Ko, H.-S.: On-line motion retargetting. In Pro- Deeploco: dynamic locomotion skills using hierarchi-
ceedings of the 7th Pacific Conference on Computer cal deep reinforcement learning. ACM Trans. Graph.
Graphics and Applications, PG ’99. IEEE Computer 36(4), 1–41 (2017)
Society (1999) Rabbani, A.H., van de Panne, M., Kry, P.G.: Anticipatory
Clavet, S.: Motion matching and the road to next-gen balance control. In Proceedings of the Seventh Interna-
animation. In Proceedings of GDC (2016) tional Conference on Motion in Games, MIG ’14,
Coros, S., Beaudoin, P., van de Panne, M.: General- pp. 71–76. ACM (2014)
ized biped walking control. In ACM SIGGRAPH Rhodin, H., Tompkin, J., In Kim, K., Varanasi, K., Seidel,
2010 Papers, SIGGRAPH ’10, pp. 130:1–130:9. H.-P., Theobalt, C.: Interactive motion mapping for
ACM (2010) real-time character control. Comput. Graph. Forum.
Coros, S., Karpathy, A., Jones, B., Reveret, L., 33(2), 273–282 (2014)
van de Panne, M.: Locomotion skills for simulated Rose, C., Cohen, M.F., Bodenheimer, B.: Verbs and
quadrupeds. In ACM SIGGRAPH 2011 Papers, adverbs: multidimensional motion interpolation. IEEE
SIGGRAPH ’11, pp. 59:1–59:12. ACM (2011) Comput. Graph. Appl. 18(5), 32–40 (1998)
Da Silva, M., Abe, Y., Popović, J.: Simulation of human Seol, Y., O’Sullivan, C., Lee, J.: Creature features:
motion data using short-horizon model-predictive con- Online motion puppetry for non-human characters. In
trol. Comput. Graph. Forum. 27(2), 371–380 (2008) Proceedings of the 12th ACM SIGGRAPH/
de Lasa, M., Mordatch, I., Hertzmann, A.: Feature-based Eurographics Symposium on Computer Animation,
locomotion controllers. In ACM SIGGRAPH 2010 SCA ’13, pp. 213–221. ACM (2013)
Papers, SIGGRAPH ’10, pp. 131:1–131:10. ACM Tan, J., Yuting, G., Karen Liu, C., Turk, G.: Learning
(2010) bicycle stunts. ACM Trans. Graph. 33(4), 1–50 (2014)
Dontcheva, M., Yngve, G., Popović, Z.: Layered acting Yamane, K., Ariki, Y., and Hodgins, J.: Animating non-
for character animation. In ACM SIGGRAPH 2003 humanoid characters with human motion data. In Pro-
Papers, SIGGRAPH ’03, pp. 409–416. ACM (2003) ceedings of the 2010 ACM SIGGRAPH/Eurographics
Gleicher, M.: Motion editing with spacetime constraints. Symposium on Computer Animation, SCA ’10,
In Proceedings of the 1997 Symposium on Interactive pp. 169–178. Eurographics Association (2010)
3D Graphics, I3D ’97, pp. 139–ff. ACM (1997) Yin, K.K., Loken, K., van de Panne, M.. Simbicon: Simple
Hodgins, J.K., Wooten, W.L., Brogan, D.C., O’Brien, J.F.: biped locomotion control. In ACM SIGGRAPH 2007
Animating human athletics. In Proceedings of the 22nd Papers, SIGGRAPH ’07. ACM (2007)
978 Interaction

them only provide simple interaction for the

Interaction users. Better 3D interaction techniques are
needed to extend the usability of mobile AR
▶ Virtual Reality: A Model for Understanding applications.
Immersive Computing In this article, we will introduce a 3D interac-
tion technique suitable for mobile AR applica-
tions developed at the mixed reality and
interaction (MRI) laboratory recently. The 3D
Interaction Techniques interaction technique had been developed concen-
trating on object manipulations.
▶ Locomotion in Virtual Reality Video Games

State-of-the-Art Work

Interaction with Mobile 3D Interaction in AR Environments

Augmented Reality There is little research on interactions of mobile
Environments AR systems with a small display. Anders
Henrysson et al. developed two interaction tech-
Jong Weon Lee and Han Kyu Yoo niques. They used an AR-enabled mobile phone
Department of Digital Contents, Sejong as a tangible interaction device. In Henrysson
University, Seoul, South Korea et al. (2005), the mobile phone itself was manip-
ulated to control an object after selecting it in a 3D
AR environment. In Henrysson and Billinghurst
Synonyms (2007), they extended the interaction technique
developed in 2005 for mesh editing. They selected
Augmented reality; Mediated reality; Mixed real- multiple points on a mesh and the selected vertices
ity; MR are locked relative to the camera. Now a user
could move the mobile phone to translate and
rotate the selected object or points after they
Definition chose the motion type.

Augmented reality is a technology that combines Touch-Based Interaction for 3D Manipulation

virtual and real worlds in real time to help users Touch-based interaction techniques have been
complete their work or to provide users new applied to manipulate 3D objections in a few
experiences. virtual reality systems. These interaction tech-
niques are categorized into two types: constrained
and unconstrained. Constrained interaction tech-
Introduction niques are able to manipulate 3D objects pre-
cisely. The constrained interaction techniques
Augmented reality technologies have been separate the control of degree of freedom (DOF)
widely applied to military, industry, medical, to restrict the movements of 3D objects. A widget,
and entertainment areas. The rapid spread of which acts as a visual guidance for the predefined
smart mobile devices such as smart phones and constraints, is typically used to restrict the move-
smart pads has made it possible to experience ments of 3D objects in the constrained interaction
AR on smart mobile devices. Various AR appli- techniques. Figure 1 shows a standard 3D trans-
cations including games have been developed formation widget. A user can select one of three
on mobile devices using sensors such as a cam- arrows in the widget to set a translation direction
era, a GPS, and an inertial sensor, yet most of or one of three circles to set a rotation axis. Any
Interaction with Mobile Augmented Reality Environments 979

error on a device where the screen elements were

too small compared to a finger.
Unconstrained interaction techniques do not
use a 3D transformation widget that visually
guides possible motions of a 3D object. Users
can transform an object along an arbitrary direc-
tion or axis with the unconstrained interaction
techniques. Users can also translate and rotate a
3D object simultaneously with the unconstrained
ones so they are typically useful for fast and
coarse manipulations.
M. Hancock et al. introduced the Sticky Tools
technique in Hancock et al. (2009) to control the
full 6DOF of objects. Users select a virtual object
by touching it with their two fingers. Users move
the two touched fingers and rotate the two touched
fingers relative to one another to manipulate the
Interaction with Mobile Augmented Reality Environ- I
ments, Fig. 1 A standard 3D transformation widget virtual object. While users manipulate the virtual
(Cohé et al. 2011) objects, user’s two fingers should stay in touch
with it. Anthony Martinet et al. developed DS3
(Depth-Separated Screen-Space) interaction tech-
user’s motions are then applied along the selected niques to manipulate 3D objects in a multi-touch
direction or the selected rotation axis. device (Martinet et al. 2012). They combined
A boxlike widget, tBox, was developed in constrained and unconstrained approaches and
Cohé et al. (2011). The edges and the faces of applied different techniques for translation and
tBox were used for translation and rotation of the rotation. The selected object was translated
selected object, respectively. Users can select and along the axis or the plane defined with one or
manipulate edges and faces of tBox easily with a two fingers. It was rotated freely using the con-
fingertip. Widgets were designed to be more tol- strain solver, which was introduced by Reisman
erable to imprecise touch inputs even though care- et al. in Reisman et al. (2009). Translation and
ful touch positioning was still necessary. Schmidt rotation were clearly separated by the number of
et al. developed a single touch interaction tech- fingers directly in contact with the object. Nicho-
nique with transient 3D widgets (Schmidt las Katzakis et al. used a mobile device as the
et al. 2008). Stroke-based gestures were used to game controller in Katzakis et al. (2011). They
create translation and rotation widgets. The stan- developed an interaction technique that could
dard click-and-drag interaction was used for control a 3D cursor on a large display without
manipulation. directly touching the large display. The plane
A few constrained interaction techniques have defined by the orientation of a mobile device
been developed for multi-touch inputs without a was casted on the large display. The user could
widget. Oscar K.C. Au et al. introduced the move the cursor on the casted plane using touch
widgetless constrained multi-touch interaction inputs on the display of the mobile device.
on a 10.1 inch display (Au et al. 2012). A user The last three interaction techniques are good
selected the constraint without directly touching solutions for a virtual environment with a touch-
the constraint mark. The orientation of two based display, but they cannot be directly applied
touched fingers was compared with the predefined to mobile AR environments with a small display.
axes to select the constraint. The constraint marks The Sticky Tools and DS3 interaction techniques
were displayed only as a visual guidance to users. require direct contacts with an object. This
This solved the fat-finger problem causing an requirement is not applicable for a mobile AR
980 Interaction with Mobile Augmented Reality Environments

system. Fingers will occupy too much area of the

display. The constraint solver could be burden-
some for the processor of the mobile device,
which has limited processing power. The interac-
tion technique proposed by Oscar K.C. Au
et al. could be applied to the device with a small
display because they do not require direct contact
with the constraint marks. The possible problems
with this technique are clutter caused by visual
guidance and two required touched fingers. The
plane casting interaction developed by Nicholas
Katzakis could be adapted to a mobile AR envi-
ronment since the position and orientation of the
mobile device are tracked in real time. This
tracked information could be used to constrain
the motion of a 3D object in the mobile AR
environment. We adapted this plane casting inter-
action to the proposed interaction techniques.

Overview
Interaction with Mobile Augmented Reality Environ-
We developed a new interaction technique for ments, Fig. 2 Dynamic constraints
mobile AR systems with following three charac-
teristics: (1) combining constrained and
unconstrained interaction techniques, (2) using
relations between real objects and a smart mobile
device, and (3) combining a way to manipulate
real objects and a touch interface of a smart
mobile device. The proposed interaction tech-
nique aims at providing intuitive and effective
interaction when a user manipulates virtual
objects in mobile AR world.

3D Interaction in Mobile AR
Environments

We designed a new interaction technique for

mobile AR systems with three characteristics
described in the earlier paragraphs. The interac-
tion technique uses the movements of a mobile
device to change constraints and a mapping ratio
dynamically as shown in Figs. 2 and 3. After
moving the mobile device, the plane created by
the orientation of the mobile device is projected
onto the coordinate of the selected virtual object in Interaction with Mobile Augmented Reality Environ-
an AR world. For example, the mobile devices ments, Fig. 3 Dynamic mapping distance
Interaction with Mobile Augmented Reality Environments 981

A and B in Fig. 2 are projected onto the coordi- B along the E0 direction using the 2D motion E on
nates of a cube object as plane A0 and plane B0 the display.
passing through the origin of the selected object The moving distance of the object is dependent
coordinate, respectively. A user can translate the on the distance of the mobile device as shown in
object along the projected plane, which is the Fig. 3. When the mobile device is located at loca-
constraint plane, by a simple drag motion shown tion A, the drag motion translates the virtual object
in Fig. 4. By changing the constraint plane, a user C to the location CA. The same drag motion on the
can translate the object to any location with simple display of the mobile device at B will translate the
drag motions on the display. Figure 5 shows the C to the location CB. The distance between C and
mapping between the translations on the AR CA is twice as long as the distance between C and
world and motions on the display. The 2D motion CB since the distance between C and A is twice as
E on the display is projected onto the constraint long as the distance between C and B. This map-
plane D as E0 . A user can move the selected object ping is represented in Eq. 1 where α is the

Interaction with Mobile Augmented Reality Environments, Fig. 4 The setting of the usability test

5
7 Likert Scale

0
1 2 3 4 5 6 7 8 9 10 11 12 13
Question Number

Interaction with Mobile Augmented Reality Environments, Fig. 5 User preference

982 Interaction with Mobile Augmented Reality Environments

mapping ratio between dp, the distance of the drag trained until their performance improvements
motion, and do, the translated distance of the vir- were saturated or they felt comfortable with the
tual object C. test. Participants generally took 30–45 min for the
training period. The number of trials and the
do ¼ dp  l  a ð1Þ learning time were measured during the training
period. The numbers of translation, rotation, and
The tapping on a mode-changing button is used to scaling operations and the task completion time
change the interaction mode between translation were measured for each trial. Before the usability
and rotation. In the rotation mode, the axis of the test, we asked participants to fill up the question-
rotation is defined as the axis orthogonal to the naires to understand participants’ backgrounds.
direction of the drag motion on the constraint The numbers of translation, rotation, and scaling
plane created by the orientation of a mobile operations and the task completion time were also
device. The axis b is orthogonal to the drag measured during the final test. After the training
motion a. The scaling is done with pinch and and the final test period, participants were asked to
spreading motions. The scaling is also constrained fill up the questionnaires shown in Table 1 to
by the projection plane defined by the orientation measure the preference of interaction techniques
of a mobile device. The ratio of the scaling is and the opinions about interaction techniques.
determined dynamically based on the distance Ten participants (four males and six females)
between the mobile device and the selected object with normal or corrected vision took part in the
similar to the translation. experiment. They were volunteers coming for the
experiment and we gave them a small gift. All
Experiments participants owned smart phones and seven
We designed and performed a user study to eval-
uate the presented interaction technique. We
Interaction with Mobile Augmented Reality Environ-
examined the subjective intuitiveness such as ments, Table 1 Questionnaires to measure the partici-
ease of use, ease to learn, naturalness, preference, pants’ preferences of the interaction techniques (7 Likert
and fun. scale)
We developed a docking task, manipulated No. Questions
virtual objects (indicated by the dotted lines), Q1 The interaction technique was easy to use
and arranged them along the real objects Q2 The interaction technique was easy to learn
(indicated by the filled rectangular) on table Q3 The interaction technique was natural to use
T (Fig. 4). We asked participants to put five virtual Q4 The interaction technique was easy to remember
characters on the top of the same real characters as Q5 It was easy to view the pattern required for using
shown in Fig. 4. Five virtual characters randomly the augmented reality system
appeared at the starting location, the lower center Q6 The augmented object was lost few times, but
they did not cause a big problem to complete the
of T. To enforce 3D manipulation, the position, the given task
orientation, and the size of each virtual character Q7 The interaction technique was generally
were randomly assigned. If each virtual object satisfactory
was closely posed with a similar size to the Q8 The interaction technique was fun
corresponding real object, it was considered as Q9 It was easy to move the augmented object to the
successfully docked and the virtual object target location
disappeared, and the next virtual one appeared at Q10 It was easy to rotate the augmented object to the
target orientation
the starting location again (see the right part of
Q11 There wasn’t a major problem to complete the
Fig. 4). The rectangular with the character M was given task
the location of a pattern used for tracking the Q12 The size of the display was suitable for the
camera of a smart phone. interaction technique
The usability test consisted of two periods: Q13 It was easy to use one hand for the interaction
training and final test periods. Participants were technique
Interaction with Mobile Augmented Reality Environments 983

participants have heard about AR. Three partici- experiment. We would fix all real objects for
pants have used AR apps before, but they only the next user experiment to understand the
used them few times. We selected young partici- behaviors of the participants better in an AR
pants for the experiment since they were generally environment.
more familiar with new technologies and more
willing to learn new technologies. In addition, our experience suggests that we
Average ratings are summarized in Fig. 5. have to modify the rotation interaction of the
Overall, the presented interaction technique presented interaction technique to provide users
achieved good ratings in all questions except with better user interactions. Participants had the
Q10 and Q13. The interaction technique was con- most difficult time when they had to rotate the
sidered easy to learn, easy to remember, and fun. augmented objects in the desired direction. Partic-
Users had difficulty applying rotation motion to ipants also provided useful comments. During the
the selected object and using the mobile device training period, they complained about discomfort
with one hand. in their arms caused by holding the smart phone
for a long period of time. This aspect regarding
discomfort should also be considered while devel-
Conclusion and Discussion oping mobile AR applications if they are to be
I
truly user-friendly.
Understanding the characteristics of mobile AR
systems can lead to the development of more
effective 3D interaction schemes in the mobile Cross-References
AR applications. Important findings from the
usability study with the presented interaction tech- ▶ Interactive Virtual Reality Navigation Using
nique can be summarized as: Cave Automatic Virtual Environment
Technology
1. The hybrid touch-based interface, combining ▶ Integrating Virtual Reality and Augmented
constrained and unconstrained interaction Reality into Advertising Campaigns: History,
techniques, is easy to learn and easy to remem- Technology, and Future Trends
ber for the given task. The participants’ famil-
iarities to the touch-based interface could affect
the results. References and Further Reading
2. Users have to view the given pattern through
their cameras for AR applications using com- Au, O.K., Tai, C.L., Fu, H.: Multitouch gestures for
constrained transformation of 3D objects. J. Comput.
puter vision techniques. Participants were not
Graph. Forum. 31(2), 651–660 (2012)
bothered much by this requirement for the pre- Cohé, A., Decle, F., Hachet, M.: tbox: A 3D transformation
sented interface. This is an encouraging result widget designed for touch-screens. In: Proceedings of
because computer vision techniques are used the 2011 Annual Conference on Human Factors in
Computing Systems, pp. 3005–3008 (2011)
often to create mobile AR applications. Partic-
Hancock, M., Cate Ten, T., Carpendale, S.: Sticky tools:
ipants also responded positively to the losses of Full 6DOF force-based interaction for multi-touch
augmented objects due to tracking failures. tables. In: Proceedings ITS’09, pp. 145–152 (2009)
3. Users do not want to move around the AR Henrysson, A., Billinghurst, M.: Using a mobile phone for
6 DOF mesh editing. In: Proceedings of CHINZ 2007,
environment yet. The geometrical relations
pp. 9–16 (2007)
between augmented virtual objects and real Henrysson, A., Billinghurst, M., Ollila, M.: Virtual object
objects are important in an AR environment, manipulation using a mobile phone. In: Proceedings of
so users have to move around the AR environ- the 2005 International Conference on Augmented Tele-
Existence (ICAT’05), pp. 164–171 (2005)
ment. In the experiment, participants preferred
Katzakis, N., Hori, M., Kiyokawa, K., Takemura, H.:
to rotate the real environment, which is the Smartphone game controller. In: Proceedings of 75th
board that contains all real objects used in the HIS SigVR Workshop, pp. 55–60 (2011)
984 Interactive Art

Martinet, A., Casiez, G., Grisoni, L.: Integrality and sepa- not fully support natural interaction, and the
rability of multi-touch interaction techniques in 3D existing 3D pop-up book has used touch-based
manipulation tasks. IEEE Trans. Vis. Comput.
Graphics. 18(3), 369–380 (2012) to interact with 3D content. Therefore, this entry
Reisman, J., Davidson, P.L., Han, J.Y.: A screen-space describes a fundamental to design an interactive
formulation for 2D and 3D direct manipulation. In: AR pop-up book with natural gesture interaction
Proceedings of UIST’09, pp. 69–78 (2009) using real hand. Subsequently, the real hand ges-
Schmidt, R., Singh, K., Balakrishnan, R.: Sketching and
composing widgets for 3D manipulation. Comput. ture tracking in handheld AR is explored to exam-
Graph Forum. 27(2), 301–310 (2008) ine how it can track user’s hands in real time.
Thus, this entry describes about gesture interac-
tion to allow the user to directly interact with the
virtual objects. The user feels more realistic to
Interactive Art interact with 3D objects using their bare hands
on 3D pop-up book.
▶ Biosensing in Interactive Art: A User-Centered
Taxonomy
Introduction

Augmented reality (AR)33 is a technology that

Interactive Augmented allows computer-generated information or digi-
Reality Pop-Up Book with tal information including text, video, 2D virtual
Natural Gesture Interaction images, and 3D virtual objects to be overlaid
for Handheld onto the real-world environment in real time
(Ismail and Sunar 2013). The main reason people
Muhammad Nur Affendy Nor’a1, Ajune Wanis intend to develop AR application is to merge the
Ismail3 and Mohamad Yahya Fekri Aladin1,2 real world into the virtual world to provide the
1
Mixed and Virtual Reality Research Lab, users with information-enhanced environment
Vicubelab, Universiti Teknologi Malaysia, Johor (Billinghurst et al. 2008). The connection
Bahru, Malaysia between these two worlds seems impossible
2
School of Computing, Faculty of Engineering, back then, but now it becomes an attraction,
Universiti Teknologi Malaysia, Johor Bahru, and its potential was very overwhelming. Usu-
Malaysia ally, the virtual elements are generated by the
3
Mixed and Virtual Reality Research Lab, computer and made to be overlaid onto the real
Vicubelab, School of Computing, Faculty of world, to enhance the user’s sensory perception
Engineering, Universiti Teknologi Malaysia, of the augmented world they are seeing or
Johor Bahru, Malaysia interacting with.
Nowadays, the concept of AR technology is
used widely in entertainment, military training,
Synonyms engineering design, robotics, manufacturing, and
other industries. AR technologies bring a lot of
Augmented Reality; PUN: Photon Unity Network advantages to perform a task especially once it
involves with design and planning. AR has the
ability to perform 3D object manipulation and can
Definition provide natural user interaction techniques (Ismail
and Sunar 2013). All developers take an advan-
Handheld augmented reality (AR) has been tage on AR technologies and believe it could help
widely used with smart and portable device in them to perform real task in virtual way easily
the applications such as education, games, besides reducing cost for real task and able to
visual experience, and information visualization. solve many issues which cannot be remedied in
However, most of the handheld applications do the real world.
Interactive Augmented Reality Pop-Up Book with Natural Gesture Interaction for Handheld 985

The level of immersion for both elements of and animated content registered on real book
virtual and real objects in AR application refers to pages, mimicking a traditional “pop-up book”
the merging of real and virtual worlds to produce (Markouzis and Fessakis 2015). AR pop-up
AR environments and visualizations where real book is a book that involves the process of over-
and digital objects coexist and interact in real laying a virtual content onto the pages of a phys-
time (Azuma et al. 2001). According to Ismail ical pop-up book. The current existing AR book
and Sunar 2013, a tracking process is very impor- that uses similar metaphor is MagicBook
tant in developing AR application and in running (Billinghurst et al. 2001). MagicBook offers the
it in real time. The main requirements for trackers user the ability to experience the full reality-
are high accuracy and little latency at a reasonable virtuality continuum. This is because the
cost. The tracking of objects in the scene amounts MagicBook itself is capable of changing the
to calculating the pose between the camera and the mode between AR and VR. Through AR display,
objects. Virtual objects can then be projected into the user is able to see a scene alike, and they could
the scene using the pose. change the view mode to an immersive virtual
environment. Another application that adopted
the AR book metaphor is AR coloring book
Augmented Reality Handheld Interface (Clark et al. 2011). The AR coloring book aims
I
at augmenting an educational coloring book with
There are three main fundamentals that can be user-generated AR virtual content.
found: tracking, display technology, and interac- Handheld interfaces have four interaction tech-
tion (Billinghurst et al. 2008). Tracking is one of niques that have been recently explored: touch-
the fundamental parts in enabling technologies in based interaction (Kim and Lee 2016), midair
AR, and it still have many problems that are gesture-based interaction (Vuibert et al. 2015),
unsolved (Ismail and Sunar 2013). Interaction device-based interaction (Samini and Palmerius
technique issues in mobile AR and multimodal 2016), and direct interaction (Hilliges et al. 2018).
AR are becoming more popular. In vision-based The traditional touch-based interaction methods for
interaction, hand and fingertip tracking and hand handheld AR cannot provide intuitive 3D interac-
gesture recognition method are widely used to tion due to a lack of natural gesture input with real-
provide an easy way to interact with virtual object time depth information (as agreed by Bai et al.
in AR (Chun and Lee 2012). A real-time vision- 2013). Therefore, this entry aims to illustrate the
based approach was introduced to manipulate the design of natural interaction techniques in 3D spaces
overlaid virtual objects dynamically in a marker- by handheld AR devices. Positions and movements
less AR system using bare hand with a single of the user’s fingertips are corresponding to the
camera (Cohen et al. 1989). It is natural that the manipulations of the virtual objects in the AR
collision between the human hand and the aug- scene (as recommended in Bai et al. 2013).
mented object can occur during manipulation of
the virtual 3D object. In AR, however, the colli-
sion happened between a virtual object and a real Augmented Reality Pop-Up Book
object; thus, the collision detection approach may
be different compared with the ways in the real There are three phases carried out to develop AR
world. Most AR-handheld applications are not pop-up book that are described in the following
applying a natural interaction, and the user inter- subsections.
actions mostly are using touch-based (Kim and
Lee 2016). Therefore, this entry describes the Phase 1: Defining Interactivity and
interaction in an interactive pop-up book with Storytelling for AR Pop-Up Book
natural gesture interaction using real hand in The interactivity for an interactive book happens
handheld interface. when it contains story and activities which
The existing AR book which is generally required the user to perform and interact. The
known as the magic book contains 3D virtual real pop-up book does offer a lot of advantages,
986 Interactive Augmented Reality Pop-Up Book with Natural Gesture Interaction for Handheld

but in the transformation to more digital and inter- pop-up book into AR transitional and tangible in
active experience, the book offers a lot more than order to measure the AR experience.
just a pile of heavy paper. Digital books recently
have been widely restructured and recycled, yet it Phase 2: Setting Up AR-Handheld User
enhances the reading experience and more inter- Interface
active than the conventional printed books. The The phase is carried out that includes determining
main advantages of a digital book are that it can be the display technique, tracking technique, and
customized to meet the reader’s prospect interaction method. This stage focuses on setting
(Markouzis and Fessakis 2015). This phase is up the handheld AR interface as shown in Fig. 1.
conducted to design and construct the 3D contents The user interface for AR application that uses the
for AR pop-up book. The 3D object built with “pop-up book” metaphor has been designed. In
animation is developed during this phase since the order to create a good AR presentation, ensuring
physical pop-up book does not in a digital mode. the virtual environment was displayed in a correct
It was a fully printed copy. alignment to merge with real environment. This
An interactive storytelling enables user to take stage is the crucial part. Next, the display tech-
part and affects the plot of the story, creating a new nique that was chosen is a handheld display
genre of narrations that is much more engaging device. The tracking technique that has been
and adaptive. Several levels of interactive story- applied in this project is a feature-based. Feature-
telling start from a simple branching plot to fully based tracking technique involves the registration
dynamic narration models. Interactive storytelling of the virtual element on top of the real marker.
constitutes a new genre of literature which prom- Sensor-based was used in this project since it
ises considerable learning effectiveness. This required the depth data to recognize the user’s
stage also defined that the appropriate 3D anima- real hand gesture features. These elements have
tion could be applied on the virtual object so the been prepared and examined to proceed with the
visual is more appealing and interesting. How- next stage, the development of the AR pop-up
ever, the storytelling has been chosen based on book.
the current available conventional pop-up book As illustrated in the diagram, it can be seen the
which is entitled Beauty and the Beast. The phys- hardware configuration. In order to overlay the
ical fairytale pop-up book is being used to provide virtual element on the top of real environment,
pop-up book with the storytelling. Therefore, we the data of 3D object are loaded binding with 2D
were transforming the existing format for real textures. In order to display the AR interface,

Interactive Augmented
Reality Pop-Up Book
with Natural Gesture
Interaction for
Handheld, Fig. 1 Setting
up of the AR-handheld
interface
Interactive Augmented Reality Pop-Up Book with Natural Gesture Interaction for Handheld 987

Interactive Augmented Reality Pop-Up Book with Natural Gesture Interaction for Handheld, Fig. 2 Natural
feature tracking process. (a) RGB image. (b) Gray-scale image. (c) Feature points

handheld device is chosen as AR display technol- image in Fig. 2a shows the original state of the
ogy. The standard vision-based tracking system marker. The marker then will be converted into
works to recognize the input, marker, and user gray scale using image processing to gray-scale
hand. It recognizes the registered marker before format as shown in Fig. 2b before it is being
it was loaded with the appropriate 3D object onto processed as image target in the form of features
I
the scene. The user’s hand required to be captured as shown in Fig. 2c. The features were recognized
by the leap motion device (Guna et al. 2014). User by the system as a unique identification. The sys-
interacts with the AR environment by using their tem will detect the marker and register the marker
bare hand as an interaction tool. The application is with a virtual element. The virtual cube, for exam-
able to recognize user’s one hand to interact with ple, will appear on the top of the marker after the
the virtual object, and the other hand holds the camera recognizes the marker. The AR user inter-
handheld device. Users can interact with the vir- face was using this tracking process to display
tual animation by performing a define gesture that animation on the top of pop-up book. The edges
is recognized by the system. of real pop-up book are being converted into
features for this project.
Phase 3: Pop-Up Book Feature Tracking
This phase is conducted to design and construct Phase 4: Developing Hand Gesture Interaction
the 3D contents for AR pop-up book. The 3D This phase focuses on exploring the gesture inter-
object built with animation is developed during action for the user to interact with AR pop-up
this phase since the physical pop-up book does not book. The study of the pop-up book concept and
in a digital mode. It was a fully printed copy. The its interactivity processes has been carried out in
phase is carried out that includes determining the Phase 3. In order to enhance the realism in AR
display technique, tracking technique, and inter- environment for conventional pop-up book, we
action method. merge the AR pop-up book with the live character,
The main challenge in AR pop-up book appli- and the story elements of the pop-up book come
cation is to ensure the registrations and hand alive. The character will follow user’s hand move-
tracking problem are effectively solved. AR pop- ment, and the story elements will activate the
up book utilizes the benefit of hand gesture rec- animation effects once user’s hands touch them.
ognition technique as an interaction tool in the AR To actualize this realism effects, user interaction is
environment. The tracking library is used to track crucial to precisely hit the characters. To look
the page of the pop-up book that utilizes a feature- more natural, the user can use their bare hands to
based tracking technique. directly contact with the virtual elements.
Figure 2 shows the natural feature tracking Therefore, hand gesture recognition method is
process. The original RGB image is captured one of the crucial parts in this project as it acts as
and converted to features so it will be recognized the input metaphor for the user to interact with the
by the camera as the target image. Printed-colored virtual object in AR environment. Sensor-based
988 Interactive Augmented Reality Pop-Up Book with Natural Gesture Interaction for Handheld

Interactive Augmented
Reality Pop-Up Book
with Natural Gesture
Interaction for
Handheld, Fig. 3 Hand
gesture recognition method

tracking device, leap motion, allows the applica- Phase 1: Acquiring Gesture Inputs
tion to read depth data and is able to track the There are three gesture inputs that have been
position of user’s hand in real world and mapping defined such as TouchGesture, SwipeGesture,
it into the virtual world (Guna et al. 2014). 3D and CircleGesture. TouchGesture represents a
hand skeleton-based interaction uses a leap virtual object that will call an appropriate anima-
motion sensor attached in front or back of a tion as a feedback once it is being touched.
mobile device to provide simultaneous manipula- SwipeGesture represents a virtual object that is
tions of 3D AR objects. By capturing the hand being swiped, while CircleGesture is being
skeleton and identifying 3D finger positions and retrieved and updated whenever user performed
orientations, we can support a more natural hand a circling gesture at designated position in the AR
gesture-based interaction in an AR scene. In addi- environment and call appropriate animation.
tion to the 3D translation-only tasks in the previ- Figure 4 shows the flow of acquiring gesture
ous works, simultaneous 3D translation and 3D inputs. The process starts when a leap motion
rotation are possible to alter the location, pose, device detects the hand interaction from the user
and size of virtual objects with hand gestures. As using the sensor, and the gestures are identified in
shown in Fig. 3, sensor-based tracking device, the pose detection. Then, the signal is sent to start
leap motion, allows the application to read depth the skeleton calibration that later leads to skeleton
data during recognition. Then, the device pro- tracking. In this project, gestures used are grab-
duces positions and orientations. It runs to track bing to grasp object, pointing to select menu, palm
the position of user’s hand in real world and to up gesture to activate menu, and pinch to rescale
map it into the virtual world. To display virtual the 3D object. The next process is to develop the
hand skeleton, the modeling process is required, natural gesture to interact with virtual object in
and to enable interaction cues, the rigid body was AR pop-up book. In the next section, the real hand
applied to the 3D model of virtual hands. Once human gestures were captured by leap motion
this process was completed, the gesture inputs are device, and recognition process was executed to
created. obtain depth data from leap motion sensor-based
tracking system.
The SwipeGesture is a gesture input where the
Natural Gesture Interaction user swipes their index finger to interact with the
virtual object of the AR environment. The gesture
This section explains on natural gesture interac- is defined in this particular project by calculating
tion which was divided into the following process. the velocity and speed of the tip of the index finger
Interactive Augmented Reality Pop-Up Book with Natural Gesture Interaction for Handheld 989

Interactive Augmented
Reality Pop-Up Book
with Natural Gesture
Interaction for
Handheld, Fig. 4 Flow of
acquiring gesture inputs

and the collision detection between the finger and I

the virtual object that can be interacted with.
The CircleGesture is a gesture input where
the user makes a circling gesture by using their
index finger to enable certain features in the AR
environment in order to interact with the virtual
object. The gesture is defined by calculating the
vector, magnitude, and angle of the circle based
on the position of the tip of the index finger of
the user. Figure 5 is executed to calculate the
angle.
The TouchGesture is a gesture input where the
user touches the virtual object by using their index
finger to enable certain features in the AR environ-
ment and interact with the virtual object. The ges-
ture is defined in this particular project by making
collision detection whenever the tip of the index Interactive Augmented Reality Pop-Up Book with
Natural Gesture Interaction for Handheld,
finger collider of the user collides with the
Fig. 5 Flow of acquiring gesture inputs
interactable virtual object in the AR environment.

PUN (Photon Unity Networking) (Network 2015)

Phase 2: Integrating Gesture with is used, so we can send and receive gesture inputs
Handheld AR to the AR handheld application.
In this phase, the gesture interaction technique is Through the PUN network protocol that is
then integrated with the handheld AR scene. The being implemented in this stage, the user hand
hand gesture interaction technique has been tracking data (position and rotation) from the
developed for the user to interact with the AR real world is being sent by desktop (sender) to
pop-up book. In order to transmit the signal from the handheld device (client or receiver). Photon
the leap motion gesture tracking device to the network always uses a master server and one
application, we need to use Internet protocol. To or more game servers. The master server
actualize this, we enable the multiplayer network- manages the currently available games and does
ing as shown in Fig. 6. The network protocol in matchmaking. Once a room is found or created,
990 Interactive Augmented Reality Pop-Up Book with Natural Gesture Interaction for Handheld

the actual gameplay is done on a game server. All The virtual hand is the representative of the real
servers run on dedicated machines. hand. Each gesture that is detected by the leap
motion sensor can be seen in the monitor. Thus,
Phase 3: Executing Gesture Interaction in AR every hand gesture such as swiping, pinching, or
Pop-Up Book pointing in the real world is replaced by the virtual
The leap motion is attached at the back of the hand. This is done to ease the system development
smartphone, and the leap motion needs to be and give the user an immersive feeling or realism.
triggered and well-connected. It is necessary to Handheld device captures the user’s bare hand to
enable the hand tracking and gesture interaction. work with real hand gesture in handheld AR scene
Gesture recognition can be achieved by using the as presented in Fig. 8.
leap motion controller. It detects the hand gesture The handheld device’s camera has synchro-
or hand signal as shown in Fig. 7. The hand nized the video input (720 HD pixel resolutions,
gesture in the real world is recognized by the 25 frames per second). It was placed in single
controller as shown in Fig. 7a, while the hand alignment with the physical pop-up book (image
gesture in the virtual world is produced as shown of the marker) and the leap motion device which is
in Fig. 7b. attached to the handheld device (Android

Server

Send posiotn of fingertips Receive posiotn of fingertips

Detect AR marker, display

AR environment. Perform
Receive hand data input gesture based on position of
from Leap Motion. right hand.
Get position right hand.

Interactive Augmented Reality Pop-Up Book with Natural Gesture Interaction for Handheld, Fig. 6 Flow of
data transmitting using PUN

Interactive Augmented Reality Pop-Up Book with Natural Gesture Interaction for Handheld, Fig. 7 Gesture
signal transferred to handheld device. (a) Real hand gesture. (b) Virtual gesture inputs
Interactive Augmented Reality Pop-Up Book with Natural Gesture Interaction for Handheld 991

Samsung), respectively, to detect the user’s real Problems and Restriction

hand skeleton as shown in Fig. 9a. A handheld
screen displayed the viewing of AR scene. On the The AR pop-up book is demonstrated in this entry
top of the table, the pop-up book was demon- as an interactive AR environment that enables
strated with the user’s fingertip as a controller to users to play with the storytelling. The gesture
get the reference point of the augmentation as interaction provides the user to directly interact
presented in Fig. 9b. with the virtual objects. The user feels more

Interactive Augmented
Reality Pop-Up Book
with Natural Gesture
Interaction for
Handheld, Fig. 8 Gesture
signal transferred to
handheld device

Interactive Augmented
Reality Pop-Up Book
with Natural Gesture
Interaction for
Handheld, Fig. 9 AR
pop-up book in handheld
screen. (a) User interacts
with AR pop-up book.
(b) Swipe the gesture, it will
bring character alive
992 Interactive Augmented Reality Pop-Up Book with Natural Gesture Interaction for Handheld

realistic to interact with 3D objects using their bare Sunar 2014; Piumsomboon et al. 2014). Handheld
hands, and the realism of the 3D objects appears on AR has been widely used with smart and portable
the top of the pop-up book in the real world. Hence, device in the applications such as education,
there are several problems that arise regarding the games, visual experience, and information visuali-
real-time 3D gesture sensing in AR pop-up book. zation. However, most of the handheld applications
The first problem is the accuracy of the hand’s have used touch-based to interact. Subsequently,
detection because when the hands move into a the real hand gesture tracking in handheld AR is
certain position, it loses the accuracy. Accuracy in explored to examine how it tracks user’s hands in
tracking is vital to ensure intuitive user interaction real time. This entry describes the gesture interac-
with the virtual elements (Lv et al. 2015). tion that allows the user to directly interact with the
The second problem is that the user feels detached virtual objects. Thus, the user feels more realistic to
from the AR environment because of the indirect interact with 3D objects using their bare hands.
interaction method. However, these problems still
persist especially when involving the precision
of the hand’s detection which can cause problem References
in the performance. It is natural that collision
between the human hand and the augmented object Azuma, R., Behringer, R., Feiner, S., Julier, S.,
can occur when manipulating the virtual 3D object. Macintyre, B.: Recent advances. In EEE Computer
Graphics and Applications, 2011(December), 1–27
In AR, however, the collision happened between a (2001)
virtual object and a real object; thus, the collision Bai, H., Gao, L., El-Sana, J., Billinghurst, M.: Markerless
detection approach may be different compared 3D gesture-based interaction for handheld augmented
with the ways in the real world. In the user’s reality interfaces. In Mixed and Augmented Reality
(ISMAR), 2013 IEEE International Symposium on,
observation, with a handheld, the screens are pp. 1–6. IEEE (2013)
often restricted and sometimes can be rotated Billinghurst, M., Kato, H., Poupyrev, I. Tangible
between portrait and landscape. Handheld is small Augmented Reality. ACM SIGGRAPH ASIA 2008
enough to hold and operate in the hand; neverthe- Courses, 7, pp. 1–10 (2008)
Billinghurst, M., Kato, H., Poupyrev, I.: The MagicBook: a
less the user cannot use their both hands since the transitional AR interface. Comput. Graph. 25(5),
other hand needs to hold the device. 745–753 (2001)
Based on the development stages described in Chun, J., Lee, S.: A vision-based 3D hand interaction for
the previous section, the standard guidelines marker-based AR. Int J Multimed Ubiquit Eng. 7(3),
51–58 (2012)
emphasize on developing the handheld AR inter- Clark, A., Dünser, A., Grasset, R.: An interactive augmented
face for AR pop-up book application that applies reality coloring book. In: Mixed and Augmented Reality
natural gesture interaction instead of touchscreen. (ISMAR), 2011 10th IEEE International Symposium on,
The AR pop-up book development is explained in pp. 259–260. IEEE (2011)
Cohen, P.R., Dalrymple, M., Moran, D.B., Pereira, F.C.,
this entry but does not study the education peda- Sullivan, J.W., Cohen, P.R., Sullivan, J.W.: Synergistic
gogy and the development stresses on AR use of direct manipulation and natural language.
technology to bring the physical book into In: Proceedings of the SIGCHI Conference on Human
more appealing and interesting handheld AR appli- Factors in Computing Systems Wings for the Mind –
CHI ’89, vol. 20, pp. 227–233. ACM Press, New York
cation. On the physical book, the virtual environ- (1989)
ment was overlaid in real time. The study on Guna, J., Jakus, G., Pogačnik, M., Tomažič, S., Sodnik, J.:
education purposes can be further explored the An analysis of the precision and reliability of the leap
potentials and future work. Also, more future motion sensor and its suitability for static and dynamic
tracking. Sensors. 14(2), 3702–3720 (2014)
work in user’s interaction for usability aspect can Hilliges, O., Kim D., Izadi S., Molyneaux D., Hodges S.E.,
be carried out such as invoking the multimodal Butler D.A.: Augmented reality with direct user inter-
interaction that may bring AR pop-up book to be action. U.S. Patent 9,891,704, issued February
more interactive when speech input complements 13 (2018)
Ismail, A.W., Sunar, M.S.: Intuitiveness 3D objects inter-
the gesture. Multimodal interaction is seen to action in augmented reality using S-PI algorithm.
advance interaction technique in AR which can Indones J Electr Eng Comput Sci. 11(7), 3561–3567
improve user’s experience in AR (Ismail and (2013)
Interactive Augmented Reality to Support Education 993

Ismail, A.W., Sunar, M.S.: Multimodal fusion: gesture and Definition

speech input in augmented reality environment.
In: Computational Intelligence in Information Systems:
Proceedings of the Fourth INNS Symposia Series on Augmented reality is a technology adding virtual
Computational Intelligence in Information Systems objects to real environment through which
(INNS-CIIS 2014), vol. 331, p. 245. Springer, Cham enabling the fusion of missing information in
(2014) real life. More formally, AR has been defined as
Kim, M., Lee, J.Y.: Touch and hand gesture-based interac-
tions for directly manipulating 3D virtual objects in a system that fulfills three characteristics (Azuma
mobile augmented reality. Multimed. Tools Appl. 75, 1997). First, it combines the real and virtual
16529 (2016) world. Second, it allows real-time interaction.
Lv, Z., Halawani, A., Feng, S., Ur Réhman, S., Li, H.: Third, it aligns real objects or places and digital
Touch-less interactive augmented reality game on
vision-based wearable device. Pers. Ubiquit. Comput. information in 3D.
19(3–4), 551–567 (2015)
Markouzis, D., & Fessakis, G.: Interactive storytelling and
mobile augmented reality applications for learning and
entertainment – a rapid prototyping perspective.
In: Interactive Mobile Communication Technologies
Introduction
and Learning (IMCL), 2015 International Conference
on, pp. 4–8. IEEE (2015) Augmented reality has the potential to change
Network, P.U.: How to Create an Online Multiplayer Game how people interact and experience their sur-
I
with Photon Unity Networking (2015)
Piumsomboon, T., Altimira, D., Kim, H., Clark, A.,
rounding environment (Sommerauer and Muller
Lee, G., Billinghurst, M.: Grasp-Shell vs gesture- 2014). It is well known that interaction in educa-
speech: a comparison of direct and indirect natural tion could result in better learning effect. Active
interaction techniques in augmented reality. involvement in learning, in the sense of being
In ISMAR 2014 – IEEE International Symposium on
Mixed and Augmented Reality – Science and Technol-
engaged, interacting, and taking part, is central
ogy 2014, Proceedings, pp. 73–82 (2014) to its effectiveness. To this end, a variety of
Samini, A., Palmerius, K.L.: A study on improving close computer-based activities have been developed
and distant device movement pose manipulation for with the aim of augmenting and extending active
hand-held augmented reality. In The 22nd ACM Sym-
posium on Virtual Reality Software and Technology
learning.
(VRST), Munich, Germany, November 02-04, 2016 Nowadays augmented reality technology is
(pp. 121–128). ACM Press (2016) emerging rapidly in educational and presenta-
Vuibert, V., Stuerzlinger, W., Cooperstock, J.R.: tional area, such as augmented reality books,
Evaluation of docking task performance using mid-air
interaction techniques. In: Proceedings of the 3rd ACM
mixed reality books, AR contents in exhibi-
Symposium on Spatial User Interaction (pp. 44–52). tions, and AR applications in different disci-
ACM (2015) plines, classroom, and laboratory. Usually,
these contents will be used in classroom or
in exhibition with multiple users at the same
time and who usually have different back-
ground; so it is important to provide the
Interactive Augmented users with a friendly, stable, and low-cost
Reality to Support Education interface.
In these circumstances, tangible interface
YanXiang Zhang and QingQing Zhao (Billinghurst et al. 2005)-based interaction for
Department of Communication of Science and augmented reality could be a good choice.
Technology, University of Science and Therefore, in this paper, authors combs the
Technology of China, Hefei, Anhui, China literature research related to AR support education
and focuses on the combination of interactive AR
and education in a variety of technical mean;, it is
Synonyms excepted that there could be a general interpreta-
tion of the application of interactive AR in
Augmented reality; Human-computer interaction education.
994 Interactive Augmented Reality to Support Education

The Educational Research on disciplines; in this section, some applications

AR-Supported Education were presented.

In 2010, Johnson et al. proposed AR technology 1. In a chemistry course

as a key educational technology for the next Su Cai, Xu Wang, and Feng-Kuang Chiang,
5 years (Johnson et al. 2010). the three researchers, designed and developed
Santoso, Yan, and Gook from the Institute of a set of inquiry-based augmented reality learn-
Ambient Intelligence (IAI) work together with ing tools. Students could control, combine, and
Sungsan Elementary School (SES) by developing interact with a 3D model of microparticles
a digital edutainment content based on Tangram using markers and conduct a series of
toy as an existed edutainment media (Santoso inquiry-based experiments (Cai et al. 2014).
et al. 2012). 2. Botany
Researchers also considered the AR technol- Based on Kolb’s experiential learning the-
ogy to be integrated in the physical classroom ory, Huang et al. develop an eco-discovery
environment (Bujak et al. 2013) and proposed AR-based learning model (EDALM) which is
AR design principles for classroom (Cuendet implemented in an eco-discovery AR-based
et al. 2013). learning system (EDALS). In a field experi-
In 2014, Mhd Wael Bazzaza, Buti Al Delail, ment at a botanical garden, 21 middle school
and M. Jamal Zemerly describes in their paper students constitute three groups participated in
how an immersive augmented reality (iAR) appli- a learning activity using different learning
cation in conjunction with a book can act as a new types and media (Huang et al. 2016).
smart learning method by engaging as many of the 3. Mathematics
user’s senses and human functions as possible In 2014, Peter Sommerauer and Oliver
(Bazzaza et al. 2014). Müller designed and conducted a pretest–post-
Through the applications of augmented reality, test crossover field experiment with 101 partic-
users can interact with virtual objects that are ipants at a mathematics exhibition to measure
interposed on real scenes around them and obtain the effect of AR on acquiring and retaining
the most natural and genuine human–computer mathematical knowledge in an informal learn-
interaction experience (Cai et al. 2014). ing environment. They hypothesized that visi-
In 2015, Murat Akçayır and other collaborators tors acquire more knowledge from augmented
investigated the effects of the use of augmented exhibits than from exhibits without AR
reality (AR) technologies in science laboratories (Sommerauer and Muller 2014).
on university students’ laboratory skills and atti- 4. Biology and Medical Science
tudes toward laboratories (Clark et al. 2011). In In 2012, Tang and Ou carried out an exper-
2016, Tien-Chi Huang, Chia-Chen Chen, and iment using AR and mobile technologies as an
Yu-Wen Chou, based on Kolb’s experiential learn- assistant tool for learning butterfly ecology
ing theory, they develop an eco-discovery (Tarng et al. 2013).
AR-based learning model (EDALM) which is Sylvain Bernhardt et al. proposed a new
implemented in an eco-discovery AR-based learn- approach to automatically register the recon-
ing system (EDALS) (Huang et al. 2016). struction from an intraoperative CT acquisition
with the static endoscopic view, by locating the
endoscope tip in the volume data (Bernhardt
Application of AR Technology in et al. 2014).
Different Disciplines 5. Painting Appreciation
Kuo-En Chang et al. designed an aug-
Many researches and developments had been mented reality auxiliary tool for painting
made to apply AR technology in different appreciation. It’s a mobile guide system that
Interactive Augmented Reality to Support Education 995

integrates art appreciation instruction with children to gain historical knowledge about
augmented reality (AR) that was designed as Medieval Amsterdam (Nincareana et al. 2013).
an auxiliary tool for painting appreciation. 9. Language
After the experiment, they made the discovery Rose and Billinghurst (1995), Barreira et al.
that most of the visitors using the mobile (2012), Miyosawa et al. (2012), and Li, S. and
AR-guide system elicited positive responses Chen, Y. (2015) developed AR tools for teach-
and acceptance attitudes (Clark and Dünser ing Japanese and English language to non-
2012). In 2016, an AR app for iOS named native speakers.
ARart could turn figures in painting works
into animating portraits with vividly expres-
sion and posture. Interfaces and Interactions in AR for
6. Early Childhood Education Education
In 2015, Rabia M. Yilmaz developed edu-
cational magic toys (EMT) with augmented Image/Marker Recognition Based e-Books
reality technology. EMT has included puzzles,
flash cards, and match cards to teach animals, Augmented Reality Books
fruits, vegetables, vehicles, objects, profes- Mark Billinghurst introduced augmented books
I
sions, colors, numbers, and shapes for children resembling print books except that their pages
5–6 years of age in early childhood education have virtual graphics superimposed on them.
(Yilmaz 2016). The virtual content can provide an animated
7. Physical Spaces scene that complements print content and, in
Sara Price and Yvonne Rogers described an some cases, supports simple interactivity
approach for developing digitally augmented (Billinghurst and Dunser 2012).
physical spaces. They claim that getting chil-
dren to interact with the physical world, Mixed Reality Book
resulting in relevant augmented digital infor- Raphael Grasset, Andreas Dunser, and Mark
mation appearing and which can subsequently Billinghurst focused creating a mixed reality
be interacted with, is what can facilitate active book based on an already published book. With
learning (Price and Rogers 2004). a mixed reality book, they propose to combine and
8. History and Archeology seamlessly merge physical and virtual content in
Martín, Díaz, Cáceres, Gago, and Gibert more meaningful ways. The representation of
presented an educational application called content can be either real or virtual or a mixture
EnredaMadrid to cope with this complexity. of both (Grasset et al. 2007).
The objective of EnredaMadrid is to teach the Adrian Clark and Andreas Dünser present a
history of the city in the seventeenth century to new experience utilizing augmented reality-
students in the activity through previous online enhanced books. Users are able to color in the
training and a later physical technological pages, and these pages are then recognized by
gymkhana (Martin et al. 2011). the system and used to produce three-dimensional
Ardito, Buono, Costabile, Lanzilotti, and scenes and textured models reflecting the artwork
Piccinno presented a MAR game called created by the users. This three-dimensional vir-
Explore! with the aim to support during a visit tual content is then overlaid on the real book
and explorations of middle school students to pages, providing a three-dimensional experience
archeological sites in Italy. Huizenga, Admiraal, using the users own content (Clark et al. 2011).
Akkerman, and Dam have conducted a research
by integrating the MAR games called Fre- Multi-Marker-Based Interaction
quency 1550. This hybrid reality game was Different from the single marker-based AR, the
developed by the Waag Society to facilitate multi-marker-based AR could allow marks to
996 Interactive Augmented Reality to Support Education

cooperate or interact with each other, one of which Head-Mounted Displays (HMDs)
can be used as a trigger, when it is scanned, can Head-mounted display is a kind of display which
trigger some interactive behavior, or can be used is worn on the head or as part of a helmet. It has a
for experiment or game. small display optic in front of one or each eye
Su Cai, Xu Wang, and Feng-Kuang Chiang use (Kesim and Ozarslan 2012).
the position of markers to present different phase In 2007, Arvanitis, Petrou, Knight, Savas,
of a structure and various combinations of atoms. Sotiriou, and Gargalakos developed a project
The markers’ behavior can be consistent with real that is CONNECT; the CONNECT concept
particle behaviors in some cases while inconsis- required student to wear a head-mounted dis-
tent in other cases. For example, when two play (HMD) and related computer-mediated
markers get closer, a new molecule can be formu- learning platform in order to visualize and
lated, which is what really happens in microworld interact physically and intellectually with
(Cai et al. 2014). learning environment that deals with instruc-
Zhang et al. (2017) developed tangible user tional materials, through “hands on” experi-
interface elements based on multi-marker recog- mentation and “minds on” reflection. In
nition for a scientific educational AR book, addition, student can also perform experiments
including virtual buttons, virtual rotate, and vir- that are not possible in school. To evaluate the
tual hotspot. The user elements were integrated usability and effectiveness of the CONNECT
into various kinds of digital presentation systems project, a study has been conducted with
by optimizing the logistic structure and interaction learners with physical disabilities (Arvanitis
design of the user interface system to realize con- et al. 2009).
venient spatial interactions.
Handheld Displays
The Markerless AR In 2009, Dunleavy, Dede, and Mitchell designed
Tai-Wei Kao and Huang-Chia Shih developed a Alien Contact!, a MAR game that focus to teach
markerless augmented reality (AR) applying for math, language arts, and scientific literacy skills to
the picture books. They used the scale-invariant middle and high school students. Alien Contact!
feature transform (SIFT) (Zhao and Ngo 2013; was designed based on Massachusetts state stan-
Zhao and Ngo 2013) to realize the markerless dards and nurtures multiple higher-order thinking
augmented reality application. skills. When the students move around to their
In order to reach the markerless mechanism, spot fields by using the Alien Contact! (Nincarean
they identify the image contours using the point- et al. 2013).
matching algorithm: scale-invariant feature trans-
form (SIFT) to deal with the black rectangular Location-Based Educational AR
framing. The ARtoolkit is used to recognize the The location-based AR systems use the position
object from database and fetch the animations data of mobile devices, determined by the Global
of the corresponding 3D objects. We collect Positioning System (GPS) or WiFi-based posi-
images from the Internet to build the database tioning systems. The location-based AR systems
and extract the SIFT features in advance (Kao enable users moving around with mobile devices
and Shih 2013). in the real environment. Users can observe
computer-generated information on the screens
Mobile Device-Based Interactive AR for of mobile devices, while the information is trig-
Education gered by the current location of the users in an
Due to the rising popularity of mobile devices environment.
globally, the widespread use of AR on mobile CityViewAR is an example of unique experi-
devices such as smartphones and tablets has ential learning. Students can use this mobile
become a growing phenomenon (Nincarean phone application to walk through the city of
et al. 2013). Christchurch and “see” buildings as they were
Interactive Augmented Reality to Support Education 997

Interactive Augmented Reality to Support Education, Fig. 1 Gesture-based interaction for classroom teaching
(Source: Zhang and Zhu (2016). With permission from Springer)

before the 2011 earthquake made it necessary to on transparent screen by using his hands or
demolish them. feet and achieve highly attractive performance.
Figure 1 shows the process of a user interacting
Interact with Virtual Objects by Gesture with a virtual object, zoom in or zoom out, move,
It will also be very wonderful to allow teacher to and rotate them. Here, different gestures will be
interact with virtual objects spatially, which could used to realize different manipulations.
bring much more real feel and deeper immersive
experience to the students also for the teacher. Tangible Augmented Reality (TAR)
Tangible augmented reality (TAR) technology
Gesture-Based Interaction opens a novel realm which integrates the
It provides the teachers with a more effective computer-generated elements into the real word.
way to communicating knowledge to the students Its applications into design education have been
by allowing the teachers to present the educational explored with a limitation to this entire area (Chen
3D contents interactively with spatial AR technol- and Wang 2008). In TAR systems, markers/tags
ogy (Bimber and Raskar 2005). In this scenario, 3D can be added to the text to identify information
virtual objects is displayed on a transparent projec- related to the descriptions in the text and are
tion screen that arranged in front of the podium, detected with an image-processing tool, such as
while students will see their teacher is interacting ARToolkit (Kato et al. 2000).
with the 3D virtual objects just like in the air, so the In 2008, Rui Chen and Xiangyu Wang pre-
students could get much deeper immersive experi- sented and evaluated one TAR system to improve
ence than that in traditional mode. the pedagogical effectiveness of experiential and
Zhang and Zhu (2016) build interaction collaborative learning process in urban design edu-
between Kinect skeleton and virtual objects, cation. For TAR systems, the initial mental image/
which allow teacher to interact with virtual objects model can be gained from reflective observation
998 Interactive Augmented Reality to Support Education

(AR) and tactilely from tangible feedback (tangible Face-to-Face Collaboration by AR

interface) (Chen and Wang 2008). Chien-Huan AR interfaces blend the physical and virtual
Chien, Chien-Hsu Chen, and Tay Sheng Jeng worlds so real objects can interact with 3D digital
had also applied TAR in their study; they aim to content and improve users’ shared understanding
use augmented reality (AR) technology to create an (Billinghurst and Kato 2002).
interactive learning system, which helps medical Hannes Kaufmann and Dieter Schmalstieg cre-
students to understand and memorize the 3D anat- ated various hybrid hardware setups, in order to
omy structure easily with tangible augmented real- complement the diverse teacher–student interac-
ity support (Chien et al. 2010). tion scenarios for educational environments, and
Mark Billinghurst had enumerated the relevant it was used in classroom. The wearable AR kits are
examples in his paper: composed of back pack computer, stereoscopic
“Young children often fantasize about being see-through head-mounted display with camera,
swallowed up into the pages of a fairy tale and and custom pinch gloves for two-handed input.
becoming part of the story. The MagicBook makes One kit can be worn by the teacher, and the second
this fantasy a reality by using a normal book as the one is available for use by students. In addition, it is
main interface object. People can turn the pages of
the book, look at the pictures, and read the text intended to be used by high school students and
without any additional technology. However, if teachers in an interactive, collaborative manner and
they look at the pages through a handheld Aug- to blend seamlessly into an everyday classroom
mented Reality display, they see three dimensional situation (Kaufmann and Schmalstieg 2003).
virtual models appearing out of the pages. The
models appear attached to the real page, so users
can see the AR scene from any perspective simply Remote Collaboration by AR
by moving themselves or the book.” (Billinghurst AR technology can also be used to support remote
et al. 2001b)
collaboration. Mark Billinghurst and Hirokazu Kato,
in an AR conferencing interface they developed in
Interactive and Collaborative Education 1998, demonstrated a user that wore a lightweight
by AR HMD (with camera) and could see a virtual image of
a remote collaborator attached to a physical card as a
In natural face-to-face collaboration, people use life-size, live virtual video window.
speech, gesture, gaze, and nonverbal cues to
attempt to communicate. In many cases, the sur- Multiscale Collaboration by AR
rounding physical world and objects also play an AR techniques can also be used to support multi-
important role, particularly in design and spatial scale collaboration, where users collaboratively
collaboration tasks (Billinghurst and Kato 2002). view a data set from different viewpoints. Mark
In a study by Bressler and Bodzin (Bressler and Billinghurst and Hirokazu Kato explored this in
Bodzin 2013), middle school students collabora- their MagicBook work (Billinghurst et al. 2001a).
tively played an inquiry-based mobile AR game Individual users of the MagicBook interface have
by using mobile devices to scan QR (quick their own independent view of the content; any
response) codes to access game-related informa- number of people can view and interact with a
tion, solve a detective case, and learn forensic virtual model as easily as they interact with a real
science. The study reported that the group play object (Billinghurst and Kato 2002).
of the vision-based AR game can increase stu-
dents’ science interest and their collaboration
skills. Conclusion and Discussion
With the development of AR technology, it
often be used to “shared space” system as a face- In this paper, the authors summarized the applica-
to-face collaboration tool, and it can be used to tions of AR technology in the field of education
support remote collaboration, as well as the multi- and tried to give an outline of the support and
scale collaboration. influence of interactive AR on education.
Interactive Augmented Reality to Support Education 999

With the rapid development of AR technology Billinghurst, M., Kato, H., Poupyrev, I.: The MagicBook –
and the popularization of AR device, AR could Moving seamlessly between reality and virtuality.
IEEE Comput. Graph. Appl. 21(3), 6–8 (2001a)
bring very wide space for education, the possibil- Billinghurst, M., Kato, H., Poupyrev, I.: The MagicBook:
ities of AR interactions for education will be A transitional AR interface. Comput. Graph. 25(5),
explored more and more, and also the utilization 745–753 (2001b)
of AR in different disciplines will be deeper and Billinghurst, M., Grasset, R., Looser, J.: Designing aug-
mented reality interfaces. Comput. Graph. 39(1), 17–22
deeper. (2005)
However, compared to other more mature Bimber, O., Raskar, R.: Spatial Augmented Reality Merg-
technologies that were applied in education, it ing Real and Virtual Worlds. A K Peters Ltd, Natick
is still not so easy to use AR in education; AR (2005)
Bressler, D.M., Bodzin, A.M.: A mixed methods assess-
creation usually needs many technical abilities ment of students’ flow experiences during a mobile
such as programming and 3D modeling and augmented reality science game. J. Comput. Assist.
more, but it is very difficult for normal teachers Learn. 29(6), 505–517 (2013)
to utilize so many technologies, while program- Bujak, K.R., Radu, I., Catrambone, R., MacIntyre, B.,
Zheng, R., Golubski, G.: A psychological perspective
mers are usually not familiar with educational on augmented reality in the mathematics classroom.
contents and the needs of interaction design Comput. Educ. 68, 536–544 (2013)
from educational principles. So an excellent AR Cai, S., Wang, X., Chiang, F.K.: A case study of Aug-
mented Reality simulation system application in a I
educational application should be the result of
chemistry course (vol 37, pg 31, 2014). Comput.
convergence team works of teachers, educational Hum. Behav. 39, 424–424 (2014)
researchers, and technical experts and Chen, R., Wang, X.: An empirical study on tangible aug-
programmers. mented reality learning space for design skill transfer.
Tsinghua Sci. Technol. 13(s1), 13–18 (2008)
Chien, C.H., Chen, C.H., Jeng, T.S.: An interactive
augmented reality system for learning anatomy
References structure. In: Proceedings of the International Multi-
conference of Engineers and Computer Scientists,
Arvanitis, T.N., Petrou, A., Knight, J.F., Savas, S., Vol. 1. Hong Kong: International Association of
Sotiriou, S., Gargalakos, M., Gialouri, E.: Human fac- Engineers (2010)
tors and qualitative pedagogical evaluation of a mobile Clark, A., Dünser, A.: An interactive augmented reality
augmented reality system for science education used by coloring book. IEEE Symp. 3D User Interf. 85, 7–10
learners with physical disabilities. Pers. Ubiquit. (2012)
Comput. 13(3), 243–250 (2009) Clark, A., Nser, A., Grasset, R.: An interactive augmented
Azuma, R. T.: A survey of augmented reality. Presence reality coloring book. 3d User Interfaces (2011)
Teleop. Virt. 6(4), 355–385 (1997) Cuendet, S., Bonnard, Q., Do-Lenh, S., Dillenbourg, P.:
Barreira, J., Bessa, M., Pereira, L.C., Ado, T., Peres, E., Designing augmented reality for the classroom.
Magalhes, L.: Mow: Augmented reality game to learn Comput. Educ. 68, 557–569 (2013)
words in different languages: case study: learning Dalim, C.C., Dey, A., Piumsomboon, T., Billinghurst, M.,
English names of animals in elementary school. In: Sunar, S.: TeachAR: an interactive augmented reality
Information Systems and Technologies (CISTI), 2012 tool for teaching basic English to non-native children.
7th Iberian Conference on, pp. 1–6 (2012) In: 2016 IEEE International Symposium on Mixed and
Bazzaza, M. W., Al Delail, B., Zemerly, M. J., Ng, J. W. P.: Augmented Reality (ISMAR-Adjunct), Merida pp. 82–
iARBook: An immersive augmented reality system for 86 (2016)
education. In: Teaching, Assessment and Learning Grasset, R., Duenser, A., Seichter, H., Billinghurst, M.:
(TALE), 2014 International Conference on. IEEE, pp. The mixed reality book: a new multimedia reading
495–498 (2014) experience. CHI'07 extended abstracts on Human fac-
Bernhardt, S., Nicolau, S.A., Agnus, V., Soler, L., tors in computing systems. ACM, pp. 1953–1958
Doignon, C., Marescaux, J.: Automatic detection of (2007)
endoscope in intraoperative ct image: Application to Huang, T.C., Chen, C.C., Chou, Y.W.: Animating eco-
ar guidance in laparoscopic surgery. Biomedical Imag- education: To see, feel, and discover in an augmented
ing (ISBI), 2014 IEEE 11th International Symposium reality-based experiential learning environment.
on. IEEE, pp. 563–572 (2014) Comput. Educ. 96, 72–82 (2016)
Billinghurst, M., Dunser, A.: Augmented reality in the Johnson, L.F., Levine, A., Smith, R.S., Haywood, K.:
classroom. Computer. 45(7), 56–63 (2012) Key emerging technologies for elementary and sec-
Billinghurst, M., Kato, H.: Collaborative augmented real- ondary education. The Education Digest, 76(1): 36
ity. Commun. ACM. 45(7), 64–70 (2002) (2010)
1000 Interactive Computer Graphics and Model-View-Controller Architecture

Kao, T.W., Shih, H.C.: A study on the markerless aug- Symposium on Mixed and Augmented Reality
mented reality for picture books. In: Consumer Elec- (ISMAR-Adjunct) (2017)
tronics (ISCE), 2013 IEEE 17th International Zhao, W.L., Ngo, C.W.: Flip-invariant SIFT for copy
Symposium on. IEEE, pp. 197–198 (2013) and object detection. IEEE Trans. Image Process.
Kato, H., Billinghurst, M., Poupyrev, I., Imamoto, K., 22(3), 980–991 (2013)
Tachibana, K.: Virtual object manipulation on a table-
top AR environment. IEEE and ACM International
Symposium on Augmented Reality, Proceeding,
pp. 111–119 (2000)
Kaufmann, H., Schmalstieg, D.: Mathematics and geome- Interactive Computer
try education with collaborative augmented reality. Graphics and Model-View-
Comput. Graph. 27(3), 339–345 (2003)
Kesim, M., Ozarslan, Y.: Augmented reality in education:
Controller Architecture
Current technologies and the potential for education.
Procedia Soc. Behav. Sci. 47, 297–302 (2012) Aaron Hitchcock and Kelvin Sung
Martin, S., Diaz, G., Sancristobal, E., Gil, R., Castro, M., Computing and Software Systems, University of
Peire, J.: New technology trends in education: Seven
years of forecasts and convergence. Comput.
Washington Bothell, Bothell, WA, USA
Educ. 57(3), 1893–1906 (2011)
Miyosawa, T., Akahane, M., Hara, K., Shinohara, K.:
Applying augmented reality to e-learning for foreign Synonyms
language study and its evaluation. In: Proceeding of
the 2012 International Conference on E-learning,
E-Business, Enterprise Information Systems, & Model-view-controller (MVC); MVC architec-
EGovernment, pp. 310–316 (2012) ture; MVC design pattern
Nincarean, D., Alia, M.B., Halim, N.D.A., Rahman, M.
H.A.: Mobile augmented reality: The potential for
education. Procedia Soc. Behav Sci. 103, 657–664
(2013) Definition
Price, S., Rogers, Y.: Let’s get physical: The learning
benefits of interacting in digitally augmented physical Interactive graphics applications are a class of
spaces. Comput. Educ. 43(1–2), 137–151 (2004)
Rose, H., Billinghurst, M.: Zengo sayu: An immersive
application that allows users to interactively
educational environment for learning japanese. Univer- update their internal states. These applications
sity of Washington, Human Interface Technology Lab- provide real-time visualization of their internal
oratory, Report No. r-95-4 (1995) states with computer graphics. The model-view-
Santoso, Y., Vignakaran, N., Goldstraw, P.: The value of
geriatric functional syndromes for targeting services.
controller (MVC) architecture is effective for
Australas J Ageing, 31, 48–49 (2012) presenting, discussing, understanding, and
Sommerauer, P., Muller, O.: Augmented reality in implementing this type of application.
informal learning environments: A field experiment in As illustrated in Fig. 1, the Model contains the
a mathematics exhibition. Comput. Educ. 79, 59–68
(2014)
application state, the View renders the model
Tarng, W., Yu, C.S., Liou, F.L., Liou, H.H.: Development graphically, and the Controller modifies the
of a virtual butterfly ecological system based on model. A User interacts with the MVC system
augmented reality and mobile learning technologies. by observing the content of the view and manip-
2013 9th International Wireless Communications
and Mobile Computing Conference (IWCMC),
ulating the controller to alter the state of the
pp. 674–679 (2013) application.
Yilmaz, R.M.: Educational magic toys developed with
augmented reality technology for early childhood
education. Comput. Hum. Behav. 54, 240–248
(2016)
Implementation Considerations
Zhang, Y., Zhu, Z.: Interactive spatial AR for classroom
teaching. In: De Paolis L., Mongelli A. (eds.) Aug- The model defines the persistent application state
mented reality, virtual reality, and computer graphics. and implements interface functions which allow it
AVR 2016. Lecture Notes in Computer Science, vol.
9768. Springer, pp. 463–470 (2016)
to be modified. The model implementation should
Zhang, Y.X., Zhu, Z., Yun, Z.: Empower VR art and AR be independent from the technologies that build
book with spatial interaction. In: 2016 I.E. International the view and controller components. For example,
Interactive Computer Graphics and Model-View-Controller Architecture 1001

the model of an image editing application should application state should be read-only and should
consist only of data structures and algorithms for not be changed.
defining and maintaining the abstract content of Figure 2 illustrates understanding GIMP, an image
images. In this way, different views and control- editor, as an MVC application. In this case, the Model
lers based on distinct libraries can be defined and (in orange), or the application state, is simply the
implemented for the same model. For example, image and information about the image. The view
view/controller implementations for a PC-version (in blue) renders and visualizes the application state as
and a Mac-version are based on the same model. different panes in the application window, and the
One important benefit of the MVC architecture controller (in green) provides the interface for the user
is the clear enforcement of separation between to manipulate and update the image.
state modification and visualization. During state
modification, the controller receives user input
and triggers the model to modify the application Context of Video Games
state. The MVC architecture ensures that the
application state rendering is a completely sepa- Modern video games are examples of interactive
rate process involving the model triggering the graphical applications. Typically, games are built
view. During this visualization stage, the based on specific game engines. As illustrated in
I

Renders View

Output Observes
Display User
Model
Application
State

Controller Manipulates

Modifies Input System

Interactive Computer Graphics and Model-View-Controller Architecture, Fig. 1 The model-view-controller

architecture

Model
Image and info
about the image
Controller

Menu System

View

Main Edit Color Channel Main Edit

Pane Pane Pane

Interactive Computer Graphics and Model-View-Controller Architecture, Fig. 2 GIMP (an image editor) as an
example MVC application
1002 Interactive Design

Interactive Computer
Graphics and Model- Game Engine
View-Controller
Architecture, Update/Draw Game Loop
Fig. 3 Modern video
games and the MVC The Game
architecture (Model)
Game State Camera (View)

Input (Controller)

Fig. 3, the game loop sub-system in the game

engine periodically triggers the game to update
and draw its state. In response, the game invokes
the game engine functionality: the camera sub- Interactive Computer Graphics and Model-View-
system to render, and input sub-system to receive Controller Architecture, Fig. 4 A Unity3D slider bar
user commands. In this way, the game is the
model responsible for defining and maintaining controller typically supports mouse down and drag
the game state, and the view and controller func- events to interactively modify the value in the
tionality are provided by the game engine. model component. A slider bar implementation
Considering a video game as an MVC applica- can choose to include an additional view by echo-
tion ensures the separation of state update and ing the numeric value in a separate textbox. The
draw operations. Game state should only be mod- corresponding controller would allow the user to
ified during the game engine update call, and only modify the numeric value in the textbox. When the
rendered during the game engine draw call. As typing functionality is disabled, the view exists
discussed in the game loop implementation, the without a corresponding controller.
update and draw call frequencies are typically
independent and can vary with the underlying
system performance. Any attempts to draw the Cross-References
game state during update cycles or change the
game state during draw cycles can easily result ▶ Character Animation Scripting Environment
in a chaotic and unmanageable system. ▶ Decoupling Game Tool GUIs from Core
Editing Operations
▶ Game Engine
Applying the MVC ▶ Game Loop and Typical Implementation
▶ Physical, Virtual, and Game World Persistence
It is interesting that the MVC architecture can be
applied to interactive graphical systems of any
scale. For example, the slider bar shown in Fig. 4
is a fully functional graphical interactive system. In Interactive Design
this case, the model is a numeric value (typically a
floating-point number), the view presents the ▶ 3D Room Layout System Using IEC (Interac-
numeric value to the user, and the controller allows tive Evaluational Computation)
the user to interactively modify the value. A typical ▶ Foundations of Interaction in the Virtual Real-
view draws icons (bar and knobs) representing the ity Medium
range and current value in the model, whereas the ▶ Virtual Reality Retailing
Interactive Room Layout 1003

Interactive Digital Literature Interactive Multimedia

Narrative
▶ Hypermedia Narrative as a Tool for Serious
Games ▶ Hypermedia Narrative as a Tool for Serious
Games

Interactive Displays
Interactive Multimedia
▶ Experiential Media: Using Machine Vision and Scenarios
Sensor-Input to Create Dynamic Real-Time Gen-
erated Media ▶ Timed Automata for Video Games and
Interaction

Interactive Game
I
Interactive Music
▶ King of Fighters, a Brief History
▶ Adaptive Music

Interactive Game Design

Interactive Music Systems
▶ Biosensing in Interactive Art: A User-Centered
Taxonomy ▶ Dynamic Music Generation: Audio Analysis-
Synthesis Methods

Interactive Genetic Algorithm

Interactive Narratives
▶ 3D Room Layout System Using IEC (Interac-
tive Evaluational Computation) ▶ Narrative Design

Interactive Learning Interactive Ray Tracing

▶ Gamification and Social Robots in Education ▶ Ray Tracing in Video Games

Interactive Room Layout

Interactive Learning Events
▶ 3D Room Layout System Using IEC (Interac-
▶ Gamification and Serious Games tive Evaluational Computation)
1004 Interactive Sound Design

appreciate this art form, such as computer anima-

Interactive Sound Design tion, digital art, and virtual reality, among others.
Thus, the main objective of this project is to
▶ Biosensing in Interactive Art: A User-Centered create an animated story that will use a CAVE, a
Taxonomy holographic projection, and a tracking system as
necessary tools to tell the story and not only as a
way to project it. With the use of this technology,
we are seeking to generate a similar or a better
Interactive Storytelling cinema-like viewing experience for the users.
In this entry, we will introduce the prototype of
▶ Narrative Design the first phase of our investigation. This small-
scale prototype simulates the performance
explained above. The goal is to test the response
of the users to this prototype.
Interactive Virtual Reality
Navigation Using Cave
Automatic Virtual Brief Description of the Project
Environment Technology
CAVE
Rodrigo Torres1 and Marlene Palafox2 A CAVE system is built using three different
1
Instituto Tecnológico y de Estudios Superiores screens. Through the use of stereo glasses, the
de Monterrey, Mexico City, Mexico user is introduced to a three-dimensional
2
Mexico City, Mexico environment.

Hologram
Synonyms A light source originated from the top of the
CAVE illuminates a contained reflective object.
CAVE; Cave automatic virtual environment The diffraction generated from the interaction
between the light and the object produces a
computer-generated 3D holographic animation.
Definition
Tracking System
Although interactive systems already exist in the A tracking system captures the user’s movement.
animation industry, nowadays there is not a sys- The gestures generated with this system stimulate
tem that combines an animated story and a vir- and put in action the holographic animation and
tual reality system complemented by the the graphic elements inside the CAVE environ-
spectator’s interaction, in order to tell that story ment (Fig. 1).
properly. The possibilities of how a story can be told are
infinite, since there are lots of combinations that
can be done between the use of the CAVE, the
Introduction hologram, and the user’s interaction.

Over the years, animation has had a great impact

on society. It has been used to entertain, but also to Description of the Prototype
inspire, inform, and educate. In the last century,
animation has been transformed in terms of find- The following elements were used in the creation
ing new ways to achieve it. The use of new tech- of this prototype in order to simulate the perfor-
nologies has provided entirely new forms to mance of the project:
Interactive Virtual Reality Navigation 1005

Interactive Virtual
Reality Navigation Using
Cave Automatic Virtual
Environment
Technology,
Fig. 1 Diagram that
illustrates the project

Interactive Virtual
Reality Navigation Using I
Cave Automatic Virtual
Environment
Technology,
Fig. 2 CAVE prototype

Three 18.500 * 11.600 Monitors When the user makes a movement with his
These monitors are used to simulate the CAVE hand, the leap motion catches it and both, the
(Fig. 2). animal and the environment, answer with an ani-
mation. The animal moves in its individual way
One 18.500 * 11.600 Monitor according to which animal is being used (duck,
This monitor is used as a light source above the firefly, or elephant). The user can change the ani-
CAVE to project the hologram. mal by making another gesture. Since each animal
has its unique voxelized environment, when the
Reflection System
animal is changed, also thus the environment
A reflective pyramid used to simulate the holo-
(Fig. 5).
graphic projection (Fig. 3).

A Leap Motion Controller

Used to track the user’s gestures.
Experiments and Results
Overview
For this prototype we designed a Unity project. In Fifty people tested the prototype (18–40 years).
this project there is a 3D model of an animal and The obtained results were very positive in
an interactive voxelized environment (Fig. 4). terms of interaction and entertainment.
1006 Interactive Virtual Reality Navigation

Interactive Virtual
Reality Navigation Using
Cave Automatic Virtual
Environment
Technology,
Fig. 3 Holographic
projection

Interactive Virtual
Reality Navigation Using
Cave Automatic Virtual
Environment
Technology, Fig. 4 The
Unity project

Interactive Virtual Reality Navigation Using Cave Automatic Virtual Environment Technology, Fig. 5 Models
used for the prototype

Ninety-five percent of the testers liked to be the The average time that testers took to under-
directors of the actions that were projected in the stand the functioning of the holographic and
CAVE (Fig. 6). CAVE system was 1.67 min. Moreover, users
Eighty-five percent of the testers expressed that were allowed to interact with the prototype for as
the interaction system was easy to understand and long as they liked, and the average time was
to learn (Figs. 7 and 8). 9.3 min.
Interactive Virtual Reality Navigation 1007

Interactive Virtual
16%
Reality Navigation Using
Cave Automatic Virtual
Environment
Easy to understand and to
Technology,
learn
Fig. 6 Interactive design
graph 84% Difficult to understand and
to learn

Interactive Virtual 3.5

Reality Navigation Using
Cave Automatic Virtual 3
Environment
2.5
Technology, Fig. 7 Time
used to understand the 2
system graphic
1.5
I
1

0.5

0
User 1
User 4
User 7
User 10
User 13
User 16
User 19
User 22
User 25
User 28
User 31
User 34
User 37
User 40
User 43
User 46
User 49
Time (minutes)

Interactive Virtual 18
Reality Navigation Using 16
Cave Automatic Virtual
14
Environment
Technology, Fig. 8 Time 12
that users liked to interact 10
with the prototype 8
6
4
2
0
User 1
User 4
User 7
User 10
User 13
User 16
User 19
User 22
User 25
User 28
User 31
User 34
User 37
User 40
User 43
User 46
User 49

Time (minutes)

Eighty percent of the testers considered that the Data indicates that the system was an easy one
aspects of the prototype that could be improved are to understand and followed the interactive design;
the number of actions that the user can make. This and very importantly, it proved to be entertaining
change would increase the possible stories or results, for them.
giving an illusion closer to the role of a storyteller.
1008 Interior Design

Further Work
Intrusion Detection System
To create a whole animated story that uses the
holographic projection and the CAVE in its narra- ▶ IPv6 Common Security Vulnerabilities and
tive structure, then test the simulation with a real Tools: Overview of IPv6 with Respect to Online
holographic projection in a real-scale CAVE with Games
several users at the same time. This test will
include:

• How the animated story functions in this iPad Animation

platform
• User’s response ▶ Exploring Innovative Technology: 2D Image
• Different and new ways in which users could Based Animation with the iPad
interact, since there will be several of them

Conclusion and Discussion IPv6

The use of new technologies can contribute a lot ▶ IPv6 Common Security Vulnerabilities and
to storytelling. They provide to the artists different Tools: Overview of IPv6 with Respect to Online
perspectives of how to push an animated project, Games
in order to come up with more interactive and
interesting results.
IPv6 Common Security
References Vulnerabilities and Tools:
Overview of IPv6 with Respect
Craig, A.B.., Sherman, W.R., Will, J.D.: Developing Vir-
tual Reality Applications – Foundations of Effective
to Online Games
Design. Elsevier, Burlington (2009)
Durlanch, N.I., Mavor, A.S.: Virtual Reality: Scientific and Mostafa Tajdini1 and Hoshang Kolivand2
Technological Challenges. National Academy of Sci- 1
Staffordshire University, Stoke on Trent, UK
ences, Washington, DC (1995) 2
Department of Computer Science, Faculty of
Elmorshidy, P. A.: The holographic projection technology.
Gulf Univ. Sci. Technol. J. Telecommun. (2010, May) Engineering and Technology, Liverpool John
First International Conference, AVR 2014: Augmented and Moores University (LJMU), Liverpool, UK
Virtual Reality – Selected Papers. First International
Conference, AVR 2014, Lecce. (2014, Sept 20)
Giglio, V. S.: Sensory Interactive Multimedia Entertain-
ment Theater. US (1996, Oct 3) Synonyms
Hariharan, P.: Basics of Holography. University Press,
Cambridge, UK (2002) Intrusion detection system; IPv6; Network secu-
The Agency for Science, Technology and Research
rity; Vulnerabilities
(A*STAR): Full-color moving holograms in high reso-
lution. ScienceDaily. Retrieved September 26, 2015
from www.sciencedaily.com/releases/2015/02/15020
4090101.htm (2015, Feb 4) Definition

An Internet Protocol Version 6 (IPv6) is the most

Interior Design recent generation of Internet Protocol.
An Intrusion Detection System (IDS) is a device
▶ 3D-Rendered Images and Their Application in or software application that monitors a network for
the Interior Design malicious activity or policy violations.
IPv6 Common Security Vulnerabilities and Tools 1009

A Network Intrusion Detection Systems (NIDS) of IPv4 and covers the biggest limitation of IPv4
is a system that analyzes incoming network traffic. which is the lack of enough addresses for all
Internet users. In recent years, the major service
providers have started to offer IPv6 addresses to
Introduction their users (Icann 2011). Based on a report from
Google on 16 Aug 2018, 23.91% of the users that
Our world has become a big network where every- access Google are over IPv6. This report shows
one connects to it by the Internet. Most people how usage of IPv6 has grown during the last
living on the earth rely on this network. They are couple of years. With IP being the Internet’s
reading news, transferring money, and checking main protocol, many constitutive Internet technol-
their emails and much more in their daily basic ogies are heavily tied to it and the change to
life. The goal of the new modern world is the version 6 resulted in updates of related protocols
availability, integrity, and confidentiality of this (Fig. 1).
network. The rapid growth and widespread use of The major changes between IPv4 and IPv6 can
electronic devices and data processing (cloud described as:
computing, web application, Internet network,
wireless networks, and private network) will • IPv6 large address:
I
raise the need for a solution that can provide a
safe and secure infrastructure for a safe commu- IPv6 is 128bit, this mean, it can provide 340
nication. To use the Internet, each device needs to trillion, trillion, trillion IPv6 addresses. That
have an Internet Protocol (IP) address. An IP means IPv6 uses 128bit address space.
address is a unique number that is assigned to
every device that is connected to the network or • IPv6 fragmentation:
Internet. The IP address enables devices to com-
municate with each other. There are two different IPv6 is no longer required to be fragmented by
versions of IPs, Internet Protocol Version 4 (IPv4) the router. All fragmentation and reassembly are
and Internet Protocol Version 6 (IPv6). IPv4 was performed by sender and receiver host(s).
developed in the early 1980s, but because of the
rapid growth of the Internet, IPv4 has been fully • Addressing:
allocated to Internet Services Providers and Inter-
net users, and then there was a shortage of IPv4 IPv6 uses three types of addresses which are
available address (Icann 2011). IPv6 was stan- unicast, multicast, and anycast. Unicast is only
dardized in 1996 to replace the current version assigned to a single node of IPv6; however, a

IPv6 Common Security 32 bits

Vulnerabilities and
Tools: Overview of IPv6
with Respect to Online Version Traffic class Flow Label
Games, Fig. 1 IPv6
header
Next
Payload length Hope limit
Header

Source IP Address

Destination IP Address
1010 IPv6 Common Security Vulnerabilities and Tools

multicast is assigned to multiple nodes in a single dropped to reduce the cost of packet handling and
multicast group. processing. However, this has been changed in IPv6
where the Extension Headers function is added. The
• Auto-configuration: Extension Headers are placed between IPv6 header
and the upper layer header in a packet, and each of
New capabilities of IPv6 that allow a new node the Extension Headers is identified by a distinct next
automatically configure IP addresses. header value. As this is an optional field, each IPv6
packet can have zero, one, or more extension
• Extension headers: headers. Each Extension Headers have multiple of
8 octets long Fig. 2.
Referring to RFC 2460 (Deering and Hinden
1998), a full implementation must include support Internet Control Message Protocol
for six extension headers, which are Hop-by-Hop Version 6 (ICMPv6)
Options, Routing (Type 0), Fragment, Destination Unlike ICMP for IPv4, ICMP for IPv6 (Conta and
Options, Authentication Headers, and Encapsulat- Deering 2006) play an important role in IPv6
ing Security Payload. network. ICMPv4 is not required in IPv4, but
ICMPv6 is a required element and therefore it
Extension Headers cannot be filtered completely. ICMPv6 has a
Apart from expanded addressing capabilities, one next header value of 58. The main reason that
of the most important and significant changes in ICMP was developed as a protocol was to be
IPv6 is the improvement of supporting extension used for tests and diagnosis on IPv4 networks.
header with the options (Deering and Hinden1998). The most important features that ICMP provides
In IPv4, some of the header fields have been are to enable the utilities such as ping and trace

IPv6 Common Security

Vulnerabilities and Ver Traffic Class Flow Label
Tools: Overview of IPv6
with Respect to Online Payload Length Next Header = UL Hop Limit
Games, Fig. 2 IPv6 40 Octets
packet with extension Source IPv6 Address
header
Destination IPv6 Address

Upper Layer Header Payload

IPv6 Packet with Extension Headers

Ver Traffic Class Flow Label

Payload Length Next Header = EH1 Hop Limit

40 Octets
Source IPv6 Address

Destination IPv6 Address

Next Header = EH2 Extension Header 1

Next Header = EH3 Extension Header 2

Next Header = UL Extension Header 3

Upper Layer Header Payload

IPv6 Common Security Vulnerabilities and Tools 1011

route to help verify end-to-end IP communication the default gateway, and other information that
and connectivity and provide information about can help them communicate.
any errors on the connection back to nodes • Echo Request and Echo Reply support the
(Davies and Mohacsi 2007). ICMPv6 messages Ping6 utility.
can be categorized into two categories (Davies • PMTUD determines the proper MTU size for
and Mohacsi 2007): communications.
Error messages: • Multicast Listener Discovery (MLD) provides
IGMP-like functionality for communicating IP
• 1 Destination Unreachable multicast join and leave.
• 2 Packet Too Big • Multicast Router Discovery (MRD) discovers
• 3 Time Exceeded multicast routers.
• 4 Parameter Problem • Node Information Query (NIQ) shares infor-
• 100 Private experimentation mation about nodes between nodes.
• 101 Private experimentation • Secure Neighbor Discovery (SEND) helps
• 127 Reserved for expansion of ICMPv6 error secure communications between neighbors.
messages • Mobile IPv6 is used for mobile communications.
I
Informational messages: Neighbor Discovery Protocol (NDP)
As defined in RFC2461, Neighbor Discovery is a
• 128 Echo Request protocol for IPv6. Since Address Resolution Pro-
• 29 Echo Reply tocol (ARP) has been removed in IPv6, both hosts
and routers use Neighbor Discovery messages to
Error messages will generate a report of any determine the link layer addresses of nodes on the
errors that occur during the message delivery. local link. When a host is connected to an IPv6
Informational messages will allow sharing of network, it sends Router Solicitation messages to
required information between nodes. As in other routers on the same link to get network information
features, attackers may use ICMP for exploitation, such as network prefix, default router, and other
and therefore sys-admin has no choice but to network parameters. Stateless Auto-Configuration
completely filter the protocol to prevent such is another feature based on Neighbor Discovery
attacks (DoS/DDoS, Evasion, Scan, Man in the Protocol which allows new hosts on the local link
Middle) (Davies and Mohacsi 2007). However, to get and configure their IPv6 address (Thomson
unlike ICMPv4, ICMPv6 cannot be filtered/ and Narten 2007) (Table 1).
blocked completely due to the important role that The transition from IPv4 to IPv6 should have
it plays in the IPv6 network. According to RFC eliminated any related security issue to the new
4890, filtering ICMPv6 on routers and firewalls is protocol. The security mechanisms for network
different from on a host. ICMPv6 is a required layer protocol should be examined in many dif-
protocol on every IPv6 network. ICMPv6 provides ferent areas. One of these areas is how Operating
the following functions (Davies and Mohacsi Systems handle the IPv6 fragmented packet and
2007): how Network Intrusion Detection Systems can
detect an attack on the IPv6 network. If used
• Neighbor Discovery Protocol (NDP), Neigh- properly by an attacker, this feature in IPv6 can
bor Advertisements (NA), and Neighbor Solic- lead to Network Intrusion Detection System
itations (NS) provide the IPv6 equivalent of (NIDS) evasion, Firewall evasion, Operating Sys-
IPv4 Address Resolution Protocol (ARP) tem fingerprint, Network Mapping, Denial of Ser-
functionality. vice (DoS)/Distributed Denial of Service (DDoS)
• Router Advertisements (RA) and Router Solic- attack, and Remote code execution attack (Ptacek
itations (RS) help nodes determine information and Newsham 1998; Erickson 2007; Chen 2014;
about their LAN, such as the network prefix, Reese 2009).
1012 IPv6 Common Security Vulnerabilities and Tools

IPv6 Common Security Vulnerabilities and Tools: Overview of IPv6 with Respect to Online Games,
Table 1 Neighbor Discovery Protocol (NDP) messages
ICMPv6 Source
type Message name address Receiver address Used for
133 Router Nodes FF02::2 (multicast Sent by hosts to locate routers on attached
Solicitation (routers link address) link
(RS) local
address)
134 Router Routers Sender of RS or Routers advertise their presence and link
Advertisement FF02::1 prefixes, MTU, and hop limits
(RA)
135 Neighbor Nodes Solicited node To query for other nodes link layer address
Solicitation multicast address or and also used for duplicate address
(NS) the target node’s detection and to verify neighbor
address reachability
136 Neighbor Nodes In response to NS In response to NS query
Advertisement sender or to FF02::1
(NA)
137 Redirect Routers Link local address To inform other nodes for better next hop
routers

IPv6 Vulnerabilities process is different in Evasion attack; in Evasion

attack the IDS rejects the packet that the end host
Despite the security improvements in IPv6, some accepts. The attacker can send some or all mali-
vulnerabilities are still common between IPv4 and cious traffic into the network without being caught
IPv6. Insertion, Evasion, and Denial of Service by the IDS.
are three different categories of attacks, which
were proposed by Ptacek and Newsham (1998) IPv6 Fragmentation Attack
for the first time. Most of the vulnerabilities are Referring to all the aforementioned (RFC2460,
common between IPv4 and IPv6 (Mali et al. 2015; 1998; RFC 3964, 2004; RFC 7123, 2014) recom-
Satrya et al. 2015; Tripathi and Mehtre 2013), and mendations of corresponding IPv6 Requests for
because of the changes that were made in the IPv6 Comment (RFCs) and previous sections, when
implantation (Deering and Hinden 1998), addi- using IPv6 Extension Headers and IPv6 Fragmen-
tional vulnerabilities arise as well. There are tation, there are potential attacks against the Oper-
many features which are new and unique to ating System (OS). In case of discrepancies
IPv6. One of them is the improved support of between the behavior of several OS, this can
headers (extensions and options) which were not lead to OS fingerprinting, Intrusion Detection
existing before in IPv4. System (IDS) insertion and IDS evasion, and
Insertion, Evasion, and Denial of Service are Firewall evasion. Furthermore, there are still
three different categories of attacks, which were some issues regarding the handling of the IPv6
proposed by Ptacek and Newsham (1998) for the fragments (Atlasis 2012). One of the simplest
first time. The aim of these attacks is to make the examples of the one of the most common attacks
IDS or victim host process different data or pro- can be fragmentation attack, which is common
cess the same data but differently. By using an between IPv4 and IPv6 (Atlasis 2017).
insertion attack, IDS accepts a packet(s) that is Several IPv6 fragmentation and overlapping
rejected by the host. The packet looks valid only methods were used to test the effectiveness of
to the IDS. The attacker can bypass the signature- some of the most popular OS, and it is found that
based IDS by inserting the traffic in such way that none of them is fully RFC compliant while most of
the signature is never matched or found. This them seem to have significant issues (Fig. 3).
IPv6 Common Security Vulnerabilities and Tools 1013

IPv6 Common Security Vulnerabilities and Tools: Overview of IPv6 with Respect to Online Games, Fig. 3 IPv6
fragmentation process
I
Operating System Fingerprint informational messages such as ECHO (request
This method is used for identification of the victim and reply), Router Advertisement, Neighbor Adver-
host. In human life, fingerprints can be used to tisement, Neighbor Solicitation (NS), and Multicast
identify a person. Similarly, an OS has its unique Listener Discovery messages for a successful attack
implementation of communication protocols by (Chen 2014).
which it can be identified. In order to identify the
OS and its version remotely and without having a ICMPv6 Amplification
direct access to that system, the attacker uses The amplification attack is considered as one of
fingerprinting to analyze certain characteristic the common security challenges in IPv4 and still
and network behavior communication (Eckstein exists in IPv6. The amplification attack allows the
and Atlasis 2011). By using such a method, the attackers to generate huge numbers of packets
attacker can easily discover the live host on the using a small number of packets and amplify it
network and identify their OS, and furthermore by to a large number of packets based on the multi-
using this method, the attacker could even reveal cast address feature.
the victim host’s missing security patches or ser- Broadcast Amplification attack also known as
vice packs. As a result, the attacker can easily use Smurf (Fig. 4) is the most well-known amplifica-
the related vulnerability to gain access to and tion attack, which is based on ICMPv6 multicast
control the end host easily (Allen 2007). address function. The attacker uses Smurf attack
to launch a DoS attack by sending an ECHO
ICMPv6 Flooding Attack request packet to a multicast address with spoofed
ICMPv6 flooding attack is one of the most com- source address of victim machine. Once all nodes
mon attacks in both IP versions. The aim of using of the targeted multicast address have received a
ICMPv6 attack is to use all of a victim’s resources packet, all nodes start to reply to the source, which
(bandwidth, CPU, and RAM) by sending a large is the victim, and flood it with a large number of
amount of traffic. The packet can contain any ECHO reply attacks. The victim will be over-
ICMPv6 type with source address referring to whelmed and cannot respond to genuine requests
another node on the network (Martin and Dunn (Martin and Dunn 2007). In addition, there is
2007). another version of Smurf attack which is called
To disturb the communications between routers rSmurf (Remote Smurf) attack that has stronger
and hosts, an attacker can use ICMPv6 error or amplification, because each packet generated by
1014 IPv6 Common Security Vulnerabilities and Tools

IPv6 Common Security Vulnerabilities and Tools: Overview of IPv6 with Respect to Online Games,
Fig. 4 Smurf attack

rsmurf6 can generate a large number of packets on parameter (Tripathi and Mehtre 2013). An
the remote LAN. As a result, one malicious packet attacker can misuse Router Solicitation and
will generate a storm of traffic on victim network. Router Advertisement packet and perform the
following attacks:
ICMPv6 Protocol Exploitation
By sending a Router Advertisement (RA) packet, • Default router is “killed”: By default, every
any node on a network can claim that they are a node has a router table to list all routers on
router. An attacker can use this feature of ICMPv6 the network. When a node does not have any
to perform a Man in the Middle (MitM) attack by record in the table, it will consider that all
presenting themselves as a router. The first destinations are on link (Narten et al. 2007).
method an attacker can use to launch a Dos/DDoS Now an attacker can send a Router Advertise-
attack is by using Router Discovery packets, ment packet with router lifetime equal to zero
which are Router Solicitation (ICMPv6 type and spoofed address. When the host receives
133) and Router Advertisement (ICMPv6 134). the Router Advertisement packet, it will delete
The second method will be using Neighbor Dis- the router record because of the lifetime, and
covery (ICMPv6 types 135 and 136) packets. The then it will redirect all packets to the destina-
third method will be using Redirect furcation tion without a router address. If the traffic is
(ICMPv6 type 137) packets. going outside of network, all packets will be
The Router Discovery process is responsible lost, and therefore an attack has occurred
for packet routing. On the IPv6 network, a host (Tripathi and Mehtre 2013).
will find a router by sending a Router Solicitation • Bogus address configuration prefix attack: As
packet to router multicast address (FF02::2). Once mentioned earlier, one feature of IPv6 is that in
the Router Solicitation packet is received by the absence of DHCP server, a node will generate
default router, in response to that packet, the their own IPv6 using Stateless Auto-Con-
router will send Router Advertisement to the figuration with subnet prefixes of Router Adver-
host. The Router Advertisement packet contains tisement messages that are received from a default
the information needed by the host such as router router (Kempf and Nordmark 2004). The router
specification, onlink prefix, and network sends Router Advertisement messages
IPv6 Common Security Vulnerabilities and Tools 1015

accordingly to all nodes to update their routing Duplicate address detection DoS attack:
table information. By sending a Router Advertise- Another feature on IPv6 network is Duplicate
ment message with invalid subnet prefix to multi- Address Detection (DAD). When a node needs a
cast address (FF02::1), an attacker can launch an new IPv6 address, it will send Neighbor Solicita-
attack. Now all nodes will generate an invoice tion to all-nodes multicast address “FF02::1” to
IPv6 address based on the invalid prefix that was check whether that IP is in use or not. If the sender
received, and all communication between hosts did not receive a reply, that means the IPv6
will be disrupted. address is free and the new node can use it. An
• Parameter spoofing: As mentioned earlier, attacker can use this as an advantage and send a
Router Advertisement messages contain net- spoofed Neighbor Advertisement packet claiming
work parameter information, and they are that the address is in use every time that node
very useful to the host to send IPv6 packets sends a request. By using such an attack, the
later. An attacker can send a Router Advertise- new nodes will not get an IPv6 address, and there-
ment message (e.g., with a small hob limit), fore there is not any connectivity (Fig. 5) (Zhao-
which contains false network parameters that Wen et al. 2007).
can disturb the packet transmission and host’s Neighbor Unreachability Detection failure:
communications. Neighbor Unreachability Detection (NUD) process
I
detects when a neighbor is unreachable. Once this
Neighbor Discovery Attack has happened, the node starts to send a Neighbor
Neighbor Solicitation and Neighbor Advertise- Solicitation packet to lost node address and waits
ment are two ICMPv6 messages that Neighbor for a Neighbor Advertisement reply for a short
Discovery Protocol (non-routing one) uses. Two period. If no Neighbor Advertisement is received,
of the most important jobs that NDP is responsible the node will delete the peer node from its Neigh-
for are neighbor unreachability and Duplicate bor Cache Entry table. An attacker can send a
Address Detection (DAD). An attacker can use malicious Neighbor Advertisement reply to a
these functions as an advantage and launch an Neighbor Solicitation request to show that the
attack. node is still alive and on the network which it is not.

IPv6 Common Security Vulnerabilities and Tools: Overview of IPv6 with Respect to Online Games,
Fig. 5 DAD attack
1016 IPv6 Common Security Vulnerabilities and Tools

ICMPv6 Redirect Message Attack • redir6: redirect traffic to you intelligently

IPv6 nodes use ICMPv6 “Redirect” message to (man-in-the-middle) with a clever ICMPv6
find a better path to their destination. The router redirect spoofer
will send “Redirect” message to a node to opti- • dos-new-ip6: detect new IPv6 devices and tell
mize the packet routing process and the delivery them that their chosen IP collides on the net-
path. The following attack types can utilize work (DOS)
ICMPv6 redirect messages:
Havij is an automated SQL injection tool that
• The traffic can be forwarded to non-existent link. takes advantage of a vulnerable web application to
• The traffic can be redirected to an existing find and exploit SQL injection vulnerabilities. An
node. This will result in the node being attacker can perform back-end database finger-
overwhelmed. print, DBMS login names and password hashes,
and much more like fetching data from a database.
Gaming company has become more attractive However, this tool is capable of accessing the
targets for attackers during the last couple of underlying file system and executing the operat-
years. Moving the gaming companies Internet ing system shell commands.
Protocol from Version 4 to Version 6 will create Acunetix is a web vulnerability scanner
some vulnerabilities which attackers can use to designed to replicate a hacker’s methodology to
launch ICMPv6, DDoS, and some other malicious find vulnerabilities like SQL injection and
activities by bypassing the detection by using DoS/DDOS attack. By using Acunetix you can
some of IPv6 new features such as Extension use an extensive feature set of both automated and
Headers and Fragmentations. manual penetration testing tools, security analysis
and repair detected threats.
Mendax is a TCP de-synchronizer that injects
Attacking Tools overlapping segments in randomly generated
order. An attacker can use Mendax to evade
This section covers some of the most NIDS. Mendax is not a router, but is a stand-
common attack tools to perform an attack on alone TCP client program which can be used by
IP level. an attacker to perform an evasion from an input
Fragrouter is a network intrusion detection text file, performs a fixed set of evasion technique,
evasion toolkit developed by Dug Sing. It imple- and sends restructured exploit to the victim host
ments most of the attacks described in the Ptacek (Gorton and Champion 2003).
& Newsham in 1998. (It features a simple rule set In Table 2 a summary of Evasion and Insertion
language to delay, duplicate, drop, fragment, etc.). attack tools is provided.
THC-IPV6-ATTACK-TOOLKIT is a col-
lection of attacking tools that can be used to test
the implementation of IPv6 network and test fire- Related Work
wall and NIDS. This collection contains the fol-
lowing tools (van Hauser 2008): Alnakhalny et al. (Saad et al. 2014) proposed a
detection method for ICMPV6 flood attack based
• parasite6: ICMPv6 neighbor solicitation/ on Dynamic Evolving Neural Fuzzy Inference
advertisement spoofer, puts you as man-in-the System (DENFIS). DENFIS is a system that
middle, same as ARP mitm (and parasite) uses online clustering to perform online and
• alive6: an effective alive scanning, which will offline learning. The proposed system is based
detect all systems listening to this address on self-machine learning. However, one impor-
• dnsdict6: paralyzed DNS IPv6 dictionary brute tant question here is if the attacker uses a mixture
forcer of method to bypass the detection, it will take time
• fake_router6: announce yourself as a router on for the machine to learn that algorithm and detect
the network, with the highest priority future attack. Because of that, attackers will
IPv6 Common Security Vulnerabilities and Tools 1017

IPv6 Common Security Vulnerabilities and Tools: Overview of IPv6 with Respect to Online Games,
Table 2 Evasion and Insertion attack tools
Tools name Developer(s) Evasion attack
Fragrouter Dug sing Most techniques described by Ptacek & Newsham
thc-ipv6 Van Hauser Multiple attacking tools including DoS, DDoS, Evasion and Insertion attack
Havij IT Sec team SQL injection and web app evasion
Acunetix Acunetix team Web app analyser and evasion test attack
Mendax Min G. Kang TCP overlapping

change their method, and therefore the detection • Not support the upper layer
method could not be useful for such attack. • Because it needs key exchange, it will use IKE
Anbar et al. (Saad et al. 2016) proposed An management, which requires a valid IPv6
Intelligent ICMPv6 DDoS Flooding attack Detec- address. So it cannot work when a new host
tion Framework (v6IIDS) Using Backpropagation joins a network and therefore is not able to
Neural Network. Their aim is to detect ICMPv6 protect Network Discovery Protocol.
Flooding attack using an Intelligent Intrusion
Detection System in an IPv6 Network (v6IIDS). Because of the complex configuration, most of
I
The proposed system detection has four pro- the users do not implement IPsec for link local
cesses. These processes are data collection and addresses.
pre-processing, traffic analysis, anomaly-based Kempf et al. (Kempf et al. 2005) proposed
detection, and ICMPv6 flooding detection. SEcure Neighbor Discovery (SEND) protocol to
Rafiee et al. (Rafiee and Meinel 2013) pro- mitigate the issue of IPsec for link local commi-
posed a new algorithm to tackle the issue with nution. SEND is an extension of NDP that adds
Cryptographically Generated Addresses (CGA) several options such as Cryptographically Gener-
[3972] and Privacy Extension [4941] in IPv6 ated Addresses (CGA), RSA Signature and
state-less configuration. The proposed method Timestamp, and Nonce Options. In addition,
uses a new way to generate Interface Identifier they introduce four new Authorization Delegation
(IID) to reduce the computing cost and prevent Discovery, Certification Path Solicitation Mes-
security theatres related to state-less configuration sage Format, Certification Path Advertisement
such as IP spoofing. However, it seems the pro- Message Format, Router Authorization Certificate
posed algorithm cannot detect Duplicated Profile and Suitability of Standard Identity Certif-
Address Detection attack on IPv6. icates (Kempf et al. 2005; Securing IPv6 2002).
Kent et al. (Kent and Seo 2005) provided Secu- A review of SEND done by Meinel et al.
rity Architecture for the Internet Protocol. In IPv6 (Alsa’Deh and Meinel 2012). They are
unlike IPv4, Internet Protocol Security (IPSec) is challenging SEND as it is not provided link
mandatory. IPSec draws a line between protected layer security and cover NDP communication
and unprotected interfaces for host or network. If confidentiality. The Cryptographically Generated
traffic want to cross the boundary, they are subject Addresses cannot assure the real node identity.
to the access control list that is specified by the Because of the structure of SEND, it will use
system admin who is responsible for IPSec con- more CPU of nodes and bandwidth to process.
figuration. These controls indicate whether In addition, if Router Authorization and Standard
packets cross the boundary unimpeded, are Identity Certificates implement into routers, It will
afforded security services via AH or ESP, or are put an extra workload on them.
discarded. Hussain et al. (Hussain et al. 2016) proposed a
IPSec provide an end-to-end security between two-stage hybrid classification (Fig. 6) method
end hosts and all intermediate nodes. IPsec has the using Support Vector Machine (SVM) as anomaly
following weaknesses (Yang et al. 2010; Arkko detection in the first stage and Artificial Neural
and Nikander 2005): Network (ANN) as misuse detection in the
1018 IPv6 Common Security Vulnerabilities and Tools

IPv6 Common Security

Vulnerabilities and Detection & Classification
Tools: Overview of IPv6
with Respect to Online Stage-1
Games, Fig. 6 Two-stage SVM Attack
Data
hybrid classification with (Anomaly) Traffic
Preprocess
SVM and ANN (Hussain
et al. 2016)

Normal
Traffic Stage-2
ANN
Network
(Misuse)
Traffic
(Normal +
Attack)

Alarm
module

second. The advantages of using SVM and ANN vulnerabilities arise as well. There are many fea-
are better classification accuracy and a low prob- tures which are new and unique to IPv6. One of
ability of false positive. The proposed system them is the improved support of headers
classifies the type of attack into four classes: (extensions and options) which were not existing
Denial of Service (DOS), Remote to Local before in IPv4.This entry reviewed an overview of
(R2L), User to Root (U2R), and Probe. The first Internet Protocol Version 6 (IPv6), IPv6 new fea-
stage is looking for any abnormal activities that tures, and some of the most common vulnerabil-
could be an intrusion, while the second stage does ities; also, a review was discussed for existing
the future analysis, and if there are any known solutions and how those solutions can mitigate
attacks, it will classify them into the four catego- the vulnerability discussed in this entry. By
ries that were already mentioned. adopting IPv6, gaming industries will become an
Data Preprocess will prepare and pre-process net- attractive target for attackers to launch an attack
work traffic in the data pre-process module. Once data such as ICMPv6 and DDoS to game companies.
has been received and pre-processed, it will be sent to These sorts of attacks already launched to Sony,
the next process, which is “Detection and Classifica- EA, and Steam really affected the gaming industry
tion.” The detection and classification process has two in terms of availability. In addition, the customer
stages: NIDS using SVM for anomaly and ANN for data on gaming companies could be in danger, as
misuse detection. The data then passes to the Alarm attackers could use IPv6 new features to bypass
module, which interprets event results on both stages the detection on NIDSs and do malicious activi-
and reports the intrusion detection activity. ties on gaming server.

References
Conclusion
Acunetix scanner. Accessed 16 Feb 2017. [Online]. Avail-
IPv6 introduces new features and capabilities. able: https://www.acunetix.com/
These results in new issues, and security issues Allen, J.M.: OS and application fingerprinting techniques,
is one of the most important among them. Most of 2007. [Online]. Available: https://www.giac.org/paper/
gsec/8496/os-application-fingerprinting-techniques/
the vulnerabilities are common between IPv4 and 113048
IPv6, and because of the changes that were made Alsa’Deh, A., Meinel, C.: Secure neighbor discovery:
in the IPv6 implantation, additional review, challenges, perspectives, and recommendations,
Islam 1019

IEEE Secur. Priv. 10(4): 26–34 (2012). [Online]. Avail- Martin, C.E., Dunn, J.H.: Internet protocol, version, Vol. 6,
able: https://doi.org/10.1109/MSP.2012.27 pp. 1–7, 2007. [Online]. Available: https://doi.org/10.
Arkko, J., Nikander, P.: Limitations of IPsec policy mech- 1109/MILCOM.2007.4455200
anisms. Lecture notes in computer science (Including Narten, T., Nordmark, E., Simpson, W., Soliman, H.:
subseries, 2005). [Online]. Available: https://doi.org/ Neighbor discovery for, IP version, Vol. 6, 2007.
10.1007/11542322_29 [Online]. Available: https://www.rfc-editor.org/info/
Atlasis, A.: Attacking ipv6 implementation using fragmen- rfc4861
tation. BlackHat Europe, 2012. [Online]. Available: Ptacek, T.H., Newsham, T.N.: Insertion, evasion and denial
http://media.blackhat.com/bh-eu-12/Atlasis/bh-eu-12- of service: eluding network intrusion detection, 1998.
Atlasis-Attacking_IPv6-WP.pdf [Online]. Available: http://www.aciri.org/vern/Ptacek-
Atlasis, A.: The impact of extension headers on IPv6 Newsham-Evasion-98.ps
access control lists real life use cases. Heidelberg Rafiee, H., Meinel, C.: SSAS: a simple secure addressing
(2017) scheme for IPv6 autoconfiguration, 2013. [Online].
Chen, J.G.Y.: Detecting DoS/DDoS attacks under IPv6, Available: https://doi.org/10.1109/PST.2013.6596063
pp. 847–855. Springer, New York City (2014) Reese, G.: Cloud Application Architectures, 1st Edition.
Conta, A., Deering, S.: Internet control message protocol 1st ed. [Place of publication not identified]: O’Reilly
(ICMPv6) for the internet protocol, version, Vol. 6, no. Media, Inc., pp. 2–4 (2009)
6, 2006. [Online]. Available: https://www.rfc-editor. Saad, R.M.A., Almomani, A., Altaher, A., Gupta, B.B.,
org/info/rfc4443 Manickam, S.: ICMPv6 flood attack detection using
Davies, E., Mohacsi, J.: Recommendations for filtering DENFIS algorithms. Indian J. Sci. Technol. 7(2),
ICMPv6 messages in firewalls, 2007. [Online]. Avail- 168–173 (2014)
able: https://www.rfc- editor.org/info/rfc4890 Saad, R.M.A., Anbar, M., Manickam, S., Alomari, E.: An I
Deering, S., Hinden, R.: Internet protocol, version, Vol. 6, intelligent ICMPv6 DDoS flooding-attack detection
1998. [Online]. Available: https://www.ietf.org/rfc/ framework (V6IIDS) using back-propagation neural
rfc2460.txt network. IETE Tech. Rev. (Institution of Electronics
Eckstein, C., Atlasis, A.: OS Fingerprinting with IPv6, and Telecommunication Engineers, India) 33(3),
Infosec reading room, SANS Institute, 2011 244–255 (2016). [Online]. Available: https://doi.org/
Erickson, J.: The art of exploitation, 1–492, 2007. [Online]. 10.1080/02564602.2015.1098576
Available: https://leaksource.files.wordpress.com/ Satrya, G.B., Chandra, R.L., Yulianto, F.A.: The detection
2014/08/hacking-the- art-of-exploitation.pdf of DDOS flooding attack using hybrid analysis in IPv6
Gorton, S., Champion, T.G.: Combining evasion tech- networks. Technology, ICoICT, 2015. [Online]. Avail-
niques to avoid network intrusion detection systems. able: https://doi.org/10.1109/ICoICT.2015.7231429
Skaion corporation, pp. 1–20, 2003. http://www. Securing IPv6 neighbor and router discovery, pp. 77–86,
Skaion.Com/Research/Tgcrsd-Raid.Pdf 2002. [Online]. Available: https://doi.org/10.1145/
Havij. Accessed 17 July 2017. [Online]. Available: https:// 570681.570690
www.darknet.org.uk/2010/09/havij-advanced-automa Thomson, S., Narten, T.: IPv6 stateless address auto-
ted-sql-injection-tool configuration, 2007. [Online]. Available: https://www.
Hussain, J., Lalmuanawma, S., Chhakchhuak, L.: A two- rfc-editor.org/info/rfc4862
stage hybrid classification technique for network intru- Tripathi, N., Mehtre, B.: DoS and DDos attacks: impact,
sion detection system. Int. J. Comput. Commun. Eng. analysis and countermeasures, 7, 2013. [Online]. Avail-
Res. 3(2): 16–27 (2016). [Online]. Available: https:// able: https://www.researchgate.net/publication/259941
doi.org/10.1080/18756891.2016.1237186 506_DoS_and_DDoS_Attacks_Impact_Analysis_
Icann, Internet protocol (ip) addresses. Beginner’s Guide. and_Countermeasures
icann, 2011. [Online]. Available: https://www.icann. van Hauser, THC-IPV6-attack-toolkit, 2008. [Online].
org/en/system/files/files/ip-addresses-beginners-guide- Available: https://github.com/vanhauser-thc/thc-ipv6
04mar11-en.pdf Yang, D., Song, X., Guo, Q.: Security on IPv6, Vol. 3,
Kempf, J., Nordmark, E.: IPv6 neighbor discovery (ND) trust pp. 323–326, 2010. [Online]. Available: https://doi.org/
models and threats. Internet Soc. 1(23), 2004. [Online]. 10.1109/ICACC.2010.5486848
Available: https://doi.org/10.17487/rfc3756 Zhao-Wen, L., Lu-hua, W., Yan, M.: Possible attacks based
Kempf, J., Zill, B., Nikander, P.: SEcure neighbor discov- on IPv6 features and its detection, 2007. [Online].
ery, 2005. [Online]. Available: https://www.rfc-editor. Available: http://master.apan.net/meetings/xian2007/
org/info/rfc3971 publication/031_lin.pdf
Kent, S., Seo, K.: Security architecture for the Internet
protocol, 2005. [Online]. Available: https://www.rfc-
editor.org/info/rfc4301
Mali, P., Phadke, R., Rao, J., Sanghvi, R.: Mitigating IPv6
vulnerabilities, 2015. [Online]. Available: https://www.
colorado.edu/itp/sites/default/files/attached-files/57971- Islam
97277_-_ronak_sanghvi_-_may_1_2015_1212_am_-_
research_paper_final_team5.pdf ▶ Healthcare Robots with Islamic Practices
1020 Islamic Calendar

recent public report (Hayes 2019) hosting over

Islamic Calendar 200,000 games, with rough estimates that the
platform likely hosts 250,000–300,000 games at
▶ Healthcare Robots with Islamic Practices the time of writing (October, 2020), based on
community estimates on daily uploads. The plat-
form gears itself toward new developers by an
especially low barrier to enter publishing games,
Islamic Prayers providing accessible access too and hosting game
jams, providing development tools, and regularly
▶ Healthcare Robots with Islamic Practices publishing information on blogs that are geared
toward starting indie developers.

Itch.io, History of History

Mason Bates1 and Sercan Şengün2,3 Prior to its creation, the original concept for Itch.io
1
Creative Technologies Program, Illinois State was to create an online interface for making a
University, Normal, IL, USA customizable video game page where indie devel-
2
Wonsook Kim School of Art, Illinois State opers could quickly post their content online for
University, Normal, IL, USA the world to see. However, the service expanded
3
Massachusetts Institute of Technology CSAIL, both prior to, and in the early years of the website.
Cambridge, MA, USA These services continued to expand as the com-
munity of the site gained prominence within the
indie development community.
Synonyms By 2015, Itch.io has grown to a more dedicated
and well-established indie development platform,
Publishing; Web Games with support for both participating in and hosting
game jams, dedicated servers to dealing with
issues, communities following developers, or
Definitions indie groups on the site; at the time, they hosted
over 15,000 games and applications, according to
Indie game: short for independent game, a game their released site statistics. Similarly, in early
typically created by a small team of developers 2016, the official release of the Itch App created
without the financial or labor backing of major a desktop application interface for the previously
production/publication studios. solely web-based site.
Indie developer: short for independent devel- From 2016 onward, a great effort to expand,
oper studio, a team of game developers working polish, and enhance existing services and plat-
without assistance from major production/publi- forms can be seen, much of which can be seen
cation studios. by the enhancing of their desktop application and
expansion of their community on their site,
including additions in customization options for
Introduction game pages and analytics for developer postings.

Itch.io is a web-based video game publishing

platform built specifically for early indie/learning User Interaction
developers. Originally released in 2013 by Leaf
Corcoran, the platform has become rapidly more Users of Itch.io largely approach the platform
popular among a niche community of developers from at least one of the following purposes:
within the wider gaming industry, as of the most The user is...
Itch.io, History of 1021

1. A developer looking to publish games other marketing materials as a separate posting

2. A developer looking for game from the finalized product to avoid giving away
development tools their game for free.
3. A developer looking to participate in
game jams
4. A gamer looking to play indie games Publishing Games on Itch.io
5. A company/group looking to host/sponsor a
game jam As Itch.io’s original conceptualization was based
around the concept of game publishers, one can
Most of the traffic of the site is geared toward still see this reflected in the site’s available publi-
the publication and purchasing/downloading of cation and developer features. Posting a game on
games, points 1 and 4, respectively. Though the Itch.io is free, with no limitations on the quantity
game jam services are also an especially notable of products you can post or the quality of posting
portion of the site (Vu and Bezemer 2020), and options. Customization for game and developer
according to released Itch.io statistics, they pages is built into the site, and a developer can,
attract a large portion of new users to the site. without opening any other application or knowing
This has been especially true recently, with the how to code, do the following:
I
top 5 game jams (2 from 2020, 2 from 2019, and
1 from 2018), accounting for a cumulative total • Post on forums
of over 45,000 individuals/teams registering for • Change the page stylizations
the events and over 10,500 games added to • Add or edit content of the game/
the site. developer page
• Add dev logs
• Change settings or pay models of the page
Playing Games on Itch.io • View game analytics
• Get embeds for websites or social media
Many games on Itch.io are entirely free to play or • Have a handful of other useful features
follow the “pay what you want” model of sales,
though the platform does allow for developers to The number of available free features, and
have a set costs for games. Additionally, with the general pro-indie/new developer attitude of the
added ability to play HTML5 games in their community, has led to itch.io gaining many of its
respective web browser, users do not even neces- developers. Additionally, Itch.io regularly posts
sarily have to log in to play some of the games on on forums and developer communities with
the platform. However, for access to the vast indie developer strategies for designing, devel-
majority of the library, a user will need to sign oping, publishing, and marketing new indie
up for a free account and download a game from a games.
given page.
A majority of games on the platform seem to be
set to the “pay what you want” model, which a Itch.io as a Political Institution
small portion of users regularly donating money
to the developers of games they enjoy, and users Itch.io, being an American company, has both
are also given the ability follow and interact with officially and unofficially taken a number of polit-
developers on forums on the game’s respective ical stances regarding political issues in the USA.
page or within the sites’ social forum directly. The most notable of these events is the support the
Once a user has downloaded a game on itch.io, platform has given for the NAACP and the Black
they permanently have access to that game and Lives Matters movement, where the platform
any new editions that come out of it, though it is raised over eight Million dollars in June of 2020
not uncommon for developers of paid game pro- in response to the George Floyd Protests through-
jects to post a game’s alpha, beta, demo, or any out the USA (Statt 2020). This money was raised
1022 ITD

by creating a purchasable indie game bundle of References

games created by African American artists or that
strongly featured socio political discussions of Hayes, S.: There are 200,000 games on itch.io. Here’s how
to find your new favorite Itch.io, Retrieved from https://
race and racial intolerance. This stance generated
itch.io/blog/108659/there-are-200000-games-on-
a small amount of political backlash from parts of itchio-heres-how-to-find-your-new-favorite
the Itch.io community but was largely praised (November 7, 2019)
both within and without the Itch.io community Maiberg, E.: Itch.io is the littlest next big thing in gaming
Vice, Retrieved from https://www.vice.com/en/article/
and drew many artists to the site for the first time
vvbkb8/itchio-is-the-littlest-next-big-thing-in-gaming
(Squirrelrampage 2020). (June 23, 2015)
Squirrelrampage: Itch.io has a Black Lives Matter bundle
featuring 742 games for $5! [Discussion Post]. Reddit.
https://www.reddit.com/r/GamerGhazi/comments/
Itch.io and Steam gxz0mi/itchio_has_a_black_lives_matter_bundle_fea
turing/ (June 6, 2020)
The expansion of the video game market, espe- Statt, N.: Itch.io’s amazing 1,500-game charity bundle
cially within the past decade after the economic surpasses $5 million goal. The Verge, Retrieved from
https://www.theverge.com/2020/6/11/21287909/itch-
recovery of the 2008 global recession, has seen a
io-bundle-for-racial-justice-equality-five-million-
dramatic rise in digital publication platforms as dollar-goal-hit (June 11, 2020)
one of, if not the most common, the ways for Vu, Q.N., Bezemer, C.-P.: An empirical study of the char-
purchasing and obtaining video games, for both acteristics of popular game jams and their high-ranking
submissions on itch.io. In: Proceedings of the Interna-
PC and console games. Its most notable compet- tional Conference on the Foundations of Digital
itor, and the largest market share for publication Games, pp. 1-11. (2020)
and distribution of digital games then, both indie
and otherwise, is of course Steam, the publication
platform owned by Valve Corporation.
Steam also hosts a very large indie community
and has significantly greater universal brand rec- ITD
ognition. However, Steam has a notably higher
bar for the posting and sale of games, with devel- ▶ User Acoustics with Head-Related Transfer
opers needing to be formally recognized and Functions
signing legal paperwork before joining and pay-
ing $100 as a starting fee and then for each addi-
tional game they look to post on the platform. This
has caused many developers to view itch as some- Iterative Design
what of a “starter platform” where they first go to
develop their skills, before producing games that ▶ User-Centered Design and Evaluation Method-
are profitable enough to then warrant the posting ology for Virtual Environments
on Steam (Maiberg 2015).

Cross-References Iterative Game Design

▶ Indie Game ▶ Analog Prototyping for Digital Game Design

J

Japanese Chess Justification

▶ Contemporary Computer Shogi ▶ 2-Simplex Prism as a Cognitive Graphics Tool

for Decision-Making

© Springer Nature Switzerland AG 2024

N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
K

K-12 Education Definitions

▶ Augmented Learning Experience for School Augmented reality (AR) augments real world
Education with virtualized contents (i.e., objects and/or
supporting information) which appears to coexist
in the same space as the real world (Palmarini
et al. 2018). Its predecessor, i.e., virtual reality
(VR), on the contrary, generates a completely
Kaizo
artificial environment of the reality.
▶ Underground Design of Kaizo Games
Key Early Verticals for Augmented
Reality

Key Early Verticals: Although the promising notions of AR and VR

Challenges and Limitations in were coined several decades ago, the technologies
Implementation of enabling AR have just recently converged to a
Augmented Reality critical point enabling people to enjoy its experi-
ences and to fully reap its benefits (Yuan 2017).
Adnan Mahmood1,2, Bernard Butler1, Hushairi AR is believed to be one of the key technology
Zen2 and Brendan Jennings1 enablers for Industry 4.0 and is anticipated to
1
Emerging Networks Laboratory, disruptively change our world in many aspects.
Telecommunications Software & Systems Group, This section briefly depicts the key early
Waterford Institute of Technology, Waterford, verticals of AR in numerous industrial sectors,
Ireland i.e., manufacturing, healthcare, logistics, design
2
Communications Research Group, Faculty of and architecture, military, and data centers
Engineering, Universiti Malaysia Sarawak, (Syberfeldt and Gustavsso 2017; Chandler 2017)
Sarawak, Malaysia along with their salient characteristics.
Manufacturing: Connected devices and wear-
able products have rapidly penetrated across the
Synonyms manufacturing industry, hence opening new doors
for innovative AR experiences. AR is currently
Augmented reality; Mixed reality; Virtual reality making rapid strides in numerous areas. It is used
© Springer Nature Switzerland AG 2024
N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
1026 Key Early Verticals: Challenges and Limitations in Implementation of Augmented Reality

for the monitoring and solving of pain points by visualizing rearrangements by incorporating
experienced on the shop floors, i.e., pain points digital representations of envisaged future settings
along with key performance indicators (KPIs) in the current warehouse environment, and in the
could be directly projected to the engineers and freight transportation as loaders could have access
maintenance teams in order to analyze and resolve to the real-time digital data about the next pallet to
the issues in real time. This is also efficient in case be loaded and its placement in the vehicle along
of production downtime due to a broken part of a with the pertinent loading instructions thus saving
machine, as teams equipped with AR capabilities the tedious process of paper-based cargo lists and
can quickly respond to the hardware problems in speeding up the freight loading process (Stoltz
almost no time. AR could further assist the pro- et al. 2017; Glockner et al. 2014).
duction teams in the physical asset-based logistics Design and Architecture: Over the past few
and status of the physical stock on the shop floor, decades, one of the key challenges confronting
and the same can be superimposed in the form of designers was to dive deep into physical space of
digital content to augment the real-time views a structure or an object that they are conceiving.
(Caricato et al. 2014; Uva et al. 2018; Chang Traditionally, 3D objects were conceived over the
et al. 2017). 2D screens. However, as of late, more meaningful
Healthcare: Healthcare is one of the most and lucrative ways have transpired and AR expe-
dominant sectors impacted by AR in numerous riences undoubtedly lies at heart of the same, i.e.,
ways, i.e., from training medical students about from the powerful 3D printing facilitating the
the human anatomy to counseling mothers strug- companies and firms to rapidly transform their
gling with breastfeeding by effectively allowing concepts into implementation thus ultimately
the counselors to see through the eyes of mothers leading to reduction in costs and securing of
via an AR wearable device, to assisting the more clients to a collaborative design process for
patients to accurately describe their past and sourcing of innovative ideas, variants, and its
existing medical conditions to their doctors, to feedbacks from the geographically distributed
enabling nurses to locate human veins conve- consumers during the product’s planning stage,
niently during intravenous injections, to facilitat- to equipping the architectural project teams and
ing the curious consumers of the pharmaceutical their clients to immerse in an interactive AR expe-
industry with the 3D views of drug actions and rience for monitoring progress of ongoing pro-
effects in the human body, to practicing minimally jects via a real-time digital modeling of a
invasive surgeries by enabling the surgeons to construction site, thus avoiding the tedious task
see through the patient (without the need for of walking clients on the construction sites and
opening them up) during the surgical planning preventing any unwanted accidents, to the spatial
and image-guided surgery, etc. (Herron 2016; augmented reality revolutionizing the automotive
Chen et al. 2017). industry by enabling the designers to assess
Logistics: One of the biggest waves of change curves and geometries more efficaciously by pro-
anticipated in logistics industry is in the form of jecting virtual data on a real vehicle model during
AR technology, i.e., in the warehouse operations, its development process (i.e., typical virtual data
wherein, notion of pick-by-vision for providing a is often being displayed on monitors and its size is
hands-free digital approach could be employed often scaled down and is not a precise reflection of
instead of a slow and error-prone pick-by-paper the reality), etc. (Chi et al. 2013; Elia et al. 2016;
approach in order to optimize picking process (the Behzadan et al. 2015).
software employed for pick-by-vision could have Military: AR has been making its stronghold
features like the barcode reading, indoor naviga- in the battlefield, i.e., from projecting precise
tion, real-time object recognition, seamless inte- maps, navigation way points, friends or foe dis-
gration with centralized warehouse management crimination, and pertinent information to a sol-
systems, etc.), in the warehouse planning to dier’s field of vision, to integrating specialized
accommodate a number of value-added services AR gadgets to a weapon control system for
Key Early Verticals: Challenges and Limitations in Implementation of Augmented Reality 1027

enhancing the mission’s effectiveness, to train- objects and people surrounding him/her, and
ing of the combat personnel for complex condi- more interestingly, what actions are being carried
tions arising in the battlefield through injecting out through their eyes, hands, and voice and all in
of virtual threats into a realistic environment for few milliseconds so as to ensure that the precise
ensuring that the troops are skilled enough to contextual information gets layered on the actual
quickly respond and operate the equipment in environment via an AR device. Hence, the
every possible scenario, to training fighter pilots design of a state-of-the-art real sense cameras
for diverse battle scenarios and certain special- (possessing the full five-finger integration),
ized cockpit operations, i.e., aerial refueling and diverse sensors, and microphones for an AR
missile deployment, as most of their trainings are device still poses a daunting challenge yet to be
conducted over flight simulators and setting up fully realized.
live combat operations could be very expensive Seamless Optical Displays: A seamless optical
(You et al. 2018; Karlsson 2016). display blending (both) physical and digital world
Data Centers: AR is anticipated to bring tre- adds another layer of complexity in the early
mendous benefits to data center planning and to a realization of AR. It is pertinent to note that VR
wide range of its operations and processes, i.e., is already utilizing high-resolution innovative
from remote management of the data centers to screens, in whose production, the smartphone
spatial tours via the 3D real-time imaging super- industry has been actively involved over the past
imposed with pertinent contextual information so few decades. In case of VR, the user typically
as to have a better understanding of local pre- glances on the screen; however, in AR, it is essen-
vailing circumstances, to the navigational guid- tial to look through the screen so as to still expe- K
ance in a data center for better identification of rience the real-world environment.
error-prone devices or installation of a new device Computing Power: Power is one of the serious
along with installation instructions, to the color challenges currently being faced by the AR indus-
labeling of identical-looking racks and cabinets try. Today, with the continuous evolution of stron-
within a datacenter to reflect status messages ger yet ever smallest processors, there would
(i.e., notifications, alarms, or warnings) or opera- certainly be (in the near future) powerful enough
tional analyses along with troubleshooting processors specifically for AR. Thus, powerful
instructions if any, to the identification of a device processors require powerful batteries and espe-
and device-related specific virtual information cially with characteristics, i.e., low consumption,
and real-time datasets by employing the QR scan- high capacity, and small enough to be compatible
ner and many more (Deffeyes 2011; Emeis et al. with lightweight AR wearable displays. Complex-
2017). ity versus implementation is an issue and balance
is still to be determined.
Scaling: Interpreting (or rendering) of the dig-
Challenges and Limitations in ital data into meaningful graphics and subse-
Implementation of Augmented Reality quently scaling it down to suit the perspective of
individual’s visual field adds a significant chal-
Despite a number of potential applications of AR lenge in the vast implementation of AR.
in modern-day industries, there are still several AR Software: One of the indispensable com-
challenges and limitations that hinders its true ponents of AR software is its competence to effi-
realization (Akayra and Akayra 2017; Zhang ciently accumulate, process, and analyze
et al. 2017). Some of these challenges and limita- potentially diverse range of inputs simultaneously
tions are discussed as follows. and transform them accordingly based on new
Low Latency Monitoring and Tracking: In digital information so as to provide the high-end
order to have an essential AR experience, it is AR experiences. For realizing the same, high-
indispensable to precisely track and subse- speed connections would thus be required to tie
quently monitor an individual’s location, the the AR software to the back-end services, thus
1028 Key Early Verticals: Challenges and Limitations in Implementation of Augmented Reality

ensuring the future AR-enabled devices to be the References

world’s most capable Internet of Things (IoT)
endpoints. Akayra, M., Akayra, G.: Advantages and challenges asso-
ciated with augmented reality for education: a system-
atic review of the literature. Educ. Res. Rev. 20, 1–11
(2017). https://doi.org/10.1016/j.edurev.2016.11.002
Conclusion Behzadan, A.H., Dong, S., Kamat, V.R.: Augmented real-
ity visualization: a review of civil infrastructure sys-
tem applications. Adv. Eng. Inform. 29(2), 252–267
The notion of AR has grown by leaps and bounds
(2015)
over the past few decades, but very few of us Caricato, P., Colizzi, L., Gnoni, M.G., Grieco, A.,
know how far it has come in reality. At the time Guerrieri, A., Lanzilotto, A.: Augmented reality appli-
of writing of this article (2018), various tech cations in manufacturing: a multi-criteria decision
model for performance analysis. IFAC Proc. 47(3),
giants such as Google, Apple, Facebook, 754–759 (2014)
etc., have launched their AR development kits Chandler, D.L.: Realizing a clearer view: new augmented
and a new wave in technology has finally set reality systems provide medical students with a Sur-
in. Also, a number of industries have started geon’s sight. IEEE Pulse. 8(5), 36–41 (2017). https://
doi.org/10.1109/MPUL.2017.2729738
embracing AR and specific applications are now
Chang, M.M.L., Ong, S.K., Nee, A.Y.C.: AR-guided prod-
being experienced in their initial forms. But how uct disassembly for maintenance and remanufacturing.
big would the AR impact be? In order to address Procedia CIRP. 61, 299–304 (2017)
this question, this article serves as a major refer- Chen, L., Day, T. W., Tang, W., and John, N. W.: Recent
developments and future challenges in medical mixed
ence for not only highlighting some of the key reality. In: 2017 I.E. international symposium on mixed
early verticals for AR in various industrial sectors and augmented reality (ISMAR), Nantes, 123–135
but also deliberates on some of the critical chal- (2017)
lenges and limitations hindering its true realiza- Chi, H.-L., Kang, S.-C., Wang, X.: Research trends and
opportunities of augmented reality applications in
tion. It is beyond any doubt that AR is soon going architecture, engineering, and construction. Autom.
to be a mainstream product and either people Constr. 33, 116–122 (2013)
would be wearing AR headsets for most of their Deffeyes, S.: Mobile augmented reality in the data center.
time or AR would replace everything we do on IBM J. Res. Dev. 55(5), 5:1–5:5 (2011)
Elia, V., Gnoni, M.G., Lanzilotto, A.: Evaluating the appli-
smartphones, act as a new interface for laptops or
cation of augmented reality devices in manufacturing
desktops along with numerous specialized appli- from a process point of view: an AHP based model.
cations across all industrial sectors. This ulti- Expert Syst. Appl. Int. J. 63, 187–197 (2016)
mately would lead to another revolutionary wave Emeis, M.W., Hendrich, R.C., Vosburgh, C.A.: Using
Augmented Reality to Assist Data Center Operators.
in the gaming industry too. United States Patent Application Publication No. US
2017/0091607 A1 (2017)
Glockner, H., Jannek, K., Mahn, J., Theis, B.: Augmented
Cross-References reality in logistics (Changing the Way We See Logis-
tics – a DHL Perspective). DHL Customer Solutions
and Innovation, Troisdorf (2014)
▶ History of Virtual Reality Herron, J.: Augmented reality in medical education and
▶ Mixed Reality training. J. Electron. Resour. Med. Libr. 13(2), 51–55
▶ Potential of Augmented Reality for Intelligent (2016)
Karlsson, M.: Challenges of designing augmented reality
Transportation Systems for military use. Master’s thesis, Ume University, Swe-
den (2016)
Acknowledgments This publication has received support Palmarini, R., Erkoyuncu, J.A., Roy, R., Torabmostaedi,
from the Science Foundation Ireland’s CONNECT pro- H.: A systematic review of augmented reality applica-
gramme, and is co-funded under European Regional tions in maintenance. Robot. Comput. Integr. Manuf.
Development Fund’s Grant Number 13/RC/2077. 49, 215–228 (2018). https://doi.org/10.1016/j.rcim.
The corresponding author would also like to acknowledge 2017.06.002
the generous support of the Ministry of Higher Education, Stoltz, M.-H., Giannikas, V., McFarlane, D., Strachan, J.,
Government of Malaysia for supporting part of the said Um, J., Srinivasan, R.: Augmented reality in warehouse
research work through its Malaysian International Schol- operations: opportunities and barriers. IFAC-
arship Grant, KPT. 600-4/1/12 JLID 2(8). PapersOnLine. 50(1), 12979–12984 (2017)
King of Fighters, a Brief History 1029

Syberfeldt, A., Gustavsso, P.: Augmented reality smart 1994, leading up to the advent of the greatly
glasses in the smart factory: product evaluation guide- awaited King of Fighters XV – the 15th edition
lines and review of available products. IEEE Access. 5,
9118–9130 (2017). https://doi.org/10.1109/ACCESS. in 2021, this entry goes over the history of the
2017.2703952 series, SNK, spinoffs, and other media, as well as
Uva, A.E., Gattullo, M., Manghisi, V.M., Spagnulo, D., the e-Sports scene.
Cascella, G.L., Fiorentino, M.: Evaluating the effec-
tiveness of spatial augmented reality in smart
manufacturing: a solution for manual working stations.
Int. J. Adv. Manuf. Technol. 94(1–4), 509–521 (2018) The Games and Their Unique Systems
Yuan, Y.: Changing the world with virtual/augmented real-
ity technologies. IEEE Consum. Electron. Mag. 6(1), Starting off, the main set of KoF games began
40–41 (2017). https://doi.org/10.1109/MCE.2016.
2614411 with the King of Fighters ‘94 for the Neo Geo,
You, X., Zhang, W., Ma, M., Deng, C., Yang, J.: Survey on which was followed with yearly releases up until
urban warfare augmented reality. ISPRS Int. J. Geo-Inf. 2003, where SNK started taking more time to
7(2), 46, 1–16 (2018) make the new editions of the game series. KoF
Zhang, W., Han, B., Hui, P.: On the networking challenges
of mobile augmented reality. In: Workshop on virtual XI released in 2005, KoF XII alongside 2002
reality and augmented reality network, New York, Unlimited Match in 2009, KoF XIII in 2010, and
24–29 (2017). https://doi.org/10.1145/3097895. KoF XIV in 2016 (Mendoza 2020).
3097900 The King of Fighters series differentiates itself
from its peers with its 3-on-3 team battle setup.
Traditionally, fighting games ask players to select
a single character to use against their opponent, K
King of Fighters, a Brief but in the KoF series players select a team of three
History characters to challenge their opponent’s three
characters. After one of the characters is downed,
Brody Corenflos2, Sam Romershausen2 and the next character in line will tag in. Characters
Newton Lee1,2 standing in the backlines can sometimes perform
1
Institute for Education, Research, and support attacks to help.
Scholarships, Los Angeles, CA, USA Along with previously discussed mechanics,
2
Vincennes University, Vincennes, IN, USA KoF employs its own set of mechanics, such as
the many jump options (short hop, short jump,
hyper hop, hyper jump, etc.) as well as the MAX
Synonyms Mode which could open up Death Combos in
certain iterations. King of Fighters has no pre-
Fighting game; Interactive game; Video games determined combos, and players have to make
up their own using basic, unique, and special
moves.
Definition

SNK: A Japanese video game hardware and soft- Spinoffs and Other Media
ware company.
KoF: The King of Fighters is a series of fight- The King of Fighters have inspired spin-off titles
ing games by SNK. such as the Maximum Impact series, which offered
KoF in a 3D environment in 2004 (Maximum
Impact), 2006 (Maximum Impact 2), and 2007
Introduction (Maximum Impact Regulation-A). Other spin-off
titles include The King of Fighters ‘94 Re-Bout
The King of Fighters (KoF) is a series that started (2004), The King of Fighters Neowave (2004),
in arcades on SNK’s own game system. Starting in The King of Fighters XI (2005), The King of
1030 Kingdom Hearts (2002): An Analysis

Fighters ‘98 Ultimate Match (2008), and The King 2013 involved a 2-0 comeback followed by a
of Fighters 2002 Unlimited Match (2009). reverse 3-0. However, the lack of documented
There are also a ton of manga for the main e-sports tournaments for The King of Fighters
series from 1994 to 2000, and more recently, one makes it difficult to expound on the KoF
for XIV. For example, The King of Fighters: e-sports scene.
A New Beginning is a Japanese shōnen manga
authored by Kyōtarō Azuma as an adaptation of
SNK’s 2016 fighting game The King of Fighters Conclusion
XIV. There are also quite a few drama CDs as well
as some animations. One of those animations is a The King of Fighters is a fighting game series that
complete retelling of KoF ‘94, called The King of has existed for 27 years as of 2021, and that has
Fighters: Destiny. There was also a movie adap- survived through the bankruptcy of its game pub-
tation The King of Fighters in 2010 starring Sean lisher SNK. Over the years, the game has run on
Faris as Kyo Kusanagi, Maggie Q as Mai many platforms including Neo Geo arcade,
Shiranui, Will Yun Lee as Iori Yagami, and Ray Atomiswave arcade, Taito Type X arcade, Sega
Park as Rugal Bernstein. However, it was none Saturn, Dreamcast, PlayStation, Game Boy, Wii,
too faithful to the source material and was panned Nintendo Switch, iOS, Android, Microsoft Win-
by critics and audiences alike (The King of dows, and Xbox. King of Fighters differentiates
Fighters). itself using its unique 3-on-3 combat and many
ways to execute attacks to such a degree that it has
received widespread appeal.
SNK Company Turmoil

After the release of KoF 2000, SNK filed for References

bankruptcy in 2001 and auctioned the intellectual
property rights for its franchises. This led to KoF Mendoza, M.: A brief history on The King of Fighters. Too
much Gaming | Video games reviews, news, & guides.
2001 and the original 2002 being created by a
www.toomuchgaming.net/blog-news/a-brief-history-
different company, called Eolith. SNK’s founder on-the-king-of-fighters (10 June 2020)
and other executives founded a new company Rotten Tomatoes: The King of Fighters reviews. Retrieved
under the name Playmore later in August of 16 May 2023 from https://www.rottentomatoes.com/
m/king_of_fighters/reviews (n.d.)
2001. In late October of the same year, Playmore
had successfully reacquired the SNK intellectual
property rights and began rehiring former
employees.
Kingdom Hearts (2002): An
Analysis
Esports Scene
Michael Phillips2 and Newton Lee1,2
1
Several editions of The King of Fighters Institute for Education, Research, and
appeared in e-sports competitions. Some tour- Scholarships, Los Angeles, CA, USA
naments have used KoF ‘95 through ‘98 Ulti- 2
Vincennes University, Vincennes, IN, USA
mate Match (some as recent as 2020). Before
the release of KoF XIV, the e-sports tourna-
ments mainly featured 2002 Ultimate Match Synonyms
and KoF XIII. A particularly outstanding finale
for one of the KoF XIII tournaments at EVO Action role-playing game; Hack-n-slash
Kingdom Hearts (2002): An Analysis 1031

Definitions Square Enix characters, they wanted to attract

both hardcore and casual players. They were
Action It is a specific type of role-playing able to create a game that could bring together
role- game where the player controls a people from all ages, young or old, to enjoy
playing character engaged in real-time combat. this series of games. Kingdom Hearts managed
game to secure the spot for the 10th best-selling
Hack- A type of combat that focuses on PlayStation 2 game of all time as well (Game
n-slash quick, hand-to-hand or melee weapon Enthusiast).
combat. Kingdom Hearts took much influence from
Square’s Final Fantasy franchise, and it consists
Kingdom Hearts (2002) of a hack-n-slash approach to its gameplay. The
main objective was to travel around defeating
Kingdom Hearts is a single-player, action role- bosses to advance to different worlds. The player
playing game that is a crossover between two controls Sora with the ability to use basic
companies and their universes. These two compa- attacks, special attacks, magic, and in some
nies consisted of Square Enix (originally known cases, dodges. Meanwhile, Donald, Goofy,
as just Square) and Disney Interactive. Square and/or another character who can replace the
Enix and Disney Interactive Studios were the two will assist Sora within battles. The player
major developers and publishers, but the series can also control the behavior of the assisting
itself has also had help from other companies to characters to a certain extent. Sora’s stats are
compile the game together. Kingdom Hearts based around his level, and the game in its K
resides within its own series (Kingdom Hearts) entirety is set around an experience point sys-
and spans over nine main games. Kingdom Hearts tem. Defeating enemies is the only way to earn
was first created on its own in-house engine, experience points which in turn increases Sora’s
Luminous Engine, but with the release of level (capping at 100). Kingdom Hearts is rela-
Kingdom Hearts 3, the game will be switching tively linear; the player moves from world to
over to Unreal Engine 4. Kingdom Hearts was a world defeating Heartless, bosses, etc., but the
PlayStation 2 exclusive, released on March player also has the chance to run into short side-
28, 2002. The player takes on the role of the quests to secure items, experience, etc. Among
main protagonist, Sora. Kingdom Hearts is rated all these systems set in place, the combat inter-
as E, but as the series progressed, the ESRB rating face seems to be the simplest. There is a com-
has change to E 10+ (Entertainment Software mand bar consisting of Attack, Magic, Items,
Ratings Board). and Specials. Lastly, there is a health bar and
Kingdom Hearts follows the story of a boy, magic bar, as well as both of the assisting char-
Sora, who sets off on an adventure after his acter’s health and magic bars.
home, Destiny Islands, is consumed by Darkness. Kingdom Hearts was a one of kind collabora-
Throughout the series of the games the player gets tion, which was perceived rather impactfully in
to watch Sora grow as a hero. In the original game the public’s eyes. Due to its massive success,
he arrives in Traverse Town and meets a new set Kingdom Hearts never had any controversies
of friends, Donald and Goofy, and then sets off to surrounding the series. At first, no one thought
find Riku, Kairi, and King Mickey. His relation- Disney and Final Fantasy would be a good com-
ship with every friend he meets enables his growth bination but were proven wrong. The original
to flourish throughout the story. The trio meets characters, especially Sora, were unique because
many heroes and villains spanning across both the player could witness how their personalities
the Disney and Final Fantasy universes. Since grew and altered throughout the stories of each of
Kingdom Hearts consists of both Disney and the nine games. Kingdom Hearts made the 21st
1032 Knowledge

spot on IGN’s top 25 PS2 games of all time. IGN Cross-References

had this to say, “The premise behind the game was
completely absurd: Disney cartoon characters ▶ Video Games
like Mickey Mouse thrown in with Square Enix
characters to make a new kind of action RPG.
However, as strange as it sounds, these two dis- References
similar franchises came together incredibly well”
(IGN). The characters – original, Disney, and Entertainment Software Ratings Board. https://www.esrb.
org/ratings/35957/Kingdom+Hearts+3/
Final Fantasy – helped develop the game into
Game Enthusiast. https://www.game-enthusiast.com/
such a hit as it is still today. Kingdom Hearts 2020/03/04/playstation-2-turns-20-today-heres-the-
suffers from linear progression but not in an top-ten-best-selling-ps2-games/
unplayable manner. Kingdom Hearts has not IGN. https://www.ign.com/articles/best-ps2-games
really changed up its graphical capabilities over
the years, but here is a YouTube video showcasing
the evolution of the series: https://www.youtube.
com/watch?v¼Wu-QQgWeCi4.
Knowledge
As stated earlier, Kingdom Hearts managed
▶ World Representation in Artificial Intelligence
to secure the spot for the 10th best-selling
PlayStation 2 game of all time as well (Game
Enthusiast). Kingdom Hearts had such a unique
way of going about its gameplay, but it could be Knowledge Representation
comparable to games like the “Tales of” series, the and Reasoning
“Persona” series, and of course the “Final Fan-
tasy” series. ▶ World Representation in Artificial Intelligence
L

Lattice Boltzmann Method for Introduction

Diffusion-Reaction Problems
The lattice Boltzmann method (LBM) is based on
Sicilia Ferreira Judice the fundamental idea of constructing simplified
Faculty of Technical Education State of Rio de kinetic models that incorporate the essential
Janeiro, FAETERJ Petropolis, Petropolis, Brazil physics of the microscopic processes, so that the
estimated properties satisfy the macroscopic
equations. LBM is especially useful for modeling
complex boundary conditions and multiphase
Synonyms
interfaces (Chen and Doolen 1998). Extensions
of this method are described in the literature,
Cellular automata; Diffusion-reaction problems;
including simulations of turbulent fluids,
LBM; Numerical methods
suspended fluids, and diffusion-reaction systems
(Wei et al. 2004).
Lattice-based models have some advantages
Definitions compared to the traditional numerical methods
(Rothman and Zaleski 1994). The simulation
• Cellular automata occurs in a regular lattice and can be efficiently
A mathematical model based on simple and implemented in a massively parallel computer.
local rules capable of generating complex Solid boundaries and multiple fluids can be intro-
behaviors. duced in a simple way, and the simulation is done
• LBM efficiently, regardless of the complexity of bound-
Lattice Boltzmann method, a numerical ary or the interface (Buick et al. 1998). In the case
method based on kinetic equations formulated of lattice gas cellular automata (LGCA), there are
on a mesoscopic scale. no numerical stability problems because their evo-
• LGCA lution follows integer arithmetic. For LBM, accu-
Lattice gas cellular automata, a specific cel- racy and numerical stability depend on the Mach
lular automaton, whose proposal is to simulate number, which is the ratio of maximum speed to
fluids using simple and local rules that imitate a the speed of sound. The computational cost of the
particle dynamics. LGCAs is lower than the cost of LBM methods.

© Springer Nature Switzerland AG 2024

N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
1034 Lattice Boltzmann Method for Diffusion-Reaction Problems

However, the system parameterization (e.g., vis- spatial term, giving us the following approxima-
cosity) is difficult to do in LGCA models, and tion of the diffusion equation in a given node i:
the dynamics obtained is less realistic than for
the LBM. T nþ1  T ni T n  2T ni þ T ni1
i
¼ a iþ1 , ð2Þ
This article will demonstrate how to use the Dt Dx2
advantages of lattice Boltzmann method to solve
diffusion-reaction problems. where T ni corresponds to the amount that is being
diffused at node i at time step n, Δt is the discrete
time step, and Δx is the discrete spatial step.
Theoretical Foundations Isolating the term of interest from Eq. (2), we
have
Diffusion is an example of a matter transport phe-
nomenon where the particles of a set move ran- aD t n
T nþ1 ¼ T ni þ T  2T ni þ T ni1 : ð3Þ
domly and spread in the medium containing them. i
Dx2 iþ1
For example, we can smell perfume because it
diffuses into the air and makes its way into Equation (3) can be rearranged as follows:
our nose. From macroscopic point of view, these
movements cause the set to move from the higher- 2aDt 2aDt
T nþ1 ¼ T ni 1  þ
concentration zones to the low-concentration
i
Dx2 Dx2
zones. Diffusion-reaction equations arise naturally T niþ1 þ T ni1
 : ð4Þ
in systems consisting of many interacting compo- 2
nents, like chemical reactions, and are widely used
to describe pattern formation phenomena in a vari- Defining
ety of biological, chemical, and physical systems.
The diffusion equation is given by 2aD t
t¼ ,
Dx2
@f @2f
¼ a 2 þ F, ð1Þ we have then
@t @x

where α(x) >0 is the diffusion coefficient in x, T nþ1

i ¼ T ni ð1  tÞ þ t 0:5T niþ1 þ 0:5T ni1 : ð5Þ
f(x,t) is the amount that is being diffused, and
F(f,x,t) is a portion of internal contribution The finite difference approximation steps were
(source term). The portion on the left is time performed in a way that could be compared to the
variation, and the first portion on the right is LBM methodology. Following the considerations
spatial variation. of Mohamad (2011), the last term in Eq. (5) is an
average of the quantity around Ti or, in other
words, represents an equilibrium term of Ti. In
Finite Difference Approximation this way, Eq. (5) can be rewritten as follows:

The following analysis was performed by T nþ1

i ¼ T ni ð1  tÞ þ tT eq
i , ð6Þ
Mohamad (Mohamad 2011), for a one-
dimensional problem; however, the extension for where T eq
i is the equilibrium term of Ti.
two-dimensional and three-dimensional problems
is analogous. Moreover, the analysis takes into Formulation of the LBM for Diffusion
account diffusion Eq. (1) without the source
term. A finite difference method will be used in The LBM is a numerical method whose equations
the temporal term, and central differences in the are formulated on a mesoscopic scale. This
Lattice Boltzmann Method for Diffusion-Reaction Problems 1035

method is able to simulate the dynamics of fluids relaxation term toward the local equilibrium
on a macroscopic scale (Chen and Doolen 1998). (Bhatnagar et al. 1954).
In the LBM, the fluid is represented by a set of The collision operator can be represented by
particles which reside in a regular lattice with the BGK approximation:
certain properties of symmetry. The dynamics
that governs the simulation involves steps of col- 1
Oi ¼  f ðx, tÞ  f eq
i ðx, tÞ , ð8Þ
lision and scattering of these particles through o i
the lattice directions, following simple rules that
satisfy the laws of conservation of mass (number where o represents a temporal relaxation on the
of particles) and momentum. The macroscopic equilibrium distribution function, related to the
behavior of the fluid is obtained through statistical diffusion coefficient on the macroscopic scale.
results on the data at the microscale. This section Equation (7) with the BGK approximation can
shows the formulation of the LBM for solving be discretized as follows:
diffusion problems.
The kinetic equation for the distribution func- f i ðx, t þ D tÞ  f i ðx, tÞ
tion fi(x,t) can be written as follows (Mohamad Dt
2011): f ðx þ Dx, t þ D tÞ  f i ðx, t þ D tÞ
þ ci i
Dx
@f i ðx, tÞ @f ðx, tÞ 1 eq
þ ci i ¼ Oi ð7Þ ¼  f i ðx, tÞ  f i ðx, tÞ : ð9Þ
@t @x o

where i varies among the possibilities of direction Replacing Δx ¼ ciΔt, we simplify to

for each cell, depending on the dimension of the L
problem. Among the available models, those that f i ðx þ Dx, t þ DtÞ  f i ðx, tÞ
represent the Navier-Stokes equations are called Dt
¼ f ðx, tÞ  f eq
i ðx, tÞ : ð10Þ
DnQb (where n refers to the dimension and b the o i
number of lattice directions), proposed by Quian
et al. (1992). For example, a 1D problem has one What leads us to
dimension and two lattice directions, so the model
is called D1Q2. However, for 2D problems, the f i ðx þ Dx, t þ DtÞ ¼ f i ðx, tÞ½1  t
most common model used is D2Q9, which takes þ tf eqi ðx, tÞ, ð11Þ
into account the diagonal directions.
The term on the left in Eq. (7) represents the where t ¼ Δt/o is called temporal relaxation.
scattering process, where the distribution func- The dependent variable f in reaction-diffusion
tions evolve along the lattice directions with Eq. (1) is related to the distribution function as
velocity ci ¼ Δx/Δt. The term on the right follows (for D2Q9 model):
represents the rate of change of the distribution
function in the collision process. According to 8

Chopard et al. (2002), the most natural way to fðx, tÞ ¼ f i ðx, tÞ: ð12Þ
i¼0
define the collision term is by averaging the
micro-dynamics and factoring it into a product The relationship between the diffusion coeffi-
of average quantities. However, for more sophis- cient α and the temporal relaxation term t can be
ticated fluids, the collision term requires a large deduced by a multi-scale expansion (Mohamad
number of floating point operations at each node 2011), providing
of the lattice at each instant of time, which would
increase the computational cost. One solution is
Dx2 1 1
to use the BGK approximation, which uses a a¼  , ð13Þ
DtD t 2
1036 Lattice Boltzmann Method for Diffusion-Reaction Problems

where D corresponds to the dimension of the term can be treated as a source term in the LBM
problem. formulation. Thus, the LBM formula for reaction-
For diffusion problems, assume constant the diffusion problems is given by (Mohamad 2011)
equilibrium distribution functions, not having
macroscopic velocity action: f i ðx þ Dx, t þ D tÞ ¼ f i ðx, tÞ½1  t
þ tf eqi ðx, tÞ
f eq
i ¼ Ai : ð14Þ þ D twi R, ð20Þ

The equilibrium distribution function must sat- where R is the reaction term. In the
isfy mass conservation and momentum: formulation, fi corresponds to the quantity being
diffused. In this case, we want to apply the diffu-
8 sion in a two-dimensional vector field:
f eq
i ¼ Y ð15Þ f i ðx, tÞ ¼ f ui ðx, tÞ, f vi ðx, tÞ .
i¼0
In practical terms, this implies a duplicate
and structure, that is, we will now have a lattice for
the component x and a lattice for the component y,
8 and apply the LBM separately as follows:
f eq
i ci ¼ 0: ð16Þ
i¼0 f ui ðx þ Dx, t þ DtÞ ¼ f ui ðx, tÞ½1  t
þ tf u,eq
i ðx, tÞ
In general, Ai ¼ wiΘ, which implies f eq
i ¼ wi Y. þ Dtwi Ru , ð21Þ
The equilibrium distribution function for the dif-
fusion problem can be chosen as follows f vi ðx þ Dx, t þ DtÞ ¼ f vi ðx, tÞ½1  t
(Mohamad 2011): þ tf v,eq
i ðx, tÞ
þ D twi Rv : ð22Þ
f eq
i ¼ wi fðx, tÞ, ð17Þ
The reaction term will be treated as the source
with f given by Eq. (12) and wi are weight factors term in the LBM (Mohamad 2011). In this way,
relative to each direction of movement. The fac- the dynamics in (20) is rewritten as follows:
tors wi shall meet the following criteria:
f i ðx þ Dx, t þ DtÞ ¼ f i ðx, tÞ½1  t
8
þ tf eqi ðx, tÞ þ DtRi , ð23Þ
wi ¼ 1: ð18Þ
i¼0
where:
In this way, the equilibrium distribution
wi R  ci
functions can be summed in all directions, Ri ¼ , ð24Þ
providing c2s
p
8 8 where cs ¼ 1= 3 is called speed of sound and wi
f eq
i ðx, tÞ ¼ wi fðx, tÞ ¼ fðx, tÞ: ð19Þ are weighting factors related to each of the i’s
i¼0 i¼0 directions.
The temporal relaxation parameter t is related to
LBM Diffusion-Reaction to Vector Field the diffusion coefficient through Eq. (13). Finally,
So far we have shown the general formulation of the equilibrium distribution will then be given by
the LBM for diffusion problems. In this section,
we will apply this formulation to the problem of
diffusion-reaction in vector fields. The reaction f u,eq
i ¼ wi uðx, tÞ, ð25Þ
Lattice Boltzmann Method for Fluid Simulation 1037

f v,eq
i ¼ wi vðx, tÞ, ð26Þ
Lattice Boltzmann Method for
where: Fluid Simulation

8 Sicilia Ferreira Judice

uðx, tÞ ¼ f ui ðx, tÞ, ð27Þ Faculty of Technical Education State of Rio de
i¼0 Janeiro, Petropolis, RJ, Brazil
8
vðx, tÞ ¼ f vi ðx, tÞ: ð28Þ
i¼0 Synonyms

Cross-References Cellular automata; Fluid simulation; LBM;

Numerical methods
▶ Fluid Simulation
▶ Cellular Automata Methods
▶ Lattice Gas Cellular Automata for Fluid Definitions
Simulation
▶ Lattice Boltzmann Method for Fluid • Cellular Automata
Simulation A mathematical model based on simple and
local rules capable of generating complex
behaviors.
References • LBM
Lattice Boltzmann method, a numerical L
Bhatnagar, P.L., Gross, E.P., Krook, M.: A model for method based in kinetic equations formulated
collision processes in gases: Small amplitude processes on a mesoscopic scale.
in charged and neutral one-component system. Phys.
Rev. 94, 511–525 (1954)
• LGCA
Buick, J., Easson, W.J., Greated, C.A.: Numerical simu- Lattice gas cellular automata, a specific cel-
lation of internal gravity waves using a lattice gas lular automaton, whose proposal is to simulate
model. Int. J. Numer. Methods Fluids. 26(6), fluids using simple and local rules that imitate a
657–676 (1998)
Chopard, B., Dupuis, A., Masselot, A., Luthi, P.: Cellular
particle dynamics.
automata and lattice Boltzmann techniques: An
approach to model and simulate complex systems.
Adv. Compl. Syst. 05, 103–246 (2002) Introduction
Chen, S., Doolen, G.D.: Lattice Boltzmann method for
fluid flows. Annu. Rev. Fluid Mech. 30, 329–364 The lattice Boltzmann method is a numerical
(1998)
method based in kinetic equations formulated on
Mohamad, A.A.: Lattice Boltzmann Method – Fundamen-
tals and Engineering Applications with Computer a mesoscopic scale, which simulates fluid dynam-
Codes. Springer, London (2011) ics on a macroscopic scale (Chen and Doolen
Quian, Y.H., d’Humires, D., Lallemand, P.: Lattice bgk 1998). In the last years, LBM has drawn the
models for navier-stokes equation. Europhys. Lett. 17,
attention of the scientific community due to its
479–484 (1992)
Rothman, D.H., Zaleski, S.: Lattice-gas models of ease of implementation and computational effi-
phase separation: Interface, phase transition and ciency. Specifically in fluid dynamics, LBM has
multiphase flows. Rev. Mod. Phys. 66, been used due to its ease of boundary conditions
1417–1479 (1994)
implementations (Chopard et al. 2002).
Wei, X., Li, W., Mueller, K., Kaufman, A.E.: The lattice-
Boltzmann method for simulating gaseous phenomena. The method originated from the lattice gas
IEEE Trans. Vis. Comput. Graph. 10(2), 164–176 cellular automata (LGCA) that, despite its advan-
(2004) tages, is limited by its discrete nature: the
1038 Lattice Boltzmann Method for Fluid Simulation

appearance of artifacts in the characteristic direc- ni ðx þ ci , t þ drÞ ¼ ni ðx, tÞ þ Di ðnðx, tÞÞ, ð1Þ
tions of the lattice and little flexibility to adjust
physical parameters and initial conditions where ni(x, t) can assume the values 0 or
(Adilson Vicente Xavier 2006; Chen and Doolen 1 representing the absence or presence of particle
1998). It was later demonstrated that the dynam- moving from the cell in the position x to the
ics of the LBM can be derived through the neighboring cell x + ci at time step t (ci are the
Boltzmann equation (He and Luo 1997a; Phi- directions of movement). The function Δi is a
lippi et al. 2006). collision operator representing the influence of
Cellular automata is a mathematical model particle collisions. The index i represents the
based on simple and local rules capable of gener- z possible directions of movement of the lattice,
ating complex behaviors. It was originally intro- that is, i ¼ 1, . . . z.
duced by John von Neumann, under the name of Conditions of mass conservation and moment
cellular spaces, as an idealization of biological conservation are imposed on the collision opera-
systems, with the particular goal to model systems tor, respectively, given by:
capable of self-reproduction (Wolfram 1994). The
LGCA is a specific cellular automaton, whose Di ðnÞ ¼ 0 and ci Di ðnÞ ¼ 0: ð2Þ
proposal is to simulate fluids using simple and i i
local rules that imitate a particle dynamics. The
essential characteristics of the microscopic inter- The physical quantities of interest are the mac-
actions that are taken into account are the laws of roscopic quantities, such as the specific mass
conservation of linear momentum and conserva- and linear momentum at a point in the system
tion of the number of particles (Chopard and Droz (Chopard and Droz 1998). The distribution
1998). which corresponds to the probability of having a
The LBM was introduced by McNamara and particle in the node x, at time step t is defined as
Zanetti (1988), where the authors show the advan- (Chopard et al. 2002):
tage of extending the Boolean dynamic of cellular
automaton to work directly with floating point N i ðx, tÞ ¼ hni ðx, tÞi, i ¼ 1, . . . , z: ð3Þ
numbers representing probabilities of particle
presence. Following the usual definition of statistical
mechanics (Chopard and Droz 1998), the local
density of particles is the sum of the probabilities
Theoretical Foundations of the microscopic variables of occupation
(Expression 3):
The LGCAs are cellular automata that simulate
fluids through simple models. In the LGCA, the z
rðx, tÞ ¼ N i ðx, tÞ: ð4Þ
fluid is represented by a set of particles which
i¼1
reside in a regular lattice with certain properties
of symmetry. The dynamics that governs the sim- Similarly, the linear momentum (ru) is given
ulation involves steps of collision and scattering by Chopard and Droz (1998):
of these particles through the lattice directions,
following simple rules that satisfy the laws of z
conservation of mass (number of particles) and rðx, tÞuðx, tÞ ¼ vi N i ðx, tÞ, ð5Þ
momentum. The macroscopic behavior of the i¼1
fluid is obtained through statistical results on the
data at the microscale. where vi are the speeds related to each direction
The dynamics of microparticles in the LGCA of movement. The time step is defined as δt and
model is described by the equation given below the displacement between the nodes of the lattice
(Daniel Reis Golbert 2009): as δx. Thus, the six possible velocities vi of the
Lattice Boltzmann Method for Fluid Simulation 1039

particles are related to their directions of motion The collision term can be simplified by consid-
by: ering that the particles motion is not correlated
before the collision step (principle of molecular
dx chaos). Thus, the collision operator can be applied
vi ¼ c: ð6Þ
dt i directly to the quantity representing the mean of
the distribution of microparticles in the region of
From LGCA to LBM Dynamics the lattice:

The dynamics of the LGCA method are related to N i ðx þ ci Dx, t þ DtÞ  N i ðx, tÞ ¼ Di ðNÞ: ð8Þ
quantities present on the microscale. In the case of
LBM, this dynamics occurs in the mesoscale Thus, we arrive at an equation similar to the
(particle distributions), where the individual lattice Boltzmann equation in the mesoscale,
movements of each particle are not visible. In defined as (Daniel Reis Golbert 2009):
the mesoscale we will work with averages taken
on regions of the lattice, so that these averages f i ðx þ ci Dx, t þ DtÞ  f i ðx, tÞ
vary smoothly in both space and time (Guo and ¼ Oi ðf ðx, tÞÞ, i ¼ 1, . . . , z, ð9Þ
Shu 2013). Thus, we will use the (3) distribution
covering an area of the lattice, rather than just a where fi is called the particle distribution function
node. The Ni value will represent mean values of (which assumes floating point values) and Ωi the
the distribution of microparticles described by the collision operator.
Boolean variables ni over a region of the lattice, as The macroscopic quantities of interest are cal-
shown in Fig. 1 (Daniel Reis Golbert 2009). culated analogously to the process given in the
Applying the calculation of the averages on the LGCA (Eqs. (4) and (5)), where the density of L
LGCA equation of motion (1) we will arrive at the particles is given by:
equation:
z

N i ðx þ ci Dx, t þ DtÞ ¼ N i ðx, tÞ þ hDi ðnÞi, ð7Þ rðx, tÞ ¼ f i ðx, tÞ, ð10Þ
i¼1

where Δx represents the new spacing and Δt the

and the linear momentum (ru) is given by:
new step of time considered in the mesoscale.
z
rðx, tÞuðx, tÞ ¼ ci f i ðx, tÞ: ð11Þ
i¼1

BGK Approximation

The LBM is a numerical method whose equations

are formulated on a mesoscopic scale, although it
is able to simulate the dynamics of fluids on a
macroscopic scale (Chen and Doolen 1998). The
dynamics of this method is governed by the lattice
Boltzmann Eq. (9).
According to (Chopard et al. 2002), the most
natural way to define the collision term is by
averaging the micro-dynamics and factoring it
into a product of average quantities. However,
Lattice Boltzmann Method for Fluid Simulation,
for more sophisticated fluids, the collision term
Fig. 1 Representation of a region of the hexagonal lattice requires a large number of floating point
1040 Lattice Boltzmann Method for Fluid Simulation

operations at each node of the lattice at each Navier-Stokes Equations (Daniel Reis Golbert
instant of time, which would increase the compu- 2009).
tational cost. One solution is to use the BGK The LBGK models are the most used among
approximation. the LBM models for fluid simulation. Among
The BGK approach for the collision operator the available models, those that represent the
uses a relaxation term toward the local equilib- Navier-Stokes equations are called DnQb (where
rium (Bhatnagar et al. 1954). Thus, the collision n refers to the dimension and b the number of
operator through the BGK approximation is given lattice directions), proposed by Quian et al.
by: (1992), whose parameter values are shown in
Table 1.
1
Oi ðf ðx, tÞÞ ¼  f ðx, tÞ  f eq
i ðx, tÞ , ð12Þ
t i
Incompressible Equilibrium Distribution
where t is the relaxation term, which is related to Function
the diffusive phenomena in the problem (viscosity
of the fluid), and f eq
i is the local equilibrium Applying a Chapman-Enskog multiscale asymp-
distribution function, which can be given by totic expansion in the lattice Boltzmann equation
Eq. (14) forward. Thus, the lattice Boltzmann and assuming that the velocity of the fluid is less
equation with BGK approximation is given by: than the velocity of sound, it can be shown that
it is possible to derive the incompressible Navier-
f i ðx þ ci Dx, t þ DtÞ  f i ðx, tÞ Stokes Equations (Chopard et al. 2002). There-
fore, it is necessary to use an incompressible equi-
1
¼ f ðx, tÞ  f eq
i ðx, tÞ : ð13Þ librium distribution function, which reduces the
t i
inherent compressibility effects of the LBM.
In this model called lattice BGK (LBGK), the
local equilibrium distribution was chosen so that
the Navier-Stokes equations can be recovered Lattice Boltzmann Method for Fluid Simulation,
asymptotically (Chen and Doolen 1998). The gen- Table 1 Parameters of some models DnQb
eral form of this equilibrium distribution is given Weights
by Chopard and Droz (1998), Guo and Shu Model Direction vector ci oi c2s
(2013): D1Q3 0, 2/3, 1/3
1 1/6
f eq
i ðx, tÞ
D1Q5 0, 6/12, 1
1, 2/12,
ð ci  u Þ ðci  uÞ2 ðu  uÞ 2 1/12
¼ roi A þ B 2
þ C 4
þD ,
cs 2cs 2c2s D2Q7 (0,0),
p
1/2, 1/4
(l/2, 3=2) 1/12
ð14Þ
D2Q9 (0,0), 4/9, 1/3
(1,0),(0,1), 1/9,
where oi are weight factors related to the direc- (1,1) 1/36
tions of the lattice, r and u are the macroscopic D3Q15 (0,0,0), 2/9, 1/3
quantities of particle density and velocity (given, (1,0,0),(0,1,0), 1/9,
respectively, by Eqs. (10) and (11)), cs is called the (0,0,1), 1/72
(1,1,1)
sound velocity, and A, B, C, and D are constant,
D3Q19 (0,0,0), 1/3, 1/3
whose values depend on the model used. These (1,0,0),(0,1,0), 1/18,
constants of the equilibrium distribution are (0,0,1), 1/36
related to the symmetry imposed on the mesh (1,1,0),(1,0,1),
models used, which are necessary to recover the (0,1,1)
Lattice Boltzmann Method for Fluid Simulation 1041

In an incompressible fluid, the density is D2Q9 Model

approximately constant (denoted by r0) and its
fluctuation (denoted by δr) must be of the order The lattice Boltzmann model known as D2Q9 has
O(M2)(M ! 0), where M ¼ j u j /cs is the Mach eight nonzero motion directions and the possibil-
number. Thus, and seeking to reduce the com- ity of having a resting particle. It is a two-
pressibility errors that arise when using the equi- dimensional model, as illustrated in Fig. 2.
librium distribution function (14), the authors in In the D2Q9 model, the time step is discrete,
He and Luo (1997b) proposed a modified version and for each lattice node, there are nine possible
of it: directions of motion, where c0 represents the lat-
tice node itself, whose velocity is zero (Chen and
f eq
i Doolen 1998; Chopard et al. 2002). In this way,
9 3 the number of possible directions is given by
¼ wi r þ r0 3ðci  uÞ þ ðci  uÞ2  ðu  uÞ :
2 2 z ¼ 0, . . ., 8.
ð15Þ The eight possible velocities vi, given by
Expression (6), are related to directions of move-
With the use of this equilibrium distribution, ment ci (Fig. 2), described by:
the incompressible Navier-Stokes equations are
recovered, with approximation orders in terms of c0 ¼ ð0, 0Þ, c1 ¼ ð1, 0Þv, c2 ¼ ð1, 1Þv,
the Mach number (Daniel Reis Golbert 2009): c3 ¼ ð0, 1Þv, c4 ¼ ð1, 1Þv, c5 ¼ ð1, 0Þv,
c6 ¼ ð1, 1Þv, c7 ¼ ð0, 1Þv, c8 ¼ ð1, 1Þv
∇  u ¼ 0 þ O M2 , e ð16Þ ð20Þ

@u Two velocities are introduced, according to the L

þ u  ∇ u ¼ ∇ P þ v∇ 2 u þ O M3 , ð17Þ
@r distance to be traveled in a time step, namely:

where P ¼ c2s r=r0 is the normalized pressure and • The slow velocities with module given by
v is the kinematic viscosity of the fluid, described v ¼ Δx/Δt in the vertical and horizontal direc-
by: tions (c1, c3, c5, c7)
p
• Fast velocities with module 2v in the diago-
2t  1 Dx2
v¼ : ð18Þ nal directions (c2, c4, c6, c8)
6 Dt
This occurs due to the different distances that
It is noted that a value equal to 0.5 for the
distributions must travel, in the same time inter-
parameter t implies in viscosity of the fluid iden-
val, to reach the neighboring nodes.
tically null, situation not contemplated in the
model. Values less than 0.5 indicate negative vis-
cosities, which would be in disagreement with the
laws of thermodynamics. From Eq. (18), we have:

Dt 1
t ¼ 3v þ : ð19Þ
Dx2 2

According to (Daniel Reis Golbert 2009), not

found in the literature, a minimum value for the
parameter t guarantees the numerical stability of
the method. Numerical instabilities appear when t Lattice Boltzmann Method for Fluid Simulation,
is close to 0.5. Fig. 2 Direction of motion in the lattice of D2Q9
1042 Lattice Boltzmann Method for Fluid Simulation

The update rule for the D2Q9 model follows 8

the BGK model described in the Eq. (13). How- rðx, tÞ ¼ f i ðx, tÞ, ð24Þ
i¼0
ever, it is common to divide the dynamics of the
LBM into two steps, namely: collision and scat-
8
tering. Figure 3 illustrates the complete dynamics 1
uðx, tÞ ¼ f i ðx, tÞci : ð25Þ
procedure divided into these steps. rðx, tÞ i¼1
The collision step is governed by the following
equation: Boundary Conditions

f i ðx, t þ DtÞ ¼ f i ðx, tÞ½1  t þ tf eq

i ðx, tÞ: ð21Þ Different types of contour conditions have been
introduced in the field of hydrodynamics for the
The scattering step is governed by another LBM. Bounce back is the simplest of them
equation given below: (Rothman and Zaleski 1997), used in boundary
conditions whose walls are not slippery, that is,
f i ðx þ Dx, t þ DtÞ ¼ f i ðx, t þ DtÞ: ð22Þ the nodes present in the walls present zero veloc-
ity in all directions. When the particles propagate
The general form of the equilibrium distribu- to a boundary node, they simply return in the same
tion function is described by Eq. (14). The author direction they were.
in (He and Luo 1997a) obtains the following Because of its simplicity, such a method cannot
values of weights oi and constants for the model adequately represent curved boundary problems
D2Q9 (see Table 1): without introducing noise into the solution, espe-
cially in poorly refined lattices. An incorrect value
9 2 3 2 may eventually cause a negative density value,
f eq
i ¼ roi 1 þ 3ðci  uÞ þ 2 ðci  uÞ  2 u , generating an error that may accumulate along
ð23Þ the evolution (Rothman and Zaleski 1997).
The periodic boundary condition is a simple
where: option to apply. Basically, it is necessary to con-
nect the ends of the domain according to the
4 1 desired directions. Figure 4 shows the topologies
o0 ¼ , oi ¼ , ði ¼ 1, 3, 5, 7Þ, oi of the domains when applying periodic boundary
9 9
1 conditions in one direction and two directions in a
¼ , ði ¼ 2, 4, 6, 8Þ: two-dimensional model.
36
Another boundary condition is proposed by
The macroscopic amounts of density (r) and Zou and He (1997), where the authors impose
velocity (u) are given, respectively, by: velocity values (not necessarily null) on the

Lattice Boltzmann Method for Fluid Simulation, Fig. 3 Steps of LBM dynamics
Lattice Boltzmann Method for Fluid Simulation 1043

centers of the border nodes. The basic idea is to walls (Fig. 5a) and frontier nodes present in the
introduce a step, between the collision step (21) corners of lattice (Fig. 5b).
and propagation step (22), to calculate the distri- The first situation shows the case of a border
butions in some directions of the boundary nodes. node present in the bottom wall. After the propa-
After the propagation, the fis from the interior gation (22), the distributions f1, f5, f6, f7, and f8 are
and from the border itself are known at the border known, because they come from the neighboring
nodes. The unknown fis as well as the density (r) nodes. According to the boundary condition pro-
of the boundary node are calculated from the mass posed by Zou and He (1997), we will impose a
conservation Eq. (24), of the linear momentum velocity value (u) on the border nodes present on
conservation Eq. (25), where boundary velocity the bottom wall. To determine the four unknown
is imposed, and through the reflection of the variables (f2, f3, f4, and r), we will use the mass
so-called non-equilibrium part (Zou and He conservation Eq. (24):
1997):
f2 þ f3 þ f4 ¼ r
f i  f eq eq
i ¼ f iþz=2  f iþz=2 , ð26Þ  ðf 0 þ f 1 þ f 5 þ f 6 þ f 7 þ f 8 Þ,
ð27Þ
where z represents the number of nonzero direc-
tions of the lattice. The D2Q9 model has z ¼ 8. the conservation equations of linear momentum
To exemplify the abovementioned boundary (25) in x direction:
condition, we show two-boundary situations in
the D2Q9 model: boundary nodes present on the f 2  f 4 ¼ rux  f 1 þ f 5 þ f 6  f 8 , ð28Þ

L
Lattice Boltzmann
Method for Fluid
Simulation,
Fig. 4 Periodic boundary
for one direction (left) and
two directions (right)

Lattice Boltzmann Method for Fluid Simulation, Fig. 5 Boundary of D2Q9 model. (a) An example of a node present
on the bottom wall. (b) An example of a node in the corner of the lattice
1044 Lattice Boltzmann Method for Fluid Simulation

and y direction: With three equations and six unknown variables,

it is not possible to determine r. In this way, the
f 2 þ f 3 þ f 4 ¼ ruy þ f 6 þ f 7 þ f 8 : ð29Þ density at the border node in the lower-right corner
will be determined by the average of the neighbor-
From Eqs. (27) and (29), we have: ing densities. However, f2, f3, f4, f5, and f6 remain
indeterminate. We then use the reflection of the non-
1 equilibrium part of the distribution f1 and f7:
r¼
1  uy
 ½f 0 þ f 1 þ f 5 þ 2ðf 6 þ f 7 þ f 8 Þ: ð30Þ f 5  f eq eq
5 ¼ f1  f1 : ð38Þ

However, f2, f3, and f4 remain indeterminate. In f 3  f eq eq

3 ¼ f7  f7 ð39Þ
this way, we use the reflection of the non-
equilibrium part of the distribution f7: From Eqs. (35, 36, 37, 38, and 39) and the
estimated density value (r), we are able to calcu-
f 3  f eq eq late the unknown variables:
3 ¼ f7  f7 : ð31Þ

2
From Eqs. (27, 28, 29, 30, and 31) we can f 3 ¼ f 7 þ ruy ð40Þ
calculate the unknown variables: 3

2
2 f 5 ¼ f 1  rux ð41Þ
f 3 ¼ f 7 þ ruy ð32Þ 3
3
1 1
1 1 1 f 4 ¼ f 8  rux þ ruy ð42Þ
f 4 ¼ f 8 þ ðf 1  f 5 Þ  rux þ ruy ð33Þ 6 6
2 2 6
1 1
1 1 1 f 6 ¼ ðr  f 0 Þ  ðf 1 þ f 7 þ f 8 Þ þ rux
f 2 ¼ f 6  ðf 1  f 5 Þ þ rux þ ruy : ð34Þ 2 3
2 2 6
1
 uuy ð43Þ
An analogous procedure is applied to the nodes 2
that are present in the corners of the lattice. As an 1 1
example, let’s look at the node in the lower-right f 2 ¼ ðr  f 0 Þ  ðf 1 þ f 7 þ f 8 Þ þ rux
2 2
corner, as shown in Fig. 5b. After propagation (22), 1
the distributions f0, f1, f7, and f8 are known. We will  ruy : ð44Þ
3
impose them a velocity value (u), and to determine
the six unknown variables (f2, f3, f4, f5, f6, and r), Algorithm 1 shows all the steps of the LBM
we will use the mass conservation Eq. (24): methodology for the D2Q9 model.

f2 þ f3 þ f4 þ f5 þ f6 Algorithm 1 D2Q9 Model algorithm.

¼ r  ðf 0 þ f 1 þ f 7 þ f 8 Þ, ð35Þ
Dx ¼ 50, Dy ¼ 50, lattice[50]
[50]. f [9] //each node has 9 fi’s
the conservation equations of linear momentum // INITIALIZATION STEP
(25) in x direction: density ¼ 1.0
velocity[50][50] ¼ 0.0
f 2  f 4  f 5  f 6 ¼ rux  ðf 1 þ f 8 Þ, ð36Þ for each x in [0, Dx  1] do
for each y in [0, Dy  1] do
for each i in [0, 8] do
and y direction: lattice[x][y]. f [i] ¼ f
eq(density, velocity[x][y], i)
f 2 þ f 3 þ f 4  f 6 ¼ ruy þ f 7 þ f 8 : ð37Þ end for
end for
Lattice Gas Cellular Automata for Fluid Simulation 1045

end for Chopard, B., Dupuis, A., Masselot, A., Luthi, P.: Cellular
// SOLVER automata and lattice Boltzmann techniques: an
repeat approach to model and simulate complex systems.
// SAVE VELOCITY FIELD FOR STOP Adv. Complex Syst. 05, 103–246 (2002)
CONDITION Daniel Reis Golbert.: Modelos de lattice-Boltzmann
for each x in [0, Dx  1] do aplicados a simulac¸a~o computacional do escoamento
for each y in [0, Dy  1] do de fluidos incompresśıveis. Master’s thesis, LNCC –
old[x][y] ¼ velocity[x][y] Laboratório Nacional de Computac¸a~o Cient́ıfica
end for (2009)
end for Guo, Z., Shu, C.: Lattice Boltzmann Method and its Appli-
// INNER LATTICE DYNAMICS cations in Engineering Advances in Computational
for each x in [1, Dx  2] do Fluid Dynamics, vol. 3. World Scientific Publishing,
for each y in [1, Dy  2] do Singapore (2013)
Scattering of node [x][y] for He, X., Luo, L.-S.: A priori derivation of the lattice
its neighbors Boltzmann equation. Phys. Rev. E. 55(6), R6333–
Collision at node [x][y] R6336 (1997a)
end for He, X., Luo, L.-S.: Lattice Boltzmann model for the incom-
end for pressible Navier-stokes equation. J. Stat. Phys. 88,
Treats boundary conditions 927–944 (1997b)
// STOP CONDITION McNamara, G.R., Zanetti, G.: Use of the Boltzmann equa-
max ¼ 0 tion to simulate lattice-gas automata. Phys. Rev. Lett.
for each x in [1, Dx  2] do 61(20), 2332–2335 (1988)
for each y in [1, Dy  2] do Philippi, P.C., Hegele, L.A., dos Santos, L.O.E.,
norm ¼ |velocity[x][y]  old Surmas, R.: From the continuous to the lattice
[x][y]| Boltzmann equation: the discretization problem and
if (norm > max) then thermal models. Phys. Rev. E. 73, 56702 (2006)
max ¼ norm Quian, Y.H., d’Humires, D., Lallemand, P.: Lattice bgk
end if models for Navier-stokes equation. Europhys. Lett. L
end for 17, 479–484 (1992)
end for Rothman, D., Zaleski, S.: Lattice-Gas Cellular
until (max >¼ delta) Automata: Simple Models of Complex Hydrody-
namics. Cambridge University Press, Cambridge
(1997)
Wolfram, S.: Cellular Automata and Complexity: Col-
lected Papers, 1st edn. AddisonWesley. http://www.
Cross-References stephenwolfram.com/publications/books/ca-reprint/.
(1994)
Zou, Q., He, X.: On pressure and velocity boundary con-
▶ Cellular Automata Methods ditions for the lattice Boltzmann bgk model. Phys.
▶ Fluid Simulation Fluids. 9(6), 1591–1598 (1997)
▶ Lattice Gas Cellular Automata for Fluid
Simulation

References Lattice Gas Cellular Automata

for Fluid Simulation
Adilson Vicente Xavier.: Animac¸a~o de fluidos via autoˆ-
matos celulares e sistemas de part́ıculas. Master’s the-
sis, LNCC – Laboratório Nacional de Computac¸a~o Sicilia Ferreira Judice
Cient́ıfica, Agosto (2006) Faculty of Technical Education State of Rio de
Bhatnagar, P.L., Gross, E.P., Krook, M.: A model for Janeiro, FAETERJ Petropolis, Petropolis, Brazil
collision processes in gases: small amplitude processes
in charged and neutral one-component system. Phys.
Rev. 94, 511–525 (1954)
Chen, S., Doolen, G.D.: Lattice Boltzmann method for fluid Synonyms
flows. Annu. Rev. Fluid Mech. 30, 329–364 (1998)
Chopard, B., Droz, M.: Cellular Automata Modeling of
Cellular automata; Fluid simulation; Lattice
Physical Systems. Cambridge University Press,
Cambridge (1998) methods; Numerical methods; Physics simulation
1046 Lattice Gas Cellular Automata for Fluid Simulation

Definitions macroscopic variables and partial differential

equations (PDE), the LGCA modeling follows
• Cellular automata the inverse path, that is, from the microscopic
Mathematical models based on simple and description based on simple rules, it tries to obtain
local rules capable of generating complex the macroscopic behavior (bottom up) without,
behaviors. however, explicitly solving any system of PDE
• Computational fluid dynamics (CFD) (Chopard and Droz 1998). The macroscopic
Area of knowledge that studies computa- behavior is obtained through computational sim-
tional methods to solve problems in fluid ulation, based on the set of rules governing the
dynamics. system, and simple interpolations.
• FHP LGCA can be seen as a simplified universe.
A specific cellular automata model to sim- Space is represented by a uniform lattice, where
ulate fluid dynamics. each cell contains Boolean data, the time step is
• Lattice gas cellular automata (LGCA) discrete, and the laws governing that universe are
Cellular automata that simulates specifi- expressed by simple rules of local collisions. They
cally fluid problems. are discrete models, based on particles whose
movement is restricted to the directions of the
lattice edges. These methods have a lower com-
Introduction putational cost and are more stable compared to
the traditional ones used in fluid simulation, due to
In recent decades, techniques based on physical the fact that it is not necessary to calculate PDE to
models for the animation of natural elements, such obtain the desired level of physical realism
as fluids (gas or liquid), have drawn the attention (Rothman and Zaleski 1997).
of the researchers in computer graphics. The moti- Among the varieties of automata of the LGCA
vation lies in the potential in the applications of type, this article will talk about the FHP model for
these techniques as well as in the complexity and its simplicity and computational efficiency
beauty of the natural phenomena involved. In (Chopard and Droz 1998). FHP was developed
particular, techniques in the field of computational to simulate two-dimensional fluids. The space is
fluid dynamics (CFD) have been implemented for discretized in a hexagonal lattice, that is, in each
fluid animation. lattice node, there are six possible directions of
Numerical methods in DFC for fluid simula- movement. In addition, a multiscale technique
tion, such as finite differences and finite elements was applied in order to demonstrate that the FHP
(Hughes 1987), try to describe a continuous sys- model is capable of reproducing Navier-Stokes
tem through the discretization of the equations behaviors for two-dimensional fluids (Frisch
that represent it. However, such methods have a et al. 1986).
high computational cost. Another alternative in
this area is the use of techniques based on cellular
automata (Wolfram 1994; Sarkar 2000; Kari FHP Model
2005). Such methods seek to obtain the macro-
scopic dynamics of the fluid through the collective The FHP model was introduced by Frisch,
behavior of numerous microscopic particles. The Hasslacher, and Pomeau (Frisch et al. 1986),
specific cellular automata to simulate fluids are where FHP represents the name initials of their
called lattice gas cellular automata (LGCA) and authors. It is a specific cellular automaton for fluid
follow this idea by simplifying the dynamics simulation, known as lattice gas cellular automata.
through simple local rules for the interactions Although the LGCAs cannot compete with tradi-
and displacement of the microscopic particles. tional techniques of computational fluid dynam-
While traditional CFD modeling attempts to ics, particularly for high Reynolds numbers,
represent a continuous medium starting from several authors have succeeded in using such
Lattice Gas Cellular Automata for Fluid Simulation 1047

models to simulate complex systems, for which lattice node, at a given instant, in a given direc-
traditional techniques are difficult to apply tion. Such a constraint is called the exclusion
(Chopard et al. 1998), such as porous media principle, which ensures that six Boolean vari-
(Chen et al. 1991) and granular media (Krolyi ables for each lattice node are sufficient to repre-
and Kertsz 1994), among others (Boghosian sent the microdynamics.
et al. 1996). In addition, from the analysis of Each particle moves at a constant velocity in
FHP microdynamics, through multiscale tech- modulus, in such a way that, in each interaction it
niques, it is possible to obtain the traditional traverses one edge of the lattice and reaches the
fluid dynamics equations (Appendix A) on a mac- neighboring node. In the absence of collisions, the
roscopic scale (Frisch et al. 1986). These facts particles keep moving along the direction speci-
motivate the study of FHP as an alternative meth- fied by their velocity vector. Collisions occur
odology for fluid simulation. when particles enter the same node at the same
FHP is a two-dimensional model and can be instant, resulting in a new local distribution of
seen as an abstraction, on a microscopic scale, of a particle velocities.
fluid. The FHP describes the motion/interaction of When exactly two particles are incident on the
particles in a discretized space in a hexagonal same node with opposite velocities, both are
lattice, as seen in Fig. 1. deflected by an angle of 60 so that after the
The microdynamics of FHP are given in terms collision a new configuration is also made with
of Boolean variables that describe the number of zero momentum. Such a deviation may occur
occupancy at each node of the lattice at each clockwise or counterclockwise, as shown in
interaction step (i.e., the presence or not of parti- Fig. 2. For reasons of symmetry, the two possibil-
cle). The particles move in discrete time steps, ities are chosen randomly, with equal probability.
with a constant velocity in modulus, pointing When exactly three particles with velocities at an L
along one of the six directions of the lattice. No angle of 120 collide, each of them returns,
more than one particle can move to the same toward the initial edge, as shown in Fig. 2. In
this way, the moment remains null and is therefore
preserved. For other configurations, the particles
continue their movement as if there was no
collision.

The FHP Microdynamic

The complete microdynamics of the FHP model
can be expressed by an evolution equation for the
occupation numbers, defined as the number ni(x,t)
of particles entering the node x at instant t with
velocity in the direction ci (Chopard and Droz
1998):

2pði  1Þ 2pði  1Þ
ci ¼ cos , sin ,
6 6
ð1Þ

where i ¼ 1,. . .,6 represents the six possible direc-

tions for each lattice node, as shown in Fig. 3.
The numbers ni can be 0 or 1, that is, presence
Lattice Gas Cellular Automata for Fluid Simulation, or not of particle in the ci direction of the lattice.
Fig. 1 The lattice of FHP model Also defined is the time step as Δt and the
1048 Lattice Gas Cellular Automata for Fluid Simulation

Lattice Gas Cellular

Automata for Fluid
Simulation,
Fig. 2 Particle collisions in
the FHP model

line, in such a way that at the next instant it will

enter the node x + Δxci with the same direction of
motion (Chopard et al. 1998). However, when a
collision occurs, one particle may be removed
from its original direction or may be diverted to
another. Let us look at the example of the collision
described in Fig. 2, where exactly two particles
compete for the same node with opposite veloci-
ties (top of the figure). In this case, before the
collision, only n2 and n5 has 1 value in the
x node. After the collision, we have two possibil-
ities: n1 and n4 or n3 and n6 will have 1 value.
Lattice Gas Cellular Automata for Fluid Simulation, Generalizing for the collision between two
Fig. 3 Directions of movement in FHP lattice
particles with opposite velocities, the particle
moving with velocity vi before the collision will
displacement between lattice nodes as Δx. Thus,
move with velocity vi  1 or vi + 1, after the
the six possible velocities vi of the particles are
collision, where i ¼ 1,. . .,6 (note that operations
related to their directions of motion by:
on the index i must return values between 1 and 6).
The binary quantity:
Dx
vi ¼ c: ð2Þ
Dt i
Di ¼ ni niþ3 ð1  niþ1 Þð1  niþ2 Þð1  niþ4 Þ
If there is no interaction between particles, the ð1  niþ5 Þ
evolution equation for ni can be written as ð4Þ
(Chopard et al. 1998):
indicates that a collision will occur when Di ¼ 1.
ni ðx þ Dx ci , t þ Dt Þ ¼ ni ðx, tÞ: ð3Þ In fact, if there is a particle in direction ci and
one in direction ci + 3, the term nini + 3 will be 1.
The Eq. 3 says that a particle entering the node Thus it suffices that there is no particle in the
x with velocity along ci will continue in a straight other directions to ensure Di ¼ 1. Thus ni  Di
Lattice Gas Cellular Automata for Fluid Simulation 1049

is the number of resultant particles in the ci Oi ¼ Di þ qDi1 þ ð1  qÞDiþ1  T i

direction due to a collision of two particles þ T iþ3 : ð9Þ
along this direction.
However, when ni ¼ 0, a new particle may The Eq. 8 can be easily implemented, and the
appear in direction ci, as a result of a collision FHP model can be efficiently simulated in a com-
between ni + 1 and ni + 4 or a collision puter. More elaborate collision operators can be
between ni  1 and ni + 2. At this point, it is created by including collisions between four par-
convenient to introduce a random Boolean ticles or even by inclusion of a resting particle
variable q(x, t), whose value is used to decide (Frisch et al. 1987). In general, the only constraint
whether a particle will be reflected to the right imposed is that the collision operator Ωi preserves
(q ¼ 1) or to the left (q ¼ 0). Thus, the 6 6
number of particles created in ci direction is the mass Oi ¼ 0 and the moment vi Oi ¼ 0:.
i¼1 i¼1
given by:
The Macroscopic Quantities
qDi1 þ ð1  qÞDiþ1 : ð5Þ The physical quantities of interest are not the
Boolean variables but the macroscopic quantities
For the case shown in Fig. 2 of a collision or the mean values, such as density and linear
between three particles, they can be scattered, momentum at a point in the system (Chopard
with one of them occupying the direction ci. The and Droz 1998). The distribution below corre-
amount (T) expressing the occurrence of a colli- sponds to the probability of having a particle in
sion between three particles ni, ni + 2, and ni + 4 can the node x, at instant t, with velocity vi defined by
be obtained by following the same principles the Eq. 2 (Chopard et al. 1998).
above, given by: L
N i ðx, tÞ ¼ hni ðx, tÞi, i ¼ 1, . . . , 6: ð10Þ
T i ¼ ni niþ2 niþ4 ð1  niþ1 Þð1  niþ3 Þ
 ð1  niþ5 Þ: ð6Þ In general, an LGCA model is characterized
by the z number of directions for each node of
Analogous to the previous case, the result of a the lattice and the spatial dimension d. For a
collision between three particles is modify the rectangular lattice with d ¼ 2, we have z ¼ 4,
number of particles in the direction ci. Thus, the whereas for a hexagonal lattice we have z ¼ 6. In
number of resulting particles in ci direction, con- some models there is the addition of a direction
sidering the disappearance of a particle due to (c0), where n0 ∈ {0, 1} will be the number of
collision at position i or the appearance of a par- particles at rest (v0 ¼ 0) (Chopard and Droz
ticle due to collision at position i + 3 can be 1998).
expressed by: Following the usual definition of statistical
mechanics, the local density of particles is the
ni  T i þ T iþ3 : ð7Þ sum of the means of microscopic variables
(expression 10):
Once the collision rules have been stipulated,
we can rewrite the Eq. 3 to the microdynamics of z
the FHP as: rðx, tÞ ¼ N i ðx, tÞ: ð11Þ
i¼1

ni ðx þ Dx ci , t þ Dt Þ ¼ ni ðx, tÞ Similarly, the linear momentum (ru) is given

þ Oi ðnðx, tÞÞ, ð8Þ by (Chopard and Droz 1998):

z
where Ωi is the collision term, defined from rðx, tÞuðx, tÞ ¼ vi N i ðx, tÞ: ð12Þ
expressions (5) and (7): i¼1
1050 Lattice Gas Cellular Automata for Fluid Simulation

From these definitions and conservation laws, law for the scalar quantity U can be written as
it is possible to estimate the macroscopic behavior (Hirsch 1988):
of the system. For this, the multiscale expansion
of Chapman-Enskog is used. The details of this @
UdO þ F  dS ¼ QV dO þ QS  dS: ð13Þ
technique can be found in the work of (Frisch et al. @t
O O S
1986). The result is the obtaining of mass and
Navier-Stokes conservation equations from the
microscopic dynamics of the FHP described in In Eq. 13, the first term to the left of the equal-
this section. ity sign represents the U variation within the vol-
ume Ω per unit time. Such variation must be equal
to the contribution of external medium due to the
Cross-References flow through the surface S, given by:

▶ Fluid Simulation  F  dS, ð14Þ

plus the contributions of volumetric (QV) and

Appendix A surface (QS) sources of U, expressed, respectively,
by the terms to the right of the equal sign in the
Mathematical Foundations Eq. 13.
Assuming continuity of flow and surface
Conservation Laws sources, one can use the Gaussian theorem, also
The most general conservation principles are known as the divergence theorem, and rewrite the
the laws of mass conservation, moment, and expression (13) as:
the first and second laws of thermodynamics.
Assuming that matter cannot be destroyed or @U
created and despising sources, the amount dO þ ∇  F ¼ QV dO þ ∇
@t
entering a volume element equals the quantity O O O O
that exits plus the amount accumulated in the  QS dO: ð15Þ
element. This principle can be expressed in
terms of transfer rates: Equation 15 is in the integral form of the con-
servation law. The differential form is obtained
Input rate ¼ output rate þ accumulation rate: directly from this, assuming any volume, and is
given by:
The following is a review of the principles of
mass conservation and movement. Initially, the @U
general expressions of these laws will be devel- þ ∇  F ¼ QV þ ∇  QS )
@t ð16Þ
oped for scalar and vector fields and then applied @U
to the specific case of fluid mechanics. þ ∇  ðF  QS Þ ¼ QV :
@t

Conservation Laws for Scalar Fields Conservation Laws for Vector Fields
Let U be a scalar quantity per unit volume, For the case where the conserved amount is
defined in an arbitrary volume Ω, fixed in described by a vector quantity U, then the flow
space, and bounded by a closed surface S. The and the term relative to the surface sources
variation of the local intensity of U occurs due become tensors, F and QS , respectively, and the
to the flow acting which expresses the contribu- term corresponding to the volumetric sources
tion of the external medium to Ω and the becomes a vector QV. Equations 15 and 16,
Q sources. The general form of the conservation respectively, assume the form (Hirsch 1988):
Lattice Gas Cellular Automata for Fluid Simulation 1051

@U the general form given by Eqs. 17 and 18. The

dO þ ∇  FdO
@t central issue now is to determine the tensors F and
O O
QS and the vector QV that appear in these
¼ QV dO þ ∇  QS dO, ð17Þ equations.
By Newton’s second law, it is known that the
O O
agents responsible for the linear momentum of a
@U physical system are the forces acting on it. Such
þ ∇  F  QS ¼ QV : ð18Þ forces may be external or internal. Let us then
@t
consider a portion of fluid bounded by a
Mass Conservation Equation surface S.
The partial differential equation that expresses As external forces, we can have force fields
conservation of mass is also known as the conti- such as the gravitational field and electromag-
nuity equation. Applying the conservation law netic field. Let fe be the volumetric density of
given by the Eq. 15 for the particular case where the resultant of these forces. In addition to exter-
U is the mass density density (r), we have: nal forces to the system, we also have internal
forces due to deformations and internal tensions
@r of the same. In this text, it is assumed that the
dO þ ∇  ðrvÞ
@t fluid is Newtonian, that is, that the internal
O O stresses are given by the expression (Hirsch
1988):
¼ QV dO þ ∇  QS dO, ð19Þ
O O
s ¼ pI þ t, ð23Þ L
where v is fluid velocity. In the differential form,
we have: where I is a unit tensor, p is the hydrostatic
pressure, and t is the tensor of tensions whose
@r components are given by (Hirsch 1988):
þ ∇  ðrvÞ ¼ QV þ ∇  QS : ð20Þ
@t
2
tij ¼ v @ i vj þ @ j vi  ð∇  vÞdij , ð24Þ
In the absence of sources, the continuity equa- 3
tion takes into account only the density and veloc-
ity of the fluid. It can be applied to all fluids, where n is the kinematic viscosity.
compressible and incompressible and Newtonian By integrating (23) along S, we have the resul-
and non-Newtonian (Shaughnessy et al. 2005). tant of the internal forces to the fluid that act on the
For incompressible fluids, the density is constant, volume Ω. As for the tensor F, this will be given
that is, it is not a function of the spatial coordinates by (Hirsch 1988):
nor of the time. The Eq. 20 results in:
F ¼ rv  v, ð25Þ
∇  v ¼ 0: ð21Þ
where:
Another particular case occurs when the den-
sity is independent only of time, resulting in: v1 v1 v1 v2 v1 v3
∇  ðrvÞ ¼ 0: ð22Þ vv¼ v2 v1 v2 v2 v2 v3 , ð26Þ
v3 v1 v3 v2 v3 v3
Moment Conservation Equation
The linear momentum, defined by (rv), is a vector with v ¼ (v1,v2,v3).
quantity, and therefore its conservation law has Thus, the expression (17) takes the form:
1052 Lattice Gas Cellular Automata for Fluid Simulation

@ ðrvÞ Dv @v
dO þ ∇  ðrv  vÞdO r ¼ r þ rðv  ∇ Þv ¼ ∇ p þ rf e : ð33Þ
@t Dt @t
O O
Therefore, we find the equations of mass con-
¼ rf e dO þ sdS: ð27Þ servation and Navier-Stokes, given, respectively,
O S by:

Substituting s with the expression (23) gives @r

þ ∇  ðrvÞ ¼ QV þ ∇  QS ð34Þ
the integral form of the conservation law of linear @t
momentum for a Newtonian fluid. The
Dv 1
corresponding differential expression is obtained r ¼ ∇ p þ v Dv þ ∇ ð∇  vÞ
directly from the integral form, given by (Hirsch Dt 3
1988): þ rf e : ð35Þ

Such equations form the basis for describing

@ ðrvÞ
þ ∇  rv  v þ pI  t ¼ rf e : ð28Þ the behavior of a fluid, particularly for computer
@t
graphics animations.
Subtracting the Eq. 20 from the left side of the
Eq. 28, multiplied by r, and assuming the source
terms are null, we find: Cross-References

r
Dv
¼ ∇ p þ ∇  t þ rf e , ð29Þ ▶ Fluid Simulation
Dt

where Dv/Dt is the material derivative, given by: References

Dv @v Boghosian, B., Coveney, P., Emerton, A.: A lattice-gas

¼ þ v  ∇ v: ð30Þ model of microemulsions. Proc. R. Soc. Lond. A. 452
Dt @t (1948), 1221–1250. (1996). https://doi.org/10.1098/
rspa.1996.0063
When the expression (24) for the tensor of Chen, S., Diemer, K., Doolen, G.D., Eggert, K., Fu, C.,
tensions of a viscous Newtonian fluid is replaced Gutman, S., Travis, B.J.: Lattice gas automata for flow
in Eq. 29, we obtain what we call Navier-Stokes through porous media. In: Proceedings of the NATO
Advanced Research Workshop on Lattice Gas Methods
equations. For constant viscosity coefficients, for PDEs: Theory, Applications and Hardware, pp. 72–84.
this equation reduces to: North-Holland Publishing Co, Amsterdam (1991)
Chopard, B., Droz, M.: Cellular Automata Modeling of
Dv 1 Physical Systems. Cambridge University Press,
r ¼ ∇ p þ v Dv þ ∇ ð∇  vÞ Cambridge (1998)
Dt 3
Chopard, B., Dupuis, A., Masselot, A., Luthi, P. Cellular
þ rf e : ð31Þ Automata and Lattice Boltzmann Techniques: An
Approach to Model and Simulate Complex Systems. In:
Advances in Complex Systems (ACS), 5, 103–246 (2002)
For incompressible fluids (∇  v ¼ 0), the
Frisch, U., Hasslacher, B., Pomeau, Y.: Lattice-gas autom-
Eq. 31 reduces to: ata for the navier-stokes equation. Phys. Rev. Lett.
56(14), 1505–1508 (1986)
Dv Frisch, U., D’Humières, D., Hasslacher, B., Lallemand, P.,
r ¼ ∇ p þ vDv þ rf e : ð32Þ Pomeau, Y., Rivet, J.-P.: Lattice gas hydrodynamics in
Dt
two and three dimension. Complex Syst. 1, 649–707
(1987)
In the case of an ideal fluid without internal Hirsch, C.: Numerical Computation of Internal and Exter-
stresses, and therefore without viscosity n ¼ 0, the nal Flows: Fundamentals of Numerical Discretization,
Eq. 32 reduces to the call Euler equation: vol. 1. Wiley, New York (1988)
Legend of Zelda Breath of the Wild and the Lens of Curiosity 1053

Hughes, T.J.R.: The Finite Element Method: Linear Static

and Dynamic Finite Element Analysis. Prentice-Hall, Learning Analytics
Inc., Englewood Cliffs (1987)
Kari, J.: Theory of cellular automata: a survey. Theor.
Comput. Sci. 334(1–3), 3–33 (2005) ▶ Design Framework for Learning to Support
Krolyi, A., Kertsz, J.: Cellular Automata Models for Gran- Industry 4.0
ular Media. In: Herrmann, H.J., Hovi, J.-P., Luding, S
(eds) Physics of Dry Granular Media, pp. 687–696.
Springer Netherlands, Dordrecht (1998). https://doi.
org/10.1007/978-94-017-2653-5_53
Rothman, D., Zaleski, S.: Lattice-Gas Cellular Automata: Learning Challenges
Simple Models of Complex Hydrodynamics. Cambridge
University Press, Cambridge (1997)
Sarkar, P.: A brief history of cellular automata. ACM ▶ Challenge-Based Learning in a Serious Global
Comput. Surv. 32(1), 80–107 (2000) Game
Shaughnessy, E.J., Katz, I.M., Schaffer, J.P.: Introduction
to Fluid Mechanics. Oxford University Press, Oxford
(2005)
Wolfram, S.: Cellular Automata and Complexity: Collected
Papers, 1st edn. AddisonWesley (1994). http://www. Learning Disabilities
stephenwolfram.com/publications/books/ca-reprint/
▶ Making Virtual Reality (VR) Accessible for
People with Disabilities

Lattice Methods

▶ Lattice Gas Cellular Automata for Fluid Learning Disability L

Simulation
▶ Computer Games for People with Disability

LBM Learning Framework

▶ Lattice Boltzmann Method for Diffusion-Reac- ▶ Design Framework for Learning to Support
tion Problems Industry 4.0
▶ Lattice Boltzmann Method for Fluid
Simulation

Legend of Zelda Breath of the

Wild and the Lens of Curiosity
Learning
Isaac Wake2 and Newton Lee1,2
1
▶ Immersive Virtual Reality Serious Games Institute for Education, Research, and
Scholarships, Los Angeles, CA, USA
2
Vincennes University, Vincennes, IN, USA

Learning Agents Synonyms

▶ Machine Learning for Computer Games Action-adventure game; open-world game

1054 Legend of Zelda Breath of the Wild and the Lens of Curiosity

Definitions known as the Korok that give the player a Korok

Seed that can be used to expand Link’s inventory by
Action-adventure game ¼ a video game that com- giving him more slots to store weapons, items,
bines elements from both action game and etc. These seeds can be found by solving puzzles or
adventure game. just flat out finding them by flipping over rocks or
Open-world game ¼ a video game featuring a shooting specific objects with Link’s bow. Giving the
virtual world in which the player can explore and player the choice to find these leads to the player
play freely without linear and structured gameplay. asking, “Will finding these benefit me or just be a
waste of time?” There is also a story behind the
world and why it has been cleansed of most of its
The Lens of Curiosity people and why a lot of the land is in ruins.
At the start of the game, the player is given
The Legend of Zelda: Breath of The Wild is a the option to head straight to the final boss of the
game where the players’ curiosity can change game, and while they can do this, it makes the game
their path and the way they think of the worlds’ way harder for the player and leaves the player with
origins. When playing Breath of the Wild players more questions than answers when they beat the
can easily find themselves climbing tall moun- final boss. They are also given the option to go to a
tains and staring into the distance wondering character named Impa that will lead them on a path
what that other mountain could hold or what this to experience the world more and dive deeper in the
oddly shaped rock may be. The way Breath of the past that Link has experienced.
Wild can bend players’ minds is very intriguing. Sure the player can beat the game right off the
In Jesse Schell’s book The Art of Game Design: bat, but what about the rest of the world? Should
A Book of Lenses, the Lens of Curiosity states that they become stronger and master the games
“to use this lens, think about the player’s true mechanics, maybe experience more of Link’s
motivations — not just the goals your game has memories? It is all up to the player. The player
set forth, but the reason the player wants to achieve cares about these questions the further they get
those goals.” (Schell, 2019) The Lens of Curiosity into the game, giving the player crucial choices
requires us to ask the following these questions: that can leave the player on the brink of survival or
on the path to destroying the evil Calamity Ganon.
1. What questions does my game put into the The player’s mind is what turns this game into
player’s mind? an experience. Players find themselves experienc-
2. What am I doing to make them care about these ing more and more of the world by solving a
questions? question they asked maybe a couple hours earlier.
3. What can I do to make them invent even more These are just some of the questions the player will
questions? ask themselves as they play Breath of the Wild.
With so much to uncover in the world and so many
At the start of the game The Legend of Zelda: things to experience, one could say there is nearly
Breath of The Wild, the main character Link an infinite amount of questions. Ultimately, Breath
awakens from a slumber and walks out of a cave to of the Wild asks the player how they will go about
find this massive and expansive world that makes the choosing a path in this expansive world. Where and
player think, “What’s in this huge world, where is how high will the player set their stakes and why
everyone, and how do I start pursuing this world?” are they setting them where they are?
Every so often the game throws something that can
catch a curious gamer’s eye; like a sparkle off the
side of a road or a bright beacon being shot into the References
sky. These moments are what really make the game’s
world so immersive and give the player a variety of Schell, J.: The art of game design: a book of lenses,
things to do. For example, there are these creatures 3rd edn. A K Peters/CRC Press (2019)
Life-Size Telepresence and Technologies 1055

Definition
Length of View
Telepresence technology refers to using a commu-
▶ 3D-Rendered Images and Their Application in nication medium to give the impression of being
the Interior Design present in an environment. Advancements in
modern communication, media, and Internet
have allowed for the widespread adoption of this
Lens technology. Using this technology can help to
save both time and money while also reducing
▶ 3D-Rendered Images and Their Application in environmental damage because it can be used as
the Interior Design an alternative to traveling to present at confer-
ences or other meetings. Telepresence systems
have the potential to make it easier for individuals
who are separated by distance to communicate
Life Simulation Game with one another.
It can be helpful in the world of business,
▶ Animal Crossing: New Horizons and Its Popu- education, and medicine, for example, allowing
larity During COVID-19 Pandemic nonexperts to perform complicated tasks under
▶ The Sims Franchise, a Retrospective of Racial the guidance of professionals. A user’s size has
Representation and Skin Tones been found to significantly influence the outcomes
of human communication aspects. If the user is
displayed in life size, there is a good chance that
Life-Size Telepresence social relationships, for example, power or domi- L
nance and persuasiveness, will be more evenly
▶ Life-Size Telepresence and Technologies distributed and natural. Therefore, this entry dis-
cusses life-size telepresence and other technolo-
gies such as videoconferencing and holographic
telepresence.
Life-Size Telepresence and
Technologies
Introduction
Fazliaty Edora Fadzli1,2 and Ajune Wanis Ismail3
1
School of Computing, Faculty of Engineering, Telepresence is a technology that allows the user
Universiti Teknologi Malaysia (UTM), Johor, to feel present at a specific location when he
Malaysia remotely transfers himself through digital repre-
2
Mixed and Virtual Environment Research Lab sentation (Fadzli et al. 2020). It uses a technique
(mivielab), ViCubeLab, Universiti Teknologi that utilizes necessary multimedia such as sounds,
Malaysia, Johor, Malaysia visions, and touch to create a sense of physical
3
Mixed and Virtual Reality Research Lab, presence at remote locations (Shen and
Vicubelab, School of Computing, Faculty of Shirmohammadi 2006). Telepresence is also
Engineering, Universiti Teknologi Malaysia, known as virtual representation, where the term
Johor Bahru, Malaysia “tele” in telepresence refers to telecommunication
technology (Kittler 2021), and the term “pres-
ence” in telepresence refers to the experience of
Synonyms existing in one place or environment (Witmer and
Singer 1998).
Digital communication; Life-size telepresence; Figure 1 shows the timeline progress of the
Telepresence telepresence system from Fadzli et al. (2020)).
1056 Life-Size Telepresence and Technologies

Life-Size Telepresence and Technologies, Fig. 1 Chronological progress of telepresence (Fadzli et al. 2020)

According to the timeline, the early phase of tele- facial expression because it has a free viewpoint
presence is the virtual space teleconferencing video combined with immersive projection tech-
using a lot of cameras to acquire the photometric nology (Fairchild et al. 2016). Besides that, it
and depth information (Fuchs et al. 1994). Three allows users to do so in such a way that users
years after, Kanade et al. (1997)) constructed a have the same transparent sense of appropriate-
virtual world from real scenes for the viewer to ness of space to activity. One of the examples of
view continuous motion in a virtual sequence. telepresence space by Beck et al. (2013) is in
However, there are discontinuities in virtual which there is a telepresence space where two
image motion. Thus, Mulligan and Daniilidis groups of users meet virtually in a life-size 3D
(2000)) proposed view-independent scene acqui- representation of the remote user. Figure 2 shows
sition for telepresence. Furthermore, Towles et al. an example of a remote user in telepresence space.
(2002)) provided the 3D telecollaboration over One of the issues of concern is telepresence
Internet2 because the view-independent scene using holoportation technology (Orts-Escolano
acquisition required transmission of rendered 3D et al. 2016) requires the user to wear HMD in
world. However, there is a hardware restriction telepresence space. The use of cumbersome hard-
and limitation on the display setup, making ware such as HMD limits face-to-face communi-
Tanikawa et al. (2005) introduce real-time and cations (Regenbrecht et al. 2004). The small
real-size transmission and presentation of a display optic in front of each eye in HMD and
human figure. In this the person’s image is cap- narrow field of view (FOV) make the device have
tured by multiple cameras, transmitted through a limitation for gaze input in face-to-face commu-
the network, and displayed on a revolving flat nications. Besides that, wearing HMD can be very
panel (Tanikawa et al. 2005). However, the issues uncomfortable for participants. According to
did not stop here. There are overlapped between Kooi et al. (Kooi and Toet 2004), binocular dis-
the viewer and the display system. Hence, Kurillo play systems such as HMD shows a different
et al. (2008) produced an immersive 3D environ- image to the left and right eye, which can affect
ment for remote collaboration and training of viewing comfort. Users feel discomfort and have
physical activities that are able to perform real- motion sickness. Figure 3 shows an example of
time 3D construction of users. telepresence space using HMD and performing
remote collaboration.
The uncanny valley (MacDorman and
Life-Size Telepresence Chattopadhyay 2016) is a phenomenon that
occurs when human replicas stimulate unintended
Telepresence space can support nonverbal com- cold, eerie emotional responses in viewers. The
munication such as body language, eye gaze, and effect was first proposed in 1970 by Masahiro
Life-Size Telepresence and Technologies 1057

Mori, who associated it with discrepancies in the

replica’s realism, with some features represented
as human and others as nonhuman (Mori 1970).
According to Jones et al. (2021), the uncanny
valley of telepresence is characterized by a sharp
drop in the continuous increase of both local and
remote users’ experience of belonging as a tele-
presence experience becomes more like a physical
presence. Yu et al. (2021) argue that overcoming
the “Uncanny Valley of Telepresence” will entail
identifying the gestalt of what is most accountable
about the comfort of belonging with others. Based
on their findings, there is a similar concept of the
uncanny valley of telepresence, which states that
the higher the simulation level of a telepresence
experience, or the more it tries to simulate or
replicate actual presence, the greater the user’s
feeling of “belonging to the space,” up to a turning
point. The user’s experience begins to decline at
this point. The illusion of “being there” raises the
expectations of their own abilities in the space, but
when those expectations are not met, they become
frustrated and believe they are not meant to be L
Life-Size Telepresence and Technologies,
Fig. 2 Telepresence space with a remote user there. In other words, they have a diminished
sense of “belonging” there.

Life-Size Telepresence and Technologies, Fig. 3 3D teleportation remote collaboration in real time
1058 Life-Size Telepresence and Technologies

Yu et al. (2021) propose a different interpretation the interaction for both sites. When the interaction
of uncanny valley related to telepresence, claiming between each side is supported, both participants
that the level of coherence may have influenced the have a better experience in communication
perception of the remote participants perceiving the through the projection of holograms based on
presented avatar. For example, MacDorman and the experiment that has been conducted where
Chattopadhyay (2016) claim that decreasing con- there are two groups of people; group A sat in
sistency in human realism in avatar images Room A and watched the projection of group B in
increases the perception of uncanny valley catego- Room B. Besides that, Pejsa et al. (2016) have
ries. A similar aspect could be argued for their claimed that life-size telepresence makes the par-
study; while the 3D virtual character avatar was ticipants appear as if they are inhabiting the same
not entirely out of consistency with the environ- space. This facilitates more natural interaction
ment, it is clear that the point cloud representation since people can fully see each other and improve
avatar was exactly fitting the style and presentation nonverbal signals such as gaze, body language,
of the environment reconstruction, as the same sys- and gestures. The life-size virtual copies of remote
tem was used for the avatar and for the environ- participants are projected into physical space, as
ment. The result shows that in order to choose the shown in Fig. 4.
representation of the user, we need to consider the
way the environment was represented as well.
Telepresence can be projected in many sizes, Comparison Between Telepresence and
either small or large. In order to achieve a great Videoconference
augmented reality collaboration environment,
life-size telepresence can be considered the per- This section shows the difference between
fect size. Life-size telepresence has the same mea- audioconferencing, videoconferencing, and tele-
surement as a real-life size. This will help the presence. Videoconferencing began in 1964 when
participants to feel more realistic and feel pres- AT&T unveiled the picturephone at the New York
ence in the remote location as according to Pejsa World’s Fair, the world’s first videoconferencing
et al. (2016). endpoint. In the year 2000, telepresence systems
Based on Mark and DeFlorio (2001) research, hit the market, bringing a more realistic approach
the large ratio of life-size telepresence supports to face-to-face meetings. However, they are not

Life-Size Telepresence and Technologies, Fig. 4 Augmented image in life-size telepresence

Life-Size Telepresence and Technologies 1059

concerned with price but rather with providing Technologies

customers with a high-quality, fully immersive
experience (Pulido 2011). This shows that video- This section will discover the technologies used
conference exists before telepresence and that for telepresence.
telepresence is the advanced technology where it
produces a better quality of video and facial Videoconferencing
expression of the user. In comparison, videocon- Telepresence has brought a revolution in the com-
ferencing produces low to medium video resolu- munication industry. The advancement of tele-
tion and displays in TV quality. presence technology has changed the way we
In recent years, real-time telepresence has are able to communicate and connect with each
attracted more and more attention. However, this other. One of the advancements of telepresence
technology has several issues that cannot be technology is videoconference technology. Tele-
avoided. One of the issues is that participants presence technology is a system that provides a
must travel to facilities that are equipped and life-size, high-definition video and stereo-quality
staffed to produce telepresence videoconferences audio, and make people experience face-to-face
(Fuchs et al. 2014). This is an issue for the user interaction at a distance (Pulido 2011). Figure 5
that is traveling. They need to find a place that has shows an example of videoconference technol-
the equipment of the telepresence or prepare the ogy, and this type of technology is very common
set of the telepresence equipment at their place. in a meeting room.
Although the risk of disruption or failure is
extremely low, it is always possible over the Internet Holographic
(Takamizawa et al. 2004). Coverage of the Internet The next technology is holographic telepresence
depends on the location of the user. If the user loca- technology. Holographic technology provides full L
tion has low coverage of the Internet, the possibility motion of projection, as well as realistic and 3D
for failure over the Internet is high. Users cannot images in real-time. As stated by Luevano et al.
perform telepresence without the Internet. Disruption (2019), a holographic telepresence system captures
Internet also can occur during telepresence. The com- and compresses images of real, remote people or
parison table is produced by Davis and Kelly (2008). surrounding objects and transfers images together
Table 1 lists a comparison of three terminologies with the sound across the Internet. Holographic use
based on their characteristics, videoconferencing, the method of recording patterns of light and pro-
HD videoconferencing, and telepresence. jecting the pattern in a three-dimensional image

Life-Size Telepresence and Technologies, Table 1 Comparison of the three terminologies

Characteristic Videoconferencing HD videoconferencing Telepresence
Cost Free to $100 s $1000s $10 k to $100ks
Picture quality Poor to good Good to excellent Excellent
Bandwidth requirement 100 s of kilobits 100 s of kilobits per second to 1 Mbps to 50 Mbps+
per endpoint per second megabits per second
No. of participants Usually two Two or more Many
Interoperability with Common Limited Very limited
other systems
Camera position Fixed or variable Fixed or variable Usually fixed
Room footprint Variable Variable Fixed and requires room
preparation
Platform Software and Usually hardware Hardware
hardware
1060 Life-Size Telepresence and Technologies

called hologram (Elmorshidy 2010), and Fig. 6 technology has the potential for remote individ-
shows the example of holographic projection. uals to view each other’s environment and also
Other than that, the projector camera is also communicate by overlaying the user’s virtual
another type of telepresence technology. This copy image in new ways into the remote location,

Life-Size Telepresence
and Technologies,
Fig. 5 Example of
videoconferencing

Life-Size Telepresence and Technologies, Fig. 6 Holographic telepresence using projector-based images
Life-Size Telepresence and Technologies 1061

Life-Size Telepresence and Technologies, Table 2 Comparison of the three terminologies

Type Advantages Disadvantages
High-definition High resolution Flat images
3D display High-resolution Narrow viewing angles
Stereoscopic Require glasses to display
Not in 3D imagery
3D holographic display “Life-like” picture Require a large amount of processing
Volumetric 3D display Constraint by the size of holographic material
Interactivity Production cost is expensive

without wearing any specialized device on the Jones, B., Zhang, Y., Wong, P.N., Rintel, S.: Belonging
user (Pinhanez and Pingali 2004). Projector- there: VROOM-ing into the Uncanny Valley of XR
telepresence. Proc. ACM Hum.-Comput. Interact.
camera-based technology by Pinhanez et al. 5(CSCW1), 1–31 (2021)
(Pinhanez and Pingali 2004) is much more simple Kanade, T., Rander, P., Narayanan, P.J.: Virtualized reality:
than holographic. The cost required for applica- Constructing virtual worlds from real scenes. IEEE
tion setup is also low and suitable for telepresence. Multimedia. 4(1), 34–47 (1997)
Kittler, W.: Tele (Τῆlε). In: Information, pp. 199–212.
Table 2 shows the advantages and disadvan- Columbia University Press (2021)
tages between high-definition, 3D displays, and Kooi, F.L., Toet, A.: Visual comfort of binocular and 3D
3D holographic display. Based on the table, 3D displays. Displays. 25(2–3), 99–108 (2004)
holographic displays have a lot of advantages Kurillo, G., Bajcsy, R., Nahrsted, K., Kreylos, O.:
Immersive 3d environment for remote collaboration
compared to other technology. However, the pro- and training of physical activities. In: 2008 IEEE
duction cost to set up a holographic is expensive. Virtual Reality Conference, pp. 269–270. IEEE
(2008) L
Luevano, L., de Lara, E.L., Quintero, H.: Professor Avatar
holographic telepresence model. In: Holographic Mate-
References rials and Applications, vol. 91 (2019)
MacDorman, K.F., Chattopadhyay, D.: Reducing consis-
Beck, S., Kunert, A., Kulik, A., Froehlich, B.: Immersive tency in human realism increases the uncanny valley
group-to-group telepresence. IEEE Trans. Vis. effect; Increasing category uncertainty does not. Cog-
Comput. Graph. 19(4), 616–625 (2013) nition. 146, 190–205 (2016)
Davis, A.W., Kelly, E.B.: Telepresence Mark, G., DeFlorio, P. (2001). An experiment using life-
vs. Videoconferencing: Resolving the cost/benefit size hdtv. In: Proceedings of the IEEE Workshop on
conundrum (2008) http://www.wainhouse.com/files/ Advanced Collaborative Environments. Mori, M.:
papers/wr-tpvc-cost.pdf. [accessed 2022-02-18] Bukimi no tani [the uncanny valley]. Energy. 7, 33–
Elmorshidy, A.: Holographic projection technology: The 35 (1970)
world is changing. arXiv preprint arXiv:1006. Mori, M.: Bukimi no tani [the uncanny valley]. Energy. 7,
0846 (2010) 33–35 (1970)
Fadzli, F.E., Ismail, A.W., Aladin, M.Y.F., Othman, N.Z. Mulligan, J., Daniilidis, K.: View-independent scene
S.: A review of mixed reality telepresence. In: IOP acquisition for tele-presence. In: Proceedings IEEE
Conference Series: Materials Science and Engineering, and ACM International Symposium on Augmented
vol. 864(1), p. 012081. IOP Publishing (2020) Reality (ISAR 2000), pp. 105–108. IEEE (2000)
Fairchild, A.J., Campion, S.P., García, A.S., Wolff, R., Orts-Escolano, S., Rhemann, C., Fanello, S., Chang, W.,
Fernando, T., Roberts, D.J.: A mixed reality tele- Kowdle, A., Degtyarev, Y., . . . Izadi, S.:
presence system for collaborative space operation. Holoportation: Virtual 3d teleportation in real-time.
IEEE Trans. Circuits Syst. Video Technol. 27(4), In Proceedings of the 29th annual symposium on user
814–827 (2016) interface software and technology (pp. 741–754)
Fuchs, H., Bishop, G., Arthur, K., McMillan, L., Bajcsy, (October 2016)
R., Lee, S., . . . Kanade, T.: Virtual space teleconferenc- Pejsa, T., Kantor, J., Benko, H., Ofek, E., Wilson, A.:
ing using a sea of cameras. In: Proceedings of the First Room2room: Enabling life-size telepresence in a pro-
International Conference on Medical Robotics and jected augmented reality environment. In: Proceedings
Computer Assisted Surgery, vol. 26 (September 1994) of the 19th ACM Conference on Computer-Supported
Fuchs, H., State, A., Bazin, J.C.: Immersive 3d tele- Cooperative Work & Social Computing,
presence. Computer. 47(7), 46–52 (2014) pp. 1716–1725 (2016)
1062 Lifestyle Brand

Pinhanez, C., Pingali, G.: Projector-camera systems for

telepresence. In: Proceedings of the 2004 ACM Linear Solving of Dupin
SIGMM Workshop on Effective Telepresence,
pp. 63–66 (October 2004) Problem
Pulido, R.: Telepresence (Dissertation) (2011). Retrieved
from http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva- ▶ Minkowski-Lorentz Spaces Applications: Res-
74449 olution of Apollonius and Dupin Problems
Regenbrecht, H., Lum, T., Kohler, P., Ott, C., Wagner, M.,
Wilke, W., Mueller, E.: Using augmented virtuality for
remote collaboration. Presence. 13(3), 338–354
(2004)
Shen, X., Shirmohammadi, S.: Telepresence. In: Furht,
B. (ed.) Encyclopedia of Multimedia. Springer, Boston
(2006)
Live Texture Mapping in
Takamizawa, N., Ichisawa, H., Murayama, Y., Smith, J., Handheld Augmented Reality
Kumquat, J.P.: Use of telepresence in informal broad- Coloring Book
casting over the Internet. In: Proceedings of the 2004
ACM SIGMM workshop on Effective telepresence,
pp. 12–15 (October 2004)
Muhammad Nur Affendy Nor’a1 and
Tanikawa, T., Suzuki, Y., Hirota, K., Hirose, M.: Ajune Wanis Ismail2
1
Real world video avatar: Real-time and realsize trans- Mixed and Virtual Reality Research Lab,
mission and presentation of human figure Interna- Vicubelab, Universiti Teknologi Malaysia, Johor
tional Conference on Augmented tele-existence,
pp. 112–118 (2005)
Bahru, Malaysia
2
Towles, H., Chen, W.C., Yang, R., Kum, S.U., Kelshikar, Mixed and Virtual Reality Research Lab,
H.F.N., Mulligan, J., . . . Lanier, J.: 3d tele- Vicubelab, School of Computing, Faculty of
collaboration over internet2. In: International Work- Engineering, Universiti Teknologi Malaysia,
shop on Immersive Telepresence, Juan Les Pins
(2002)
Johor Bahru, Malaysia
Witmer, B.G., Singer, M.J.: Measuring presence in virtual
environments: A presence questionnaire. Presence.
7(3), 225–240 (1998) Synonyms
Yu, K., Gorbachev, G., Eck, U., Pankratz, F., Navab, N.,
Roth, D.: Avatars for teleconsultation: Effects of avatar
embodiment techniques on user perception in 3D Augmented reality; Live texturing; Texture
asymmetric telepresence. IEEE Trans. Vis. Comput. mapping
Graph. 27(11), 4129–4139 (2021)

Definition
Lifestyle Brand
Nowadays, many coloring books have been
▶ Professional Call of Duty Player Matthew using augmented reality (AR) as the enabler
“Nadeshot” Haag: An e-Sports Case Study to display virtual objects in the real world.
Real-time texture mapping is an essential ele-
ment that brings AR coloring more advanced.
The augmented 3D character is textured
according to the video stream coloring. Act
Linear Solving of Apollonius as a guideline to deliver real-time texturing
problem mapping for augmented reality coloring appli-
cations. This entry explains the live texturing
▶ Minkowski-Lorentz Spaces Applications: Res- process in AR, the live texture mapping for the
olution of Apollonius and Dupin Problems AR coloring book.
Live Texture Mapping in Handheld Augmented Reality Coloring Book 1063

Introduction AR environment after they color it. The Korea-

based company AIARA also produces an AR
AR is growing not just in entertainment but also in coloring book that combines the AR and Internet
education. AR books are a product that uses AR of Things (IoT), allowing the merge of analogue
technologies and becomes new development of and digital media.
entertainment and educational models.
An interactive augmented coloring book intro-
duced by Clark et al. (2012) is additionally one of Live Texture Mapping in AR
the real leaps forward of AR education. The tech-
nology features the significance of direct intuitive- Earlier work has been found in Clark et al. (2012)
ness with the content by enabling the user to when they create scenes in 3D and textured
control the result AR 3D content with their shades models that refer to user creation. After that,
of shading, giving more impact on the learning actual book pages will become the base for plac-
strategy since interactivity promotes learning by ing the 3D content in the real world. With this, the
activating specific cognitive processes (Nor’a user can feel a 3D experience with their content.
et al. 2019). Figure 1 shows the two examples of the AR char-
AR books can become physical or digital cop- acter as 3D content on top of the page. In 2015, the
ies with both text and illustrations. Technology enhancement suggested a live texturing process,
usage, such as smartphones or tablets, can display and it has been defined in Magnenat et al. (2015)
the 3D content on top of the AR book. This as illustrated in Fig. 2. As described in Zhao et al.
augmentation can be achievable once the camera (2017), it presented a texture process that gener-
points toward the page of the AR Book. Usually, ates a texture map for 3D augmented reality char-
the AR Book will provide the mobile application acters from 2D colored drawings using a lookup L
to be downloaded by the smartphone that allows map. Considering the movement of the mobile
the application to track the page and display the device and drawing, they have given an efficient
content based on the page that we track. A video method to track the drawing surface.
clip, audio clip, image, or even 3D object can Figure 2 shows the adapted pipeline defined in
become one of the AR content that the AR Book Magnenat et al. (2015) of the static content crea-
can display. tion and live surface tracking, texturing, and ren-
HITLab NZ. ColAR has researched the idea of dering pipelines. The pipeline has stated that it
the AR book for more than 10 years which allows needs to be run only once. A live pipeline tracks
users to color the book page and view the 3D the drawing and overlays the augmented character
model in the real world with colored textures on top of it, with a texture created from the draw-
(Cho et al. 2016). Usually, the AR color book ing in real-time using the lookup map. As in
requires a paper book and smartphone application Fig. 2, in the Live Pipeline box, the camera
to use with the AR book. The user needs to color image captures a set of colored drawing pages.
the paper book first before the texture is mapped Five boxes have been running through, and the
onto a 3D character and displayed in an AR envi- following points described all the processes
ronment. With the AR coloring book, the children involved:
can view 3D animation of the character and
painted color when they use a smartphone to • Image Processing. The input image converts
take real-time video. the colored page into grayscale and uses image
Quiver created a ColAR application that processing to produce drawing lines. The black
already sells coloring page through the online line has been produced by applying adaptive
market, and users can download the application thresholding. Figure 3 shows the process
from the apps store to view 3D characters in the involved.
1064 Live Texture Mapping in Handheld Augmented Reality Coloring Book

Live Texture Mapping in Handheld Augmented Reality Coloring Book, Fig. 1 3D content on top of the page

Live Texture Mapping in Handheld Augmented Reality Coloring Book, Fig. 2 Live texturing pipeline in AR
Color Book

• Template selection. It shows the content pipe- what has been drawn. Here it is called a real-
line where the UV mapping has been time painting.
implemented to quickly patch the image into • Surface tracking. The template that has been
characters so their textures will correspond to selected needs to retrieve the reference images,
Live Texture Mapping in Handheld Augmented Reality Coloring Book 1065

Live Texture Mapping in Handheld Augmented Reality Coloring Book, Fig. 3 Image processing: (a) Colored
image. (b) Marker texture color for image processing. (c) 3D AR object with colored texture

and this surface tracking will align the camera onward, this will be converted into various local
with the captured image to render the reference or global priors to have an inpainted image as
image. physically logical and visually satisfying as
• Texture creation and mesh rendering. Used expected (Guillemot and Le Meur 2013). The L
when to open the camera, there will be a real diffusion-based inpainting method denotes the
live environment as a background image for image’s unknown part and fills in with the source
AR scene, and pose estimation to render the 3D from the known region. These methods are gen-
scene with the live texture mapping in real- erally relevant for accomplishing straight lines
time. and curves and inpainting small areas conceivable
(Guillemot and Le Meur 2013).
The term diffusion comes from using the
Image Inpainting Technique smoothness priors, which come from proliferating
nearby data by analogue with physical marvels
Image inpainting is defined as a process to restore like warmth proliferation in physical structures
the damaged area of the image. Several kinds of and smoothness constraints. Inpainting utilizing
research have been done to reconstruct images dispersion easily spreads nearby picture structures
from message overlay or scratches, loss conceal- from the outside to the inside of the gap, “emulat-
ment in an image, and occlusion problem when ing” the motion of expert painting restorators
rendered after the camera captured the image (Guillemot and Le Meur 2013). It then uses the
(Casas et al. 2017). Marker hiding has been stud- considered data to assume smoothness limitation
ied to hide markers when the camera tracked the and iteratively regularized and creates a smoother
marker (Korkalo et al. 2010). image in an endless array. However, diffusion-
In image inpainting, there are still some prob- based is not suitable for the recovery texture of
lems concerning ill-posed inverse issues with no large areas, and it also tends to be a blur. The
well-defined unique solution. It is necessary to inpainting process is used only in the first frame
define image priors to solve the problem. This to generate the texture instead of inpainting for
method assumes the pixel from the known and every frame to ensure consistency, and then the
unknown parts to have the same measurable prop- generated texture is overlaid on the marker area
erties or geometrical complexes. From that point according to the camera pose. Multisensory has
1066 Live Texturing

been recently explored instead of the diffusion-

based inpainting method for AR drawing Localization
(Meesuwan 2021). However, today, an advanced
diffusion-based inpainting method has been ▶ Cross-cultural Game Studies
explored for real-time inpainting 4 K color images ▶ Spatial Audio and Sound Design in the Context
on contemporary GPUs (Kämper and Weickert of Games and Multimedia
2021).

References Locomotion
Casas, L., Kosek, M., Mitchell, K.: Props alive: ▶ Locomotion and Human Tracking in
A framework for augmented reality stop motion ani-
mation. In: 2017 IEEE 10th Workshop on Software
Healthcare Robots
Engineering and Architectures for Realtime Interactive ▶ Navigation Artificial Intelligence
Systems (SEARIS), pp. 1–4. IEEE (2017)
Cho, K., Kim, H., Lee, Y.: Augmented reality coloring
book with transitional user interface. Indian J. Sci.
Technol. 9(20), 1–5 (2016)
Clark, A., Dunser, A., Grasset, R.: An interactive aug- Locomotion and Human
mented reality coloring book. In: IEEE Symposium Tracking in Healthcare Robots
on 3D User Interface (March 2012)
Guillemot, C., Le Meur, O.: Image inpainting: Overview
and recent advances. IEEE Signal Process. Mag. 31(1),
Patrick C. K. Hung, Inon Wiratsin and
127–144 (2013) Nannapat Meemongkolkiat
Kämper, N., Weickert, J.: Domain decomposition algo- Faculty of Business and IT, Ontario Tech
rithms for real-time homogeneous diffusion inpainting University, Oshawa, ON, Canada
in 4K. arXiv preprint arXiv:2110.03946 (2021)
Korkalo, O., Aittala, M., Siltanen, S.: Light-weight marker
hiding for augmented reality. In: 2010 IEEE Interna-
tional Symposium on Mixed and Augmented Reality, Synonyms
pp. 247–248. IEEE (2010)
Magnenat, S., Ngo, D.T., Zünd, F., Ryffel, M., Noris, G.,
Rothlin, G., ... Sumner, R.W.: Live texturing of aug-
Healthcare robots; Human tracking; Locomotion;
mented reality characters from colored drawings. IEEE Robot navigation; Route planning
Trans. Vis. Comput. Graph. 21(11), 1201–1210 (2015)
Meesuwan, W.: The development of a drawing and color-
ing application by augmented reality technology based
on the concepts of multisensory. J. Educ. Naresuan
Definition
Univ. 23(2), 295–309 (2021)
Nor’a, M.N.A., Ismail, A.W., Aladin, M.Y.F.: Interactive Computer Computer vision is the Artificial
augmented reality pop-up book with natural gesture Vision Intelligence (AI) system
interaction for handheld. In: Lee, N. (ed.) Encyclopedia
of Computer Graphics and Games. Springer, Cham
incorporated with other scientific
(2019). https://doi.org/10.1007/978-3-319-08234-9_ fields, such as signal processing
365-1 and neurobiology, to interpret and
Zhao, H., Huang, P., Yao, J.: Texturing of augmented gain high-level understanding from
reality character based on colored drawing. In: 2017
IEEE Virtual Reality (VR), pp. 355–356. IEEE (2017)
digital images or videos.
Healthcare Healthcare robot refers to a
Robot machine programmed by a
computer capable of assisting
Live Texturing humans in the medical field. In
addition, it is able to provide care
▶ Live Texture Mapping in Handheld Augmented and support to disabled patients
Reality Coloring Book and the elderly.
Locomotion and Human Tracking in Healthcare Robots 1067

Medicine Medicine dispenser refers to the record, can be integrated into the healthcare robot
dispenser item that releases medications at a for enhancing its capabilities.
specific time and assists elderly Beyond the conventional purview of surgical
persons and those who may have and rehabilitative robots, robots perform different
impaired abilities to comply with activities and purposes in the medical/health and
their recommended medication. social care sectors. In this scenario, the robot may
navigate to search for patients inside a specific
Introduction location. However, inefficient navigation in a
socio-physical environment may cause the search
A platform with multi-terrain navigation capabil- to be delayed, identifying emergencies. Thus, com-
ities is necessary for a healthcare robot to move puter vision techniques can aid the navigation sys-
through varied geographical landscapes. The tem in healthcare robots, such as placing the
locomotion feature plays a significant role. This patient, evading obstacles, and detecting dangerous
feature allows the robot to navigate the areas, objects. For example, a visual-based robot naviga-
especially inside a house with many rooms. Fur- tion method for the catadioptric optical system was
thermore, another significant function in proposed by Winters et al. (2000). They presented a
healthcare is human tracking. A healthcare robot method for converting images from a catadioptric
should have both locomotion and human tracking camera to bird’s-eye view. These images are used
abilities to navigate an area (e.g., a house) and to control the orientation of the robot movement. In
look for the target person or patient when it is the addition, a teaching and replay approach was pro-
right time for medication. posed to train the robot to navigate indoor and
outdoor environments (Chen and Birchfield
2006). In the teaching phase, the robot is manually L
Motivation and Background guided along the desired path, and then in the
playback phase the robot follows that course inde-
The World Health Organization (WHO) reported pendently. This method allows the robot to auto-
that the number of disabled persons is dramati- matically extract the feature points along the
cally increasing (WHO 2011). It is related to trajectory during the training phase. The advantage
demographic changes and a rise in chronic health of this technique is it does not require any calibra-
problems, among other factors. Therefore, scaling tion on the robot’s camera. However, the change in
up disability services in primary healthcare, par- the environment can affect the robot navigation
ticularly rehabilitation therapies, is urgently process since the feature points are also changed.
required. These disabled persons require contin- Obstacle avoidance is another issue for mobile
ual supervision to complete simple tasks, such as robots. The neuro-fuzzy technique is used to con-
eating and drinking. Moreover, a constant moni- struct an intelligent obstacle-avoidance approach
toring process is also necessary to avoid any to autonomous navigation of a mobile robot in
emergencies. As a result, it is prudent to encour- unfamiliar surroundings (Wang et al. 2004). The
age technology that aids in detecting and robot is integrated with four infrared sensors to
responding to emergency circumstances. Many detect the distance from the obstacles to the robot
camera monitoring systems for patient behavior itself. The neuro-fuzzy model then processes this
detection are available in the market today. How- distance information to determine the optimal
ever, these systems could invade in-house privacy, actions to avoid the obstacles. Estimating slip
but they also lack the function of aiding patients. angle is also a prominent issue for wheel-based
Another solution is to create mobile healthcare robots since there are some situations where the
robots to assist the patients. Healthcare robots robots need to travel through rough terrain envi-
can track the behavior of the targeted patient. ronments. Hough transform incorporated with
The auxiliary programs, such as time scheduling, fuzzy logic was applied to estimate the angle of
medicine detection, and digital patient’s medical inclination of the wheel trace with respect to the
1068 Locomotion and Human Tracking in Healthcare Robots

vehicle reference frame (Reina et al. 2008). It between corresponding picture points’ horizontal
detects the deviation in the wheel trace in real coordinates from the acquired image pair (Kadri
time. This deviation implies that the terrain has a et al. 2019). The output disparity map presents the
change of inclination angle. estimation of the position of features in the image
pair. Then, the disparity map will be processed by
a segmentation algorithm for distinguishing
Structure of Learning System objects that appear in the image (Yoon et al.
2017). The disparity map segmentation image
The architecture of the locomotion and human represents different objects at different distances
tracking system is shown in Fig. 1. to the robot’s cameras. Next, feature extraction is
The architecture consists of four modules: applied to the disparity map segmentation to
image acquisition module, vision module, deci- extract 2D and 3D features (Tiwari et al. 2013).
sion module, and robot control module. The These features are then used to object detection
details are presented in the following subsections. algorithms. Finally, the object detection algorithm
will identify objects in the image pair (Redmon
et al. 2016).
Image Acquisition Module

The image acquisition module controls the robot Decision Module

cameras. Traditionally, two cameras placed hori-
zontally apart are used to capture two different The decision module takes the information from
perspectives of a scene, analogous to human bin- sensors, including object information and distance
ocular vision. This module synchronizes video from the robot, to decide which action to do next.
frames from two cameras, removes noisy signals, The reinforcement learning model can be used
and sends the set of digital images to the vision to implement the decision module (Sutton and
module. Barto 2018).

Vision Module Robot Control Module

The vision module is the process for extracting 3D The robot control module controls the robot’s
information from the set of digital images taken hardware to perform the action given by the deci-
from two cameras. The process is illustrated in sion module. Note that the robot can contain more
Fig. 2. than one control module for various purposes
Block matching algorithm is applied to gener- (Brooks 1986). The list of control modules essen-
ate a disparity map that encodes the difference tial for healthcare robots are as follows:

Locomotion and Human Tracking in Healthcare Robots, Fig. 1 System architecture

Locomotion and Human Tracking in Healthcare Robots 1069

Locomotion and Human Tracking in Healthcare Robots, Fig. 2 Vision module process L

1. Cruiser module (Basu and Redi 2004) Cross-References

This module controls the robot’s move-
ment. It decides the voltage level that will be ▶ Healthcare Robots with Islamic Practices
fed to the motors to control the speed and the
direction.
2. Linear motion module for robot hand move- References
ment (Catalano et al. 2012)
This module controls the simple action of Basu, P., Redi, J.: Movement control algorithms for reali-
zation of fault-tolerant ad hoc robot networks. IEEE
the robot’s arms, which are up, down, left, and
Netw. 18(4), 36–44 (2004)
right. Brooks, R.: A robust layered control system for a mobile
3. Rotation motion module for robot hand move- robot. IEEE J Robotics Automat. 2(1), 14–23 (1986)
ment (Osswald and Wörn 2001) Catalano, M.G., Grioli, G., Serio, A., Farnioli, E., Piazza,
C., Bicchi, A.: Adaptive synergies for a humanoid
This module controls the complex move-
robot hand. In: The 2012 12th IEEE-RAS International
ment of the robot’s arms, which are grabbing Conference on Humanoid Robots (Humanoids 2012),
and rotating the wrists. pp. 7–14 (2012)
Chen, Z., Birchfield, S.T.: Qualitative vision-based mobile
robot navigation. In: The 2006 IEEE International Con-
Furthermore, the control modules need to be ference on Robotics and Automation (ICRA 2006),
dynamic, robust, stable, and responsive since the pp. 2686–2692 (2006)
Kadri, I., Dauphin, G., Mokraoui, A., Lachiri, Z.: Stereo-
environment varies from time to time. Finally, it scopic image coding performance using disparity-
should be fast enough to interact within a compensated block matching algorithm. In: The 2019
suitable time. Signal Processing: Algorithms, Architectures,
1070 Locomotion in Virtual Reality Video Games

Arrangements, and Applications (SPA), Poznan, Definitions

Poland (2019) 5 pages
Osswald, D., Wörn, H.: Mechanical system and control
system of a dexterous robot hand. In: Proceedings of Locomotion in a virtual reality video game is the
the IEEE-RAS International Conference on Humanoid travel in virtual world in order to move to a desired
Robots, Tokyo, Japan (2001) location.
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You
only look once: Unified, real-time object detection.
In: The IEEE Conference on Computer Vision and
Pattern Recognition, Las Vegas, NV, USA, Introduction
pp. 779–788 (2016)
Reina, G., Ishigami, G., Nagatani, K., Yoshida, K.: Vision- Most of the virtual reality (VR) systems, including
based estimation of slip angle for mobile robots and
planetary rovers. In: The 2008 IEEE International Con- virtual reality video games, often require harmo-
ference on Robotics and Automation, niously designed components such as audio and
pp. 486–491 (2008) visual elements, task design, virtual worlds, as
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An well as interaction and locomotion techniques.
Introduction. MIT Press (2018)
Tiwari, A., Goswami, A.K., Saraswat, M.: Feature extrac- Locomotion is among the most important and
tion for object recognition and image classification. Int. very commonly used tasks in 3D virtual reality
J. Eng. Res. Technol. (IJERT). 2(10), games (Bowman et al. 2004). Small position and
1238–1246 (2013) rotation changes of the virtual viewpoint can be
Wang, X., Yang, S.X., Meng, M.Q.: Intelligent obstacle
avoidance for an autonomous mobile robot. In: The performed by head movements in immersive vir-
Fifth World Congress on Intelligent Control and Auto- tual reality systems that have head tracking capa-
mation (IEEE Cat. No. 04EX788), vol. 5, bilities. Such systems often use head-mounted
pp. 4656–4660 (2004) displays to present the virtual world to the player.
WHO: World report on disability. World Health Organiza-
tion (2011). Retrieved from https://www.who.int/ However, if the game requires a larger amount of
disabilities/world_report/2011/report.pdf travel than the real-world area, then a different
Winters, N., Gaspar, J.A., Lacey, G., Santos-Victor, J.: locomotion technique needs to be used.
Omni-directional vision for robot navigation. In: The Recently in the late 2010s, many new genera-
IEEE Workshop on Omnidirectional Vision (Cat.
No. PR00704), pp. 21–28 (2000) tion, head-mounted displays became available,
Yoon, J.S., Rameau, F., Kim, J., Lee, S., Shin, S., which are characterized as being highly
Kweon, I.S.: Pixel-level matching for video object immersive and affordable. HTC Vive, Oculus
segmentation using convolutional neural networks. Rift, and Sony PlayStation VR are among the
In: The 2017 IEEE International Conference on
Computer Vision (ICCV), Venice, Italy, most prevalent ones of these new generation
pp. 2186–2195 (2017) head-mounted displays. Nowadays, gamers can
also use their high-end smart phones as displays
for Samsung Gear VR and Google Daydream VR
headsets. Although immersive virtual environ-
ments that use head-mounted displays are usually
Locomotion in Virtual Reality suitable for being explored on foot, there is
Video Games a common major limitation of these systems.
They either cannot track the position of the
Evren Bozgeyikli user’s head or the position tracking is limited to
School of Information, University of Arizona, a small area. One approach can be designing the
Tucson, AZ, USA virtual environment not larger than the tracked
physical area and implementing a 1:1 mapping
for the virtual position of the player, based on
Synonyms the real-world position of the user. Although
there are some well-known VR game examples
Interaction techniques; Locomotion techniques; that use this technique, such as the game titled
Virtual reality systems “Job Simulator”; this is a significantly major
Locomotion in Virtual Reality Video Games 1071

limitation for game design. For VR games or “Freeze-Turn,” “2:1-Turn” (Williams et al.
applications with larger virtual environments, 2007), and “virtual distracters” (Kohli et al.
this 1:1 mapping technique would not work, 2005; Peck et al. 2010). The aim of these accom-
since users will eventually go outside the real panying methods is to make the user walk
tracking space. To overcome this problem, several towards the center of the tracked area.
different locomotion techniques have been pro- Another approach that aimed to enable real
posed in the literature. This entry reviews VR walking in virtual environments was known
locomotion techniques that have been widely as “environment change.” In this approach, the
researched or used in commercial VR systems. virtual environment’s architecture was changed
according to the user’s navigation. In one tech-
nique, orientation of virtual doors was changed
Virtual Reality Locomotion Techniques (Suma et al. 2011). In another technique, self-
overlapping virtual spaces were used, which
Algorithm-Based Techniques were referred to as “impossible spaces” (Suma
These locomotion techniques are constructed on et al. 2012) and “flexible spaces” (Vasylevska
algorithms without depending on any specific et al. 2013). In parallel lines, some portal door
hardware. This category includes real walking techniques emerged to minimize the need for real
(redirected walking) and gesture-based locomo- walking: “Arch-Explore” (Bruder et al. 2009) and
tion techniques (walking-in-place and flying/ barrier tape (Freitag et al. 2014).
leaning).
Gesture-Based Techniques
Real Walking These techniques aim to utilize body gestures,
Although it is shown to be the most presence- instead of making users really walk. Walking-in- L
enhancing (Usoh et al. 1999) and less cognitively place, flying, and leaning are among the most
demanding (Marsh et al. 2013; Ruddle et al. commonly used ones. In walking-in-place, users
2011) locomotion technique, the real walking perform marching gesture but do not actually
technique has the physical limitation of the track- move in any direction in real-world (Slater et al.
ing area (Whitton et al. 2005), which can make it 1995a, b). Important attributes of walking-in-
impossible to utilize in small-sized physical place method are as follows: step detection ges-
spaces. In order to work around this limitation, ture (Feasel et al. 2008; Nilsson et al. 2013a, b;
redirected walking was introduced. Redirection Templeman et al. 1999; Terziman et al. 2010;
can be described as altering the visual cues in Wendt et al. 2010); step detection device (Feasel
virtual reality to keep the users inside the tracked et al. 2008; Kim et al. 2012; Wendt et al. 2010;
area (Razzaque et al. 2001). This technique Williams et al. 2011; Zielinski et al. 2011); start
makes it possible to map large virtual environ- and stop latency (Feasel et al. 2008); smooth
ments to smaller tracked areas. In this technique, speed control (Whitton and Peck 2013); visual
gains are applied to the locomotion data to alter gain range, which was reported as 1.65–2.44
user’s visual cues. Keeping these gain values (Nilsson et al. 2014).
within the unnoticeable range is critical in order Another gesture-based locomotion technique
not to break user’s immersion. The range for which have been commonly used in VR is flying
unnoticeable gain values were reported as fol- (Guy et al. 2015; Robinett and Holloway 1992;
lows: Translation 14% and 26%; Rotation Ware and Osborne 1990). In this flying method,
20% and 49%; and Curvature Radius 22 m user automatically moves the virtual view through
(Steinicke et al. 2008, 2009, 2010). Although some gestures. This technique offers easy imple-
the virtual viewpoint of the user is altered, they mentation and ease of use; however, it was
can still reach to the edge of the physical tracked reported to be less realistic and less presence-
area. For such cases, accompanying methods enhancing as compared to other locomotion tech-
were developed such as “Freeze Backup,” niques (Usoh et al. 1999). The main issues
1072 Locomotion in Virtual Reality Video Games

associated with flying were low degree of control Sibert 1996; Zhixin and Lindeman 2015). Main
in speed and low resemblance to real life walking advantages are affordability in terms of price,
interaction. A similar locomotion technique is simplicity, and familiarity from real-life experi-
leaning, in which users utilize their torso to con- ences. Some studies resulted in findings of joy-
trol speed and rotation of locomotion. Different stick providing better user experience in terms of
methods or tools have been utilized for detecting locomotion as compared to other techniques
shifts in body weight such as “Nintendo Wii (Cirio et al. 2009; Nabiyouni et al. 2015), whereas
Balance Board” (Valkov et al. 2010) and other studies reported findings that indicate
“Human Joystick” (Harris et al. 2014). Leaning joystick provided worse user experience in terms
method was reported to provide higher degree of of locomotion (Peck et al. 2011, 2012; Riecke
control; however, it requires more balance and et al. 2010; Ruddle and Lessels 2006; Whitton
body control skills. A recently popularized VR et al. 2005).
locomotion technique is referred as “Point and
Teleport” that enables users to point to a place Mobile Tools
and instantly get teleported there (Bozgeyikli Mobile tools are those that can be worn on user’s
et al. 2016). body or moved around. Due to their complex
nature, these tools are not widely available for
Tool-Based Techniques typical consumers; however, these have been
These locomotion techniques rely on tools utilized in some experimental studies. For virtual
that give accurate position and rotation data. reality locomotion, wearable tools such as “Cyber
These tools can be stationary (walkers and Boots” that included pressure foot sensors (Insook
standard controllers) or mobile (wearables and Ricci 1997); “Waraji” that included a sandal
and robots). with rotary sensors (Barrera et al. 2004); a shoe
with six pressure sensors (Matthies et al. 2013);
Stationary Tools and “Powered Shoes” that included roller skates
Stationary tools are fixed to a surface, which is (Iwata et al. 2006) are used. In an experimental
usually the ground, and users climb on them for study referred to as “CirculaFloor,” mobile robots
locomotion. They can work with physical effort were utilized as locomotion tools on which users
exerted by user or through a triggering method. could freely walk while not actually moving in
Physical exercise tools such as treadmills and any direction in the real-world (Iwata et al. 2005).
stationary bicycles have been used for VR loco- As a drawback, all of these mobile tools require
motion (Brooks 1987). In this approach, user body-movement coordination and balancing
wears a head-mounted display and uses these skills from user’s end.
tools for moving in the virtual world. In the recent
years, several different treadmills have been uti-
lized for VR locomotion: unicycle; unidirectional Conclusion
and omnidirectional treadmills (Darken et al.
1997; Iwata 1999a, b); torus design-based tread- Locomotion is an important part of virtual real-
mill (Schwaiger et al. 2007; Souman et al. 2008); ity systems that may affect many factors related
low-friction surface (Jiung-Yao 2003; Suryajaya to user experience. Although there are several
et al. 2009), ball bearings and belt combination locomotion techniques that have been widely
referred to as “CyberCarpet” (De Luca used in VR, each has its limitations or draw-
et al. 2013). backs. There is still need for exploration and
Another type of stationary locomotion tools is improvement in this area. The introduction of
standard controllers. Among the most commonly the new generation VR devices in late 2010s
used ones are joysticks, joypads, and keyboards. It can make it easier to explore locomotion and
was reported that standard controllers are the most make it possible to introduce new locomotion
prevalent VR locomotion devices (Darken and ideas.
Locomotion in Virtual Reality Video Games 1073

Cross-References Harris, A., Nguyen, K., Wilson, P.T., Jackoski, M.,

Williams, B.: Human joystick: Wii-leaning to translate
in large virtual environments. In: Proceedings of the
▶ Interaction with Mobile Augmented Reality 13th ACM SIGGRAPH International Conference on
Environments Virtual-Reality Continuum and its Applications in
▶ Interactive Virtual Reality Navigation Using Cave Industry, pp. 231–234. ACM (2014)
Automatic Virtual Environment Technology Insook, C., Ricci, C.: Foot-mounted gesture detection and
its application in virtual environments. In: IEEE Inter-
national Conference on Systems, Man, and Cybernet-
ics. Computational Cybernetics and Simulation, vol.
References 4245, pp. 4248–4253. (1997)
Iwata, H.: The Torus Treadmill: realizing locomotion in
Barrera, S., Takahashi, H., Nakajima, M.: Hands-free nav- VEs. IEEE Comput. Graph. Appl. 19, 30–35 (1999a)
igation methods for moving through a virtual landscape Iwata, H.: Walking about virtual environments on an infi-
walking interface virtual reality input devices. In: nite floor. In: Proceedings of IEEE Virtual Reality,
Proceedings of Computer Graphics International, pp. 286–293. (1999b)
pp. 388–394. (2004) Iwata, H., Yano, H., Fukushima, H., Noma, H.:
Bowman, D.A., Kruijff, E., LaViola, J.J., Poupyrev, I.: 3D CirculaFloor [locomotion interface]. IEEE Comput.
User Interfaces: Theory and Practice. Addison-Wesley, Graph. Appl. 25, 64–67 (2005)
Boston (2004) Iwata, H., Yano, H., Tomioka, H.: Powered shoes. In:
Bozgeyikli, E., Raij, A., Katkoori, S., Dubey, R.: Locomo- ACM SIGGRAPH 2006 Emerging Technologies,
tion in virtual reality for individuals with autism spec- p. 28. ACM (2006)
trum disorder. In: Proceedings of the 2016 Symposium Jiung-Yao, H.: An omnidirectional stroll-based virtual real-
on Spatial User Interaction, pp. 33–42. ACM (2016) ity interface and its application on overhead crane
Brooks Jr, F.P.: Walkthrough – a dynamic graphics system training. IEEE Trans. Multimedia. 5, 39–51 (2003)
for simulating virtual buildings. In: Proceedings of the Kim, J., Gracanin, D., Quek, F.: Sensor-fusion walking-in-
1986 Workshop on Interactive 3D Graphics, pp. 9–21. place interaction technique using mobile devices. In:
ACM (1987) IEEE Virtual Reality Short Papers and Posters (VRW), L
Bruder, G., Steinicke, F., Hinrichs, K.H.: Arch-explore: pp 39–42. IEEE (2012)
a natural user interface for immersive architectural Kohli, L., Burns, E., Miller, D., Fuchs, H.: Combining
walkthroughs. In: IEEE Symposium on 3D User Inter- passive haptics with redirected walking. In: Proceed-
faces 3DUI 2009, pp. 75–82. (2009) ings of the 2005 International Conference on Aug-
Cirio, G., Marchal, M., Regia-Corte, T., Lecuyer, A.: The mented Tele-Existence, pp. 253–254. ACM (2005)
magic barrier tape: a novel metaphor for infinite navi- Marsh, W., Kelly, J., Dark, V., Oliver, J.: Cognitive
gation in virtual worlds with a restricted walking demands of semi-natural virtual locomotion. Presence.
workspace. In: Proceedings of the 16th ACM Sympo- 22, 216–234 (2013)
sium on Virtual Reality Software and Technology, Matthies, D.J.C. et al.: ShoeSoleSense: proof of concept
pp. 155–162. ACM (2009) for a wearable foot interface for virtual and real envi-
Darken, R.P., Sibert, J.L.: Navigating large virtual spaces. ronments. In: Proceedings of the 19th ACM Sympo-
Int. J. Hum. Comput. Interact. 8, 49–71 (1996) sium on Virtual Reality Software and Technology,
Darken, R.P., Cockayne, W.R., Carmein D.: The omni- pp. 93–96. ACM (2013)
directional treadmill: a locomotion device for virtual Nabiyouni, M., Saktheeswaran, A., Bowman, D.A.,
worlds. In: Proceedings of the 10th Annual ACM Sym- Karanth, A.: Comparing the performance of natural,
posium on User Interface Software and Technology, semi-natural, and non-natural locomotion techniques
pp. 213–221. ACM (1997) in virtual reality. In: IEEE Symposium on 3D User
De Luca, A., et al.: Motion control of the CyberCarpet Interfaces 3DUI 2015, pp. 3–10. IEEE (2015)
Platform. IEEE Trans. Control Syst. Technol. 21, Nilsson, N.C., Serafin, S., Laursen, M.H., Pedersen, K.S.,
410–427 (2013) Sikstrom, E., Nordahl, R.: Tapping-In-Place: increas-
Feasel, J., Whitton, M.C., Wendt, J.D.: LLCM-WIP: ing the naturalness of immersive walking-in-place
low-latency, continuous-motion walking-in-place. In: locomotion through novel gestural input. In: IEEE
IEEE Symposium on 3D User Interfaces 3DUI 2008, Symposium on 3D User Interfaces 3DUI 2013,
pp. 97–104. (2008) pp. 31–38. IEEE (2013a)
Freitag, S., Rausch, D., Kuhlen, T.: Reorientation in virtual Nilsson, N.C., Serafin, S., Nordahl, R.: The Perceived
environments using interactive portals. In: IEEE Sym- naturalness of virtual locomotion methods devoid of
posium on 3D User Interfaces 3DUI 2014, explicit leg movements. In: Proceedings of Motion on
pp. 119–122. (2014) Games, pp. 155–164. ACM (2013b)
Guy, E., Punpongsanon, P., Iwai, D., Sato, K., Nilsson, N.C., Serafin, S., Nordahl, R.: Establishing the
Boubekeur, T.: LazyNav: 3D ground navigation with range of perceptually natural visual walking speeds for
non-critical body parts. In: IEEE Symposium on 3D virtual walking-in-place locomotion. IEEE Trans. Vis.
User Interfaces 3DUI 2015, pp. 43–50. (2015) Comput. Graph. 20, 569–578 (2014)
1074 Locomotion in Virtual Reality Video Games

Peck, T.C., Fuchs, H., Whitton, M.C.: Improved redirec- walking techniques. IEEE Trans. Vis. Comput. Graph.
tion with distractors: a large-scale-real-walking loco- 16, 17–27 (2010)
motion interface and its effect on navigation in virtual Suma, E.A., Clark, S., Krum, D., Finkelstein, S., Bolas, M.,
environments. In: Proceedings of 2010 I.E. Virtual Warte, Z.: Leveraging change blindness for redirec-
Reality Conference (VR), pp. 35–38. IEEE (2010) tion in virtual environments. In: Proceedings of 2011
Peck, T.C., Fuchs, H., Whitton, M.C.: An evaluation I.E. Virtual Reality Conference (VR), pp. 159–166.
of navigational ability comparing redirected free explo- IEEE (2011)
ration with distractors to walking-in-place and Suma, E.A., Lipps, Z., Finkelstein, S., Krum, D.M.,
joystick locomotion interfaces. In: Proceedings of Bolas, M.: Impossible spaces: maximizing natural
2012 I.E. Virtual Reality Conference (VR), walking in virtual environments with self-overlapping
pp. 55–62. IEEE (2011) architecture. IEEE Trans. Vis. Comput. Graph. 18,
Peck, T.C., Fuchs, H., Whitton, M.C.: The design and 555–564 (2012)
evaluation of a large-scale real-walking locomotion Suryajaya, M., Lambert, T., Fowler, C.: Camera-based
interface. IEEE Trans. Vis. Comput. Graph. 18, OBDP locomotion system. In: Proceedings of the
1053–1067 (2012) 16th ACM Symposium on Virtual Reality Software
Razzaque, S., Kohn, Z., Whitton, M.C.: Redirected Walk- and Technology, pp. 31–34. (2009)
ing. University of North Carolina at Chapel Hill (2001) Templeman, J., Denbrook, P., Sibert, L.: Virtual locomo-
Riecke, B.E., Bodenheimer, B., McNamara, T.P., tion: walking in place through virtual environments.
Williams, B., Peng, P., Feuereissen, D.: Do we need Presence. 8, 598–617 (1999)
to walk for effective virtual reality navigation? Physical Terziman, L., Marchal, M., Emily, M., Multon, F.,
rotations alone may suffice. In: Proceedings of the Arnaldi, B., Lécuyer, A.: Shake-your-head: revisiting
Spatial Cognition International Conference, walking-in-place for desktop virtual reality. In: Pro-
pp. 234–247. (2010) ceedings of the 17th ACM Symposium on Virtual
Robinett, W., Holloway, R.: Implementation of flying, Reality Software and Technology, pp. 27–34. (2010)
scaling and grabbing in virtual worlds. In: Proceedings Usoh, M., Arthur, K., Whitton, M.C., Bastos, R., Steed, A.,
of the 1992 Symposium on Interactive 3D Graphics, Slater, M., Frederick, P., Brooks, J.: Walking > walking-
pp. 189–192. (1992) in-place > flying, in virtual environments. In: Proceed-
Ruddle, R.A., Lessels, S.: For efficient navigational search, ings of the 26th Annual Conference on Computer
humans require full physical movement, but not a rich Graphics and Interactive Techniques, pp. 359–364.
visual scene. Psychol. Sci. 17, 460–465 (2006) (1999)
Ruddle, R.A., Volkova, E., Bulthoff, H.H.: Walking Valkov, D., Steinicke, F., Bruder, G., Hinrichs, K.:
improves your cognitive map in environments that are A multi-touch enabled human-transporter metaphor
large-scale and large in extent. ACM Trans. Comput. for virtual 3D traveling. In: Proceedings of 2010 I.-
Hum. Interact. 18, 1–20 (2011) E. Symposium on 3D User Interfaces (3DUI),
Schwaiger, M., Thummel, T., Ulbrich, H.: Cyberwalk: an pp. 79–82. (2010)
advanced prototype of a belt array platform. In: Pro- Vasylevska, K., Kaufmann, H., Bolas, M., Suma, E.A.:
ceedings of IEEE International Workshop on Haptic, Flexible spaces: dynamic layout generation for infinite
Audio and Visual Environments and Games, walking in virtual environments. In: Proceedings of
pp. 50–55. (2007) 2013 I.E. Symposium on 3D User Interfaces (3DUI),
Slater, M., Steed, A., Usoh, M.: The virtual treadmill: pp. 39–42. (2013)
a naturalistic metaphor for navigation in immersive Ware, C., Osborne, S.: Exploration and virtual camera
virtual environments. In: Proceedings of Eurographics control in virtual three dimensional environments.
Workshops on Virtual Environments 0 95, pp. 135–148. Proc. SIGGRAPH. Comput. Graph. 24, 175–183
(1995a) (1990)
Slater, M., Usoh, M., Steed, A.: Taking steps: the influence of Wendt J.D, Whitton M.C, Brooks F.P.: GUD WIP: gait-
a walking technique on presence in virtual reality. ACM understanding-driven walking-in-place. In: Proceed-
Trans. Comput. Hum. Interact. 2, 201–219 (1995b) ings of 2010 I.E. Virtual Reality Conference (VR),
Souman, J.L., et al.: CyberWalk: enabling unconstrained pp. 51–58. (2010)
omnidirectional walking through virtual environments. Whitton, M.C., Peck, T.C.: Stepping-Driven Locomotion
ACM Trans. Appl. Percept. 8, 1–22 (2008) Interfaces. Human Walking in Virtual Environments:
Steinicke, F., Bruder, G., Jerald, J., Frenz, H., Lappe, M.: Perception, Technology, and Applications,
Analyses of human sensitivity to redirected walking. pp. 241–262. Springer, New York (2013)
In: Proceedings of the 2008 ACM Symposium on Vir- Whitton M.C et al.: Comparing VE locomotion interfaces.
tual Reality Software and Technology, pp. 149–156. In: Proceedings of IEEE Virtual Reality (VR),
(2008) pp. 123–130. IEEE (2005)
Steinicke, F., Bruder, G., Hinrichs, K., Jerald, J., Frenz, H., Williams, B., Narasimham, G., Rump, B., McNamara, T.P.,
Lappe, M.: Real walking through virtual environments Carr, T.H., Rieser, J., Bodenheimer, B.: Exploring large
by redirection techniques. J. Virtual Real. Broadcast. virtual environments with an HMD when physical
6(2) (2009) space is limited. In: Proceedings of the 4th Symposium
Steinicke, F., Bruder, G., Jerald, J., Frenz, H., Lappe, M.: on Applied Perception in Graphics and Visualization,
Estimation of detection thresholds for redirected pp. 41–48. (2007)
Loot Boxes: Gambling-Like Mechanics in Video Games 1075

Williams, B., Bailey, S., Narasimham, G., Li, M., reward mechanisms; Randomized monetization
Bodenheimer, B.: Evaluation of walking in place on a methods; Surprise mechanics; Virtual Gashapons
Wii balance board to explore a virtual environment.
ACM Trans. Appl. Percept. 8, 1–14 (2011)
Zhixin, Y., Lindeman, R.W.: A multi-touch finger gesture
based low-fatigue VR travel framework. In: Proceed- Definition
ings of 2015 I.E. Symposium on 3D User Interfaces
(3DUI), pp. 193–194. (2015)
Zielinski, D.J., McMahan, R.P., Brady, R.B.: Shadow Loot boxes are mechanics often found in video
walking: an unencumbered locomotion technique for games that provide the player with randomized
systems with under-floor projection. In: Proceedings of virtual rewards. Some loot boxes can be paid for
2011 I.E. Virtual Reality Conference (VR), with real-world money and therefore share struc-
pp. 167–170. IEEE (2011)
tural and psychological similarities with
gambling.

Locomotion Techniques Introduction

▶ Locomotion in Virtual Reality Video Games “Loot boxes” is a colloquial catch-all terminology
used to describe software features, typically found
in video games, that provide the player with ran-
domized virtual rewards (Drummond and Sauer
Loot Box 2018). The player must satisfy an “eligibility con-
dition” to engage with (or “open”) the loot box:
▶ Counter-Strike Global Offensive, an Analysis This could be by defeating a certain in-game L
enemy, by obtaining a certain virtual item, by
watching embedded commercials, or (more
importantly) through purchasing using fiat cur-
Loot Boxes: Gambling-Like rency (or real-world money) (Nielsen and
Mechanics in Video Games Grabarczyk 2019). Once the player engages with
the loot box, a “random procedure,” of potentially
Leon Y. Xiao1,3,4, Laura L. Henderson1,2, Rune K. varying degrees of complexity (Ballou et al.
L. Nielsen4, Paweł Grabarczyk4 and Philip W. S. 2020), is used to determine what virtual rewards
Newall5 the player will obtain. The virtual “rewards” that
1
The Honourable Society of Lincoln’s Inn, the player obtains may be merely cosmetic items
London, UK that, e.g., change the color of the player’s armor,
2
The City Law School, City, University of or may, alternatively, influence gameplay more
London, London, UK significantly by, e.g., unlocking additional game
3
School of Law, Queen Mary University of content or increasing the player’s in-game power
London, London, UK (Xiao 2021). These “rewards” may be transferable
4
Center for Computer Games Research, IT (or “sold”) to other players, in exchange for real-
University of Copenhagen, København, Denmark world money (Drummond et al. 2020b), or may be
5
Experimental Gambling Research Laboratory, restricted by the video game company for use only
School of Health, Medical and Applied Sciences, inside the in-game economy by the original player
CQUniversity, Sydney, NSW, Australia who engaged with the loot box (Xiao 2020a). The
act of engaging with a loot box may be
represented in-game as the player literally open-
Synonyms ing a box containing loot; however, the loot box
mechanic can also be visually represented in other
Blind bags; Blind boxes; Booster packs; Gachas; forms, e.g., as tearing open a card pack, spinning a
Gatchas; Loot crates; Prize crates; Random prize wheel, or receiving a capsule from a “gacha”
1076 Loot Boxes: Gambling-Like Mechanics in Video Games

Loot Boxes: Gambling-Like Mechanics in Video Games, Table 1 Four categories of loot boxes proposed by Nielsen
and Grabarczyk (2019)
Category Description Regulatory positions
Embedded- Costs real-world money to engage, and its Regulated as gambling in Belgium (Belgische
Embedded reward does have real-world value Kansspelcommissie [Belgian Gaming Commission]
2018), the Netherlands (Kansspelautoriteit [The
Netherlands Gambling Authority] 2018), the UK
(UK Gambling Commission 2017), and most other
countries that regulate gambling as a licensable activity
Banned as gambling in most countries that more heavily
prohibit gambling
Embedded- Costs real-world money to engage, but its Regulated as gambling in Belgium (Belgische
Isolated reward does not have real-world value Kansspelcommissie [Belgian Gaming Commission]
2018)
Unregulated in most other countries
Isolated- Does not cost real-world money to engage, Unregulated in most countries
Embedded but its reward does have real-world value
Isolated- Does not cost real-world money to engage, Unregulated in most countries
Isolated and its reward does not have real-world
value

vending machine. Regardless of their visual rep- Global Offensive) generated US$528,000 in 1 day
resentation, all loot boxes share the structural in one country alone, thus hinting at the immense
characteristics of being triggered by satisfying an size of the global loot box market (Zendle et al.
“eligibility condition,” involve a “random proce- 2020b).
dure,” and provide “rewards,” as described above. In terms of the historical context and develop-
Nielsen and Grabarczyk (2019) proposed a ment of loot boxes, it has been suggested that
framework for classifying various implementations using loot boxes to monetize video games was
of loot boxes into four categories which focuses on inspired by how collectible sports cards and fan-
whether the “eligibility condition” requires spend- tasy trading cards (e.g., Magic: The Gathering)
ing real-world money to satisfy, and whether the are sold in blind, randomized packs in order to
“rewards” can be transferred to other players in encourage players to buy more packs and increase
exchange for legal tender, and therefore possesses revenue (Nielsen and Grabarczyk 2019; Švelch
real-world value, as shown in Table 1. 2020; Xiao 2021). These randomized packs were
designed to contain rare cards, known as “chase
cards,” that were less likely to be included in
Paid Loot Boxes in Context packs than other cards and were therefore more
sought-after and monetarily valuable. The con-
Loot boxes that players have to purchase with sumer was thereby encouraged to purchase more
real-world money to engage with are packs in order to obtain such rare “chase cards,”
implemented by companies as monetization but they would more often only obtain less valu-
methods in video games, known as “micro- able, duplicate cards that they already possessed
transactions,” which represent an alternative, or when they try to “chase” rare cards.
complementary, business model to selling copies Loot boxes are implemented in highly popular
of the software or providing subscription-based home console games, e.g., the Ultimate Team
services (Petrovskaya and Zendle 2020). Analysis Packs in Electronic Art’s FIFA games (Electronic
of loot box-spending data has revealed that the Arts 2019). Presently, loot boxes are prevalent in
loot boxes of one single game (Counter-Strike: video games, particularly on mobile platforms,
Loot Boxes: Gambling-Like Mechanics in Video Games 1077

e.g., Android and iOS: In 2019, 59% of the online slot machines, and not correlated with more
highest-grossing iPhone games in the UK traditional gambling formats, e.g., playing bingo
contained loot boxes, while 36% of the 50 Highest or the lottery in person (Zendle 2020). The rela-
grossing PC games on Steam contained loot boxes tionship between loot boxes and gambling may be
(Zendle et al. 2020a). Compared to in the UK, weaker in non-Western countries where gambling
which represents the Western video game market, is more heavily prohibited, rather than regulated
loot boxes are significantly more prevalent in as a licensable activity, e.g., China, where lottery
China: In 2020, 91% of the 100 highest-grossing products represent the only legally available com-
iPhone games contained loot boxes (Xiao et al. mercial gambling opportunity (Xiao et al. 2021a).
2021b). This reflects that video game markets in Cultural differences and legal differences in gam-
different countries may implement loot boxes to bling product availability may affect the relation-
different degrees. Video games containing loot ship between loot boxes and gambling and remain
boxes are also generally given low age ratings: a direction for future research.
95% of the highest-grossing iPhone games Further, as in gambling contexts (Deng et al.
containing loot boxes were deemed suitable for 2021; Muggleton et al. 2021), the vast majority of
children aged 12+ (Zendle et al. 2020a). This loot box revenue is generated by a small minority
suggests that children are regularly exposed to of players spending significant amounts of money
loot boxes and can readily purchase them. The (Zendle et al. 2020b). This small minority of
UK Gambling Commission’s survey (2019) players have been identified as generally being
found that 23% of 11- to 16-year-olds reported players with problem gambling issues, rather
paying real-world money for loot boxes. than players with high personal incomes, thus
suggesting that video game companies may be
disproportionally profiting from potentially vul- L
Potential Harms: Links with Problem nerable consumers (Close et al. 2021).
Gambling Researchers have also suggested that cognitive
biases that are present in gambling contexts, e.g.,
Paid loot boxes, because of the fact that players the gambler’s fallacy and loss chasing, which lead
spend real-world money to engage with them and to maladaptive gambling, may also apply to loot
because of their randomized nature, are structur- box purchasing behavior and lead to maladaptive
ally and psychologically similar to gambling loot box overspending (King and Delfabbro 2018;
(Drummond and Sauer 2018). This encompasses Nielsen and Grabarczyk 2019; Xiao 2021).
Embedded-Embedded and Embedded-Isolated Finally, it has yet to be determined whether
loot boxes under Nielsen and Grabarczyk’s cate- engagement with loot boxes in childhood affects
gorization (2019). Further, loot box purchasing a person’s risk of developing gambling problems
has been found to be positively correlated with later in life.
problem gambling severity in more than a dozen
empirical studies in Western countries (Garea
et al. 2021), e.g., the USA (Zendle and Cairns Regulation by Law and Industry
2019), Australasia (Drummond et al. 2020a), Self-Regulation
Denmark (Kristiansen and Severin 2019), and
Germany (von Meduna et al. 2020). Players with Paid loot boxes have been the subject of regula-
higher problem gambling severity tend to spend tory scrutiny by gambling regulators and
more money purchasing loot boxes (Zendle and policymakers in many countries because of their
Cairns 2018). In Western countries, loot box similarities with gambling and because of the link
spending appears to be more strongly correlated between loot box purchasing and problem gam-
with relatively “gamified” gambling games, e.g., bling severity (Cerulli-Harms et al. 2020). In
1078 Loot Boxes: Gambling-Like Mechanics in Video Games

particular, concerns about children engaging with affects video game companies’ commercial inter-
loot boxes have been raised because they may be ests. Conversely, nonregulation would continue to
more susceptible to overspending money and expose players, including vulnerable consumers,
more in need of consumer protection measures to potential loot box harms, and is arguably inad-
(Wardle and Zendle 2021; Zendle et al. 2019). equate and unsatisfactory because of the
Different jurisdictions are regulating the various precautionary principle of public health (Digital,
categories of loot boxes to varying extents (Xiao Culture, Media and Sport Committee of the House
2021), as shown in Table 1. Applying existing of Commons (UK) 2019), which states that the
gambling law has been how loot boxes have lack of scientific certainty cannot justify regula-
become regulated in nearly all countries: Various tory inaction in a situation of high potential risk.
countries diverge as to which of the two categories Middle ground approaches that lie between a
of paid loot boxes (i.e., Embedded-Embedded and blanket ban and nonregulation have also been
Embedded-Isolated loot boxes as defined by Niel- proposed and adopted in certain countries, e.g.,
sen and Grabarczyk) constitutes gambling and is restricting loot box purchasing only when a player
therefore regulated: Nearly all countries agree that attempts to go above a certain maximum spending
Embedded-Embedded loot boxes constitute gam- limit (Drummond et al. 2019; Xiao 2020b, 2021).
bling, but only a small minority of countries (e.g., Other consumer protection measures, which
Belgium) have taken the position that Embedded- have been applied in gambling contexts
Isolated loot boxes also constitute gambling. It is (Livingstone et al. 2019), have also been identi-
rather paradoxical that randomized physical fied as being potentially applicable to loot boxes
sports and trading card packs, which arguably (King and Delfabbro 2019; Xiao and Henderson
inspired loot boxes, are generally not considered 2021). The most prominent nonrestrictive regula-
to be a form of gambling and have thereby evaded tory measure is the disclosure of “winning” prob-
regulatory scrutiny. Such physical packs legally abilities, which reveals how likely a player is to
constitute gambling in most countries because: obtain a particular reward, as implemented in
They are bought with real-world money; their Mario Kart Tour (2019). This would require
content is randomized; and the content has real- video game companies to reveal and publish the
world monetary value because it can be sold to exact probabilities of obtaining each randomized
other people. Future research should consider loot box reward. The video game industry has
why such physical Embedded-Embedded loot increasingly imposed this requirement as
boxes are not considered to be gambling (Zendle self-regulation, e.g., by Apple (Kuchera 2017),
et al. 2021). Google (Gach 2019), and the major hardware pro-
The simplest regulatory solution is to ban the viders and game publishers (Entertainment Soft-
sale of loot boxes. This has effectively been done ware Association (ESA) 2019). This measure has
in Belgium where all paid loot boxes have been also been adopted as law in China (presently the
determined to be gambling and where no gam- only country to do so), which has led to wide-
bling licenses have been granted to video game spread compliance; however, the prominence and
companies for the sale of loot boxes (Belgische accessibility of disclosure have been identified as
Kansspelcommissie [Belgian Gaming Commis- being suboptimal (Xiao et al. 2021b). A survey of
sion] 2018). This prevents Belgian players from Chinese video game players found that 85% of
purchasing loot boxes and thus shields them from loot box purchasers reported seeing probability
potential harms. However, this ban in Belgium disclosures (meaning that they have been reason-
has led to the removal of many video games that ably widely seen by players); however, only 19%
rely on loot boxes to generate revenue and which of this group reported spending less money on
can no longer be profitably operated in that coun- loot boxes as a result of seeing the disclosures
try (Xiao 2021). A blanket ban does not offer (Xiao et al. 2021a). This suggests that loot box
players freedom to play the video games they probability disclosures may be of limited effec-
want or to engage with loot boxes and negatively tiveness at reducing loot box spending even if
Loot Boxes: Gambling-Like Mechanics in Video Games 1079

they are accessibly and prominently displayed 2019; Xiao and Henderson 2021; Xiao and
such that all players can see them (Xiao and Newall 2021).
Newall 2021). The video game industry has been
widely supportive of loot box probability disclo-
sure as an industry self-regulatory measure aimed Conclusion: Directions for Future
at ensuring consumer protection (Entertainment Research
Software Association (ESA) 2019), but whether
self-regulation is effective remains to be assessed Loot boxes, and paid loot boxes in particular,
by future research. represent a relatively novel mechanic in video
Loot boxes and gambling may share many games. Despite increasing research attention
structural similarities; however, they are dissimi- being paid to the issue, further research is
lar in at least one regard: how the company makes required to fill in the existing knowledge gaps.
money. Gambling operators lose money when the Future correlational research between loot boxes
player wins money, which is why gambling oper- and gambling should examine whether loot box
ators must ensure that the gambling games are purchasing is more strongly correlated with
designed such that the operator is more likely engagement with specific gambling products,
than the player to win. In contrast, a video game rather than engagement with gambling in gen-
company does not directly lose money when the eral. Existing research has largely utilized self-
player wins a reward, valuable or otherwise, as it reported data: Transparent collaboration with the
does not directly cost money to give players these video game industry may provide more reliable
virtual rewards. However, it should also be noted data. Indeed, qualitative methods may assist in
that a video game company would face an indirect better understanding individual players’ experi-
loss when a player wins a valuable reward (Xiao ences with loot boxes (Nicklin et al. 2021), and L
2020c): A player may stop purchasing a loot box gauging players’ views as to the implementation
after receiving their desired reward, meaning that and regulation of loot boxes (Petrovskaya and
the video game company may stop making money Zendle 2021): This is especially relevant for
from that player after the valuable reward is given Embedded-Isolated loot boxes because, although
out. In order to be sustainable, many loot box this category represents the vast majority of paid
systems rely on frequent updates with new loot boxes implemented in video games, these
rewards, but each new reward costs money for mechanics have no obvious counterparts in
the video game company to develop, meaning nondigital contexts, and therefore there is no
that companies would have to expend more costs translatable literature from other fields. Further,
to develop more new content if players are able to cross-cultural perspectives would clarify
more easily obtain desirable rewards. Further, the whether players in various countries are
value and desirability of a reward would decrease experiencing loot boxes differently. Addition-
if all players managed to obtain it: It would no ally, neuroscience perspectives may shed light
longer be a “rare” reward with which players on how player’s cognition is affected when
could impress others. However, despite the poten- engaging with loot boxes: Such perspectives are
tial for indirect loss, video game companies are prominent in research on gambling disorder;
still financially able to give out their most valuable however, as of yet, they are missing from the
rewards more frequently than traditional gam- loot box literature. Finally, the prevalence of
bling operators. This means that loot box con- serious problems with loot box spending has
sumer protection methods do not have to be never been assessed, and it is not known whether
limited to what has been done in gambling con- such problems are caused by exposure to loot
texts, and that loot box consumers could be addi- boxes or are instead symptoms of preexisting
tionally protected by novel features of ethical underlying issues. In conclusion, despite recent
game design, e.g., allowing players to win valu- advances made by the literature, loot boxes
able rewards more often (King and Delfabbro remain an area deserving of further research.
1080 Loot Boxes: Gambling-Like Mechanics in Video Games

References Garea, S.S., Drummond, A., Sauer, J.D., Hall, L.C., Wil-
liams, M.N.: Meta-analysis of the relationship between
Ballou, N., Gbadamosi, C.T.T., Zendle, D.: The hidden problem gambling, excessive gaming and loot box
intricacy of loot box design: a granular description of spending. Intern. Gambl. Stud. Advance Online Publi-
random monetized reward features. (2020). https://doi. cation, 1–20 (2021). https://doi.org/10.1080/
org/10.31234/osf.io/xeckb 14459795.2021.1914705
Belgische Kansspelcommissie [Belgian Gaming Commis- Kansspelautoriteit [The Netherlands Gambling Authority]:
sion]: Onderzoeksrapport loot boxen [Research Report Onderzoek naar loot boxes: Een buit of een last? [Study
on Loot Boxes]. (2018) into Loot Boxes: A Treasure or a Burden?] (2018)
Cerulli-Harms, A., Münsch, M., Thorun, C., Michaelsen, King, D.L., Delfabbro, P.H.: Predatory monetization
F., Hausemer, P.: Loot Boxes in Online Games and schemes in video games (e.g. ‘loot boxes’) and internet
Their Effect on Consumers, in Particular Young Con- gaming disorder. Addiction. 113, 1967–1969 (2018).
sumers. Policy Department for Economic, Scientific https://doi.org/10.1111/add.14286
and Quality of Life Policies (EU) (2020) King, D.L., Delfabbro, P.H.: Video game monetization
Close, J., Spicer, S.G., Nicklin, L.L., Uther, M., Lloyd, J., (e.g., ‘loot boxes’): a blueprint for practical social
Lloyd, H.: Secondary analysis of loot box data: are responsibility measures. Int. J. Ment. Health Addiction.
high-spending “whales” wealthy gamers or problem 17, 166–179 (2019). https://doi.org/10.1007/s11469-
gamblers? Addict. Behav. 117, 106851 (2021). https:// 018-0009-3
doi.org/10.1016/j.addbeh.2021.106851 Kristiansen, S., Severin, M.C.: Loot box engagement and
Deng, X., Lesch, T., Clark, L.: Pareto distributions in problem gambling among adolescent gamers: findings
online casino gambling: Sensitivity to timeframe and from a national survey. Addict. Behav. 103,
associations with self-exclusion. Addict. Behav. 120, 106254 (2019). https://doi.org/10.1016/j.addbeh.2019.
106968 (2021). https://doi.org/10.1016/j.addbeh.2021. 106254
106968 Kuchera, B.: Apple Adds New Rules for Loot Boxes,
Digital, Culture, Media and Sport Committee of the House Requires Disclosure of Probabilities. (2017). https://
of Commons (UK): Immersive and Addictive Technol- web.archive.org/web/20200821095535/https://www.
ogies: Fifteenth Report of Session 2017–19. (2019) polygon.com/2017/12/21/16805392/loot-box-odds-
Drummond, A., Sauer, J.D.: Video game loot boxes are rules-apple-app-store
psychologically akin to gambling. Nat. Hum. Behav. 2, Livingstone, C., Rintoul, A., de Lacy-Vawdon, C.,
530–532 (2018). https://doi.org/10.1038/s41562-018- Borland, R., Dietze, P., Jenkinson, R., Livingston, M.,
0360-1 Room, R., Smith, B., Stoove, M., Winter, R., Hill, P.:
Drummond, A., Sauer, J.D., Hall, L.C.: Loot box limit- Identifying Effective Policy Interventions to Prevent
setting: a potential policy to protect video game users Gambling-Related Harm. Victorian Responsible Gam-
with gambling problems? Addiction. 114, bling Foundation, Melbourne (2019)
935–936 (2019). https://doi.org/10.1111/add.14583 Muggleton, N., Parpart, P., Newall, P., Leake, D.,
Drummond, A., Sauer, J.D., Ferguson, C.J., Hall, L.C.: The Gathergood, J., Stewart, N.: The association between
relationship between problem gambling, excessive gambling and financial, social and health outcomes in
gaming, psychological distress and spending on loot big financial data. Nat. Hum. Behav. 5,
boxes in Aotearoa New Zealand, Australia, and the 319–326 (2021). https://doi.org/10.1038/s41562-020-
United States – a cross-national survey. PLOS One. 01045-w
15, e0230378 (2020a). https://doi.org/10.1371/jour- Nicklin, L.L., Spicer, S.G., Close, J., Parke, J., Smith, O.,
nal.pone.0230378 Raymen, T., Lloyd, H., Lloyd, J.: “It’s the attraction of
Drummond, A., Sauer, J.D., Hall, L.C., Zendle, D., winning that draws you in” – a qualitative investigation
Loudon, M.R.: Why loot boxes could be regulated as of reasons and facilitators for videogame loot box
gambling. Nat. Hum. Behav. 4, 986 (2020b). https:// engagement in UK gamers. J. Clin. Med. 10,
doi.org/10.1038/s41562-020-0900-3 2103 (2021). https://doi.org/10.3390/jcm10102103
Electronic Arts: Pack Probability in FIFA Ultimate Team. Nielsen, R.K.L., Grabarczyk, P.: Are loot boxes gambling?
(2019). https://www.ea.com/games/fifa/news/fifa- Random reward mechanisms in video games.
pack-probabilities ToDIGRA. 4, 171–207 (2019). https://doi.org/10.
Entertainment Software Association (ESA): Video Game 26503/todigra.v4i3.104
Industry Commitments to Further Inform Consumer Petrovskaya, E., Zendle, D.: The Battle Pass: A Mixed-
Purchases. (2019). https://www.theesa.com/perspec Methods Investigation into a Growing Type of Video
tives/video-game-industry-commitments-to-further- Game Monetisation (2020). https://doi.org/10.31219/
inform-consumer-purchases/ osf.io/vnmeq
Gach, E.: Google Now Requires App Makers to Disclose Petrovskaya, E., Zendle, D.: Predatory monetisation?
Loot Box Odds. (2019). kotaku.com/google-now- A categorisation of unfair, misleading, and aggressive
requires-app-makers-to-disclose-loot-box-odd- monetisation techniques in digital games from the per-
1835134642 spective of players. J. Bus. Ethics. Advance Online
Lowest% 1081

Publication, (2021). https://doi.org/10.1007/s10551- box probability disclosure regulations in China. Behav.

021-04970-6 Public Policy. Advance Online Publication, (2021b).
Švelch, J.: Mediatization of a card game: magic: the gath- https://doi.org/10.1017/bpp.2021.23
ering, esports, and streaming. Media Cult. Soc. 42, Zendle, D.: Beyond loot boxes: a variety of gambling-like
8 3 8 – 8 5 6 ( 2 0 2 0 ) . h t t p s : / / d o i . o r g / 1 0 . 11 7 7 / practices in video games are linked to both problem
0163443719876536 gambling and disordered gaming. Peer J. 8,
UK Gambling Commission: Virtual Currencies, eSports e9466 (2020). https://doi.org/10.7717/peerj.9466
and Social Gaming – Position Paper. (2017) Zendle, D., Cairns, P.: Video game loot boxes are linked to
UK Gambling Commission: Young People and Gambling problem gambling: results of a large-scale survey.
Survey 2019: A Research Study Among 11–16 Year PLoS One. 13(11), e0206767 (2018). https://doi.org/
Olds in Great Britain. (2019) 10.1371/journal.pone.0206767
von Meduna, M., Steinmetz, F., Ante, L., Reynolds, J., Zendle, D., Cairns, P.: Loot boxes are again linked to
Fiedler, I.: Loot boxes are gambling-like elements in problem gambling: results of a replication study.
video games with harmful potential: results from a PLoS One. 14(3), e0213194 (2019). https://doi.org/
large-scale population survey. Technol. Soc. 63, 10.1371/journal.pone.0213194
101395 (2020). https://doi.org/10.1016/j.techsoc. Zendle, D., Meyer, R., Over, H.: Adolescents and loot
2020.101395 boxes: links with problem gambling and motivations
Wardle, H., Zendle, D.: Loot boxes, gambling, and prob- for purchase. R. Soc. Open Sci. 6, 190049 (2019).
lem gambling among young people: results from a https://doi.org/10.1098/rsos.190049
cross-sectional online survey. Cyberpsychol. Behav. Zendle, D., Meyer, R., Cairns, P., Waters, S., Ballou, N.:
Soc. Netw. 24, 267–274 (2021). https://doi.org/10. The prevalence of loot boxes in mobile and desktop
1089/cyber.2020.0299 games. Addiction. 115, 1768–1772 (2020a). https://
Xiao, L.Y.: Which implementations of loot boxes consti- doi.org/10.1111/add.14973
tute gambling? A UK legal perspective on the poten- Zendle, D., Petrovskaya, E., Wardle, H.: How do loot
tial harms of random reward mechanisms. Intern. boxes make money? An analysis of a very large dataset
J. Ment. Health Addiction. Advance Online Publica- of real Chinese CSGO loot box openings. (2020b).
tion, (2020a). https://doi.org/10.1007/s11469-020- https://doi.org/10.31234/osf.io/5k2sy
00372-3 Zendle, D., Walasek, L., Cairns, P., Meyer, R., Drummond, L
Xiao, L.Y.: People’s Republic of China legal update: the A.: Links between problem gambling and spending on
notice on the prevention of online gaming addiction in booster packs in collectible card games: a conceptual
juveniles (Published October 25, 2019, effective replication of research on loot boxes. PLoS One. 16,
November 1, 2019). Gaming Law Rev. 24, 51–53 e0247855 (2021). https://doi.org/10.1371/journal.
(2020b). https://doi.org/10.1089/glr2.2019.0002 pone.0247855
Xiao, L.Y.: Conceptualising the loot box transaction as a
gamble between the purchasing player and the video
game company. Intern. J. Ment. Health Addiction.
Advance Online Publication, (2020c). https://doi.org/
10.1007/s11469-020-00328-7
Xiao, L.Y.: Regulating loot boxes as gambling? Loot Crates
Towards a combined legal and self-regulatory con-
sumer protection approach. Interact. Entertain. Law
Rev. 4(1), 27–47 (2021). https://doi.org/10.4337/ ▶ Loot Boxes: Gambling-Like Mechanics in
ielr.2021.01.02 Video Games
Xiao, L.Y., Henderson, L.L.: Towards an ethical game
design solution to loot boxes: a commentary on King
and Delfabbro. Int. J. Ment. Health Addiction. 19,
177–192 (2021). https://doi.org/10.1007/s11469-019-
00164-4 Loud-speaker Reproduction
Xiao, L.Y., Newall, P.W.S.: Probability disclosures are not
enough: reducing loot box reward complexity as a part
of ethical video game design. (2021). https://psyarxiv.
▶ Sound Spatialization
com/nuksd/
Xiao, L.Y., Fraser, T.C., Newall, P.W.S.: Opening Pan-
dora’s loot box: Novel links with gambling, and player
opinions on probability disclosures and pity-timers in
China. (2021a). https://psyarxiv.com/837dv/
Lowest%
Xiao, L.Y., Henderson, L.L., Yang, Y., Newall, P.W.S.:
Gaming the system: suboptimal compliance with loot ▶ Speedrunning in Video Games
1082 Ludification

Ludii was developed as part of the European

Ludification Research Council (ERC) funded Digital Ludeme
Project with one of its primary purposes being for
▶ Gamification the reconstruction of historical games from partial
rulesets based upon the available evidence
(Browne et al. 2019b). However, this is only one
application, and Ludii provides a range of features
Ludii General Game System intended to help the modern game designer pro-
for Modeling, Analyzing, and totype, fine tune, and discover new designs.
Designing Board Games

Cameron Browne, Éric Piette, Matthew The Ludii System

Stephenson and Dennis J. N. J. Soemers
Maastricht University, Maastricht, Netherlands The Ludii system is based on the notion of the
ludeme, which can be described as a game related
concept or element of play that is relevant to the
Definition equipment and/or rules of a game (Browne 2021).
Ludemes constitute the fundamental building
The Ludii computer program is a complete gen- blocks of which games are composed.
eral game system for digitally modeling, ana- Games are defined for Ludii as structured
lyzing, and designing a wide range of games. ludeme trees in the form of LISP-like symbolic
These include traditional tabletop games such as expressions, according to a custom grammar that
board games and dice games, in addition to card constitutes the Ludii Game Description Language
games, graph games, mathematical games, puz- (L-GDL). For example, the game Tic-Tac-Toe can
zles, simulations, and simple video games. be defined as follows:
Ludii supports stochastic (chance) elements,
hidden information, adversarial and cooperative (game "Tic-Tac-Toe"
modes of play, and any number of players from (players 2)
1 to 16. (equipment {
(board (square 3))
The system differs from existing general
(piece "Disc" Each)
game playing (GGP) programs in a number of })
ways. Its underlying ludemic model allows a (rules
wider range of games to be described more easily (play (move Add (to (sites
Empty))))
and succinctly than other approaches, and it is
(end
intended as a tool for game design as much as (if
game playing. Ludii belongs the “hybrid” class (if Line 3)
of GGP approaches that allows extensible (result Mover Win)
)
higher-level game descriptions (Kowalksi
)
et al. 2020). )
The Ludii distribution comes with over 1000 )
predefined games and a number of default artifi-
cial intelligence (AI) agents for playing and ana- This ludemic model for describing games
lyzing these, in addition to new games authored allows a wide range of games to be described
by users. An open Ludii AI API is provided to simply and succinctly. The fact that ludemes
facilitate the system’s use as a platform for general encapsulate key game concepts makes such
game-based AI research. Ludii’s Java code base is descriptions highly conducive to automated
freely available under a Creative Commons (CC manipulations such as the evolution of new
BY-NC-ND 4.0) license. games from existing rulesets (Browne 2009).
Ludii General Game System for Modeling, Analyzing, and Designing Board Games 1083

Architecture Once the Manager module verifies that this

Figure 1 shows an overview of the main compo- move is valid, it is passed to the Core module,
nents of the Ludii system. Additional information where the state of the game is updated.
on the relevant modules are provided in the fol-
lowing sections. Grammar
The game description (written using the The custom Ludii Grammar is an Extended
L-GDL) is initially passed into the Grammar mod- Backus-Naur Form (EBNF) style grammar
ule, which first expands and parses the game consisting of a set of production rules used to gen-
description to check that it is syntactically valid. erate game descriptions. The grammar is generated
Once this initial check is done, the game descrip- automatically from the Ludii code base using a
tion is then compiled, transforming it into its class grammar approach in which all keywords,
internal logic format within the Core module. rules, and instantiations are derived directly from
This internal logic contains all the relevant infor- their corresponding Java classes (Browne 2016).
mation about the game, including the equipment, This approach provides a 1:1 correspondence
rules, and supplementary metadata. The metadata between the L-GDL and the underlying Java code
provides additional information about the game at all times, effectively making the Ludii Gram-
which is not related to how it is played. This mar a snapshot of the current class hierarchy and
includes textual information about the game – making the system easily extensible. New func-
such as its rules, author, and historical details – tionality can be added by simply implementing
but also graphical information about how the the relevant Java classes, which will then be auto-
game should be visualized. matically incorporated into the grammar the next
The rules section of the Core module is then time that Ludii is launched.
used by the Manager module, which is responsi- Each game description is contained in a plain L
ble for coordinating the moves of the game. This text file with *.lud extension. When a game
module takes input from either a human user, via description is loaded into the Ludii system, the
the Player module, or an Agent, via the AI following steps are performed to compile the
module. description into an executable Game object, as
The equipment section of the Core module is shown in Fig. 1:
used by the ViewController module, which is
responsible for constructing the relevant graphics 1. Expand: The game description is expanded
for the containers and components used in the into a plain text string to resolve certain meta-
game (view) as well as how interface inputs should language features and decorations.
be converted into game moves (controller). Any 2. Parse: The resulting string is parsed for cor-
graphical options specified in metadata are also rectness according to the current Ludii
used by the view section of the ViewController Grammar.
module, to supplement or override what is defined 3. Compile: The Java classes corresponding to
in the equipment. For example, a metadata line the game description keywords are recursively
could be added to the game that changes the size instantiated with the specified parameters to
of a piece or the color of the board. produce an executable Game object in Java
The Player module displays the visuals pro- bytecode.
vided by the ViewController module to the user.
When the user interacts with these visuals, the Internal Model
controller converts these inputs into a logical The game representation and the transitions
move and sends this to the Manager module. between states are described in the following
The Player module provides different visuals and sections.
interaction handling depending on if the user is
operating on their own local version of Ludii Game Representation
(desktop) or the remote website version (web). A game is defined as a 4-tuple of ludemes:
1084 Ludii General Game System for Modeling, Analyzing, and Designing Board Games

Ludii General Game System for Modeling, Analyzing, and Designing Board Games, Fig. 1 Architecture
overview of the Ludii general game system

• Players: provides the information about the • Mode: corresponds to the game control and
players (number of players, direction of each describes if the game is played alternatingly
player, ...). or simultaneously.
Ludii General Game System for Modeling, Analyzing, and Designing Board Games 1085

• Equipment: describes all the information • who[l]: The index of the owner of the compo-
about the containers used in the game (boards nent at l, or 0 if there is no component.
and hands) as well as the components (i.e., the • count[l]: The number components (of a single
game pieces). type) at l.
• Rules: describes the initial state, how different • state[l]: The local state of the component at l, or
components interact with each other, what 0 if there is no component.
moves can be made, and the conditions to • value[l]: The value of the component at l, or
reach a terminal state. 0 if there is no component.
• rotation[l]: The rotation index of the compo-
Figure 2 shows the components of a game in nent at l, or 0 if there is no component.
Ludii.
Different representations are implemented to
State Representation minimize the memory footprint and to optimize
A Ludii game state s encodes which player is to the time needed to access necessary data for
move in s as well as which player was moving in reasoning on any game. These representations
the previous state and which player is going to are:
move in the next state.
Each container of a game is modeled as a • Flat state: For games played on one single site
graph defined by a set of cells C, a set of type without stacking.
vertices V, and a set of edges E. Each playable • Graph state: For games played on multiple site
location l ¼ hci, ti, si, lii is specified by its con- types without stacking.
tainer c ¼ hC, V, Ei, a site type ti . • Stack state: For stacking games played on one
∈ {Cell, Vertex, Edge}, a site index si 0, and single site type. L
a level li 0. Every location specifies a type of site • Graph Stack state: For stacking games played
in a specific container at a specific level. Each on multiple site types.
container of the game has its state representation • Deduction Puzzle state: For puzzles
called container state. corresponding to a Constraint Satisfaction
A container state cs is implemented as a col- Problem (Piette et al. 2019).
lection of data vectors for each playable site. The
different data vectors are: Figure 3 shows the relations between the dif-
ferent state representations. Thanks to these dif-
• what[l]: The index of the component at l, or 0 if ferent state representations, Ludii is able to model
there is no component. a very large set of various games. Figure 4 shows
an overview of the Ludii games library (1019
games in version 1.3.0).
Players Mode

Trial and State Transitions

A Ludii successor function is given by T :
ðS∖Ster Þ  A 7! S, where S 6is the set of all the
Game Ludii game states, Ster the set of all the terminal
states, and A the set of all possible lists of actions,
where a single list of actions is a move that a
player can select. Given a current state s∈S∖Ster ,
Equipment Rules
and a list of atomic actions A ¼ ½ai ∈A, T com-
putes a successor state s0 ∈S.
A trial t is a sequence of states si and action
Ludii General Game System for Modeling, Analyzing,
and Designing Board Games, Fig. 2 The game lists Ai : s0, A1, s1, . . ., sf1, Af, sf such that f 0,
components and for all i ∈ {1, . . ., f},
1086 Ludii General Game System for Modeling, Analyzing, and Designing Board Games

Ludii General Game interface

System for Modeling, State
ContainerState
Analyzing, and
Designing Board Games,
Fig. 3 The different state
representations

DeductionPuzzle Flat Stacks

DeductionPuzzleLarge Graph GraphStacks

Ludii General Game System for Modeling, Analyzing, and Designing Board Games, Fig. 4 Some example
visualizations of games from the Ludii Library

• The played action list Ai is legal for the mover moves of si þ 1 and stores them in the trial for
(si  1). any caller to access them quickly without needing
• States are updated: si ¼ T ðsi1 , Ai Þ. to compute them again.
• Only sf may be terminal: A trial is over when all players are inactive and
s0 , . . . , sf 1 \ S ter ¼ ;. associated with a rank. The outcome of the game
corresponds to the ranking of the players.
When a new state si þ 1 is reached after apply- In short, a trial t provides a complete record of
ing Ai selected from the list of legal moves for a a game played from start to end, including all the
state si, Ludii computes the new list of legal moves made stored in a list. Any reasoning on any
Ludii General Game System for Modeling, Analyzing, and Designing Board Games 1087

game can be parallelized using separate trials per For example, Fig. 5a shows a game with pieces
thread. All the data members of a game are con- played on the vertices, edges, and cells of the board
stant and can therefore be shared between threads. graph. Figure 5b shows a board game played only
A thread will be able to use a trial t to compute on the cells but in which pieces may stack.
any playout from any state. In any given game, a component (or a stack of
In Ludii, a single object called Context is used components) can be placed on any location
to store references to the game, the state represen- corresponding to a graph element and a level.
tation s, and the trial t. For any operation such as Two different graph elements can have differ-
computing the graph of a container, computing the ent relations: Adjacent, Orthogonal, Diagonal,
initial state s0, or computing the legal moves for a Off Diagonal, or All. The complete definition of
state s, Ludii evaluates a tree of ludemes by call- each of these relations is provided in Browne et al.
ing a method eval(context) to evaluate it (2021). These relationships are summarized for
according to the current state. The Context also the regular tilings in Table 1.
contains the random number generator used for Ludii supports the following direction types:
any stochastic operations in the corresponding
trial and the value of the model – alternating or • Intercardinal directions: N, NNE, NE,
simultaneous – used to apply moves or compute ENE, E, ESE, SE, SSE, S, SSW, SW,
the legal moves in a specific game state. WSW, W, WNW, NW, and NNW.
Alternating-move models expect only a single • Rotational directions: In, Out, CW
player to select a move at a time, whereas (clockwise), and CCW (counterclockwise).
simultaneous-move models expect all active • Spatial directions for 3D games: D, DN,
players to select a move at every time step, and DNE, DE, DSE, DS, DSW, DW, DNW
simulations simply apply all legal moves automat- and U, UN, UNE, UE, USE, US, USW, L
ically. More details about the internal model can UW, and UNW.
be found in Piette et al. (2021). • Axial directions subset (for convenience):
N, E, S, and W.
Board Representation • Angled directions subset (for convenience):
In Ludii, the board shared by all players is NE, SE, SW, and NW.
represented internally as a finite graph defined
by a triple of sets G ¼ hV, E, Ci in which V is a Each graph element has a corresponding
set of vertices, E a set of edges, and C a set of cells. set of absolute directions and relative

Ludii General Game System for Modeling, Analyzing, and Designing Board Games, Fig. 5 A game played on
vertices, edges, and cells (a) and a game played only on cells (b)
1088 Ludii General Game System for Modeling, Analyzing, and Designing Board Games

Ludii General Game System for Modeling, Analyzing, and Designing Board Games, Table 1 Relations for the
regular tilings

Relation Square Triangular Hexagonal

All

Adjacent

Orthogonal

Diagonal

Off-Diagonal

directions to associated graph elements of to any relation type (Adjacent,

the same type. Absolute directions can be Orthogonal, Diagonal, Off Diago-
any of the above direction types in addition nal, or All).
Ludii General Game System for Modeling, Analyzing, and Designing Board Games 1089

Relative directions from an element are defined )

by the direction in which a component is facing, )
(end
the number of rightward steps of the component, (if
and the graph relation to use at each step (no Moves Next)
(Adjacent by default). Relative directions are: (result Mover Win)
Forward, Backward, Rightward, )
)
Leftward, FR, FRR, FRRR, FL, FLL, )
FLLL, BR, BRR, BRRR, BL, BLL, or )
BLLL.

Game Logic Metarules

The logic of an L-GDL game (Piette et al. 2021) is In Ludii, a metarule, defined with the ludeme
computed from its rules, ludeme which is defined meta, is a global rule applied in each state
by the following ludemes which correspond to s reached after applying the move decided
different rule types: by the player. That rule can modify the state
s or can add/remove some moves from the list
1. meta, of legal moves. In the Game of the Amazons,
2. start, the swap rule is defined using the metarule
3. play, (swap) (line 12). Here, after the first player
4. end. has finished their turn, the second player has
one more legal move allowing them to swap
with the other player. The metarules are
For example, the following L-GDL code
optional. L
describes the Game of the Amazons
(Amazons: ludii.games/details.php?keyword ¼
Amazons) with an additional swap rule (a rule Starting Rules and Initial State
that allows the second player to swap colors When a game is creating after being compiled, the
after the first move to reduce any first move state s1 corresponds to all the variables set to
advantage). This game is used as example in their default values and no piece placed in any
the next sections. playable location. The starting rules (lines 13–20)
of the Game of the Amazons are used to place the
(game "Amazons" queens on the expected locations to create the
(players 2) initial state. These rules define a list of movements
(equipment { A0 applied to the state s1 to build s0. The starting
(board (square 10))
rules are optional.
(piece “Queen” Each
(move Slide (then (moveAgain)))
) Playing Rules and Move Generation
(piece "Dot" Neutral) The playing rules of a game describe how to
}) generate the legal moves of the mover for any
(rules
(meta (swap))
current state si. These legal moves are defined in
(start { the Play ludeme through its Moves ludemes. The
(place "Queen1" playing rules of the Game of the Amazons are
{"A4" "D1" "G1" "J4"} described in lines 21–26.
)
(place "Queen2"
At each state si, the Moves ludeme used to
{"A7" "D10" "G10" "J7"} describe the playing rules are evaluated according
) to the state and return a list of k legal moves
}) M : hm1, . . ., mi, . . ., mki stored in the trial t.
(play
(if (is Even (count Moves))
As described in Section, the transition between
(forEach Piece) two successive states si and siþ1 is possible as a
(move Shoot (piece "Dot0")) sequence of atomic actions Ai applied to si. Such a
1090 Ludii General Game System for Modeling, Analyzing, and Designing Board Games

sequence is modeled as a move m : ha1, . . ., ai, . . ., effect Moves ludeme (moveAgain) is evalu-
ani, where n is the number of actions in Ai.. ated when a slide move is decided, setting the
An atomic action a is the only operator able to next player be the current mover.
modify the state after its creation. Consequently, • Arithmetic functions return one or many
when a player selects their move m from the list of numerical values. The arithmetic functions
legal moves available in the trial for the state si, are composed of many different functions
this state is updated by applying successively each according to the type of numerical values
atomic action in the list of actions composing the returned (array, integer, range, or real). As
move m. examples, the ludemes (count Sites “Board”)
and (count Players) return the number of sites
Ending Rules and Terminal States in the board, and the number of players,
The ending rules describe when and how play can respectively.
terminate for one or more of the players. In the • Logic functions return a Boolean value. The
Game of the Amazons, the ending rule (lines most common logic functions start by (is ...),
27–32) checks whether the next player has no such as (is Even (count Moves)), which returns
legal moves; if this is the case, the game is over true if the number of moves played so far
and the current player wins. is even.
In Ludii, any conditions to reach an ending • Region functions return one or many playable
state are described in the ending rules, followed sites. The most common region functions start
by the description of the outcome of at least one by (sites ...), such as (sites Board), which
player. In games with two players or fewer, an returns a list of all the sites on the board.
ending rule describes a terminal state ster, but for • Direction functions return one or many abso-
games with more players, the game can continue lute directions. For example, (directions
if play did not yet terminate for at least two of the {Rightward, Leftward}) is returning the abso-
players. lute directions corresponding to the right and
left of the current direction of a piece.
Functions
All the ludemes defining the rules are functions Ludii Player
that are evaluated according to a state s returning a The Ludii Player provides the graphical user inter-
specific type of data. Five types of functions exist face (GUI) aspect of Ludii. This includes both the
in Ludii: visuals and controls for playing games, as well as
additional software options to improve the user/
• Moves functions return a list of moves. The developer experience (e.g., remote online games,
Moves ludemes starting by (move ...) describes a built-in editor, game analysis tools, advanced
a decision move, all the other Moves ludemes graphical settings, etc.). This is something that is
are effect moves. To make the computation of either missing or severely lacking in most other
the legal moves efficient, the effect moves general game systems.
which have to be applied before the decision An example screenshot of the main Ludii
action ad are distinguished from those that Player GUI is shown in Fig. 1. This example
have to be applied after, corresponding to the demonstrates an in progress game of Shogi. The
consequences of the decision and described left side of the player shows the current state of the
using the ludeme (then ...). Due to that distinc- game board. The top right area of the player
tion, only the non-consequence moves are fully displays details about each player, including who
evaluated during the computation of the legal is controlling them and the contents of their hand.
moves, and the consequences are evaluated The bottom right area provides supplementary
only when a specific move has been selected information about the game, such as the moves
to be applied by the player. In the context of the that have been made, ludeme description, agent
slide movement of the Game of Amazons, the analysis results, etc. A range of menu options at
Ludii General Game System for Modeling, Analyzing, and Designing Board Games 1091

the top of the Ludii Player also provides many User Interface
other alternative features. The Ludii Player provides the graphical user inter-
A few of the user-friendly features offered face (GUI) aspect of Ludii, including both the
by the Ludii Player, and their uses for research, visuals and controls for playing games. There
are now described. Firstly, being able to visu- are currently two different version of the Ludii
ally see and play the games described using the Player: the Desktop Player, which is used when
L-GDL makes testing and verifying the correct- running Ludii locally on any standard PC, and the
ness of game descriptions much easier. The Web Player, which is used when interacting via
benefit of this point should not be understated, the Ludii Portal Website.
as there have been several cases of games being
described for alternative systems which were Ludii Desktop Player
later found to be incorrect. Secondly, the heu- An example screenshot of the Ludii Desktop
ristics and strategies of agents can be easily Player GUI is shown in Fig. 6. The left side of the
viewed to see their current performance and if player shows the current state of the game board.
there are any obvious weaknesses in their The top right area of the player displays details
behavior. Humans can also play directly against about each player, including who is controlling
agents to help determine if they are at a human- them and the contents of their hand. The bottom
level playing strength. Lastly, providing a user- right area provides supplementary information
friendly interface is more inviting to the general about the game, such as the moves that have been
public and encourages other game design made, ludeme description, agent analysis results,
enthusiasts to create their own games, leading etc. A range of menu options at the top of the Ludii
to a larger range of games for research pur- Desktop Player also provides many other addi-
poses. Ludii currently includes over 1000 tional features, including but not limited to: L
games which were created by members of the
general public, with new games being added • The ability to play remote games and tourna-
frequently. ments with other Ludii users online.

Ludii General Game System for Modeling, Analyzing, and Designing Board Games, Fig. 6 The graphical user
interface (GUI) of the Ludii Desktop Player for an in progress game of Shogi
1092 Ludii General Game System for Modeling, Analyzing, and Designing Board Games

• A built-in editor for creating, modifying, and benefit of the web version, however, is that it
saving Ludii game descriptions. can be played on almost any device with an
• Game analysis tools for evaluating and com- internet connection and does not require the
paring games across a variety of metrics. user to install Java beforehand.
• Multiple graphical settings, such as anima-
tions, move highlighting, cell coordinates, etc. Ludii Portal
• The ability to select different game options and Both the Ludii Web Player and the download link
rule sets. for the Ludii Desktop Player can be accessed via
the Ludii Portal (ludii.games). This portal also
provides additional information about Ludii and
the games within it. Some of main web pages that
Ludii Web Player can be accessed from this portal include:
An example screenshot of the Ludii Web
Player GUI is shown in Fig. 7. This picture • The Ludii Game Library (ludii.games/library),
was taken from a mobile smartphone device in which displays images and category informa-
portrait mode. Other devices may arrange cer- tion for all games within Ludii. Selecting a
tain elements such as the player hands differ- game from this library will open a Web Player
ently, but are otherwise functionally identical. instance of that game.
As can be seen, the Web Player contains less • The Ludii Downloads Page (ludii.games/
features that the Desktop Player, essentially download), which contains links for down-
only allowing the user to play the game loading the Ludii Desktop Player, as well as
against other AI or human opponents. The other Ludii documentation.

Ludii General Game

System for Modeling,
Analyzing, and
Designing Board Games,
Fig. 7 Ludii Web Player
GUI for mobile devices,
showing Chess
Ludii General Game System for Modeling, Analyzing, and Designing Board Games 1093

• The Game Concepts Search Page (ludii.games/ 2012), GRAVE (Cazenave 2015), MAST
searchConcepts), which can be used to search (Finnsson and Björnsson 2008), Progressive His-
for games with a specific combination of over tory (Nijssen and Winands 2011), and NST (Tak
700 defined concepts. et al. 2012). It also includes training techniques and
• The Ludeme Tree Page (ludii.games/ variants of MCTS that are guided by trained fea-
ludemeTree), which displays an interactive tures, which are described in other publications
hierarchy tree for all the ludemes within the (Browne et al. 2019a; Soemers et al. 2020).
L-GDL. Another search technique implemented in
Ludii is αβ-search (Knuth and Moore 1975),
Artificial Intelligence with MaxN (Luckhardt and Irani 1986), Paranoid
Ludii provides an API for game-playing agents search (Sturtevant and Korf 2000), and BRS+
using any artificial intelligence (AI) techniques to (Esser et al. 2014) extensions for games with
be developed and used to play any of Ludii’s more than two players. Unlike MCTS, these tech-
games from within its GUI-based player as well niques require heuristic evaluation functions –
as command-line programs and competitions generally based on domain knowledge – to com-
(Stephenson et al. 2019). The API for agents pro- pare the “desirability” of various states. A variety
vides them with a forward model; given any of heuristics, most of which were found to be
(current) game state, this may be used to generate fairly generally useful across multiple games
lists of legal moves, generate successor states (Browne 2009), are included in Ludii for this
resulting from the application of moves, query purpose. Typical examples include a material
whether or not a game state is terminal or any heuristic to count weighted sums of types of
rankings have already been determined, and so pieces owned by players or terms that compute
on. This is similar to the API provided by the proximity to board centers, corners, sides, and L
General Video Game AI (GVGAI) framework so on.
(Perez-Liebana et al. 2019) for its collection of
video games. This interface is sufficient for typi- Ludii Database
cal tree search algorithms as commonly used for All relevant information about each official Ludii
GGP, such as Monte-Carlo Tree Search (MCTS) game (i.e., those which are included within the
(Kocsis and Szepesvári 2006; Browne et al. 2012; code Ludii software and repository) is stored
Coulom 2007). There is also support for tensor inside the Ludii Game Database (LGD). This
representations of states and actions to be gener- data can be decomposed into two main types,
ated, the use of which has been demonstrated in a game-related and evidence-related. The
bridge between Ludii and the Polygames evidence-related data is primarily stored only for
(Cazenave et al. 2020) framework of deep learn- games that are relevant to the goals of the DLP. As
ing approaches for games. Various types of con- such, a large portion of games in the LGD does
straints can be specified for agents, such as not have any evidence-related information. As
processing time per move, maximum iteration this information is unlikely to be useful outside
count, and maximum search depth; different con- of this archaeological context, this section will
straints may be more or less suitable for different focus primarily on the game-related data. This
experiments or use cases. game-related data can further be split into three
Based on this interface, several standard algo- subsections: games, rulesets, and ludemes. Each
rithms have already been implemented and game can be thought of as being composed of one
included directly in Ludii, as well as new tech- or more rulesets, with each rulesets being made up
niques developed and proposed specifically in the of multiple ludemes.
context of DLP and Ludii. In GGP, one of the most While the distinction between a game and a
commonly used search algorithms is MCTS. Ludii ruleset is not exact, two sets of rules/equipment
includes implementations of several variants and can be considered different games if they come
common extensions, such as UCT (Browne et al. from different places or existed in different time
1094 Ludii General Game System for Modeling, Analyzing, and Designing Board Games

Ludii General Game Game Rulesets Ludemes

System for Modeling,
Analyzing, and Ruleset A Track
Description
Designing Board Games,
+ Ruleset B Step
Fig. 8 Overview of the
main Ludii Game Database Metadata
Ruleset C Jump
groups and their
relationships Connect

Line

Evidence

Game Evidence Ruleset Evidence Ludeme Evidence

periods. If they cannot be separated, however, then provide the complete opposite. As a result of this
they are considered different rulesets of the same change, it is these rulesets that are evaluated and
base game. This can lead to two rulesets of the analyzed when it comes to gameplay, rather than
same game with very different rules/equipment, the game itself.
such as two different sets of rules that have been
suggested for a historical game with largely Ludemes
unknown rules. It can also lead to two distinct A ludeme is single elemental building block of a
games with very similar rules/equipment, to con- game. Multiple ludemes can be combined
sider the possibility that these games were created together to create a description of a specific
independently in different places and times. piece of equipment or rule that a game uses.
A rough outline of the LGD structure is shown Each ludeme is stored in the LGD and is associ-
in Fig. 8. ated with the rulesets which use it. If a game has
any ruleset that uses a particular ludeme, then by
Games extension, that game will also be considered as
Each game entry in the LGD describes a specific using that ludeme.
game (i.e., a single .lud game description) in
Ludii. Each of these game entries will also have
at least one ruleset associated with it, although it
Cross-References
can have more. Auxiliary metadata information
▶ Monte-Carlo Tree Search
about each game is also stored, such as plain
English descriptions of the game and its rules,
any aliases, publication details, and so on.
References
Rulesets
Browne, C.B.: Automatic generation and evaluation of
A ruleset is a defined set of ludemes which recombination games. Phd thesis, Faculty of Informa-
describe the specific rules and equipment that is tion Technology, Queensland University of Technol-
used to play a certain game. These rulesets could ogy, Queensland, Australia (2009)
Browne, C.: A class grammar for general games. In:
be speculative in nature or can simply be a known Advances in Computer Games, vol. 10068 of Lecture
variant of an established game (e.g., the different Notes in Computer Science, pp. 167–182, Leiden
scoring systems for the game Go). Playing the (2016)
same game with a different ruleset can often lead Browne, C.: Everything’s a ludeme: well, almost every-
thing. In: Proceedings of the XIIIrd Board Game Stud-
to different gameplay experiences. One ruleset
ies Colloquium (BGS 2021), Paris (2021)
could make a game long and biased, with little Browne, C., Powley, E., Whitehouse, D., Lucas, S., Cowl-
room for strategic play while another could ing, P.I., Rohlfshagen, P., Tavener, S., Perez, D.,
Ludosemiotics 1095

Samothrakis, S., Colton, S.: A survey of Monte Carlo Nijssen, J.A.M., Winands, M.H.M.: Enhancements for
tree search methods. IEEE Trans. Comput. Intell. AI multi-player Monte-Carlo tree search. In: van den
Games. 4(1), 1–49 (2012) Herik, H.J., Iida, H., Plaat, A. (eds.) Computers and
Browne, C., Soemers, D.J.N.J., Piette, E.: Strategic fea- Games (CG 2010), vol. 6515 of Lecture Notes in Com-
tures for general games. In: Proceedings of the 2nd puter Science, pp. 238–249. Springer, Kanazawa,
Workshop on Knowledge Extraction from Games Japan (2011)
(KEG), pp. 70–75 (2019a) Perez-Liebana, D., Liu, J., Khalifa, A., Gaina, R.D.,
Browne, C., Soemers, D.J.N.J., Piette, È., Stephenson, M., Togelius, J., Lucas, S.M.: General video game AI: a
Conrad, M., Crist, W., Depaulis, T., Duggan, E., Horn, multitrack framework for evaluating agents, games,
F., Kelk, S., Lucas, S.M., Neto, J.P., Parlett, D., and content generation algorithms. IEEE Trans.
Saffidine, A., Schädler, U., Silva, J.N., de Voogt, A., Games. 11(3), 195–214 (2019)
Winands, M.H.M.: Foundations of digital Piette, C., Piette, É., Stephenson, M., Soemers, D.J.N.J.,
archæoludology. Technical report, Schloss Dagstuhl Browne, C.: Ludii and XCSP: playing and solving
Research Meeting, Germany (2019b) logic puzzles. In: 2019 IEEE Conference on Games
Browne, C., Piette, É., Stephenson, M., Soemers, D.J.N.J.: (CoG), pp. 630–633 (2019)
General board geometry. In: Advances in Computer Piette, É., Browne, C., Soemers, D.J.N.J.: Ludii game logic
Games (ACG 2021) (2021) guide. https://arxiv.org/abs/2101.02120 (2021)
Cazenave, T.: Generalized rapid action value estimation. Soemers, D.J.N.J., Piette, É., Stephenson, M., Browne, C.:
In: Yang, Q., Woolridge, M. (eds.) Proceedings of the Manipulating the distributions of experience used for
Twenty-Fourth International Joint Conference on Arti- self-play learning in expert iteration. In: Proceedings of
ficial Intelligence (IJCAI 2015), pp. 754–760. AAAI the 2020 IEEE Conference on Games, Osaka, Japan,
Press, Buenos Aires, Argentina (2015) pp. 245–252. IEEE (2020)
Cazenave, T., Chen, Y.-C., Chen, G.W., Chen, S.-Y., Chiu, Stephenson, M., Piette, É., Soemers, D.J.N.J., Browne, C.:
X.-D., Dehos, J., Elsa, M., Gong, Q., Hu, H., Khalidov, Ludii as a competition platform. In: Proceedings of the
V., Li, C.-L., Lin, H.-I., Lin, Y.-J., Martinet, X., Mella, 2019 IEEE Conference on Games (COG 2019),
V., Rapin, J., Roziere, B., Synnaeve, G., Teytaud, F., pp. 634–641, London (2019)
Teytaud, O., Ye, S.-C., Ye, Y.-J., Yen, S.-J., Zagoruyko, Sturtevant, N.R., Korf, R.E.: On pruning techniques for
S.: Polygames: improved zero learning. ICGA J. 42(4), multi-player games. In: Proceedings of the Seventeenth L
244–256 (2020) National Conference on Artificial Intelligence and
Coulom, R.: Efficient selectivity and backup operators in Twelfth Conference on Innovative Applications of
Monte-Carlo tree search. In: van den Herik, H.J., Artificial Intelligence, pp. 201–207. AAAI Press, Aus-
Ciancarini, P., Donkers, H.H.L.M. (eds.) Computers tin, Texas (2000)
and Games, vol. 4630 of LNCS, pp. 72–83. Springer, Tak, M.J.W., Winands, M.H.M., Björnsson, Y.: N-grams
Turin, Italy (2007) and the last-goodreply policy applied in general game
Esser, M., Gras, M., Winands, M.H.M., Schadd, M.P.D., playing. IEEE Trans. Comput. Intell. AI Games. 4(2),
Lanctot, M.: Improving best-reply search. In: van den 73–83 (2012)
Herik, H., Iida, H., Plaat, A. (eds.) Computers and
Games. CG 2013, vol. 8427 of Lecture Notes in Com-
puter Science, pp. 125–137. Springer, Cham (2014)
Finnsson, H., Björnsson, Y.: Simulation-based approach to
general game playing. In: The Twenty-Third AAAI Ludo Game
Conference on Artificial Intelligence, pp. 259–264.
AAAI Press, Chicago, Illinois (2008)
Knuth, D.E., Moore, R.W.: An analysis of alpha-beta prun-
▶ Augmented Reality Ludo Board Game with
ing. Artif. Intell. 6(4), 293–326 (1975) Q-Learning on Handheld
Kocsis, L., Szepesvári, C.: Bandit based Monte-Carlo plan-
ning. In: Fürnkranz, J., Scheffer, T., Spiliopoulou,
M. (eds.) Machine Learning: ECML 2006, vol. 4212
of Lecture Notes in Computer Science (LNCS),
pp. 282–293. Springer, Berlin, Heidelberg (2006) Ludonarrative
Kowalksi, J., Miernik, R., Mika, M., Pawlik, W., Sutowicz,
J., Szykula, M., Tkaczyk, A.: Efficient reasoning in ▶ Narrative in Video Games
regular boardgames. In: Proceedings of the 2020
IEEE Conference on Games, pp. 455–462. IEEE,
Osaka, Japan (2020)
Luckhardt, C.A., Irani, K.B.: An algorithmic solution of
n-person games. In: Proceedings of the Fifth AAAI Ludosemiotics
National Conference on Artificial Intelligence,
pp. 158–162. AAAI Press, Philadelphia, Pennsylvania
(1986) ▶ Semiotics of Computer Games
M

Machine Intelligence Definitions

▶ Computer Games and Artificial Intelligence Video game modding: It is the process of modifi-
cation of a video game by players or fans, for
changing the gaming experience by allowing
others to get and use these modifications, which
Machine Learning are referred to as “mods.”
Ensemble learning: Use of multiple learning
▶ Audio and Facial Recognition CAPTCHAs for algorithms to obtain better performance than that
Visually Impaired Users obtained by using any of these algorithms in
▶ Classical Learning Method in Digital Games isolation
▶ Fall Risk Detection in Computer Vision Transfer learning: It is the process of reusing
▶ Human Interaction in Machine Learning (ML) knowledge from previously learned source tasks
for Healthcare to bootstrap learning of target tasks (Braylan
et al. 2016).

Machine Learning for Introduction

Computer Games
There are multiple ways in which some or all of
Amol D. Mali the research at the intersection of machine learn-
Computer Science Department, University of ing and computer games can be surveyed. One can
Wisconsin-Milwaukee, Milwaukee, WI, USA survey the applications of one machine-learning
paradigm in computer games in one genre. One
can survey the applications of multiple machine-
Synonyms learning paradigms in computer games in one
genre. One can survey the applications of one
Clustering; Deep learning; Game AI; Hidden machine-learning paradigm in computer games
Markov Models; Learning agents; Nearest neigh- from multiple genres. One can survey the appli-
bors; Neural networks; Regression; Reinforce- cations of multiple machine-learning paradigms
ment learning in computer games from multiple genres. Justesen

© Springer Nature Switzerland AG 2024

N. Lee (ed.), Encyclopedia of Computer Graphics and Games,
https://doi.org/10.1007/978-3-031-23161-2
1098 Machine Learning for Computer Games

and others have co-authored such a survey DEEP SPECTRUM features using Bag-of-
(Justesen et al. 2020). Machine-learning frame- Audio-Words. They considered recognition of
works, representations, architectures, and algo- the following six genres: Action or Shooter,
rithms covered in their survey include Deep Arcade or Platform, Fighting, Racing, Sports,
Q-network (DQN), Distributional DQN, Deep and Simulation or World Building. They trained
recurrent Q-learning (DRQN), Hierarchical a linear SVM (support vector machine) classifier.
DQN (h-DQN), LSTM (long short-term memory) They found that Racing games were the easiest to
network, hierarchical deep reinforcement- recognize due to automotive noises and Simula-
learning network (H-DRLN), Neural Turing tion and World Building games were the most
Machines (NTMs), recurrent memory Q-network difficult to recognize.
(RMQN), feedback recurrent memory Q-network
(FRMQN), teacher-student curriculum learning
(TSCL) framework, Independent Q-learning Game Balancing
(IQL), multiagent bidirectionally coordinated
network (BiC-Net), convolutional neural- A game is unbalanced when it is too easy or too
network-fitted Q-learning (CNNFQ), LSTM- hard for the human player. This imbalance can
DQN architecture, deep reinforcement relevance cause the player to lose interest in playing the
net (DRRN), deep distributed recurrent game. According to Andrade and others (2005),
Q-network (DDRQN), Double DQN, dueling dynamically balancing a game using reinforce-
DQN, and Prioritized DQN. Q-learning is a ment learning by giving a negative reward to the
reinforcement-learning algorithm. computer-controlled agent when the game is too
One can survey genre-independent applica- easy or too hard is not a good solution to the
tions of one machine-learning paradigm in com- game-balancing problem, as this solution can
puter games. One can survey genre-independent result in unbelievable behaviors like the
applications of multiple machine-learning para- computer-controlled agent not using defense at
digms in computer games. A survey can also be all after hitting the character controlled by the
driven by tasks related to computer games and human player. Their challenge-sensitive action-
machine learning. So one can survey uses of selection mechanism requires the computer-
machine learning to solve tasks related to com- controlled agent to periodically use the challenge
puter games. This chapter is a survey that covers function to evaluate if it is at the same level as the
some of the important applications of different player. According to the challenge function pro-
machine-learning paradigms to computer games posed by them for a fighting game, whether the
such that the methodology in these applications is game is easy, moderately hard, or difficult,
either genre-independent or can be generalized to depends on the difference in the lives of the
make it applicable to additional genres. This computer-controlled character and the human-
survey is organized by the abstract computer- controlled character. They used modified rein-
game-related tasks addressed by one or more forcement learning. According to their approach,
machine-learning paradigms. the computer-controlled agent progressively
chooses sub-optimal actions if the game is too
hard. Their modification to reinforcement learn-
Recognizing Genre Based on Audio ing resulted in the computer-controlled agent
exhibiting win-loss balance.
Amiriparian and others (2020) introduce game-
audio-based genre recognition and present an
approach to solve this problem. They extract the Testing Hypotheses
following three feature representations from
game-audio files: knowledge-based acoustic fea- Akbar and others (2021) used ensemble learning
tures, DEEP SPECTRUM features, and quantized to test if inclusion of adult content allows a mod to
Machine Learning for Computer Games 1099

get more endorsements. The ensemble included in (Dey and Child 2013). Their approach priori-
recurrent neural networks (RNNs) and decision tizes the behaviors in a behavior tree based on
forest models. They found that adult content their utilities.
played an important role in determining the
endorsement count of a mod.
Behavior Cloning

Transferring Learning for Fulfilling Since playing against superhuman opponents can
New Tasks be frustrating to human players, Diels and Kazmi
(2021) propose cloning behavior of human
Braylan and others (2016) propose GRUSM players to create realistic opponents. They evalu-
(General ReUse of Static Modules) for transfer ate a behavioral clone of a game-playing agent by
learning. A GRUSM network is a 3-tuple G’ ¼ comparing action distributions and by comparing
(M’, S0 , T’) where M’ is a traditional neural play-style data. They tried to clone a
network (feedforward or recurrent) containing reinforcement-learning agent that was not allowed
the new nodes and connections unique to the to improve after the training phase, a random
target task, with input and output nodes agent sampling actions uniformly, and a human
corresponding to inputs and outputs defined by player. They show that cloning the behavior of a
the target domain; S0 is a (possibly empty) set of human player is more difficult than cloning the
pointers to recruited source networks; and T’ is a behavior of other agents since humans play least
set of weighted transfer connections between consistently.
nodes in M’ and nodes in source networks
(Braylan et al. 2016). Each source network can
be a traditional neural network or a GRUSM Learning from Visual Game Display
network. Source networks are existing neural
M
modules. NPCs (non-playing characters) are broadly
defined as visible components of a game that are
under the control of the computer, and that either
Improving Behavior Trees work with or against the human player (Fink et al.
2007). Fink and others use machine learning to
A behavior tree represents behaviors at multiple learn the behavior of NPCs from just the graphical
levels of abstraction such that the behaviors at output of the game generated during game play.
the lowest level are executable. Behavior trees They claim that this can be useful for knowing
can be used in computer games for controlling what the human player may learn during game
computer-controlled characters (non-playing play. The learnt behavior can be compared with
characters). The root of a behavior tree for the implemented behavior to know how well the
preys in (Dey and Child 2013) is the selector human player can learn the implemented behav-
node. Its children are Retreat, Idle, and Attack. ior. They claim that their method can be used to
The children of Retreat are Flee and SeekSafety. re-engineer the game if game code is not avail-
The children of Idle are Graze and Explore. The able. They use a similarity function and nearest-
children of Graze are Forage and Eat. The chil- neighbor rule.
dren of Explore are Flock and Wander. The chil-
dren of Attack are Charge and Assist. Assist
means attacking the nearest predator agent Personalizing Challenges
targeting a neighboring prey agent. The main
goal in (Dey and Child 2013) is development of The framework (Georgiou and Demiris 2016) for
an approach for suggesting changes to a behavior altering the segments of a racing track according
tree, to assist game designers. Q-learning is used to model of the user uses linear regression for
1100 Machine Learning for Computer Games

finding weights of physiological and non- computer games, and simulations. It is much eas-
physiological metrics. Physiological metrics ier to create diverse and unforeseen roads and
include metrics based on eye tracking and head surroundings fast at a much lower cost in virtual
pose. Non-physiological metrics include metrics world, to get a large amount of data for machine
obtained from player inputs and game outputs. learning for autonomous driving.

Selecting Animations Better Recognizing Social Context

Gillies (2009) points out that the individuality of a Li and others (2010) introduce a multiplayer, col-
real actor’s performance should be reflected not laborative version of Tetris. Each of the two
only in the appearance and animation of the char- players can control one of the two falling pieces.
acter, but also in the AI that controls the character’s Each player played on a separate computer sepa-
behavior and interactions, in games containing rated by a shelf in their experiments. They
characters based on real actors. I-O HMM (input- recorded the players’ eye movements. The four
output hidden Markov model) is used in this work social contexts considered by them were NN
to learn the transition probabilities for the finite- (a novice playing with another novice), NE
state machine and the function that selects edges of (a novice playing with an expert), EN (an expert
the motion graph based on the current state. playing with a novice), and EE (an expert playing
A motion graph allows animations to be generated with another expert). EN and NE are different
by sequencing preexisting animations. Each edge categories since the type of the first player in
of this graph is associated with an animation clip. these combinations is different, and this matters
There is a probability for each edge of the motion in interpretation of the statistical information
graph. A node of the motion graph is a point at found for the first player in a pair. They found
which an animation clip can be selected. that gaze on self was strongest for players in NN
pairs. They found that gaze on self was weakest
for expert players when playing with novice
Making Agents Human-Like players. Action features they used were intended
to capture playing style. Action features were
To make computer-controlled characters believ- found using type of moves (rotations, translations,
able by having them move like human-controlled drops, and downs) and distances associated with
characters, Gorman and Humphrys (2006) these moves on the grid. They used SVM (support
recorded locations of the character controlled by vector machine) classifiers. They show that it is
the human player and clustered these locations. possible to recognize social context based on gaze
Each of these clusters is a node in the directed and action features.
graph which has an edge from node i’ to node j’ if
the human-controlled character was observed to
move from i’ to j’. They used reinforcement learn- Relating Game-Related Preferences to
ing. The rewards were designed to encourage the Social-Network Data
computer-controlled character to follow the routes
of the human-controlled character. Lim and Harrell (2013) used social-networking
information to predict the likelihood of players
customizing their profile in several ways associ-
Using Games to Improve Machine ated with the monetary values of the players’
Learning avatar. They used clustering and SVMs. They
show that a strong relationship exists between a
Greengard (2017) is a brief review of work relat- player’s real-world identity and virtual identity
ing autonomous driving, machine learning, within games.
Machine Learning for Computer Games 1101

Automated Story Generation There can be multiple cameras, e.g., chasing cam-
era, front camera, and high-view camera. The
Martin and others (2018) define automated story camera used affects what the human player sees
generation as the problem of automatically on the screen, and this affects gaming experience
selecting a sequence of events, actions, or words, considerably. The approach in (Passos et al. 2009)
that can be told as a story. They formalize auto- uses a neural network for each camera and decides
mated story generation as sampling from the prob- which camera to use in the current situation based
ability distribution over successor events when a on the classifications provided by the neural
sequence of events is given. They use a recurrent networks.
encoder-decoder network to produce the probabil-
ity distribution. They introduce event2sentence,
which is a neural network for translating an Detecting and Adapting to Novelty
event into natural language. in Games

Peng and others (2021) point out that policies of

Generation of Game Stages reinforcement-learning agents are often fine-tuned
to the mechanics of the environment, but assum-
Nam and others (2022) train reinforcement-learning ing that the state space, action space, and transi-
agents to generate diverse and enjoyable stages of a tion dynamics of the world do not change over the
game. A stage is a series of events in a turn-based life of the agent is not practical in many domains.
role-playing game. They represent each stage using They define open-world novelty as a change in the
a matrix containing real numbers. An empty stage is observation types, action types, environment
represented by a matrix filled with zeros. Actions fill dynamics, or any other change in the world
an incomplete stage matrix sequentially by deciding mechanics that makes it different from the world
one or more of its elements. The stage-evaluation originally used to train the agent. They propose a
M
function serves as the reward function. deep-reinforcement-learning technique for
detecting and adapting to open-world novelty in
games. Their system maintains two knowledge
Predicting Opponent’s Strategy graphs. One of them represents the state of the
world and the other represents the rules of the
StarCraft is a real-time strategy (RTS) game requir- game. Their system detects novelty by detecting
ing tactics, strategies, and resource management. a change to static entities and relations of the state-
Opponent’s territory is not fully visible in this war knowledge graph or to the rule graph.
game due to fog. Park and Cho’s approach (Park and
Cho 2012) involves sending a scouting unit to the
opponent’s territory to find information about the Automated Creation of Faces of Game
opponent’s structures and units, so that it can be Characters
used for machine learning. They used 13 machine-
learning algorithms. These included algorithms using Shi and others (2022) point out that customizing
decision trees and neural networks. Opponent’s strat- the face of a character is very time-consuming
egy is predicted to be fast or slow. The opponent’s even for experienced players as there are too
strategy is fast if it attacks first, otherwise it is slow. many parameters whose values need to be
adjusted. Their method automatically creates
face of a character based on a single photograph
Camera Selection of face. Their method transforms the input
image into a set of in-game facial parameters.
Passos and others (2009) demonstrate an intelli- They use deep convolutional neural networks
gent editor agent for deciding shot transition. (CNNs).
1102 Machine Learning for Computer Games

Detecting Player Burnout follows: dribbling (one player possesses the ball
and the other player follows), given-and-go (the
Smerdov and others (2021) point out that phys- ball is passed back and forth between two
iological data may provide new insights into a players), ball retrieval (one player runs toward
player’s behavior. They assert that though the resting ball and passes it to the other player
in-game data can be useful in player-skill assess- right away), long pass (the ball is passed over a
ment, match analysis, match-outcome predic- longer distance and not returned), and solo (the
tion, and post-match analysis, models trained player with the ball does not pass the ball but
on in-game data can become obsolete because constantly moves in one direction, and the other
of new patches or changes in game mechanics. player runs alongside).
They collected sensor data from humans playing
eSports. This data is about pulse, saccades (rapid
eye movements between fixation points), muscle References
activity, galvanic skin response (GSR), mouse
activity, keyboard activity, facial-skin tempera- Akbar, G., Tandra, V., Qomariyah, N.: Skyrim game mods
ture, electroencephalogram (EEG), environmen- endorsement prediction with machine learning. In: Pro-
tal temperature, relative humidity, carbon- ceedings of International Seminar on Machine Learn-
ing, Optimization, and Data Science (ISMODE),
dioxide level, pulse-oximeter data, and move-
pp. 157–162 (2021)
ments of chair, hand, and head. They used this Amiriparian, S., Cummins, N., Gerczuk, M.,
data and machine learning to predict outcomes of Pugachevskiy, S., Ottl, S., Schuller, B.: “Are you
encounters extracted from in-game logs. They playing a shooter again?!” Deep representation learn-
ing for audio-based video game genre recognition.
used logistic regression, KNN (k-nearest neigh- IEEE Trans. Games. 12(2), 145–154 (2020)
bors algorithm), GRU (gated recurrent unit) neu- Andrade, G., Ramalho, G., Santana, H., Corruble, V.:
ral network, and transformer neural network. Challenge-sensitive action selection: an application in
Their observations include the following: game balancing. In: Proceedings of the IEEE/WIC/
ACM International Conference on Intelligent Agent
(i) The feature indicating good in-game activity
Technology (IAT), pp. 194–200 (2005)
the most is the intensity of gamma waves for the Braylan, A., Hollenbeck, M., Meyerson, E., Miikkulainen,
T7 electrode located near the left temple. R.: Reuse of neural modules for general video game
(ii) Higher heart rate and eye activity increase playing. In: Proceedings of the 30th AAAI (American
Association for Advancement of Artificial Intelligence)
the player’s performance. (iii) The feature most Conference, pp. 353–359 (2016)
important in predicting that the player is likely to Dey, R., Child, C.: QL-BT: enhancing behavior tree design
lose the next fight is vertical movement of the and implementation with Q-learning. In: Proceedings
chair. (iv) Higher environmental temperature, of the IEEE Conference on Computational Intelligence
in Games (CIG) (2013)
humidity, and carbon-dioxide level decrease the Diels, L., Kazmi, H.: Behaviorally cloning river raid
player’s performance. (v) Higher facial-skin tem- agents. In: Proceedings of AAAI (Association for
perature, GSR, and pupil diameter lower the Advancement of Artificial Intelligence) Conference
probability of winning the next fight, as these (2021)
Fink, A., Denzinger, J., Aycock, J.: Extracting NPC behav-
are related to a lower concentration, higher men- ior from computer games using computer vision and
tal load, and stress. machine learning techniques. In: Proceedings of the
IEEE Symposium on Computational Intelligence and
Games (CIG), pp. 24–31 (2007)
Georgiou, T., Demiris, Y.: Personalized track design in car
Classification of Team Behaviors racing games. In: Proceedings of IEEE Conference on
Computational Intelligence in Games (CIG),
A Hidden Markov Model (HMM) is trained for pp. 1–8 (2016)
each team behavior (Thurau et al. 2006). They Gillies, M.: Learning finite-state machine controllers from
motion capture data. IEEE Trans. Comput. Intell. AI
classified observation data into one of five behav- Games. 1(1), 63–72 (2009)
iors. The names of these behaviors followed by Gorman, B., Humphrys, M.: Towards integrated imitation
their descriptions from (Thurau et al. 2006) are as of strategic planning and motion modeling in
Madden NFL and Infinite Inspiration 1103

interactive computer games. ACM Comput. Entertain.

4(4), 1–13 (2006) Madden NFL and Infinite
Greengard, S.: Gaming machine learning. Commun.
ACM. 60(12), 14–16 (2017) Inspiration
Justesen, N., Bontrager, P., Togelius, J., Risi, S.: Deep
learning for video game playing. IEEE Trans. Games. Matthew Clark2 and Newton Lee1,2
12(1), 1–20 (2020) 1
Institute for Education, Research, and
Li, W., Nussli, M.-A., Jermann, P.: Gaze-quality-assisted
automatic recognition of social contexts in collabora- Scholarships, Los Angeles, CA, USA
2
tive Tetris. In: Proceedings of International Conference Vincennes University, Vincennes, IN, USA
on Multimodal Interfaces and the Workshop on
Machine Learning for Multimodal Interaction (ICMI-
MLMI), pp. 1–8 (2010)
Lim, C.-U., Harrell, D.: Modeling player preferences in Synonyms
avatar customization using social network data. In:
Proceedings of the IEEE Conference on Computational Sports video game
Intelligence in Games (CIG) (2013)
Martin, L., Ammanabrolu, P., Wang, X., Hancock, W.,
Singh, S., Harrison, B., Riedl, M.: Event representa-
tions for automated story generation with deep neural Definition
nets. In: Proceedings of the 32nd AAAI (Association
for Advancement of Artificial Intelligence) Confer- Sports A genre of games that replicates how a
ence, pp. 868–875 (2018)
Nam, S.-G., Hsueh, C.-H., Ikeda, K.: Generation of game video sport is played in a video game format
stages with quality and diversity by reinforcement game
learning in turn-based RPG. IEEE Trans. Games.
14(3), 488–501 (2022) History of Madden NFL
Park, H., Cho, H.-C.: Prediction of opponent’s early-stage
strategy for StarCraft AI using scouting and machine
learning. In: Proceedings of the Workshop at The lens of infinite inspiration proposes that
SIGGRAPH Asia (WASA), pp. 7–12 (2012) instead of looking at a game for ideas, one should
M
Passos, E., Montenegro, A., Clua, E., Pozzer, C., Azevedo, look at everything else. If one looks only at a
V.: Neuronal editor agent for scene cutting in game
cinematography. ACM Comput. Entertain. 7(4), game or games like the one they are creating, the
1–57 (2009) game might seem like an imitation. Instead, look
Peng, X., Balloch, J., Riedl, M.: Detecting and adapting to for inspiration in the world. In Jesse Schell’s book
novelty in games. In: Proceedings of AAAI “The Art of Game Design: A Book of Lenses,” the
(Association for Advancement of Artificial Intelli-
gence) Conference (2021) Lens of Infinite Inspiration states that “to use this
Shi, T., Zou, Z., Shi, Z., Yuan, Y.: Neural rendering for lens, stop looking at your game, and stop looking
game character auto-creation. IEEE Trans. Pattern at games like it. Instead, (Schell, 2019) look
Anal. Mach. Intell. (PAMI). 44(3), 1489–1502 (2022) everywhere else.” The Lens of Infinite Inspiration
Smerdov, A., Somov, A., Burnaev, E., Zhou, B., Lukowicz,
P.: Detecting video game player burnout with the use of asks the following questions:
sensor data and machine learning. IEEE Internet Things
J. 8(22), 16680–16691 (2021) 1. What is an experience I have had in my life that
Thurau, C., Hettenhausen, T., Bauckhage, C.: I would want to share with others?
Classification of team behaviors in sports video
games. In: Proceedings of the 18th International 2. In what small way can I capture the essence of
Conference on Pattern Recognition (ICPR), that experience and put it into my game?
pp. 1188–1191 (2006)
Madden NFL 19 is a sports video game about
the professional American football league. This
game captures the experience of a game that some
Macroscopic Simulation cannot play due to physical handicaps or a lack of
means. The game inspires young children to go
▶ Crowd Evacuation Using Simulation out and try football in real life. It was created
Techniques because someone wanted to share an experience
1104 Making Virtual Reality (VR) Accessible for People with Disabilities

they had with other people (Madden NFL 19, willing to create new technology to allow for a
2018). better experience.
The next part of the lens of infinite expression To make a game something special, it needs to
is conveying an experience into the game. In what have an inspiration, such as personal experience.
ways can an experience in the game match the Imitating other people’s inspiration can lead to a
experience the players will feel? The four ele- game that lacks emotion. Use an experience and
ments that need to be conveyed are mechanics, the four basic elements to create a game from a
story, esthetic, and technology. unique inspiration.
Mechanics are rules and controls that the game
will have; its what makes a game feel real. Mad-
den NFL 19 uses many mechanics to enrich the
References
gameplay. The game has a mechanic the allows
players to perform skill moves. Skill moves are Madden NFL 19 New Features – EA SPORTS. Retrieved
things like hurdling over a defender or attempting September 9, 2018, from https://www.easports.com/
to knock the ball out of the offender’s hands. madden-nfl/features
These mechanics give the player a glimpse of Schell, J.: The Art of Game Design: A Book of Lenses. A K
Peters (2019)
what it is like to play in a professional football
game. Using mechanics to imitate real life expe-
riences makes a game interesting and unique.
The next element that is used to capture the
experience is the story. This can enhance the Making Virtual Reality
experience by allowing the player to make (VR) Accessible for People
choices throughout the game that will determine with Disabilities
the ending. In Madden NFL 19, there is a story
mode that has a young man fighting his way up Marco Antonio Martínez Cano, Carolina Padilla
the ladder to play in the professional league. The Velasco and Steve Bakos
player gets a feel on the experience many young Ontario Tech University, Oshawa, ON, Canada
men have as they struggle to play professional
football. The story is about the life of the game
that is portrayed. Synonyms
The esthetic is how a game looks and sounds.
The art and soundtrack will immerse the player Accessibility; Augmented reality; Blindness;
into the setting of the game to bring out the expe- Cognitive disabilities; Learning disabilities;
rience through their senses. Madden NFL 19 Mixed reality; Physical disabilities; Sensory
makes the art as realistic as possible. The devel- disabilities; Virtual reality; Visual impairment
opers scanned the players to make the in-game
models nearly identical to the real players. They
made realistic stadiums with a soundtrack that Definition
enhances the experience. If the games’ art and
sound match the experience, the game will Virtual reality (VR) is a simulated digital environ-
stand out. ment that obstructs the real world, creating an
The technology is anything that makes the immersive experience that can imitate real-world
game possible. Inspiration can allow for new tech- physical properties.
nology for a game to be developed. In Accessibility in technology refers to the prac-
Madden NFL 19, the developers made use of tice of making technology and digital environ-
microphones to allow the play to give voice com- ments usable by as many people as possible.
mands to the game. An example of this is saying Physical disabilities are medical conditions
time-out so the game will call time-out for you. that diminish an individual’s capacity for physical
Technology can make a game very unique if one is movement.
Making Virtual Reality (VR) Accessible for People with Disabilities 1105

Cognitive disabilities or disorders are medical will experience a disability in some form or
conditions that affect an individual’s brain func- another at some point of their life. Disabilities
tion by limiting the brain’s capacity to collect, can take various forms, physical, cognitive, sen-
process, understand, or store information. sory, or other. Each disability brings its challenges
Sensory disabilities are neurological disorders to the individual’s life and unique accessibility
that affect the human brain’s ability to process needs for the user to engage with VR. Headsets
sensory information, that is, sight, hearing, are often bulky, causing accessibility issues for
touch, taste, and smell. users with weak neck and shoulder strength.
According to The 360 Guy (2022), VR headsets
range in weight from 468 grams (1 lb, 0.5 oz) to
Introduction 644 g (1 lb, 6.7 oz). Users who cannot support this
weight while performing actions required by the
Virtual reality (VR) requires the use of special- VR application have few options.
ized hardware to allow the user to partake in an Motion controllers often require the user to
interactive virtual experience. Examples of this hold them at specific positions, point them in a
hardware include headsets and motion control- direction, and make mid-air motions (Heilemann
lers to interact with the virtual environment. et al. 2021). Disabilities that prevent someone
Handling a headset requires the usage of the from raising their arms and moving their hands
user’s body parts, like neck and shoulders, and can prevent them from using motion controllers
upper body strength to support it while being and many VR applications that depend on these
susceptible to movement. These headsets rely controllers for input. An implicit assumption of
on the user’s eyes, often leaving little-to-no these motion controllers is that the user has one or
room for glasses. In addition, motion controllers both hands available and has fine and gross motor
require the use of the user’s hands and arms to control to perform precise and large movements
take part in most VR gaming experiences, imme- (Mott et al. 2020). This may not always be the
M
diately excluding disabled users who may not be case, as many physical disabilities can severely
able to use this hardware as intended, keeping limit an individual’s ability to perform these
people with disabilities from participating in required motions, thus preventing them from
most VR experiences. However, as VR technol- engaging with VR and presenting a significant
ogy becomes more popular and its development barrier to entry.
and techniques are further explored, some alter- Sensory disabilities can also impact the user’s
natives have been proposed to make sure people ability to partake in VR. Users with visual disabil-
with disabilities can experience VR. Through the ities may find text instructions difficult to read,
article, examples of accessible approaches for the requiring the application to support descriptive
development of VR gaming will be presented, audio or additional audio cues to guide them.
focusing on people living with physical impair- However, including these additional accessibility
ments, visual impairments, and cognitive features is a choice made by the developers of the
disabilities. application. Conversely, users with auditory dis-
abilities can experience difficulty hearing impor-
tant sounds or speech and may depend on the
Disabilities and Their Impact application’s developers, including subtitles as
an accessibility aid.
According to the World Health Organization in Each manufacturer of VR equipment has their
World Health Organization (2021), over one bil- standards, locking games to specific hardware that
lion people (15% of the world’s population) are may not support the accessibility features a dis-
living with some form of disability. This number abled user requires (W3C Working Group 2021).
is growing because of demographic trends, the This presents an additional barrier to entry as not
increasing rate of chronic health conditions, and having the accessibility of VR hardware standard-
other causes. Therefore, it is likely that everyone ized across different manufacturers leaves the
1106 Making Virtual Reality (VR) Accessible for People with Disabilities

inclusion of accessibility features to each manu- environments. People with disabilities that cannot
facturer’s discretion. hold motion controllers can now engage with VR
by using their hands as the controller to a limited
degree. Integrating a Microsoft Kinect 2.0 or
VR Gaming for People with Physical Kinect 360 allows a user’s hands to act as a
Impairment motion controller for environments requiring a
player “touching” something within them. The
Physical disabilities can prevent individuals from VR environment now uses the position of the
using VR for a variety of reasons. Depending on user’s hands in place of the controller, thus
the disability, different actions required by the enabling people with disabilities that prevent
virtual environment can be difficult to impossible them from holding or grasping a motion controller
for a disabled person to perform. Because the VR to engage with VR (WalkinVRDriver 2022d).
headset tracks the position of the user’s head in 3D The VR game Half-Life: Alyx, developed by
space and various applications expect the user to Valve Corporation, is an example of a game with
stand, people who cannot stand negatively impact many accessibility options for physically disabled
their experience. users. Players can choose their dominant hand to
WalkinVR is an assistive tool that enables a wield weapons and play in a seated mode where
third party to alter the position of the VR user in crouching and standing are done with the control-
3D space, use action buttons on the controller, or ler instead of physical movements (Rad 2020).
grip game objects (WalkinVRDriver 2022a). For The game provides additional accessibility fea-
individuals that cannot perform actions such as tures such as subtitles and closed captions, reduc-
standing, crouching, or lying prone, WalkinVR ing the strength and flickering of lights in-game,
also provides assistive functionality that enables and disabling “Barnacle Lift” (i.e., preventing an
users to create additional controller bindings that in-game enemy from lifting the player character
mimic these actions. By adding this functionality, off the ground. Instead, the player takes damage
users with disabilities that prevent these actions and remains in place).
can partake in environments that require them Half-Life: Alyx also lets users change how
(WalkinVRDriver 2022b). they select their weapons from the in-game con-
Another accessibility feature WalkinVR text menu. Instead of using their hand to point at
provides is adjusting the position of a motion con- and select a weapon, the user can do this using
troller in VR. Environments can have physical their head. Disabled users with difficulty with fine
expectations of the user, such as raising the motion motor skills have other options that may be more
controller to a specific height (e.g., eye or shoulder accessible to them.
level) or extending the motion controller a distance
away from their body, which a disabled user may
find difficult or impossible. The user can also VR Gaming for People with Learning
change the scaling of real-world movements to Disabilities
virtual movements (i.e., an 8 cm movement on the
real controller can translate to a 30 cm movement The approach of adopting video games as a way
within the virtual environment). WalkinVR’s for people with cognitive disabilities to develop or
functionality of adjusting the motion controller’s improve learning skills has been around for a
position and increasing the scaling of real-world couple of years, and this approach has been
to virtual moments enables disabled users to over- applied to different types of disabilities such as
come these barriers to entry (WalkinVRDriver Autism, Dyslexia, Dyscalculia, Dysgraphia,
2022c). etc. In addition, some video games and mobile
And finally, another accessibility issue that applications have been developed with the inten-
WalkinVR addresses is the requirement of using tion of identifying whether a user might have
motion controllers to engage with virtual some type of learning disability rather than
Making Virtual Reality (VR) Accessible for People with Disabilities 1107

immediately addressing a specific type. Results in visual-motor integration, the second editions of
have shown improvements in the different areas the Bruininks-Oseretsky Test of Motor Profi-
where it has been applied. However, with the ciency and Developmental Test of Visual Percep-
recent spread and popularity of VR, specifically tion were implemented to analyze changes in
in the gaming sector, the development of VR function for visual-motor integration and to iden-
games directed to address cognitive disabilities tify changes in visual perception.
constitutes the next step for further gaming devel- Results showed the effectiveness of VR and
opers. The possible advantages of a fully computer game-based therapy to approach cogni-
immersive gaming experience to enhance and tive disabilities by appealing to visual perception
facilitate learning in different areas should be and motor function in children with these types of
explored to create a more inclusive gaming com- disabilities, demonstrating a considerable
munity and to take advantage of new technologies improvement of scores for both examination
as a helping tool for people with cognitive methods after exposure to VR gaming therapy
disabilities. for specific elements such as visual motor integra-
A project was founded in 2018 by the US tion and general visual perception.
Department of Education, aiming to use VR to Research conducted by Kim et al. (2019)
help students with learning disabilities in schools focused on VR intervention’s effectiveness as a
all over the country. The University of Kansas therapy method for people with mild cognitive
developed the “VOISS: Virtual Reality Opportu- impairment or dementia, gathering results from
nities to Implement Social Skills” project to 11 different studies highly focused on VR. Kim
address the challenges in the development of et al. compared the results from studies that used
social skills for students with disabilities. VOISS a full-immersive VR experience to studies using
uses a VR Head-mounted Display (HMD) set that semi-immersive technology and found a small-
allows subjects to enter a controlled virtual envi- to-medium effect ratio in patients exposed to
ronment where they can walk around in different VR intervention techniques, especially in fac-
M
locations like hallways, locker rooms, classrooms, tors such as physical fitness, cognition, and
etc. Recreating interaction scenarios with emotion. Aiming to set boundaries for a stan-
computer-driven avatars allows the user to evalu- dardized guideline for VR intervention in
ate an interaction’s positive or negative conse- patients, the research intended to summarize
quences. The fundamental goal is to use VR to evidence to validate the use of VR in this
explicitly teach students social skills and the healthcare area.
proper way to apply them in natural environments
(Gera 2018).
According to cognitive science, virtual experi- VR Gaming for People with Visual
ences can enhance cognitive flexibility in non- Impairment
digital environments. To become an adequate
learning tool, digital environments must keep a Most video games rely on visual elements to
sense of realism. This realism creates a relation- guide the user, leading to unintended exclusion
ship between the consequences of learned actions for people with some degree of visual impairment.
in a virtual environment and the real world. Even though some features like screen reading
Ahn’s work intended to elaborate on the effects and options for size adjustment exist in some
of combining VR and a computer video game for games, these approaches do not ensure people
cognitive-based therapy with a group of 13 chil- with sight loss can experience gaming to its full
dren between the ages of 7 and 13 years and the extent. The lack of inclusive options increases
capacity to play a game for 20 min and follow with emerging technology, which is the case
verbal instructions (Ahn 2021). Subjects took part with VR gaming. However, it is important to
in four sports games using a Wii console and a point out that blind individuals develop other
motion-based VR game. To study the differences senses to a larger extent, such as hearing,
1108 Making Virtual Reality (VR) Accessible for People with Disabilities

smelling, and touching. That is why VR gaming is to the blind community, as it takes advantage of
an excellent area of opportunity to enhance the the elements built into the Oculus Quest VR head-
user’s experience while gaming without the use of set. With its haptics, tracking, auditory, and voice
visual stimuli. Making the most of other elements systems, a nonvisual experience is provided to the
like audio and haptics can ensure an entertaining user. The game consists of a car race against four
gaming experience. Recent developments were artificial intelligence opponents on a three-and-a-
made to create VR gaming options for the blind half-mile virtual track. The first one to complete
community. However, mainstream VR games are three laps over the track wins. Haptic feedback is
not designed to adapt to the needs of people with the primary source for providing the player with
sight loss. information about the racetrack without any
One game prototype is The Enclosing Dark, visual input. Also, with the controllers’ hand ges-
created by Gluck and Brinkley in 2020. It is a VR tures tracking, the player can provide input to the
adventure game that was designed as a game game. In addition, vocal input is used to commu-
option for the visually impaired. Pre-game train- nicate with the artificial intelligence pit crew.
ing teaches the user how to understand the audio Finally, audio output is used to create a more
and haptics and interact with the virtual environ- immersive experience, as it was not suitable for
ment. Audio and haptics are combined to provide information transmission because it was too slow
feedback about the invisible landscape where the to create a fast-paced experience (Gluck
game takes place. Audio is mainly used to com- et al. 2021).
municate with the user and describe the actions of
the player’s avatar, from footsteps to rotation.
Spatial audio is applied to create a simulation of Conclusion and Discussion
the natural environment, allowing the user to
build a mental map through the auditory output. It is important to consider the area of opportu-
Haptics is a medium to deliver supplemental nity within the design of inclusive VR video
information about the virtual environment games. Nowadays, mainstream VR games are
through three different vibration patterns. The usually not accessible to people with disabil-
first one is used to aid in the location of obstacles, ities. However, headsets expand possibilities
the second one provides feedback when hitting an and allow those types of games to have a lot of
enemy, and the third one gives additional infor- potential for making the most of their features
mation about the surroundings (Gluck and and creating an accessible experience for every-
Brinkley 2020). one. Elements like haptics, controllers, visual
In 2021, a VR archery game that had no output, and audio output should be considered
visual elements was created. The main purpose when designing a VR game. There are a few
behind it was to assess if early blind adults developments and adaptations of video games
would show less developed head-trunk coordi- to make them accessible. However, the work
nation, as sensory disabilities have a direct done is not enough to ensure that VR gaming
impact on motor development. It was tested by is accessible for people with disabilities. In
letting early blind and sighted individuals play it addition, the advantages obtained from the use
and compare the results. The hardware used was of VR gaming in different therapy techniques
the game engine Unity 3D, BOSE over-ear head- have been shown in recent research, highlight-
phones, and a LG Google Nexus 4 smartphone. ing the importance of pushing VR gaming to
The user had to lead an arrow to hit the center of become more accessible for people with disabil-
a target by a combination of head and trunk ities. Aiming for a more inclusive society and
rotations around the vertical axis (Esposito gaming community, people with disabilities can
et al. 2021). benefit from entertainment improving the state
Also, in 2021, Racing in the Dark was devel- of their condition through VR therapy in some
oped. It is a fast-paced VR racing game accessible cases.
Mario Kart, an Analysis of Its Absence from Esports 1109

Cross-References WalkinVRDriver: Gameplay and VR gaming with the help

of a second person. https://www.walkinvrdriver.com/
gameplay-and-vr-gaming-with-the-help-of-a-second-
▶ Accessibility of Virtual Reality for Persons person/ (2022a)
with Disabilities WalkinVRDriver: Virtual motion and rotation. https://
▶ Immersive Technologies for Accessible User www.walkinvrdriver.com/virtual-motion-and-rotation/
Experiences (2022b)
WalkinVRDriver: Adjusting the position of the controller
▶ Mindfulness, Virtual Reality, and Video Games in virtual reality. https://www.walkinvrdriver.com/
adjusting-the-position-of-the-controller-in-virtual-
reality/ (2022c)
References WalkinVRDriver: Tracking disabled or spastic hands.
https://www.walkinvrdriver.com/tracking-disabled-or-
spastic-hands/ (2022d)
Ahn, S.: Combined effects of virtual reality and computer World Health Organization: Disability and health. https://
game-based cognitive therapy on the development of www.who.int/news-room/fact-sheets/detail/disability-
visual-motor integration in children with intellectual and-health (2021)
disabilities: a pilot study. Occup. Ther. Int. 2021,
1–8 (2021). https://doi.org/10.1155/2021/6696779
Esposito, D., Bollini, A., Gori, M.: Early blindness limits
the head-trunk coordination development for horizon-
tal reorientation. Front. Hum. Neurosci. 15, 312 (2021). Management
https://doi.org/10.3389/fnhum.2021.699312
Gera, E.: How VR is being used to help children with learning ▶ Game Development Leadership Tips
disabilities, autism. https://variety.com/2018/digital/fea
tures/voiss-interview-vr-hmd-1203086576/ (2018)
Gluck, A., Brinkley, J.: Implementing “the enclosing dark”:
a VR auditory adventure. J. Technol. Persons Disabil. 8,
149–159 (2020). https://scholarworks.csun.edu/bitstre Many-Light Rendering
am/handle/10211.3/215985/2197%20Implementing%20
The%20Enclosing%20Dark%20A%20VR%20Auditory M
%20Adventure.pdf?sequence¼1 ▶ High-Performance Many-Light Rendering
Gluck, A., Boateng, K., Brinkley, J.: Racing in the dark:
exploring accessible virtual reality by developing a rac-
ing game for people who are blind. Proc. Hum. Factors
Ergon. Soc. Annu. Meet. 65, 1114–1118 (2021). https://
doi.org/10.1177/1071181321651224 Mario Kart, an Analysis of Its
Heilemann, F., Zimmermann, G., Münster, P.: Accessibil- Absence from Esports
ity guidelines for VR games – a comparison and syn-
thesis of a comprehensive set. https://www.frontiersin. John Hoback2 and Newton Lee1,2
org/articles/10.3389/frvir.2021.697504/full (2021) 1
Kim, O., Pang, Y., Kim, J.-H.: The effectiveness of virtual Institute for Education, Research, and
reality for people with mild cognitive impairment or Scholarships, Los Angeles, CA, USA
2
dementia: a meta-analysis. BMC Psychiatry. 19(1), Vincennes University, Vincennes, IN, USA
219 (2019). https://doi.org/10.1186/s12888-019-2180-x
Mott, M., Cutrell, E., Franco, M.G., Holz, C., Ofek, E.,
Stoakley, R., Morris, M.R.: Accessible by design: an
opportunity for virtual reality. https://ieeexplore.ieee. Synonyms
org/document/8951960 (2020)
Rad, L.: Half-life: Alyx accessibility options – full guide. Racing games
GameSpot. Retrieved November 26, 2022, from
https://www.gamespot.com/articles/half-life-alyx-
accessibility-options-full-guide/1100-6475045/ (2020,
Mar 23) Definition
The 360 Guy: The ultimate VR headset comparison table:
every VR headset compared. https://www.threesix
tycameras.com/vr-headset-comparison-table/ (2022)
Mario Kart is a Multiplayer Racing Game created
W3C Working Group: XR accessibility user requirements. by Nintendo. It allows up to eight players to race
https://www.w3.org/TR/2021/NOTE-xaur-20210825/ to the finish line by using various items to attempt
(2021) to take the lead.
1110 Mario Kart, an Analysis of Its Absence from Esports

Introduction winner (factor #3). In the offline mode, when a

player wins first place in a grand prix, they are
As of December 2021, Mario Kart 8 is Nintendo’s rewarded with a trophy. If they get first place in all
highest selling Switch game, with over 43 million four races, they get a gold trophy for mastering the
copies sold (Nintendo.co.jp). Despite the popular- tournament. In online tournaments, the players are
ity and the competitive gameplay, Mario Kart has randomly assigned to a track. The points you earn
limited popularity in the professional esports either increase or decrease your score on the world
scene. This entry discusses the positive and neg- leaderboards (factor #4). From this, we can
ative factors effecting Mario Kart in order to deduce that the game is in line with all four goals.
determine why it is missing from the professional
esports scene (Esports Talks, 2018).
Four Negative Factors

Four Positive Factors However, there are four factors that negatively
hurt Mario Kart in the esports world. Chris
William Collis, author of “Book of Esports: The Blain, a writer for Esportstalk.com, listed several
Definitive Guide to Competitive Video Games,” reasons the game is missing from professional
states four main factors that make a game an esports. His first point was the lack of tourna-
esports: the player’s skills, the community sur- ments. Mario Kart tournaments do not have offi-
rounding the game, how accessible the game is cial leagues, unlike many other popular games.
to players, and how rewarding the game is to play The tournaments are mostly fan run, so the prize
(Collis, 2020 p.5). Another source from the pools are too low for players to spend the time
esports-news.co.uk article titled “What Factors getting skilled at the game. His second point was
Help Create a Successful Esports Games” states the RNG factor. He stated the game has too much
that “Providing a game that is easy to learn yet randomness and luck, but the game is quite boring
hard to master, delivering quality audience expe- without having at least some items available. His
riences with spectator modes, title must have third point was that many of the courses are
widespread appeal, and having simple goals set designed to allow players to cheat and take short-
for players to achieve” (esports-news.co.uk) cuts, which can present an unfair advantage. His
(Esports News UK, 2021). final point was the lack of team elements in the
In short, the most important factors for an game – a feature that is common in almost all
esports game are: popular esports games (Blain). It should be
noted that his fourth point is no longer valid
1. Simple to learn but requires skills to master because Nintendo has added team races in a
2. Has a large community game update.
3. Has a simple goal to achieve Blain made some good points in his article.
4. Must be rewarding to become skilled The game relies heavily on RNG. Many of the
races could end because another player got lucky.
How does this relate to Mario Kart 8? As However, luck is a factor that is common in most
stated in the introduction, it is currently games. For example, a player could draw a good
Nintendo’s highest selling switch game. It would card right when they need it in Hearthstone. The
be reasonable to say that it has a large community cheating on tracks is a big factor, but this could be
backing (factor #2). The game is also quite easy to alleviated by either banning the cheatable tracks
learn for new players but requires a large amount in official tournaments or adding rules to disqual-
of skills to use items and drift abilities at the right ify players who cheat. From his points, the only
time (factor #1). The goal of the game is to com- negative factor that cannot be easily removed is
plete four races and earn points to be crowned the the lack of official tournaments.
MCG 1111

Issues with Tournaments professional leagues are to take off, Nintendo

must step up, offer bigger prizes, and most of all,
The biggest negative factor for Mario Kart is the listen to their fan base.
limited number of tournaments. In fact, Nintendo
only recently started hosting official Mario Kart
tournaments in 2021. They signed an agreement References
with PlayVS to create an official high school
league, but not much has been done otherwise Bailey, K.: Nintendo is bringing Super Smash Bros.,
Splatoon 2, and Mario Kart 8 to High School Esports
(ign.com). The closest thing that most players
with New Partnership. IGN, 27 May 2021., https://
get to official tournament is the online matches www.ign.com/articles/nintendo-playvs-varsity-esports
that Nintendo occasionally host. These tourna- Collis, W.: Book of Esports: the definitive guide to com-
ments require you to be on the top of the world petitive video games. RosettaBooks, New York (2020)
Which Factors Help Create a Successful Esports Game? –
leaderboard at the end of the 3-day tournament in Esports News UK (2021). https://esports-news.co.uk/
order to be crowned the winner. The winner of the 2021/09/09/which-factors-help-create-a-successful-
tournament wins 2500 Eshop credits, which trans- esports-game/
late to roughly $25, which does not even cover Why Mario Kart Is Absent in Competitive Esports –
Esports Talk (2018). https://www.esportstalk.com/
half the cost of Mario Kart 8.
blog/why-mario-kart-is-absent-in-competitive-esports-
The unofficial leagues do not reward much 10883/
either. According to esportsearnings.com, the
highest earning player of 2022 earned $450 for
the whole year. This is nothing when compared to
Dota 2. The top earner, Mira, won $3,696,653.
The 500th place winner made $2800, which is six Marketing
times what Mario Kart paid out
(esportsearnings.com). ▶ Virtual Reality Retailing
M

Conclusion and Discussion

Massively Multi-player Online
Mario Kart 8 has the potential to be a professional Games, MMOG
esports game. It has all the common traits needed
to gain popularity. There are certainly some fac- ▶ Peer-to-Peer Gaming
tors that need balancing, such as RNG and issues
with track designs. Since Nintendo still actively
adds new content to Mario Kart in 2022, there is
still a potential. The biggest factor hurting the Massively Multiplayer Online
game’s popularity in esports is Nintendo itself. Role-Playing Game
While Epic, Tencent, and other companies have (MMORPG)
been actively involved in the esports scene since
the release of their games, Nintendo has only ▶ Virtual World, a Definition Incorporating Dis-
recently started to allow officially licensed tributed Computing and Instances
leagues in 2021. They have focused primarily on
building tournaments for their other games, such
as Super Smash Bros Ultimate and Splatoon 2,
while ignoring their cash cow Mario Kart. The MCG
lack of official tournaments means that the win-
ning prizes are too small to be enticing. If ▶ Mobile Cloud Gaming
1112 MCTS

MCTS MEEGA+

▶ Monte-Carlo Tree Search ▶ MEEGA+, Systematic Model to Evaluate Edu-

cational Games

Mean Average Precision

(MAP)
MEEGA+, Systematic Model to
▶ American Sign Language Detection Evaluate Educational Games

Giani Petri, Christiane Gresse von Wangenheim

and Adriano Ferreti Borgatto
Measurement Instrument Department of Informatics and Statistics (INE),
Federal University of Santa Catarina (UFSC),
▶ MEEGA+, Systematic Model to Evaluate Edu- Florianópolis, SC, Brazil
cational Games

Synonyms

Educational game; Evaluation model; Measure-

Mechanics ment instrument; MEEGA+; Serious games

▶ Design of Alienation in Video Games

Definitions

MEEGA+ is a systematic model to analyze edu-

Mediated Reality cational games (digital and nondigital ones) in
order to evaluate their perceived quality from the
▶ Augmented Reality for Maintenance students’ perspective in the context of computing
▶ Interaction with Mobile Augmented Reality education (Petri et al. 2016, 2017a).
Environments

Introduction

Medical Education In the last years, games have also been used
for different purposes than entertainment, being
▶ Virtual Reality Proton Beam Therapy Unit: more and more used in educational contexts (Abt
Case Study on the Development 2002; Connolly et al. 2012; Battistella and Gresse
von Wangenheim 2016). Educational games are
supposed to be an effective and efficient instruc-
tional strategy for teaching and learning in diverse
Medical Robot knowledge areas such as mathematics, health,
computing, and nutrition (Connolly et al. 2012;
▶ Healthcare Robots with Islamic Practices Calderón and Ruiz 2015). Especially in
MEEGA+, Systematic Model to Evaluate Educational Games 1113

computing education, there is a vast variety of user experience, and learning (Savi et al. 2011).
educational games to teach computing competen- Yet, although demonstrating an acceptable valid-
cies mainly in higher education (Battistella and ity and reliability, a more comprehensive analy-
Gresse von Wangenheim 2016). The majority are sis of the initial version of the MEEGA model
digital games, principally PC (Personal Com- has identified improvement opportunities mainly
puter) games, with a considerable trend also to related to an overlap of theoretical concepts of
nondigital ones (paper and pencil, board games, the factors motivation and user experience (Petri
etc.). Predominant are simulation games, which et al. 2017b). Consequently, the model has been
allow students to practice competencies through revised resulting in a new version, the MEEGA+
the simulation of real-life situations in a realistic model, which evaluates educational games in
environment while keeping them engaged in terms of usability and player experience. Thus,
the game (Battistella and Gresse von Wangenheim this article presents the design and evaluation of
2016). On the other hand, there also are several the MEEGA+ model, as an evolution of the
games designed to teach computing aiming MEEGA model proposed by Savi et al. (2011).
at learning objectives at lower cognitive levels.
Typically, these games are used to review and
reinforce knowledge taught beforehand using dif- The MEEGA+ Evaluation Model
ferent instructional strategies (Battistella and
Gresse von Wangenheim 2016). These games The objective of the MEEGA+ model is to ana-
are expected to be an effective and efficient strat- lyze educational (digital and nondigital) games in
egy for computing education (Backlund and order to evaluate their perceived quality from the
Hendrix 2013). However, these claims seem not students’ perspective in the context of computing
rigorously established as most evaluations of edu- education (Petri et al. 2016, 2017a).
cational games are performed in an ad hoc manner
in terms of research design, measurement, data Evaluation Dimensions
M
collection, and analysis (Calderón and Ruiz With respect to this objective, the perceived qual-
2015; Petri and Gresse von Wangenheim 2017) ity is evaluated in terms of quality factors and
due to the absence of models that provide a more dimensions. Based on a literature review (Petri
systematic support for the evaluation of educa- and Gresse von Wangenheim 2017) and a system-
tional games (Petri and Gresse von Wangenheim atic analysis of the initial version of the MEEGA
2016). Existing models for the evaluation of model (Petri et al. 2017b), a set of dimensions
games typically focus only on specific quality was defined to be measured by the MEEGA+
factors, such as usability (Omar and Jaafar model: focused attention, fun, challenge, social
2008), engagement (Brockmayer et al. 2009; interaction, confidence, relevance, satisfaction,
Norman 2013), or player experience (Denisova usability, and perceived learning (Table 1). The
et al. 2016; Abeele et al. 2016), not providing a dimension usability is further fragmented into
more comprehensive analysis of the games’ qual- five subdimensions: learnability, operability, aes-
ity, and, in particular, not evaluating their impact thetics, accessibility, and user error protection.
on learning, being one of the main objectives of A detailed description of the definition of dimen-
educational games. In this context, the MEEGA sions can be found in Petri et al. (2016, 2017a).
model (Savi et al. 2011) being developed since
2011 seems to be the most widely used evaluation Measurement Instrument Items
model in practice (Calderón and Ruiz 2015; Petri Data collection is operationalized through a mea-
and Gresse von Wangenheim 2017). MEEGA is a surement instrument (questionnaire). The mea-
model developed for the evaluation of the quality surement instrument items were derived based
of educational games in terms of motivation, on the defined dimensions, improving the initial
1114 MEEGA+, Systematic Model to Evaluate Educational Games

MEEGA+, Systematic Model to Evaluate Educational Games, Table 1 MEEGA+ measurement instrument items
Item
Dimension/Subdimension No. Description
Usability Aesthetics 1 The game design is attractive (interface, graphics, cards, boards, etc.)
2 The text font and colors are well blended and consistent
Learnability 3 I needed to learn a few things before I could play the game
4 Learning to play this game was easy for me
5 I think that most people would learn to play this game very quickly
Operability 6 I think that the game is easy to play
7 The game rules are clear and easy to understand
Accessibility 8 The fonts (size and style) used in the game are easy to read
9 The colors used in the game are meaningful
10 The game allows customizing the appearance (font and/or color) according to
my preferences
User error 11 The game prevents me from making mistakes
protection 12 When I make a mistake, it is easy to recover from it quickly
Confidence 13 When I first looked at the game, I had the impression that it would be easy for
me
14 The contents and structure helped me to become confident that I would learn
with this game
Challenge 15 This game is appropriately challenging for me
16 The game provides new challenges (offers new obstacles, situations, or
variations) at an appropriate pace
17 The game does not become monotonous as it progresses (repetitive or boring
tasks)
Satisfaction 18 Completing the game tasks gave me a satisfying feeling of accomplishment
19 It is due to my personal effort that I managed to advance in the game
20 I feel satisfied with the things that I learned from the game
21 I would recommend this game to my colleagues
Social interaction 22 I was able to interact with other players during the game
23 The game promotes cooperation and/or competition among the players
24 I felt good interacting with other players during the game
Fun 25 I had fun with the game
26 Something happened during the game (game elements, competition, etc.)
which made me smile
Focused attention 27 There was something interesting at the beginning of the game that captured my
attention
28 I was so involved in my gaming task that I lost track of time
29 I forgot about my immediate surroundings while playing this game
Relevance 30 The game contents are relevant to my interests
31 It is clear to me how the contents of the game are related to the course
32 This game is an adequate teaching method for this course
33 I prefer learning with this game to learning through other ways (e.g., other
teaching methods)
Perceived learning 34 The game contributed to my learning in this course
35 The game allowed for efficient learning compared with other activities in the
course

version of the MEEGA questionnaire, and cus- Keller 1987; Tullis and Albert 2008; Sindre and
tomizing and unifying existing standardized ques- Moody 2003; Sweetser and Wyeth 2005; Poels
tionnaires found in literature (Savi et al. 2011; et al. 2007; Gámez 2009; Takatalo et al. 2010;
MEEGA+, Systematic Model to Evaluate Educational Games 1115

O’Brien and Toms 2010; Wiebe et al. 2014; Fu and Spanish at: http://www.gqs.ufsc.br/meega-a-
et al. 2009; Mohamed and Jaafar 2010; Zaibon model-for-evaluating-educational-games/ under
and Shiratuddin 2010; Zaibon 2015; Brooke the Creative Commons License.
1996; Davis 1989). Table 1 shows the defined
items for the MEEGA+ measurement instrument
for each dimension/subdimension. Evaluation of the MEEGA+ Model

Response Format In order to evaluate the reliability and construct

The response format of the items of the MEEGA+ validity of the MEEGA+ measurement instru-
measurement instrument is a 5-point Likert scale ment, 40 case studies were conducted, evaluating
with response alternatives ranging from strongly 18 different games applying the MEEGA+ model
disagree to strongly agree (DeVellis 2016). The with a population of 718 students (Petri
use of the Likert scale in its original 5-point for- et al. 2018).
mat allows to express the opinion of the individual Reliability refers to the degree of consistency
(student) under the object of study (educational of the instrument items, typically, measured
game) with greater precision, besides allowing the through the Cronbach’s alpha coefficient
individual being more comfortable to express (Trochim and Donnelly 2008). Analyzing the
their opinion, using a neutral point and, thus, data collected, the value of the Cronbach’s
increasing the quality of the answers (Dawes alpha for all items of the MEEGA+ measurement
2008). instrument is considered excellent (α ¼ 0.928).
This result demonstrates that the answers to the
Research Design items are consistent and precise, indicating the
In order to conduct the evaluation in a quickly and reliability of the MEEGA+ measurement
nonintrusive way not interrupting the normal flow instrument.
of the class, a case study design was chosen for Construct validity refers to the ability to
M
the evaluation that allows an in-depth research actually measure what it purports to measure,
of an individual, group, or event (Wohlin et al. involving convergent and discriminant validity,
2012; Yin 2017). The general study design is a which is obtained through the degree of correla-
one-shot post-test only design, in which the case tion between the instrument items (Trochim and
study begins with the application of the treatment Donnelly 2008). In order to identify the number
(educational game) and afterwards the MEEGA+ of factors that represents the responses of the
questionnaire is answered by the students to items of the MEEGA+ measurement instrument,
collect the respective data. Adopting this research a factor analysis was performed. The results
design, the evaluation objective is assessed based indicate that the structure of the MEEGA+ is
on the students’ perceptions (perceived quality). based on an evaluation model with two quality
factors and their dimensions, as presented in
Data Analysis Fig. 2.
Data collected in the case study are analyzed in Analyzing the correlation between the items
terms of frequency distribution (through bar of the two quality factors (usability and player
graphs) and central tendency (median) for each experience), the results show that there is a large
quality factor (usability and player experience) correlation between most of the items within each
and their dimensions (Fig. 1). The MEEGA+ quality factor. This indicates that a convergent
model provides a spreadsheet for the analysis validity can be established for the two quality
of the data collected, assisting in the organization factors (usability and player experience). In the
of the information and automatic generation of same way, items of different quality factors pre-
graphs visualizing the results of an evaluation. sent a small correlation, and, thus, provide evi-
The complete material of the MEEGA+ model dence of discriminant validity. This indicates that
is available in English, Brazilian Portuguese, the measurement instrument of the MEEGA+
 1116

MEEGA+, Systematic Model to Evaluate Educational Games, Fig. 1 Example of data analysis graph provided by the MEEGA+ model
MEEGA+, Systematic Model to Evaluate Educational Games
MEEGA+, Systematic Model to Evaluate Educational Games 1117

Quality factors Dimensions

Focused Attention

Fun

Challenge
Player
Experience
Social Interaction
Cronbach’s
alpha α=.856
Confidence

Relevance
Quality of
Educational
Satisfaction
Games

User error protection

Learnability

Usability Operability Cronbach’s

alpha α=.930
Aesthetics

Accessibility M

MEEGA+, Systematic Model to Evaluate Educational Games, Fig. 2 Decomposition of the quality of educational
games in the MEEGA+ model

model is valid for evaluating the quality of edu- order to evaluate the quality of educational
cational games. games as a basis for their improvement and effec-
tive and efficient adoption in practice. And,
although it has been originally developed for
Conclusions the evaluation of games for computing educa-
tion, the MEEGA+ model can be used and
The MEEGA+ model, an evolution of the adapted for the evaluation of games to teach
MEEGA model, aims at evaluating the perceived other knowledge areas.
quality of educational games focusing on usabil-
ity and player experience from the students’ per-
spective in the context of computing
education. Results of a statistical evaluation of Cross-References
the MEEGA+ measurement instrument demon-
strate satisfactory reliability and construct valid- ▶ Augmented Learning Experience for School
ity. Thus, the MEEGA+ model can provide a Education
reliable and valid measurement instrument for ▶ Game Player Modeling
game creators, instructors, and researchers in ▶ Games and the Magic Circle
1118 MEEGA+, Systematic Model to Evaluate Educational Games

References O’Brien, H.L., Toms, E.G.: The development and evalua-

tion of a survey to measure user engagement. J. Am.
Abeele, V.V., Nacke, L.E., Mekler, E.D., Johnson, D.: Soc. Inf. Sci. Technol. 61(1), 50–69 (2010)
Design and preliminary validation of the player expe- Omar, H.M., Jaafar, A.: Playability heuristics evaluation
rience inventory. Symposium Computer-Human Inter- (PHE) approach for Malaysian educational games.
action Play, pp. 335–341. Austin (2016) Symposium Information Technology, pp. 1–7. Kuala
Abt, C.C.: Serious Games. University Press of America, Lumpur (2008)
Lanhan (2002) Petri, G., Gresse von Wangenheim, C.: How to evaluate
Backlund, P., Hendrix, M. (2013). Educational games - Are educational games: a systematic literature review.
they worth the effort? A literature survey of the effec- J. Univ. Comp. Sci. 22(7), 992–1021 (2016)
tiveness of serious games. Proc. of the 5th Int. Conf. on Petri, G., Gresse von Wangenheim, C.: How games for
Games and Virtual Worlds for Serious Applications, computing education are evaluated? A systematic liter-
Poole, GB. ature review. Comp. Educ. 107, 68–90 (2017)
Battistella, P.E., Gresse von Wangenheim, C.: Games for Petri, G., Gresse von Wangenheim, C., Borgatto, A.F.:
teaching computing in higher education – a systematic MEEGA+: an evolution of a model of educational
review. IEEE. Tech. Eng. Educ. 9(1), 8–30 (2016) games. Tech Rep INCoD/GQS.03.2016.E. INCoD/
Brockmyer, J.H., Fox, C.M., Curtiss, K.A., McBroom, E., INE/UFSC. http://www.incod.ufsc.br/wp-content/
Burkhart, K.M., Pidruzny, J.N.: The development of uploads/2016/07/Relatorio-Tecnico-INCoD_GQS_
the game engagement questionnaire: a measure of 03_2016_Ev11.pdf (2016). Accessed 26 Oct 2017
engagement in video game-playing. Exp. Soc. Psy. Petri, G., Gresse von Wangenheim, C., Borgatto, A.F.:
45(4), 624–634 (2009) Evolução de um Modelo de Avaliação de Jogos para o
Brooke, J.: SUS-A quick and dirty usability scale. Usab. Ensino de Computação. CSBC/WEI, São Paulo
Eval. Ind. 189(194), 4–7 (1996) (2017a) (in Portuguese)
Calderón, A., Ruiz, M.: A systematic literature review on Petri, G., Gresse von Wangenheim, C., Borgatto, A.F.:
serious games evaluation: an application to software A large-scale evaluation of a model for the evaluation
project management. Comp. Educ. 87, 396–422 (2015) of games for teaching software engineering. ICSE-
Connolly, T.M., Boyle, E.A., MacArthur, E., Hainey, T., SEET, pp. 180–189. Buenos Aires (2017b)
Boyle, J.M.: A systematic literature review of empirical Petri, G., Gresse von Wangenheim, C., Borgatto, A.F.:
evidence on computer games and serious games. Design and evaluation of a model for the evaluation
Comp. Educ. 59(2), 661–686 (2012) of games for computing education. Computing Educa-
Davis, F.D.: Perceived usefulness, perceived ease of use, tion (2018) (submmited)
and user acceptance of information technology. Poels, K., Kort, Y.D., Ijsselsteijn, W.: It is always a lot
MIS Q. 13, 319–340 (1989) of fun!: exploring dimensions of digital game experi-
Dawes, J.: Do data characteristics change according to the ence using focus group methodology. Future Play,
number of scale points used? An experiment using pp. 83–89. Toronto (2007)
5-point, 7-point and 10-point scales. J. Mark. Res. Savi, R., Gresse von Wangenheim, C., Borgatto, A.F.:
50(1), 61–77 (2008) A Model for the Evaluation of Educational Games for
Denisova, A., Nordin, A.I., Cairns, P.: The convergence of Teaching Software Engineering. SBES, São Paulo
player experience questionnaires. Symposium (2011) (in Portuguese)
Computer-Human Interaction Play, pp. 33–37. Austin Sindre, G., Moody, D.: Evaluating the effectiveness of
(2016) learning interventions: an information systems case
DeVellis, R.F.: Scale Development: Theory and Applica- study. Conference on Information System, Paper 80.
tions, 4th edn. SAGE, Thousand Oaks (2016) Naples (2003)
Fu, F., Su, R., Yu, S.: EGameFlow: a scale to measure Sweetser, P., Wyeth, P.: GameFlow: a model for evaluating
learners’ enjoyment of e-learning games. Comp. player enjoyment in games. Comput. Entertain. 3(3),
Educ. 52(1), 101–112 (2009) 1–24 (2005)
Gámez, E.H.: On the core elements of the experience of Takatalo, J., Häkkinen, J., Kaistinen, J., Nyman, G.:
playing video games. Dissertation, UCL Interaction Presence, involvement, and flow in digital games.
Centre, Department of Computer Science, London In: Bernhaupt, R. (ed.) Evaluating User Experience in
(2009) Games: Concepts and Methods, pp. 23–46. Springer,
Keller, J.: Development and use of the ARCS model of London (2010)
motivational design. J. Instr. Dev. 10(3), 2–10 (1987) Trochim, W.M., Donnelly, J.P.: Research Methods Knowl-
Mohamed, H., Jaafar, A.: Development and potential anal- edge Base, 3rd edn. Atomic Dog Publishing, Mason
ysis of heuristic evaluation for educational computer (2008)
game (PHEG). Computer Science Convergence Infor- Tullis, T., Albert, W.: Measuring the User Experience:
mation Technology, pp. 222–227. Seoul (2010) Collecting, Analyzing, and Presenting Usability Met-
Norman, K.L.: GEQ (Game engagement/experience ques- rics. Morgan Kaufmann, Amsterdam (2008)
tionnaire): a review of two papers. Interact. Comput. Wiebe, E.N., Lamb, A., Hardy, M., Sharek, D.: Measuring
25(4), 278–283 (2013) engagement in video game-based environments:
Meta Artificial Intelligence and Artificial Intelligence Director 1119

investigation of the user engagement scale. Comput.

Hum. Behav. 32, 123–132 (2014) Mental Rotation
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C.,
Regnell, B., Wesslén, A.: Experimentation in Software
Engineering. Springer, Berlin/Heidelberg (2012) ▶ Training Spatial Skills with Virtual Reality and
Yin, R.K.: Case Study Research and Applications: Design Augmented Reality
and Methods, 5th edn. Sage, Beverly Hills (2017)
Zaibon, S.B.: User testing on game usability, mobility,
playability, and learning content of mobile game-
based learning. J. Tekn. 77(29), 131–139 (2015)
Zaibon, S.B., Shiratuddin, N.: Heuristics evaluation strat- Mesh Parameterization
egy for mobile game-based learning. Wirless Mobile
Ubiquitous Technologies in Education, pp. 127–131.
Kaohsiung (2010) ▶ UV Map Generation on Triangular Mesh

Meta Artificial Intelligence

Melee and Artificial Intelligence
Director
▶ Super Smash Bros.: A Brief History
Daiki Satoi and Yuta Mizuno
SQUARE ENIX Co., Ltd., Tokyo, Japan

Memory Synonyms
M
▶ Preserving the Collective Memory and Re-cre- Dynamic difficulty adjustment; Dynamic game
ating Identity Through Animation balancing; Pacing

Definition
Mental Disorder
Meta Artificial Intelligence (Meta AI, as known as
▶ Virtual Reality Therapy AI Director) is the set of processes used to control
a whole game from a meta perspective.

Mental Health of Indigenous Introduction

Peoples
The main AI used in games can be classified into
▶ Indigenous Knowledge for Mental Health, Data three types (Miyake 2017): character AI (see the
Visualization ▶ “Character Artificial Intelligence” chapter) for
representing characters, navigation AI (see the
▶ “Navigation Artificial Intelligence” chapter)
for making environment information easy to han-
Mental Immersion dle, and Meta AI, which is described in this entry.
Meta AI is AI for controlling the whole game
▶ Virtual Reality: A Model for Understanding from the meta perspective (Fig. 1). For example,
Immersive Computing difficulty is generally adjusted in advance based
1120 Meta Artificial Intelligence and Artificial Intelligence Director

Meta Artificial
Intelligence and
Artificial Intelligence
Director, Fig. 1 The basic
idea of Meta AI. Character
AI and Navigation AI work
inside the game world. On
the other hand, Meta AI
controls the whole game
from the meta perspective

on several standards such as “easy,” “normal,” Meanwhile, Meta AI is a technique advocated

and “hard.” However, by introducing Meta AI, it by Will Wright (2005) and utilized in his games
is possible to dynamically change the difficulty such as The Sims (Wright 2000). According to
and game balance according to the player’s esti- Wright, Meta AI creates an experience by infor-
mated emotions calculated by the gameplay data, mation flow, pacing, and a simple player model.
making it possible to realize the optimum diffi- This includes a peer AI, which controls agents,
culty setting for individual players and game sit- and a sub-AI, which governs simulations. There-
uations (see the “Combat Meta AI” section). In fore, Meta AI is not limited to battles and can be
addition, it is difficult to optimally configure all understood to be an AI, which broadly controls
events and objects beforehand in the wide game the whole game.
environment referred to as an open world. Using Accordingly, in this entry AI Director is
Meta AI makes it possible to dynamically control defined that optimizes certain parts of a game as
the events, characters, and other game objects a type of Meta AI and interpret the technology
encountered by players (see the “Non-combat referred to as AI Director also as Meta AI.
Meta AI” section). Furthermore, Meta AI can
make fun games actively through not only con-
trolling the existing game but also generating Combat Meta AI
contents such as music, terrain, or stories (see
the “Conclusion and Discussion” section). In this section, the authors describe Meta AI in a
combat scene. Combat Meta AI can be broadly
Meta AI Versus AI Director classified as (1) an approach that dynamically
AI Director is a similar term to Meta AI. adjusts the difficulty of the game according to
AI Director is a technology introduced for real- the changing battle situation and the state of the
izing adaptive dramatic pacing in order to increase player throughout the battle and (2) an approach
the ability to replay (Booth 2009a, b). In Left that dynamically adjusts the behavior, place-
4 Dead (Booth 2008), this was accomplished ment, and other game parameters of a nonplayer
through AI, which controls the intensity of the character (NPC) depending on the positional
player by controlling the spawning pace of relationship of the characters and the battle sit-
enemy characters. uation. In this section, this is referred to as
Meta Artificial Intelligence and Artificial Intelligence Director 1121

pacing Meta AI, and tactical Meta AI, that is more adequate in speed of the tension
respectively. than others by using two events. Such game
Application of many of the methods intro- design techniques are widely known as pacing
duced in this section is envisaged in situations (Schell 2014; Rogers 2014).
where players and NPCs battle in the game, such
as first-person shooter (FPS) fighting games or Dynamic Difficulty Adjustment
role-playing games (RPG). However, they are In many games, the pace and difficulty of the
widely applicable to games that are designed to game is predetermined by the game designer and
make players challenge the game system or NPC does not change dynamically during play. The
in some way, such as puzzle games, card games, game designer carefully adjusts the parameters
board games, and other game parameters. such that most players (or a representative player)
experiences enjoyment. As a result, ten enemies
Pacing Meta AI appear in the first minute, and there is a 5-s break
When and in what number should enemies make before a boss. However, as the skills of players
an appearance in a game where the enemies and the game situations are diverse, it is not easy
appear and are defeated one after another, such for every player to enjoy exactly the same content.
as in Space Invaders (Nishikado 1978)? For Depending on the person, the game may be too
example: easy or boring or it may be too difficult and make
the player abandon the game. The basic idea of
(a) Ten enemies appear per minute and continue pacing Meta AI is to dynamically control the
to do so thereafter. contents of the game according to the skill of the
(b) Ten enemies appear in the first minute, 20 ene- player and the game situation, such that various
mies appear in the next 1 min, and the number players experience a more interesting game.
of enemies that appear continues to increase at Within pacing Meta AI, the particular function
the same pace. that dynamically adjusts the difficulty level is
M
(c) Ten enemies appear in the first minute, 20 ene- called dynamic difficulty adjustment (DDA).
mies appear in the next 1 min, and after a 5-s Various approaches are proposed for the
break, the boss appears. method of creating a pacing Meta AI. The most
classical method is to make difficulty adjustment
Of these cases, the third is an adequate pace for rules in a form integrated with the game system.
the flow of the game and would probably be found For example, Xevious (Endo et al. 1983) imple-
interesting. Figure 2 shows the player’s expected ments a simple Meta AI (Miller 2004; Cerny et al.
chart of tension or intensity (called “interest 2005; Miyake 2017). The player’s hidden “diffi-
curve”) for each cases. Case (c) makes change culty” evaluation score defines the enemy

Meta Artificial Intelligence and Artificial Intelligence increases in a constant pace. (c) Using the pacing technique
Director, Fig. 2 Examples of the interest curve. (a) Ene- with the “Start a break” and “Boss appears” events
mies appear in a constant pace. (b) Number of enemies
1122 Meta Artificial Intelligence and Artificial Intelligence Director

appearance pattern like “if the difficulty is 10 then where beginners continue to fight strong enemies
three Type-A enemies appear” and “if the diffi- or advanced players are matched with weak ene-
culty is 20 then four Type-A and five Type-B mies. Such a concept is modeled as flow theory
enemies appear.” As long as the player proceeds and the state where the skill and the challenge are
in the game, increasingly stronger enemies balanced is called a flow state (Csikszentmihalyi
appear, but when the player is defeated, the ene- 1990). Flow theory has influenced game design
mies return to weakness. This dynamically real- theory and the DDA method (Salen and
izes a degree of difficulty that is just right for the Zimmerman 2003; Hunicke and Chapman 2004;
player. Chen 2007). Additionally, a framework for
designing games to facilitate the achievement of
Emotion-Based Methods a flow state has been proposed (Cruz and Uresti
In recent years, methods of separating the Meta AI 2017).
from the game system and indirectly controlling it The balance between the skill of the player
by digitizing the player and the game situation and the challenge is considered to be particularly
have been developed. In Left 4 Dead, Meta AI important in games where pacing cannot be
carries out pacing based on “emotional intensity” based on the number of enemies spawned and
in order to realize repeated game playability and the outcome of the game greatly depends on
maintain interest (Booth 2009a, b). For example, player skill, such as in fighting games. There-
if a player is damaged by an enemy, the evaluation fore, several techniques for dynamically
value of emotional intensity increases in propor- adjusting the behavior patterns of enemy char-
tion to the amount of damage. The game flow is acters in fighting games have been proposed in
divided into four phases (build up, sustain peak, order to match enemy characters with players
peak fade, and relax) in order to pace the emo- such that beginners and intermediate players
tional intensity throughout the game and the can improve their skill while enjoying gameplay
phases shift according to a player’s behavior and more. Demediuk et al. proposed an enemy AI
emotional intensity. A similar Meta AI has been that realizes DDA by trying to bring the differ-
introduced in Warframe (Sinclair and McGregor ence in health points (HP) between the player
2013), where the speed of increasing intensity and the enemy close to zero by changing the
changes according to the growth of the player action selection policy and the Monte Carlo
character (Brewer et al. 2013; Brewer 2014). Tree Search (MCTS) evaluation function
Satoi and Mizuno proposed a Meta AI using a (Demediuk et al. 2017). In addition, Ishihara
two-dimensional emotion model composed of et al. introduced a term for reliability of behavior
“hope of winning” and “fear of losing” to measure into the MCTS evaluation function and adjusted
and influence the player’s various emotions (Satoi parameters according to the game situation,
and Mizuno 2019). For example, in an action thereby realizing an enemy AI that can play
game, if the battle becomes deadlocked, Meta AI with skills similar to that of the player and
estimates the player’s current emotion as bored which seems natural (Ishihara et al. 2018).
and then tries to change it toward the opposite side
of the emotion map by adjusting the enemies’ Focusing on the Overall Engagement
behavior. Xue et al. proposed a method for increasing the
overall engagement of a level-progression game
Balancing the Skills and Challenges such as a puzzle game. This method increases the
In the cases mentioned so far, the authors have cumulative number of levels played until the
focused on the pattern of change in difficulty and player leaves the game and the total duration of
intensity according to the game progression, but it gameplay time. This was implemented it in a
is also important to balance the skills of players match three game released by Electronic Arts
and challenges. If players are to enjoy the game (Xue et al. 2017). The game progress is modeled
over a long period, situations should be avoided as a stochastic graph composed of level up
Meta Artificial Intelligence and Artificial Intelligence Director 1123

transition, retry transition, and churn transition. NPC according to the position of the character or
The optimum degree of difficulty in each state is battle situation.
calculated by dynamic programming.
Deciding Spawning Position
Tactical Meta AI In Left 4 Dead (Booth 2009a, b), players move
In the previous section, the authors discussed a towards the exit while fighting enemy NPC within
method of deciding when and by how much to a level of intricate terrain. However, Meta AI
adjust the contents of a game. In action games, dynamically spawns several kinds of enemy
real time strategy (RTS) games, and FPS, in which NPC and weapon items in order to realize unex-
a battle develops in a vast virtual space, it is pected game play that can be repeated many
necessary to take into account “where” the game times. For example, the majority of enemies,
contents are to be adjusted, that is, the spatial which are called mob, spawn in areas close behind
element. For example, when you want to make the player and are not directly visible. Meanwhile,
an enemy NPC appear (spawn) to increase the boss enemies are spawned in areas that are close to
intensity for a player, it makes no sense if it the player and are not directly visible on the
spawns at a distant location the player passed player’s predicted path of travel (referred to as
some time ago. Conversely, if the NPC spawns “Golden path”) (Fig. 3) (Jack and Vehkala
suddenly in front of the player, it probably causes 2013). In order to realize such spawning, it is
a sense of discomfort or unnatural. Therefore, in necessary to dynamically analyze the positional
game design it is necessary to calculate a “suitable relationship between the terrain and the player.
location” at which to spawn the enemy NPC. Left 4 Dead models terrain information for the
There are a great number of methods for Meta AI. First, the terrain is divided into a number
constructing tactics and strategies in game AI of subareas. Then information for each subarea for
(Robertson and Watson 2014; Rabin 2017). In the remaining distance (flow distance) to the exit
this section, the authors introduce some examples is embedded using a navigation mesh. Next, the
M
of tactical Meta AI that dynamically adjust the subarea around the player is dynamically calcu-
behavior, placement, and other characteristics of lated. This is called an active area set (AAS) and

Meta Artificial Intelligence and Artificial Intelligence Director, Fig. 3 An example of enemy spawning by Meta
AI. Spawning areas are close to the Golden path and invisible by the player
1124 Meta Artificial Intelligence and Artificial Intelligence Director

signifies the entire area to be handled by the Meta things occur as emergent events based on “loca-
AI. Combining AAS and flow distance makes it tion and area,” “time of day and weather,” “fre-
possible to easily and dynamically calculate quency,” “priority and probability,” and other
places suitable for spawning enemies and items. factors (Varnier 2014). As a result, events such
as a soldier and an elephant starting to battle and
Combination with Procedural Level the elephant overturning a jeep occur irrespective
Generation of the player’s behavior. Moreover, objects such
A similar mechanism is introduced in Warframe as prisons and animals are dispersed to have a
(Brewer et al. 2013; Brewer 2014), but in contrast density that results in players encountering events
to Left 4 Dead, levels are automatically generated. at an appropriate frequency. In addition, by world
Therefore, an influence map is used to analyze the profiling and adaptive spawning, the number of
movement status of players instead of flow dis- scavenger animals is increased when there is a big
tance. Using the position of a player as a heat battle.
source, the temperature of the areas surrounding In Assassin’s Creed: Origins (Guesdon and
the player are matched to that heat source produc- Ismail 2017), Meta AI was utilized in order to
ing areas where the temperature rises or falls as the make maximum use of limited resources in a
player moves. The area where the temperature vast terrain and at the same time to make the
rises is regarded as the destination to which the characters feel as realistic as possible (Lefebvre
player is heading and in which enemies spawn and 2018). Objects in games are broadly divided into
the area where the temperature falls is regarded as two types, moving objects (dynamic objects) such
the place from which the player moves away and as animals and vehicles and nonmoving objects
enemies there are stopped or deleted. (static objects) such as bases and garages. For
efficient searching, in Assassin’s Creed: Origins,
Natural Cooperating dynamic objects can be optimally arranged by
Meanwhile, in the Meta AI of Final Fantasy XV dividing the game environment according to
(Tabata 2016) attention is given to the behavior of invisible destination objects called “station” and
ally NPCs (Miyake 2018). When a player and “child position.” Stations own some child posi-
three ally NPCs fight together, the Meta AI ana- tions, and unoccupied stations are selected by a
lyzes the battle situation and directs an appropriate dynamic object such as a character. At this time,
ally NPC to help the player when in a difficult the density of objects is controlled with uniformly
situation or to follow when the player is running sized cells dividing the level. The numbers of
away. This avoids unnatural behaviors such as dynamic objects existing in one cell are managed
three allies suddenly coming to help the player simultaneously by Meta AI.
all at once and realizes a more natural cooperative
behavior.
Conclusion and Discussion

Noncombat Meta AI In this entry, the authors explained Meta AI,

which is an AI that controls the entire game.
In this section, the authors describe cases of Meta They classified and explained the two types of
AI with a focus on elements not directly related to combat Meta AI: pacing Meta AI, which controls
combat scenes. pacing according to the progress of the game, and
In games with very extensive levels, which are tactical Meta AI, which controls spatial tactical
called open worlds, it is difficult for a game crea- behavior. In addition, they explained that the
tor to appropriately arrange all events in advance. occurrence of events during the game and the
Accordingly, in Far Cry 4 (Hutchinson and Méthé arrangement of objects, and other game elements
2014), Meta AI is utilized as an encounter man- were controlled by noncombat Meta AI. In this
ager and objects are arranged so that various section, they conclude the entry by considering
Meta Artificial Intelligence and Artificial Intelligence Director 1125

the direction of the future development of possible component in the game, such as game
Meta AI. rules, stories, quests, terrain, music, and others.
Important functions of Meta AI include (Shaker et al. 2016; Short and Adams 2017). In
(1) understanding the player and the game situa- addition, in the game industry, the proliferation of
tion and (2) controlling the contents of the game. procedural techniques is proceeding in many
For understanding the player and the game titles, including Warframe and Far Cry 4 as was
situation, algorithms to calculate the state previously discussed, and No Man’s Sky (Murray
(intensity) of the player from the data obtained et al. 2016). By combining Meta AI and proce-
from the game controller input are built ad hoc; dural technology, it is possible to realize a differ-
however, it is costly to construct the model this ent procedural generation of the gaming
way and difficult to ensure accuracy. In recent experience for each player.
years, there have been many extensive collections In this way, the capabilities of Meta AI to
of large-scale game play data using telemetry and realize more interesting games are dramatically
their utilization is progressing in visualization of increasing. Designing interesting games by utiliz-
data (Pascale 2016), game balance adjustment ing these technologies will be a challenging future
(Mouret and Athanassoff 2018), and cheat detec- work. Additionally, the importance of developing
tion by deep learning (McDonald 2018). Further- Meta AI technology in both theory and practice is
more, approaches to measure the physiological likely to increase further.
information of players (biofeedback with respect
to the game) and more direct estimation of the
player’s psychological state are promising. For Cross-References
example, Ambinder created a version of Meta AI
in Left 4 Dead 2 in which the player’s skin con- ▶ Character Artificial Intelligence
ductance level (SCL) was used as the intensity. In ▶ Navigation Artificial Intelligence
a comparative experiment with conventional ver-
M
sions, it was found that the version using biofeed-
back had higher player satisfaction (Ambinder References
2011). It is expected that Meta AI will be able to
acquire more useful information by developing Ambinder, M.: Biofeedback in gameplay: how valve mea-
sures physiology to enhance gaming experience. In:
methods to increase the amount, quality, and pro-
Game Developers Conference 2011 (GDC 2011)
cessing of data in the future. (2011)
For controlling of the game contents, integra- Booth, M.: Left 4 Dead. Valve Software (2008)
tion with procedural content generation (PCG) Booth, M.: Replayable cooperative game design: Left
4 Dead. In: Game Developers Conference 2009 (GDC
technology is considered important. By using
2009) (2009a)
PCG, Meta AI will be able to create the game Booth, M.: The AI systems of Left 4 Dead. In: The 5th
worlds as well as control the existing game AAAI Conference on Artificial Intelligence and Inter-
world. For example, PCG is useful to make active Digital Entertainment (AIIDE 2009) (2009b)
Brewer, D.: The living AI in Warframe’s procedural space
unique experience to each player because it can
ships. In: Game AI Conference (2014)
change games dynamically. However, PCG tech- Brewer, D., Cheng, A., Dumas, R., Laidacker, A.: AI post-
nology needs the reference parameter for deciding mortems: Assassin’s Creed III, XCOM: enemy
how to change game contents for each player. If unknown, and Warframe. In: Game Developers Con-
ference 2013 (GDC 2013) (2013)
Meta AI guesses the player is angry and it wants Cerny, M., Hocking, C., Iwatani, T., Mizuguchi, T.,
the player to become more excited, Meta AI may Rigopulos, A.: International game designers panel. In:
ask the procedural music system to make the Game Developers Conference 2005 (GDC 2005)
BGM more spectacular and may ask the proce- (2005)
Chen, J.: Flow in games (and everything else). Commun.
dural conversation system to make the enemy’s
ACM. 50(4), 31–34 (2007)
dialogue to fuel the player’s excitement. Many Cruz, C.A., Uresti, J.A.R.: Player-centered game AI from a
PCG technologies have been proposed for every flow perspective: towards a better understanding of past
1126 Microscopic Simulation

trends and future directions. Entertain. Comput. 20, Satoi, D., Mizuno, Y.: Changing the game: measuring and
11–24 (2017) influencing player emotions through meta AI. In: Game
Csikszentmihalyi, M.: Flow: The Psychology of Optimal Developers Conference 2019 (GDC 2019) (2019)
Experience. Harper & Row, New York (1990) Schell, J.: The Art of Game Design: A Book of Lenses,
Demediuk, S., Tamassia, M., Raffe, W.L., Zambetta, F., Li, 2nd edn. A K Peters/CRC Press, New York (2014)
X., Mueller, F.: Monte Carlo tree search based algo- Shaker, N., Togelius, J., Nelson, M.J.: Procedural Content
rithms for dynamic difficulty adjustment. In: Proceed- Generation in Games: A Textbook and an Overview of
ings of the IEEE Symposium on Computational Current Research. Springer, New York (2016)
Intelligence and Games (CIG 2017) (2017) Short, T., Adams, T.: Procedural Generation in Game
Endo, M., Toyama, S., Ono, H.: Xevious. Namco (1983) Design. A K Peters/CRC Press, New York (2017)
Guesdon, J., Ismail, A.: Assassin’s Creed: Origins. Ubisoft Sinclair, S., McGregor, S.: Warframe. Digital Extremes
(2017) (2013)
Hunicke, R., Chapman, V.: AI for dynamic difficulty Tabata, H.: Final Fantasy XV. Square Enix (2016)
adjustment in games. In: Challenges in Game Artificial Varnier, J.: Far Cry’s AI: a manifesto for systemic
Intelligence AAAI Workshop, pp. 91–96 (2004) gameplay. In: Game/AI Conference 2014 (2014)
Hutchinson, A., Méthé, P.: Far Cry 4. Ubisoft (2014) Wright, W.: The Sims. Maxis, Electronic Arts (2000)
Ishihara, M., Ito, S., Ishii, R., Harada, T., Thawonmas, R.: Wright, W.: AI: a design perspective. In: The 1st AAAI
Monte-Carlo tree search for implementation of Conference on Artificial Intelligence and Interactive
dynamic difficulty adjustment fighting game AIs hav- Digital Entertainment (AIIDE 2005) (2005)
ing believable behaviors. In: Proceedings of the IEEE Xue, S., Wu, M., Kolen, J., Aghdaie, N., Zaman, K.A.:
Symposium on Computational Intelligence and Games Dynamic difficulty adjustment for maximized engage-
(CIG 2018) (2018) ment in digital games. In: Proceedings of the 26th
Jack, M., Vehkala, M.: Spaces in the sandbox: tactical International Conference on World Wide Web (WWW
awareness in open world games. In: Game Developers 2017) Companion, pp. 465–471 (2017)
Conference 2013 (GDC 2013) (2013)
Lefebvre, C.: Virtual insanity: meta AI on “Assassin’s
Creed: Origins.” In: Game Developers Conference
2018 (GDC 2018) (2018)
McDonald, J.: Robocalypse now: using deep learning to com- Microscopic Simulation
bat cheating in “Counter-Strike: Global Offensive.” In:
Game Developers Conference 2018 (GDC 2018) (2018)
Miller, S.: Auto-dynamic difficulty. Scott Miller’s Game
▶ Crowd Evacuation Using Simulation
Matters Blog. http://dukenukem.typepad.com/game_ Techniques
matters/2004/01/autoadjusting_g.html (2004). Accessed
31 Jan 2019
Miyake, Y.: Current status of applying artificial intelli-
gence in digital games. In: Nakatsu, R., Rauterberg,
M., Ciancarini, P. (eds.) Handbook of Digital Games Mindfulness
and Entertainment Technologies, pp. 253–292.
Springer, New York (2017) ▶ Mindfulness, Virtual Reality, and Video Games
Miyake, Y.: Eos is alive: the AI systems of “Final Fantasy
XV.” In: Game Developers Conference 2018 (GDC
2018) (2018)
Mouret, G., Athanassoff, L.: Intelligent game design on
“Rainbow Six Siege.” In: Game Developers Confer- Mindfulness, Virtual Reality,
ence 2018 (GDC 2018) (2018)
Murray, S., Duncan, G., Doyle, R., Ream, D.: No Man’s
and Video Games
Sky. Hello Games (2016)
Nishikado, T.: Space Invaders. Taito (1978) Mehmet Kosa1 and Ahmet Uysal2
Pascale, M.: Unified telemetry, building an infrastructure 1
Department of Information Systems, Middle
for Big Data in games development. In: Game Devel-
opers Conference 2016 (GDC 2016) (2016)
East Technical University, Ankara, Turkey
2
Rabin, S. (ed.): Game AI Pro 3. A K Peters/CRC Press, Department of Psychology, Middle East
New York (2017) Technical University, Ankara, Turkey
Robertson, G., Watson, I.: A review of real-time strategy
game AI. AI Mag. 35(4), 75–104 (2014)
Rogers, S.: Level Up! The Guide to Great Video Game
Design, 2nd edn. Wiley, Hoboken (2014) Synonyms
Salen, K., Zimmerman, E.: Rules of Play: Game Design
Fundamentals. MIT Press, Cambridge (2003) Mindfulness; Video games; Virtual reality
Mindfulness, Virtual Reality, and Video Games 1127

Definition system was found to be effective in reducing

chronic pain (Gromala et al. 2011). In another
Virtual reality (VR) is “A high-end user-computer study, using virtual reality was shown to be more
interface that involves real-time simulation and effective than nonvirtual reality condition in
interactions through multiple sensorial channels” people’s learning of mindfulness-based stress
(Burdea Grigore and Coiffet 1994). reduction (Gromala et al. 2015). Similarly, virtual
Mindfulness is defined as the intentional pay- reality supported mindfulness meditation environ-
ing attention to the present moment (Kabat-Zinn ments were shown to increase mindfulness
and Hanh 2009). skills and psychological well-being (Crescentini
et al. 2016). In a case study it was shown that
virtual reality-supported mindfulness intervention
Introduction decreases negative feelings along with urges to
commit suicide or self-harm (Nararro-Haro et al.
Mindfulness is characterized by positive emo- 2016). Other attempts include developing virtual
tional states and well-being (Brown and Ryan reality systems that aim to improve mindfulness
2003), and it is researched in many contexts with enjoyable experiences (Amores et al. 2016)
such as health, business, and education and virtual reality systems that are combined with
(Aviles and Dent 2015). Mindfulness results in neurofeedback to provide deep relaxation, pres-
being more sensitive to the environment and ence and deep levels of meditation (Kosunen
more open to new information (Langer and et al. 2016).
Moldoveanu 2000a, b). With the advent of virtual
reality and wide use of digital applications,
the research on mindfulness in virtual reality Mindfulness in Digital Games
and other digital applications is also gaining
importance. Games may be considered at the intersection of
M
There is already a considerable amount of technology and mindfulness, in the sense that they
applications in the mobile market targeting solely may provide full immersion and engagement for
mindfulness, which have variety of features, sup- the player leaving little to no cognitive room for
port different platforms, and have free or paid worrying or other thoughts (Sliwinski et al. 2017).
versions (Plaza et al. 2013). These interactive Although application of mindfulness to the digital
applications are mostly used as support tools for games context is relatively new, there are a
the nurturing of mindfulness (Sliwinski et al. few studies that incorporate mindfulness theory
2017). Although more research is needed on the to video game research. Some of these studies
interaction between technology and mindfulness, approach the topic from a theoretical perspective,
research shows that mobile-based electronic inter- and mainly examine the relation between mind-
ventions for mindfulness is potentially effective fulness and other user experience constructs. For
for stress reduction (Lyzwinski et al. 2017). instance, one study showed that mindfulness was
positively associated with frequency of game
play, length of game play sessions, total number
Mindfulness in Serious Virtual Reality of games played, and the age begun gaming
Applications (younger meaning higher scores) (Gackenbach
and Bown 2011). Another study investigated the
There are several studies which show that virtual relationship between social anxiety and state
reality systems and their applications help induc- mindfulness in online first person shooter (FPS)
ing mindfulness or increase people’s mindfulness games (Gavriloff and Lusher 2015). It was found
skills, which in return leads to improved well- that more socially anxious players were less mind-
being. For instance, augmenting mindfulness in ful in the game. Finally, a project shows that how
chronic pain patients by the use of a virtual reality an innovative use of Minecraft with mindfulness
1128 Mindfulness, Virtual Reality, and Video Games

principles may contribute to digital story telling systems. Thus, research on the long-term accep-
(Butler et al. 2016). tance of virtual reality systems might be needed
There are also other studies, which approach before delineating the processes about mindful-
video games as tools for improving mindfulness. ness in virtual reality.
For instance, researchers provided several
examples for how digital games can improve
mindfulness (Sliwinski et al. 2015). Another References
study describes how procedural content
generation-driven games may increase mindful- Amores, J., Benavides, X., Maes, P.: Psychicvr: increasing
mindfulness by using virtual reality and brain computer
ness of the players and help learning (Smith and
interfaces. In: Proceedings of the 2016 CHI Conference
Harteveld 2013). Extended Abstracts on Human Factors in Computing
Although scarce, there is also some research Systems, CHI EA ’16, San Jose, 7–12 May 2016.
on mindfulness and virtual reality games. For ACM, New York, p. 2 (2016)
Aviles, P.R., Dent, E.B.: The role of mindfulness in leading
instance, one study examined whether a gamified
organizational transformation: a systematic review.
virtual reality system helps gamers learn about J. Appl. Manag. Entrep. 20(3), 31–55 (2015)
mindfulness practices (Choo and May 2014). Brown, K.W., Ryan, R.M.: The benefits of being present:
Another study explains the design of a virtual mindfulness and its role in psychological well-being.
J. Pers. Soc. Psychol. 84(4), 822 (2003)
reality game to help people practice breathing
Burdea Grigore, C., Coiffet, P.: Virtual Reality Technology.
exercises and mindfulness (Patibanda et al. 2017). Wiley-Interscience, London (1994)
Butler, D., Brown, M., Críosta, G.M.: Telling the Story of
MindRising: Minecraft, Mindfulness and Meaningful
Learning. International Association for Development
Conclusion and Discussion of the Information Society. Paper presented at the Inter-
national Conferences on Internet Technologies & Soci-
Taken together, these studies show that there is ety (ITS), Education Technologies (ICEduTECH), and
potential for use of virtual reality in improving Sustainability, Technology and Education (STE), Mel-
bourne, 6–8 December 2016 (2016). https://eric.ed.
mindfulness and wellbeing. Although mindful- gov/?id=ED571584
ness in virtual reality and games appear to be Choo, A., May, A.: Virtual mindfulness meditation:
separate lines of research, it is likely that these virtual reality and electroencephalography for health
lines of research will intersect in future. gamification. In: Games Media Entertainment (GEM),
2014 IEEE, Toronto, 22–24 October 2014. IEEE, pp.
Mindfulness in computing is a relatively fresh 1–3 (2014). https://doi.org/10.1109/GEM.2014.
research area with limited number of studies, 7048076. http://ieeexplore.ieee.org/abstract/document/
and the literature on mindfulness in virtual 7048076/
reality games is even scarcer. Thus, more Crescentini, C., Chittaro, L., Capurso, V., Sioni, R.,
Fabbro, F.: Psychological and physiological responses
research is needed to understand the processes to stressful situations in immersive virtual reality:
related to mindfulness in virtual reality and differences between users who practice mindfulness
other digital applications. The use of neuro- meditation and controls. Comput. Hum. Behav. 59,
feedback in mindfulness-related virtual reality 304–316 (2016)
Gackenbach, J., Bown, J.: Mindfulness and video
gaming looks promising for future applications. game play: a preliminary inquiry. Mindfulness. 2(2),
From a theoretical perspective, the associations 114–122 (2011)
between mindfulness and long-studied phenom- Gavriloff, D., Lusher, J.: Social anxiety and mindfulness
ena such as immersion, flow, and engagement in online gamers. Comput. Games J. 4(1–2), 123–132
(2015)
in virtual reality domain might also provide inter- Gromala, D., Song, M., Yim, J.D., Fox, T., Barnes, S.J.,
esting future research directions. However, the Nazemi, M., Shaw, C., Squire, P.: Immersive VR: a non-
use of virtual reality is not widespread as of the pharmacological analgesic for chronic pain?
time of this writing. Moreover, to the best of our In: Proceeding CHI EA’11, CHI ’11 Extended Abstracts
on Human Factors in Computing Systems, Vancouver,
knowledge there are no studies that examine the 7–12 May 2011. ACM, New York, pp. 1171–1176
acceptance or long-term adoption of these (2011)
Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems 1129

Gromala, D., Tong, X., Choo, A., Karamnejad, M.,

Shaw, C.D.: The virtual meditative walk: virtual real- Minkowski-Lorentz Spaces
ity therapy for chronic pain management. In: Pro-
ceedings of the 33rd Annual ACM Conference on Applications: Resolution of
Human Factors in Computing Systems, CHI ’15, Apollonius and Dupin
Seoul, 18–23 April 2015. ACM, New York, pp. Problems
521–524 (2015)
Kabat-Zinn, J., Hanh, T.N.: Full Catastrophe Living: Using
the Wisdom of Your Body and Mind to Face Stress, Lionel Garnier1, Jean-Paul Bécar2,
Pain, and Illness. Delta. Penguen Random Company, Lucie Druoton1, Laurent Fuchs3 and
New York (2009) Géraldine Morin4
Kosunen, I., Salminen, M., Järvelä, S., Ruonala, A., 1
Arts et Métiers, University of Burgundy-
Ravaja, N., Jacucci, G.: RelaWorld: Neuroadaptive
and immersive virtual reality meditation system. Franche-Comté, Dijon, France
2
In: Proceedings of the 21st International Confer- LAMAV-CGAO, FR CNRS 2956 EA 4015,
ence on Intelligent User Interfaces, IUI ’16, Valenciennes, France
Sonoma, 07–10 March 2016. ACM, New York, 3
Université de Poitiers, Chasseneuil, France
pp. 208–217 (2016) 4
Langer, E.J., Moldoveanu, M.: Mindfulness research and Laboratoire IRIT, Université Paul Sabatier,
the future. J. Soc. Issues. 56(1), 129–139 (2000a) Toulouse, France
Langer, E.J., Moldoveanu, M.: The construct of mindful-
ness. J. Soc. Issues. 56(1), 1–9 (2000b)
Lyzwinski, L.N., Caffery, L., Bambling, M.,
Edirippulige, S.: A systematic review of electronic Synonyms
mindfulness-based therapeutic interventions for
weight, weight-related behaviors, and psychological Linear solving of Apollonius problem; Linear
stress. Telemed. e-Health. (2017). https://doi.org/10. solving of Dupin problem
1089/tmj.2017.0117
Nararro-Haro, M.V., Hoffman, H.G., Garcia-Palacios, A.,
Sampaio, M., Alhalabi, W., Hall, K., Linehan, M.: The
use of virtual reality to facilitate mindfulness skills Definitions
M
training in dialectical behavioral therapy for borderline
personality disorder: a case study. Front. Psychol. 7,
1573 (2016) Introduction of Minkowski-Lorentz spaces to
Patibanda, R., Mueller, F., Leskovsek, M., Duckworth, J.: simplify Euclidean 2- or 3-dimensional problems.
Life Tree: Understanding the Design of breathing exer-
cise games. In: Proceedings of the Annual Symposium
on Computer-Human Interaction in Play, CHI PLAY
’17, Amsterdam, 15–18 October 2017. ACM, New Introduction
York, pp. 19–31 (2017)
Plaza, I., Demarzo, M.M.P., Herrera-Mercadal, P., The Apollonius problem is the determination, in a
García-Campayo, J.: Mindfulness-based mobile
applications: literature review and analysis of cur-
plane, of a circle tangent to three given geometric
rent features. JMIR mHealth uHealth. 1(2), e24 circles which corresponds to 23 ¼ 8 Apollonius
(2013) cases using oriented circles. The use of the
Sliwinski, J., Katsikitis, M., Jones, C.M.: Mindful gaming: Minkowski-Lorentz space permits to choice one
how digital games can improve mindfulness. In:
Human-Computer Interaction, pp. 167–184. Springer,
over eight cases. Moreover, it generalizes when an
Cham (2015) oriented circle becomes an oriented line, the new
Sliwinski, J., Katsikitis, M., Jones, C. M.: A review of number of solutions is given using the same algo-
interactive technologies as support tools for the culti- rithm whereas the new number of solutions is
vation of mindfulness. Mindfulness. 8(5), 1150–1159
(2017)
multiplied by 4 in the usual Euclidean affine
Smith, G., Harteveld, C.: Procedural content generation as space. The centers of the circles tangent to two
an opportunity to Foster collaborative mindful learning. given circles are on a conic. So, solving one of the
In: Proceedings of the SIGCHI Conference on Human Apollonius problems (with oriented circles) leads
Factors in Computing Systems, CHI ’14, Toronto, 26
April–1 May 2014. ACM, New York, pp. 917–926
to the computation of the intersection of two
(2014) conics whereas the problem is linear on the circles
1130 Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems

space Λ3 in the Minkowski-Lorentz space L3,1. Bécar 2017; entry “▶ Theory of Minkowski-
The Dupin problem is the determination of a Lorentz Spaces”; Garnier et al. 2017, 2018;
surface tangent to three given oriented spheres in Garnier and Druoton 2013)
E 3. The surface is called a Dupin Cyclide
(Darboux 1873, 1887, 1917; Dupin 1822; Dutta
et al. 1993; Forsyth 1912; Garnier 2007; Pratt n1-Spheres Tangent to Three Given
1990, 1995); its degree is at most 4. To determine n1-Spheres
the Dupin cyclide, the computation of the two
Dupin cyclide principal circles in a plane is nec- In this section, the authors consider three oriented
essary. Then, to solve Dupin problem leads to n1-spheres, and the goal is to determine the
solve the Apollonius problem. One can define a oriented n – 1-spheres tangent to the three afore-
Dupin cyclide as a canal surface in two different mentioned oriented n1-spheres. If n ¼ 2 (resp.
equivalent ways. A Dupin cyclide is the envelop n ¼ 3), this problem is known as Apollonius
of a one-parameter family of oriented spheres problem (resp. Dupin problem). Using
centered on a conic (Druoton 2013; Druoton Minkoswki-Lorentz space, the Algorithm 1 per-
et al. 2013a, 2014; Forsyth 1912; Langevin et al. mits to resolve these problems.
2015). As a cubic Dupin cyclide is the envelop of One can note that all formulas in (1) are linear
one-parameter family of oriented spheres centered and force that the n – 1-spheres to have the same
on a parabola, the authors generalize and solve orientation at the tangency points. Formula (2)
this problem in E 2 and E 3 in section “Family of permits to find the representation of the n1-
n1-Spheres Centered onto a Parabola of E n”. To spheres on Λn + 1.
simplify the solving of these problems, the Each equation given in Formula (1) defines a
authors use the Minkowski-Lorentz space (Bécar hyperplane in the Minkoswki-Lorentz space. The
et al. 2016; Druoton et al. 2013b; Garnier and dimension of each hyperplane is n + 1, and the

Minkowski-Lorentz 4
Spaces Applications:
Resolution of Apollonius 3 M1a
and Dupin Problems,
Fig. 1 Solution of the Ω1
2
Apollonius problem in E 2 M1b
when all the radius r0, r1, 1 C1
and r2 are nonnegative. The
radii of the circles which are M0a Ωb
0 M0b
solutions of our problem Ω0
have not the same sign; ra
(resp. rb) is nonnegative
-1 C0 Ωa Cb
(resp. nonpositive)
-2 C2
M2b
-3
Ω2
-4

-5
M2a
-6 Ca

-7
-5 -4 -3 -2 -1 0 1 2 3 4 5 6 7
Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems 1131

dimension of the set defined by Formula (1) is !

Onþ2 s2 ¼ 1 ð2Þ
n1. Using E 2 (resp. E 3), the authors obtain a line
(resp. a two-plane), and the intersection between
3. For i in 〚0, 2〛, from the solutions ss of the
this line and Λn + 1 is two points (resp. a circle for
equations given by Formulas (1) and (2), compu-
the Lorentz form or a parabola isometric to a line).
tation of tangency points using the light-like vec-
tors ss!si
Algorithm 1: Solving of Apollonius-Dupin
4. Computation of the oriented n1-spheres Ss
Problems
of E n from the solutions ss of the equations given
Input: Three oriented n1-spheres S0, S1, and S2
by Formulas (1) and (2)
of E n, n ∈ {2,3}
Output: A family of oriented n 1-spheres
1. For i in 〚0, 2〛, computation of the
tangent (with the same orientation) to S0, S1, and S2
representation si in Ln + 1,1 of the oriented n1-
sphere Si
Linear Solving of Apollonius Problem
2. Determination of s by
The Figs. 1 and 2 shows a solution of the Apollo-
nius problem in E 2 with the same geometric cir-
! ! cles (Table 1).
Onþ2 s  Onþ2 s0 ¼ 1
! ! In Fig. 1, Formulas (1) and (2) lead to
Onþ2 s  Onþ2 s1 ¼ 1 ð1Þ
! !
Onþ2 s  Onþ2 s2 ¼ 1

and

p p p p
M
32 6061  1058 12 6061  203 6 6061  286 8 6061  278
sa , , ,
6975 2325 2325 465
p p p p
32 6061  1058 12 6061  203 6 6061  286 8 6061  278
sb , , ,
6975 2325 2325 465

which define the circles Ca and Cb of centers Ωa The radius of Ca is

and centers Ωb and of radii ra and rb. The coor-
p
dinates of Ωa are 48 6061 þ 1587
ra ¼ ’ 4:86646835086
1094
p p
4 6061 þ 1363 5 6061  473 whereas the radius of Cb is
,
1094 547
p
’ ð1:53053902924, 1:5763475731Þ 48 6061 þ 1587
rb ¼ ’ 1:96518864336
1094
whereas the coordinates of Ωb are The light-like vectors which define the points
of tangency are defined in Table 2.
p p
4 6061 þ 1363 5 6061  473 In the Fig. 1, the radius r2 is changed into its
,
1094 547 opposite, and the Fig. 2 shows the solution of this
new Apollonius problem in E 2.
’ ð0:961234279272, 0:1530856993Þ Formulas (1) and (2) lead to
1132 Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems

Minkowski-Lorentz 6
Spaces Applications:
Resolution of Apollonius 5
and Dupin Problems,
Fig. 2 Solution of the 4
Apollonius problem in E 2,
the radius r2 used in Fig. 1,
3 C1
is changed into its opposite
Ωa Ω1
2 M1a
M1b
C0
1
Ω0
0
M0a M0b
-1 Ca
M2b
-2
C2
-3
Ω2 Ωb
-4
M2a
-5

-6

-7 Cb

-8
-5 -4 -3 -2 -1 0 1 2 3 4 5 6 7

p p p p
32 319 þ 34 12 319  181 6 319 þ 478 8 319  86
sa , , ,
2325
p 775
p p775 p 155
32 319 þ 34 12 319  181 6 319 þ 478 8 319  86
sb , , ,
2325 775 775 155

which define the circles Ca and Cb of centers Ωa The radius of Ca is

and centers Ωb and of radii ra and rb. The coor-
dinates of Ωa are p
16 319  17
p p ra ¼ ’ 3:840
70
4 319 þ 83 319  5
, ’ ð0:165, 1:837Þ
70 7 whereas the radius of Cb is

whereas the coordinates of Ωb are p

16 319  17
p p rb ¼ ’ 4:325
4 319 þ 83  319  5 70
,
70 7
The light-like vectors which define the points
’ ð2:206, 3:266Þ of tangency are defined in Table 3.
Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems 1133

Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems,

Table 1 Characteristic of circles in the Apollonius problem in Figs. 1 and 2 and their representations on Λ3
Name Center Radius On Λ3 Figure
!
C0 Ω0(2, 0) r0 ¼ 1 O4 s0 ¼ !
eo  2e! 3 !
1 þ 2 e1
Figures 1 and 2
! !
C1 Ω1(3, 2) r1 ¼ 1 O4 s1 ¼ eo þ 3e! ! !
1 þ 2e2 þ 6 e1
Figures 1 and 2
Cþ Oþ rþ ¼2 ! 1! ! !
2 ð0, 4Þ O4 sþ Figure 1
2 ¼ 2 eo  2e2 þ 3 e1
2 2

C O r !
2 2 ð0, 4Þ 2 ¼ 2 O s ¼  1 e þ 2e  3 e!
4 2
 !
2 o
!
2 1
Figure 2

Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems,

Table 2 Determination of the tangency points in the Apollonius problem in Fig. 1
Circles Light-like vectors Point of tangency
C0 and Ca m!0a ’ ð1, 2:913, 0:408, 4:326Þ M0a ’ (2.913, 0.408)
C1 and Ca m!1a ’ ð1, 3:380, 2:925, 9:990Þ M1a ’ (3.380, 2.925)
C2 and Ca m!2a ’ ð1, 1:068, 5:691, 16:764Þ M2a ’ (1.068, 5.691)
C0 and Cb m!0b ’ ð1, 1:001, 0:051, 0:503Þ M0b ’ (1.001, 0.051)
C1 and Cb m!1b ’ ð1, 2:312, 3:485, 3:485Þ M1b ’ (2.312, 3.485)
C2 and Cb m! ’ ð1, 0:485, 2:060, 2:239Þ
2b M2b ’ (0.485, 2.060)

Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems,

Table 3 Determination of the tangency points in the Apollonius problem in Fig. 2
M
Circles Light-like vectors Point of tangency
C0 and Ca m!
0a ’ ð1, 2:762, 0:647, 4:025Þ M0a ’ (2.762, 0.647)
C1 and Ca m!
1a ’ ð1, 3:998, 2:057, 10:110Þ M1a ’ (3.998, 2.057)
C2 and Ca m!
2a ’ ð1, 0:057, 2:001, 2:003Þ M2a ’ (0.057, 2.001)
C0 and Cb m!
0b ’ ð1, 1:210, 0:613, 0:920Þ M0b ’ (1.210, 0.613)
C1 and Cb m!
1b ’ ð1, 2:851, 1:011, 4:575Þ M1b ’ (2.851, 1.011)
C2 and Cb m! ’ ð1, 1:900, 4:631, Þ
2b M2b ’ (1.900, 4.631)

Figure 3 shows the solution of the Apollo- Formulas (1) and (2) lead to
nius problem in E 2 when a circle is a hyper-
plane defined by the point P 0 (3,0) and the 40 7 20 66
sa , , ,
normal unit vector !
e1 ð1, 0Þ and the radius of C1 141 47 47 47
has changed into its opposite, i.e., r1 ¼ 1; Cþ2 sb ð0, 1, 0, 2Þ
is kept. The representation of the line is given
by which define the circle Ca of center Ωa and of
radius ra and the hyperplane Cb defined by the
! point Ωb (2,0) and the unit vector !
e1 . The coordi-
O4 s0 ¼ e!1 þ 3 e!
1
nates of Ωa are
whereas the representation of the circle C1 is
21 3
, ’ ð0:525, 1:5Þ
40 2
! !  3e!  2e
!  6 e!
O4 s0 ¼ e o 1 2 1 and the radius of Ca is
1134 Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems

Minkowski-Lorentz 6
Spaces Applications:
Resolution of Apollonius 5
and Dupin Problems,
Fig. 3 Solution of the 4
Apollonius problem in E 2,
the authors change the circle C1
3
S0 used in Fig. 1 into a
hyperplane (a line) M1b Ω1 M1a
2 Ωa
M0a
1
P0 →
e1 Ωb →
e1
0

-1
Ca
-2
C2 M2a
-3
C0
Ω2
-4 M2b

-5 Cb

-6

-7
-5 -4 -3 -2 -1 0 1 2 3 4 5 6 7

Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems,

Table 4 Determination of the tangency points in the Apollonius problem in Fig. 3
Circles Light-like vectors Point of tangency
C0 and Ca m! ’ 1, 3, 3 , 45
0a 2 8 M0a ’ 3, 32
C1 and Ca m!
1a ’ ð1, 3:980, 2:198, 10:337Þ M1a ’ (3.980, 2.198)
C2 and Ca !
m2a ’ ð1, 0:190, 2:009, 2:036Þ M2a ’ (0.190, 2.009)
C0 and Cb m! !
0b ¼ e1 M0b ¼ 1
C1 and Cb !
m1b ’ ð1, 2, 2, 4Þ M1b ’ (2, 2)
C2 and Cb m!
2b ’ ð1, 2, 4, 10Þ M2b ’ (2, 4)

141 radius. The Dupin problem is now on

ra ¼ ’ 3:525
40 consideration.

The light-like vectors which define the points Linear Solving of Dupin Problem
of tangency are given in Table 4. One can note that The authors replace the circles given in Table 1
if the authors change the orientation of the line C0, by spheres, and the authors add a zero as last
the solution is two circles with nonnegative component: the centers of the spheres belong to
Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems 1135

the plane of equation z ¼ 0 in E 3. The represen- cyclides (Cayley 1873; Darboux 1887, 1917;
tations of these spheres on Λ4 are given in Druoton 2013; Druoton et al. 2014; Dupin
Table 1; the expressions on Λ3 and Λ4 are the 1822; Garnier and Bécar 2017; Pratt 1990,
same. The solution of the Dupin problem leads 1995). These surfaces are canal surfaces in
to a cylinder of revolution or a cone of revolu- two ways: the centers of the spheres which
tion or cubic or quartic surface: the Dupin define a quartic Dupin cyclide belong to an

Minkowski-Lorentz Spaces Applications: Resolution two spheres Ss,1 and Ss,2 tangent to the previous spheres, M
of Apollonius and Dupin Problems, Fig. 4 Solution of and two characteristic circles. (b) The three initial spheres
Dupin problem in E 3 from the Apollonius problem in E 2 S0, S1, and S2, the quartic ring Dupin cyclide, and two
given in Fig. 1. (a) The three initial spheres S0, S1, and S2, characteristic circles

Minkowski-Lorentz Spaces Applications: Resolution two spheres Ss,1 and Ss,2 tangent to the previous spheres,
of Apollonius and Dupin Problems, Fig. 5 Solution of and two characteristic circles. (b) The three initial spheres
Dupin problem in E 3 from the Apollonius problem in E 2 S0, S1, and S2, the quartic horned Dupin cyclide, and two
given in Fig. 2. (a) The three initial spheres S0, S1, and S2, characteristic circles
1136 Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems

12
11
10
9
8
7
6
5
4
3
2
1
0
-1
-2
-3
-4
-5
-6
-7
-8
-9
-10
-11
-12
-12-11-10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8

Minkowski-Lorentz Spaces Applications: Resolution circles have been drawn for the values
of Apollonius and Dupin Problems, Fig. 6 The 0,  14 ,  12 ,  34 ,  0:85,  0:95, and  1 in For-
parabola P and some circles (n – 1-spheres) of the one- mulas (3) and (4)
parameter family F in E 2, p ¼  1, and q ¼ 5. These

ellipse and a hyperbola; and the centers of the ða, c, mÞ ’ ð3:416, 0:766, 1:451Þ
spheres which define a cubic Dupin cyclide
belong to two parabolas. Moreover, each and figure shows the solution of this Dupin prob-
sphere of a family of spheres which generates lem (Fig. 4).
the Dupin cyclide is tangent to the all spheres of From Fig. 2, the parameters of the quartic
the other family. horned Dupin cyclide are
The circles computed in Figs. 1, 2, and 3 are
called principal circles and permit the determina- ða, c, mÞ ’ ð4:082, 2:748, 0:243Þ
tion of Dupin cyclides parameters.
From Fig. 1, the parameters of the quartic ring and figure shows the solution of this Dupin prob-
Dupin cyclide are lem (Fig. 5).
Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems 1137

Minkowski-Lorentz 12
Spaces Applications: Δ
Resolution of Apollonius 11
and Dupin Problems, 10
Fig. 7 The parabola P and
9
some circles (n – 1-spheres)
of the one-parameter family 8
F and their orthogonal 7
n1-spheres in E 2, p ¼ 1
and q ¼ 5 and the envelope 6
of F which is the union 5
between a circle C and a
line Δ. These circles have 4
been drawn for the values 0, 3
 12, and  1 in Formulas (3) 2
and (4)
1
0
C
-1
-2
-3
-4
-5
-6
-7
-8
M
-9
-10
-11
-12
-7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12

Family of n1-Spheres Centered onto a where t ∈ ℝ.

Parabola of E n The one-parameter family F of oriented n–1-
spheres is defined: the centers are Ω(t), and the
Let p et q be two reals such as radii are

pq < 0
q  ðp  qÞt2
rðtÞ ¼ , t∈ℝ ð4Þ
2
The parabola P is defined as the set of Ω(t)
defined like this
and the envelope defined by the family F is com-
puted. Figure 6 shows an example in E 2 with
! q ðp  qÞ 2 !
O n OðtÞ ¼ þ t e1 p ¼  1 and q ¼ 5.
2 2
The representation of the family F in the
þ ðp  qÞt !
e 2 ð3Þ Minkowski-Lorentz space is
1138 Minkowski-Lorentz Spaces Applications: Resolution of Apollonius and Dupin Problems

Minkowski-Lorentz Spaces Applications: Resolution spheres (n1-spheres) of the one-parameter family. (b)
of Apollonius and Dupin Problems, Fig. 8 Generation The cubic Dupin cyclide and three characteristic circles
of a cubic Dupin cyclide in E 3. (a) The parabola P and three

•
! 2 ! t2 pðp  qÞ ! and s ð0Þ is the representation of the hyperplane
Onþ2 sðtÞ ¼ e þ e
q  ðp  qÞt2 o
q  ðp  qÞt2 1 defined by the point O n and the unit normal vector
q þ ðp  qÞt2 ! 2ðp  qÞt ! sign ðqðp  qÞÞ !e2 . Figure 7 shows an example in
þ e þ e E 2 with p ¼  1 and q ¼ 5; the envelope is the
q  ðp  qÞt2 1 q  ðp  qÞt2 2
circle C and the line Δ.
and the derivative sphere (15) is Figure 8 shows an example in E 3 with p ¼  1
and q ¼ 5; the canal surface is a cubic Dupin
! 2t ðp  qÞ cyclide. Using the construction in the plane of
• !
Onþ2 s ðt Þ ¼ eo equation z ¼ 0, the two principal circles which
ðp  qÞ2 t 2 þ qðq  pÞ
permit the computation of the Dupin cyclide
pqðp  qÞt
þ e!
1
parameter are calculated. Moreover, in this
ðp  qÞ2 t 2 þ qðq  pÞ plane, the Figs. 6 and 7 are obtained.
2t q ðp  qÞ !
þ e1
ðp  qÞ2 t 2 þ qðq  pÞ
ðp  qÞ ððp  qÞt 2 þ qÞ ! Conclusion
þ e2
ðp  qÞ2 t 2 þ qðq  pÞ
This article presents some applications of the
which generates, it t 6¼ 0, the n1-sphere of center Minkowski-Lorentz space and its interest for
• computer graphics. In section “n – 1-Spheres Tan-
OðtÞ defined by gent to Three Given n – 1-Spheres,” the represen-
! tation of spheres in the Minkowski-Lorentz space
2
•
! þ ðp  qÞt þ q !
O n OðtÞ ¼ qe e2 permits to solve classical geometric problems like
1
2t Apollonius problems or Dupin problems. First, as
the point at infinity is seen as any point, we get a
and of radius
more general solution including spheres and
hyperplane. Second, a lot of quadratic computa-
• ðp  qÞ2 t2 þ qðq  pÞ tions in the Euclidean space become linear in this
r ðt Þ ¼
2tðp  qÞ space. In section “Family of n–1-Spheres
Mixed Reality 1139

Centered onto a Parabola of E n,” an envelope of a tangents at a vertex. In: XIV Mathematics of Surfaces,
family of oriented circles or spheres is computed. pp. 237–276, Birmingham, Royaume-Uni, 11–13 sep-
tember (2013)
Garnier, L., Bécar, J.-P., Druoton, L.: Canal surfaces as
Bézier curves using mass points. Comput. Aided
References Geom. Des. 54, 15–34 (2017)
Garnier, L., Bécar, J.-P., Druoton, L., Fuchs, L., Morin, G.:
Bécar, J.P., Druoton, L., Fuchs, L., Garnier, L., Langevin, Theory of Minkowski-Lorentz Spaces, pp. 1–17.
R., Morin, G.: Espace de Minkowski-Lorentz et espace Springer International Publishing, Cham (2018)
des sphères: un état de l’art. In: GTMG 2016, Dijon, Langevin, R., Sifre, J.-C., Druoton, L., Garnier, L.,
France, Mars 2016. Le2i, Université de Bourgogne Paluszny, M.: Finding a cyclide given three contact
Cayley, A.: On the cyclide. Q. J. Pure Appl. Math. 12, conditions. Comput. Appl. Math. 34, 1–18 (2015)
148–165 (1873) Pratt, M.J.: Cyclides in computer aided geometric design.
Darboux, G.: Sur une Classe Remarquable de Courbes et Comput. Aided Geom. Des. 7(1–4), 221–242 (1990)
de Surfaces Algébriques et sur la Théorie des Pratt, M.J.: Cyclides in computer aided geometric design
Imaginaires. Gauthier-Villars, Paris (1873) II. Comput. Aided Geom. Des. 12(2), 131–152 (1995)
Darboux, G.: Leçons sur la Théorie Générale des Surfaces,
vol. 1. Gauthier-Villars, Paris (1887)
Darboux, G.: Principes de géométrie analytique. Gauthier-
Villars, Paris (1917)
Druoton, L.: Recollements de morceaux de cyclides de Minutiae Extraction
Dupin pour la modélisation et la reconstruction 3D.
PhD thesis, Université de Bourgogne, Institut de ▶ Fingerprint Verification Based on Combining
Mathématiques de Bourgogne, avril (2013) Minutiae Extraction and Statistical Features
Druoton, L., Langevin, R., Garnier, L.: Blending canal
surfaces along given circles using Dupin cyclides. Int.
J. Comput. Math., 1–20 (2013a)
Druoton, L., Garnier, L., Langevin, R.: Iterative construc-
tion of Dupin cyclide characteristic circles using non- Mixed Reality
stationary iterated function systems (IFS). Comput. M
Aided Des. 45(2), 568–573 (2013b). Solid and Physical
1 2
Modeling 2012, Dijon Vlasios Kasapakis , Damianos Gavalas and
Druoton, L., Fuchs, L., Garnier, L., Langevin, R.: The non- Elena Dzardanova2
degenerate Dupin cyclides in the space of spheres using 1
Department of Cultural Technology and
geometric algebra. AACA. 23(4), 787–990 (2014).
ISSN 0188-7009 Communication, University of the Aegean,
Dupin, C.P.: Application de Géométrie et de Méchanique à Mytilene, Greece
la Marine, aux Ponts et Chaussées, etc. Bachelier, Paris 2
Department of Product and Systems Design
(1822) Engineering, University of the Aegean,
Dutta, D., Martin, R.R., Pratt, M.J.: Cyclides in surface and
solid modeling. IEEE Comput. Graph. Appl. 13(1), Ermoupoli, Greece
53–59 (1993)
Forsyth, A.R.: Lecture on Differential Geometry of Curves
and Surfaces. Cambridge University Press, Cambridge Synonyms
(1912)
Garnier, L.: Mathématiques pour la modélisation
géométrique, la représentation 3D et la synthèse Augmented Reality; Augmented Virtuality; Vir-
d’images. Ellipses, Paris (2007). ISBN: 978-2-7298- tual Environments; Virtual Reality; Virtuality
3412-8 Continuum
Garnier, L., Bécar, J.P.: Nouveaux modèles géométriques
pour la C.A.O. et la synthèse d’images: courbes de
Bézier, points massiques et surfaces canal. Editions
Universitaires Européennes, Saarbrucken (2017). Definitions
ISBN 978-3-639-54676-7
Garnier, L., Druoton, L.: Constructions of principal
patches of Dupin cyclides defined by constraints: four The Virtuality Continuum represents a scale
vertices on a given circle and two perpendicular which extends from the completely Real
1140 Mixed Reality

Mixed Reality, Fig. 1 Virtuality continuum

Environment (RE) to the completely Virtual Envi- virtual elements, based on the location of the
ronment (VE) with Mixed Reality (MR) laying in fiducial marker (Azuma et al. 2001). Figure 2a
between. MR refers to environments anywhere shows an example of marker-based AR, using
between the extremes of the Virtuality Contin- the popular AR development framework Vuforia
uum, wherein real world and virtual world objects (https://www.vuforia.com/), where the RE is pro-
are presented together in a single display. Aug- jected to the user through a web-camera feed and
mented Reality (AR) superimposes computer- augmented by a 3D model based on the location of
generated objects upon the RE, while Augmented a fiducial marker.
Virtuality (AV) blends real-world elements into Marker-less AR is based on computer vision
the VE. Virtual Reality (VR) refers to entirely and motion tracking technology, to accurately
synthetic worlds, which may mimic the physical place 3D objects on top of the RE which the user
properties of the real world, wherein the user can sees through the device’s camera feed (Azuma
be totally immersed. et al. 2001). A popular framework for marker-
less AR development is ARCore (https://devel
opers.google.com/ar/discover/). ARCore blends
Introduction virtual objects with the RE, employing
(a) motion tracking, to estimate the phone’s posi-
The Virtuality Continuum encompasses all possi- tion and orientation in relation to the world and
ble variations of blending real and virtual ele- ensure accurate projection of virtual objects;
ments into a single environment, with the RE (b) perception of the environment, to detect the
and VE laying at the ends of the scale. MR spans dimensions and positioning of flat horizontal sur-
between RE and VE, referring to environments faces so as to place virtual objects on them;
which integrate virtual and physical elements (c) light estimation, to properly estimate the
within a coherent space (see Fig. 1), with the RE’s light conditions, thus overlay the virtual
two most popular MR paradigms being AR and objects upon the RE with appropriate shadows
AV (Milgram and Kishino 1994). and lighting (see Fig. 2b).
The third method for realizing AR applications
is sensor-based AR, wherein multiple commodity
Augmented Reality (AR) sensors are utilized to extract device’s rotation and
direction. This information allows the appropriate
AR applications may be implemented through positioning of virtual elements on top of the RE,
three different methods. Marker-based AR is which the user sees using the camera feed or see-
based on vision tracking and relies upon the place- through capability of the device. Typically,
ment of fiducial markers into the RE, which are sensor-based AR utilizes GPS to acquire user
then tracked via a camera feed, projected onto location, providing accurate outdoors positioning
displays of devices such as smartphones, Head- with respect to virtual elements, which are often
Mounted Displays (HMDs), or Personal Com- bound to real locations (Zhou et al. 2008).
puters (PC). In marker-based, the RE is streamed Figure 2c illustrates an example of sensor-based
through the camera feed and superimposed by AR with GPS integration, created with the AR
Mixed Reality 1141

Mixed Reality, Fig. 2 (a) Marker-based AR; (b) marker-less AR; (c) sensor-based AR with GPS integration

M
Mixed Reality, Fig. 3 (a) Using real hands and objects into the VE; (b) full body representation into VE using motion
capture

application development framework Wikitude embedding real-world camera feeds and a few user
(https://www.wikitude.com/), where the RE is movements into the VE (Regenbrecht et al. 2004).
superimposed with virtual elements based on the However, recent technological advances, such
device’s rotation, direction, and location. as data gloves (Silva et al. 2013) and motion
The outspread of camera- and sensor-enabled capture (Chan et al. 2011), fueled the develop-
smartphones and HMDs has resulted in the estab- ment of AV worlds, facilitating the integration of
lishment of AR as the most widespread type of real elements into the VE. Figure 3a demonstrates
MR, with numerous applications in a variety of an AV example where a data-glove, powered by
fields, such as gaming, tourism, military, and med- Arduino (https://www.arduino.cc/), provides
icine (Azuma et al. 2001). information of the user’s finger position and rota-
tion, while the hand’s overall position, along with
the one of another real object, is tracked using a
Augmented Virtuality (AV) motion capture system by Vicon (https://www.
vicon.com/). This implementation enables con-
AV refers to the augmentation of the VE with real current interaction with both real and virtual
elements, aiming at enriching the overall user expe- objects. Figure 3b illustrates another example of
rience (Schnabel et al. 2007). AV allows users to AV, wherein the motion capture system by Vicon
navigate within a completely synthetic world, is integrated with an Oculus Rift (https://www.
enabling interaction with either fictional or real oculus.com/rift/) HMD to transfer the full body
objects (Ternier et al. 2012). AV projects developed movement of a user from the RE to the VE in real
in the past are scarce and have mostly focused on time.
1142 Mixed Reality and Immersive Data Visualization

Conclusion technology. IEEE Trans. Learn. Technol. 4, 187–195

(2011)
Milgram, P., Kishino, F.: A taxonomy of mixed reality
In 1994, Milgram and Kishino foresaw that visual displays. IEICE Trans. Inf. Syst. 77,
delimiting AR and AV worlds would be hard, as 1321–1329 (1994)
future technological advances would allow seam- Regenbrecht, H., Lum, T., Kohler, P., Ott, C., Wagner, M.,
less blending between RE and VE (Milgram and Wilke, W., Mueller, E.: Using augmented virtuality for
remote collaboration. Presence Teleop. Virt. 13,
Kishino 1994). Their prediction has been con- 338–354 (2004)
firmed to a certain degree, since the predominance Schnabel, M.A., Wang, X., Seichter, H., Kvan, T.: From
of either RE or VE in recent AR and AV worlds is virtuality to reality and back. In: Proceedings of the
equivocal. MR remains the most widespread term International Association of Societies of Design
Research, p. 15 (2007)
used to describe the area between RE and VE and Silva, L., Dantas, R., Pantoja, A., Pereira, A.: Development
represents a particularly promising field of of a low cost dataglove based on arduino for virtual
research and commercial exploitation. reality applications. In: Proceedings of the International
Conference on Computational Intelligence and Virtual
Environments for Measurement Systems and Applica-
tions (CIVEMSA), IEEE, pp. 55–59 (2013)
Cross-References Ternier, S., Klemke, R., Kalz, M., Van Ulzen, P., Specht,
M.: ARLearn: augmented reality meets augmented vir-
▶ 3D Puzzle Games in Extended Reality tuality. J. Univers. Comput. Sci. 18, 2143–2164 (2012)
Zhou, F., Duh, H.B.-L., Billinghurst, M.: Trends in aug-
Environments mented reality tracking, interaction and display: a
▶ Augmented Reality for Maintenance review of ten years of ISMAR. In: Proceedings of the
▶ Augmented Reality Entertainment: Taking 7th International Symposium on Mixed and Aug-
Gaming Out of the Box mented Reality, IEEE/ACM, pp. 193–202 (2008)
▶ Everyday Virtual Reality
▶ Experiential Media: Using Machine Vision and
Sensor-Input to Create Dynamic Real-Time
Generated Media Mixed Reality and Immersive
▶ History of Augmented Reality Data Visualization
▶ Interaction with Mobile Augmented Reality
Environments Isabel Cristina Siqueira da Silva
▶ Key Early Verticals: Challenges and Limita- UniRitter Laureate International Universities,
tions in Implementation of Augmented Reality Porto Alegre, Brazil
▶ Making Virtual Reality (VR) Accessible for
People with Disabilities
▶ Mixed Reality Synonyms
▶ Mixed Reality, Gamified Presence, and Story-
telling for Virtual Museums Augmented reality; Holography; Immersion;
▶ Origin of Virtual Reality Information visualization; User experience;
▶ Shadow Shooter: All-Around Game with e- Virtual reality
Yumi 3D
▶ Virtual Reality and Robotics
Definitions

References Mixed reality is related to the hybrid visualization

of virtual objects superimposed in real scenarios
Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., with the possibility of interaction with them by the
MacIntyre, B.: Recent advances in augmented reality.
users.
IEEE Comput. Graph. Appl. 21, 34–47 (2001)
Chan, J.C., Leung, H., Tang, J.K., Komura, T.: A virtual Data visualization places the information/data
reality dance training system using motion capture in focus, distinguishing it from the unnecessary
Mixed Reality and Immersive Data Visualization 1143

information and facilitating the understanding of From such concepts, the MR works with
correlated data, thus allowing the recognition of virtual objects (VR) inserted in real scenes
patterns and facilitating inferences about different (AR) as part of this universe (holography), allo-
concepts. wing the direct interaction of the user with
virtual objects with or without the use of spe-
cific devices for this purpose, such as HMD or
Introduction VR/AR glasses.
Besides MR, data (or information) visualiza-
Currently, the visualization of information and tion is another important emergent technology.
data can be improved by the use of techniques Considering which the amount of data in the
related to the concepts of mixed reality (MR) and world is growing faster than ever before, data
immersive user experience. While the MR is visualization techniques help people make sense
related to the merging of real and virtual worlds of the data and turn it into insights (Card et al.
to produce new environments and visualizations, 1999; Mazza 2009). In this context, the idea of
the immersive user experience refers to the quality immersive data visualization is to offer full inter-
that an interaction offers to the user, allowing action with the data, allowing easier recognition
him/her to feel connected to the system and and retention of patterns (Kreylos et al. 2008;
being part of the presented data visualization. Manning et al. 2008; Han et al. 2012; Donalek
MR combines characteristics of virtual reality et al. 2014; Geryk 2015; Olshannikova
(VR), augmented reality (AR), and holography. et al. 2015).
The VR deals with the virtual environment inter- In this context, this article presents and dis-
action, generated by computer, that allow people cusses concepts of MR applied to data visualiza-
to visualize, manipulate, and interact with repre- tion field in order to improve the analysis by users
sentations of a three-dimensional (3D) scenario through an immersive interaction.
and objects in real time. For this, the user interacts
M
with the VR through specific devices such as the
head-mounted display (HMD) or VR glasses, in Background
order to improve the feeling of immersion during
the interaction (Milgram and Kishino 1994; According Kim (2005), the VR consists of
Burdea and Coiffet 2003; Bowman et al. 2004; reproducing a synthetic experience representing
Krevelen and Poelman 2010). a context of virtual simulation to the user. The VR
The AR, on the other hand, allows the visual- system has three essential components: one or
ization of virtual objects combined with scenes more screens, a set of sensors that detect the
from real environments with mobile devices such movements and stimulate the user, and a mobile
as smartphones and tablets or AR glasses and device, console, or computer, which controls the
head-mounted display. whole experience (Fig. 1).
So the real and virtual environments are However, one of the major problems of the
connected, enriching the real world with current VR devices is the motion sickness related
computer-generated virtual objects or other to the user’s movement in the virtual world and
technological devices that seem to coexist in the the monitoring of the image displayed on the
same space and run in real time in an interactive screen (Hettinger and Riccio 1992). This problem
way (Azuma 1997; Azuma et al. 2001; Milgram occurs due to the need to generate stereoscopic
et al. 1994). images (Fig. 2), that is, two scenes being one for
Another way of visualizing and interacting each eye in order to produce a high degree of
with 3D images is through the concept of holo- realism in the sensation of depth, which can
graphy (Leith 1972; Schnars et al. 2015). cause a drop in performance. If the update time
A hologram is a 3D image obtained from the is too long, this will cause nausea and/or
projection of light on two-dimensional figures. discomfort.
1144 Mixed Reality and Immersive Data Visualization

Mixed Reality and

Immersive Data
Visualization, Fig. 1 VR
glasses with a smartphone
inside displaying the video

Mixed Reality and

Immersive Data
Visualization, Fig. 2 VR
stereoscopic images from
the “Romans From Mars
360” game (Majesco
Entertainment 2017)

While VR deals with user immersion in a vir- software modeling, particle model, and optical
tual world, the AR allows the visualization of reflection (Collier et al. 1971). Figure 4 presents
virtual objects combined with scenes from real the optical reflection hologram, a common tech-
environments with mobile devices such as nique based on a plane screen that reflects the 2D
smartphones, tablets, or AR glasses (Fig. 3). AR images in a translucent and reflective material in
provides the user an interaction natural and direct an inverted pyramid format, where the hologra-
with the environment. phy forms at its center.
Both VR and AR technologies need These concepts related to MR are possible to
headwear, eyewear, or mobile device in order to apply to data visualization by promoting such an
allow the user interaction. The holography, immersive experience for the user. In this context,
instead, presents the scene or object in real visualization is a form of communication that
space (floating in midair) without the need for transcends application and technological bound-
3D glasses or similar, allowing the viewer to look aries because it offers a way to data discovery
around objects and see them from a slightly dif- (Defanti et al. 1989). According to Ware (2008),
ferent perspective, as they would in real life. visualization used to be mental images that people
Then, this leads to a more comfortable and natu- formed while they thought about something, but
ralistic viewing experience. Examples of tech- now the term is related to a graphical representa-
niques for generating holograms include tion of some data or concept.
Mixed Reality and Immersive Data Visualization 1145

The immersive data visualization should allow The next section discusses the contribution of
the user to analyze the dataset and/or information the concepts related to MR in order to extract
in order to identify patterns growing, which may value from the data by increasing the user under-
be indicative of trends, and the discovery and standing and improving decision-making capabil-
extraction of new, useful, and interesting knowl- ity through immersive interaction.
edge from databases (mainly nonconventional
databases) (Marr 2017).
Data Visualization and Immersive User
Experience with Mixed Reality

Bowman and McMahan (2007) already affirmed

that the goal of immersive virtual environments
was to let the user experience a computer-
generated world as if it were real, producing a
sense of presence, or “being there,” in the user’s
mind. According to Lemoine et al. (2003), the
success of applications involving virtual elements
depends on the degree of immersion, comfort, and
the nature of the interaction.
Therefore, in MR, the user has the ability to
explore the virtual environment and the real world
at the same time. The MR combines characteris-
tics of VR with AR through the insertion of virtual
objects in the real world and allowing the
immersive interaction of the user with such
M
objects, producing new environments in which
physical and virtual items coexist and interact in
real time (Fig. 5).
There are three basic ideas involving MR and
the user experience: (1) immersion, related to the
Mixed Reality and Immersive Data Visualization,
feeling of being inside environments; (2) interac-
Fig. 3 AR example of a 3D virtual elephant projected
over a real scene generated with Augmented software tion, which corresponds to the ability of the user to
(Augmented 2018) instantly modify the virtual objects projected in

Mixed Reality and Immersive Data Visualization, Fig. 4 3D hologram obtained from a video (Hologram Project by
Kiste 2014) of 2D images projected over a translucent and reflective material in pyramid format
1146 Mixed Reality and Immersive Data Visualization

Mixed Reality and Immersive Data Visualization, Fig. 5 MR generated with Linq mixed-reality headset (Stereolabs
2017; Steele 2016)

the real world through actions on these; and • How many dimensions are involved?
(3) involvement that is the degree of user motiva- • What are the data structures?
tion with certain activity. These concepts can aid • What kind of interaction is required?
the analysis of data, since they amplify the process
of insight, transforming the data and/or amplify- Moreover, the classical Shneiderman’s visual
ing the data as images (Card et al. 1999; Spence information-seeking “mantra” (Shneiderman
2007). 1996) gives more directions about important
When an image is analyzed, a cognitive pro- features in a visualization: “First, overview, then,
cess begins and perceptual mechanisms are acti- zoom and filtering, and finally, details on
vated to identify patterns and segment elements. demand.”
The correct mapping of data to visualization is Graphs are the most intuitive form of data
crucial, since one can discard relevant information visualization by their both hierarchical and
or exceed the amount of irrelevant information relational characteristics. An interactive
(Ware 2004, 2008). Thus, the image should limit graph or tree solves part of the problem, allo-
the amount of information that the user receives wing the user to highlight the information in
while keeping him/her aware of the total informa- focus through selection, but the overlapping
tion space and reducing the cognitive effort (Ward edges are still a problem. Moreover, as the
et al. 2015; Silva 2017). dataset grows, incorporating new concepts
Regarding the common visualization tech- (and their relationships) increases the visuali-
niques, there are two main problems: defining zation complexity. Although one can add
which visual structures should be used to repre- interaction to solve part of the problem, allo-
sent the data and defining the location of such wing the user to select the information he/she
visual structures in the display area (Silva 2017). wants to put into focus, overlapping edges
These problems involve the proposition of ade- remain a problem.
quate visual structures with visual attributes and Besides, data representation in a three-
their location, and reaching a solution involves dimensional (3D) space allows the user to navi-
responding to the following questions: gate through in-depth visual representations,
rotating, expanding, and selecting the desired
• What is the problem? items. However, 3D charts on flat screen can
• What is the nature of the data? make information difficult to understand and
Mixed Reality and Immersive Data Visualization 1147

Mixed Reality and Immersive Data Visualization, Fig. 6 3D hologram obtained from “Holostruction” project
generated with HoloLens (Microsoft Asia News Center 2017; Microsoft HoloLens 2017)

compare because these views require the user semiautomatic means for driving the visual explo-
immersion and depth perception. ration in a immersive way and replacing tradi-
In this context, stereo visualization improves tional 2D charts by interactive and immersive
user immersion through perception related to 3D colorful visualizations where the user can M
the shapes and proportions. The natural inter- “dive into” data and see patterns that are not
action with data enhances the intuitive user discernible on 2D charts.
experience from an extra dimension of informa- Figure 7 presents an image from the
tion. According Marr (2017), by presenting Virtualitics (Donalek 2017), an immersive and
data that wraps around the user, more than the collaborative data exploration platform that
traditional 3D become available. As well as merges artificial intelligence, big data, and
placement on X, Y, or Z coordinates, data MR. In this image, it is possible to visualize a
points can be distinguished by size, color, collaborative and customizable shared space to
transparency, as well as direction and velocity analyze data, build virtual dashboards, and pre-
of movement. sent and discuss insights.
Figure 6 presents an image that applies MR to Besides data visualization and analysis, the
data visualization through the exhibition of inter- possibilities of MR applications are extensive as,
active overlaying holograms and enables the user for example, military drills based on MR will give
to view and manipulate these within a visible, the army capabilities beyond what the human
real-world context. being possesses today (Fig. 8). They will be able
In relation to data analytics visualization, if the to view information in real time, share it with
MR techniques are combined with visual analyt- other comrades, if necessary share their field of
ics, it is possible to amplify the cognition and vision, add virtual objects to the training combat
reduce exploration time of a dataset, allowing field, etc.
the recognition of patterns and facilitating infer- Education, health, arts and entertainment,
ences about different concepts. MR and visual architecture and design, and security are areas
analytics can improve both quality and efficiency that will also benefit from MR technology applied
of data visualization systems, providing to data visualization.
1148 Mixed Reality and Immersive Data Visualization

Mixed Reality and Immersive Data Visualization, Fig. 7 3D dashboard of Virtualitics Immersive Platform (Donalek
2017)

Mixed Reality and Immersive Data Visualization, Fig. 8 3D visualization simulations of a Royal Australian Air
Force (RAAF) and Department of Defense project that uses HoloLens (Odom 2017)

Conclusion interaction became necessary in order to facilitate

the comprehension of domains represented in
Over the last decade, there has been a steady diverse systems.
increase of data made available in several forms, However, it is not simple to create a visualiza-
and efficient methods for data visualization and tion and interaction that displays effectively all
Mixed Reality and Immersive Data Visualization 1149

the information and, at the same time, allows opportunities. Computer. 22(8), 12–25 (1989).
the user to easily perform various operations Los Alamitos, CA
Donalek, C.: Why Does (immersive) Data Visualization
on the dataset. The MR technology can contribute Matter? Technical Report from Virtualitics Blog.
to these aspects once it replicates and extends (2017) Available in https://www.virtualitics.com/why-
natural and physical interaction through advanced immersive-data-visualization-matter/
interfaces. Donalek, C., Djorgovski, G., Davidoff, S., Cioc, A.,
Wang, A., Longo, G., Norris, J., Zhang, J., Lawler, E.,
MR combines concepts of VR, AR, and holo- Yeh, S., Mahabal, A., Graham, M., Drake, A.:
graphy to promote innovation and improve the Immersive and Collaborative Data Visualization
user experience in the interaction of these with Using Virtual Reality Platforms, IEEE International
graphical representations. This is possible due Conference on Big Data, pp. 609–614, Washington,
DC (2014)
to advances in techniques of computer vision,
Geryk, J.: Using visual analytics tool for improving data
graphic processing, and rendering, among others. comprehension. In: Proceedings of the 8th International
MR applied to data visualization allows an Conference on Educational Data Mining, pp. 327–334.
immersive and approximate real-world experi- International Educational Data Mining Society.
Madrid, Spain (2015)
ence to the user in order to make it difficult to
Han, J., Kamber, M., Pei, J.: Data Mining Concepts and
distinguish physical reality from virtual reality. Techniques, 3rd edn. Morgan Kaufmann, Amsterdam
(2012)
Hettinger, L.J., Riccio, G.E.: Visually induced motion
sickness in virtual environments. Presence Teleop.
Cross-References Virt. Environ. 1(3), 306–310 (1992)
Hologram Project by Kiste: Available in https://www.
▶ Cognitive Processing of Information youtube.com/watch?v¼Y60mfBvXCj8 (2014)
Visualization Kim, G.: Designing Virtual Reality Systems: The Struc-
tured Approach. Springer, Secaucus (2005)
▶ Holography, History of
Krevelen, D.W.F.V., Poelman, R.: A survey of augmented
▶ Information Presentation Methods in Virtual reality technologies, applications and limitations. M
Reality Int. J. Virtual Real. 9(2), 1–20 (2010)
Kreylos, O., Bawden, G.W., Kellogg, L.H.:
Immersive visualization and analysis of LiDAR
data. In: Bebis, G., et al. (eds.) Advances in Visual
References Computing. ISVC 2008. Lecture Notes in Computer
Science, vol. 5358. Springer, Berlin/Heidelberg
Aumented: Available in http://www.augment.com/ (2018) (2008)
Azuma, R.: A survey of augmented reality. Presence Leith, E.N.: Dennis Gabor, holography, and the Nobel
Teleop. Virt. 6(4), 355–385 (1997) Prize. Proc. IEEE. 60(6), 653–654 (1972)
Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., Lemoine, P., Vexo, F., Thalmann, D.: Interaction tech-
MacIntyre, B.: Recent advances in augmented reality. niques: 3D menus-based paradigm. In: Proceedings
IEEE Comput. Graph. Appl. 21, 34 (2001) of First Research Workshop on Augmented Virtual
Bowman, D.A., McMahan, R.P.: Virtual reality: how much Reality (AVIR 2003), Geneva (2003)
immersion is enough? Computer. 40(7), 36–43 (2007) Majesco Entertainment: Romans From Mars 360. Available
Bowman, D.A., Kruijff, E., LaViola, J.J., Poupyrev, I.: 3D in http://www.majescoent.com/ (2017)
User Interfaces: Theory and Practice. Addison Wesley Manning, C., Raghavan, P., Schütze, H.: Introduction to
Longman Publishing, Redwood City (2004) Information Retrieval, 1st edn. Cambridge University
Burdea, G., Coiffet, P.: Virtual Reality Technology. Wiley- Press, New York (2008)
IEEE Press, New York (2003) Marr, B.: How VR and AR Will Change How We Visualize
Card, S.K., Mackinlay, J.D., Shneiderman, B.: Read- Data. Technical Report from Forbes Finds. Forbes
ings in Information Visualization: Using Vision to Media LLC (2017)
Think. Morgan Kaufmann Publishers, San Francisco Mazza, R.: Introduction to Information Visualization,
(1999) 1st edn. Springer, London (2009)
Collier, R.J., Collier, C.B., Burckhardt, L.H.: Lin: Optical Microsoft Asia News Center: Oyanagi Construction and
Holography. Academic, New York (1971) Microsoft Japan partner on “Holostruction” project
Defanti, T.A., Brown, M.D., Mccormick, B.H.: Visualiza- using Microsoft HoloLens. Available in https://news.
tion: expanding scientific and engineering research microsoft.com/apac/2017/05/03/oyanagi-construction-
1150 Mixed Reality Serious Games

microsoft-japan-partner-holostruction-project-using-
microsoft-hololens/ (2017) Mixed Reality, Gamified
Milgram, P., Kishino, F.: A taxonomy of mixed reality
visual displays. IEICE Trans. Inf. Syst. E77-D(12), Presence, and Storytelling for
1321–1329 (1994) Virtual Museums
Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Aug-
mented reality: a class of displays on the reality- George Papagiannakis1,2, Efstratios
virtuality continuum. Telemanipulator Telepresence
Technol. SPIE. 2351, 282–292 (1994) Geronikolakis1,2, Maria Pateraki2, Victor M.
Odom, J.: Royal Australian Air Force Using HoloLens to López-Menchero3, Michael Tsioumas4, Stella
Experiment with Augmented Reality. Technical Report Sylaiou5, Fotis Liarokapis6, Athina
from Next Reality. Available in https://next.reality. Grammatikopoulou7, Kosmas Dimitropoulos7,
news/about/ (2017)
Olshannikova, E., Ometov, A., Koucheryavy, Y., et al.: Nikolaos Grammalidis7, Nikolaos Partarakis2,
Visualizing Big Data with augmented and virtual real- George Margetis2, Giannis Drossis1,2, Martha
ity: challenges and research agenda. J. Big Data. 2, Vassiliadi8, Alan Chalmers9, Constantine
22 (2015) Stephanidis1,2 and Nadia Magnenat-Thalmann10
Schnars, U., Falldorf, C., Watson, J., Jüptner, W.: Digital 1
holography. In: Digital Holography and Wavefront Computer Science Department, University of
Sensing. Springer, Berlin/Heidelberg (2015) Crete, Heraklion, Greece
2
Shneiderman, B.: The eyes have it: a task by data type Foundation for Research and Technology Hellas,
taxonomy for information visualizations. In: IEEE Heraklion, Greece
Visual Languages, no. UMCP-CSD CS-TR-3665, 3
pp. 336–343. College Park (1996) Spanish Society of Virtual Archaeology, Seville,
Silva, I.C.S.: Data Visualization and Augmented Reality in Spain
4
Health Education. LAP LAMBERT Academic Publish- Hellenic Ministry of Culture and Sports, Service
ing, Saarbrücken (2017) of Modern Monuments and Technical Works of
Spence, R.: Information Visualization: Design for Interac-
tion, 2nd edn. Pearson/Prentice Hall, Harlow (2007) Central Macedonia, Thessaloniki, Greece
5
Steele, B.: The Linq mixed reality headset blends the Hellenic Open University, Patras, Greece
6
real and the virtual. Technical Report from Engadget. Masaryk University, Ponava, Brno, Czech
Available in https://www.engadget.com/2016/11/17/ Republic
stereolabs-linq-mixed-reality-headset/ (2016) 7
Stereolabs, Inc: Available in: https://www.linqmr.com/ Information Technologies Institute, CERTH,
press/ (2017) Thessaloniki, Greece
8
Ward, M., Grinstein, G., Keim, D.: Interactive Data Visu- Department of Philology, School of Philosophy,
alization: Foundations, Techniques, and Applications. Aristotle University of Thessaloniki,
A K Peters/CRC Press, Boca Raton (2015)
Ware, C.: Information Visualization: Perception for Thessaloniki, Greece
9
Design, 2nd edn. Morgan Kaufmann, San Francisco WMG, University of Warwick, Coventry, UK
10
(2004) MIRALab, University of Geneva, Geneva 4,
Ware, C.: Visual Thinking: For Design. Morgan Kaufmann Switzerland
Series in Interactive Technologies, 1st edn. Morgan
Kaufmann, Amsterdam (2008)

Synonyms

Augmented reality; Cultural heritage;

Mixed Reality Serious Games Gamification; Holographic augmented reality;
Mixed reality; Storytelling; Virtual reality
▶ Gamification and Serious Games

Definition

Mixed Reality Stories Mixed reality as display technology, gamification

as motivational element, and storytelling as inter-
▶ Tabletop Storytelling action metaphor while maintaining the feeling of
Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums 1151

presence are identified as key elements for defin- collections-centered models, museums have
ing the next generation of virtual museums. changed.
Museums can act as primus inter pares (first
among equals) and create a direct communication
Introduction with the visitors, in which museum is the commu-
nicator and the visitor is the receiver and vice
Storytelling, presence, and gamification are three versa, providing the visitor the opportunity to
basic fields that need to be taken into account actively participate in the story. Museums can
when developing novel mixed reality applications have a conversation with the visitor, which can
for cultural heritage, based on the recent renais- share her/his experiences and personal views. To
sance of commercial VR and AR hardware. This this end, recent AR/VR commercial h/w techno-
survey aims to cover a gap in the bibliography and logical advances enable the use of sophisticated
the last relevant surveys of Papagiannakis et al. tools to deliver virtual museum stories and infor-
(2008), Jung et al. (2011), Foni et al. (2010), and mation in a number of ways for experience
Anderson et al. (2009) which are more than enhancement, knowledge construction, and
7 years old, whereas some of these topics are meaning making (Sylaiou et al. 2009).
mentioned in several different chapters within
Ioannides et al. (2017) but not in single, compar- Storytelling Using AR and MR (Merging AMI
ative study. Section “The Role of Storytelling in Installations with Mobile Devices and Physical
Cultural Heritage” of this survey covers the state Artifacts Through Stories)
of the art in storytelling for cultural heritage and
virtual museums, section “The Role of Presence in Storytelling in Mixed Reality
Cultural Heritage” the role of presence, and sec- Static visualizations have been traditionally
tion “The Role of Gamification in Cultural Heri- employed to support storytelling in the form of
tage” the usage of gamification principles. In the text, diagrams, and images. The adoption of
M
last section “Survey of Recent MR Methods for dynamic approaches utilizing state-of-the-art 2D
Virtual Museums,” a comparison among latest and 3D graphics is emerging in an effort to
methods in the above areas is presented. explore the full potential of interactive narration.
Rather than simply constituting an additional
layer to exhibitions, the enhancement of CH insti-
The Role of Storytelling in Cultural tutions through interactive MR exhibits added
Heritage value to the overall user experience (Marshall
et al. 2016), especially if combined with person-
Storytelling in Museums alization to each user’s interests (Partarakis et al.
Museums have realized the value of storytelling 2016). Interactive storytelling in MR environ-
the second half of the twentieth century. Nowa- ments bridges digital and physical information,
days museums find themselves competing with augmenting the real world and offering interaction
the large offer of cultural products coming not which corresponds to the user’s actions in the
only from the cultural sector but also from the physical space. Storytelling can be employed for
entertainment industry. Therefore, museums associating tangible and intangible information;
need to differentiate and make their products such an example is Huang and Huang (2013),
more appealing and attractive to a variety of audi- where the authors compound information for the
ences. Influenced by the “new museology” con- promotion of indigenous cultural heritage.
cept and the transformation of the museological
practice, which refers to a shift in the social roles Storytelling Authoring
of museums encouraging new styles of commu- Authoring refers to the process of creating narra-
nication and expression, in contrast to the “cul- tions that form a digital story. In terms of
tural authority” of museums based on classic, authoring, the most common story types are
1152 Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

character-based stories (Cavazza et al. 2002), lin- environment needs to recreate this dynamic
ear timeline-based stories (Bimber et al. 2003; range of a scene. Where this is not possible
Drossis et al. 2013b), and ontology-based stories using existing technology, techniques such as
(Casillo et al. 2016). In terms of authoring virtual tone mapping need to be used.
worlds, Lu et al. (2008) present an editing envi-
ronment for facilitating the construction of 3D High-Dynamic-Range (HDR) Environments
museums. Additionally, narratives description The dynamic range in a scene is defined as the
and structuring is also performed using authoring ratio between the darkest part of the scene and the
tools. Ardito et al. (2017) create a similar tool for brightest. While the human visual system is able
storytelling creation and customization, focusing to adapt to the full range of light in a scene,
on the aspects of smart objects integrated in CH traditional (also known as low-dynamic-range
installations. Modern cameras are equipped with (LDR)) imaging is not able to capture or display
depth sensing capabilities allowing the real-time a dynamic range of more than 256 to 1 (8 stops).
scanning of surroundings with sufficient detail. High-dynamic-range (HDR) imaging, on the
Such cameras are increasingly being adopted by other hand, by using 32 bit IEEE floating point
consumer smartphones, such as the Google Tango values to represent each color channel, can cap-
compatible devices (Klingensmith et al. 2015). ture and display all the visual data in a scene.
Therefore, such approaches can facilitate aug- Failure to capture the full range of visual data
mented reality environment rapid prototyping can lead to important information being missed
and authoring in a straightforward manner with- due to under- or overexposed regions in the
out requiring user expertise on computer science. image. Even if a scene has been captured, or
created in HDR, detail can still be lost when the
image is subsequently displayed on a device
The Role of Presence in Cultural Heritage which is not HDR. In such a case, the HDR
images need to be tone mapped in order to attempt
“Presence” refers to the phenomenon of people to preserve the perception of the real scene on the
behaving and feeling as if they “are there” in the LDR device. Many tone mapping operators
virtual world created by computer displays (TMOs) have been presented over the years.
(Ioannides et al. 2017). It is an incredibly power- More recently new TMOs have needed to be
ful sensation, which is unique to MR, as it is not developed specifically for displaying 360o HDR
possible to recreate it in any other medium. images on head-mounted displays (HMDs) which
are not yet HDR display devices. This is because
The Role of Illumination in MR Presence for the lighting in a full environment can vary signif-
Cultural Heritage icantly depending where the person is looking.
HDR imaging is especially important in MR envi-
Introduction ronments to ensure the virtual objects are relit with
There are two key components that are necessary the same levels of lighting as in the real scene. As
in order to achieve the right illumination and such, HDR is increasingly being been used within
thereby a high sense of presence in a MR cultural CH applications to improve their authenticity.
heritage environment: authentic appearance of
the light and the correct dynamic range. Prior to The Impact of Virtual Narrators to Presence in
the introduction of electricity, past societies relied Virtual Museums
entirely on daylight and lighting from flames for One of the important aspects which enhance the
illumination. Any MR cultural heritage environ- feeling of presence in CH (Papaefthymiou and
ment which is lit by modern lighting would thus Papagiannakis 2017) is the interaction with and
not be authentic. In addition, the real world con- behavior of the virtual narrators. These narrators
tains a wide range of lighting conditions, from should have humanlike behaviors, so that the vis-
dark shadows to bright sunshine. The MR itors will feel like the virtual narrator that is
Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums 1153

speaking to them is real and that he/she is in the Mobile AR Interactive Applications for Virtual
same room with them. An issue that rises at this Museums
point is how a virtual narrator can be as realistic as Mobile augmented reality systems (MARs) are
possible, with humanlike behavior and character- increasingly currently being tested in rich content
istics. For a virtual character to look as real as heritage environments by both creative profes-
possible, it is not enough just to create him/her sionals and laymen. Recent mobile hardware
programmatically with code. It would be more such as GPU-enhanced, multicore smartphones
realistic if that character could be reconstructed and novel untethered AR headsets (e.g., HoloLens
out of a real human. That way, the appearance of by Microsoft) pave the way for a new breed of AR
the character would resemble the appearance of services and applications; however there are a
the real person based on whom this character was number of issues to consider which regard the
reconstructed. This can be realized by scanning a aspects of information presentation and physical
real person with the Occipital™ depth sensor, interaction (Papaefthymiou and Papagiannakis
specially designed for this task. The photogram- 2017; Papagiannakis 2017; Ioannides et al.
metric method is one of the best methods yet to 2017; Kateros et al. 2015; Li and Duh 2013).
reconstruct the texture of a real person. By scan- With respect to direct hand interaction, there are
ning the person with the special sensor mentioned limitations in handheld devices where the free
before and applying the texture, a 3D model of hand is used for interaction with the 3D objects
that person can be created. That way, a 3D model (e.g., small screen size, unsuitability of point and
of a real person will be able to be used in computer click gestures for manipulation, finger occlusion),
graphics and mixed reality applications, in order whereas in HMDs hands are an intuitive input
to tell a story and communicate with the people channel although bimanual interaction cannot
that will use the specific mixed reality application, ensure better performance than single-hand inter-
as shown in Figs. 1 and 2 (Papaefthymiou and action, and it is important to optimally assign
Papagiannakis 2017). functions to two hands. For example, in recently
M

Mixed Reality, Gamified Presence, and Storytelling gestures through Microsoft HoloLens, in the ITN-DCH
for Virtual Museums, Fig. 1 The hologram of the priest project (Papaefthymiou and Papagiannakis 2017)
of the Asinou Church and the viewer interacting via
1154 Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

Mixed Reality, Gamified Presence, and Storytelling narrator) using the Structure Sensor (Papaefthymiou and
for Virtual Museums, Fig. 2 Digitization of the priest of Papagiannakis 2017)
Asinou Church (to reconstruct his 3D model as a virtual

introduced ARKit by Apple, which represents a providing an enjoyable experience, mixed reality
commercial tipping point in terms of mass-market installations can facilitate cultural awareness, his-
adoption of AR technology, interactions are kept torical reconstruction, and heritage awareness.
simple with object’s movements restricted on a State-of-the-art approaches are not limited to
plane, and the possibility of recognition of installations in indoor spaces (Grammenos et al.
conflicting gestures. On the HMD side Microsoft 2012), but can also involve vehicles that act as
HoloLens, although it has promised a hands-free portable kiosks (Zidianakis et al. 2016).
experience, interaction can be frustrating given
the limited field of view and the misreading of Setups and Interaction
gestures. Although the promise of MARs is turn- Interactive installations in public spaces such as in
ing into reality, the technical challenges with CH institutions have certain requirements in terms
respect to computational efficiency, information of interaction and setup. The installations need to
retrieval from different data sources, markerless adapt to fit to the space available, provide content
detection, and hand gesture recognition perfor- which interests both domain experts and non-
mance still affect the overall user experience expert users, and also present thorough informa-
when interacting with MARs. tion on demand (Mortara et al. 2014). At the same
time, the system design should provide informa-
Immersive Experiences for Interaction with tion immediately and support straightforward
Cultural Heritage interaction techniques. Multiuser interaction with
public displays is an open issue and constitutes an
Mixed Reality Installations active area of research. Once people approach the
Mixed reality has the potential not only to increase interactive display, they decide their actions with
motivation to learn but also to raise interest on regard to the system. Especially in the context of
CH. Thus, CH institutions can increase their MR applications, the establishment of interaction
appeal and enhance visitor engagement through with a public display involves transitioning from
interactive installations that include public dis- implicit to explicit interaction (Vogel and
plays (Partarakis et al. 2017). In addition to Balakrishnan 2004) as the users become engaged
improving the aesthetic experience, mixed reality to the pervasive display.
environments have influence on visitor experi- Tangible interaction is a form of interaction
ence, thus favoring the probability of revisiting a with mixed reality installations in which physical
specific attraction (Jung et al. 2016). Apart from items act as mediators between the users and the
Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums 1155

environment. One aspect of tangible interaction The Role of Gamification in Cultural

refers to physical objects, which are simply the Heritage
medium of communicating information. For
instance, physical paper is employed as a portable Gamification has been applied to a number of
display, augmenting maps or glass-protected application domains including CH, and it is a
models (Grammenos et al. 2011). Interactive popular approach to increase the entertainment
maps (Margetis et al. 2017) are a mixed reality and thus the motivation factor of users
application, where printed maps are augmented (Vassileva 2012). It is different from computer
with additional multimedia information. Further- games even if it shares a lot of common theories
more, tangible interaction can involve objects and practices of development. It is considered to
which have meaningful substance with a semantic be one of the more recent developments of com-
meaning, such as smart objects (i.e., physical puter game design principles that have been
items equipped with sensors such as RFID tags). researched since their appearance (Liarokapis
Smart objects can facilitate related information in et al. 2017a). A major objective of gamification
its context of use (Petrelli et al. 2016). is the improvement of users’ engagement that can
provide positive effects, but the effects are heavily
Bridging Worlds: Combining AR, MR, and VR dependent on personalization (Hamari et al.
Digital cultural heritage content is combined with 2014). Recently, an interactive tool called Photo-
physical 3D replicas (Bugalia et al. 2016) in order Trip was presented that can autonomously recom-
to provide a virtual tour in architectural sites using mend CH locations along travel itineraries (Bujari
MR. This approach has the advantage of moving et al. 2017). The application can identify these
in a wall-projected 3D virtual environment while points of interest by gathering pictures and related
placing a physical model in front of the users to information from social media and then provide
act as a minimap assisting navigation. Head- suggestions and recommendations.
mounted displays are employed in the context of Gamification elements also exist in a number
M
CH foundations for holographic AR and addition- of simulations ranging from serious games to
ally support VR visualizations (Pedersen et al. virtual and augmented reality. Similarly, edutain-
2017). VR can act as a medium to visualize CH ment and gamification are argued to be common
exhibits otherwise unavailable to users as well as recurring themes in the fields of education and
reenactments of historical events offering hands- cultural communication (Mancuso et al. 2017).
on experience regarding unattainable content. In terms of virtual reality, gamification elements
Such indicative examples of VR usage are ele- were recently shown in a kinesthetic application
ments which are under conservation or even sto- of sculpturing Cycladic figurines (Koutsabasis
len (The Museum of Stolen Art 2017) and such as and Vosinakis 2017). The user takes the role of
the battle of Thermopylae (Christopoulos et al. an ancient craftsman who creates a figurine with
2011), respectively. A different approach for bare hand in a virtual reality environment. In
mixing realities involves using tangible objects another study, an immersive virtual underwater
as souvenirs which summarize narratives as they environment was developed to explore Mazotos
were presented to users (Petrelli et al. 2017), act- shipwreck site and get relevant information about
ing as personalized keepsakes of their visit in CH it and its contents (Liarokapis et al. 2017b).
institutions. Finally, an interesting approach at In the following sections, key recent MR pro-
Geevor Tin Mine Museum (Jung et al. 2017) jects with rich gamification elements are
combines AR, VR, and 3D printed objects with presented.
the common goal of adding value to the overall
user experience. The technologies are employed Playful Interaction with Cultural Heritage
for enriching visits (AR), providing access to Exhibits (Improved UX)
inaccessible areas (VR), and souvenirs which the Immersion is based on covering physically a per-
users can keep, respectively (3D printing). son’s stimuli, namely, vision, spatialized sound,
1156 Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

and haptic feedback (Bowman and McMahan accurate reconstructed 3D models, the Re-Play
2007). Haptic feedback (Ott et al. 2007) is exam- platform computes the displacement map and
ined in literature as an additional means of assigns weights to each coordinate in the dis-
enhancing immersion. Kosmalla et al. (2017) placement map based on EMG measurements.
combine tactile feedback from physical worlds To increase the feeling of immersion, photo-
with a virtual rock climbing environment in com- realistic 3D models acquired using a 3D scanner
bination with full body movement and exertion. have been employed to animate and render the
Immersion is also strongly related to the interac- virtual sports heroes.
tion process: in addition to perceiving a mixed
reality (MR) application with human senses, the Gamification Applications for Folklore Dance
interaction modality employed constitutes a Intangible cultural heritage (ICH) includes frag-
decisive factor in feeling of immersion and the ile expressions mainly transmitted orally or by
overall user experience. Contrary to mission- gestures from one generation to the next. In
critical domains such as a working environment, order to support the learning and transmission
CH applications belong to a field in which users of ICH expressions, gamification can also be
are more open to novel alternate interaction used. Based on the theory of “experiential learn-
modalities. Requirements such as precision and ing” (Kolb, D.A., 1984, Experiential learning:
efficiency are not fundamental in such a context, Experience as the source of learning and devel-
as users may be willing to sacrifice precision and opment, Englewood.), one of the main pillars of
overrule tiredness for entertainment and playful game-based learning, the acquisition of learning
user experience (Drossis et al. 2013a). Playful- is achieved by observing, reflecting, mentally
ness encourages exploration, fosters creativity, representing, and enacting movements. In accor-
and stimulates social interaction by entertaining dance with this theory, within the i-Treasures
users and allowing them to escape from the real- (Dimitropoulos et al. 2018) project, a number
ity (Tsekleves and Darby 2017). Playful interac- of game-like applications for sensorimotor
tion is employed for attracting users toward learning of specific dance types and other activ-
public installations and therefore facilitating ities involving full body gestures were designed
user engagement (Williamson and Sundén and developed to support the learning and trans-
2015). mission of a number of ICH expressions. Spe-
cifically, a serious game application for
Re-Play: A Cultural Heritage Project that transmitting ICH knowledge concerning the
Allows to Replay Ancient Games Greek traditional dance “Tsamiko” was devel-
In Europe only, there are over 3000 Traditional oped (Kitsikidis et al. 2015) and shown in the
Sports and Games (TSG). The EU project figure below. The game was structured as a set of
Re-Play has focused on two families of TSG activities, aiming to teach different variations of
(Gaelic and Basque) that are integral to the fabric the dance allowing users to learn by either
of their communities of origin and have success- observing experts’ prerecorded movements in
fully staved off this trend of convergence. The the Observe mode or by starting practicing the
group at MIRALab, University of Geneva, has dance step sequences respectively in the Practice
focused on the development of a realistic real- mode (Fig. 3).
time animation and rendering platform that During the Practice mode the learner is
enabled the visualization of the virtual national expected to imitate the moves of the expert
sports heroes as well as the visualization of ava- avatar displayed on the screen. The addition
tars representing the local heroes (Tisserand et al. of a virtual tutor to correct/manipulate/guide
2017). This state-of-the-art animation-rendering the user by providing visual and audio feed-
system includes dynamic muscle effects, which back was also supported in order to encour-
are modeled over the skin using a novel GPU age the learners’ “participation” and
approach. Based on existing MRI datasets and engagement.
Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums 1157

Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums, Fig. 3 Observe and Practice screen of
Tsamiko’s gamified application within the i-Treasures EU project (Kitsikidis et al. 2015)

Survey of Recent MR Methods for Virtual partial one. Moreover, some applications that run
Museums tethered with a computer and do not support VR
or AR provided no immersion at all. The majority
In Table 1, a summary of key papers in the last of these key papers use mobile AR or holographic
7 years, after the last relevant survey paper from AR, and that explains the majority of the partial
Anderson et al. (2009), is presented. Although immersion entries on Table 1. A recommendation
there is no specific MR method that features at this point would be that there should be more
gamified storytelling with heightened interaction MR applications that support desktop or mobile
that still maintains full immersion and the feeling VR with full immersion, as it creates a more
of presence, several conclusions and recommen- realistic experience for the viewers because they
dations for next lines of research can be drawn and do not have access to the real world and the
summarized. feeling of presence is respectably higher than it
M
The MR technologies that are used in the key is with partial immersion. These technologies can
papers (located in Table 1) contribute to the pres- also be categorized in two additional categories,
ervation of cultural heritage, each one with its own tethered and untethered, according to if an instal-
level of storytelling, presence, gamification, inter- lation needs to be connected to a PC or not
action, and tracking methods. Since all of the (standalone device), respectively. An example of
installations below are MR applications, many of an untethered MR technology, which can be found
them take into account the gamification field, in in the table below, is the Papaefthymiou and
order for those applications to be interesting and Papagiannakis (2017), which uses the Apple
fun for the viewers. Some of them include story- iPad Pro device in order to run. It uses Apple’s
telling elements (e.g., Papaefthymiou and ARKit for camera tracking. The viewers’ move-
Papagiannakis 2017; Pedersen et al. 2017), ments are not limited by cables (since no cables
which are used to inform the viewers about the are used), which enhances the feeling of presence
story of a monument, for instance, thus contribut- as their movements would not be limited if they
ing in cultural heritage curation. Furthermore, it were exploring the real monument. Also,
can be noted that most of the MR methods below gamification and storytelling elements are
support partial immersion and few of them sup- supported, and along with the feeling of presence
port full immersion. The term “immersion” is and freedom of movements, all those elements
included in purpose as it can be easily quantified create the perfect experience for the viewers.
based on the display, whereas “presence” is elu- Another version of this work runs on the Micro-
sive and depends on many parameters and thus soft HoloLens holographic AR HMD, which is
difficult to provide that holds true throughout a also an example of untethered AR.
simulation. VR HMDs thus support full immer- On the other hand, there are some great MR
sion, whereas AR and holographic AR support applications (e.g., Pedersen et al. 2017; Drossis
Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums, Table 1 Comparison of recent MR methods for virtual museums
1158

Mixed reality Intangible

method MR installation Gamification Storytelling Interaction Tracking Immersion heritage
Anderson et al. Rome Reborn serious game x x – No
(2009) Ancient Pompeii application x x – No
Parthenon Project x x – No
Virtual Egyptian Temple x x Walking/ Cave Automatic Full No
movement Virtual
Environment
(CAVE)
The Ancient Olympic Games x x Navigation wand – No
of the VR system
Virtual Priory Undercroft x x – No
Commercial historical games x x – No
Bugalia et al. 3D printed models, projector, camera, laser x x Laser pointer IR camera, Partial No
(2016) pointer project camera,
view camera
Dong et al. VR application using Focus 3DX 330 Faro x Depending on VR Depending on Full No
(2017) scanner, Spheron PanoCam, two GoPro Hero platform VR platform
4 cameras for scanning, and utility for
rendering
Drossis et al. 3D model prototype (depth sensor, touch x Touch screen, RGB-D sensor, Partial No
(2015) screen, interactive cube, projection) interactive cube, Kinect or Asus
walking/movement Xtion camera
Gimeno et al. Region of Valencia map, seven pointers, and x Walking/ Map, pointers, Partial No
(2011) Hostess movement Kinect camera
Grammenos Tabletop augmented reality system x Finger-based input Projector and Partial Yes/no
et al. (2011) pieces of white
paper
Javornik et al. Apple iPad Pro x Touch screen Inner iPad camera Partial Yes
(2017)
Kitsikidis et al. Microsoft Kinect Sensor x Motion capture Skeleton tracking Partial Yes
(2015) with Kinect
Sensor
Kosmalla et al. HTC Vive with Leap Motion x Controllers Positional and Full No
(2017) hand tracking
Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums
Koutsabasis Leap Motion x Hand gestures Hand and finger Partial No
and Vosinakis tracking
2017
Liarokapis et al. HTC Vive x Controllers Motion and laser Full No
(2017b) based
Margetis et al. Interactive map tabletop system x Handwriting, Projector, high- Partial Yes/no
(2017) gestures, touch/ resolution
click camera, depth
sensor
Nakevska et al. CAVE x x Back projection on Pressure sensors Full Yes/no
(2017) walls
Papaefthymiou Oculus Rift x x Controllers Rotational and Full No
and positional
Papagiannakis (sensors)
2017
Papaefthymiou Apple iPad Pro x x Touch screen Apple’s ARKit Partial No
and for camera
Papagiannakis tracking
2017
Papaefthymiou Microsoft HoloLens x x Voice commands, Body motion Partial No
and gestures
Papagiannakis
2017
Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

Papagiannakis DELL P4 M50 Mobile Workstation x Real-time Full No

et al. (2004) markerless
camera tracking
Pedersen et al. Meta developer Kit x x Gestures Markerless Partial No
(2017) surface tracking,
head movement
SpatialStories Toolkit for VR/AR platform x x Depending on Depending on Full/partial No
(2018) VR/AR platform VR/AR platform depending on
the platform
Tisserand et al. Computer with Kinect Sensor for traditional x Movement Kinect Sensor Partial Yes
(2017) sports preservation
1159

M
1160 Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

et al. 2015; Tisserand et al. 2017), which support References

gamification and/or storytelling elements, but
they limit the movements of the viewers due to Anderson, E.F., McLoughlin, L., Liarokapis, F., Peters, C.,
Petridis, P., de Freitas, S.: Serious games in cultural
the existence of cables or need of a connection
heritage. In: Ashley, M., Liarokapis, F. (eds.) The 10th
with a desktop PC. Although these applications International Symposium on Virtual Reality, Archaeol-
are impressive and greatly contribute to cultural ogy and Cultural Heritage VAST – State of the Art
heritage preservation, they restrict the freedom of Reports, St Julians, Malta, 22–25 Sept 2009, pp. 29–48
Ardito, C., Buono, P., Desolda, G., Matera, M.:
movement to the users, which in some cases may
Empowering CH experts to produce IoT-enhanced
disrupt the feeling of presence. Based on this visits. In: UMAP ‘17 25th Conference on User Model-
discussion, it is recommended for MR applica- ing, Adaptation and Personalization, Bratislava, 9–12
tions to be able to run in MR installations that do July 2017, pp. 327–328. ACM, New York (2017)
Bimber, O., Encarnação, L.M., Schmalstieg, D.: The vir-
not restrain the movements of the viewers in any
tual showcase as a new platform for augmented reality
way. Without restriction of movements and with digital storytelling. In: IPT/EGVE03 Imersive Pro-
full immersion, the experience of a viewer will gection Technologies/Eurographics Virtual Environ-
reach very high levels. Finally, SpatialStories ments, Zurich, 22–23 May 2003, pp. 87–95. ACM,
New York (2003)
(Foni et al. 2010), a very recent commercial effort,
Bowman, D.A., McMahan, R.P.: Virtual reality: how much
is a toolset for real-time interactive VR/AR expe- immersion is enough? IEEE Comput. 40(7), 36–43
riences featuring storytelling for non- (2007)
programmers. Although it has not yet been fully Bugalia, N., Kumar, S., Kalra, P., Choudhary, S.: Mixed
reality based interaction system for digital heritage. In:
tested, from information gathered from their VRCAI ‘16 The 15th International Conference on Vir-
website, its manual, and its videos, it poses a tual-Reality Continuum and its Applications in Indus-
promising commercial solution in contributing to try, Zhuhai, 3–4 December 2016, pp. 31–37. ACM,
MR cultural heritage applications. New York (2016)
Bujari, A., Ciman, M., Gaggi, O., Palazzi, C.E.: Using
gamification to discover cultural heritage locations
from geo-tagged photos. Pers. Ubiquit. Comput.
21(2), 235–252 (2017)
Conclusions Casillo, M., Colace, F., De Santo, M., Lemma, S.,
Lombardi, M., Pietrosanto, A.: An ontological
In conclusion, the ability to innovate in the field of approach to digital storytelling. In: MISNC, SI, DS
2016 The 3rd Multidisciplinary International Social
MR gamified storytelling while maintaining pres- Networks Conference, SocialInformatics 2016, Data
ence in the virtual museums of the future will be Science 2016, Union, 15–17 August 2016, p. 27.
directly related to the ability to find synergies ACM, New York (2016)
between the above fields of knowledge, without Cavazza, M., Charles, F., Mead, S.J.: Character-based
interactive storytelling. IEEE Intell. Syst. 17(4),
forgetting that new technologies can play a tran- 17–24 (2002)
scendental role in this equation. And as the quin- Christopoulos, D., Mavridis, P., Andreadis, A.,
tessential reductive machine, MR technologies Karigiannis, J. N.: Using virtual environments to tell
are employed as the device that will restore its the story: the battle of Thermopylae. In: 2011 Third
International Conference on Games and Virtual Worlds
unity through multiplicity and fragmentation. for Serious Applications (VS-GAMES), pp. 84–91.
Hence this work needs to be considered as a National Technical University Great Amphitheater,
further step toward reconciliation and a renewed Athens (2011)
mutual beneficial relationship between humani- Dimitropoulos, K., Manitsaris, S., Tsalakanidou, F.,
Denby, B., Buchman, L., Dupont, S., Nikolopoulos,
ties, social sciences, and computer science. S., Kompatsiaris, Y., Charisis, V., Hadjileontiadis, L.,
Pozzi, F., Cotescu, M., Ciftci, S., Katos, A., Manitsaris,
A. and Grammalidis, N.: A Multimodal Approach for
Cross-References the Safeguarding and Transmission of Intangible Cul-
tural Heritage: The Case of i-Treasures. IEEE Intell
Syst, pp. 1–1 (2018)
▶ Gamification and Serious Games Dong, Y., Webb, M., Harvey, C., Debattista, K., Chalmers,
▶ Gamification in Crowdsourcing Applications A.: Multisensory virtual experience of tanning in
Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums 1161

medieval Coventry. In: EUROGRAPHICS Workshop Jung, Y., Kuijper, A., Fellner, D.W., Kipp, M., Miksatko,
on Graphics and Cultural Heritage, Graz, 27–29 Sept J., Gratch, J., Thalmann, D.: Believable virtual charac-
2017 ters in human–computer dialogs. Eurographics
Drossis, G., Grammenos, D., Bouhli, M., Adami, I., (STARs). 2011, 75–100 (2011)
Stephanidis, C.: Comparative evaluation among Jung, T., tom Dieck, M.C., Lee, H., Chung, N.: Effects of
diverse interaction techniques in three dimensional virtual reality and augmented reality on visitor experi-
environments. In: International Conference on Distrib- ences in museum. In: Information and Communication
uted, Ambient, and Pervasive Interactions, pp. 3–12. Technologies in Tourism 2016, pp. 621–635. Springer,
Springer, Berlin/Heidelberg (2013a) Cham (2016)
Drossis, G., Grammenos, D., Adami, I., Stephanidis, C.: Jung, T.H., Jung, T.H., tom Dieck, M.C., tom Dieck, M.C.:
3D visualization and multimodal interaction with tem- Augmented reality, virtual reality and 3D printing for
poral information using timelines. In: IFIP Conference the co-creation of value for the visitor experience at
on Human–Computer Interaction, pp. 214–231. cultural heritage places. J. Place Manag. Dev. 10(2),
Springer, Berlin/Heidelberg (2013b) 140–151 (2017)
Drossis, G., Ntelidakis, A., Grammenos, D., Zabulis, X., Kateros, S., Georgiou, S., Papaefthymiou, M.,
Stephanidis, C.: Immersing users in landscapes using Papagiannakis, G., & Tsioumas, M.: A Comparison of
large scale displays in public spaces. In: International Gamified, Immersive VR Curation Methods for
Conference on Distributed, Ambient, and Pervasive Enhanced Presence and Human-Computer Interaction
Interactions, pp. 152–162. Springer, Cham (2015) in Digital Humanities. International Journal of Heritage
Foni, A., Papagiannakis, G., Magnenat-Thalmann, N.: in the Digital Era, 4(2), 221–233. (2015). https://doi.
A taxonomy of visualization technologies for cultural org/10.1260/2047-4970.4.2.221
heritage applications. ACM J. Comput. Cult. Herit. Kitsikidis, A., Kitsikidis, A., Dimitropoulos, K., Uğurca,
3(1), 1–21 (2010) D., Bayçay, C., Yilmaz, E., Tsalakanidou, F., . . .,
Gimeno, J., Olanda, R., Martinez, B., Sanchez, F.M.: Grammalidis, N.: A game-like application for dance
Multiuser augmented reality system for indoor exhi- learning using a natural human computer interface. In:
bitions. In: IFIP Conference on Human–Computer International Conference on Universal Access in
Interaction, pp. 576–579. Springer, Berlin/Heidelberg Human–Computer Interaction, pp. 472–482. Springer,
(2011) Cham (2015)
Grammenos, D., Michel, D., Zabulis, X., Argyros, A.A.: Klingensmith, M., Dryanovski, I., Srinivasa, S., Xiao, J.:
PaperView: augmenting physical surfaces with Chisel: real time large scale 3D reconstruction onboard
location-aware digital information. In: TEI ‘11 Fifth a mobile device using spatially hashed signed distance
M
International Conference on Tangible, Embedded, and fields. In: Robotics: Science and Systems, vol. 4
Embodied Interaction, Funchal, 22–26 January 2011, (2015). https://doi.org/10.15607/rss.2015.xi.040
pp. 57–60. ACM, New York (2011) Kosmalla, F., Zenner, A., Speicher, M., Daiber, F., Herbig,
Grammenos, D., Zabulis, X., Michel, D., Padeleris, P., N., Krüger, A.: Exploring rock climbing in mixed real-
Sarmis, T., Georgalis, G., . . ., Adam-Veleni, P.: Mace- ity environments. In: CHI ‘17 CHI Conference on
donia from fragments to pixels: a permanent exhibition Human Factors in Computing Systems, Denver, 6–11
of interactive systems at the archaeological museum of May 2017, pp. 1787–1793. ACM, New York (2017)
Thessaloniki. In: Euro-Mediterranean Conference, Koutsabasis, P., Vosinakis, S.: Kinesthetic interactions in
pp. 602–609. Springer, Berlin/Heidelberg (2012) museums: conveying cultural heritage by making use
Hamari, J., Koivisto, J., Sarsa, H.: Does gamification of ancient tools and (re-) constructing artworks. Virtual
work? – a literature review of empirical studies on Reality. pp 1–16. (2017). https://doi.org/10.1007/
gamification. In: Proceedings of the 47th Hawaii Inter- s10055-017-0325-0
national Conference on System Sciences (HICSS), Li, N., Duh, H.: Cognitive issues in mobile augmented
IEEE Computer Society, Waikoloa, pp. 3025–3034 reality: an embodied perspective. In: Huang, W., et al.
(2014) (eds.) Human Factors in Augmented Reality Environ-
Huang, C.H., Huang, Y.T.: An annales school-based seri- ments. Springer, New York (2013). https://doi.org/10.
ous game creation framework for Taiwanese indige- 1007/978-1-4614-4205-9_5
nous cultural heritage. J. Comput. Cult. Herit. 6(2), Liarokapis, F., Petridis, P., Andrews, D., de Freitas, S.:
9 (2013) Multimodal Serious Games Technologies for Cultural
Ioannides, M., Magnenat-Thalmann, N., Papagiannakis, Heritage, Mixed Reality and Gamification for Cultural
G. (eds.): Mixed Reality and Gamification for Cultural Heritage, Part V, pp. 371–392. Springer International
Heritage. Springer, Heidelberg (2017). https://doi.org/ Publishing (2017a). https://doi.org/10.1007/978-3-
10.1007/978-3-319-49607-8 319-49607-8_15
Javornik, A., Rogers, Y., Gander, D., Moutinho, A.: Liarokapis, F., Kouřil, P., Agrafiotis, P., Demesticha, S.,
MagicFace: stepping into character through an aug- Chmelík, J., Skarlatos, D.: 3D modelling and mapping
mented reality mirror. In: CHI ‘17 CHI Conference on for virtual exploration of underwater archaeology
Human Factors in Computing Systems, Denver, 6–11 assets. In: Proceedings of the 7th International Work-
May 2017, pp. 4838–4849. ACM, New York (2017) shop on 3D Virtual Reconstruction and Visualization of
1162 Mixed Reality, Gamified Presence, and Storytelling for Virtual Museums

Complex Architectures and Scenarios, ISPRS, Napflio, Partarakis, N., Grammenos, D., Margetis, G., Zidianakis,
pp. 425–431 (2017b) E., Drossis, G., Leonidis, A., Stephanidis, C.: Digital
Lu, W., Zeng, D., Pan, J.: An XML-based scene description cultural heritage experience in ambient intelligence. In:
language for 3D virtual museum. In: 30th International Mixed Reality and Gamification for Cultural Heritage,
Conference on Information Technology Interfaces, pp. 473–505. Springer, Cham (2017)
2008, ITI 2008, Croatia Hotel, Cavtat/Dubrovnik, Pedersen, I., Gale, N., Mirza-Babaei, P., Reid, S.: More
pp. 445–450. IEEE (2008) than meets the eye: the benefits of augmented reality
Mancuso, S., Muzzupappa, M., Bruno, F.: Edutainment and holographic displays for digital cultural heritage.
and gamification: a novel communication strategy for J Comput. Cult. Herit. 10(2), 11 (2017)
cultural heritage. J Pub Archaeology. 1, 79–89 (2017) Petrelli, D., Dulake, N., Marshall, M.T., Pisetti, A., Not, E.:
Margetis, G., Ntoa, S., Antona, M., Stephanidis, C.: Voices from the war: design as a means of understand-
Interacting with augmented paper maps: a user experi- ing the experience of visiting heritage. In: CHI ‘16 CHI
ence study. In: Proceedings of the 12th Biannual Con- Conference on Human Factors in Computing Systems,
ference of the Italian SIGCHI Chapter (CHITALY San Jose, 7–12 May 2016, pp. 1033–1044. ACM,
2017), Cagliari, 18–20 Sept 2017 New York (2016)
Marshall, M.T., Dulake, N., Ciolfi, L., Duranti, D., Petrelli, D., Marshall, M.T., O’Brien, S., McEntaggart, P.,
Kockelkorn, H., Petrelli, D.: Using tangible smart rep- Gwilt, I.: Tangible data souvenirs as a bridge between a
licas as controls for an interactive museum exhibition. physical museum visit and online digital experience.
In: TEI ‘16 Tenth International Conference on Tangi- Pers. Ubiquit. Comput. 21(2), 281–295 (2017)
ble, Embedded, and Embodied Interaction, Eindhoven, SpatialStories: A toolset for real-time interactive VR/AR
14–17 February 2016, pp. 159–167. ACM, New York experiences. http://apelab.ch/spatialstories (2018).
(2016) Accessed 6 Jan 2018
Mortara, M., Catalano, C.E., Bellotti, F., Fiucci, G., Houry- Sylaiou, S., Liarokapis, F., Kotsakis, K., Patias, P.: Virtual
Panchetti, M., Petridis, P.: Learning cultural heritage by museums, a survey and some issues for consideration.
serious games. J. Cult. Herit. 15(3), 318–325 (2014) J. Cult. Herit. 10(4), 520–528 (2009)
Nakevska, M., van der Sanden, A., Funk, M., Hu, J., The Museum of Stolen Art.: http://mosa.ziv.bz/ (2017).
Rauterberg, M.: Interactive storytelling in a mixed real- Accessed 22 Dec 2017
ity environment: the effects of interactivity on user Tisserand, Y., Magnenat-Thalmann, N., Unzueta, L.,
experiences. Entertain. Comput. 21, 97–104 (2017) Linaza, M.T., Ahmadi, A., O’Connor, N.E., Zioulis,
Ott, R., Thalmann, D., Vexo, F.: Haptic feedback in mixed- N., Zarpalas, D., Daras, P.: Preservation and
reality environment. Vis. Comput. 23(9), 843–849 gamification of traditional sports. In: Ioannides, M.,
(2007) Magnenat-Thalmann, N., Papagiannakis, G. (eds.)
Papaefthymiou, M., Papagiannakis, G.: Gamified aug- Mixed Reality and Gamification for Cultural Heritage,
mented and virtual reality character rendering and ani- pp. 421–446. Springer, Cham (2017)
mation enabling technologies. In: Ioannides, M., Tsekleves, E., Darby, A.: The role of playfulness and
Magnenat-Thalmann, N., Papagiannakis, G. (eds.) sensory experiences in design for public health and
Mixed Reality and Gamification for Cultural Heritage, for ageing well. In: Sensory Arts and Design. Blooms-
pp. 333–357. Springer (2017). https://doi.org/10.1007/ bury p. 49–66. 18 p (2017)
978-3-319-49607-8 Vassileva, J.: Motivating participation in social computing
Papagiannakis, G.: Gamification and serious games. In: applications: a user modeling perspective. User Model.
Lee, N. (ed.) Encyclopedia of Computer Graphics and User-Adap. Inter. 22(1–2), 177–201 (2012)
Games. Springer, Cham., https://doi.org/10.1007/978- Vogel, D., Balakrishnan, R.: Interactive public ambient
3-319-08234-9_90-1, ISBN 978-3-319-08234-9 displays: transitioning from implicit to explicit, pub-
(2017) lic to personal, interaction with multiple users. In:
Papagiannakis, G., Schertenleib, S., Ponder, M., Arevalo- UIST ‘04 17th Annual ACM Symposium on User
Poizat, M., Magnenat-Thalmann, N., Thalmann, D.: Interface Software and Technology, Santa Fe, 24–
Real-time virtual humans in AR sites. In: IEE Visual 27 October 2004, pp. 137–146. ACM, New York
Media Production (CVMP04), London, pp 273–276 (2004)
(2004) Williamson, J.R., Sundén, D.: Enter the circle: blending
Papagiannakis, G., Singh, G., Magnenat-Thalmann, N.: spherical displays and playful embedded interaction in
A survey of mobile and wireless technologies for aug- public spaces. In: Proceedings of the 4th International
mented reality systems. J Comput. Anim. Virtual Symposium on Pervasive Displays (PerDis ‘15),
Worlds. 19(1), 3–22 (2008) Saarbruecken, 10–12 June 2015, pp. 195–200. ACM,
Partarakis, N., Antona, M., Zidianakis, E., & Stephanidis, New York (2015)
C.: Adaptation and content personalization in the con- Zidianakis, E., Margetis, G., Paparoulis, S., Toutountzis,
text of multi user museum exhibits. In B.N. De Carolis, T., Stratigi, K., Paparoulis, G., Stephanidis, C.: Turning
C. Gena, T. Kuflik, & F. Nunnari (Eds.), Proceedings of an electric cargo vehicle into a portable interactive
the 1st Workshop on Advanced Visual Interfaces for information kiosk. In: International Conference on
Cultural Heritage (AVI*CH 2016), Bari, Italy, 7-10 Human–Computer Interaction, pp. 463–469. Springer,
June. Published by: CEUR-WS (Vol. 1621) (2016) Cham (2016)
Mobile Cloud Gaming 1163

MMO Mobile Cloud Gaming

▶ World of Tanks, MMO Strategy Freemium Di Wu1, Yihao Ke1, Jian He2, Yong Li3 and
Game Min Chen4
1
Department of Computer Science, Sun Yat-sen
University, Guangzhou, China
2
Department of Computer Science, University of
MMORPG Texas at Austin, Austin, TX, USA
3
Department of Electronic Engineering, Tsinghua
▶ Detecting and Preventing Online Game Bots in University, Beijing, China
4
MMORPGs School of Computer Science and Technology,
▶ Disney Toontown Online, a Massively Multi- Huazhong University of Science and Technology,
player Online Role-Playing Game Wuhan, China
▶ Fantasy XVI Online, a Massively Multiplayer
Online Role-Playing Game
▶ World of Warcraft, a MMORPG with Synonyms
Expansions
MCG

Mobile Applications Definitions

▶ Indigenous Language Revitalization with Mobile cloud gaming (MCG) is a new type of
Stories and Games mobile gaming, in which games are stored, syn-
M
chronized, and rendered in the remote cloud plat-
form and delivered to mobile users using video
streaming technology.
Mobile Applications for
Behavior Change

▶ Mobile Persuasive Applications Introduction

As a killer application in the mobile app market,

the growth of mobile gaming is very impressive in
Mobile Augmented Reality the past decade. The major obstacle that may
hinder the further growth of mobile gaming is
▶ Conceptual Model of Mobile Augmented Real- the constraint of limited resources on mobile
ity for Cultural Heritage devices. However, it is fortunate that recent
advances in cloud computing shed lights on the
feasibility of playing high-end video games on
mobile devices.
Mobile cloud gaming (MCG) enables mobile
Mobile Augmented Reality for users to play games in the cloud and thus mitigates
Cultural Heritage the tension between high requirements of video
games and limited resources on mobile devices.
▶ Conceptual Model of Mobile Augmented Real- By offloading computation-intensive tasks to the
ity for Cultural Heritage powerful cloud platform, the capacity of mobile
1164 Mobile Cloud Gaming

devices can be significantly augmented (Wei Cai nature of wireless channels, it is necessary to
et al. 2013; Zhang et al. 2013; Wen et al. 2014). ensure that game video streaming can adapt to
Unlike traditional mobile gaming, mobile the changes of wireless network conditions.
cloud gaming offers many novel features: Firstly, 4. Limited Battery Life: In addition to providing
with mobile cloud gaming, there is no need for sufficiently short latency for real-time gaming
mobile users to constantly upgrade their mobile and reacting to user inputs timely, it is neces-
devices; secondly, mobile users are allowed to sary to take energy efficiency of mobile
start game playing instantly, without downloading devices into account. It is critical to reduce
and installing mobile game apps; thirdly, mobile energy consumption of computation, commu-
games can be platform-independent under the nication, and display for mobile cloud gaming.
mode of mobile cloud gaming. There is no need
for game developers to build a separate game The above challenges raise a number of ques-
app for each mobile platform (e.g., iOS, Android). tions that should be addressed. Among them, the
first critical question is how to better evaluate the
QoE of mobile gamers. Only when the QoE target
Technical Challenges and Problems is clear can the MCG service provider know how to
provision and configure its cloud resources in a
Compared to common cloud gaming, the features reasonable way. The QoE of game players in
of mobile devices pose some additional chal- MCG systems can be generally modeled by a set
lenges for mobile cloud gaming. The main chal- of objective and subjective factors. Wang and Dey
lenges of delivering MCG services to mobile (2009) characterized user QoE by video settings,
devices include the following aspects: network factors, image quality, etc. Game Mean
Opinion Score (GMOS) (Schaefer et al. 2002) has
1. Heterogeneity of Mobile Devices: There exists been widely used to map these factors to the value
significant diversity among hardware configu- of QoE. However, these QoE models often assume
rations of mobile devices (including screen that players are in the same physical context when
size, resolution, bandwidth, and CPU/GPU). playing games. When using mobile devices to play
The MCG service provider should consider games, the physical context can be highly dynamic
the heterogeneity of mobile devices when since the mobility of users will incur certain
delivering video game streams to mobile changes of their surrounding environment (e.g.,
devices. where the player is, what the player is doing, and
2. Diversity of Mobile Games: Different game so on) (Benford et al. 2005). Mobile devices pro-
genres have different QoE (Quality of Experi- vide an opportunity to sense the physical context
ence) requirements. For example, a slight by analyzing information from multiple sensors
increase of interaction delay is intolerable for such as motion sensors, light sensors, and acoustic
the first-person shooter (FPS) games, while it sensors. It is important to integrate the availability
may not be observable for the war strategy of context sensing when designing a more realistic
games. Therefore, the decision on MCG QoE model of mobile cloud gaming.
resource provisioning should consider the dif- The provisioning of MCG resources also
ference on QoE requirements among different should be able to maximize the QoE of gamers
game genres. (including latency, video quality, etc.). In the
3. Unreliable Wireless Transmission: The mobile meanwhile, due to the cost of delivering high
devices are connected with the MCG platform QoE, the MCG service provider also needs to
via wireless channels. However, the conditions take service cost into account during optimization.
of wireless channels are intrinsically unreliable To achieve the tradeoff between the monetary
and vary with time. It makes the transmission cost incurred by provisioning cloud resources
of video game streams easy to violate the strict (e.g., CPU, GPU, bandwidth) and service quality
latency requirements. Due to the stochastic (e.g., delay, frame rate, resolution) experienced by
Mobile Cloud Gaming 1165

mobile game players (Gao et al. 2016; Hu et al. are computed on cloud gaming servers, they have
2016), the MCG service provider should dynam- to be compressed before being streamed. This can
ically adjust the amount of provisioned cloud be done in one of the three data compression
resources according to the variations of user schemes: (i) video compression, which encodes
demand. Chuah et al. (2014) provided a compre- 2D rendered videos, (ii) graphics compress, which
hensive survey on how to efficiently leverage encodes 3D structures and 2D textures, and (iii)
hardware resources to satisfy the demand of hybrid compression, which combines both video
graphics rendering and video coding in cloud and graphics compress.
gaming. It is of more interests to analyze the
complexities of scheduling cloud resources from
a broader perspective. As the difference among Architecture and Design of MCG Systems
game genres affects the demand pattern of cloud
resources, the resource efficiency can be It is challenging to architect a cost-effective MCG
improved by optimizing resource allocation platform that can provide users with high QoE. As
among different game genres jointly. However, a delay-sensitive service, game players are sensi-
as indicated by Chen et al. (2014), it requires tive to interaction delay during game playing. The
further effects to solve the problem of consolidat- MCG platform should be able to adjust resource
ing multiple MCG tasks (e.g., rendering, video allocation for mobile users dynamically in order
coding and transmission) on cloud servers. to meet the delay constraint. According to the
Streaming gaming video in such a stringent distance to mobile devices, the authors briefly
time-constrained scenario is also a major chal- classify existing MCG architectures into two cat-
lenge in MCG. One solution to cope with this egories, namely, Remote Cloud and Edge Cloud
problem is data compression. After game scenes (as shown in Fig. 1).
M
Mobile Devices MCG Platform

Edge Cloud
Fixed Cloudlets

Mobile Cloudlets
WiFi AP

Remote Cloud
3G/4G BS

3G/4G BS

Mobile Cloud Gaming, Fig. 1 Architectures of mobile cloud gaming systems

1166 Mobile Cloud Gaming

Remote Cloud refers to the architecture in the remote cloud, researchers have performed
which a MCG service provider relies on a remote quite a few research studies on video codec opti-
cloud platform to deliver the mobile cloud gaming mization, network transmission, and QoE adapta-
service to mobile users. The architecture of tion jointly. As stated in Jarschel et al. (2011), the
Remote Cloud has been widely adopted by lead- QoE of cloud gaming is highly related to down-
ing MCG service providers, such as Ubitus and stream packet loss and downstream delay, which
G-Cluster. Physically, the remote cloud can be indicates the necessity of connecting the nearest
provisioned on a set of geographically distributed and fastest server.
data centers in order to serve mobile users in Edge Cloud is another attractive architecture
different regions. proposed to address the latency problem of remote
As the MCG service provider can directly rent cloud, in which cloudlets (Satyanarayanan et al.
cloud resources (e.g., VM, bandwidth, storage) 2009) close to the mobile device are responsible
from the third-party cloud service providers for task offloading in order to meet the strict
(e.g., Amazon, Microsoft), the construction and latency requirement. Generally, a cloudlet is a
maintenance of the remote cloud infrastructure kind of resource-rich computing entity with a
can benefit significantly from mature services of broadband network connection in the proximity
cloud service providers. Remote cloud could be of mobile devices. There are two kinds of cloud-
very reliable and powerful. It is also easy to lets: fixed cloudlets (e.g., desktop PC, gateway,
expand the scale of provisioned resources and set-top-box) and mobile cloudlets (e.g., tab-
according to dynamic user demands. lets, pads). As a mobile device is well connected
The major issue with the mode of the remote with nearby cloudlets, the intensive computation
cloud is network latency, as most large cloud tasks (e.g., frame rendering, video encoding) can
service providers only deploy their data centers be completely migrated to the nearby cloudlets.
at a limited number of locations. For players who In spite that edge cloud is a promising
are far away from any cloud data center, network approach, several intrinsic problems should be
latency will deteriorate their gaming experience addressed before its successful deployment in
significantly. Figure 2 shows the response delay reality. First, the MCG service provided by cloud-
when selecting different data centers when using lets is not reliable and depends on the existence of
the remote cloud. The requirements on high video cloudlets in the proximity. The problem is even
quality and high frame rate will further exacerbate worsened for the case of mobile cloudlets. Sec-
the latency problem. To achieve real-time trans- ond, the decentralized nature of cloudlets makes it
mission of game video streams under the mode of hard to be operated and managed. For an edge

Mobile Cloud Gaming,

800
Fig. 2 Response delay
when selecting different
700
Response Delay (ms)

data centers in remote cloud

600

500

400

300

200

DC1 DC2 DC3

Mobile Cloud Gaming 1167

cloud with many volunteer cloudlets, the cost of resources to deliver a smooth gaming experience
cloudlet management and maintenance will be for mobile game players.
nontrivial for a MCG service provider. Third, However, the architecture of hybrid cloud will
effective incentive mechanisms should be adopted also complicate the system design. In the real
to encourage individuals to deploy more cloud- scenario, task partitioning among mobile devices,
lets. To provide MCG services, a cloudlet should cloudlets, and remote cloud servers will become
contribute its own resources (including CPU very sophisticated. The decision on task
cycles, battery energy, and disk space). Without partitioning needs to take the constraints on
monetary incentives, it is difficult if not impossi- response delay, resource availability, energy con-
ble to stimulate resource sharing between cloud- sumption, and bandwidth conditions into account.
lets and mobile devices. In addition, the issues of In the current stage, the research on hybrid cloud
security and privacy caused by task offloading is still in its infancy.
should also be considered.
In the spectrum of architectural design of MCG
platforms, Remote Cloud can be regarded as one Potential Directions and Opportunities
end and Edge Cloud could be the other end. Either
of them has its own pros and cons as the authors In this section, the authors discuss a few directions
have explained in the above. The authors believe and opportunities for future research on mobile
that, Hybrid Cloud, which is a combination of cloud gaming. Figure 3 provides an illustrative
edge cloud and remote cloud, is a better architec- example on the directions covered here, which
ture to support mobile cloud gaming. It can seam- will be explained one by one in the following
lessly integrate both proximate and remote cloud parts.

Multi-Screen Teleportation
M

Social Switching
Community A

WiFi
Augmented mobile UI /3G
/4G
MCG
Resource
Optimizer
Social
Community B

Context Sensing

Game Players Mobile Devices MCG Platform

Mobile Cloud Gaming, Fig. 3 An illustrative example on potential opportunities on mobile cloud gaming
1168 Mobile Cloud Gaming

Augmented User Interface for Mobile Devices experiences. Social mobile cloud gaming creates
User interface plays an important role for user QoE a virtual living-room experience, by allowing
of mobile cloud gaming. Particularly, many mobile remote mobile gamers to play games together,
devices (e.g., smartphones, pads) cannot support watch game playing of other friends, and com-
the traditional physical keyboard and mouse as the municate via various communication modalities
input, and only touch screen is available as the (such as text, graphics, audio, video, and so on).
gaming control. Therefore, it is critical to augment For example, twitch-like live video game broad-
the current user interface of mobile devices to casting (Hamilton et al. 2014) has already
better support mobile cloud gaming. The tracking attracted significant attention from game players.
of a mobile device can be realized by analyzing A gaming broadcaster often belongs to the same
received signals such as RF signals and acoustic social community with his viewers. As the
signals. By combining the arrival sequences of exchange of information flows between mobile
measured signal angles from two or more receivers players is supported by the MCG platform, the
(e.g., antennas, speakers), the movement of a abundant resources in the cloud can better sup-
mobile device can be precisely sensed (Vasisht port the increasing number of mobile players.
et al. 2014; Agrawal et al. 2011). The augmented The MCG system can be highly scalable with
interaction techniques will significantly enhance the support of cloud infrastructure.
gaming experiences on mobile devices.
Multiscreen Teleportation of MCG Sessions
Environment-Aware Mobile Game Playing Nowadays, mobile gamers normally interact
Different from fixed game consoles (e.g., desktop with multiple screens in their daily living envi-
PCs, gaming boxes), mobile cloud gaming ronment (Hu et al. 2014). The sizes of these
enables gamers to play games anywhere at any- screens range from large (e.g., TV screen) to
time. Such high mobility implies that the sur- small (e.g., smartphone screen). To enjoy a better
rounding environment of a gamer may keep gaming experience, gamers are willing to tele-
changing during game playing. With a rich set of port ongoing gaming sessions from a small
sensors (e.g., microphone, light sensor, camera, screen to a large screen if without any interrup-
GPS, accelerometer) available on today’s mobile tion, and vice versa. Imagine that a user is
devices, it is possible for mobile devices to sense playing a game on his smartphone when taking
the physical context of their surrounding environ- the subway back to home. Upon arriving at
ment (e.g., indoor or outdoor, fixed or moving, home, he can teleport his gaming session to the
light or dark). Moreover, the emergence of wear- TV screen and play the game with a joystick.
able devices (e.g., smartwatches, smartglasses) After leaving his apartment for dinner, he can
shows a bright future to sense fine-grained phys- switch the gaming session back to his
ical context. By adaptively generating game con- smartphone. Such seamless teleportation among
tents according to the changing physical context, multiple screens will surely enhance the gaming
mobile game players can enjoy enriched gaming experience of players. With the MCG platform,
experiences. As an example, the MCG platform gaming session migration can be easily
can adapt the quality, frame-rate, and contents of supported by the back-end cloud infrastructure.
game video streams according to the light condi- As a promising solution to enable high-end
tion and moving speed of a mobile user. Such video games to be played on resource-constrained
environment-aware adaptation can improve the mobile devices, the authors believe that mobile
user QoE significantly. cloud gaming will definitely have a bright future
in the coming era.
Integration of Social Networks with MCG
By integrating social links among gamers with
the MCG system, mobile cloud gaming can pro- Cross-References
vide a user-centric gaming environment, which
dramatically transforms the traditional gaming ▶ Cloud for Gaming
Mobile Persuasive Applications 1169

References Wen, Y., Zhu, X., Rodrigues, J., Chen, C.: Cloud mobile
media: reflections and outlook. IEEE Trans. Multime-
Agrawal, S., Constandache, I., Gaonkar, S., Roy dia 16, 885–902 (2014)
Choudhury, R., Caves, K., DeRuyter, F.: Using mobile Zhang, W., Wen, Y., Guan, K., Kilper, D., Luo, H., Wu, D.:
phones to write in air. In: Proceedings of the 9th Energy-optimal mobile cloud computing under sto-
International Conference on Mobile Systems, Appli- chastic wireless channel. IEEE Trans. Wirel. Commun.
cations, and Services – MobiSys ‘11, pp. 15–28 12, 4569–4581 (2013)
(2011)
Benford, S., Magerkurth, C., Ljungstrand, P.: Bridging the
physical and digital in pervasive gaming. Commun.
ACM 48, 54–57 (2005)
Chen, K., Huang, C., Hsu, C.: Cloud gaming onward:
Mobile Devices
research opportunities and outlook. In: 2014 I.-
E. International Conference on Multimedia and Expo ▶ Exploring Innovative Technology: 2D Image
Workshops (ICMEW), pp. 1–4 (2014) Based Animation with the iPad
Chuah, S., Yuen, C., Cheung, N.: Cloud gaming: a green
solution to massive multiplayer online games. IEEE
Wirel. Commun. 21, 78–87 (2014)
Gao, G., Hu, H., Wen, Y., Westphal, C.: Resource provi-
sioning and profit maximization for transcoding in Mobile Game
clouds: a two-timescale approach. IEEE Trans. Multi-
media, 19, 836–848 (2016)
▶ Gardenscapes and Homescapes, Casual Mobile
Hamilton, W., Garretson, O., Kerne, A.: Streaming on
twitch. In: Proceedings of the 32nd Annual ACM Con- Games
ference on Human Factors in Computing Systems –
CHI ‘14, pp. 1315–1324 (2014)
Hu, H., Wen, Y., Luan, H., Chua, T., Li, X.: Toward
multiscreen social TV with geolocation-aware social
sense. IEEE MultiMedia 21, 10–19 (2014) Mobile Interface
Hu, H., Wen, Y., Chua, T., Huang, J., Zhu, W., Li, X.: Joint M
content replication and request routing for social video ▶ Tracking Techniques in Augmented Reality for
distribution over cloud CDN: a community clustering Handheld Interfaces
method. IEEE Trans. Circuits Syst. Video Technol. 26,
1320–1333 (2016)
Jarschel, M., Schlosser, D., Scheuring, S., Hoßfeld, T.: An
evaluation of QoE in cloud gaming based on subjective Mobile Persuasive
tests. In: 2011 Fifth International Conference on Inno-
vative Mobile and Internet Services in Ubiquitous
Applications
Computing (2011)
Satyanarayanan, M., Bahl, P., Caceres, R., Davies, N.: The Damianos Gavalas1, Vlasios Kasapakis2 and
case for VM-based cloudlets in mobile computing. Elena Dzardanova3
IEEE Pervasive Comput. 8, 14–23 (2009) 1
Schaefer, C., Enderes, T., Ritter, H., Zitterbart, M.: Subjec-
Department of Product and Systems Design
tive quality assessment for multiplayer real-time Engineering, University of the Aegean,
games. In: Proceedings of the 1st Workshop on Net- Ermoupoli, Greece
work and System Support for Games – NETGAMES 2
Department of Cultural Technology and
‘02, pp. 74–78 (2002)
Vasisht, D., Wang, J., Katabi, D.: RF-IDraw. In: Proceed-
Communication, University of the Aegean,
ings of the 6th Annual Workshop on Wireless of the Mytilene, Greece
3
Students, by the Students, for the Students – S3 ‘14, Department of Product and Systems Design
pp. 1–4 (2014) Engineering, University of the Aegean,
Wang, S., Dey, S.: Modeling and characterizing user
experience in a cloud server based mobile gaming
Mytilene, Greece
approach. In: GLOBECOM 2009 – 2009 I.-
E. Global Telecommunications Conference,
pp. 1–7 (2009) Synonyms
Wei Cai, Leung, V., Min Chen: Next generation mobile
cloud gaming. In: 2013 I.E. Seventh International Sym-
posium on Service-Oriented System Engineering, Mobile applications for behavior change;
pp. 551–560 (2013) Mobile-based behavior change support systems
1170 Mobile Persuasive Applications

Definition applications, through presenting the most typ-

ical behavioral change goals they pursue,
Mobile persuasive applications are native or reviewing representative popular applications,
mobile web applications with inbuilt persuasive and discussing the elementary principles con-
features that influence the attitudes or behaviors sidered in the design of mobile persuasive
of users. systems.

Introduction Attitude and Behavioral Traits

Addressed by Mobile Persuasive
B. J. Fogg has introduced the concept of Applications
“persuasive technology” to frame the domain of
research, design, and applications of persuasive Mobile devices represent a useful instrument for
computers Fogg 2002). Persuasive technology persuasive applications as they exhibit several
refers to applications and interactive media desired characteristics and satisfying features
which implement psychological principles of (e.g., the requirement to assess and motivate the
persuasion (like credibility, trust, reciprocity, physical activity of a person may be satisfied by
and authority) aiming at positively altering persuasive applications which take advantage of
users’ disposition and behavior toward a the readings of smartphones’ built-in sensors)
predetermined goal. (Kasapakis and Gavalas 2015). The diffusion
It is now widely recognized that apart from of mobile devices allows to reach a much wider
relying on humans to influence others (e.g., life audience while apps are easy to disseminate
coaches, nutritionists, etc.), the conscious design through the established mobile application
of persuasive computer applications as a means markets; mobile devices come with an array
for influencing human behaviors can bring about of inbuilt sensors and may easily couple with
positive changes in many domains, including wearable devices (such as smart wristbands,
health, personal lifestyle and well-being, safety, smartwatches, etc.) to accurately estimate a
and environment. This underscores the impor- variety of user activities, for instance, the type,
tance of persuasive technology, that is, its capacity pace, duration, and location of physical
to effectively influence the attitudes or behaviors activity (e.g., number of steps taken and/or
of users so as to ultimately serve the personal stairs climbed), which may then be translated to
and/or the social good (e.g., adoption of healthy meaningful statistical indicators (e.g., calorie
diet habits, promotion of sustainable lifestyles consumption).
toward green travels, or household waste Recently, persuasion principles have been
reduction). Recent advances in computing, partic- applied to the design of mobile applications
ularly mobile computing, leverage untapped which aimed at such diverse behavioral change
opportunities to impact users’ behavior for goals (Gardeli et al. 2017; Langrial et al. 2013,
the better. In fact, field studies have provided 2017):
evidence that smartphone applications achieve
higher adherence to behavior change tasks (e.g., • Stimulation of enhanced physical activity:
change exercise and nutrition habits to treat “fitness” apps supporting walking, running,
obesity) compared to their web counterparts hiking, and cycling workout
or other paper-based programs with respect to • Improvement of well-being and adoption
retention, engagement, and effectiveness (Carter of healthy lifestyles: monitoring of
et al. 2013). physical exercise, nutrition intake, smoke
The key objective of the article is to cessation, alcohol intake reduction, weight
explore the area of mobile persuasive management, etc.
Mobile Persuasive Applications 1171

• Addressing health problems (e.g., obesity, Persuasion Design for Mobile

heart disease, type 2 diabetes) and adherence Applications
to prescribed medication intake
• Mood monitoring, stress, and anger Principles of social psychology, context-aware
management decision-making, as well as engagement strate-
• Adoption of low-carbon lifestyle, e.g., gies are commonly utilized as means of
encouragement to embrace environmentally persuasive design throughout the design process
friendly transport practices of products and tools intended to influence posi-
• Preservation of shared physical resources tive behavioral changes. Oinas-Kukkonen and
and environmental awareness Harjumaa (2009) summarized their empirical
• Improvement of driving behavior and safety findings to several postulates that should be
taken into account when designing or evaluating
The number of mobile persuasive applica- persuasive systems. Firstly, persuasive systems
tions, serving the abovementioned behavioral should decompose persuasion to a multilevel pro-
change goals, has proliferated in the recent cess, as users are more likely to be persuaded
years. These applications employ several behav- when performing a series of incremental tasks.
ior change techniques (Conroy et al. 2014) to Designers should also take into account that
enhance awareness, increase motivation, and technology is not neutral as it critically affects
engage users with the task/activity at hand user behavior; hence, this should be communi-
(a theoretical discussion of the key design con- cated to users in order not to lower the level of
siderations for such applications is provided in persuasion. Furthermore, users are more likely to
the next Section). For instance: the Fabulous – be persuaded when making commitments, and
Self-Care app (https://play.google.com/store/ therefore persuasive systems should facilitate
apps/details?id¼co.thefabulous.app) supports users in completing those commitments by
users in goal setting through monitoring tasks organizing user tasks accordingly and offering
M
completion toward adopting healthy habits (see consistency. Moreover, it is important that persua-
Fig. 1a); the Runtastic Running guides (https:// sive systems should support both a direct route
play.google.com/store/apps/details?id¼com. to persuasion that focuses on the quality and
runtastic.and roid) users throughout running/jog- punctuality of the information and also an indirect
ging sessions via a voice coach while also pro- route that uses peripheral cues to associate
viding detailed feedback on their performance positivity with the message without requiring
(see Fig. 1b); the Home Workout app (https:// much effort or information processing. Finally,
play.google.com/store/apps/details?id¼home persuasive systems should be useful to users and
workout.homeworkouts.noequipment) rewards rather usable while also being discreet so as not to
users with badges when successfully completing interrupt users when performing primary tasks.
a weekly program of exercises (see Fig. 1c); So far, the Persuasive Systems Design (PSD)
the QuitNow! Quit Smoking app (https://play. model has been the most generic conceptual
google.com/store/apps/details?id¼com.EAGIN framework in assistance of persuasive systems
software.dejaloYa) allows social interaction designers (Oinas-Kukkonen and Harjumaa
and progress comparison among its users to 2009). The PSD model underlines the significance
further motivate them in quitting smoking (see of persuasion context by addressing the intent
Fig. 1d). It is noted that the illustrated apps are (intended change), the event (use, users, and
not chosen as the best manifestations of the per- the context), and the strategy (persuasive
suasive technology but, instead, as representa- message and delivery route). Moreover, it iden-
tive examples of mobile persuasive applications tifies four basic design principles to be
on the basis of their popularity (>1M incorporated in persuasive applications, some of
downloads). which have been adopted from Fogg (2002):
1172 Mobile Persuasive Applications

Mobile Persuasive Applications, Fig. 1 (a) Fabulous – (c) Home Workout app badge rewards (image courtesy of
Self-Care app, health goal setting (image courtesy of Leap Fitness Group); (d) QuitNow! Quit Smoking, social
TheFabulous); (b) Runtastic Running, self-monitoring of interaction with other users performing alike tasks. (Image
running performance (image courtesy of runtastic GmbH); courtesy of Fewlaps)
Mobile Persuasive Applications 1173

Mobile Persuasive Applications, Table 1 Methods employed by popular persuasive applications for addressing the
persuasive design principles of the PSD model
QuitNow! Quit
Title Fabulous – Self-Care Runtastic Running Home Workout smoking
Scope Supporting users in Motivating users to Persuading users to Supporting users to
adopting healthy exercise (running/ exercise (gymnastics) quit smoking
habits jogging)
Primary Collection of user data Collection of user data Collection of user data Collection of user data
task (e.g., sleep and (e.g., age, weight, (e.g., age, weight, (e.g., number of
support training habits); height); provision of height); provision of cigarettes smoked per
illustration of detailed custom training plans training plans day); illustration of
progress charts based according to user goals according to user goals detailed progress
on user goals (e.g., weight loss (e.g., full body charts
plan); illustration of exercise plan);
detailed progress illustration of detailed
charts progress charts
Dialogue Task reminders via Voice coach guidance Voice coach guidance Badge/achievements
support notifications/alarms; throughout training throughout training rewards; task
voice coach messages sessions; leaderboard sessions; badge/ reminders via
support; task achievements rewards; notifications/alarms;
reminders via task reminders via provision of an
notifications/alarms notifications/alarms automated chat bot
which supports users
with scientific facts
relevant to quitting
smoking
Credibility Provision of verified Usage of GPS for The application can be Provision of an
support scientific data via accurate running/ linked and transfer automated chat bot
personalized messages jogging distance data to Google Fit which supports users M
about the importance measurement; (https://www.google. with scientific facts
of each healthy habit integration with com/fit/), the official relevant to quitting
wearable fitness fitness tracker app of smoking; provision of
trackers to further Google a FAQ section with
increase the accuracy verified methods for
of measurements quitting smoking
Social Users can share and Users can share and – Users can share and
support compare their progress compare their progress compare their progress
with friends through a with friends through a with friends through a
dedicated service and dedicated service and dedicated service and
popular social media popular social media popular social media
platforms platforms platforms

• Primary task support, which involves support words, images, symbols, or sounds based on
of the user to carry out her primary task user behaviors), providing virtual rewards (as a
through reduction (decomposition of complex credit for performing the target behavior), and
behaviors into simple tasks), guidance along reminding users of their tasks.
the attitude change process, and provision of • Credibility support, which relates with
information tailored to users’ potential needs designing systems so that they are more
or interests, allowing users to self-monitor their credible through inculcating trustworthiness
own performance. (providing information that is truthful, fair,
• Dialogue support, namely, enabling users to and unbiased), offering content incorporating
receive feedback while moving toward their expertise (i.e., knowledge, experience, and
target behavior through offering praise (via competence), highlighting the real people or
1174 Mobile Persuasive Applications

Mobile Persuasive Applications, Fig. 2 (a) Zombies, Run! story missions and training plans; (b) Zombies, Run!
progress charts and mission details shared in ZombieLink

organizations behind the system’s content or most successful examples of gamification for
services, and providing means to verify the persuasive systems is Zombies, Run! (https://
accuracy of content. play.google.com/store/apps/details?id¼com.sixto
• Social support which relates with fostering start.zombiesrunclient). In effect, users are
user motivation through social learning immersed in running game sessions, being
(observing other users who perform similar provided voice guidance to achieve exercise
target behaviors), social comparison goals while also trying to survive the “zombie
(comparing performance with the performance apocalypse.” Also, users can share and compare
of other users), and normative influence their progress with other users via the
(means for gathering together people who aim complementary community website ZombieLink
at the same goal). (https://zombiesrungame.com/zombielink/home/,
see Fig. 2).
Table 1 provides an overview on the methods
employed by the example applications illustrated
in Fig. 1 to address the basic design principles of Conclusions and Research Prospects
the PSD model
Further to the persuasive design principles of Sufficient evidence already exists that smartphone
the PSD, gamification (i.e., the use of game design applications may effectively support people in
elements in non-gaming contexts) is increasingly pursuing behavior change goals. Nevertheless,
used as a design strategy in the development of despite the recent conceptualization of generic
persuasive systems (Cugelman 2013; Gardeli methodologies and guidelines for designing
et al. 2017). The design of gamified applications persuasive systems (Oinas-Kukkonen and
entails the incorporation of appropriate Harjumaa 2009), the selection and combination
motivational and playful elements in the core of of design elements and technologies that effec-
interactive systems in order to attract and retain tively support behavioral change represent an
users’ interest while achieving their goals open research issue. There is a growing need
(Papagiannakis 2018). Thus, the gamification for field studies (involving real users for relative
approach is particularly relevant to the design of prolonged time periods) in a variety of usage
mobile persuasive applications. One of the situations and persuasion contexts that will allow
Modeling and Mesh Processing for Games 1175

deeper understanding of the persuasion context 8th International Conference on Persuasive Technol-
and will eventually provide rich feedback on ogy (Persuasive 2013), pp. 7–13 (2013)
Langrial, S., Karppinen, P., Lehto, T., Harjumaa, M.,
various design choices with respect to the target Oinas-Kukkonen, H.: Evaluating mobile-based
group and the desired outcomes (Langrial behavior change support systems for health and well-
et al. 2017). being. In: Behavior Change Research and Theory.
Although the PSD model represents a solid Academic Press. pp. 69–85 (2017)
Oinas-Kukkonen, H., Harjumaa, M.: Persuasive systems
framework that suggests must-have design ele- design: key issues, process model, and system features.
ments of persuasive applications, the formulation Commun. Assoc. Inf. Syst. 24(1), 28 (2009)
of design strategies that specifically suit the unique Papagiannakis, G.: Gamification and serious games.
characteristics of mobile applications is still in In: Lee, N. (ed.) Encyclopedia of Computer Graphics
and Games. Springer, Cham (2018)
need. For instance, the effect of contextual factors
easily captured by mobile devices (e.g., location,
activity, environmental and social context, etc.)
could be considered in a revised PSD model.
Potential revisions of PSD should also include the
Mobile-Based Behavior
incorporation of selected gamification principles.
Change Support Systems

▶ Mobile Persuasive Applications

Cross-References

▶ Games and the Magic Circle

▶ Gamification
Mocap
▶ Gamification of Modern Society: Digital
▶ Sketch-Based Posing for 3D Animation
Media’s Influence on Current Social Practices
▶ Pervasive Games
M
▶ Transformational Games
Mod
References
▶ Counter-Strike Global Offensive, an Analysis
Carter, M.C., Burley, V.J., Nykjaer, C., Cade, J.E.: Adher-
ence to a smartphone application for weight loss com-
pared to website and paper diary: pilot randomized
controlled trial. J. Med. Internet Res. 15(4), e32 (2013) Modding
Conroy, D.E., Yang, C.H., Maher, J.P.: Behavior change
techniques in top-ranked mobile apps for physical
activity. Am. J. Prev. Med. 46(6), 649–652 (2014) ▶ Underground Design of Kaizo Games
Cugelman, B.: Gamification: what it is and why it matters
to digital health behavior change developers. JMIR
Serious Games. 1(1), e3 (2013)
Fogg, B.J.: Persuasive Technology: Using Computers to
Change What We Think and Do. Morgan Kaufmann, Modeling and Mesh
San Francisco (2002) Processing for Games
Gardeli, A., Vosinakis, S., Englezos, K., Mavroudi, D.,
Stratis, M., Stavrakis, M.: Design and development of
games and interactive installations for environmental Jin Huang
awareness. EAI Trans. Serious Games. 4(12), e5 (2017) Zhejiang University, Hangzhou, China
Kasapakis, V., Gavalas, D.: Pervasive gaming: status,
trends and design principles. J. Netw. Comput. Appl.
55, 213–236 (2015)
Langrial, S., Stibe, A., Oinas-Kukkonen, H.: Practical Synonyms
examples of mobile and social apps using the out-
come/change design matrix. In: Proceedings of the Digital geometry processing; Polygonal modeling
1176 Modeling and Mesh Processing for Games

Definition into smaller ones, and the extrude operation (right

of the inset) builds a prism using an existing face
Mesh modeling is the process of constructing a as the base (Fig. 1). Other advanced modeling
polygonal mesh from scratch by basic operations methods usually adopt other surface representa-
or editing an existing mesh for a desired geometric tions, such as NURBS and implicit surface, and
shape. Mesh processing includes various operations then convert the modeling result into a mesh.
which modify input mesh for specific purpose, Mesh processing refers to applying certain
which usually preserve overall shape or local details. operations on a mesh to turn it into a desired
There are various methods to represent a 3D one. Thus, some modeling methods can also be
surface, such as NUBRS and implicit surface. viewed as mesh processing. For games, simplifi-
Among these methods, polygonal mesh is a very cation (left of the inset) (Fig. 2) and subdivision
popular one for game industry, for it is simple, flex- (right of the inset) could be the most important
ible, and efficient. We introduce the modeling and processing over an input mesh (center of the
processing of polygonal mesh, or, for short, mesh. inset), which adjust the mesh to balance the accu-
A mesh is composed of some basic elements: racy and performance.
vertices (red dots), edges (black lines), and faces Simplification turns a mesh into an approxi-
(light blue areas). Each vertex stands for a specific mated one with less number of faces, which sac-
position in a 3D space, each edge connects two rifices the quality to reduce the storage and
vertices, and each face is a surface patch bounded rendering cost. The common strategy to simplify
by a loop of edges. Theoretically, a face is a mesh is to recursively remove basic elements
bounded by any number of edges, but triangular (e.g., vertices, edges, or faces) in order. The order
and quadrilateral faces are the most common ones. is critical and usually determined according to the
error introduced by removing the elements. The
error can be measured in many different ways.
A typical measurement is named as “Quadric
Error Metric,” (Garland et al. 1997) which mea-
sures the change of geometric shape by an easy-
to-compute quadric form in terms of the vertex
positions. Other measurements could take the
change of normal distribution, texture, and other
factors into account. During the simplification,
one could also apply application-dependent
restrictions to the procedure, such as maximal
edge length, minimal angle, and sharp feature
preserving, etc.
On the contrary, subdivision increases the
number of faces for better accuracy. It uses a set
of topological rules to change and add more ele-
To model a surface by mesh, one can first ments into a mesh and then put the updated verti-
create the vertices and then build the edges and ces into certain positions according to geometric
faces. Such a very preliminary way is very ineffi- rules. Recursively refining a mesh can result in a
cient. A widely applied method is to modify a sequence of mesh with finer and finer resolution.
simple mesh into a desired one by repeatedly The limit one is smooth almost everywhere and
applying several basic operations. Two of the tightly related to B-spline and other algebraic
most important operations are split and extrude. surface. The most simple and widely used subdi-
The split operation (center of the inset) cuts a face vision schemes include Catmull-Clark (Stam
Modeling and Mesh Processing for Games 1177

Modeling and Mesh Processing for Games, Fig. 1 starting from a cube (left), use split (middle) and extrude (right)
operations to model a shape

M
Modeling and Mesh Processing for Games, Fig. 2 Simplify (left) and subdivide (right) an input mesh (middle)

1998), Doo and Sabin (1978), and Loop (1987). change the detail of the mesh but roughly keep its
The above subdivision schemes refine the mesh in shape. Such operations view the shape represented
a whole and may introduce too many elements by mesh as a signal and modify the high frequency
unnecessarily. The adaptive subdivision strategy part. Such types of mesh processing are usually
only refines part of the model that needs more related to Laplacian-based methods and tightly
elements. Modern graphics processing unit related to spectral analysis (Taubin 1995). For
(GPU) is able to perform subdivision very effi- example, the smoothing operation reduces high
ciently. Therefore, to render a high quality surface, frequency details from the input mesh (left of the
one can just send a simplified mesh with a small inset), which is similar to a low-pass filter and
number of faces into GPU, and let GPU tessellate results in a mesh with smoother shape (right of
it into a high-quality mesh with a large number of the inset). On the contrary, some of the mesh pro-
faces. Integrating simplification and subdivision cessing, such as deformation and animation, keep
does not only help rendering, but can also be the detail but change the overall shape. Many state-
helpful in network environment. of-the-art deformation and animation methods also
Mesh processing also includes a lot of other found the base on Laplacian of the mesh and adopt
operations, which does not change the topology many differential geometry concepts to character-
but only changes the vertex positions, such as ize the “detail” to be preserved (Huang et al. 2006)
smoothing and enhancement. These two operations (Fig. 3).
1178 Model-View-Controller (MVC)

Modeling and Mesh Processing for Games, Fig. 3 parameterization turns a 3D surface mesh (left) to a 2D planar one
(middle), which helps to map a texture image back to the mesh (right)

References

Doo, D., Sabin, M.: Behavior of recursive division surfaces

near extraordinary points. Comput. Aided Des. 10(6),
356–360 (1978)
Garland, M., Heckbert, P.S.: Surface simplification using
quadric error metrics. In: Proceedings of the 24th
Annual Conference on Computer Graphics and Inter-
active Techniques (SIGGRAPH’97) (1997)
Hormann, K., Lévy, B., Sheffer, A.: Mesh parameteriza-
tion: theory and practic. In: ACM SIGGRAPH 2007
Courses (SIGGRAPH ’07) (2007)
Huang, J., Shi, X., Liu, X., Zhou, K., Wei, L.-Y., Teng,
S.-H., Bao, H., Guo, B., Shum, H.-Y.: Subspace gradi-
ent domain mesh deformation. ACM Trans. Graph.
25(3), 1126–1134 (2006)
Loop, C.: Smooth Subdivision Surfaces Based on Trian-
gles. M.S. Mathematics thesis, University of Utah
(1987)
Stam, J.: Exact evaluation of Catmull-Clark subdivision
surfaces at arbitrary parameter values. In: Proceed-
ings of the 25th Annual Conference on Computer
Graphics and Interactive Techniques
(SIGGRAPH’98) (1998)
Taubin, G.: A signal processing approach to fair surface
General mesh processing even includes some design. In: Proceedings of the 22nd Annual Conference
operations that neither change the topology nor on Computer Graphics And Interactive Techniques
geometry. Take parameterization as an example (SIGGRAPH’95) (1995)
(Hormann et al. 2007), it computes a mapping
(middle of the inset) from a planar domain to a
mesh (left of the inset) so that a planar signal can
be transported onto the mesh (right of the inset). Model-View-Controller (MVC)
Texturing is a typical application of parameteriza-
tion, which enhances the appearance of a mesh by ▶ Interactive Computer Graphics and Model-
mapping an image onto the mesh. View-Controller Architecture
Monte-Carlo Tree Search 1179

executing the selection, play-out, expansion, and

Monte-Carlo Tree Search backpropagation steps. Such an iteration is called
a full simulation. After a certain number of simu-
Mark H. M. Winands lations, a move is chosen to be played in the actual
Department of Data Science and Knowledge game. This final move selection is based on the
Engineering, Maastricht University, Maastricht, highest score or alternatively the number of times
The Netherlands being sampled. The detailed structure of MCTS is
discussed by explaining the four steps below.

Synonyms Selection
Selection chooses a child to be searched based on
MCTS; UCT previous information. It controls the balance
between exploitation and exploration. On the
one hand, the task consists of selecting the move
Definition that leads to the best results so far (exploitation).
On the other hand, the less promising moves still
Monte-Carlo Tree Search (MCTS) (Coulom have to be tried, due to the uncertainty of the
2007; Kocsis et al. 2006) is a best-first search simulations (exploration).
method that does not require a positional evalua- Several selection strategies (Browne
tion function. It is based on a randomized explo- et al. 2012) have been suggested for MCTS such
ration of the search space. Using the results of as BAST, EXP3, and UCB1-Tuned, but the most
previous explorations, the algorithm gradually popular one is based on the UCB1 algorithm
builds up a game tree in memory and successively (Auer et al. 2002), called UCT (Upper Confidence
becomes better at accurately estimating the values Bounds applied to Trees) (Kocsis et al. 2006).
of the most promising moves. MCTS consists of UCT works as follows. Let I be the set of nodes
M
four strategic steps, repeated as long as there is immediately reachable from the current node p.
time left (Chaslot et al. 2008b). The steps, The selection strategy selects the child b of node
outlined in Fig. 1, are as follows: p that satisfies Formula 1:

1. In the selection step, the tree is traversed from ln np

the root node downward until a state is chosen, b∈arg max i∈I vi þ C  ð1Þ
ni
which has not been stored in the tree.
2. Next, in the play-out step, moves are chosen in
where vi is the value of the node i, ni is the visit
self-play until the end of the game is reached.
count of i, and np is the visit count of p. C is a
3. Subsequently, in the expansion step, one or
parameter constant, which can be tuned experi-
more states encountered along its play-out are
mentally (e.g., C ¼ 0.4). The value of vi should lie
added to the tree.
in the range [0, 1]. In case a child has not been
4. Finally, in the backpropagation step, the game
stored in the tree or has not been visited yet, a
result r is propagated back along the previously
default value is assumed. For example, the maxi-
traversed path up to the root node, where node
mum value that a node could obtain by sampling
statistics are updated accordingly.
(i.e., vmax ¼ 1) is taken.

Structure of MCTS Play-Out

When in the selection step a state is chosen, which
MCTS usually starts with a tree containing only has not been stored in the tree, the play-out starts.
the root node. The tree is gradually grown by Moves are selected in self-play until the end of the
1180 Monte-Carlo Tree Search

Repeated X times

Selection Play-out Expansion Backpropagation

A selection strategy is One simulated One or more nodes The result is propagated
used to traverse the tree game is played are created back in the tree

Monte-Carlo Tree Search, Fig. 1 Outline of Monte-Carlo Tree Search (adapted from Chaslot et al. 2008b; Winands
et al. 2010)

game is reached. This task might consist of result is backpropagated as a tuple of size N,
playing plain random moves or – better – semi- where N is the number of players. For instance,
random moves chosen according to a simulation if Player 1 and Player 3 both reach a winning
strategy. Smart simulation strategies have the condition in a 3-player game, then the result r is
potential to improve the level of play significantly. returned as the tuple 12 , 0, 12 . Propagating the
The main idea is to play interesting moves based values back in the tree is performed similar to
on heuristics. In the literature this play-out step is maxn (Sturtevant 2008).
sometimes called the roll-out or simulation. To compute the value vi of a node i, a back-
propagation strategy is applied. Usually, it is cal-
culated by taking the average of the results of all
Expansion
simulated games made through this node (Coulom
Expansion is the procedure that decides whether
2007), i.e., vi Ri,j/ni, where j is the player to
nodes are added to the tree. Standard the follow-
move in its parent node p and Ri,j r the
t t,j
ing expansion strategy is sufficient in most cases:
one node is added per simulation (Coulom 2007). cumulative score of all the simulations.
The added leaf node L corresponds to the first
state encountered during the traversal that was
not already stored. This allows to save memory MCTS Enhancements
and reduces only slightly the level of play.
Over the past years, several enhancements have
Backpropagation been developed to improve the performance of
Backpropagation is the procedure that propa- MCTS (Browne et al. 2012). First, there are
gates the result r of a simulated game t back many ways to improve the selection step of MCT-
from the leaf node L, through the previously S. The major challenge is how to choose a prom-
traversed nodes, all the way up to the root. If a ising node when the number of simulations is still
game is won, the result of a player j is scored as rt, low. Domain-independent techniques that only
j ¼ 1, in the case of a loss as rt,j ¼ 0, and a draw as use information gathered during the simulations
rt,j ¼ 0.5. To deal with multiplayer games, the are Transposition Tables, Rapid Action Value
Monte-Carlo Tree Search 1181

Estimation (RAVE), and Progressive History MCTS tree. When the allotted search time is up,
(Childs et al. 2008; Gelly et al. 2012; Nijssen the results of the different trees are combined. In
and Winands 2011). Techniques that rely on leaf parallelization, one tree is traversed using a
hand-coded domain knowledge are, for instance, single thread. Subsequently, starting from the leaf
Move Groups, Prior Knowledge, Progressive node, play-outs are executed in parallel for each
Bias, and Progressive Widening/Unpruning available thread. Once all threads have finished,
(Chaslot et al. 2008b; Childs et al. 2008; Gelly the results are backpropagated. When using tree
et al. 2012). The used heuristic knowledge may parallelization, one tree is shared, in which all
consist of move patterns and even static board threads operate independently. For shared mem-
evaluators. When a couple of these enhancements ory systems, tree parallelization is the natural
are successfully incorporated, the C parameter of approach that takes full advantage of the available
UCT becomes usually very small or even zero. bandwidth to communicate simulation results
Next, the play-outs require a simulation strategy (Enzenberger and Müller 2010).
in order to be accurate. Moves are chosen based on
only computationally light knowledge (Gelly
et al. 2012) (e.g., patterns, capture potential, and Historical Background
proximity to the last move). Adding computation-
ally intensive heavy heuristic knowledge in the Classic search algorithms such as A*, αβ search,
play-outs (such as a 1- or 2-ply search using a full or Expectimax require an evaluator that assigns
board evaluator) has been beneficial in a few games heuristic values to the leaf nodes in the tree. The
such as Chinese Checkers and Lines of Action. 15-puzzle and the board games backgammon,
When domain knowledge is not readily available, chess, and checkers are instances where this
there exist various domain-independent techniques approach has led to world-class performance.
to enhance the quality of the play-outs, including However, for some domains constructing a strong
the Move-Average Sampling Technique (MAST), static heuristic evaluation function has been a
M
Last-Good-Reply Policy, and N-Grams (Tak rather difficult or an even infeasible task.
et al. 2012). The principle of these techniques is Replacing such an evaluation function with
that moves good in one situation are likely to be Monte-Carlo sampling was proposed in the early
good in other situations as well. 1990s. Abramson (1990) experimented with these
The basic version of MCTS converges to the so-called Monte-Carlo evaluations in the games
game-theoretic value, but is unable to prove it. of tic-tac-toe, Othello, and chess. In 1993 Bernd
The MCTS-Solver technique (Winands et al. Brügmann was the first to use Monte-Carlo eval-
2010) is able to prove the game-theoretic value uations in his 9  9 Go program Gobble. The
of a state with a binary outcome (i.e., win or loss). following years, the technique was incorporated
It labels terminal states in the search tree as a win in stochastic games such as backgammon
or loss and backpropagates the game-theoretic (Tesauro et al. 1997) and imperfect-information
result in a maxn way (Nijssen and Winands games such as bridge (Ginsberg 1999), poker
2011). For games with multiple outcomes (e.g., (Billings et al. 1999), and Scrabble (Sheppard
win, loss, or draw), the technique has been 2002).
extended to Score Bounded Monte-Carlo Tree In the early 2000s, the Monte-Carlo approach
Search (Cazenave and Saffidine 2011). received new interest in the Computer Go domain
Finally, to utilize the full potential of a multi- (Bouzy and Helmstetter 2004). Bruno Bouzy’s
core machine, parallelization has to be applied in Monte-Carlo Go engine Indigo had some limited
an MCTS program. There exist three different success as the main challenge was to effectively
parallelization techniques for MCTS: (1) root combine Monte-Carlo evaluations with game-tree
parallelization, (2) leaf parallelization, and search. The breakthrough came when Coulom
(3) tree parallelization (Chaslot et al. 2008a). In presented the MCTS approach at the 2006 Com-
root parallelization, each thread has its own puters and Games Conference (Coulom 2007). He
1182 Monte-Carlo Tree Search

subsequently demonstrated its strength by win- based strategy game Total War: Rome II and for
ning the 9  9 Go tournament at the 12th ICGA tactical assault planning in the real-time strategy
Computer Olympiad with his MCTS engine game Wargus (Balla et al. 2009). The MCTS
Crazy Stone. Simultaneously Kocsis and framework has also shown promise in the General
Szepesvári (Kocsis et al. 2006) introduced the Video Game AI Competition (Perez et al. 2014),
MCTS variant UCT. Its selection strategy became where the goal is to build an agent that is capable
the standard for many MCTS engines (Browne of playing a wide range of (simple) video games.
et al. 2012). Techniques such as RAVE, Prior MCTS has also been applied in puzzle games
Knowledge, Progressive Bias, and Progressive such as SameGame (Schadd et al. 2012) where it
Widening (Chaslot et al. 2008b; Gelly is hard to design an admissible evaluation func-
et al. 2012) were needed to make MCTS effective tion for A* or IDA*. As these games are close to
in many challenging domains such as 19  19 Go. scheduling and optimization problems, MCTS
Parallelization (Enzenberger et al. 2010; Gelly has been introduced in real-life applications.
et al. 2012) has enabled MCTS to compete with They are, for instance, high energy physics
human Go Grandmasters. As of 2014, an MCTS (Ruijl et al. 2014), patient admission scheduling
engine can beat a 9-dan professional player with (Zhu et al. 2014), and interplanetary trajectory
only a four-stone handicap, whereas a decade ago planning (Hennes et al. 2015).
20 stones could be given.

Future Directions
Applications
MCTS does not require a positional evaluation
In the past few years, MCTS has substantially function, overcoming partially the knowledge
advanced the state of the art in several abstract acquisition bottleneck. It is therefore a promising
games (Browne et al. 2012), in particular Go method when an agent has to play a wide range of
(Gelly et al. 2012), but other two-player deter- games as is fostered in the General (Video) Game
ministic perfect-information games include Ama- Playing competitions. However, for MCTS to
zons (Lorentz 2008), Hex (Arneson et al. 2010), work effectively, search-control knowledge is
and Lines of Action (Winands et al. 2010). required to guide the simulations. Domain-
MCTS has even increased the level in multi- independent techniques are able to boost the deci-
player games such as Chinese checkers sion quality of an MCTS engine, but for achieving
(Sturtevant 2008) and games with stochasticity expert level hand-coded domain knowledge is
and/or imperfect information such as Kriegspiel incorporated to grasp high-level context. Instead
(Ciancarini and Favini 2010), Lord of the Rings: of being hand-coded by the programmer, a future
The Confrontation (Cowling et al. 2012), and research direction is to automatically discover,
Scotland Yard (Nijssen and Winands 2012). In extract, represent, and tune this control knowl-
the General Game Playing competition, where an edge during online search.
agent has to play many different abstract games MCTS has been quite successful in abstract
without any human intervention, MCTS has games; however, the number of successful appli-
become the dominant approach as well cations in modern video games with high fidelity
(Björnsson and Finnsson 2009). is rather limited. There are three challenges for
Besides application to abstract games, MCTS applying MCTS in these games. (1) In these video
has made inroads in the video game domain. It has games, the action space is large if not infinite, and
been applied in the arcade game Ms. Pac-Man for the state space is often continuous. For MCTS to
controlling either the Ghosts or the Pac-Man work effectively, the game world has to be
(Nguyen and Thawonmas 2013; Pepels abstracted automatically in such a way that
et al. 2014). The technique has been used for (i) the number of possible moves is limited and
resource allocation and coordination in the turn- (ii) the number of moves required to finish the
Monte-Carlo Tree Search 1183

game is reduced as well. (2) These games have a Samothrakis, S., Colton, S.: A survey of Monte Carlo
high degree of uncertainty, not only due to Tree Search methods. IEEE Trans. Comput. Intell. AI
Games 4(1), 1–43 (2012)
non-determinism (the outcome of a move cannot Cazenave, T., Saffidine, A.: Score bounded Monte-Carlo
be predicted) or imperfect information (certain Tree Search. In: van den Herik, H.J., Iida, H., Plaat,
information is hidden for a player) but also A. (eds.) Computers and Games (CG 2010). Lecture
because of incomplete information (the behavior Notes in Computer Science, vol. 6515, pp. 93–104.
Springer, Berlin (2011)
of the physics engine may be unknown). For Chaslot, G.M.J.-B., Winands, M.H.M., van den Herik, H.
non-determinism and imperfect information, J.: Parallel Monte-Carlo Tree Search. In: van den Herik,
MCTS enhancements have been investigated to H.J., Xu, X., Ma, Z., Winands, M.H.M. (eds.) Com-
a limited number of abstract games (Cowling puters and Games (CG 2008). Lecture Notes in Com-
puter Science, vol. 5131, pp. 60–71. Springer, Berlin
et al. 2012), but even less for video games. Deal- (2008a)
ing with incomplete information in the MCTS Chaslot, G.M.J.-B., Winands, M.H.M., van den Herik, H.
framework is a largely unexplored terrain. J., Uiterwijk, J.W.H.M., Bouzy, B.: Progressive strate-
(3) Due to the real-time property the amount of gies for Monte-Carlo Tree Search. New Math. Nat.
Comput. 4(3), 343–357 (2008b)
deliberation time is limited. MCTS has to generate Childs, B.E., Brodeur, J.H., Kocsis, L.: Transpositions and
a sufficient number of simulations in a short time move groups in Monte Carlo Tree Search. In: Hingston,
as otherwise the decision quality is quite low P., Barone, L. (eds.) Proceedings of the
(Björnsson and Finnsson 2009). 2008 IEEE Symposium on Computational Intelligence
and Games, pp. 389–395. IEEE, Piscataway, NJ, USA
(2008)
Ciancarini, P., Favini, G.P.: Monte Carlo Tree Search in
Kriegspiel. AI J. 174(11), 670–684 (2010)
References and Further Reading Coulom, R.: Efficient selectivity and backup operators in
Monte-Carlo Tree Search. In: van den Herik, H.J.,
Abramson, B.: Expected-outcome: A general model of Ciancarini, P., Donkers, H.H.L.M. (eds.) Computers
and Games (CG 2006). Lecture Notes in Computer
static evaluation. IEEE Trans. Pattern Anal. Mach.
Intell. 12(2), 182–193 (1990) Science, vol. 4630, pp. 72–83. Springer, Berlin
M
Arneson, B., Hayward, R.B., Henderson, P.: Monte Carlo (2007)
Tree Search in Hex. IEEE Trans. Comput. Intell. AI Cowling, P.I., Powley, E.J., Whitehouse, D.: Information
Games 2(4), 251–258 (2010) set Monte Carlo Tree Search. IEEE Trans. Comput.
Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis Intell. AI Games 4(2), 120–143 (2012)
of the multiarmed bandit problem. Mach. Learn. Enzenberger, M., Müller, M.: A lock-free multithreaded
47(2–3), 235–256 (2002) Monte-Carlo Tree Search algorithm. In: van den
Balla, R.K., Fern A.: UCT for tactical assault planning in Herik, H.J., Spronck, P. (eds.) Advances in Computer
real-time strategy games. In: Boutilier, C. (ed.) Pro- Games (ACG 2009). Lecture Notes in Computer Sci-
ceedings of the Twenty-First International Joint Con- ence (LNCS), vol. 6048, pp. 14–20. Springer, Berlin
ference on Artificial Intelligence (IJCAI-09), (2010)
pp. 40–45. AAAI Press, Menlo Park, CA, USA (2009) Enzenberger, M., Müller, M., Arneson, B., Segal, R.: Fuego –
Billings, D., Peña, L., Schaeffer, J., Szafron, D.: Using an open-source framework for board games and Go engine
probabilistic knowledge and simulation to play poker. based on Monte Carlo Tree Search. IEEE Trans. Comput.
In: Hendler, J., Subramanian, D. (eds) Proceedings of Intell AI Games 2(4), 259–270 (2010)
the Sixteenth National Conference on Artificial Intelli- Gelly, S., Kocsis, L., Schoenauer, M., Sebag, M., Silver,
gence and Eleventh Conference on Innovative Appli- D., Szepesvári, C., Teytaud, O.: The grand challenge of
cations of Artificial Intelligence, pp. 697–703. AAAI computer Go: Monte Carlo Tree Search and extensions.
Press/The MIT Press, Menlo Park, CA, USA (1999) Commun. ACM 55(3), 106–113 (2012)
Björnsson, Y., Finnsson, H.: CadiaPlayer: A simulation- Ginsberg, M.L.: GIB: Steps toward an expert-level bridge-
based General Game Player. IEEE Trans. Comput. playing program. In: Dean, T. (ed.) Proceedings of the
Intell. AI Games 1(l), 4–15 (2009) Sixteenth International Joint Conference on Artificial
Bouzy, B., Helmstetter, B.: Monte-Carlo Go develop- Intelligence (IJCAI-99), vol. 1, pp. 584–589. Morgan
ments. In: van den Herik, H.J., Iida, H., Heinz, Kaufmann, San Francisco, CA, USA (1999)
E.A. (eds.) Advances in Computer Games 10: Many Hennes, D., Izzo, D.: Interplanetary trajectory planning
Games, Many Challenges. IFIP Advances in Informa- with Monte Carlo Tree Search. In: Yang, Q.,
tion and Communication Technology, vol. Wooldridge, M. (eds.) Proceedings of the Twenty-
135, pp. 159–174. Kluwer, Boston (2004) Fourth International Joint Conference on Artificial
Browne, C.B., Powley, E., Whitehouse, D., Lucas, S.M., Intelligence (IJCAI 2015), pp. 769–775. AAAI Press,
Cowling, P.I., Rohlfshagen, P., Tavener, S., Perez, D., Menlo Park, CA, USA (2015)
1184 Motion and Posture Analysis

Kocsis, L., Szepesvári, C.: Bandit based Monte-Carlo

Planning. In: Fürnkranz, J., Scheffer, T., Spiliopoulou, Motion Capture
M. (eds.) Machine Learning: ECML 2006. Lecture
Notes in Artificial Intelligence, vol. 4212,
pp. 282–293. Springer, Berlin (2006) ▶ Sketch-Based Posing for 3D Animation
Lorentz, R.J.: Amazons discover Monte-Carlo. In: van den
Herik, H.J., Xu, X., Ma, Z., Winands, M.H.M. (eds.)
Computers and Games (CG 2008). Lecture Notes in Com-
puter Science, vol. 5131, pp. 13–24. Springer, Berlin (2008)
Nguyen, K.Q., Thawonmas, R.: Monte Carlo Tree Search for Motion Matching
collaboration control of Ghosts in Ms. Pac-Man. IEEE
Trans. Comput. Intell. AI Games 5(1), 57–68 (2013) ▶ Motion Matching: Data-Driven Character Ani-
Nijssen, J.A.M., Winands, M.H.M.: Enhancements for
multi-player Monte-Carlo Tree Search. In: van den mation Systems
Herik, H.J., Iida, H., Plaat, A. (eds.) Computers and
Games (CG 2010). Lecture Notes in Computer Science,
vol. 6151, pp. 238–249. Springer, Berlin (2011)
Nijssen, J.A.M., Winands, M.H.M.: Monte Carlo Tree
Search for the hide-and-seek game Scotland Yard. Motion Matching: Data-
Trans. Comput. Intell. AI Games 4(4), 282–294 (2012) Driven Character Animation
Pepels, T., Winands, M.H.M., Lanctot, M.: Real-time Systems
Monte Carlo Tree Search in Ms Pac-Man. IEEE
Trans. Comput. Intell. AI Games 6(3), 245–257 (2014)
Perez, D., Samothrakis, S., Lucas, S.M.: Knowledge-based Adan Häfliger
fast evolutionary MCTS for general video game Cygames Research, Cygames, Inc., Tokyo, Japan
playing. In: Proceedings of the IEEE Conference on
Computational Intelligence and Games (CIG 2014),
pp. 68–75 (2014)
Ruijl, B., Vermaseren, J., Plaat, A. van den Herik, H.J.: Synonyms
Combining simulated annealing and Monte Carlo Tree
Search for expression simplification. In: ICAART Distance Matching; Motion Matching
2014, pp. 724–731 (2014)
Schadd, M.P.D., Winands, M.H.M., Tak, M.J.W., Uiterwijk,
J.W.H.M.: Single-player Monte-Carlo Tree Search for
SameGame. Knowl.-Based Syst. 34, 3–11 (2012) Definition
Sheppard, B.: World-championship-caliber Scrabble.
Artif. Intell. 134(1–2), 241–275 (2002) Motion matching is a data-driven technique for
Sturtevant, N.R.: An analysis of UCT in multi-player
games. ICGA J. 31(4), 195–208 (2008) animating virtual characters in real-time with a
Tak, M.J.W., Winands, M.H.M., Björnsson, Y.: N-Grams high degree of realism. This method relies on a
and the last-good-reply policy applied in general game large set of preprocessed motion data, typically
playing. IEEE Trans. Comput. Intell. AI Games 4(2), captured using motion capture devices. In a game
73–83 (2012)
Tesauro, G., Galperin, G.R.: On-line policy improvement loop, the motion matching system continually
using Monte-Carlo search. In: Mozer, M.C., Jordan, M. searches a motion database to find the most suit-
I., Petsche, T. (eds.) Advances in Neural Information able short motion clip to fulfill the user’s inten-
Processing Systems, vol. 9, pp. 1068–1074. MIT Press, tions and the game requirements. The chosen
Cambridge, MA, USA (1997)
Winands, M.H.M., Björnsson, Y., Saito, J.-T.: Monte Carlo motion is then seamlessly blended with the cur-
Tree Search in Lines of Action. IEEE Trans. Comput. rently playing character animation to create real-
Intell. AI Games 2(4), 239–250 (2010) istic transitions.
Zhu, G., Lizotte, D., Hoey, J.: Scalable approximate policies
for Markov decision process models of hospital elective
admissions. Artif. Intell. Med. 61(1), 21–34 (2014)
Introduction

Motion and Posture Analysis Video games and other interactive applications are
constantly striving for higher levels of graphical
▶ Data Gloves for Hand and Finger Motion fidelity and immersion. While visuals are
Interactions certainly important, the realism of the dynamic
Motion Matching: Data-Driven Character Animation Systems 1185

interactions between characters and their virtual length, this component is typically assumed to be
worlds may be even more crucial to achieving a equal to one. For cartoony characters, the scale
sense of immersion. Therefore, animation systems component could vary.
play a vital role in modern game development. As Equation 1 is used to compose an affine trans-
graphics continue to evolve and become more formations A0 over A1. This equation first translates
detailed, there is a corresponding need for equally A1 by rotating it with the q component of A0 and
detailed world dynamics and animations. An adding the p component of A0. It then computes the
effective animation system must be able to handle new orientation of A1 by composing q0 with q1,
an increasing number of interactions and support effectively applying the two rotations in order.
the creation of rich virtual worlds.
p ¼ q0  p 1 þ p 0 :
Conventional animation systems, which rely ð1Þ
heavily on hand-crafted state machines and para- q ¼ q0  q 1 :
metric blending of animations, are not suitable for
many scenarios because they do not scale well. Skeletal Mesh Animation
Manually modeling every potential interaction Virtual characters are typically represented using
with a state machine is impractical as the number skeletal meshes, which consist of a set of poly-
of possible state transitions increases combinato- gons that make up the character’s surface and a
rially. To address this issue, motion matching hierarchy of interconnected bones used to deform
(Büttner and Clavet 2015) was developed as a the polygons. The polygons are composed of ver-
data-driven animation technique that does not tices, and the way in which each bone influences
explicitly model every possible interaction. the movement of the polygons it is associated with
Instead, it searches an animation database for the is determined by skin weights, which are assigned
most suitable short sequence for a given set of to each vertex of the mesh. This process, known as
circumstances. mesh skinning, is used to bind the character’s
Motion matching has been widely adopted by surface to its bones. However, the specifics of M
game studios, including For Honor (Clavet 2016), mesh skinning are not covered here.
EA Sports UFC 3 (Harrower 2018), The Last of It is useful to model a chain of bones such that
Us 2 (Michal and Zhuravlov 2021), and Control rotating a bone automatically applies the same
(Ilkka and Ville 2021). rotation to its children. To do this, each bone i is
I ðiÞ
modeled using an affine transformation Ai that
is defined relative to the affine space of its parent
I ðIðiÞÞ
Character Animation AIðiÞ , where I(i) is a function that maps bone i to
its parent. This is achieved by applying the inverse
In order to create an animation system, a mathe- transformation of the parent, AI(i)i ¼ A1I(i)  Ai.
matical model of the character and its movements The pose of a character at frame k is then
over time must be developed. This includes defin- represented as an ordered set of N affine trans-
ing the foundational concepts and data structures I ð1Þ I ðN Þ
formations, Pk ¼ AW 0,k , A1,k , . . . , AN,k : The
used to represent character animations. Typically,
animations are made up of discrete character root bone’s affine transformation, AW 0 , is special
poses that are stored and interpolated through as and is defined relative to the coordinate system of
time progresses. the game world, as indicated by the W superscript.
The central element in character animation is The above construction has the property that
the affine transformation, which concisely translating the whole set of bones in the game
describes a position and direction in an affine world only involves translating AW 0 : However,
space. An affine transformation A is defined by a when rendering the character, converting every
translation p ∈ ℝ3 and a rotation q∈3 , both relative transformations to the game world coor-
relative to an affine space. Affine transformations dinate system is necessary. Each bone has exactly
can also include a scale component, but in the case one parent, meaning that one can compute every
of human bones, which are rigid and of constant world space affine transformations in a single
1186 Motion Matching: Data-Driven Character Animation Systems

Motion Matching: Data-

Driven Character
Animation Systems,
Fig. 1 The skeletal mesh
of a freely available 3D
model. (© Unity
Technologies Japan/UCL)
The multiple polygons
composing the characters
are delineated with black
lines, and the bone
hierarchy is displayed with
red spheres and stripes

ordered pass starting from the root bone using values such as the desired character position in
forward kinematics. For example, if the bones 60 frames or the foot starting position at the start
with indices 1 and 2 have parents indexed as of the sought short sequence. This obtained pose
0 and 1, respectively, the calculations will be sequence is then intelligently blended into the
0
A 1 ¼ AW0  A1 followed by A2 ¼ A1  A12 : already playing animation. When the goals of
(Fig. 1). the animation system are updated or a fixed inter-
A skeletal mesh animation is therefore a val of time has passed, a new search is performed
sequence of character poses: to find an appropriate short pose sequence and
blend it in. This continuous blending, coupled
s ¼ ½P0 , P1 , . . . :, Pn  ð2Þ with the goal of minimizing the new sequence’s
disparity through the search process, is key to
where n is the number of frames in the animation. synthesizing smooth character motion.
For simplicity, it is assumed that the animations To enable the search, the poses Pk of each
are densely sampled at each frame and that the animation sequence s are concatenated to form
sample rate is equal to the application frame rate. K consecutive frames. An algorithm is then used
Playing back an animation in this context is equiv- to search for a frame k⁎ ∈ ℕ : k⁎ < K from which to
alent to flipping through the pages of a flipbook. start playing back a pose sequence. Figure 2 illus-
trates the database structure that contains a global
index mapping to each existing pose. During the
Motion Matching search, the selected frame k⁎ best conforms to the
animation system’s goals. To find the most similar
Motion matching is a technique that works by frame, a set of numerical features that characterize
defining the high-level goals of the animation the situation at each frame k is extracted and
system and searching a database for the short compared to query features, which encode the
subset of an animation that best matches those desired properties of the wanted sequence. Choos-
goals. The system defines goals with concrete ing the appropriate information to put in the
Motion Matching: Data-Driven Character Animation Systems 1187

Motion Matching: Data-Driven Character Animation frame. Also, a mapping per action in the dataset is
Systems, Fig. 2 Schematic of a motion database. All maintained to search only a relevant subset of the data.
pose sequences are concatenated, forming K frames Frames at the end of sequences are invalid as they have no
indexing an array of poses. Each frame is associated with defined future and therefore are not valid candidates to start
a numerical feature vector that describes the situation at the playing back the animation

features, which consists of designing a query see how each frame of data forms a smooth trajec-
space, is fundamental to a working implementa- tory in an hypothetical feature space.
tion of motion matching. Motion matching is essentially making search
Section “Query Space” introduces concrete queries, and the design of the possible queries
M
feature definition and usage, section “Search” pre- impacts the quality of the generated motion. It is
sents the search algorithm, section “Blending” necessary to have simple and general queries. To
describes the motion blending procedure, section improve the generality of spatial features, they are
“Procedural Adjustments” introduce post- encoded in a relative coordinate system based on
processing steps, and section “Making Queries” the character’s facing direction projected to the
touches upon how to create the query features. ground, which is called the character reference
frame. The character reference frame is described
Query Space with an affine transform r which can be applied to
The features reflect the high-level goals of the search any transform as r1  A encoding A in the
because they specify the values that the searched character reference frame. By applying r1, spa-
motion sequence should approach. For example, by tial features and pose sequences are made invari-
considering a character’s in-game right foot position ant to translations and rotations in the plane
as a feature during the search, the system prioritizes relative to the character. For example, the pose
motion that starts with the same right foot position. sequence and spatial features of a character walk-
Similarly, by adding the in-game character’s desired ing forward in any cardinal direction are the same
future right foot position as a feature, the search is in the character reference frame since it is forward
steered toward selecting a motion that will move the with respect to the character’s direction. Thanks to
right foot to that position. The first example is a the invariance provided by the character reference
pose-conserving feature, while the second example frame, it is not necessary to record a separate
is a controlling feature. It is important to balance the walking forward animation for every possible
influence of pose-conserving and controlling fea- direction.
tures to achieve an acceptable trade-off between Each frame k in the database is associated with
responsiveness and smoothness. In Fig. 3, one can a pose and a set of numerical features. The j-th
1188 Motion Matching: Data-Driven Character Animation Systems

query space 2D projection with poses

Walk
Crouch
Closest

frame 71
frame 72
frame 73

frame 74

frame 75

or
g hb
n ei
st
se
clo frame 76

frame 77
frame 78
query frame 79
frame 80 frame 81

Motion Matching: Data-Driven Character Animation query point. The frames form a discretized trajectory in the
Systems, Fig. 3 An hypothetical 2D query space made feature space and the distance between the query and each
from the features of multiple frames of motion data and a frame can be computed

feature at frame k is computed as fj, k ∈ ℝ where K

j ∈ {0, . . ., J} and J is the number of features. 1 2
sj ¼ f j,k  mj ð5Þ
Concatenating every frame’s features vertically K k¼0
forms the feature matrix F.
Each feature is also optionally tuned with a
f 0,0 ... f 0,K custom weight wj yielding the final scaled feature
w f
F¼ ⋮ ⋱ ⋮ ð3Þ as dj,k ¼ j sj j,k :
f J,0 ... f J,K Those feature are compared with a query vec-
tor q ¼ [d0,q, d1,q, . . ., dJ,q, ] which is computed in
To account for the variable scales of the fea- real time from the game situation. Searching the
tures (rows), the mean mj and standard deviation sj best frame turns into a comparison between q and
of each feature are computed, where fj,k is the j-th every scaled columns of F.
row and k-th column of the feature matrix F as Holden et al. (2020) and Bergamin (2021)
shown below: shared a set of spatial features found to be effec-
tive for locomotion. Figure 4 illustrates three con-
K ventional motion matching features. Firstly,
1
mj ¼ f j,k ð4Þ steering and velocity control are achieved by
K k¼0 matching the character reference frame positions
Motion Matching: Data-Driven Character Animation Systems 1189

cost of one recorded frame is measured using

Foot in
Eq. 6, which calculates the squared distance with
contact each feature vector.

Trajectory J
2
ck ¼ dj,q  dj,k ð6Þ
j¼0

In the end, the pose sequence associated with

the frame with the lowest total cost k⁎ ¼ arg mink
(3) ck will be blended in. A variation proposed in
Häfliger and Kurabayashi (2021) relies on a spe-
cific distance function per feature instead of a
(2)
generic distance function, removing the need to
compute the scaled features dj,k but instead nor-
malizes the separate costs.
The search algorithm scales with the number of
compared frames, leading to a trade-off between
(1)
the amount of variation in the dataset and the
retrieval speed. Fortunately, there is a lot of liter-
Motion Matching: Data-Driven Character Animation
ature on optimizing the k-nearest neighbors algo-
Systems, Fig. 4 An illustration of effective spatial fea-
tures for locomotion extracted from frames of motion data. rithm to accelerate the search while guaranteeing a
(1) A group of future positions of the character. (2) The valid result. In most cases, only the frame with the
positions and velocities of the feet. (3) The hip velocity. The minimum cost is relevant (closest neighbor), and
group in (1) is made of control features, while (2) and (3) are
pose-conserving features. Feet in contact with the ground
it is not necessary to compute the exact score of M
are also marked for procedural adjustments each frame. To accelerate the search, the early out
matching variation shown in Algorithm 1 can be
used. This method involves skipping a candidate
0.33, 0.66, and 1 second in the future relative to
frame as soon as its partial cost sum exceeds the
the current character reference frame. This
score of the current best frame.
matching compares candidate trajectories starting
at every frame with the desired trajectory formed
from a game-pad. Secondly, matching the charac- Algorithm 1 Early out matching
ter’s current foot velocities and positions relative Input: A reference to the motion database and
to the character reference frame maintain consis- the query vector
tent foot placement and reduce foot sliding arti- Output: The index of the pose sequence in the
facts. Thirdly, matching the character’s current hip DB with the lowest cost, k⁎
velocity relative to the character reference frame 1: d0,q, d1,q, . . ., dJ,q query vector
stabilizes the character’s orientation. 2: c⁎ c0
3: k⁎ 0
Search 4: for k ¼ 1, 2, . . . ,K do
The motion matching search is identical to finding 5: ck 0
the closest neighbor in the k-nearest neighbors 6: for j ¼ 0, 1, . . . , J do
algorithm (Holden 2018). The idea is straightfor- 7: dj,k motion database
ward; a transition cost between the query vector 8: ck ck + (dj,q  dj,k)2
q and each frame’s feature in the database is used 9: if ck > c⁎ then
to rank neighbors. Figure 3 illustrates the geomet- 10: skip to next k
rical closeness of an hypothetical query vector 11: end if
with recorded features in a 2D feature space. The end for
1190 Motion Matching: Data-Driven Character Animation Systems

12: if ck < c⁎ then Blending

13: k⁎ k After the search, the optimal frame k⁎ from where
14: c⁎ ck to start playing back the pose sequence is
15: end if obtained. However, if the new sequence is directly
end for played, a noticeable jump in the character pose
16: return k⁎ will be visible. One solution called cross-fading
consists of interpolating toward the new sequence
Another conventional acceleration is achieved in a short period by gradually translating/rotating
by pre-computing a hierarchy of bounding vol- each bone position/rotation toward their new
umes per feature vector. The cost of the bounding values. For human motion data, such interpolation
volume is checked before the feature vectors produces natural transitions (Safonova and
inside the volume, skipping a majority of score Hodgins 2005).
computation when the bounds are well tuned. In Assuming two sequences s0 and s1 of length n,
practice, as hinted by the database layout in Fig. 2, each pose is merged using a mix function: Pi ¼
only frames whose action corresponds to the cur- mix P0i , P1i , ni ∀i ∈0::n, where mix applies a lin-
rent user-selected action are considered. During ear interpolation (Lerp) to each bone position and
transitions, it is also advised to consider past a spherical linear interpolation (Slerp) to each
actions. A mapping between actions and their rotations. The factor ni controls the influence
corresponding frames is stored to enable this between the old and the new sequences, when
pruning of the search. i ¼ n only the new pose is considered. The Slerp
It is also possible to accelerate motion matching and Lerp functions are described in Eq. 7 for
by tolerating some retrieval imprecision. Tech- positions p0, p1, quaternions q0, q1, and coeffi-
niques such as quantization or dimensionality cient α.
reduction applied to every dj,k can reduce the com-
putational burden of the search. A dimensionality
reduction technique, PCA (Pearson 1901; Hotelling Lerpðp0 , p1 , aÞ ¼ ð1  aÞ  p0 þ a  p1 :
a ð7Þ
1933), diminishes the feature vector sizes by pro- Slerpðq0 , q1 , aÞ ¼ q0 q1
0 q1 :
jecting them to a lower-dimension coordinate sys-
tem. Consider a feature matrix X ∈ ℝKJ that Cross-fading is not preferred in practice, as
contains K mean normalized feature vectors of it requires evaluating all blended sequences,
J dimensions (one vector per row). PCA provides which is resource intensive in scenarios where
a projection matrix W ∈ ℝJJ composed of J eigen- there may be more than two blended sequences.
vectors ordered by their influence on the total var- In complex games, the actual motion results
iance of the projected features. A subset of the most from multiple procedural adjustments applied
influential columns of the projection matrix on top of the raw poses (Ilkka and Ville 2021),
WL ∈ ℝJL where L < J is extracted to project such as rotating the character’s head toward a
features to a lower-dimension coordinate system relevant object or enforcing that the character
by applying the truncated transformation Xpca ¼ doesn’t traverse a wall. Evaluating multiple
XWL. Despite the lousy projection, the projected sequences is too computationally expensive;
features still conserve most of the variance of the therefore, Inertialization (Bollo 2018), which
non-optimized features. Furthermore, a quantiza- only requires the evaluation of the new
tion of the feature values, for example, by using a sequence and often produces more natural tran-
reduced number of bits, can also accelerate the sitions, is preferred. This technique works by
speed; Büttner (2019) introduced such a scheme interpolating between the source and target
by encoding vectors with short codes. Finally, poses through an approximated smooth curve
Holden et al. (2020) showed how deep neural net- that considers the initial velocities of the source
works could approximate the motion matching sequence joints and estimates suitable
algorithm enabling constant inference time. accelerations.
Motion Matching: Data-Driven Character Animation Systems 1191

Procedural Adjustments the gaps. Finally, designers can mark valid frame
Motion matching is often limited by issues such as sequences in the data to remove duplicates, fix
foot sliding due to the blending of dissimilar poses poor acting, and customize the system’s runtime
or the feet going through the ground in complex behavior.
terrain. Adjusting the pose data of a found In addition to having high-quality data, it is
sequence to mitigate these artifacts and add other also important to generate query features that
effects depends on the specific requirements of correspond to the recorded features, or the
a game. matches will be of poor correspondence. Pose-
One approach to addressing issues with the conserving features can be used as-is, since they
feet is to mark each frame index k with two are already derived from previously played back
boolean values indicating whether the right or data. Control features, on the other hand, can be
left foot is in contact with the ground. When more challenging to craft, as they must approx-
playing a sequence in which a foot should be in imate natural human locomotion while being
contact, the system can use this information to generated from a game controller. One common
ensure that the foot reaches the ground and does approach is to simulate the evolution of a spring
not move too much horizontally (on non-slippery to generate natural and parameterized future
surfaces). To accomplish this, the system can trajectories. Holden (2021b) presents live exam-
adjust the position of the foot to stay on top of ples and rigorous derivations of such spring
the ground geometry beneath it. This can be simulation for games. The spring behavior can
achieved by performing a raycast, which be tuned through multiple parameters to match
involves shooting a virtual ray to detect the ter- the desired control scheme and generate appro-
rain geometry by finding the closest collisions. priate queries.
For a typical foot-knee-hip bone chain, the
adjustment can be made by finding the knee and
hip rotations that allow the foot to reach the Conclusion M
desired location. In general, this problem of find-
ing such rotations is known as Inverse Kinemat- This entry delves into the use of motion matching
ics (Aristidou et al. 2018). There is an efficient techniques in character animation systems for
closed-form solution for a two-joint kinematic video games. It presents a mathematical descrip-
chain in a fixed plane, and readers can refer to tion of motion matching, the main components of
Holden (2017) for an intuitive derivation. a motion matching-based animation system, and
practical guidelines followed by industry
Making Queries practitioners.
To be effective, the animations played back by Motion matching synthesizes responsive,
motion matching must be of high quality and smooth motion by selecting the most appropriate
include sufficient variations to cover a wide animation subsequence for the current game situ-
range of gameplay scenarios. Traditionally, ation and blending it with the currently playing
motion clips are recorded through motion capture, animation. The construction of the query space, a
either by having the actor perform a precisely retrieval space that separates potential starting
orchestrated choreography or by allowing them points of animation subsequences through
to move freely and randomly. Structured and featurization at each frame, is a key element of
unstructured motion clips can both be useful in the algorithm. By comparing recorded frame fea-
this context. Because of the vertical symmetry of tures with features constructed at runtime, it is
the human body, the data can be mirrored to possible to find the closest frame of animation
reduce the number of necessary takes by half in from which to start blending. Forming queries
well-planned sessions. However, it is difficult to that match the data and the prepared query space
anticipate and capture every possible transition, so is crucial to achieve satisfactory motion smooth-
unstructured takes can also be useful in filling in ness and responsiveness.
1192 Motion Planning in Computer Games

Those interested in learning more about Holden, D.: Code vs data driven displacement (2021a).
motion matching techniques can refer to https://theorangeduck.com/page/code-vs-data-driven-
displacement
(Bergamin 2021) for a thorough mathematical Holden, D.: Spring-it-on: The game developer’s spring-
definition, (Häfliger and Kurabayashi 2022) and roll-call (2021b). https://theorangeduck.com/page/
(Holden et al. 2020) for implementation details, spring-roll-call
and (Holden 2021a) for a standalone open-source Holden, D., Kanoun, O., Perepichka, M., Popa, T.: Learned
motion matching. ACM Transactions on Graphics.
implementation. 39(4), 1–13 (2020). https://doi.org/10.1145/3386569.
3392440
Hotelling, H.: Analysis of a complex of statistical variables
Cross-References into principal components. J Educ Psychol. 24(6),
417–441 (1933)
Häfliger, A., Kurabayashi, S.: Dynamic motion matching:
▶ Character Animation Scripting Environment Context-aware character animation with subspaces
ensembling. In: Proceedings of the 2021 IEEE Interna-
tional Symposium on Multimedia, (ISM ’21),
pp. 115–122. IEEE, New York (2021). https://doi.org/
References 10.1109/ISM52913.2021.00028
Ilkka, K., Ville, R.: Take ’control’ of animation. In: Game
Aristidou, A., Lasenby, J., Chrysanthou, Y., Shamir, A.: Developers Conference 2021, Informa PLC, (GDC
Inverse kinematics techniques in computer graphics: ’21) (2021). https://www.gdcvault.com/play/1026988/
A survey. Computer Graphics Forum. 37, 35–58 (2018) Animation-Summit-Take-CONTROL-of
Bergamin, K.: High-performance data-driven control of Michal, M., Zhuravlov, M.: Motion matching in ‘the last of
physically-based human characters. Master’s thesis, us part ii’. In: Game Developers Conference 2021,
McGill University (2021) Informa PLC, (GDC ’21) (2021). https://www.
Bollo, D.: Inertialization: High-performance animation tran- gdcvault.com/play/1027118/Motion-Matching-in-
sitions in ‘gears of war’. In: Game Developers Confer- The-Last
ence 2018, Informa PLC, (GDC ’18) (2018). https:// Pearson, K.: Liii. On lines and planes of closest fit to
www.gdcvault.com/play/1025331/Inertialization-High- systems of points in space. The London, Edinburgh,
Performance-Animation-Transitions and Dublin Philosophical Magazine and Journal of
Büttner, M.: Machine learning for motion synthesis and Science. 2(11), 559–572 (1901). https://doi.org/10.
character control in games. In: Symposium on Interac- 1080/14786440109462720
tive 3D Graphics and Games, ACM, (I3D ’19) (2019). Safonova, A., Hodgins, J. K.: Analyzing the physical cor-
https://i3dsymposium.org/2019/keynotes/I3D2019_ rectness of interpolated human motion. In: Proceedings
keynote_MichaelButtner.pdf of the 2005 ACM SIGGRAPH/Eurographics Sympo-
Büttner, M., Clavet, S.: Motion matching-the road to next sium on Computer Animation, pp 171–180. ACM
gen animation. In: Game Developers Conference 2015, (2005). https://doi.org/10.1145/1073368.1073392
Informa PLC, (GDC ’15). (2015)
Clavet, S.: Motion matching and the road to next-gen
animation. In: Game Developers Conference 2016,
Informa PLC, (GDC ’16) (2016). https://www.
gdcvault.com/play/1023280/Motion-Matching-and-
The-Road Motion Planning in Computer
Häfliger, A., Kurabayashi, S.: Dynamic motion matching: Games
Design and implementation of a context-aware anima-
tion system for games. International Journal of Seman-
tic Computing. 16(02), 189–212 (2022). https://doi.
Amol D. Mali
org/10.1142/S1793351X22400086 Computer Science Department, University of
Harrower, G.: Real player motion tech in’ea sports ufc 3’. Wisconsin-Milwaukee, Milwaukee, WI, USA
In: Game Developers Conference 2018, Informa PLC,
(GDC ’18) (2018). https://www.gdcvault.com/play/
1025228/Real-Player-Motion-Tech-in
Holden, D.: Simple two joint ik (2017). https:// Introduction
theorangeduck.com/page/simple-two-joint
Holden, D.: Character control with neural networks and Planning motions of characters is necessary in
machine learning. In: Game Developers Conference
some computer games because of several reasons.
2018, Informa PLC, (GDC ’18) (2018). https://www.
gdcvault.com/play/1025389/Character-Controlwith- Positions of existing obstacles or opponents
Neural-Networks change, making predefined paths unsuitable.
Motion Planning in Computer Games 1193

New obstacles may be introduced in the gaming indirect or delayed or immediate reward and
environment unpredictably, making motion plan- punishment.
ning necessary. Position of goal, like the treasure Navigation mesh: A navigation mesh is
to be seized, may change unpredictably, making obtained by partitioning the free space into con-
motion planning indispensable. Even if it is pos- vex polygons or polyhedrons, commonly referred
sible to predefine paths that are guaranteed to to as cells. A collision-free path can then be found
remain collision-free in some games, motion plan- by searching a graph capturing connectivity of
ning is needed to synthesize diverse paths to avoid these cells and joining subpaths contained in
predictability and unnatural motion. Some games these cells.
have too many moving characters to predefine Roadmap: In the context of motion planning,
paths for, and motion planning is important for a roadmap is a graph such that each of its vertices
them even if they are nonplaying characters is a configuration of a moving agent like a point
because they enhance player experience. Even if robot and each edge represents transition from one
there are very few moving characters, there may configuration to another through movement. If a
be too many combinations of motion constraints roadmap is connected and is stored for future use,
to be satisfied at different times, making pre- a path from an initial configuration to goal con-
computation of motion plans infeasible. Other figuration can be found by connecting each of
reasons for motion planning include expansion them to some vertex in the roadmap, instead of
of the gaming environment enabled by portals, finding from scratch.
different speeds of characters, and multiple loco- Waypoints: They are points between start and
motion modes of characters. In some computer goal configurations used to guide an agent’s
games, the challenge lies in deciding where to navigation.
move next and not how to move there. In some Voronoi diagram: Given a two-dimensional
computer games, the challenge lies in finding how shape and a set of n points called seeds or sites, its
to move to goal and not selecting goal. Some of Voronoi diagram is a collection of points
M
the references in this article do not describe a partitioning the shape into n regions such that all
computer game, but some ideas in them are usable points in each region are at least as close to one
in computer games. The emphasis of this article is seed as any other seed.
on representative heuristics, strategies, tactics, Generalized Voronoi diagram (GVD): Given
challenges, geometric and logical representations, a shape S with shapes inside it, like a polygonal
types of constraints to be satisfied, and evaluation boundary with polygons within it, the GVD of S is
metrics in motion planning in computer games the collection of points such that each of these
and not on physics of motion and algorithmic points is equidistant from at least two shapes in
details. S closest to it.
Medial axis: The medial axis of a shape is the
set of centers of largest balls that can fit inside the
Terminology shape such that each ball meets the boundary of
the shape only tangentially at least two points.
Degrees of freedom: The number of degrees of The medial axis changes if the object is
freedom (DOFs) of a moving agent is the mini- deformed.
mum number of parameters whose values need to Motion clip: It is a short video containing
be specified to completely specify the position and motion of one or more objects.
orientation of each part of the agent with respect to Motion graph: A vertex of a motion graph is a
a fixed coordinate system. motion clip or component of a motion clip. An
Desirability map: It shows the desirability of edge of this graph represents that a transition from
visiting individual locations in a gaming world. one clip to another is possible, making compound
The desirability is usually computed by combin- motion possible by joining motion clips
ing the values of attributes related to direct or connected by the edge. The graph can have cycles.
1194 Motion Planning in Computer Games

Motion database: A motion database used for connected by 18 joints. The object carried by the
computer animation contains motion clips. mannequin has 6 degrees of freedom. Since it is
Inverse kinematics: It is a process which finds extremely hard to find a collision-free path for a
positions and orientations of parts of a moving system with 59 degrees of freedom, the system is
agent which ensure that a specific subsystem of first treated as a 9-DOF system by constraining the
the agent (usually the end effector like a gripper) configurations of the mannequin and the object.
is in the desired configuration. Once a collision-free path is found, the frozen
Sampling-based motion planning: It is a par- parts of the system are animated in a collision-
adigm which involves creating samples free manner to create a natural motion.
(geometric configurations) of the moving agent, The path planning approach in (Juarez-Perez
randomly or with a bias, and connecting them to and Kallmann 2018) associates a clearance, pref-
create a roadmap which is used for finding a erence, and cost with each behavior. They con-
collision-free path. sider three behaviors – (i) frontal walking,
(ii) constrained frontal walking, and (iii) lateral
walking or sidestepping. Arms are relaxed during
Motion Planning for One Agent frontal walking, but this behavior needs higher
torso clearance. Frontal walking is the most pre-
Motion planning for one agent is relevant when ferred behavior and costs the least. Constrained
there is only one moving agent or there are mul- frontal walking needs lower torso clearance, costs
tiple agents that move fully independent of each more, and is less preferred. Lateral walking is the
other without having to worry about collisions least preferred behavior among the three, costs the
among them, but collisions with nonmoving most, but needs the least clearance, making it
objects are to be avoided. Kovar and others indispensable for navigation through narrow pas-
(Kovar et al. 2002) introduced motion graphs for sages. For efficiency, their approach doesn’t check
efficient composition of motions to create com- for collisions between certain obstacles (like short
pound motions. Rahim and others (Rahim et al. obstacles) and torso of the moving character since
2009) report on creation of motions for animated such collisions are impossible. Their layered
characters using motion graphs derived from approach initially finds a collision-free path
motion clips from a motion database. These using the costliest behavior (sidestepping) which
motion clips contain running, walking, jumping, is also the least preferred behavior and needs the
and other types of movement. They review dis- least clearance. The layered approach then tries to
tance metrics used to find similar poses in clips to reduce the path cost by improving the initial path
find transitions possible between motion clips or using more preferred behaviors which also cost
their components, to establish the edges of a less and need higher clearance. The path planning
motion graph. The virtual agent in (Rashidian approach in (Kapadia et al. 2013) handles hard
et al. 2014) has to visit at least one goal from and soft constraints. Each soft constraint has a
each of the k groups of goals, starting from the weight showing how important its satisfaction
given initial configuration in presence of static is. Satisfaction of hard constraints is mandatory.
obstacles. This is referred to as multigroup motion Satisfaction of soft constraints is optional. Some
planning in (Rashidian et al. 2014). Their of the nodes and edges in their hybrid state-
approach includes finding a tour for the virtual transition graph are obtained by partitioning the
agent without considering obstacles and using free space into triangles in a specific way. Some
sampling-based motion planning to consider nodes and edges in this graph are obtained by
obstacles. Their experiments included a ground laying a uniform grid over the free space such
vehicle and an aerial vehicle. The digital manne- that a node in the grid has up to eight neighbors.
quin in (Arechavaleta et al. 2004) has 53 degrees Some nodes and edges in this graph are obtained
of freedom. It is composed of 20 rigid bodies by laying triangles over the free space for
Motion Planning in Computer Games 1195

representing spatial relations satisfying spatial a combo of an environment and collision-free

constraints on a path. Examples of spatial rela- motion, significantly reducing or eliminating the
tions include an agent being on the right or left of computational complexity of motion planning
an object or in the line of sight of a patrolling because motion planning is then needed only for
guard. Nodes and edges added to the hybrid characters or environment not based on motion
state-transition graph because of triangular cells clips. It is possible to compose character motions
and their adjacency allow faster synthesis of paths and then include nonmoving objects in a manner
and reduction in path length. Nodes and edges that avoids motion conflicts.
added to the graph by uniform grid allow synthe-
sis of paths satisfying diverse constraints.
It is pointed out in (Lee et al. 2006) that cap- Motion Planning for Adversaries
turing motion data in a physical environment and
adapting it to a virtual environment is a bottleneck The motion planning approach suitable when
in computer animation. This is because creating or there are adversaries depends on their count, sep-
renting complex/specialized physical environ- aration distance possible, speeds, directions of
ments and having humans perform various actions movement, and other constraints like staying vis-
in them and creating digital approximations of ible to nonplaying agents, if any. Park and others
real environments and real characters can be (Park et al. 1997) present a computer game
time-consuming and pricey. They propose a containing two boxing robots such that one of
data-intensive scalable approach in which a com- them is controlled by the human player and the
plex virtual environment is composed of small other is computer-controlled. The human player’s
units and the motions in this environment are arm movements are tracked with 3D position sen-
composed of the motions associated with these sors attached to wrists and elbows. The motion
small units, e.g., a large playground can be created planner for the computer-controlled boxer finds
by populating it with curved slides and straight offensive motions to hit the human-controlled
M
slides, and complex motion in this environment boxer and defensive motions to prevent being hit
can be composed of walking, climbing, and slid- on its hittable surface. The motion planner sam-
ing motions. A building block (also known as a ples the position of fist of the human-controlled
motion patch) in their work consists of an boxer at a fixed frequency and uses it along with
environment-building unit like a curved slide passage of fixed time without a punch from the
and specific motions available with it. Motions human-controlled boxer to decide whether to be
available in an environment-building unit are offensive or defensive. Hittable areas of each
reusable in a copy of it. Their approach is usable robot are two predefined rectangles which are
for multiple agents too. They report that an office discretized assuming that there are finite hit
environment can be generated using a desk-chair points. Blocking a punch with a forearm is the
combo and square ground panel. The motion only defense strategy. The motion planner com-
patches for an office environment they identified putes the ratio of the distance from the defending
based on 40 minutes of motion are (i) sitting down forearm of the human-controlled boxer and the
or standing up, (ii) working at a desk, (iii) chatting distance from the hitting fist of the robot-
at a desk, (iv) standing idle at a place away from controlled boxer for individual exposed hittable
all desks, (v) chatting at a place away from all points on the human-controlled boxer and targets
desks, (vi) disputing at a place away from all the point with highest ratio during offense. Inverse
desks, (vii) making a presentation while standing kinematics is used for motion planning.
away from all desks, and (viii) walking and then Shamgah and others (Shamgah et al. 2016)
stopping. Instead of generating an environment present an abstract model in which the attacking
with obstacles and then finding collision-free robot tries to move to reach its target cell in a grid
paths in it, the approach in (Lee et al. 2006) offers with the defending robot trying to move to capture
1196 Motion Planning in Computer Games

the attacking robot. The number of strategies for The crowd simulation model in (Liu and Chen
each player in this reach-avoid game is the num- 2008) allows different types of agents. Their imple-
ber of cells it can move to next. The payoff for the mentation has many birds, deer, and tigers. Besides
defender for strategy combination (i,j) is e(ai, t)/e having different speed and acceleration, they exhibit
(ai, dj) where function e is the Euclidean distance different behaviors. Birds fly or rest. Deer wander,
between its arguments, ai is the cell in which the march toward goal, or evade tigers. Tigers wander,
attacker will end up by following strategy i (same trace deer, and attack. There is a finite-state machine
as moving to cell i), t is the target cell the attacker (FSM) for each kind of animal. The FSMs specify
is trying to reach, and dj is the cell in which the conditions for transition between behaviors.
defender will end up by following strategy j (same The main objective of the crowd simulation
as moving to cell j). The attacker wants to move to approach in (Chang and Li 2008) is maintaining
minimize this payoff. The defender wants to move the shape of the crowd, e.g., a square or convex
to maximize this payoff. This model can be real- nonagon. Opposing troops in a battle can form
ized in computer games in multiple ways, e.g., different shapes. Their approach first generates a
having one attacker and one defendant with mul- path for the shape representing the crowd. If
tiple target cells that vanish if not reached in cell- needed, the shape is deformed to avoid obstacles.
dependent time, having multiple attackers and one Individual agents exhibit different behaviors
defendant, having one attacker and multiple within the shape, e.g., changing or maintaining
defendants, and allowing an attacker to be cap- speed or orientation or staying still. A gamified
tured a certain number of times. Instead of gener- approach to crowdsourced design of floor plans is
ating the payoff matrix with all strategy presented in (Chakraborty et al. 2017). Human
combinations to guide the movement of players don’t plan motions of characters. They
computer-controlled characters, a partial payoff design floor plans by choosing the locations and
matrix with randomly chosen strategy combina- numbers of doorways, walls, and pillars, starting
tions can be used. This will not only be computa- with the layout provided by an architect or
tionally efficient, but it will also often result in designer. A player can change density of crowd
suboptimal choices for computer-controlled char- and distribution of agent speed, acceleration, and
acters which will make them more believable and agent width and height. After designing the floor
boost the morale of the human player by increas- plan, a human player can run simulation which
ing his/her chance of winning. ends after evacuation of the environment is com-
plete. Players are ranked based on evacuation
time. This work can be viewed as a way to verify
Motion Planning for Crowds that motion planning and execution meeting cer-
tain constraints is possible in the given environ-
Li and others (Li et al. 2017) present an ment. By running simulations with different
implemented and evaluated hybrid approach to parameters, one can find how motion-friendly an
model motions of agents in a crowd. In an agent- environment is. Some approaches to crowd simu-
based approach to crowd simulation, each agent is lation involve guiding the moving characters
autonomous. In cellular-automaton approach to along medial axis or the edges of a Voronoi dia-
crowd simulation, agents have no autonomy at all. gram or a GVD or some other type of roadmap.
In the hybrid approach in (Li et al. 2017), each agent Though this can result in slow movement of
is autonomous and has a positive tolerance value characters due to waiting and make motion
which is the number of time steps for which it can look unnatural and predictable, it makes
wait in its current cell before moving to the next cell. collision avoidance easier. The approach in
When multiple agents want to move to the same (Oliva and Pelechano 2013) dynamically gener-
cell, the agent which waited the longest in its current ates waypoints for characters with different radii
cell has the highest probability of moving to the cell. to avoid these problems.
Motor Disability 1197

Conclusion Liu, Y., Chen, L.. A group behavior model of real-time

crowds in virtual environments. In: Proceedings of
IEEE International Conference on Robotics, Auto-
Motion planning in computer games has used mation, and Mechatronics (RAM), pp. 866–872.
some of the advances in robot motion planning, (2008)
computational geometry, computer animation, Oliva, R., Pelechano, N.: A generalized exact arbitrary
and search in artificial intelligence. Still, high clearance technique for navigation meshes. In: Pro-
ceedings of ACM SIGGRAPH Conference on Motion
computational complexity associated with large in Games (MIG), 2013. pp. 81–88. Dublin, Ireland
state spaces and loss of completeness and opti- (2013)
mality typically associated with scalable motion Park, S., Hwang, Y., Lee, S., Kang, S., Cho, K., Han, Y.,
planning techniques is inherited by computer Kim, M., Lee, C.: Human-computer competition in
game situation: Motion planning for boxing. In: Pro-
games using these techniques. There is scope for ceedings of IEEE/RSJ Conference on Intelligent
research on modifying techniques and representa- Robots and Systems (IROS), pp. 134–139. Grenoble,
tions used in robot motion planning and search in France (1997)
artificial intelligence, using the freedom to include Rahim, R., Suaib, N., Bade, A.: Motion graph for charac-
ter animation: Design considerations. In: Proceedings
fiction in computer games. of IEEE International Conference on Computer Tech-
nology and Development (ICCTD), pp. 435–439.
(2009)
References Rashidian, S., Plaku, E., Edelkamp, S.: Motion planning
with rigid-body dynamics for generalized traveling
salesman tours. In: Proceedings of the 7th International
Arechavaleta, G., Esteves, C., Laumond, J-P.: Planning ACM SIGGRAPH Conference on Motion in Games
fine motions for a digital factotum. In: Proceedings of (MIG), pp. 87–96. ACM, New York (2014)
IEEE/RSJ International Conference on Intelligent Shamgah, L., Karimoddini, A., Homaifar, A.: A symbolic
Robots and Systems (IROS), pp. 822–827. (2004) motion planning approach for the reach-avoid problem.
Chakraborty, N., Haworth, B., Usman, M., Berseth, G., In: Proceedings of IEEE International Conference on
Faloutsos, P., Kapadia, M.: Crowdsourced co-design Systems, Man, and Cybernetics (SMC), 2016.
of floor plans using simulation-guided games. In: Pro- pp. 3955–3960. Budapest, Hungary (2016)
M
ceedings of ACM SIGGRAPH Conference on Motion
in Games (MIG), pp. 1:1–1:5. ACM, New York (2017)
Chang, J-Y., Li, T-Y.: Simulating virtual crowd with fuzzy
logics and motion planning for shape template. In:
Proceedings of IEEE Conference on Cybernetics and
Intelligent Systems (CIS), pp. 131–136. (2008)
Juarez-Perez, A., Kallmann, M.: Fast behavioral locomo- Motion Tracking
tion with layered navigation meshes. In: Proceedings of
ACM SIGGRAPH Symposium on Interactive 3D ▶ Tracking Techniques in Augmented Reality for
Graphics and Games (i3D), pp. 8:1–8:6. (2018) Handheld Interfaces
Kapadia, M., Ninomiya, K., Shoulson, A., Garcia, F.,
Badler, N.. Constraint-aware navigation in dynamic
environments. In: Proceedings of ACM SIGGRAPH
Conference on Motion in Games (MIG), pp. 89–98.
Dublin 2, Ireland. (2013)
Kovar, L., Gleicher, M., Pighin, F.: Motion graphs. In:
Proceedings of the 29th Annual Conference on Com-
Motivation
puter Graphics and Interactive Techniques
(SIGGRAPH), pp. 473–482. San Antonio, Texas ▶ Videogame Engagement: Psychological
(2002) Frameworks
Lee, K., Choi, M., Lee, J.: Motion patches: Building blocks
for virtual environments annotated with motion data.
ACM Trans. Graph. 25(3), 898–906 (2006)
Li, T., Yao, Y., Tang, W., Yao, F.: A hybrid pedestrian
motion modeling approach for crowd simulation. In: Motor Disability
Proceedings of 10th IEEE International Symposium on
Computational Intelligence and Design (ISCID),
pp. 103–107. Hangzhou, China (2017) ▶ Computer Games for People with Disability
1198 Movie-Making of Spatiotemporal Dynamics in Complex Systems

Introduction
Movie-Making of
Spatiotemporal Dynamics in This entry describes the development and use of
Complex Systems the art of movie-making in order to display spa-
tiotemporal dynamics in complex systems such
Jeffery Jonathan Joshua (‫ )ישוע‬Davis1 and as brains. This qualitative tool has been used in
Robert Kozma2 visualizing, analyzing, and understanding brain
1
The Embassy of Peace, Whitianga, New Zealand dynamics as a new form of the art of encephalog-
2
CLION, Department Mathematical Sciences, raphy (Davis et al. 2015; Davis and Kozma 2013).
University of Memphis, Memphis, TN, USA This tool has also been applied in order to visual-
ize individual heart rate variability (HRV) in
group dynamics in the study of psychophysiolog-
Synonyms ical coherence and its relationship to inner peace
and social harmony (Davis et al. 2018; Heart
The art of electroencephalography Coherence Ratio per Participant 2018). It would
be possible, in principle, to apply this method to
study any complex system that can be described as
Definitions an array of signals that comprises a region of a
system, which can be described as a whole in
Electroencephalography, or EEG, is an electro- itself, for example, regions of the visual cortex.
physiological monitoring device comprised of This comes as a new and more advanced way to
multiple electrodes (small, flat, metal discs with display images with the purpose of visual discrim-
thin wires) placed on the scalp that send signals to ination associated with the different brain cogni-
a computer in order to noninvasively measure and tive states or heart coherent states, for example, in
record electrical activity on the scalp. EEG can be order to better understand and formulate theories
used in cognitive research or to diagnose condi- of different stages of cognitive processes related to
tions such as epilepsy and sleep disorders. the creation of knowledge and meaning, inten-
Biofeedback systems are computerized tional action, and value-based decision making
devices that provide information about the activ- (Davis and Kozma 2013; Davis et al. 2015a;
ity of physiological measures, such as heart rate Kozma and Davis 2018). The movies allow us to
variability, in order to learn how to modulate at visualize different patterns of behavior in different
will specific body functions to improve energy conditions produced by different stimuli based
management, health, self-mastery, and general on experimental data. By careful observation of
well-being. Biofeedback systems may include each of these movies, the researcher learns to
EMG machines, ECG, heart rate variability, res- identify different structures and visual patterns
piration, and EEG among others. where, for example, large-scale synchronizations
Heart rate variability (HRV) is a measure of and desynchronizations can be observed across
the patterns prescribed by inter-beat intervals of different frequency bands (Davis et al. 2013;
time and the functioning of the heart. HRV has Myers et al. 2014).
been described as a psychophysiological bio-
marker to assess coherent or stressful states asso-
ciated with respiration, cognition, and emotions Overview of the Methodology
(McCraty et al. 2009).
Psychophysiological coherence has been A thorough description of this methodology, par-
widely described as a state conducive to optimal ticularly applied to brain data, has been presented
cognitive performance, improved health and also in previous work (Davis et al. 2015) together with
associated to inner balance, peace, and harmony an introduction to the different costs and benefits
(McCraty et al. 2009). associated to it in terms of time consumption (hours
Movie-Making of Spatiotemporal Dynamics in Complex Systems 1199

of work). Generally speaking, in order to more Working Time

efficiently produce a greater amount of movies 1000
800

Time (hours)
from brain data, for example, the more man and
computer power is required. However, with a fair 600
amount of people, like a team of two (2), and two 400
(2) relatively powerful laptops, a considerable 200
amount of movies can be made. It is important to 0
note that in “The Art of Encephalography to 1 2 3 4 5 6 7 8 9 10
Understand and Discriminate Higher Cognitive Steps
Functions Visualizing Big Data Imaging Using
Movie-Making of Spatiotemporal Dynamics in Com-
Brain Dynamics Movies” (Davis et al. 2015), plex Systems, Graph 1 Working time per step in the
39 experiments (runs) in 4 bands for 4 indices movie-making methodology
(signals), for a total of 624 individual movies,
were produced for a total work time of around identify, in the case of brain dynamics, the different
1623 h, which equates to around 10 or 5 months stages in the manifestation of, for example, a
(1623 h) with 1 or 2 fulltime analysts, respectively, hypothesized cycle of creation of knowledge and
each working 5 days a week for 8 h a day. To meaning. These types of studies have been
illustrate this, it is important to note that the meth- conducted both in animals and humans to capture
odology is comprised of ten (10) steps as follows: brain dynamics in response to salient stimuli (Davis
and Kozma 2012; Kozma et al. 2012), as well as the
Step 1: Signal preprocessing and movie genera- study of brain dynamics in different cognitive states
tion (~52 h) (Davis et al. 2016; Davis et al. 2017).
Step 2: Movie downloads to computer (~4 h)
Step 3: Organization and editing of projects Movie-Watching and Pattern Recognition
(~13.3 h) It is conjectured that movie viewing will allow a
M
Step 4: Synchronization of different runs or better understanding of learning, adaptation, and
experiments into one movie that displays cognition in general, by allowing researchers to
them all at the same time (~960 h) familiarize themselves with very distinct patterns
Step 5: Corrections and art editing (~96 h) of the behavior of a complex system, like, for
Step 6: Rendering and further editing (~224 h) example, group coherence and synchronization
Step 7: Exporting edited movie (~128 h) between individuals in different group activities
Step 8: Finalizing corrections and speed calibra- (McCraty 2017; McCraty et al. 2017; Timofejeva
tion (~8 h) et al. 2017). In the case of cognition and con-
Step 9: Final rendering (~42 h) sciousness, many of the philosophical implica-
Step 10: Exporting and publishing final movie to tions associated with intentional behavior and
website (~96 h) value-based decision-making that have been
published (Davis et al. 2015a; Davis et al. 2017;
The distribution of the workload across all steps Kozma and Davis 2018) highlighted the value of
is presented in Graph 1. Note that, Step 4 together using brain dynamics movies in the uncovering of
with Step 6 are the major time-consuming activities different cognitive states and the cycle of creation
requiring manpower skills, and therefore, in order of knowledge and meaning.
to cut that time, more people, computers, software,
and working space are needed. Further Comments on the Art of
Encephalography
Movie-Making and the Art of In summary, it can be said that viewing brain
Encephalography via EEG Measurements dynamics movies will allow a significant impres-
These movies allow researchers to explore the tem- sion of brain events for different measurements,
poral evolution of spatial patterns where they can brain events across bands and the different stages
1200 Movie-Making of Spatiotemporal Dynamics in Complex Systems

of the cycle of creation of knowledge and mean- One can imagine every element of this matrix
ing, as well as the design of better monitors and to be a pixel or a point to be displayed in a surface
biofeedback systems that will have to meet the plot. If one assigns a color to each point and
challenge of efficiently processing large amounts displays the surface in each time step (t), then
of data online. With that in mind, many the color patterns may reveal an orderly move-
researchers interested in the study of conscious- ment. When this set of surface plots is captured as
ness and cognition, for example, will be able to frames of a movie and they are played at a certain
discuss their insights after watching brain dynam- speed, then the emergence of a spatiotemporal
ics movies in a set of iterative brainstorming ses- pattern may be observed. An example could be a
sions. One can imagine and be positive that this simulated set of signals that could represent elec-
kind of activity and approach will allow the emer- trodes implanted on the cortex of an epileptic
gence of better theories about brain functions, patient, in order to measure brain activity via
cognition, and consciousness in general. It is ECoG. The following fictional example portrays
important to note that this way of visualizing a square array of 1212 as if it was placed
brain dynamics has been inspired by the work of on the cortex of a patient with epilepsy, similar
Walter J. Freeman who was a great pioneer and to the ones described in recent studies (Heck
luminary of systems neuroscience. et al. 2016). The simulated patterns that emerge
could look like the following sequence of surface
plots in different moments of simulated data
Methodology and Applications plotting amplitude values in the range of [0,10],
as in Fig. 1.
Usually complex systems are represented as a set Similar to the above simulated signals, recent
of signals or time series which share a certain studies using this methodology have shown that
correlational or causal structure, and when viewed spatiotemporal brain dynamics could be tracked
independently as a collection of plots over the in terms of amplitude modulation patterns that
same time axes, a substantial amount of informa- could reveal responses to stimuli (Davis and
tion related to the spatial configurations and pat- Kozma 2013; Davis et al. 2013). This same meth-
terns that arise in each moment or time step (t) is odology could be applied to brain dynamics mea-
lost. In order to capture the spatiotemporal sured on the scalp of a human, where the
dynamics, the need arises to display the signals, electrodes are placed all over the scalp and their
xi(t) with i ¼ 1,2,. . .N2 in a new configuration, as associated signals are displayed for each point in
follows: the matrix, in each time step (t). In recent studies,
the dominant frequency band (theta, alpha, beta,
or gamma) has been studied under different cog-
nitive modalities and these kinds of movies were
used to track changes in the dominant frequency
band (representing brain dynamics) for each part
of the cortex (Davis et al. 2016, 2017). Figure 2
depicts two (2) moments where the change in the
dominant frequency band can easily be appreci-
ated for the arrangement of electrodes placed on
the scalp that were obtained in the study just cited
before (Davis et al. 2017). The displays show a
spatiotemporal landscape related to one partici-
pant, in one particular time, in one particular
modality (EEG Dunedin Movies 2018).
Similar movies can be developed based on
studies of group dynamics using the heart rate
Movie-Making of Spatiotemporal Dynamics in Complex Systems 1201

Movie-Making of Spatiotemporal Dynamics in Com- 4 different consecutive times and the evolution of the
plex Systems, Fig. 1 Simulated signal amplitudes system for t ¼ 0 (top left), t ¼ 1 (top right), t ¼ 2 (bottom
(144) placed in an imaginary 1212 array showing left) and t ¼ 3 (bottom right)

variability (HRV) or a coherence ratio measure of Davis and Kozma 2018). Studies conducted in
each participant as the signal to be displayed in recent years using this methodology have shown
each time step (t), showing a group spatiotemporal that brain dynamics associated with intentional
landscape as shown in (Davis et al. 2018; Heart behavior have their foundation and are deeply
Coherence Ratio per Participant 2018) (Fig. 3). rooted in the creation of knowledge and meaning
It is important to note that these kinds of stud- that takes place in mind-brain dynamics (Davis
ies with the aid of complex systems movies, like et al. 2015a; Davis 2018; Kozma and Davis 2018).
brain dynamics movies, could assist in the under-
standing of consciousness associated to different
cognitive states, as in meditation and relaxation, Conclusions and Future Perspectives
or energy-consuming activities like reading diffi-
cult material, revealing different patterns in both A methodology has been presented for displaying
brain and HRV dynamics (Davis et al. 2017; signals related to complex systems like brains and
1202 Movie-Making of Spatiotemporal Dynamics in Complex Systems

EEG Frequency per channel in 1s window EEG Frequency per channel in 1s window
second: 51 second: 57
45 45

40 40
45 45
40 35 40 35
35 35
30 30 30 30
25 25
20 25 20 25
15 15
20 20
10 10
5 15 5 15
0 0
ch 1 ch 1
10 10
ch 6 ch 25 ch 6 ch 25
ch 11 ch 24 5 ch 11 ch 24 5
ch 23 ch 23
ch 16 ch 22 ch 16
0 ch 22 0
Left side channels ch 21 ch 21 Back channels Left side channels ch 21 ch 21 Back channels

Movie-Making of Spatiotemporal Dynamics in Com- electrodes are anchored in zero (red), showing two (2) dif-
plex Systems, Fig. 2 Displays of the dominant fre- ferent moments in the evolution of human brain dynamics
quency band for 19 electrodes that cover the scalp and for t ¼ 51 s (left) and t ¼ 57 s (right)
are arranged in a 55 array where six (6) dummy

Movie-Making of Spatiotemporal Dynamics in Com- (23, 24 and 25) are left in zero (0). Left: Baseline activity
plex Systems, Fig. 3 Display of the HRV coherence around 8:36 a.m. (low group coherence) and Right:
ratio for twenty two (22) participants in two (2) different A group meditation activity around 6:20 p.m. (high group
activities over an array of 55 where three (3) positions coherence)

communities among others, where the displays quality movies, very close to online times biofeed-
are turned into movies based on sequential frames, back system displays. These kinds of new tech-
where the evolution and dynamics of the system nologies could facilitate an efficient and
can be qualitatively studied in space and time. The comfortable qualitative interaction between com-
challenges associated with the production of puters and humans through brain-computer inter-
these movies are various, requiring efficient big- faces, where humans can presumably learn to
data processing algorithms and computer tech- modulate their own internal cognitive and psycho-
nologies that minimize the latencies between physiological states and in so doing, minimize
frame displays. This will ideally allow good stress and improve well-being. The use of this
Movie-Making of Spatiotemporal Dynamics in Complex Systems 1203

methodology can be foreseen in the study of cog- Davis, J.J., Kozma, R., Freeman, W.J.: Neurophysiological
nition and consciousness by increasing the exper- evidence of the cognitive cycle and the emergence of
awareness. Paper presented at International Joint Con-
imental database that is derived from EEG ference on Awareness Science and Technology and
measurements on the scalp of human participants Ubi-Media Computing (iCAST-UMEDIA), Aizu-
under different modalities presumably leading to Wakamatsu, Japan, 2–4 Nov 2013
different cognitive states. It also can be Davis, J.J.J., Kozma, R., Freeman, W.J.: The art of enceph-
alography to understand and discriminate higher cog-
conjectured that with a careful visual and statisti- nitive functions visualizing big data imaging using
cal analysis (qualitative and quantitative), brain dynamics movies. Procedia Comp. Sci. 53(1),
researchers will be able to uncover the intrinsic 56–63 (2015)
relationship between signals associated to respi- Davis, J.J, Kozma, R., Lin, C-T, Freeman, W.J.: Spatio-
temporal EEG pattern extraction using high-density
ration and brain-heart dynamics. It has been scalp arrays. Paper presented at The 2016 International
shown that human beings can generate psycho- Joint Conference on Neural Networks (IJCNN), Van-
physiological states which are reflected in HRV couver, 24–29 July 2016
and that are conducive to cognitive clarity and Davis, J.J.J., Lin, C.-T., Gillett, G., Kozma, R.: An integra-
tive approach to analyze EEG signals and human brain
stress reduction (Heck et al. 2017). It is likely dynamics in different cognitive states. J. Artif. Intell.
that in the future, the use of this methodology in Soft Comput. Re. 7(4), 287–299 (2017)
the implementation of biofeedback systems will Davis, J.J.J., Schübeler, F. Kozma R.: Heart rate variability
assist human beings in the mastery of inner peace, dynamics and its implications for individual psycho-
physiological coherence in community dynamics while
general well-being, and social harmony, where in Meditation or other presumably beneficial activities.
ideally one will be able to self-monitor heart- Open Science Framework. https://osf.io/rywq4/
brain-respiration and other physiological signals (2018). Accessed 06 Apr 2018
almost in online time (minimal latency) and all EEG Dunedin Movies. The Science of Peace website.
https://thescienceofpeace.weebly.com/experiments%
signals at the same time. 2D%2Dresearch1.html. Accessed 01 Apr 2018
Heart Coherence Ratio per Participant (First HRV Paper).
The Science of Peace website. https://thescien
M
ceofpeace.weebly.com/experiments%2D%2Dresearch.
References html. Accessed 01 Apr 2018
Heck, D.H., McAfee, S.S., Liu, Y., Babajani-Feremi, A.,
Davis, J.J.: Pragmatic information, Intentionality & Con- Rezaie, R., Freeman, W.J., Wheless, J.W.,
sciousness. J. Conscious. Explor. Res. 9(2), 113–123 Papanicolaou, A.C., Ruszinkó, M., Kozma, R.: Corti-
(2018) cal rhythms are modulated by respiration. BioRxiv.
Davis, J.J.J., Gillett, G., Kozma, R.: Revisiting Brentano https://www.biorxiv.org/content/early/2016/04/16/
on consciousness: striking correlations with Electrocor- 049007 (2016). Accessed 28 Mar 2018
ticogram findings about the action-perception cycle and Heck, D.H., McAfee, S.S., Liu, Y., Babajani-Feremi, A.,
the emergence of knowledge and meaning. Mind Mat- Rezaie, R., Freeman, W.J., Wheless, J.W.,
ter. 13(1), 45–69 (2015a) Papanicolaou, A.C., Ruszinkó, M., Sokolov, Y.,
Davis, J.J., Kozma, R.: Analysis of Phase Relationship in Kozma, R.: Breathing as a fundamental rhythm of
ECoG using Hilbert Transform and Information Theo- brain function. Front. Neural Circuits. 10(115),
retic Measures. Paper presented at The 2012 Interna- (2017). https://doi.org/10.3389/fncir.2016.00115
tional Joint Conference on Neural Networks (IJCNN), Kozma, R., Davis, J.J.J.: Why do phase transitions matter
Brisbane, 10–15 June 2012 in minds. J. Conscious. Stud. 25(1–2), 131–150 (2018)
Davis, J.J., Kozma, R.: Creation of Knowledge & Meaning Kozma, R., Davis, J.J., Freeman, W.J.: Synchronized min-
Manifested via Cortical Singularities in Cognition. ima in ECoG power at frequencies between Beta-
Paper presented at The 2013 I.E. Symposium Series gamma oscillations disclose cortical singularities in
on Computational Intelligence (SSCI) Cognitive Algo- cognition. J. Neurosci. Neuroeng. 1(1), 13–23 (2012)
rithms, Mind, and Brain (CCMB), Singapore, 16–19 McCraty, R.: New Frontiers in heart rate variability and
Apr 2013 social coherence research: techniques, technologies,
Davis, J.J.J., Kozma, R.: Visualization of Human Cogni- and implications for improving group dynamics and
tive States Monitored by High-density EEG Arrays. outcomes. Front. Public Health. 5(267), (2017).
Paper presented at The 3rd International Neural Net- https://doi.org/10.3389/fpubh.2017.00267
work Society Conference on Big Data and Deep Learn- McCraty, R., Atkinson, M., Stolc, V., Alabdulgader, A.A.,
ing (INNS BDDL), Sanur – Bali, Indonesia, 17–19 Apr Vainoras, A., Ragulskis, M.: Synchronization of human
2018 [in press] autonomic nervous system rhythms with geomagnetic
1204 MR

activity in human subjects. Int. J. Environ. Res. Public

Health. 14(7), 770 (2017). https://doi.org/10.3390/ Multiplayer Game
ijerph14070770
McCraty, R., Atkinson, M., Tomasino, D., Bradley, R.T.:
The coherent heart: heart–brain interactions, psycho- ▶ Disney Toontown Online, a Massively Multi-
physiological coherence, and the emergence of system- player Online Role-Playing Game
wide order. Integral Rev. 5(2), 10–115 (2009) ▶ Fantasy XVI Online, a Massively Multiplayer
Myers, M.H., Kozma, R., Davis, J.J.J., Ilin, R.: Phase cone
detection optimization in EEG data. Paper presented at Online Role-Playing Game
The 2014 International Joint Conference on Neural ▶ Overwatch: Team-Based Multiplayer First-Per-
Networks (IJCNN), Beijing, 6–11 July 2014 son Shooter Game
Timofejeva, I., McCraty, R., Atkinson, M., Joffe, R., ▶ World of Tanks, MMO Strategy Freemium
Vainoras, A., Alabdulgader, A.A., Ragulskis, M.: Iden-
tification of a Group’s physiological synchronization Game
with Earth’s magnetic field. Int. J. Environ. Res. Public
Health. 14(9), 998 (2017). https://doi.org/10.3390/
ijerph14090998
Multiplayer Games

▶ Sociality of Digital Games

MR

▶ Interaction with Mobile Augmented Reality

Environments Multiplayer Online Game,
MOG

▶ Peer-to-Peer Gaming
Multicolor Rainbow Hologram

▶ Holography as an Architectural Decoration

Multiplayer Online Gaming
Architecture

Multiculturalism ▶ Online Gaming Architectures

▶ Diversity in Gaming and the Metaverse

Multiplayers

Multiplayer Cooking ▶ Game Prosumption

▶ On Computer Games About Cooking

Multiuser Virtual Environment

Multiplayer First-Person (MUVE)
Shooter
▶ Virtual World, a Definition Incorporating Dis-
▶ Destiny and Destiny 2, an Analysis of an FPS tributed Computing and Instances
Multi-user Virtual Environments for Education 1205

massively-multiplayer online role-playing games

Multi-user Virtual (MMORPGs) (Dieterle and Clarke 2009). Newer
Environments for Education generations of these environments are called
immersive virtual worlds or metaverses (Dalgarno
Dilek Doğan1, Murat Çınar2 and Hakan Tüzün2 and Lee 2010). All of these virtual worlds are
1
Department of Informatics, Ankara University, displayed online in three-dimensions (3D), and
Ankara, Turkey users can move and interact in simulated 3D
2
Department of Computer Education and spaces (Dickey 2005) that can be changed and
Instructional Technology, Faculty of Education, modified. Users are represented as modifiable
Hacettepe University, Ankara, Turkey 3D avatars that can interact with other 3D avatars
and do many activities such as walking, flying,
running, jumping, and dancing. Avatars can be
Synonyms designed based on a user’s real-life appearance
or imaginary appearance (Fig. 1).
3DMUVE: Three-dimensional multi-user virtual Three-dimensional multi-user virtual environ-
environment; HUDs: Head-up displays; LSL: ments (3D MUVEs) can be used for different
Linden script language; MMORPGs: Massively- purposes such as socializing, entertainment, edu-
multiplayer online role-playing games; MOOs: cation, or business. A number of opportunities
MUD, object-oriented; MUDs: Multi-user dun- offered by 3D online virtual environments
geons; OpenSim: OpenSimulator compared to 2D equivalents have led to great
expectations, especially in the field of education.
Using 3D MUVEs for education has the potential
Definition to create a constructivist learning environment
where learners’ interactions and communications
Multi-user virtual environments (MUVEs) are using avatar-to-avatar activities can challenge
M
structured with three-dimensional objects, in them to figure things out for themselves
which users can actively navigate their avatars to (Educause 2006). Social organizations formed in
different areas of the immersive environment. 3D virtual microenvironments might make a sig-
OpenSimulator (OpenSim) is an open source nificant contribution to learners’ self-perception
multi-platform and multi-user 3D application and moral values in the light of personal and
server. social values. In this respect, 3D MUVEs seem
to be well suited not only for cognitive and psy-
chomotor learning areas but also for supporting
Introduction affective learning (Barab et al. 2005; Bers 2001).
Spatial knowledge representation provided by
Online multi-user virtual worlds have been used 3D MUVEs contributes to designing authentic
since the late 1970s (Achterbosch et al. 2007; learning environments, creating opportunities for
Shield 2003). Initially, these environments were experiential learning or learning contexts, and
text-based interaction called multi-user dungeons providing a rich learning experience that includes
(MUDs). With the transformation of MUDs to more effective collaboration and increases stu-
MOOs (MUD, object-oriented), users started to dents’ participation and motivation (Dalgarno
modify these worlds (Tüzün 2006). Advances in and Lee 2010). In addition, students are encour-
information and communication technologies aged to exhibit active involvement and learn by
have driven the MOOs’ evolution, resulting in experience using applied activities (Coffman and
diverse human computer interfaces such as Klinger 2008). 3D MUVEs contain teaching and
multi-user virtual environments (MUVEs) and learning activities such as problem-based
1206 Multi-user Virtual Environments for Education

Multi-user Virtual Environments for Education, Fig. 1 Avatar examples for 3D MUVEs

learning, inquiry-based learning, game-based • It opens up new learning spaces and custom-
learning, role-playing, virtual quests, collabora- ized environments for rehearsal and explora-
tive simulations, collaborative construction, tion, experimentation and design, production,
design courses, language teaching and learning, and user-generated content.
virtual laboratories, virtual fieldwork, and attend- • It provides broader capabilities for learner-led
ing classes (Duncan et al. 2012). activity as well as problem-based and explor-
These environments’ educational potential is atory learning.
summarized as follows (Kluge and Riley 2008; • It provides learners with interaction via text,
Freitas and Veletsianos 2010; González et al. voice chat, or some animation movements.
2013; Papachristos et al. 2014; Antonio et al. • It offers remote access.
2015; Dad et al. 2016): • It allows creation of a parallel world without
limits to creativity and possibilities as the
financial, spatial, and material constraints and
• It reorganizes and extends social interactions
the laws of physics are not applicable.
and collaborations.
• It provides a free environment for learning.
• It supports active participation or “learning by Three-Dimensional Multi-user Virtual
doing” via authentic learning activities. Environments for Authentic Learning
• It increases learner engagement, motivation,
collaboration, and communication. 3D MUVEs can be used for pedagogical
• It presents new opportunities and additional classroom activities that are costly, complex,
scope for creativity in learning such as role- and even dangerous for learners and educa-
playing and mentoring. tors. 3D MUVEs offer opportunities to design
• It supports deeper learning by embedding sim- authentic learning environments that focus on
ulations that are difficult to replicate in the real real-world complex problems and their solu-
world including buying, selling, constructing tions, using role-playing exercises, problem-
buildings, dancing, clubbing, and even learn- based activities, case studies, and participat-
ing and training. ing in virtual communities of practice
Multi-user Virtual Environments for Education 1207

(Lombardi 2007). Herrington et al. (2002) Users utilize these environments for different pur-
identified ten characteristics of authentic poses such as education, business, and entertain-
learning environments: ment. They may not always be appropriate for
education because of user safety concerns, and
• Authentic activities have real-world relevance. authorization is limited to a user’s land and avatar.
• Authentic activities are ill-defined, requiring In addition, there is a fee to customize some
students to define the tasks and sub-tasks of these environments. In these cases, the open
needed to complete the activity. source platform called OpenSimulator can be
• Authentic activities comprise complex tasks to used to create custom 3D MUVEs without fees
be investigated by students over a sustained or limitations.
period.
• Authentic activities provide the opportunity for
students to examine the task from different OpenSimulator
perspectives, using a variety of resources.
• Authentic activities provide the opportunity to OpenSimulator (OpenSim) written in C# is not a
collaborate. virtual world. It is an open source multi-platform,
• Authentic activities provide the opportunity to multi-user 3D application server to create your
reflect. own virtual world (OpenSimulator 2017a)
• Authentic activities can be integrated and released under the BSD License. It provides the
applied across different subject areas and lead ability to customize and design virtual worlds for
beyond domain-specific outcomes. developers. OpenSim has features to support
• Authentic activities are seamlessly integrated developers and users including:
with assessment.
• Authentic activities create polished products • It runs on both a localhost and server using the
valuable in their own right rather than as prep- Windows and Unix/Linux operating systems.
M
aration for something else. • It supports personal computers as a server.
• Authentic activities allow competing solutions • It supports online, multi-user 3D environments
and diverse outcomes. from one to thousands of simulators.
• It supports different sizes of 3D virtual spaces.
3D MUVEs represent a powerful media for • Users access the same world at the same time.
instruction and have the ability to adapt to differ- • It supports real-time Physics Simulations.
ent learner needs (Mascitti et al. 2012). In addi- • It supports users creating or modifying
tion, they can provide innovative ways to create 3D content in real time.
challenging tasks in context (Iqbal et al. 2010). • It supports using scripting including LSL
Thus, they can act as venues for authentic learn- (Linden Script Language)/OSSL and C#.
ing. Learners are part of a constructed environ- • It supports different database engines such
ment and are engaging with the simulated as SQLite, MySQL, and MSSQL to store all
environment, which is similar to real-life interac- content.
tions (Farley 2016). There are some 3D MUVEs • It supports instant messaging by friends or
that allow designers and users to design a virtual groups.
environment such as Worlds.com (1994), Active • It supports loading different modules for con-
Worlds (1995), Traveler (1996), Whyville (1999), figuration settings.
Moove (2000), Second Life (2003), There (2003), • It supports using external or internal VoIP
IMVU (2004), Kaneva (2004), vSide (2006), services such as Freeswitch or Vivox.
OsGrid (2007), Smeet (2007), Smallworlds
(2007), PlayStation Home (2008), Twinity The latest version of OpenSim was released on
(2008), Blue Mars (2009), and Onverse (2009) August 15, 2017 to users. However, some public
(Pearce et al. 2015; Tüzün and Özdinç 2016). distributors, such as diva distribution, add their own
1208 Multi-user Virtual Environments for Education

modules and configurations and later share their scene plug-in modules, although it is a highly
repackaged OpenSim with users through the BSD complex system (OpenSimulator 2017c). Some
license. It contains the standard OpenSim plus add- of these plug-ins are for the startup of virtual
ons that support more features and tools that make worlds such as the region management plug-in.
the process of running and upgrading the virtual Other plug-ins are responsible for extending the
world easier (Diva 2013). Running in diva distribu- functions of the virtual worlds such as voice or the
tion is easier than OpenSim binary packages. effect of clouds and fog (Sun et al. 2010).
OpenSim can be started in stand-alone or grid After the installation process, a viewer must
mode. Stand-alone mode refers to operating all the be install as a user interface that allows users
data services in a single process when run as one to modify or add connection settings to grids.
or many regions. On the other hand, grid mode The most popular of these viewers are Firestorm,
refers to using separate machines when multiple Singularity, Cool VL, Kokua, Alchemy, and
OpenSim instances run on different machines. Radegast Metaverse Client (OpenSimulator
Running in grid mode is more complicated than 2017d). Most of these viewers are available for
running in stand-alone mode (OpenSimulator Windows, Linux, and MacOSX systems.
2017b). According to the number of users and A default avatar and an empty island will be
intended use of the virtual worlds, the technical displayed in the viewer in the first uploaded envi-
specifications for computers and servers will be ronment (Fig. 2).
different. For example, if a server is used for After installation, OpenSim offers unlimited
20–25 users who perform tasks in virtual worlds, possibilities for users and designers based on
the following specifications will be sufficient: their level. User powers are determined by User
Level and Title (OpenSimulator 2017e).
• CPU: 4 Dual-core
• RAM: 8GB • If a user’s level is 0, the user is defined as
• Bandwidth: If 20 users log in simultaneously, default without any permissions.
10 MB/sec is necessary because each avatar or • If a user’s level is 1, the user may rename
a user will use a minimum of 500 KB. objects without modifying permissions.
• Network Latency: Pings between the client and • If a user’s level is 100, the user may toggle
server should be better than 350 ms. It is impor- character geometry, take copy, set to linden
tant and critical on both upload and download content, claim public land, and take ownership
to the simulator. It will affect avatar movement of an object.
and object or avatar position changes. • If a user’s level is 150, the user can enable land
auctions.
If a server is used for 30–34 users who design • If a user’s level is 200 or 250, the user has full
in virtual worlds at the same time with VoIP, the powers in the virtual world.
technical specifications for the computers and
servers must be improved. In addition, the number Initially, users’ characters, called avatars, are
of objects used is a critical issue for these envi- displayed in the default view. However, they have
ronments so storage capacity is important for an inventory and appearance options. Users can
designers. According to this example, the follow- customize their avatars’ shape, skin, hair, eyes,
ing specifications will be sufficient: clothes, etc. and design their avatars’ outfit.
OpenSim supports file formats and extensions
• CPU: 20 GHz as follows:
• RAM: 32 GB
• HDD: 300 GB SAS 10 K • Video: Flash (.swf), QuickTime (.mov), AVI
• Bandwidth: Unlimited, 1000 Mbps Uplink (.avi), Mpeg (.mpeg), and RealNetworks
Stream (.smil)
OpenSim provides an unlimited ability to • Audio: MP3 (.mp3), WAV (.wav)
customize virtual world applications easily with • Text: Text (.txt)
Multi-user Virtual Environments for Education 1209

Multi-user Virtual Environments for Education, Fig. 2 OpenSimulator default context on viewer

• Image: Bitmap (.bmp), Jpeg (.jpeg), Mpeg • Content: All objects have the ability to store
(.mpg, .mpeg), Portable Networks Graphics script or animation files.
(.png), Macintosh PICT (.pict), Silicon
Graphics (.sgi), Graphics Interchange Format Avatars interact with 3D objects and other ava-
(.gif), Targa (.tga), and Tagged Image File tars by touching them or script triggered behav-
M
Format (.tiff, .tif) iors. In addition, Head-Up Displays (HUDs) are
• 3D Model: Digital Asset Exchange (.dae), useful objects for interaction. These objects can be
COLLADA (COLLAborative Design Activ- attached to an avatar to create custom interfaces
ity), and Extensible Markup Language (.xml) on a user’s screen. Some interactions, messages,
• Compressed files: Compressed TAR Archive or textures can be added in the 2D view on HUDs.
file (.tgz), OpenSimulator Archive (OAR), and OpenSim provides many opportunities for
Inventory Archives (.iar) authentic learning utilizing customized virtual
• Animation: Biovision Hierarchy Animation environments. The National Aeronautics and
File (.bvh, .anim) Space Administration (NASA) sponsored one of
the largest projects with OpenSim, Virtual Mis-
3D objects are designed using basic objects sions, and Exoplanets (vMAX), between 2014
called prims such as a cube or cylinder. Autho- and 2017. vMAX developed a 3D virtual world
rized users create prims, determine their position, using OpenSim to engage middle school students
scale, movement, and rotation, and combine them. and educators. The overall project goal was to
Prims have different features: create a comprehensive NASA resource to engage
students, educators, and the public in the search
• General: Object’s name and description, for worlds beyond Earth. In addition, it aimed to
owner, and permissions. increase student engagement in STEM (Science,
• Object: Object’s x-y-z location, rotation, size, Technology, Engineering, and Mathematics),
type, and physical features. knowledge of exoplanet missions, and awareness
• Features: Object’s light cast and flexible path. of NASA-related careers (NASA n.d.). In this
• Texture: Textures can be 2D images or web project, STEM includes astronomy and physics,
contents on 3D objects. technology such as telescopes and satellites, and
1210 Multi-user Virtual Environments for Education

the engineering process and mathematics as References

applied to orbits and measurements. In addition,
participants learn about exoplanets and related Achterbosch, L., Pierce, R., Simmons, G.: Massively
multiplayer online role-playing games: the past, pre-
methods of detection as well as investigate via
sent, and future. ACM Comput. Entertain. 5(4) (2007).
authentic learning activities using 3D MUVEs. https://doi.org/10.1145/1324198.1324207
Their avatars look like an astronaut and they com- Antonio, C.P., Lima, J.P.C., Mota Alves, J.B.,
plete tasks. Marcelino, R., Silva, J.B., Simão, J.P.S.: Remote exper-
iments and 3D virtual worlds in education. In: Pro-
ceedings of 2015 3rd Experiment at International
Conference (Exp At’15), pp. 157–158. Ponta Delgada
Conclusion and Discussion (2015). https://doi.org/10.1109/EXPAT.2015.7463216
Barab, S.A., Thomas, M.K., Dodge, T., Carteaux, B.,
3D immersive technologies, which initially came Tüzün, H.: Making learning fun: Quest Atlantis,
a game without guns. Educ. Technol. Res. Dev. 53(1),
into daily life with modern computer games, have 86–107 (2005)
attracted research interest in the education field Bers, M.U.: Identity Construction Environments: Devel-
because of their pedagogical affordances such as oping Personal and Moral Values through the Design of
enhanced learner engagement, motivation, and a Virtual City. J. Learn. Sci. 10(4), 365–415 (2001).
Coffman, T., Klinger, M.B.: Utilizing virtual worlds
positive attitudes together with their openness to
in education: the implications for practice. Int.
explore, design, and manipulate 3D objects. Today, J. Soc. Sci. 2(1), 29–33 (2008)
there are increasing attempts to use 3D virtual Dad, A.M., Davies, B.J., Kear, A.: 3D virtual worlds:
learning environments in both face-to-face and dis- business and learning opportunities. Int. J. Adv.
tance learning settings to provide learners with Comput. Sci. Appl. 7(1), 7–20 (2016)
Dalgarno, B., Lee, M.J.W.: What are the learning
more realistic and authentic learning environments. affordances of 3-D virtual environments?
OpenSim is an open source platform for build- Br. J. Educ. Technol. 41(1), 10–32 (2010). https://doi.
ing MUVEs. Unlike development platforms that org/10.1111/j.1467-8535.2009.01038.x
offer ready-made services for accessing 3D spaces Dickey, M.D.: Three-dimensional virtual worlds and dis-
tance learning: two case studies of Active Worlds as a
and 3D objects that can be used in these areas, medium for distance education. Br. J. Educ. Technol.
OpenSim offers designers more manipulative 36(3), 439–451 (2005)
design environments where all users can access Dieterle, E., Clarke, J.: Multi-user virtual environments
the same design environment synchronously. for teaching and learning. In: Pagani, M. (ed.) Ency-
clopedia of Multimedia Technology and Networking,
Designers can also create 3D tools and objects 2nd edn, pp. 1033–1040. Idea Group, Inc., Hershey
that can be shared with other designers. Designers (2009)
can modify shared 3D elements easily in accor- Diva: Home. Available online at https://github.com/diva/
dance with their purposes. In addition, user security d2/wiki (2013). Retrieved on 27 Sept 2017
Duncan, I., Miller, A., Jiang, S.: A taxonomy of virtual
is high in OpenSim because of the use of private worlds usage in education. Br. J. Educ. Technol. 43(6),
servers that are not accessible to everyone. It is 949–964 (2012)
important to know how to integrate pedagogy Educause: 7 Things you should know about virtual
into these environments as well as how to use 3D worlds. Available online at https://library.educause.
edu/resources/2006/6/7-things-you-should-know-
environments created with OpenSim. However, the about-virtual-worlds (2006). Retrieved on 18 Oct
lack of guidance on how to organize instructional 2017
design elements and pedagogical arrangements in Farley, H.S.: Chapter 6, The reality of authentic learning in
3D MUVEs is a major limitation for educators. virtual worlds. In: Gregory, S., Lee, M.J.W.,
Dalgarno, B., Tynan, B. (eds.) Learning in Virtual
Worlds: Research and Applications, pp. 129–149.
Available online at https://eprints.usq.edu.au/28739/3/
Cross-References farley_2015.pdf (2016). Retrieved on 18 Oct 2017
Freitas, S., Veletsianos, G.: Editorial: crossing
boundaries: learning and teaching in virtual worlds.
▶ Social-, Mobile-, and Multi-Player-Games and Br. J. Educ. Technol. 41(1), 3–9 (2010)
Their Impact on Today’s Online Entertainment González, M.A., Santos, B.S.N., Vargas, A.R., Martin-
Industry Gutiérrez, J., Orihuela, A.R.: Virtual worlds.
Multivariate Visualization Using Scatterplots 1211

Opportunities and challenges in the 21st century. Pro- Tüzün, H., Özdinç, F.: The effects of 3D multi-user virtual
cedia Comput. Sci. 25, 330–337 (2013) environments on freshmen university students’ concep-
Herrington, J., Oliver, R., Reeves, T.: Patterns of engage- tual and spatial learning and presence in departmental
ment in authentic online learning environments. Aust. orientation. Comput. Educ. 94, 228–240 (2016)
J. Educ. Technol. 19(1), 59–71 (2002)
Iqbal, A., Kankaanranta, M., Neittaanmaki, P.: Engaging
learners through virtual worlds. Procedia Soc. Behav.
Sci. 2, 3198–3205 (2010)
Kluge, S., Riley, L.: Teaching in virtual worlds: opportu- Multivariate Visualization
nities and challenges. Issues Inform. Sci. Inf. Technol. Using Scatterplots
5, 127–135 (2008)
Lombardi, M.M.: Authentic learning for the 21st century:
an overview. In: Educase Learning Initiative- Advanc- Fabrizio Lamberti1, Federico Manuri2 and
ing Learning Through IT Innovation. Available online Andrea Sanna2
at http://www.lmi.ub.edu/cursos/s21/REPOSITORIO/ 1
Dipartimento di Automatica e Informatica,
documents/Lombardi_2007_Authentic_learning.pdf
(2007). Retrieved on 18 Sept 2017
Politecnico di Torino, Torino, Italy
2
Mascitti, I., Fasciani, M., Stefanellil, C.: Virtual worlds in Dipartimento di Automatica e Informatica,
education: Avatar St.Art and euroversity action- Politecnico di Torino, Turin, Italy
research initiatives. Highlight. 1(2), 253–265 (2012)
NASA: Virtual Missions and Exoplanets (vMAX). Avail-
able online at https://informal.jpl.nasa.gov/museum/
CP4SMP/virtual-missions-and-exoplanets-vmax Synonyms
(n.d.). Retrieved on 18 Sept 2017
OpenSimulator: What is OpenSimulator? Available online Scatter chart; Scatter diagram; Scatter graph; Scat-
at http://opensimulator.org/wiki/Main_Page (2017a).
Retrieved on 27 Oct 2017
tergram; Scatterplot
OpenSimulator: OpenSimulator simulator configuration
file. Available online at http://opensimulator.org/wiki/
Configuration (2017b). Retrieved on 27 Oct 2017
OpenSimulator: 0.9.0.0 Release notes. Available online at
Definition M
http://opensimulator.org/wiki/0.9.0.0_Release (2017c).
Retrieved on 27 Oct 2017 Multivariate visualization by scatterplots is the
OpenSimulator: Compatible viewers. Available online at usage of diagrams to visualize sets of data that
http://opensimulator.org/wiki/Compatible_Viewers have more than three variables.
(2017d). Retrieved on 27 Oct 2017
OpenSimulator: UserLevel. Available online at http://
A scatterplot is a chart or mathematical dia-
opensimulator.org/wiki/Userlevel (2017e). Retrieved gram displaying a set of data as a collection of
on 27 Oct 2017 points using Cartesian coordinates, usually
Papachristos, N.M., Vrellis, I., Natsis, A., Mikropoulos, T.A.: defined by horizontal and vertical axes. Each
The role of environment design in an educational multi-
user virtual environment. Br. J. Educ. Technol. 45(4),
point on the chart represents two variables, x and
636–646 (2014) y, calculated independently to form bivariate pairs
Pearce, C., Blackburn, B.R., Symborski, C.: Virtual worlds (xi, yi). A functional relation between x and y is
survey report. Available online at http://cpandfriends. not necessary. The purpose of a scatterplot is to
com/wp-content/uploads/2015/03/vwsurveyreport_
final_publicationedition1.pdf (2015). Retrieved on
reveal (if existing) the relation between the
27 Oct 2017 displayed variables.
Shield, L.: MOO as a language learning tool. In:
Felix, U. (ed.) Language Learning Online: Towards
Best Practice, pp. 97–122. Swets and zeitlinger, Lisse
(2003)
Introduction
Sun, B., Wu, H., Zhao, H., Hu, X.:. Research and applica-
tion on plug-in technology in OpenSim. In: 2010 Inter- Multivariate visualizations deal with the chal-
national Conference on Audio Language and Image lenge of displaying sets of data with three or
Processing (ICALIP), 23–25 Nov 2010, Shanghai
(2010)
more variables: this peculiar feature poses two
Tüzün, H.: Educational computer games and a case: Quest kinds of problems. First, most of the charts and
Atlantis [in Turkish]. Hacettepe Univ. J. Educ. 30, diagrams usually adopted to visualize data cannot
220–229 (2006) display more than three dimensions adequately.
1212 Multivariate Visualization Using Scatterplots

Multivariate
Visualization Using
Scatterplots, Fig. 1 An
example of scatterplot
diagram

Second, the effectiveness of the visual effects out relationships, patterns, or outliers, a
adopted to represent different variables deterio- scatterplot provides a suitable visualization tool
rates when the number of variables increases. for multivariate data due to its intrinsic features.
Scatterplots may be considered, among the
different types of data visual representations, as
one of the most useful and versatile, especially in Usage
statistics. According to (Miller 1995), the term
first appeared as Scatter Diagram in a 1906 article Different scenarios lead to different tasks when
in Biometrika, “On the Relation Between the dealing with multidimensional visualization tech-
Symmetry of the Egg and the Symmetry of the niques. As defined by Valiati (2005) and further
Embryo in the Frog (Rana temporaria)” by J. W. described by Pillat et al. (2005), five major tasks
Jenkinson. However, the term only came into can be considered as objectives a user might want
wide use in the 1920s when it began to appear in to fulfill when using a visualization tool to display
textbooks, e.g., F. C. Mills, Statistical Methods of or analyze multivariate data: identify, determine,
1925. The Oxford English Dictionary gives the compare, infer, and locate. Scatterplots can be
following quotation from Mills: “The equation to used to assess all these different tasks and have
a straight line, fitted by the method of least squares been applied to data in many different fields of
to the points on the scatter diagram, will express use, such as automotive, finance, pharmacology,
mathematically the average relationship between environment, weather forecast, telecommunica-
these two variables.” Fig. 1 provides an example tion, food, and many others.
of scatterplot diagram.
Scatterplots are mainly appreciated for their Identify
ability to reveal nonlinear relationships between This task refers to any action of finding, discover-
variables. Moreover, scatterplots are typically ing, or estimating visually:
used to identify correlations between variables,
with a certain confidence interval. Another usage • Properties like symmetrical or asymmetrical
for the scatterplot is to compare similar data sets. distribution, values or dispersion
Since the main problem of multivariate data is to • Correlation, data dependency or independency
correctly understand and analyze them, pointing • Similarities or differences
Multivariate Visualization Using Scatterplots 1213

Multivariate
Visualization Using
Scatterplots, Fig. 2 This
scatterplot suggests a
correlation between the two
displayed variables

• Clusters as a result of similarity, continuity, Compare

proximity, or closed shapes This task takes place when the user wants to com-
• Thresholds, patterns, data variation pare data that have been previously identified,
located, visualized, or determined. The user may
The identify task takes place anytime the user compare data to analyze dimensions, data items,
analyzes the chart with the purpose of finding, clusters, properties, proportion, values, locations
M
estimating, or discovering new information and distances or visual characteristics. The compare
about the data. The task ends when the user finds task is an analytic task the user performs specifically
the information he/she was looking for or the if he/she compares data items displayed in the
current goal changes. Figure 2 shows an example graphical visualization. Figure 4 shows a scatterplot
of scatterplot that clearly suggests a linear corre- configuration that enhances the comparison task.
lation between the displayed variables.
Infer
Determine This task refers to the action of inferring knowl-
This task corresponds to the action of calculating, edge from the visualized information, such as
defining, or precisely designating values such as: defining hypotheses, rules, probabilities or trends,
attributes of cause and effect. This task usually
– Mean, median, variance, standard deviation, takes place after determining, identifying, or com-
amplitude, percentile paring information, and it is performed as part of
– Sum, differences, proportions the mechanism of data analysis, thus it may not be
– Correlation coefficients, probabilities, or other completed at once, requiring consecutive applica-
statistics such as hypotheses test tions of the other visualization tasks. By analyzing
Fig. 1, it is possible to infer a hypothesis, e.g., that
This task begins when the user needs to calcu- the y variable is the cause of the trend of the data.
late a specific value and ends up when the calcu-
lation is completed. Figure 3 shows a scatterplot Locate
that allows to derive the precise value of each This task refers to the actions of searching and
point in order to compute precise calculations finding information in the graphic representation:
such as the mean value. they can be data points, values, distances, clusters,
1214 Multivariate Visualization Using Scatterplots

Multivariate
Visualization Using
Scatterplots, Fig. 3 A
scatterplot visualization that
simplify the computation of
the mean

Multivariate
Visualization Using
Scatterplots, Fig. 4 A
scatterplot configuration
that enhances comparison

properties, or other visual characteristics. The task difficult to correctly visualize and analyze all the
begins when the user starts examining the visual data. In order to overcome this problem, different
representation and finishes when he/she recog- solutions have been proposed through the years to
nizes the desired information. Figure 5 shows a enhance the scatterplot.
scatterplot visualization that enhances the identi-
fication of outliers. Adding Dimensions
Even if the basic scatterplot may display only two
variables, various techniques have been
Dimensions researched and adopted through the decades to
increase the dimensionality of scatterplots by
The main problem when using the scatterplot to one, two, or even several additional dimensions.
visualize multivariate data is that its basic version A bidimensional planar scatterplot of two vari-
is limited to only two variables, thus making it ables X and Y can display additional variables
Multivariate Visualization Using Scatterplots 1215

Multivariate
Visualization Using
Scatterplots, Fig. 5 A
scatterplot visualization that
enhances the identification
of outliers

Multivariate
Visualization Using
Scatterplots, Fig. 6 A
scatterplot with an
additional variable
visualized as color
M

by correlating them to one or more graphical together an arbitrary number of scatterplots,

features of the plotted points. both different or complementary, such as in the
case of a scatterplot matrix, without cluttering
Color or visibly degrading any of them. This solution
One approach is to show a third dimension can increase significantly the effectiveness of
through a color map. Colored points on a such visualization with respect to the sum of
scatterplot may suggest similarity among values the individual unlinked scatterplots. Colors
of the same dataset or correspondence among can also be used to enhance the perception of a
points of different datasets. Moreover, this cor- variable already displayed by another effect
relation may be perceived without drawing any (such as an axis). Figure 6 shows a scatterplot
connecting line. This technique is particularly that displays an additional variable through
powerful since it could also be used to link colors.
1216 Multivariate Visualization Using Scatterplots

Size the orientation of the shape. Usually, a dot or line

A further option to provide an additional dimen- is drawn orthogonally to the perimeter of the
sion to the scatterplot is to vary the size of the shape to better identify the reference point for
points. Anyway, this option may lead to occlu- the orientation. Figure 9 shows a scatterplot that
sion problems if the plot does not provide proper displays an additional variable through the orien-
scaling on the two axis. Figure 7 shows a tation of the points.
scatterplot with a variable mapped on the size
of the points. Error Bars
The uncertainty is the variability related to a
Shape specific variable of the dataset for each point. It
Another approach is to add a third dimension provides a generic idea of how precise the mea-
changing the shape of the points. Instead of surement of the reported value is or how far from
using only points, each element of the dataset the recorded value the real value might be. This
could be drawn as different kinds of glyphs information is usually reported through error
depending on a third variable. This option leads bars if it is related to a variable mapped on the
to further possibilities in terms of the paradigm x or y axis (or both). Figure 10 shows three
used to choose the shape. One option is to display examples of error bars. Error bars require addi-
the points as “flowers,” relating the variable to the tional space around the points to be correctly
number of “petals” to display. Another option is to displayed due to the chance of overlapping
display polygons and relating the number of sides between points. For this reason, they are usually
to the variable. Moreover, various glyphs, clearly adopted only if the points of the scatterplots are
distinct among them, could be used to represent very scattered and occlusions do not occur. Oth-
different datasets. Figure 8 shows a scatterplot erwise, the use of error bars would greatly affect
that uses the shape of the points to display addi- the understandability of the representation. As a
tional information. result, the use of error bars limits the number of
different graphical effects that could be com-
Orientation bined on the same scatterplot and should be
Another possibility when displaying points as avoided when displaying more than three or
shapes is to represent a third dimension changing four variables.

Multivariate
Visualization Using
Scatterplots, Fig. 7 A
scatterplot with a variable
mapped on the size of the
points
Multivariate Visualization Using Scatterplots 1217

Multivariate
Visualization Using
Scatterplots, Fig. 8 A
scatterplot that uses the
shape of the points to
display additional
information

Multivariate
Visualization Using
Scatterplots, Fig. 9 A
scatterplot that displays an
additional variable through
orientation
M

Multivariate Visualization Using Scatterplots, Fig. 10 Three examples of error bars

1218 Multivariate Visualization Using Scatterplots

Multivariate
Visualization Using
Scatterplots, Fig. 11 A
scatterplot that displays
multiple variables through
different effects

Adding More Dimensions Concurrently Scatterplot Matrix

It is possible to use simultaneously more than one
of these techniques, independently, to obtain even The simplest approach to adapt the scatterplot to
high visual dimensionality. Figure 11 shows an multivariate data is to produce a series of
example of such a scatterplot. However, this is scatterplots for each pair of variables and display
recommended only if the graphical effects are them together on a single screen or page. This
clearly distinguishable, otherwise the visual clar- visualization technique is called scatterplot matrix
ity and benefits of displaying more dimensions at and for k variables it requires k(k-1)/2 pairs and
the same time will promptly worsen. Many stud- therefore scatterplots. Unfortunately, this solution
ies, like the one by (Demiralp et al. 2014), have presents a major problem: analyzing all the
been carried out to understand how visualization scatterplots may require a lot of time, depending
design can benefit from taking into consideration on the number of variables, thus this solution
perception, as different assignments of visual is not optimal when dealing with time-related
encoding variables such as color, shape, and size tasks. To overcome this problem, different visual-
could strongly affect how viewers understand ization techniques may be adopted to interact with
data. the dataset and simplify data comprehension.
Figure 12 shows an example of scatterplot matrix.
Dynamic Visualizations
Even if scatterplots are typically used to display Brushing
static data, nevertheless they can be very useful Brushing is the action of selecting a subset of the
when applied to display data that could change points displayed on the scatterplot. Four brushing
dynamically, moreover if the change may be con- operations have been defined by Becker and
trolled by the user. More complex graphical Cleveland (1987): highlight, shadow highlight,
effects such as animation may be adopted in this delete, and label. To perform these operations, it
case to enhance the comprehension of data as they is necessary to resize a rectangle, called the brush,
change over time. This is the case of data charac- over one of the scatterplots. The corresponding
terized by one or more time-related variables, points on each different scatterplot are then
such as stocks values in finance or weather condi- affected by the chosen operation. The brush can
tions in forecasting. be moved to different regions of the scatterplot by
Multivariate Visualization Using Scatterplots 1219

100 100

90 90
OVERALL SCORE 2015

OVERALL SCORE 2015

80 80

70 70

60 60

50 50

40 40

30 30
10 20 30 40 50 60 70 80 90 100 10 20 30 40 50 60 70 80 90 100
ACADEMIC REPUTATION SCORE EMPLOYER REPUTATION SCORE

100 100

90 90
OVERALL SCORE 2015

OVERALL SCORE 2015

80 80

70 70

60 60

50 50

40 40

30 30
10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100
FACULTY STUDENT SCORE CITATIONS PER FACUALTY SCORE

100 100

90 90
M
OVERALL SCORE 2015

OVERALL SCORE 2015

80 80

70 70

60 60

50 50

40 40

30 30
0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100
INTERNATIONAL FACULTY SCORE