Bachelorarbeit im Studiengang Audiovisuelle Medien

Real-Time Cinematography in a Virtual

Production Environment

Developing a Virtual Camera Prototype for Commercial Productions

vorgelegt von Anna Eschenbacher

an der Hochschule der Medien Stuttgart

Erstprüfer: Prof. Uwe Schulz

Zweitprüfer: Kai Götz

am 17. Oktober 2018

Abstract

Virtual production technologies have successfully been applied to various high-end film, TV and
games productions over the last decade. The technological advancements of real-time graphics
and motion capture technology driven by the games industry allow for a facilitated development
of virtual production tools. Therefore, smaller visual effects studios or even students can now
benefit from virtual production. One sector in need of improvement is the advertising industry
characterized by tight scheduling and little room for creative expermentation.

The aim of this paper, is to examine the potential of a virtual camera system for commercial
productions as well as providing a comprehensive overview about the fundamentals of virtual
production and recent developments regarding game engines and real time filmmaking. Thus,
a prototypical virtual camera system has been developed and tested by an expert group. The
results show that a virtual camera system for commerical work allows for various benefits such as
more creative freedom, less unnecessary iterations as well as faster and more intuitive working.
However, the virtual production has to be reliable and customizable to support established
workflows instead of interfering with them.

Target group
This paper targets individuals who are involved with computer generated imagery (CGI), visual
effects (VFX) and advertising industry and have a certain interest in the development of the
VFX industry in regard to virtual production. This includes everyone from students to artists to
industry professionals as well as clients and agencies interested in the use of novel technologies.
The reader should have basic knowledge about CGI and VFX workflows. A general interest in
immersive media such as virtual reality and augmented reality as well as game engine technology
makes it easier to understand the different components and features, which are necessary to create
a virtual camera system.

iv
Kurzfassung
Virtual Production Technologien wurden in den letzten zehn Jahren erfolgreich bei Film, TV
und Games Produktionen eingesetzt. Der technologische Fortschritt von Echtzeit-Grafiken und
Motion-Capture-Technik wurde vor allem von der Games Industrie angetrieben. Diese Weiter-
entwicklung erlaubt vereinfachte Entwicklung von Virtual Production Geräten. Daher haben
nun auch kleinere VFX Studios und sogar Studenten die Möglichkeit von Virtual Production
Technologien zu profitieren. Besonders die Werbefilm Industrie ist auf Verbesserung der Pro-
duktionsabläufe angewiesen, da sie von knapper Zeitplanung und wenig kreativem Freiraum
geprägt ist. Diese Arbeit untersucht das Potenzial eines virtuellen Kamerasystems im Einsatz für
kommerzielle Produktionen. Des Weiteren, wird ein grundlegender Überblick zur Anwendung
von Virtual Production vermittelt, so wie zu aktuellen Entwicklungen im Bereich Game Engines
und Echtzeit-Filmaufnahmen. Aus diesem Grund wurde ein Prototyp eines virtuellen Kamerasys-
tems entwickelt und von einer Expertengruppe getestet. Die Evaluation ergab, dass ein virtuelles
Kamerasystem bei Werbefilmproduktionen für mehr kreativen Freiraum sorgt, weniger unnötige
Iterationen erforderlich sind, sowie schnelleres und intuitiveres Arbeiten erlaubt. Jedoch müssen
Virtual Production Technologien zuverlässig und einfach konfigurierbar sein, um bestehende
Arbeitsabläufe zu unterstützen statt sie zu behindern.

Zielgruppe
Diese Bachelorarbeit ist für alle, die sich mit der Visual Effects- (VFX), Computer Generated
Imagery- (CGI) und Werbefilm-Branche befassen und ein gewisses Interesse in die Weiteren-
twicklung der VFX Industrie im Bezug auf Virtual Production haben. Dazu gehört jeder von
Studenten, über Artists bis hin zu Branchenexperten. Des Weiteren betrifft es Kunden und
Agenturen, welche sich für den Einsatz von neuen Technologien interessieren. Ein Grundwissen
im Bereich CGI und VFX Workflows wird vorausgesetzt. Durch ein generelles Interesse für
Virtual und Augmented Reality, sowie für Game Engines können die notwendigen Komponenten
zur Entwicklung eines virtuelles Kamerasystem leichter nachvollzogen werden.

v
Nomenclature

AR Augmented Reality

CG Computer graphics

CGI computer generated imagery

DCC Digital content creation

DOF Degrees of freedom

DOP Director of photography

GUI Graphical user interface

HMD Head mounted display

Mocap Motion capturing

Previs Previsualization

UI User interface

VCS Virtual camera system

VFX Visual effects

VR Virtual Reality

vi
Outline
1. Introduction 1
1.1 Scope of Thesis 2

2. Fundamentals of Virtual Production 3

2.1 Implementation of Virtual Production into Existing Workflows 5

3. Virtual Production Technologies 10

3.1 Motion Capture Technologies 10
3.2 Real-Time Graphics 16
3.2.1 Origins of Real-Time Filmmaking 17
3.2.2 Recent Developments 17
3.3 Virtual Cinematography 21
3.3.1 Virtual Camera in Full CG 24
3.3.2 Recent Developments for Virtual Camera Systems 26
3.4 Demands for Virtual Production Technologies in Commercial Productions 31

4. Developing a VCS Prototype 32

4.1 User Interface 32
4.1.1 Required Input and Output Devices 33
4.1.2 Selected Hardware and Software Components for ALICE RT 35
4.2 ALICE RT Prototype Implementation 40
4.2.1 User Interface Design 40
4.2.2 Setup and Configuration 41
4.2.3 Operating ALICE RT 43

5. Evaluation 47
5.1 Test Scenario for Evaluation 47
5.2 Results 51
5.2.1 User Group Review 51
5.2.2 Personal Review 58

6. Conclusion 61

Appendix 65
A. Glossary 65
B. Bibliography 66
C. List of Figures 70

vii
1 Introduction

Filmmaking has always been a collaborative process to tell captivating stories. By introducing
visual effects (VFX) visually anything could be told. VFX is an umbrella term for ”any creation,
alteration, or enhancement of imagery for moving media that can not be accomplished during
live-action shooting” (Fink 2015, p. 2). It started with simply stopping the camera and adjusting
the scene before resuming, continued with the use of techniques such as analog matte paintings,
miniature models, stop-motion, rear projection, and pyrotechnics. Around the 1970s more
complex visual effects were used such as motion control systems and the integration of computer
generated imagery (CGI) (Fink 2015, p. 3). With an increasing amount of CG content, the
filmmaking process became more and more departmentalized, resulting in a fragmented workflow.
Simultaneously, film sets started to look increasingly abstract to the lead creatives with the use of
blue and green screens. Thus, maintaining a shared vision of the project became more difficult.
In search of regaining collaborative and interactive methods, where the film is seen as a holistic
entity again, real-time and motion capture technologies were put together under the term virtual
production.

This fusion of technologies has been used and analyzed on several high-end film, TV and games
productions. However, the application of virtual production in advertising is still very uncommon,
especially in regard to smaller and mid-sized VFX studios. This paper explores the potential
benefits, challenges, and areas of use for a virtual production tool in advertising.

A detailed description of the concept of virtual production and its implementation into a traditional
filmmaking workflow demonstrates the potential of virtual production tools. Followed by an
approach to various technologies necessary to develop such tools, particularly in regard to
virtual camera systems. Motion capture methods, real-time graphics and the concept of virtual
cinematography are elaborated in this chapter. The recent progress for these technologies
highlights the relevance of assessing their application in cinematic projects.

1
Moreover, an overview of recent virtual camera systems is given. To examine the use of virtual
production technologies in advertising, the conception, development, and application of a virtual
camera prototype was conducted and is described in this paper. Subsequently, an evaluation of the
virtual camera prototype with industry professionals is performed and assessed to verify assumed
advantages and potential application of virtual production tools in advertising. The user group
suggests additional features, describes possible deficits and recognizes the benefits of these tools,
regardless of the prototypical state of the tool. The provided feedback serves as a basis for further
development of the respective virtual camera system.

1.1 Scope of Thesis

The present paper is published at Stuttgart Media University in the winter term 2018/2019 as
bachelor thesis by Anna Eschenbacher. The development of the virtual camera system prototype
was realized at INFECTED, a post-production and CG content studio based in Hamburg, Germany.
Furthermore, the test conducted for evaluation was carried out and supported by INFECTED as
well. The development of the virtual camera prototype ALICE RT started out in 2017 as part
of an internship at INFECTED absolved by the author. Necessary practical work, in regard to
the virtual camera prototype examined in this paper, was the conceptual design, the creation
of a requirements specification for a virtual camera system, the graphical user interface (GUI)
design as well as initial GUI programming, and the preparation of the evaluation questionnaire as
well as the evaluation process itself were performed by the author. The software development
and hardware configuration were carried out by INFECTED employees Dennis Gabriel and Tim
Edelmann, and one external programmer. Dennis Gabriel acted as the lead engineer on this
project. Since the INFECTED employees were only able to work on the prototype in between
commissioned work, the development spans over one and a half years starting in April 2017.

2
2 Fundamentals of Virtual Production

If you have a look at a film set of any blockbuster movie today, chances are you will find
actors surrounded by a blue or green screen. Sometimes this might only be for a background
replacement, other times everything including the actor will be replaced by digital objects and
characters.

Figure 2.1: On set of ’Doctor Strange’ (2016) inside a green screen studio (left), the final image
(right)

In order to still be able to visualize the final composition of a scene while being on set, certain
tools are necessary to blend together digital and real content. These tools are part of the concept
of virtual production. The official definition for virtual production by the Virtual Production
Committee is:

”Virtual Production is a collaborative and interactive digital filmmaking process

which begins with virtual design and digital asset development and continues in an
interactive, nonlinear process throughout the production.” (Beck 2015, p.444)

This definition is likely to change with technological developments over the next few years as
well as the more frequent use of virtual production technologies. Subsequently, this will yield
new workflow models and new points of intersection between different departments.

3
Since the demands and needs for VFX films are ever changing and evolving, there is no checklist
of technologies which define every virtual production tool. Girish (Balakrishnan 2016), virtual
production supervisor at VFX Studio MPC, describes it as a design methodology that requires
a blending of advanced visualization technologies with rapid prototyping. This makes virtual
production an iterative workflow to creatively work through ideas from concept to final.

The best known project depending on virtual production is ’Avatar’ (2009). However, it was
not the first virtual production nor is it the most advanced anymore. The first motion captured
performance done by Industrial Light & Magic (ILM) was in ’The Mummy’ (1999). Until then,
no one had captured the movements of a principal actor and applied them to a CG character (ILM
1999). Since then, many new advancements have been made. Especially over the last three to
four years, virtual production has developed rapidly and is now being used on a growing number
of films (Morin 2018). The main reason for this is the increasing use of CGI content and the
artistic benefits related to virtual production, which are explained more elaborately throughout
this thesis.

For a better understanding of how virtual production is changing existing production workflows,
the next chapter will showcase the integration of virtual production into a traditional production
workflow.

4
2.1 Implementation of Virtual Production into Existing
Workflows

Before the use of any VFX, the possibilities for being able to visualize stories were limited, yet the
director was able to interact with everything on set, whether that is actors, the set environment or
any practical effects. When CGI came along in the late 1970s (AMC 2018), virtually anything was
achievable, which gave the opportunity to tell new stories convincingly. Directors were able to
realize films, which would have been impossible until then, due to the laws of physics or excessive
production costs (Patel 2009, p. 3). All of this digital content was created in postproduction.
While the director would still feedback and approve the VFX, there was no direct involvement
in the creative process. Since the beginning of CGI, the average amount of VFX shots in action
or sci-fi heavy movies has increased from a few hundred to over 2000 today (Serrabassa 2014).
Taking a look at the twenty worldwide most grossing films of all time, seventeen of those are
VFX-heavy and the remaining three are fully CG-animated films (Box Office Mojo 2018).

Previsualization

When digital effects started to take up a greater amount of films, the need to previsualize digital
assets ahead of production became critical. Previsualization (previs) enables the director to regain
some control and share the vision for the film early on. It is a collaborative process, in which
preliminary versions of a shot are generated predominantly by 3D animation tools. This allows
filmmakers to visually explore creative ideas, plan technical solutions, and communicate a shared
vision for efficient production (Beck 2015, p. 46). The director can channel in on emotional
events in the story, that define the characters even before a scene has been shot. At the same
time, the production designer works together with the previs team to roughly build the world
digitally before investing in any material costs. Keeping production costs under control is another
major reason for the concept of previsualization Wolfe (2012). The sooner creative choices can
be made, the less time and money is wasted on other possible outcomes.

5
Previs enhanced the production workflow in many ways, yet there was still a gap between pre-
and postproduction concerning digital content. There was no real possibility for the director to
interact with CG Elements and digital characters during the on set production process. With
ongoing developments in tracking technologies and computer graphics, it became possible to
create tools that enable directors to interact with VFX as effectively and immersive as with live
action. These tools are the beginning of virtual production technologies (Patel 2009, p. 3).

Virtual Production

Virtual production starts with preproduction and continues until the final frame is rendered. By
blending traditional boundaries and enhancing the production with the use of real-time computer
graphics, the director is finally able to interact with virtual and real elements at the same time.
The lead creative team, such as the director or cinematographer, regained their ability to directly
control the creative process by operating tools like a virtual camera system (VCS) on set. Real-
time graphics enable instant feedback and form the base of virtual production according to
the Chairman of the Virtual Production Committee, David Morin. It starts with digital asset
development and continues in an interactive process throughout production (Morin 2018).

A closer look at the different possible stages of previsualizing helps to get a better understanding
of the advantages of virtual production and the use of a shared asset pipeline. Generally, each
production step can be improved by creating the respective form of visualization as shown in the
figure below.

Figure 2.2: The process of previsualization throughout production

6
Pitchvis offers the opportunity to support the vision for a project visually, so the investors can
have a first look at the potential outcome. A sequence or idea of the project is visualized in a
quick, yet aesthetically pleasing way so the mood and general feeling of a film can be conveyed.

Figure 2.3: Pitchvis by Halon Entertainment: ’300: Rise of an Empire’ (left), ’Crash Site’ (right)

D-vis, which stands for design visualization allows for an early design collaboration between
different departments. Locations can be virtually scouted, digital assets are created, designed
and shared with the postproduction department for a common understanding of what is intended.
During this phase, ideas and possible compositions can be explored and experimented with (Beck
2015, p. 46).

Figure 2.4: The different stages of visualization on ’The Jungle Book’ (2016): (top left to bottom
right) D-vis, Tech-vis, live-action shoot, final shot.

Tech-vis encompasses a technical breakdown before shooting a scene regarding accurate physical
properties and dimensions for the real world set, which helps define production requirements.

7
This includes information about camera, lighting, design and scene layout. Technical previs is
often used in action or VFX heavy scenes, so they can be tested and planned in detail while still
conserving resources for the actual production (Beck 2015, p. 46).

On-Set Previs is where virtual production is the most prominent. Real-time technologies are used
to evaluate and interact with on set performance of both real and CG content. A live composite
of footage with digital environments and preanimated or live motion captured CGI characters
is the most common use of virtual production on set. Being able to combine everything in real
time can be crucial for the execution of a scene particularly for stunts or combat sequences. This
kind of visualization is coined as stunt vis. For example, the combat scenes in ’Suicide Squad’
(2016) consist of a complex choreography with a large number of characters of radically different
heights. The stunt performers would wear motion capture suits so that their performance could
immediately be applied and upscaled in real-time to the larger CG models.

Animatrik CTO Brett Ineson, who supervised this project commented that “this gives real-time
feedback to the director. He could see how the disproportionate actor character combinations
would play out with each other and make sure the sequence would just work” (Ineson 2016).
Knowing how the scenes are playing out before moving forward helps to decrease the high
amount of changes in postproduction. Additionally, after a selection of certain combat scenes
has been made, the motion captured data was post-processed and handed off to the animation
department to continue to work on these sequences. This shows that communication and sharing
of assets between the on-set production team and post production are crucial, thus without it the
whole production would be painfully slow and uneconomical.

In postvis, CGI elements and live action plates are combined in a more thorough way than live
on set. It is still only a rough version compared to the final image, nevertheless, it is essential for
director and editor to have an insight on what is actually in the shot instead of having to imagine
what might be there, how fast will it move and how large will it be. Postvis is a first test on how
well the various digital and real elements interact together (Beck 2015, p. 57). Since creating the
final VFX shots will take up most of the time in postproduction, postvis is a placeholder which
can be shown to test audiences and used as a planning tool for VFX producers (Beck 2015, p.
46).

8
Summarising the above, it can be said that virtual production and previs create better commu-
nication between all departments from start to finish. A shared vision of the project helps to
discuss and eliminate problems as soon as they arise. Clint Reagon, previs supervisor at Halon
Entertainment, describes it as ”the reason to have previz [is] that everybody can see and say, oh
wait there is a problem, and we can all address it. It is kind of a catch-all where you can see the
movie beforehand and catch those problems you really do not want to have [during production].”
(Reagan 2017a). It provides guidance for the whole project, reduces unnecessary iterations in
post production and decreases high production costs by planning digitally first with the help of
real-time technologies.

A good example for this is the film ’The Jungle Book’ (2016) directed by Jon Favreau, which was
a complete virtual production project. The only real character in this film is Mogli, everything
else was added in post production. The use of previs, a virtual camera system and other motion
captured data was essential to create rough versions of scenes that enabled meaningful creative
choices and instant feedback. Furthermore, The Jungle Book finished ahead of schedule and
under budget, when it usually is the other way around in the VFX industry (Morin 2018).

Figure 2.5: On set of The Jungle Book (2016). Neel Sethi surrounded by blue screen.

A change towards a clear vision for CG content is very much in need, due to the overused mantra
”fix it in the post”. This way of thinking caused the VFX crisis over the last decade, in which
many studios had to restructure or even declare bankruptcy. One of the reasons for this is the
fixed bidding system, where a VFX studio has to estimate how much a project will cost including
changes, which cannot be predetermined (Leberecht 2014). As a result, VFX studios are now
making use of state of the art technology to improve communication throughout the whole process
of filmmaking and eliminate unnecessary iterations early on.

9
3 Virtual Production Technologies

It is becoming more and more common to use virtual production during film, TV or game
productions. Companies like Halon and The Third Floor fully specialize in previsualization. Not
only high end productions profit from using virtual production, but also low budget projects are
able to use it thanks to commercialised use of various technologies. This chapter focuses on the
main hard- and software behind virtual production in particular with regard to virtual camera
systems for full CG films. Thus, techniques such as compositing will not be elaborated in this
chapter.

Virtual production blends together several technologies, which were originally developed for
other purposes. For example, the use of game engines to create real-time graphics for film or the
tracking technology for VR systems. These differing technologies are now merged together to
fulfill needs of virtual production.

3.1 Motion Capture Technologies

Motion capture (mocap) is the process of tracking and recording the position as well as movement
of objects or people. The recorded data will be applied to virtual assets either after further
processing or in real-time during production (Gleicher 1999, p. 1). Depending on what is being
tracked, different information about the position is needed. Starting with rigid body objects, such
as a camera, only the rotation and position, consisting of x, y and z coordinates, is measured.
Adam Ealovega (2014, p. 26), creative director at Stargate Studios, states that ”at the core to
any evolution of virtual production is incredibly accurate camera positional metadata”, which is
necessary to perfectly align virtual and real space. More complex than rigid body object tracking,
is motion capturing a character by additionally predefining every single joint.

10
This can deliver high-fidelity character animation, difficult to achieve in any other method (Root
2015, p. 385). The next step after capturing characters movements is performance capture
in which the face, fingers or subtle expressions are recorded simultaneously with the body
performance (Knopp 2014, p. 30), as shown in the figure below. The more detailed a performance
can be captured, the more potential has a CG character to be convincing and captivating. Andy
Serkis (2011), critically acclaimed and well known motion capture actor, states that "Performance-
capture technology is really the only way that we could bring characters to life. It’s the way that
Gollum was brought to life, and King Kong, and the Na’vi in Avatar ... it’s really another way of
capturing an actor’s performance. That’s all it is, digital make-up.”

Figure 3.1: The use of performance capture in War for the Planet of the Apes (2017), (left to
right) CG renders, original principal photography plate, final shot.

The most frequently made use of tracking technologies are described in the following paragraphs.
The focus is primarily on positional tracking for rigid body objects in regard to virtual camera
systems.

Non-Optical Tracking

Acoustic tracking describes the measurement of the elapsed time for an acoustic signal to travel
from an emitter to a receiver. When placing several transmitters around the tracked area and
various receivers on the tracked object, the orientation, and position of the object relative to the
transmitters can be defined by the different times of arrival to the respective receivers. Acoustic
tracking only allows relatively low capture rates. It requires time-consuming calibration and
is prone to measurement errors due to ambient noise. Thus, the technology is usually used in
combination with other tracking methods to provide better accuracy (Road to VR 2014).

11
Another technique is magnetic tracking, which measures the magnitude of the magnetic field
while moving around the tracking volume. Similar to acoustic tracking, a static emitter sends out
an electromagnetic field, which is received by a sensor attached to the tracked object. Depending
on where the object is located in relation to the emitter, the induced current changes and can
be analyzed for positional and rotational data. In a controlled environment, magnetic tracking
is precise. Though, the method is susceptible to interference from other electric devices or
ferromagnetic materials within the tracked area (Faisstnauer 1997, p. 22).

Inertial tracking is based on the measured position and angular orientation by accelerometers
and gyroscopes. It provides low latency, high update rates, and is inexpensive compared to other
tracking technologies. The technology consists of small sensors, which do not interfere with
freedom of movement when attached to objects or actors. In fact, they are small enough to be
integrated inside of smartphones, where they facilitate features such as step counting. However,
inertial tracking is known for positional drifting, which subsequently makes the captured data
unusable for virtual production (Road to VR 2014).

Optical Tracking

A different approach to motion capture is optical tracking, where various cameras are used to
aquire tracking data. If the camera is attached to the tracked device and scans the surrounding
area, it can be described as inside-out tracking. The reversed method is outside-in tracking, where
cameras in a fixed position are oriented towards the tracked object, thereby creating a tracking
volume surrounding the object (Xinreality 2018).

Figure 3.2: Outside-in tracking on a mocap stage.

Tracking cameras work either with or without markers. Without any markers, certain information
about the tracked object, such as shape, color or patterns, is needed. The known features are
continuesly searched within the recorded image by using comparison algorithms.

12
This can be the geometry, edges and color transitions for a known 3D model. Markerless tracking
can also work when a certain amount of generic information is provided for objects such as a face
or body. This method is used for facial capturing within social media apps such as Snapchat to
create animated selfies (Le 2018). When using tracking markers, they can either be applied to the
tracked object or the surrounding environment, depending on wheter it is a inside-out or outside-in
technique. These markers are then recognized by one or several cameras and run through various
algorithms to determine position and orientation within the tracking area. Markers can be active
or passive. Passive markers are retroreflective and reflect light for detection, which is emitted
by the tracking cameras. In contrast, self-luminous active markers like infrared LEDs flash
periodically to get optically tracked by cameras (Road to VR 2014).

In general, optical tracking can provide precise positions and orientation of the tracked objects.
However, a line of sight between marker and camera is crucial for the system to function. Thus,
sufficient markers and cameras have to be installed, to compensate the loss of one or more
tracking references with the remaining captured data. Since the tracking cameras surround the
tracked objects in a outside-in method, the size of the tracking volume generated by the cameras
is restricted. Active markers allow for a larger volume, because of their improved visibility and
reduced susceptibility for reflections and atmospherics. Depending on how elaborate the optical
tracking setup is, it can be quite time consuming and consist of expensive technology. Another
kind of marker is a predefined pattern such as a QR code. When several of these markers are
arranged in a known way, they can be recognized by the cameras and used to triangulate the
position.

Figure 3.3: Blackbird in use for ’The Human Race’ produced by the Mill

This technique was used by the Mill for their costum virtual production car rig Blackbird, where
a CG car can be tracked in real-time onto the Blackbird rig as shown bellow (The Mill 2017).

13
This is very usefull when the desired car is not yet released by the client or only available as a
digital concept car.

Depth map tracking is a rather new way of tracking. Measuring the time of flight of a light photon
traveling away from and back to the camera is one of various technologies used to calculate a
depth map. The captured data can then me processed into a 3D model of the scene. However, a
depth map lacks accuracy and requires more computational power compared to any fixed tracking
marker method. This technique is still very useful for gesture recognition and for more scientific
work. For instance, it is used for Microsoft’s Kinect and LiDAR instruments (Road to VR 2014;
LiDAR 2018).

Choosing the most suitable tracking system depends on various factors like the size of the tracking
volume, required viewing direction, environmental conditions, low latency, high accuracy, budget,
time, necessary degrees of freedom and other factors depending on die individual project. The
term ’degrees of freedom’ (DOF) plays an important role when comparing tracking technology.
The DOF of an object describes its ability to move around in a three-dimensional space, which has
up to six DOF. Three of those represent translational movement along the x, y, and z-axis, while
the remaining three represent rotational movement designated as yaw, pitch, and roll (Snyder
2016). While it is crucial to be able to move around in 6 DOF, it might be necessary to lock
certain movements in order to imitate the camera movement on e.g. a tripod or rails.

Figure 3.4: The six degrees of freedom.

Often times manufacturers end up using a hybrid solution, which combine two or more tracking
technologies to compensate the respective deficits. This sensor fusion can be found in VR
products like the HTC Vive Lighthouse inside-out tracking system, which uses magnetic and
inertial tracking with a gyroscope, an accelerometer, and a magnetometer to measure rotation, as
well as optical tracking with photosensors and infrared lasers for positional tracking.

14
The spacial resolution is very accurate at 0.3mm and the latency very low at 22ms, which is
necessary for effective gaming products (Niehorster, Li, and Lappe 2017).

Thanks to the success of VR technology over the last three to four years, a large amount of effort
has been put into developing new positional tracking methods. Since head mounted displays
(HMDs) are now mainly produced for private end consumers, the price is rather inexpensive
compared to established motion capture tracking technolgies used within the VFX industry.
Furthermore, these technologies can be found in many virtual production prototypes of succesfull
VFX studios. For instance, ILM used a Vive controller for their VCS for ’Rogue One’ (2016) and
Steven Spielberg scouted his locations for ’Ready Player One’ (2018) with an HTC Vive HMD
as can be seen in the figure below.

Figure 3.5: ILMs virtual camera system for ’Rogue one’ (left), Spielberg VR scouting for Ready
Player One (right)

15
3.2 Real-Time Graphics

Besides an accurate and functioning motion capture system, the virtual production process
requires a highly efficient real-time graphics engine. A traditional 3D animation production
consists of various steps, the main ones being modelling, rigging, animation, texturing, shading,
lighting and rendering (Dunlop 2014, p. 6). Especially the last step rendering can take up huge
amounts of valuable time during a CGI production since every single frame needs to be rendered
individually. With 24 frames per second, this can take hours, or days, for a large render farm with
hundreds of processors to render. For instance, in Disneys Zootopia (2016) one frame could take
up to 100 hours to render (Lussier 2016).

Figure 3.6: A still from Zootopia before (left) and after rendering (right).

When creating previs, one can already prerender the 3D animation before shooting, so a rough
version of the CG content can than be integrated into the live-action footage while on set. However,
being able to adjust and interact with CG assets allows for a huge amount of freedom for directors.
This is what makes virtual production so valuable, being able to adjust a scene in real-time and
immidiatley see the new output. Beside the real-time factor, the visual quality of CG content in
virtual production, such as physically correct lighting or convincing effects simulation, ensures
better blending between real and virtual worlds for directors and cinematographers.

Fortunately, there is one media sector which has always depended on real-time graphics: the
video-game industry. Game engines make it possible to visualize complex graphics in real-time
by relying on powerful graphics processing units (GPUs) and an highly optimized render process.
Traditionally, a game engines is a software developement environment used for the creation
of video games. They exist in a variety of features to assist game developers with common
game-related tasks, like rendering, animation, physhics, networking, scripting and video support
for cinematics (Ward 2008).

16
3.2.1 Origins of Real-Time Filmmaking

Over the last decades the games and film industries have been converging increasingly. Many
games use non-interactive scenes to support the narrative of a game. These cinematic scenes
can either be pre-rendered or rendered in real-time inside the game engine (Taylor 2015, p.
877). Machinimas are another use of cinematic sequences created within game engines. The
term machinima is a word blend of machine cinema and goes back to recorded videos from the
ego-shooter game Quake in 1996. It is described as the art of making animated movies in a
3D virtual environment in real-time (Marino, P. 2004, p. 2). They were born out of fan made
short films consisting of available assets from the respective game. Consequently, game engine
manufactureers startet adding features for filmmaking. Thus, the so called ’machinimators’ were
able to use their own character models, sets, animation and even camera settings to support their
own narrative, which enabled a new form of virtual filmmaking (Elson and Riedl 2007, p. 8). In
recent years, huge VFX companies are making use of the filmmaking tools inside game engines
and preexisting boundaries between the games and film industry fade away.

3.2.2 Recent Developments

Game engines allow for earlier creative decision-making, faster iteration and revision process,
reduced production time, and higher quality. The improvment in quality can be explained with the
fact that ”it’s generally accepted that more iterations lead to better results, so the ability to iterate
faster leads either to more iterations, or time saved. Or both.” (Unreal Engine 2018a, p. 10).
This makes game engines a perfect fit for virtual production requirements. Not only film editing
features and familiar 3D environments similar to DCC tools are necessary, but also effective
import and export features to support existing workflows are in need. Some game engines have
focused on developing such tools for real-time filmmaking over the last few years, which can
be acknowledged by looking at several projects that emerged from these engines. The most
prominent game engines in regard to film-making are Unreal Engine and Unity.

Disney approached Unity in 2018 to create three short films, namely ’Baymax Dreams’, based on
the popular Baymax character from ’Big Hero 6’ (2014) to explore efficiencies using real-time
technology. This required restructuring the familiar pipeline into a parallel, non-linar workflow.
The finished scripts were directly put into Unity as a first previs version.

17
Thus, skipping the traditional storyboard process. Additinally, the previs version served as a
basis to built upon for the final film, as opposed to starting over with a second more sophisticated
version of the film. Consequently, all work went into the final product. Using a game engine
enabled the creator to work in a virtual stageplay. They were able to look at a scene from any
angle, at any time, which helped finding the right choice of framing in an intuitiv and quick
way. All departements started to work from the beginning within the same context. Thus, when
reviewing the current state of the project, every department received feedback at the same time
(Unity 2018a). On the contrary, in a traditionl workflow, one department would hand over their
finished sequence the next department and so forth. This explains the excessive amount of time
needed for implementing feedback in a linear workflow, when every adjustment has to go through
the whole production chain. With ’Baymax Dreams’ the work of various departmentens was
viewed holistically in an edit, emulating more of a bullseye target model instead of a linear
waterfall model, as illustrated in the left figure bellow (Unity 2018a).

Figure 3.7: Unity’s real-time workflow for animation based on iterative cycles.

Another example of a full CG production in a game engine is the Canadian children’s TV show
’Zafari’ produced by Digital Dimension. It is the first broadcast television show that is being
rendered completely inside Unreal Engine 4 with a production schedule of one episode per week,
which is around twelve minutes long (Libreri 2018).

18
According to the show’s creators, reasons to choose a game engine were huge gains in cost
savings, higher productivity, and an increasing quality of final product. Real-time graphics enable
Digital Dimension to experiment with lighting and effects so far that they can create the final
image inside the game engine without any need of compositing. Furthermore, it only takes
them between three to four hours to render an entire episode of ’Zafari’. Thus, the team size
can be reduced in the lighting, rendering and compositing departements, resulting in decreasing
production costs, while the productivity is increasing (Unreal Engine 2018a, p. 9)

Besides stylized animation films, there have been several photorealistic projects carried out in
game engines. One of the most recent ones being ’Book of the Dead’ (2018) created by the
Unity Demo team to showcase their new scribtable render pipeline, which allows for further
customization of post-processing and shading among other features. The uitilized assets in ’Book
of the Dead’ are largely based on photogrammetry-scanned objects and textures to enhance the
realistic look and feel (Unity 2018b). At the same time, Unreal Engine has been working on
extending filmmaking features specifically for virtual production. They enhanced the pipeline
integration by making Unreal Engine compatible with the USD file format, by adding a pipeline
API supporting the programming language Python, by integrating the project managment software
Shotgun, and further features (Mayeda 2018). Additionally, Unreal Engine has released several
demos showcasing state of the art real-time raytracing graphics possible with Nvidias new
RTX GPU technology. This enables high-end real-time rendered images with dynamic lighting,
including realistic reflections and diffuse global illumination as shown in the figure below (Unreal
Engine 2018b).

Figure 3.8: Stills from the ’The Speed of Light’ demo created by Unreal Engine and Nvidia.

19
Aside from full CG projects, game engines are increasingly used for previs. Companies special-
ized on previsualization, such as Halon Entertainment, stated that they have re-trained their artist
from traditional DCC tools to game engines since the real-time factor enables them to rapidly
visualize ideas and immediately adjust assets, lighting, or camera, if necessary (Reagan 2017b).

To conclude, working with real-time graphics inside game engines may require less artists than
the respective project would have needed for a traditional 3D production with conventional DCC
tools. Thus, fewer job opportunities can be a consequence.

Established budgetary models have to be rearranged to fit a non-linear workflow. In general,

more coordination, and more sophisticated production management is necessary to establish end
maintain a new, more complex pipeline, when all departments are working parallel as opposed to
a successive workflow model. However, once an effective, functioning pipeline is consolidated,
the advantages of real-time graphics clearly outweigh the drawbacks. Being able to interact
and change a scene in real-time allows for fast revision. The production itself resembles the
live-action process, where the director has more freedom and control over his project throughout
the whole production. For example, during editing, the camera can still be repositioned, which
would not be possible with time-consumingly image sequences. Though game engines are used
increasingly to produce various CG content, the industry standard is still DCC tools such as
Autodesk Maya. Game engines still lag behind on many features and the capability to achieve the
same high-end output. However, this might change over the next few years, with the high amount
of work being put into enhancing game engines for real-time filmmaking.

20
3.3 Virtual Cinematography

Previsualizing has always been a part of the content creation process. Ansel Adams defines
previsualization as the single most important factor in photography and describes it as ”the ability
to anticipate a finished image before making the exposure” (Adams 1980, p. 1). Continuing with
moving images, directors such as Alfred Hitchcock placed a high value on previsualizing his
films through storyboarding and detailed camera plans (Ferster 1998). Moving on to full CG
films or digital previs as described in chapter 2.1, entire scenes are created inside a DCC Tool
including a CG or virtual camera to frame each shot. A virtual camera is mathematically true to a
physical camera and has the same kind of components such as depth of field, field of view, sensor
size, lens type etc. (Lin 2016a). This enables the application of cinematographic principles to a
computer graphics scene, referred to as virtual cinematography (Okun 2015, p. 445). Various
camera movements, like a dolly, crane or handheld shot, can be recreated inside the virtual scene
by animating the camera.

It seems like a natural progression that James Cameron took the next step of previsualizing by
using a SimulCam for Avatar (2009) as soon as the technology was advanced enough. Essentially,
it is a combination of a motion-captured camera displaying live action footage while the green or
blue screen is already replaced by a virtual environment created in previs. This allows proper
cinematography and evaluation for a shot, where real-world actors or sets are combined with
virtual ones (Sargeant, Morin, and Scheele 2015) .

Figure 3.9: Virtual production on set of ’The Jungle Book’: Mowgli surrounded by blue screen
(left), replacement of the blue screen with previs footage (right).

The SimulCam technology was originally developed at the Swiss Federal Institute of Technology
in Lausanne in 1997 and stands for blending and synchronizing two or more video sequences
(Compute Scotland 2010).

21
Originally, this was mainly used by sports broadcasters to let two athletes virtually compete
against each other (Dartfish 2013). Another use for SimulCam tenchnology can be found in many
virtual television studios today. This enables shows like newscasts, game shows etc. to present
their content in a modern way with virtual elements surrounding them (Lauterbach 2009).

Figure 3.10: Virtual studio for ZDF built in 2009 equipped with motion control cameras (left),
virtual studio for The Weather Channel with costum mixed reality features launched
in 2015 (right).

Going back to the film industry, another milestone for the use of a SimulCam was in ’The Real
Steel’ (2011). The main robot boxing match was previsualized, rendered, edited, and approved
by the director before principal photography began. During the live-action shoot, the camera
operator would see the prerendered fight on his camera display together with the real world
environment on set. This enabled intuitive framing and dynamic camera movement for the fight
scene (Beck 2015, p. 74).

’The Jungle Book’ (2016) used various virtual production technologies to plan and execute the
final shots. The first step was in preproduction when the director John Favreau determined the
approximate size, scale, and placement of the animals while being fully immersed in the virtual
world with an HMD. Subsequently, Favreau and his cinematographer Bill Pope used a virtual
camera (v-cam) to block out the scene and shoot rough camera ideas on a mocap stage, where
several stunt performer would act as animals. The virtual camera in use basically consisted of
”a custom carbon-fiber rig that incorporated a high-resolution OLED monitor connected to two
joysticks and an arrangement of motion-capture markers, which allowed the device to be tracked
throughout the stage, thereby capturing all of Pope’s virtual-camera movements.” (Goldman
2016, p. 37). Afterwards, a master character animation consisting of motion-captured movements
refined with hand-keyed animation by CG Artists was put together into scenes.

22
Moving on to the virtual cinematography volume of Digital Domain, these scenes could then be
layouted virtually with camera positions and movements. Giving Pope great freedom to test out
possible camera angles, different lenses, etc. before committing to them for the actual live-action
shoot. A custom digital light kit was particularly useful to experiment with various lighting
scenarios according to the cinematographer (Goldman 2016, p. 38).

Figure 3.11: Comparison of a final shot (left) and the respective live action shoot (right) of ’The
Jungle Book’.

On the acutal set, Pope had a SimulCam viewfinder attached to the camera, which showed him
the actor inside the virtual environment. Taking a look a the figure above, it is clear that having
a previs version of the surrounding environment helps framing the shot instead of blue screen.
This is feasible with SimulCam technology. ”Bill Pope was able to not only direct the framing
of characters in shots — either live in a motion-capture volume with actors, or virtually with
animated characters — but could also direct how dappled light through trees fell on a character,
live-controlling the depth of field, seeing how motion blur might affect an action sequence, and
tonally dictating the overall mood of a shot” (Goldman 2016, p. 41). By using their own costum
render engine ’Photon’ based on the foundation of Unity and MotionBuilder, the lead creatives
were able to see multiple rendered views of the virtual scene. This synergy of specialist working
with costum developed technology allowed for a new creative and collaborative process, where
the digital and real world supplement each other pefectly. Favreau stresses that his collaboration
with Pope was strikingly familiar. “My cinematographer was a partner with me in the same way
he would be in a liveaction film,” the director says. “You scout together, rehearse together, and
then, finally, you capture the images. Lots of times on effects films, you sit over someone’s
shoulder while they work in a box, designing previs. Or in animation, you do layout with an artist.
Here, I incorporated all the department heads that I’m used to collaborating with on live-action
movies” (Favreau 2016, p. 35).

23
3.3.1 Virtual Camera in Full CG

Of course, aside from VFX movies with a certain amount of CG Elements, there are also full
CG films where camera work often gets unnoticed. However, virtual camera systems can also be
applied to fully animated films. As described at the beginning of this chapter, a virtual camera
inside a DCC tool has the same features as a real one. A cinematographer for animated films
considers the same rules of visual language such as framing, composition, camera movement
or depth of field. Yet, there are some differences in the process of creating the final shot.
Traditionally the cinematographic work is done by layout artists imitating real-world camera
movements (Creative Skillset 2018). In live-action, a camera operator is able to add very subtle
movements as well as involuntary ones from breathing or doing a weight shift, which adds to the
authenticity of the camera. Yet, the natural feel of a handheld shot it is very difficult to achieve
simply by putting on keyframe after another. As a solution, the camera movement is captured
with a similar virtual camera rig as used for VFX productions. These virtual camera systems
are essentially consisting out of a screen with motion capture tracking points attached. This rig
was developed to replace the computer mouse which traditionally controls the CG camera inside
the virtual world. The CG camera motion is now driven by the captured position of the virtual
camera rig in real-world space.

Figure 3.12: Patrick Lin working with a virtual camera system at Pixar (left), framing of a shot
from ’Inside Out’ with a virtual 50mm lens (right)

Pixar started this process on ’The Blue Umbrella’ (2013) since they wanted real physical move-
ment for the camera. They continued to use this technology on ’Inside Out’ (2015) primarily
to support the story. Patrick Lin, director of photography (DOP) at Pixar, describes the virtual
cinematography process as follows. First, a shot goes through rough layout, also referred to as
previs, where the 2D storyboard version is translated into a 3D environment.

24
Creation of a virtual lens kit, deciding on technical aspects like aspect ratio, rough animation
blocking, lighting, and preliminary camera movement are all part of this phase. When ready,
the rough layout is handed over to the following CG departments. In the final layout phase, the
previs shots are swapped with the completed ones including final animation and effects. The
layout department creates a final camera pass, which is where virtual production can be applied
(Badgerow 2012; Lin 2016a). Live-action and animation cinematography are very similar in the
sense that they are both trying to tell a story visually with the same factors such as framing, lens
choice, camera angle. Additionally, one can react to characters both in live-action productions
and in CG films in the final layout phase, where animation has been integrated already. However,
the big difference is the spontaneity to reacting to light. Seeing the final rendered version of a
lit scene is still at the very end of production with conventional DCC tools. Thus, a simplified
description of the difference between live-action and full CG films by Patrick Lin (2016b) is:

”In animation, it is camera, action, lights, instead of lights, camera, action.”

Though, in view of recent developments regarding real time graphics inside game engines,
creating a live-action film and producing a full CG film become more and more alike.

25
3.3.2 Recent Developments for Virtual Camera Systems

Over the last ten years, various virtual camera systems have been developed. The devices range
from high to low budget and different areas of use. This section highlights some of the most
recent ones.

Both Vicon and Optitrack are studios specialised on motion capture technology. Their work
ranges from movement science, VR, object tracking, to motion capture for film and television.
Optitrack developed the Insight VCS, which works within a mocap stage constructed out of
their self-developed tracking cameras. It is compatible with Autodesk Maya and MotionBuilder
and has been used by previs companies, such as Halon Entertainment. Vicon has built a similar
virtual camera tool, which has been used to film Siren, the digital human created by Epic Games
(OptiTrack 2018; VICON 2018). Both systems work with outside-in tracking and passive markers,
they require expensive equipment and time-consuming preparation.

Figure 3.13: Virtual camera by Vicon (left), Insight VCS by OptiTrack (right)

One leading VCS at the moment is Expozure from Digital Monarch Media. The company was
established in 2014 by Wes Potter and Habib Zargarpour, both having years of experience in the
film and gaming industry. They serve as a contract engineering and software development firm
to support predominantly high-end VFX studios with virtual production tasks (Digital Monarch
Media 2018). Potter (2018) states that “the goal is to bring the creative process back into the
shots on set as if it was all practical shooting ... bringing directors front and center with their
films again”. Since they started out, they have worked on some of the biggest virtual production
movies, such as ’The Jungle Book’ (2016), ’Blade Runner’ (2017), and ’Ready Player One’
(2018). Expozure is their virtual cinematography platform. Its API is open, thus, Digital Monarch
Media can adjust to whichever systems their clients are using.

26
However, the graph node editing system is based on Unity (Zargarpour 2018). Features embedded
into Expozure are, for example, rig and lens simulations, real-time lighting and scene manip-
ulation, shot playback, SMTPE timecode, data export as well as import, character and scene
streaming, and real-time communication to multiple Expozure and DCC instances.

Zargarpour demonstrated Expozure at FMX, Siggraph and IBC 2018. Showcasing various
features, among which the multi-user control stands out the most from other VCSs. This feature
enables collaborative work in virtual cinematography, e.g. one user can move around the camera
while the second one pulls focus. For the hardware setup, they have been using a Lenovo
Phap 2 Pro smartphone with the Google Project Tango tracking platform. Together they offer a
built-in time-of-flight camera, an IR projector, and a fish-eye motion camera, which allows for
motion tracking, depth perception, and area learning. This technology provides high accuracy,
theoretically, the capture volume can be as big as the respective room, no additional hardware for
tracking is needed, but Tango has been discontinued by Google.

Figure 3.14: VCam development of Digital Monarch Media

Besides Expozure, they provide further virtual production tools functioning together with Ex-
pozure, namely Logos, a logic node engine, Hermes, their production network environment,
and Cyclopz the virtual camera operating system for AR technology (Digital Monarch Media
2018). Digital Monarch Media is currently leading the development of virtual camera tools as an
independent company since they deliver full service to large-scale VFX Studios.

They have worked with ILM and Digital Domain to realize Ready Player One (2018), which was
another milestone for virtual cameras and virtual production in general. Since half of the film
takes place in a virtual world it stands to reason to produce it in a similar way.

27
Besides a virtual camera rig and SimulCam, which has been described in the previously mentioned
projects, Steven Spielberg had a virtual camera inside VR to scout and create his shots. He would
put on an HMD and use a Vive controller as the input device to frame his scene. When he
wanted to change the virtual set, an artist next to him could make those adjustments in real-time
(Cofer 2018). Furthermore, VR was used to prepare the actors and help them visualize their
surroundings. Spielberg states that ”it is confusing for any actor or director to walk onto a bare
naked set and try to imagine what is there. I asked each of the actors to also put on the headset
and enter the virtual set, so we didn’t have to imagine” (Spielberg 2018). Additionally, he would
watch the actors perform via a VR Headset, as shown in the figure below.

Figure 3.15: The different methods of virtual production for ’Ready Player One’ for Steven
Spielberg.

Besides external virtual production providers Industrial Light and Magic (ILM) has an entire
department dedicated to producing such content. ILM created ILMxLAB, which is their inter-
active sandbox for any virtual content (ILM 2018). Since the studio is equipped with high-end
technology to produce all of their projects in various media fields, such as film, VR, AR, games,
they are constantly working on new tools to improve the production. For ’Rogue One: A Star
Wars Story’, the director Gareth Edwards was able to frame full CG shots with an in-house
developed virtual camera system (Anderton 2017). The VCS consists of an iPad connected to
game controllers and an HTC VIVE controller as tracking device.

28
Figure 3.16: Virtual Camera System used for Rogue One (left), VCAT hardware setup for the
3ds Max plugin (right)

A similar setup to ILMs VCS has been developed by the small company Marui in 2017 called
VCAT. Marui are using an android tablet with a Vive tracker attached. However, their plugin
supports both the Oculus Rift and the HTC Vive tracking systems and all android devices with
WiFi. The plugin is compatible with Autodesk 3ds Max (Marui 2017). It appears to be a good
approach for a low budget VCS except for the limited compatibility with DCC tools. Though, the
VCS needs to be tested for further assessment.

Thanks to VR technology, further developments with VCSs have been made. Using VR tracking
systems seem to be the obvious choice to develop low budget prototypes. Since these devices
derive from the VR world, they are also being used in this environment. Kert Gartner, who
specialized on creating third-person VR game trailers, developed a virtual camera rig, where
a Vive controller and an iPhone are attached to a steadicam. Additionally, he uses an Xbox
controller to create more cinematic shots compared to the handheld ones. This enables him to
shoot smooth VR trailers with different authentic camera movements (Gartner 2018).

Owlchemy Labs, a VR and games studio from Texas, has developed a ’Mobile Spectator Camera’.
This, allows users to see their friends in VR through a smartphone (Owlchemy Lab 2018). The
Mobile Spectator application uses ARCore, Google’s AR platform.

The motion tracking is a combination of optical inside-out tracking using the smartphones camera
and inertial measurements from the devices integrated measuring instruments. Visually distinct
feature points are detected by the camera and used to track a change in location while the inertial
sensors measure the position and orientation relative to the calibrated starting point (Google
2018a).

29
The application receives a video stream of the VR scene via WiFi from the PC that is also running
the VR game. The spectator can see the players avatar, while the player can see a CG version of
the camera in VR to enable interaction (Owlchemy Lab 2018).

Figure 3.17: Mobile Spectator by Owlchemy Lab (left), Kert Gartner’s virtual camera setup
(right)

Taking everything into consideration, there are many promising approaches to virtual camera
systems. However, one needs to take into account, what the VCS will be used for and which
features have higher priority than others. Factors such as being able to freely move around,
positional accuracy, available budget, required features of the virtual camera and compatibility
with specific softwares, need to be defined. Another importany factor is, which technology is
already available to the user. If a VR tracking system is already set up, this might be the best
solution, or like Steven Spielberg in Ready Player One (2018), one can use the motion capture
system, which is already used to record the actors movements.

30
3.4 Demands for Virtual Production Technologies in
Commercial Produdctions

”As artistic demands on computer graphic technologies continue to increase in the

face of ever-tightening schedules and smaller budgets, the film and television industry
is undergoing a new paradigm shift. This shift is being driven by real-time technology.
The next evolution in content production will unquestionably rely on being real-
time, interactive, immersive, programmable, non-linear, and viewer-customizable.”
(Unreal Engine 2018a, p. 2)

Thanks to the recent progress concerning tracking technology, with AR and VR tracking systems,
and the increasing quality of real-time graphics, driven by game engine manufacturers, the
potential for new virtual production tools is flourishing. Being able to work on full CG projects
interactively and immersively opens up new creative freedom. Though these tools are deployed
more and more often, they are mainly used within the high-end film, games and TV industry.
However, with decreasing hardware costs and facilitated access to fitting software such as game
engines, virtual production tools can be used for projects with a wider range of budgets. One
industry being in high demand for improvement is the advertising industry. INFECTED, a
post-production and CG content studio for commercial work has experienced the ever-increasing
demand for high-end CG content in commercials over the last decade with tight scheduling and
budgeting restraints. Thus, introducing virtual production technologies and its benefits into the
advertising industry for smaller studios could enhance working methods just as it does for the
high-end film, games and TV industry.

INFECTED produces various CG content both for conventional output formats such as TV,
cinema and online commercials, and more modern output formats like real-time content for
VR or AR applications. For conventional work, as well as AR and VR trailers, camera layout
and animation is always required. Consequently, the use of a virtual camera system appears to
be a suitable start to introduce virtual production into the preexisting workflow at INFECTED.
However, there are no purchasable virtual camera applications available yet. At the same time,
developing a customizable application allows for more adaptability for the respective projects and
tasks. Thus, the next chapter focuses on the development of an in-house virtual camera system
prototype specifically for INFECTED.

31
4 Developing a VCS Prototype

In July 2017 INFECTED started developing a virtual camera system called ALICE RT. The goal
of ALICE RT is creating a virtual production tool, which will improve the production of full CG
commercials. The application consists of two components: first, smoothly streaming a 3D scene
from a game engine to a motion tracked tablet computer, while the tablet acts as a virtual camera
within the scene. Secondly, the tablet serves as a remote control to apply changes to the 3D scene,
such as camera settings, lighting, and scale. Depending on the production, the recorded camera
movement can be saved and exported to conventional DCC tools or postprocessed and finished
within the game engine.

In contrast to a virtual camera in a 3D software, the operator can move freely and intuitively
within a real room and explore any shot size and perspective via the screen of a tablet computer
with a tracking device attached. This allows directors, DOPs, CG artists or other creatives to
share and realize their vision for the project, by creating detailed references or even final camera
movements and compositions.

The following sections will showcase the required components and features necessary to create
such a prototype. Subsequently, an evaluation of ALICE RT will point out still existing deficits
as well as further potential areas of use for a virtual camera system in advertising.

4.1 User Interface

Aside from the actual functionality of an application the user interface (UI) is one of the most
important aspects. A human-computer interface is the point of intersection and communication
between a human and a computer. The aim is to achieve intuitive human-computer interaction,
where the human can effectively operate a device.

32
Hence, a good UI design is crucial to maintain a user-friendly interface. In order to create a
UI that is both operational and simple to use, the design should be a balance between technical
functionality and clear visual elements. (Sodiya 2014, p. 19)

There are various types of user interfaces, differentiated by forms of input and output. Relevant
for a virtual camera system are the following ones. A graphical user interface (GUI), which
receives input via devices such as a mouse or keyboard and outputs information graphically on a
display. Another UI is the touch user interface, which is essentially a GUI using a touchscreen
display for input and output (Sodiya 2014, p. 20). For optimal interaction between the user and a
virtual camera system, necessary requirements should be clarified, in order to select fitting input
and output devices.

4.1.1 Required Input and Output Devices

The idea of ALICE RT is to create a virtual camera system that supports the creation of fully
animated films, by making the cinematography process more intuitive and immersive compared
to a traditional workflow. There are different methods to built virtual camera systems, yet some
components are always needed while others depend on the area of application.

In order to let real-world movement drive a camera inside a virtual space, all motion needs to be
recorded. Being free to move is an important factor when the cinematographer is trying to find
the perfect angle and camera movement within a shot. Thus, a tracking device should incorporate
six DOF for full flexibility. The tracking volume needs to be big enough to be able to easily move
around objects like for example a car, or other common objects of utility, which are often used
in advertising. Subsequently, the virtual camera rig should be tracked wirelessly to avoid being
hampered by cables. Since even very subtle camera movements are crucial when filming a scene,
the spatial resolution has to be precise. Another factor is the latency of tracking technology, which
has to be kept to a minimum, so the virtual and real world are as close to in sync as possible.
Otherwise, when the user moves the VCS and it takes several seconds for the virtual image to
catch up, the option to react to animation would be compromised.

33
A significant factor for navigating through a virtual scene is the correct perception of the virtual
and real world. At best, this is achieved from a user-specific, egocentric perspective, e.g. provided
by a tracked tablet device or an HMD. These devices enable to move freely and synchronous in
the real and virtual set (Helzle and Spielmann 2015, p. 9). The term spatial input was defined in an
early study about the development of mobile 3D user interfaces in 1994. These interfaces can be
described as spatial input devices, which allow manipulation and navigation of a virtual space like
a virtual camera system does. The study concludes, that spatial input devices with touchscreens
enable the user to perform direct 2D manipulation tasks independent of their position within
the virtual space (Hinckley et al. 1994). Since then, various hybrids of 2D and 3D interaction
have been developed. Using a touchscreen device such as a tablet computer enables mobility,
high-resolution imagery, touch-based input and shared visual output for everybody involved. Of
course, in regard to recent developments with HMDs over the last three to four years, choosing a
VR Headset as UI would be another possibility. A virtual camera inside VR could look like the
one Steven Spielberg used for Ready Player One (2018) as shown in the figure below.

Figure 4.1: Steven Spielberg layouting shots in VR for Ready Player One (2018). A rectangular
frame attached to the controllers is used to layout shots.

However, an HMD has certain disadvantages compared to a tablet computer when used as a
VCS, which need to be taken into consideration. First of all, HMDs are still an unknown territory
for many people, being completely sealed off from the real world might cause uncertainty
and insecurity towards their surroundings. The isolation also complicates communication with
collaborators. Furthermore, having a visual connection between the virtual and the real world
helps to navigate inside a room, when executing camera movements. Though, the process of
editing a virtual set before shooting would make more sense inside VR.

34
Even though this might change within the next years of technological advancement especially in
regard to augmented reality (AR), at the moment using a tablet device seems to be the best option
for creating virtual camera movements.

In general, the price of each component needs to be considered, since the application is not part
of a big budget production but rather a research project, which has been developed between
commissioned work.

4.1.2 Selected Hardware and Software Components for ALICE RT

The following hardware and software components were chosen after taking the necessary require-
ments and budget limits into account as well as evaluating similar virtual camera systems.

Hardware

The hardware should consist of lightweight, robust and intuitive input and output devices. As
described in the previous chapter, a touchscreen tablet device like an Apple iPad Pro complies
with these requirements. The 12,9-inch display offers a well-sized window into the virtual world.
Since it is a consumer product it is still within the price range, easily accessible and has a large
number of features and extensions available both for hardware and software. One feature, which
might become crucial in the future is the ARKit tracking technology, which was officially released
in 2017. Currently, it does not fulfill the demands for ALICE RT, mainly because the accuracy
is to low. However, when its tracking technology advances, it might become a substitute for
an external tracking system. Another useful extension for the iPad are specifically designed
game controllers, which offer an improved ergonomic shape compared to a bare tablet device.
Moreover, each one of the individual buttons can be assigned to a particular feature. Hence, users
will not need to take their hands off the controller to make adjustments.

In addition to the iPad and its fitting controller, the Vive tracker is attached to the virtual camera
rig to record its movements. The Vive tracker is part of Valves Lighthouse tracking system.
Besides the tracker, two Vive Base stations need to be set up. The tracker provides six DOF
so the virtual camera can be oriented and moved in every possible way. The tracking volume
encompasses an area of four by three meters, which can be extended by adding more Lighthouse
Base stations. Another decisive advantage of the Lighthouse tracking system is that INFECTED
already has the technology setup and is familiar with it.

35
The hardware rig is composed as follows: The iPad is protected by a simple hard plastic case.
The game controller is strapped around the back of the iPad and connected via lighting port. The
Vive tracker is screwed onto a metal bracket, which is glued to the hard plastic case, as shown in
figure 4.2.

The PC is a 64-bit gaming workstation. The graphics card needs to be powerful enough to run
the respective Unreal Engine scene at the desired quality. Besides visualizing the virtual scene
a certain amount of GPU and CPU resources are needed for video encoding as well as a fast
drive to save recorded videos. INFECTED is using a GTX 1080 graphic card and an M.2 card.
Professional gaming switches and WiFi components should be used for ideal network connection.

Figure 4.2: Hardware setup of virtual camera rig ALICE RT

Software

The software development is divided into two parts, the ALICE RT application, and the ALICE
RT plugin. The iPad application is created with the cross-platform framework Xamarin inside
Visual Studio and mainly written in the scripting language C#. The user interface was done in
XAML. The plugin inside Unreal Engine is based on scripting with C++ and Unreal Engines
node based Blueprint system. The choice in favor of Unreal Engine as game engine was made
due to fact that it is one of the leading game engines together with Unity with ever-emerging
new filmmaking features (Unreal Engine 2018a). Unreal Engine enables high-end visuals on a
real-time basis. Another reason for Unreal Enginge is the fact that INFECTED is familiar with
the software, whereas Unity would have required preceded training.

36
Data Transfer

While in use, the virtual camera system ALICE RT establishes two connections between the
tablet app and the Unreal Engine plugin. One transfers the camera remote controls via TCP to
Unreal Engine. Simultaneously, a video stream of the virtual scene is being transferred to the
tablet in H.264 format.

Figure 4.3: Data transfer between ALICE RT plugin and app.

Communication

The communication between Unreal Engine and the tablet application can be established through
IPv4 or IPv6 Network connection. The PC should use a wired connection, while an optimal
connection for the tablet would be a WiFi hotspot with 5 GHz 802.11ac and high signal quality.

The communication is based on the network protocols TCP/IP and UDP. In general, network pro-
tocols define conventions and rules for transmission and communication between network devices
(Baun 2018, p. 148). It makes sense to use both protocols for different tasks to achieve efficient
data transfer. Transmission control protocol (TCP) ensures reliable, ordered, and error-checked
delivery. If a transferred data segment gets lost or damaged, TCP requests a retransmission. This
is useful for transmitting camera settings as well as establishing the connection between ALICE
RT app and a specific Unreal Engine project with the ALICE RT plugin (Baun 2018, p. 151). In
contrast to TCP, user datagram protocol (UDP) does not provide a reliable data transfer. This, is
very useful for fast and efficient transmissions, such as video streaming. For example, if a frame
gets lost during transfer, UDP continues to send the following frames, while TCP would resend
the lost frame which causes video buffering.

37
Furthermore, UDP allows direct data transmission between hosts without any prior communica-
tion as long as they have the same port number. This makes it possible to find Unreal Engine
projects inside the tablet application without having to manually search for them (Baun 2018, p.
149-150). Hence, first of all, the iPad app and the Unreal Engine plugin can find each other via
UDP, then they can connect with TCP, and subsequently the two components can communicate
by using both protocol types.

Camera Settings

Unreal Engine provides two camera actors by default. One with basic and one with advanced
camera settings. Traditionally, they are both meant to visualize games during normal game
view while playing as well as narrative game cinematics. The cine camera actor, which is the
advanced version, offers all required camera controls for ALICE RT. Thus, when camera settings
are changed inside the app, the adjustments are transferred to Unreal Engine and directly feed
into the respective setting inside cine camera actor.

Tracking Data

The VIVE tracker is captured by the Valve Lighthouse tracking system. Valve provides a
SteamVR software development kit (SDK) and detailed instructions on how to integrate motion
controllers into Unreal Engine. The Vive tracker collects rotational angles and three-dimensional
coordinates, which then drive a CG representation of the tracker inside the game engine. Both the
virtual and real tracker are connected to the camera substitutes. Hence, when the VCS in the real
world is being moved, the virtual version is driven by the captured tracking data.

Figure 4.4: HTC Vive Tracker

38
Scene Transfer

To create a 2D image as output, Unreal Engine needs to render each frame, which goes through a
chain of various effects and adjustments. Some are integrated by default such as tone mapping
the high dynamic range colors into low dynamic range that a display can output and some can be
added manually, like e.g. color grading or a vignette (Unreal Engine 2018c). The Alice RT plugin
uses FFmpeg to pick up the rendered frames at the end of that chain, to transcode them into an
H.264 file, and then sends the video file to the ALICE RT app. Since the scene is broadcasted as
an H.264 compressed video file and not as the actual 3D scene with all its elements, the stream
is independent of the complexity of the scene. The other possibility would be, to render the
scene on the iPad, although this would be a too high computational effort for the iPad for more
elaborate scenes. It is very crucial to achieve a video stream with almost no latency, so the tracked
movements of the VCS will match with the video output on the iPad screen.

Save and Export

Screenshots of a specific position can be recorded with the snapshot feature. This includes the x,
y and z coordinates and rotation as well as an image reference. The position is saved as FBX file,
while the snapshot feature within the ALICE RT app simply accesses the screenshot function
inside Unreal Engine to save an image as PNG file. FBX is a 3D asset exchange format owned
by Autodesk in which CG objects and their animation data can be saved. The choice for FBX
is due to the fact that it is one of the most commonly used exchange formats between various
DCC tools (Autodesk 2018). Generally, camera position and animation can only be exported as
FBX file from Unreal Engine while in editing mode. Therefore, it would not be possible to use
the snapshot feature during production, when Unreal Engine is in play mode. Thus, the export
feature is included inside the ALICE RT plugin, so FBX files can be saved during production as
well. Besides a snapshot, the whole scene recording can be exported too. This includes a H.264
file of the captured take and an FBX file of the camera movement.

Another promising file format is USD, the Universal Scene Description format, developed by
Pixar to allow for seamless translation and interchange of assets between multiple mediums
(Pixar, 2017). However, this format is still very new. Thus, software developers are just now
starting to implement import and export features for various DCC tools. Consequently, FBX
might be replaced with USD as soon as it allows for facilitated pipeline integration.

39
4.2 ALICE RT Prototype Implementation

4.2.1 User Interface Design

Designing a GUI is an iterative process, in which features, functions and the face of a UI are
designed, implemented, evaluated and readjusted (Sodiya 2014, p. 31). Developing a fitting GUI
for ALICE RT started with hand-drawn sketches, which gave a base for discussion between the
team members. After establishing a rough design, it was recreated inside Adobe Illustrator and
went through further iterations. Subsequently, the final design was implemented in XAML, an
XML based user interface markup language developed by Microsoft (Microsoft 2017). To ensure
flexibility regarding tablet devices and their associated display size, the design was programmed
responsive with percentage values instead of fixed ones (Gardner 2011, p. 14).

The design concept incorporates a clean style, where only frequently used features are visible
at all times, and the main focus remains on the camera viewport. It is based on the design
principles of Google’s Material Design. This embodies a meaningful, focused, immersive and
coherent design method (Google 2018b). Another influence was interfaces of applications such as
photography apps, film camera displays, and similar VCSs. The design gives the user a familiar
and self-explanatory impression since it is based on known structures. Thus, a user guide is not
essential to work with ALICE RT. However, a short introduction is advisable for efficient use of
the application.

The choice of color is based on the corporate design of INFECTED. The primary color is a dark
gray, which functions as a background for the toolbar and both panels. A bright red color, as
well as a light green and gray, are used sparingly, yet purposefully to indicate icons and their
activity or inactivity. For example, while recording the record button is blinking red, and inside
the advanced settings menu, toggles switch from light gray to green as optical feedback. All Icons
integrated on the toolbar represent their function graphically, thus work without any written text.
Only value based settings such as aperture, focal length, focus distance, and playback timecode
consist of numeric characters to enable immediate information about their current value.

40
 Figure 4.5: UI design of Alice RT.

4.2.2 Setup and Configuration

ALICE RT consists of two components. One being an Unreal Engine plugin, which streams a
virtual scene to an iPad application. The second component is the iPad app, which sends back
information like camera and general scene setting updates. Additionally, the VIVE tracking
system delivers position coordinates to Unreal Engine, tracked by the VIVE tracker attached to
the iPad.

ALICE RT Unreal Plugin

Setting up ALICE RT on engine side begins with adding the plugin to either your project plugins
or engine plugins, depending on whether it should be available to all projects or only a specific
one. The ALICE RT app on the tablet can be connected by enabling the plugin inside the plugin
browser and restarting the scene. Furthermore, certain settings can be adjusted or left at their
default configuration, such as optinal password protection. The connection can be deactivated,
the network port number can be adjusted and a password can be defined if necessary. Saving
setting includes optional fbx and video export of the recorded camera movements, as well as a
preferred target directory. To complete the setup the ’ALICE RT PAWN’ file must be added from
the Unreal Engine content browser to the virtual scene. By doing so, the tablet will receive a
video stream of the scene and the ALICE RT pawn will move according to the VCSs position.
A pawn is an object that can be placed into a virtual scene in Unreal Engine and furthermore
receive input from a controller or the position from a tracking device as in this case (Unreal
Engine 2018d).

41
The ALICE RT pawn position equals the center of the previously calibrated center from the
Steam VR room setup within the real world. This must be considered when placing the pawn
inside the virtual scene since the camera operator will only be able to move within the limitations
of the physical space and the tracking volume. However, the pawn position can be changed at any
time during production.

ALICE RT Application

After installing and running the ALICE RT app on an iPad a login screen appears (see figure 4.6).
If necessary the port number can be adjusted by selecting the settings button in the lower left
corner. During the login phase, the app searches the network for an open Unreal Engine project
with a running Alice RT plugin. Fitting results are listed with the project name and IP address.
The desired project can be selected by a basic tap to the display, which is the most intuitive and
direct method of selection. This method combined with physical buttons on the game controller
is mainly used for interacting with the app. If a password has been established previously inside
the ALICE RT plugin settings, a lock icon will be shown next to the project name and the user
will be prompted to type it in. Subsequently, the user will be forwarded to the main UI.

Figure 4.6: ALICE RT login screen.

42
4.2.3 Operating ALICE RT

The interface is structured in a simple and familiar way as previously described (see chapter 4.2.1
User Interface Design). The main interface is composed of the video stream of the virtual scene
and a toolbar at the bottom of the screen equipped with essential camera settings. The toolbar
camera settings are designated in the following figure 4.7:

Figure 4.7: Toolbar with camera settings.

Snapshot allows the user to capture and save a screenshot and the coordinates of the current
camera position. The screenshot can be used as an image reference, e.g. to swiftly create a
storyboard or shotlist for a project. Moving on to 25 frames per second, the record button starts
or stops recording the camera movement. Captured sessions can be rewatched and reviewed by
selecting the recordings button, which lets a panel with all recorded videos appear. Besides the
shot number, further information such as duration, date and time of capture are displayed. When
in re-watch mode, the playback button starts or stops the playback of a previously recorded
scene. Back in the main interface, the playback button starts or stops the animation of the CG
scene inside Unreal Engine. This enables camera operators to react to the animation within a
scene, to capture moving characters or objects.

To change the depth of field of a shot, one can adjust the aperture by simply scrolling up- or
downwards to in- or decrease the f-stop. The following aperture values are offered for selection:
1.4, 2.0, 2.8, 4.0, 5.6, 8.0, and 16. These values cover most of the conventional full-stop f-numbers
used for cinematography. The focal length can be adjusted in the same way as the aperture.
It would make sense to use a zoom lens for the virtual camera since there is no difference in
quality compared to a virtual prime lens with fixed focal lengths. However, certain prime lenses
are preferably used by cinematographers. Thus, ALICE RT offers the following values as focal
length ranging from wide to long-focus lenses: 18mm, 35mm, 50mm, 85mm, and 105mm. If
necessary, further values for aperture or focal length can easily be integrated.

43
Another value displayed on the toolbar is the focus distance measured in meters. It can also be
adjusted by scrolling up- or downwards. Apart from the main interface and playback panel a third
panel can be blended in, the advanced settings, which consists of further camera and general
scene settings. Advanced settings, recordings, playback, record button and snapshot all function
with a single tap to the respective button. Furthermore, all main settings have an assigned button
on the game controller to intuitively operate the VCS. The keys are assigned as shown in the
figure below.

Figure 4.8: Key layout on game controller.

Advanced Camera Settings

A modification of an f-stop value in a real-world environment affects the exposure of an im-

age. However, this only changes the depth of field inside a game engine. Thus, if brightness
adjustments are necessary one can use the gain slider. Another setting is smoothing, which
stabilizes one’s movements to imitate a steady-cam. Axis Lock enables fixating specific motion
axes similar to a tripod. So, while the VCS has six DOF (see figure 3.4) with the VIVE tracker
attached, using all of them might not be necessary or even counterproductive. If, for example, the
camera operator wants to create a dolly shot where no tilt function is needed, and the horizon
should always be straight, at least two degrees of freedom are not required and would only
be aggravating. Thus, it makes sense to exploit task-specific needs to reduce dimensionality
(Hinckley et al. 1994, p. 4). Under the segment Focus, a center cross and camera grid can be
inserted to support framing. Additionally, one can choose between focus modes such as manual
focus, tap to focus or grid focus. Manual focus equals an adjustment of the focus distance, as can
be seen in the toolbar. Tap to focus is similar to focusing with a smatphone since a basic tap to
the display suffices, to focus the selected object.

44
With this method a ray is casted out from the selected point into the scene and whichever object
collides first with the ray, will be set in focus. Grid focus is comparable to various autofocus
points within a photo camera, where different grids can be chosen from and by pushing a specific
button the camera will focus on the selcted area.

Figure 4.9: The main user interface of ALICE RT with blended in panels.

Traveling and Navigation

The virtual camera is driven by the tracked VCS movement in the real world. In other words, the
translation and rotation of the ALICE RT camera rig are mapped onto the virtual camera inside
Unreal Engine. Navigation feels natural, by using a device with six DOF and visual feedback
since you see where you are moving through the tablet display opposed to navigating a camera
with a computer mouse. However, this movement is restricted by the size of the physical room
as well as the VIVE Pro tracking area, which is currently at least four by three meters (VIVE
2018). One integrated option to maintain flexibility is Scaling the virtual scene smaller, which
allows one to tread a larger virtual area within the confines of the physical room. Another one is
Position Mode, which lets the user reposition themselves inside the virtual space.

Figure 4.10: Operating Alice RT

45
Miscellaneous Settings

Inside the Setup tab format adjustments can be made. This includes aspect ratio, the transparency
of the letterbox, an anamorphic lens look adjustment and a camera shadow, which can be used
when using ALICE RT as a light previs tool. Moreover, further data, e.g. sensor size, shot number
or timecode, can be blended in or out and the ALICE RT session can be stopped, which brings
the user back to the login screen.

46
5 Evaluation

In order to asses the demand and interest for virtual production tools and evaluate the ALICE
RT prototype, a test has to be conducted. First, the test scenario and its limitations are described
in section 5.1 as well as a specification of the aspects to be examined. Section 5.2 will then
showcase the results of the survey separated in the user group opinion and the personal review of
the author.

5.1 Test Scenario for Evaluation

The test was conducted inside the showroom at the INFECTED office in Hamburg, which is
primarily used for testing and presenting VR projects, either internally or for potential clients and
agencies. Thus, the room is already equipped with the VIVE Lighthouse tracking technology, a
workstation, which meets the requirements as stated in chapter 4, a 50” flat screen TV as monitor
fixed to the wall and enough free space to move around the tracking volume of three by four
meters. The test was prepared by Dennis Gabriel and Anna Eschenbacher. Dennis Gabriel works
as real-time supervisor at INFECTED and provided the newest version of ALICE RT at the time
of the test. Anna Eschenbacher executed and supervised the trial, and set up the subsequent
survey for all participants.

Each user was able to test ALICE RT individually for approximately twenty minutes according
to the following scheme. Initially, a short description of virtual cameras systems and possible
areas of application was given. Followed by hands-on experience with the ALICE RT prototype.
The various settings together with the operation of the device were explained as an introduction.
Each participant went through three different virtual sets, all real-time rendered inside Unreal
Engine. Beginning with a static car surrounded by a natural environment.

47
This setting has been created by INFECTED to demonstrate the current real-time capabilities of
Unreal Engine, which can be presented to potential clients. It is suitable as the first scenario, due
to the simplicity of virtual elements as well as lack of motion, which enables the user to focus on
the virtual camera itself and to not be distracted by a complex, moving scene. Each participant
was encouraged to walk around the virtual world, e. g. walk inside of the car and try out different
camera angles as well as features of ALICE RT such as focal length, aperture, focus methods,
and scene scaling.

Figure 5.1: Test scenario 1, VW T-Roc

When all features were tested and a certain familiarity with the VCS was reached, each user
moved on to the next scenario. The second virtual set consisted of a dark and moody cellar room
in which a fight scene would take place. The entire scene was created by Unreal Engine and
is provided as free of charge learning content for cinematic sequences (Unreal Engine 2018e).
Since the scene has animated characters, the playback feature of ALICE RT can be tested. The
participants were able to frame the animated fight scene while moving around the tracking
volume to find various camera perspectives. The last scenario was a slow-motion showdown
scene, originally provided by Unreal Engine as a free VR Demo (Ivey 2015). This scene was
suitable for experimenting with orientation and framing of wider shot sizes on a virtual set since
it was not restricted to a room, but a large urban street. Furthermore, the user could react to the
ongoing animation in a calm way compared to the previous rather hectic fight scene.

48
 Figure 5.2: Test scenario 2, the fight scene (left), test scenario 3, the showdown (right)

Subsequently, each participant received a previously composed questionnaire to obtain the

required information, which is defined in the following paragraph. ALICE RT is still in a
prototypical development phase, hence, various functions were not working yet, even though they
were already integrated into the user interface design. This includes extended camera settings
such as axis lock and smoothing, the position mode as well as the playback feature of previously
recorded scenes. However, these features were not absolutely necessary to get first hands-on
experience with ALICE RT and to grasp the potential of a virtual camera system. Additionally,
the video transfer of the Unreal Engine scene to the ALICE RT app still suffered from a too high
amount of latency, presumably caused by time loss for encoding and streaming data on a high
bandwith. This led to the decision of simply showing the live video on the flat screen connected to
the workstation and no video on the iPad. Fortunately, this method works very well except when
facing the opposite direction of the flat screen. Each participant was informed about the planned
additional features and asked to take them into consideration when filling out the questionnaire.

Figure 5.3: Snapshots of the ALICE RT test.

49
To plan an evaluation one has to define under which aspects the test should be examined. The
required information can be divided into two aspects. First, the demand for virtual production
tools in general and secondly the VCS prototype ALICE RT specifically. Since the advantages
of virtual production mentioned in this thesis are either gathered from books, papers or reports
from various large scale film productions, it is necessary to prove whether they are confirmed by
artists mainly working within the advertising industry. Assuming these advantages are true, a
demand for improvement from conventional VFX productions is needed. Regarding the VCS
ALICE RT, aspects such as usability of the UI, benefits of existing, intended or still missing
features, ergonomics of the hardware components, and orientation within the virtual space need
to be examined. Furthermore, the need for possible alternatives such as VR solutions or different
hardware setups should be surveyed. At last, it is important to check whether a general interest in
working with ALICE RT on future projects exists.

50
5.2 Results

5.2.1 User Group Review

The user group consists of various INFECTED employees, fellow students and other creatives
from agencies. Each user was able to test ALICE RT as described in chapter 5.1 and filled out
a short questionnaire to feedback their experience. The questionnaire was distributed as a link
to an online survey, which was filled out by the user group within 24 hours after the test. In
total 18 people participated in the survey. Their line of work varies between live-action film,
animation, VFX, interactive media and games, post production, and IT. The job titles are as
follows: (senior) CG artists, business development manager, colorists, media design apprentices,
motion graphics artist, media engineer, animation director, managing partner of the agency Jung
von Matt, software developer, producer, editor, lead compositor, and audiovisual media students.
Thus, their work experience ranges with 44% still being students, 33% have experience of more
than five years while four participants classify as young professionals with up to five years of
work experience.

Figure 5.4: Level of experience (top), fields of experience (bottom)

51
Demand for virtual production

The beginning of the survey focuses on the general demand for virtual production and previous
experience in this area. When asked what seems to be in need for optimization in a traditional
VFX pipeline the users could select from a set of predefined answers. They could agree on the
facts, that the workflow is too linear and goes through too many interrations, the feedback loops
are too excessive, and budgets, as well as time, are insufficient for the desired end products.
The user group mostly disagrees with the statement, that traditional VFX productions have little
creative environment and suffer from bad communication. Subsequently, the conventional VFX
pipeline is not considered in an entirely negative way. Though, an improvement in various areas,
such as pipeline and workflow optimization, is clearly in need. Surprisingly, the creative freedom
in existing VFX pipelines seems to be sufficient for the user group, although, one of the main
benefits of virtual production is the enhanced creative environment.

Figure 5.5: Proposed areas in need for optimization in a traditional VFX pipeline

Except for one participant, everyone was already familiar with the term virtual production.
Additionally, a third of the user group stated that they have already worked within a virtual
production environment. When asked about the advantages of virtual production time saving
and facilitated communication was listed most frequent. Other keywords listed were intuitive,
effective, experimental and interactive working, merging real and virtual world as well as lower
expenses. One user stated that “virtual production enables creatives simplified access to previously
rather technical task, such as camera animation”.

52
The mentioned advantages are confirmed by the following task, where participants could either
agree or disagree with several benefits of virtual production, which are commonly found when
researching virtual production projects. Only the aspect of higher quality divides the opinion of
the user group, which can assumingly be traced back to the fact that real-time graphics are still
lacking some of the quality time-consuming rendered images deliver. However, the quality of
virtual production should not be reduced to real-time graphics, but the enhancement of the entire
production, which includes more time to creatively shape the final rendered images.

Figure 5.6: Assumed advantages of virtual production

Prototype ALICE RT

The second part of the questionnaire concentrates on the assessment of the virtual camera system
ALICE RT. First, the participants were asked about the experience with related tools, to better
evaluate whether previous knowledge is necessary or helpful. 44% have preceding experience as
camera operators. 56% of the participants have used a virtual camera inside a DCC Tool, while
no one has used a virtual camera system such as ALICE RT before. When filtering for experience
with cameras in general, either a virtual camera in a DCC Tool or as camera operator, 13 of
the 18 participants state previous know-how. The user group agrees on the fact that ALICE RT
constitutes an advantage over conventional hardware input devices such as a computer mouse
and keyboard. However, 78% would like to have a more traditional camera rig, e. g. a shoulder
rig, in addition to the current game controller iPad setup. One participant stated that “it really
does depend on the project” and another would like “all possible camera configurations” such as
a follow focus ring.

53
The functionality of iPad and game controller combination is neither terrible nor excellent with
3.67 points on a scale from one to five. This might be due to the fact that the game controller
extension does not fit perfectly tight to the iPad, which causes a slight feeling of instability.
Moreover, traditionally trained personnel might find game-like input devices alienating. Other
game controllers for iPads need to be tested, so a more robust version of the virtual camera rig
can be created. Aside from the instability, the virtual camera rig seems to be a good basis, which
needs to be adapted according to the needs of the respective project. The orientation within the
virtual world was rated at 4.06, which is a rather good result. However, this should again be tested
as soon as the scene transfer feature is working, which should produce even better results.

Another approach to virtual production tools are HMDs, so one can be fully immersed within the
virtual environment. Only five out of the eighteen users would prefer an HMD. Fifteen have tried
out VR before, while seven of those have experienced motion sickness. Thus, motion sickness
can be one of the reasons against choosing an HMD, though two out of the five that would prefer
an HMD have already experienced it. Further reasons against choosing to be fully immersed in
VR can be the isolation from the real world, which can lead to communication problems on set.

Figure 5.7: Results for the user interface of ALICE RT

To improve the user interface of ALICE RT different aspects have to be assessed. Each aspect is
rated from one to five, while one represents very bad and 5 stands for very good. In general, the
user interface is rated positively ranging between 3.94 to 4.22. However, the standard deviation
counts an average of 0.91, which shows that the user group has diverging opinions about specific
UI categories. With intuitivity being rated at 3.94, further testing on where to assign which feature
to the game controller is necessary. Clear arrangement is graded with 4.11 points, slightly better
than comprehensibility with 3.94. Thus, a more thorough introduction might be needed as well as
possible labeling of the game controller functions to achieve quick understanding.

54
This can be achieved by creating a video tutorial to introduce all features. A more elaborate
method would be an integrated user guide when first working with the application. Content and
features of ALICE RT are rated at 4.17, which proves the usefulness of the developed and planned
features. Further suggestions are listed in the following paragraph. The UI design of ALICE
RT was well received with 4.22 points, same as the quality of entertainment. This could explain
the result of the following question, where the user group was asked about further interest in
spending time with the virtual camera system. 94% of the participants replied with yes.

Particularly relevant for further development and improvement of ALICE RT are suggestions
for additional features. The proposed functions can be grouped into the categories navigation,
camera, and UI. Navigation includes a reset function for the position and height within the virtual
scene. This proved to be an essential necessity, as discovered during the test scenario. Otherwise,
when changing the scene scale value several times, one can end up floating a few meters above
the ground, which might not be intended. Another suggestion is a save function for a number of
positions and their respective orientation. This would enable the user to rapidly switch preselected
positions, which can be very helpful when trying to capture a scene from different angles.

The last requested feature concerning navigation is a live scene editing feature to move assets
around. Currently, ALICE RT is primarily a camera tool not necessarily meant for scouting a
scene. However, scene adjustments can quickly be made within the Unreal Engine editor. Adding
features for scouting is definitely intended, though fine-tuning the camera tool is the highest
priority at the moment.

Proposed camera features are an imitation of dolly and crane movements, which is already
planned. Additionally, the path of a camera movement should have a playback function in order
to redo the pan movement for example. At the moment the camera movement can only be
reviewed wihtout the ability to adjust single components. The orientation of the camera needs to
be independent of the camera position to achieve this feature, which seems very useful to get the
perfect shot. The UI category includes further extensions such as a motion-sensitive wheel to
pull focus organically. Another extension would be a second input tool to enable adjustments by
multiple users. This can be used for sharing tasks, for example, one user can drive the dolly while
the second user pans the camera. Suggested adjustments within the existing UI are a two finger
zoom on the iPad screen to change the focal length and directly editable values for f-stop and
focal length as opposed to the currently predetermined values.

55
A completely different idea concerning the UI is using an HMD and attaching a frame to a
controller to frame shots, as it was done for Ready Player One (2018) by Steven Spielberg.

Conclusively, the participants were asked if they would use ALICE RT for a future production
and in which area. Seventeen out of eighteen would use the VCS, while one user explains his
decision against ALICE RT with the fact that it is not fully developed yet. Out of the other
participants, seven would apply the virtual production tool for previs, five for full CG productions
and the remaining five would use it for both areas.

Additional Testing

Aside from the user group review, mentioned in the previous section, an expert group of pro-
fessional camera operators and directors of photography (DOPs) was able to test ALICE RT at
an early state in June 2018. The test production was very similar to the one mentioned above.
Test scenario one, the car, and two, the fight scene, were tested by the four camera experts.
Every one of them spent about an hour on testing ALICE RT, during which suggestions and
observations were noted by Dennis Gabriel. At this date, the state of the VCS was very early,
though, sufficient enough to recognize the potential of such a device. The expert group consisted
of Fabian Hothan and Malte Goy, two very experienced DOPs, who commonly work on high-end
car commercials and the two videographers Lukas Willasch and Neels Feil, who mostly deal with
smaller commercials with either handheld or steadicam shots.

Several problems exist within their current, established workflow. First, they usually do not see
the car until the actual shooting day. This is due to the concealment of new car models and
the tight schedules within the advertising industry. This gives DOPs very little time to create a
suitable light setup and no time to experiment with framing. Thus, they only capture shots that
will definitely work. All camera operators state that the time constraints are a major problem for
their work. Furthermore, Hothan and Goy agree that the current previs available to them is too
rough and vague to understand the features and shape of the respective car. Another problem is
the fact that cars get replaced often times in post production. During these CG productions a
similar sized model equipped with tracking markers, and hopefully in the same color, is used for
the live-action shoot, which will then be replaced by the correct model by 3D and compositing
artists.

56
In this case, DOPs can only guess how the light will look on the CG model. In addition, the
clients request more involvement especially during commercials which are partially CG. More
and more car commercials end up being full CG productions, either because the confidentiality of
the car model or because the intended film is to complex or surreal for a live-action shoot. These
productions cause a slight decline in work assignments for DOPs.

Figure 5.8: The VW T-Roc inside Unreal Engine with surrounding light sources.

While testing ALICE RT the following feedback was given by the expert group in regard to the
above-mentioned problems. The virtual production tool is very much suitable for previs. Ahead
of production as well as during the shoot to plan the next camera position. Furthermore, the
high-end DOPs would like to use it as a tech previs tool to design the light setup and see whether a
light source, unintentional shadows or the reflection of the camera would be in the frame. The UI
is well received due to its big display. However, the two videographers would prefer a shoulder rig
or camera handle as an extension. Further propositions are adding the look of an anamorphic lens,
which is already implemented in the UI design. Other wishes are preanimated car movements
like ’accelerate’ or ’fall back’ and a ’hook mode’, where the camera is linked to the movement
of the car. Thus, when the camera is not being moved, only the virtual environment passes by.
In general, the feedback was very positive. They described it as a tool with a huge amount of
potential, many areas of use, improved insight for clients, and an opportunity for better and faster
previs (Hothan et al. 2018).

57
5.2.2 Personal Review

The test production allowed for feedback from a diverse group of creatives. Several oberservations
can be gathered from the test production. First, the workflow inside Unreal Engine is already
working very smootly. The ALICE RT plugin can easily be added to a virtual scene and the
connection establishment between Unreal Engine and the Alice RT application works very well.
Moreover, the VIVE tracker is immediatly recognized and feeds the positional and rotational data
for the virtual camera. The video transfer works already, though, the latency is significantly too
high, which needs to be improved. The functionality of the video transfer has the highest priority
for ALICE RT at this moment. Additionally, intended and newly suggested features should be
implemented to the virtual production tool. The additional features highly depend on the area
of use, which can only be decided when a client chooses a virtual production as opposed to the
conventional production workflow. One problem INFECTED encountered is that clients often
show interest but do not have designated budgets for content inside VR, AR and similar new
technologies, such as virtual production. Thus, a new budget divisions for such content needs
to be created, which proves to be difficult especially in large companies with strictly regulated
hierarchies like automobile manufacturers. However, using virtual production can shift the way a
budget is distributed over a production or even save money, by shortening the production time and
averting unnecessary iterrations, as it did for the canadian children’s TV show ’Zafari’ (2018).

For example, a camera operator, which usually would not be involved with the CG camera
movement and framing, would have to be payed for working with ALICE RT. On the contrary,
the 3D artist would no longer spend time on animating the camera. In an ideal situation the
time to capture the camera with ALICE RT, including the time needed to import and export the
recorded data, would be less than the time a 3D artists would have needed for the same task. In
this situation, the CG artist could focus more on creatively working with the respective DCC
tool. This requires a smooth and steady workflow between Unreal Engine and the DCC tool.
The workflow has not been tested and refined yet, and again depends on the comissioned future
projects. At best, the whole project is designed and executed inside Unreal Engine, like the VW
T-Roc car showcase, so everything can be optimised for a real-time output. Even if most of the
work is carried out inside a game engine, often times the assets are still created with advanced
DCC tools, e.g. Autodesk Maya. Factors such as complexity of the scene, including the polygon
count, need to be considered and adjusted for game engines.

58
Besides the fact that the respective 3D artist will have more time for creative tasks, new job
opportunities open up for traditional camera operators, which previously ceased due to the
convergence to produce more partial and full CG commercials. These professionals bring years
of experience with composition, look development, camera movement, light setup, choice of lens
and further camera knowledge to the project. This knowledge might by familiar with very few
CG artists as well, though, usually is not the case since their profession is a different one. On
large-scale CG projects most artists have their area of expertise, whether that may be animating,
rigging, rendering or any other CG task. These projects also have layout artists specialised on
camera related tasks. However, in smaller companies many artists serve as generalists or at least
specialise on several tasks. Thus, it is even more important to find a solution like virtual camera
systems, which can be operated by trained DOPs. Sönke Heuer (2018), colorist at INFECTED
and participant of the test production, stated that he has occasionally noticed less aesthetic choices
of framing in in-house CG productions, which do not accure in live-action productions, where a
trained DOP carefully planned the shot composition.

Another observation made by the author during the test production was that participants with
previous camera experience interacted differently with ALICE RT. For example, they immidiately
started looking for new camera angles by moving around the tracking volume, played around
with depth of field, and experimented with different camera movements. On the other hand, less
experienced participants would use ALICE RT more cautiously, and rather move around the VCS
to look around the virtual world without particularly framing anything specific.

The hardware setup of ALICE RT should be adjusted according to the needs of the intended
production. Both user groups suggested additional areas of use, besides the virtual camera
functionality for full CG films, such as light setup, previs and scouting with a virtual set editing
tool. ALICE RT is still very much a prototype, though, it shows a huge amount of potential,
which is resonated positively by the user group.

59
Figure 5.9: Still frames taken from the test production

60
6 Conclusion

As examined at the beginning of this paper, the development of virtual production was driven
by the growing discontent of a linear production workflow within the VFX Industry. It became
clear that, when working with an increasing amount of CG content, novel solutions have to be
developed, which allow for an interactive, collaborative, and intuitive way of working. Further-
more, the technological achievements in the fields of tracking systems and real-time graphics
accelerate the ongoing progress in virtual production. The fusion of these technologies, which
has only been possible for the last two to three years, enables the realization of projects from low
to high budgets in various areas of use.

One central tool in this environment is a virtual camera system. Various approaches for a virtual
camera were examined, which showed that the usability of such a prototype highly depends on
the application environment. Additionally, the inspected virtual camera systems are generally
developed for the high-end film, gaming, and TV industry. They only support specific software
and are usually not accessible to the public. Thus, a customizable virtual camera prototype
was developed at the post-production and CG content studio INFECTED. The virtual camera
prototype is based on the accurate Lighthouse VR tracking system from Valve, the quickly
evolving game engine Unreal Engine 4 and an iPad with controllers as GUI. The result is a
relatively low-cost model compared to high-end setups depending on expensive motion capture
technology. Epic Games has just released a virtual camera application for Unreal Engine very
similar to the one developed at INFECTED, which confirms the current relevance of virtual
camera systems. The area of use for ALICE RT is exclusively commercial productions, which
has been characterized by time sensitive scheduling, where little to no room is left for creative
experimentation. Thus, the advertising industry is in need of change, which can be provided by
applying the techniques of virtual production.

61
To examine the potential for using a virtual camera system in advertising, as well as verify the
assumed advantages and challenges of virtual production, a test with the prototype ALICE RT was
conducted. Concurrently, the user interface and its features were evaluated as a basis for further
development. The practical trial consisted of three different scenarios to showcase the various
features. The setup was quick and simple compared to elaborate mocap stages. The test and the
affiliated questionnaire helped to understand exisiting problems with the hardware rig, exposed
missing features, confirmed the need of existing ones, and verifies assumed advantges such as
increasing intuitivity, interactivity, and efficiency, less time wasted for unnecessary iterrations and
more time for creative tasks. Notwithstanding the provisional state of ALICE RT, the potential for
a virtual camera rig for commercial productions was recognized by the user group. In particular,
the feedback given by professional DOPs made it clear that virtual production technologies can
enhance their daily work by enabling them to previsualize their shots in detail, e.g testing various
light setups in a virtual world. This allows for more flexibility and freedom to experiment, which
is not possible in a traditional production due to the lack of time to physically try out different
light layouts and camera angles on set.

However, the feedback also showed, that depending on the area of use, ALICE RT needs to be
easily adaptable for the respective project. Besides implementing the planned features and the
functionality of the video transfer of the scene with low latency, further extensions and features
such as a custom shoulder rig or a light asset library need to be integrated. Furthermore, the
virtual camera system needs to be more reliable and less prone to errors than the current prototype,
when used in production. Otherwise, the tool could disrupt the production workflow instead of
improving it. In regard to the entire production pipeline, sharing assets from conventional DCC
tools to game engines and back needs to go as smooth as possible and only requires little effort.
Over the next few months of further development of ALICE RT, more experimental tests need to
be performed to aggregate additional feedback from industry professionals.

As soon as ALICE RT is ready for production, it could potentially benefit the advertising industry
in scouting, previs, and full CG projects. However, one should not disregard the fact, that any
new technology will have to prove its advantages during production to be acknowledged as a
necessary tool. Though, taking a look at projects, which made use of virtual production like
’Zafari’ or ’The Jungle Book’, it is clear that the benefits outweigh the deficits.

62
Especially the enthusiasm expressed by the producers and directors indicates that there will be a
lot more immersive, interactive, and intuitive projects widespread over the media industry since
they are the ones deciding how to execute future projects. Another film highly anticipated by the
virtual production community will be ’Avatar 2’ (2020) by James Cameron, which might once
again revolutionize the VFX industry.

Over the next few years, the use of real-time graphics is almost certain to increase further,
due to the demand for real-time based technology such as VR, AR, games, as well as film,
TV, and advertising. 3D assets created for a franchise will be shared in various media outputs
thanks to shared asset pipelines. Virtual production will be applied to various sectors in the
media industry to ensure collaborative work with detailed visualization and less of a chance
for miscommunication. With acceptance and continuous application of virtual production tools,
the industry will get to a point, where ”virtual will be dropped, and this will just be considered
production.” as stated by David Morin, the Chairman of the Virtual Production Committee
(Giardina 2016).

63
64
A. Glossary
3D ARTIST. Digital artist creating three-dimensional HTC VIVE. Virtual reality headset developed
models, rigs or animations in DCC tools. by HTC and Valve Corporation.

ACCELEROMETER. Device for measuring the LIVE-ACTION. Form of cinematography that uses
acceleration of an object.1 actors and actresses instead of animation or animated
pictures. 1
ANIMATION. Movement created by manipulation of
moving imagery.1 MACHINIMA. Use of real-time computer
graphics engines to create a cinematic production.1
ASSET. Any 3D object created with a DCC tool. MOTION CAPTURE. Process of recording the
movement of objects or people and using that
CINEMATOGRAPHY. Art and science of capturing information to animate digital assets in 2D or 3D
moving images. 2 computer animation.1

COMPOSITING. Combination of elements into an POST-PRODUCTION. Post-production includes all

stages of production occurring after shooting or recording
integrated final shot.
individual program segments.1

CENTRAL PROCESSING UNIT. CPU. Electronic

circuitry within a computer that carries out the instructions POSITIONAL TRACKING. Positional tracking registers
of a computer program by performing the basic arithmetic, the exact position of the HMDs, controllers or other
logical, control and input/output operations. 1 objects, due to recognition of the rotation (pitch, yaw and
roll) and recording of the translational movements.

DIGITAL CONTENT CREATION TOOL. DCC tool.

PREVIS. Visualizing of complex scenes in a movie before
Software used for creation of electronic media. 1
filming.1

DEGREES OF FREEDOM. Number of independent

RAYTRACING. Rendering technique for generating an
motions that are allowed to a body. 1
image by tracing the path of light as pixels in an image
plane and simulating the effects of its encounters with
FRAMING. Presentation of visual elements in an image, virtual objects.1
especially the placement of the subject in relation to other
objects. Framing can make an image more aesthetically
REAL-TIME. A real-time system has been described as
1 one which controls an environment by receiving data,
objects.
processing them, and returning the results sufficiently
quickly to affect the environment at that time. 1
GRAPHICS PROCESSING UNIT. GPU. Specialized
processor for rendering graphics.1
RENDERING. Generating an image from a model by
means of computer programs. 1
GRAPHICAL USER INTERFACE. GUI. Type of
interface that allows users to interact with electronic
devices through graphical icons and visual indicators. 1 SCOUTING. Scouting is the process of searching a
suitable location for a film. Virtual scouting describes the
process of composing and adjusting virtual sets.
GREEN SCREEN. A primary green backdrop that is
placed behind the subject to be photographed so that the
VIRTUAL CAMERA SYSTEM. Hardware setup to
background can be extracted. 2
control a virtual camera.

GYROSCOPE. Device for measuring orientation. 1

VISUAL EFFECTS. VFX. Global term describing effects
that cannot be created with standard filming techniques. 2
HEAD-MOUNTED DISPLAY. HMD. display device,
worn on the head. Has a small display optic in front of one
(monocular HMD) or each eye (binocular HMD). 1

1
Wikipedia (2018). www.wikipedia.org
2
Goulekas (2001). Visual Effects in a Digital World.

65
B. Bibliography
AMC (2018). Visual and Special Effects Film Milestones. Retrieved from: http://www.filmsite.
org/visualeffects9.html (visited on 09/02/2018).
Anderton, E. (2017). .
Retrieved from: https://www.slashfilm.com/rogue-one- virtual-camera-system/ (visited on 10/04/2018).
Ansel, A. (1980). The Camera.
Autodesk (2018). FBX: Adaptable File Formats for 3D Animation Software. Retrieved from:
https://www.autodesk.com/products/fbx/overview (visited on 09/17/2018).
Badgerow, D. (2012). A Stack Of Drawings: What is Layout, anyway? Retrieved from:
http://badgerart.blogspot.com/2012/09/what-is-layout-anyway.html (visited on 09/10/2018).
Balakrishnan, G. (2016). Unite 2016 - Unity for Films [video file]. Retrieved from:
https://www.youtube.com/watch?v=tvcu4oIcFuA (visited on 08/30/2018).
Baun, C. (2018). Computernetze kompakt. eBook Springer.
Beck, M. (2015). The VES handbook of visual effects. Industry Standard VFX Practices and Procedures. Focal
Press.
Box Office Mojo (2018). All Time Worldwide Box Office Grosses. Retrieved from:
https://www.boxofficemojo.com/alltime/world/ (visited on 09/01/2018).
Cofer, Grady (2018). HTC VIVE x READY PLAYER ONE - Utilizing VIVE In The Filmmaking Process
[video file]. Retrieved from: https://www.youtube.com/watch?v=W_6vTqIyPmM (visited on 08/12/2018).
Compute Scotland (2010). Avatar-style technique come to PC with motion. Retrieved from:
https://www.computescotland.com/avatar-style-technique-come-to-pc-with-motion-controller-3799.php (visited
on 09/06/2018).
Creative Skillset (2018). Layout Artist (3D computer animation). Retrieved from: http://creativeskillset.
org/job_roles/365_layout_artist_3d_computer_animation (visited on 09/10/2018).
Dartfish (2013). Dartfish User Guide. Retrieved from: http://ronngardsgolf.se/onewebmedia/UserGuide_7.0.pdf
(visited on 06/09/2018)
Digital Monarch Media (2018). Digital Monarch Media Info. Retrieved from: https://digitalmonarch.media/
(visited on 10/04/2018).
Dunlop, R. (2014). Production Pipeline Fundamentals for Film and Games. Sek 5. Focal Press.
Ealovega, A. (2014). Dreamspace: Virtual Production Methods, Guidelines and Scenarios. Retrieved from:
https://www.dreamspaceproject.eu/dyn/1429609964686/Dreamspace_D2.1.1_v04_030414.pdf (visited on
09/25/2018).
Elson, D., Riedl, M. (2007). A Lightweight Intelligent Virtual Cinematography System for Machinima
Production. Retrieved from: http://www.dtic.mil/ docs/citations/ADA464770 (visited on 09/05/2018).
Faisstnauer, C. (1997). Navigation and Interaction in Virtual Environments.
Favreau, J. (2016). American Cinematographer The Jungle Book. Retrieved from:
http://girishbalakrishnan.com/redesign/wp-content/uploads/2018/04/american-cinematographer-may-2016-the-
jungle-book.pdf (visited on 08/12/2018).
Ferster, B (1998). Previsualization Article. Retrieved from: http://www.stagetools.com/previs.htm (visited on
09/06/2018).
Fink, M.(2015). The VES handbook of visual effects. Industry Standard VFX Practices and Procedures. Focal P.

66
Gardner, B. (2011). Responsive Web Design: Enriching the User Experience. Retrieved from:
http://www.webdesignblog.gr/wp-content/uploads/2012/03/5.pdf (visited on 09/10/2018).
Gartner, K. (2018). Virtual Cinematography for VR Trailers. Retrieved from:
http://www.kertgartner.com/virtual-cinematography-for-vr-trailers/ (visited on 10/05/2018).
Giardina, C. (2016). Virtual Production for 'Jungle Book' Detailed During HPA Tech Retreat. Retrieved from:
https://www.hollywoodreporter.com/behind-screen/virtual-production-jungle-book-detailed-866395 (visited on
10/04/2018)
Gleicher, M. (1999). Animation from observation: Motion capture and motion editing. Retrieved from:
http://portal.acm.org/citation.cfm?doid=345370.345409 (visited on 09/04/2018).
Goldman, M. (May 2016). American Cinematographer The Jungle Book. Retrieved from: https://goo.gl/qg7E2a
(visited on 08/12/2018).
Google (2018 a). ARCore Overview. Retrieved from: https://developers.google.com/ar/discover/ (visited on
10/05/2018).
Google (2018 b). Material Design Principles. Retrieved from: https://material.io/design/introduction/#principles
(visited on 09/14/2018).
Helzle, V., Spielmann, S.(2015). Draft Specification and Interface description of Virtual Production Editing
Tools. Retrieved from: https://www.dreamspaceproject.eu/dyn/1429609964635/DREAMSPACE_D5.1.1_v2.1_
060215.pdf (visited on 09/13/2018).
Hinckley, K. et al. (1994). A survey of design issues in spatial input. Retrieved from:
http://portal.acm.org/citation.cfm?doid=192426.192501 (visited on 09/12/2018).
Hothan, F. et al. (2018). ALICE RT Feedback by DOPs.
ILM (1999). The Mummy. Retrieved from: https://www.ilm.com/vfx/the-mummy/ (visited on 09/01/2018).
ILM (2018). ILMxLAB Immersive Entertainment. Retrieved from: https://www.ilmxlab.com/ (visited on
10/04/2018).
Ineson, B. (2016). -Vis. Retrieved
from: https://www.awn.com/news/animatrik-provides-performance-capture-tech-suicide-squad-stunt-vis (visited
on 09/02/2018).
Ivey, C. (2015). Showdown Cinematic VR Experience Released for Free! Retrieved from:
https://www.unrealengine.com/en-US/blog/showdown-cinematic-vr-experience-released-for-free (visited on
10/02/2018).
Knopp, T. (2014). Virtual Production Methods, Guidelines and Scenarios. Retrieved from:
https://www.dreamspaceproject.eu/dyn/1429609964686/Dreamspace_D2.1.1_v04_030414.pdf (visited on
09/04/2018).
Lauterbach, T. (2009). Retrieved from:
http://www.vizrt.com/casestudies/36757/Behind_the_scenes_at_ZDF_Europes_biggest_virtual_studio (visited
on 09/06/2018).
Libreri, K. (2018). State of Unreal GDC 2018 [video file]. Retrieved from:
https://www.youtube.com/watch?v=jwKeLsTG12A (visited on 10/06/2018).
Lin P. (2016 a). Patrick Lin - Director of Photography FMX 2016 Part 3. Retrieved from:
https://www.youtube.com/watch?v=9xXKiizQxn4&t=2s (visited on 09/06/2018).
Lin, P. (2016 b). AWN - FMX 2016 Professional Spotlight: Patrick Lin [video file]. Retrieved from:
https://www.youtube.com/watch?v=3cWbGbulByg (visited on 09/10/2018).
Le, J. (2018). . Retrieved from:
https://goo.gl/7u7euN (visited on 09/26/2018).

67
Leberecht, S. (2014). Life after [video file] Retrieved from: https://www.youtube.com/watch?v=9lcB9u-9mVE
(visited on 10/07/2018).
LiDAR (2018). How does LiDAR work. Retrieved from: http://www.lidar-uk.com/how-lidar- works/ (visited on
09/27/2018).
Lussier, G. (2016). One Animal in Zootopia Has More Individual Hairs Than Every Character in Frozen
Combined. Retrieved from: https://io9.gizmodo.com/one-animal-in-zootopia-has-more-individual-hairs-than-e-
1761542252 (visited on 09/04/2018).
Marino, P. (2004). The art of machinima. Scottsdale: Paraglyph Press.
Marui (2017). VCAT MARUI PlugIn. Retrieved from: https://www.marui-plugin.com/vcat/ (visited on
10/05/2018).
Mayeda, R. (2018). Virtual Production in Unreal Engine 4.20. Retrieved from:
https://www.youtube.com/watch?v=yTYI_8a7kPM (visited on 10/06/2018).
Microsoft (2017). XAML. Retrieved from: https://docs.microsoft.com/de-de/dotnet/framework/wpf/advanced/
xaml-overview-wpf
Morin, D. (2018). The state of Virtual Production in 2018 at FMX 2018. Retrieved from:
https://fmx.de/program2018/event/13196.
Niehorster, D., Li, L., Lappe M. (2017). The Accuracy and Precision of Position and Orientation Tracking in the
HTC Vive Virtual Reality System for Scientific Research. Retrieved from:
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5439658/ (visited on 09/27/2018).
Okun, J.(2015). The VES handbook of visual effects. Industry Standard VFX Practices and Procedures. Focal P.
OptiTrack (2018). Insight VCS - A professional virtual camera system. Retrieved from:
http://images.autodesk.com/apac_grtrchina_main/files/the_new_art_of_virtual_ moviemaking_-
_autodesk_whitepaper1.pdf (visited on 09/01/2018).
Owlchemy Lab (2018). Mobile Spectator AR Spectator Camera Experiment. Retrieved from:
https://owlchemylabs.com/owlchemy-mobile-spectator-ar-spectator-camera/ (visited on 10/05/2018).
Pixar (2017). Universal Scene Description. Retrieved from: https://graphics.pixar.com/usd/docs/index.html
(visited on 09/15/2018).
Potter, W. (2018). Story Tellers of Tomorrow. Retrieved from: https://digitalmonarch.media/2018/02/02/story-
tellers-of-tomorrow/ (visited on 10/04/2018).
Reagan, C. (2017 a). AWN - FMX 2017 Professional Spotlight: Clint Reagan - Part 1. FMX 2017. Retrieved
from: https://www.youtube.com/watch?v=A4QkWd9oy4A (visited on 09/02/2018).
Reagan, C. (2017 b). AWN - FMX 2017 Professional Spotlight: Clint Reagan - Part 2. FMX 2017. Retrieved
from: https://www.youtube.com/watch?v=Q-Mmmb1ROxE (visited on 10/08/2018).
Road to VR (June 2014). Overview of Positional Tracking Technologies for Virtual Reality. Retrieved from:
https://www.roadtovr.com/overview-of-positional-tracking-technologies-virtual-reality/ (visited on 09/25/2018).
Root, J. (2015). The VES handbook of visual effects. Industry Standard VFX Practices and Procedures. Focal P.
Sargeant, B., Morin, D., Scheele, J. (2015). The VES handbook of visual effects. Industry Standard VFX
Practices and Procedures. Focal Press.
Serkis, A. (2011). The Guardian: W go ape over motion-capture acting. Retrieved from:
https://www.theguardian.com/film/2011/aug/12/andy-serkis-motion-capture-acting (visited on 09/25/2018).
Serrabassa, R. (2014). VFX Shots Race. Retrieved from: http://www.upcomingvfxmovies.com/svfx-shots-race/
(visited on 09/01/2018).
Snyder, C. (2016). Intro to VR: Degrees of Freedom. Retrieved from:
http://www.leadingones.com/articles/intro-to-vr-4.html (visited on 09/13/2018).

68
Sodiya, S. (2014). User Interface Design and Ergonomics. Retrieved from: https://goo.gl/cFsXGe (visited on
09/10/2018).
Spielberg, S. (2018). HTC VIVE x READY PLAYER ONE - Utilizing VIVE In The Filmmaking Process
[video file]. Retrieved from: https://www.youtube.com/watch?v=W_6vTqIyPmM (visited on 08/12/2018).
Taylor, R. (2015). The VES handbook of visual effects. Industry Standard VFX Practices and Procedures. Focal
Press.
The Mill (2017). The Mill: The Human Race - Behind The Scenes. Retrieved from:
http://www.themill.com/portfolio/3517/the-human-race---behind-the-scenes- (visited on 09/27/2018).
Unity (2018 a). Baymax Dreams. Retrieved from: https://unity.com/de/node/375 (visited on 10/06/2018).
Unity (2018 b). Book of the Dead. Retrieved from: https://unity3d.com/book-of-the-dead (visited on
10/06/2018).
Unreal Engine (2018 a). Why Real-Time Technology is the Future of Film and Television Production. Retrieved
from: https://cdn2.unrealengine.com/Unreal+Engine%2FWhy-Real-Time-Technology-is-the-Future-of-Film-
and-Television-Production-ecc14547ab8a340d4 c00e2f931adfcf3a201478d.pdf (visited on 09/20/2018).
Unreal Engine (2018 b). Porsc
Speedster Concept. Retrieved from: https://goo.gl/TMrgyy (visited on 10/06/2018).
Unreal Engine (2018 c). Rendering Overview. Retrieved from: https://docs.unrealengine. com/en-
us/Engine/Rendering/Overview (visited on 09/18/2018).
Unreal Engine (2018 d). Pawn. Retrieved from: https://docs.unrealengine.com/en-us/Gameplay/
Framework/Pawn (visited on 09/13/2018).
Unreal Engine (2018 e). Matinee. Retrieved from: https://docs.unrealengine.com/en-
us/Resources/Showcases/MatineeFightScene (visited on 10/02/2018).
Vicon (2018). Virtual Camera. Retrieved from: https://goo.gl/Deo5ab (visited on 10/04/2018).
VIVE (2018). VIVE Enterprise Professional Grade VR. Retrieved from: https://enterprise.vive.com/eu/ (visited
on 09/13/2018).
Ward, J. (2008). What is a Game Engine? Retrieved from:
http://www.gamecareerguide.com/features/529/what_is_a_game_.php (visited on 09/05/2018).
Wolfe, J. (2012). A New Look at Virtual Production. Retrieved from: https://www.awn.com/vfxworld/new-
look-virtual-production (visited on 09/01/2018).
Xinreality (2018). Inside-out tracking - Virtual Reality and Augmented Reality. Retrieved from:
https://xinreality.com/wiki/Inside-out_tracking (visited on 09/26/2018).
Zargarpour, H. (2018). Siggraph 2018 - Using a Real-Time Engine in Movie Production [video file]. Retrieved
from: https://www.youtube.com/watch?v=U_NG7WfoI7s (visited on 10/04/2018).

69
C. List of Figures
Fig. 2.1: On set of Doctor Strange. Retrieved from: http://www.weiquvr.com/m/view.php?aid=99

Fig. 2.2: The process of previsualization throughout production. Eschenbacher, A. (2018)

Fig. 2.3: Pitchvis. Retrieved from: https://vimeo.com/225877552

Fig. 2.4: . Retrieved from:

http://girishbalakrishnan.com/redesign/portfolio-posts/the-jungle-book/

Fig. 2.5: Virtual Production set of The Jungle Book. Retrieved from: https://www.awn.com/vfxworld/adam-
valdez-talks-jungle-book

Fig. 3.1: Mocap in War for the Planet of the Apes. Eschenbacher, A. (2018), based on:
https://www.cartoonbrew.com/vfx/performance-capture-creature-apes-created-war-planet-apes-152357.html

Fig. 3.2: Outside-in tracking. Retrieved from: https://www.optitrack.com/

://vimeo.com/206263410

Fig. 3.4: Six degrees of freedom. Retrieved from: http://kilograph.com/virtual-reality-6dof/

Fig. 3.5: ILMs virtual camera system for Rogue On (left), Spielberg VR scouting for Ready Player One (right).
Retrieved from: Left: https://vrscout.com/news/rogue-one-director-vr-plan-films-digital-scenes/
Right: https://www.youtube.com/watch?v=W_6vTqIyPmM

Fig. 3.6: Zootopia before and after rendering. Retrieved from: https://io9.gizmodo.com/one-animal-in-zootopia-
has-more-individual-hairs-than-e-1761542252

Fig. 3.7: real-time workflow for animation. Retrieved from: https://unity.com/de/node/375

https://goo.gl/TMrgyy

Fig. 3.9: . Retrieved from:

https://www.youtube.com/watch?v=zTebgHNSe_4

Fig. 3.10: Virtual studio for ZDF (left), The Weather Channel Studio (right). Retrieved from:
Left: http://hosoyaschaefer.com/projects/zdf-news-studio/
Right: https://www.newscaststudio.com/2018/04/05/weather-channel-augmented-reality-future-group/

Fig. 3.11: Comparison of the final shot and the live action shoot. Retrieved from:
https://www.awn.com/vfxworld/adam-valdez-talks-jungle-book

Fig. 3.12: Patrick Lin working with a virtual camera system at Pixar. Retrieved from:
https://camerimage.pl/en/pixar-na-camerimage-2015/

Fig. 3.13: Virtual camera by Vicon (left), Insight VCS by OptiTrack (right). Retrieved from:
Left: https://www.youtube.com/watch?v=NW6mYurjYZ0
Right: https://optitrack.com/about/press/20161115.html

70
Fig. 3.14: VCam development of Digital Monarch Media. Retrieved from:
https://www.youtube.com/watch?v=U_NG7WfoI7s

https://www.independent.co.uk/arts-entertainment/films/news/ready-player-one-cgi-vfx-motion-capture-behind-
the-scenes-steven-spielberg-a8434426.html

Fig. 3.16: Virtual Camera System used for Rogue One. Retrieved from: https://vrscout.com/news/rogue-one-
director-vr-plan-films-digital-scenes/

Fig. 3.17: Mobile Spectator by Owlchemy Lab. Retrieved from: https://owlchemylabs.com/owlchemy-mobile-

spectator-ar-spectator-camera/

Fig. 3.18: Steven Spielberg layouting shots in VR for Ready Player One. Retrieved from:
https://www.youtube.com/watch?v=W_6vTqIyPmM

Fig. 4.2: Hardware setup for Alice RT. Eschenbacher, A. (2018)

Fig. 4.3: Data transfer between Alice RT plugin and App. Eschenbacher, A. (2018)

Fig. 4.4: Vive Tracker. Retrieved from: https://www.cgtrader.com/3d-models/electronics/video/htc-vive-tracker

Fig. 4.5: UI design of ALICE RT. Eschenbacher, A. (2018)

Fig. 4.6: ALICE RT app login screen. Eschenbacher, A. (2018)

Fig. 4.7: Toolbar with camera settings. Eschenbacher, A. (2018)

Fig. 4.8: Key layout game controller. Eschenbacher, A. (2018)

Fig. 4.9: Main UI of ALICE RT. Eschenbacher, A. (2018)

Fig. 4.10: Operating ALICE RT. Eschenbacher, A. (2018)

Fig. 5.1: Test Scenario 1 with VW T-Roc. INFECTED (2018)

Fig. 5.2: Test Scenario 2 (left) and Test Scenario 3 (right). Retrieved from:
Left: https://docs.unrealengine.com/en-us/Resources/Showcases/MatineeFightScene
Right: https://www.unrealengine.com/en-US/blog/showdown-cinematic-vr-experience-released-for-free

Fig. 5.3: Snapshots of the ALICE RT Test. Eschenbacher, A. (2018)

Fig. 5.4: Level of experience, fields of experience. Retrieved from: https://www.umfrageonline.com.

Fig. 5.5: Need for optimization in a traditional VFX. Retrieved from: https://www.umfrageonline.com.

Fig. 5.6: Assumed advantages of virtual production. Retrieved from: https://www.umfrageonline.com.

Fig. 5.7: Results for the user interface of ALICE RT. Retrieved from: https://www.umfrageonline.com.

Fig. 5.8: VW T-Roc inside Unreal Engine. Eschenbacher, A. (2018)

Fig. 5.9: Still Frames. Eschenbacher, A. (2018)

71