Keywords

1 Introduction

United Nations Educational, Scientific and Cultural Organization (UNESCO) have officially announced the folk dance of the Mazu Festival Celebration as “Intangible Cultural Heritage of Humanity” in August 2009 [1]. Among the Mazu Festival Celebration image performance, the woodcut prints by Lin [2] are the most representative images. Our research team already completed the transition from woodcut prints into animation arts, the results have accepted by the Digital Humanities organization in 2014 [3], SIGGRAPH 2015 conference [4] and the HCI International 2017 [5]. The international experts and scholars all agree with the integrated studies of humanities, technology and art, especially the White Crane Dance is about to disappear. In recent years, HMD devices have become more and more compact in appearance and provide an immersive space environment for the viewer. The HMD becomes the ideal development model for this research and provides users with natural interactive experience. In order to develop the VR of folk dance and print art, our team have two major purposes to accomplish in this research. There are:

White Crane Dance in VR

White Crane Dance is a typical double play performance that consists of two characters, Crane and Fairchild. For viewing White Crane Dance in VR, we have to convert the completed White Crane Dance animation from video to VR format, this process required to overcome the data error when the animation file convert to different platform, we used Unity as VR platform and the finished 3D Max animation was input to unity. Users wearing VR glasses can experience the White Crane Dance, and are free to walk around to watch different perspectives, as the user next to the Crane and Fairchild dance scene. As mentioned in the previous study, the movement of Crane and Fairchild was captured by the folk dances. The character modeling and material are referenced to Chih-Shin Lin’s print style. The combination of folk dance and print art makes this VR experience both in folk dance culture and art style.

Interactive Experience of VR

With the interactive and display technology innovation, and gradually be used in various forms of cultural relics collection and education. This research aims to promote the interaction between user and virtual character. With the theme of “White Crane Dance”, wet developed the 3D avatar in a VR environment. This research first adopted RGB-D image capture technology, which features easy to carry, without adding any tracker on the body, the new device such as Kinect 2.0 has been able to automatically calculate the shadow interference phenomenon, capture the accuracy and quality significantly improved. To achieve this goal, we required to integrate the motion capture system with the HMD module, detect the user’s action and drive the corresponding 3D avatar image content, presented in the user’s environment. Through the HMD, the 3D avatar calculated by the foregoing 3D environment is projected onto the user’s eyes so that the user can watch both the demo and the played roles. The role of the 3D avatar is the state of operation, allowing users to experience White Crane Dance effect.

2 Literature Research

2.1 Mazu Faith and White Crane Dance

Mazu is a sea god faith centered on the southeastern coast of China. With its worship and praise of Mazu’s ethics, good deeds and love as the core. Mazu belief is Taiwan’s first world-class heritage of faith, has a history of thousands years. At present, Mazu culture spread to more than 20 countries and regions in the world. Mazu temple is the main place for cultural activities and the customs. Typical folk dance groups are the forms of temple fairs. There are more than 5,000 Mazu temples and more than 200 million believers in the world. Among the folk dance groups, White Crane Dance is an original display in Tainan, south of Taiwan. According to the elder’s descriptions, the written materials and the relics, Bao-An Temple in Tainan wanted to escort Kong-Fu God in 1928. The people then got the god’s message to protect him by displaying White Crane Master (Fig. 1(a)) as the battle front [6]. White Crane Dance became a significant temple activity of Bao-An Temple (Fig. 1(b)). The story describes a man wearing a mask of fairy, holding the red cloth fairy fan, make-up for Fairchild. Another man is put on the head of the crane disguised as a crane, left and right hands tied wings, you can open and close the wings to swing. Fairchild tease play crane, crane dances with the Fairchild’s fan ups and downs, moving parts with both harmonious funny and dynamic rhythm.

Fig. 1.
figure 1

(a) Statute of White Crane Master, (b) Dancers in front of Bao-An Temple

Full size image

2.2 Artist Chih-Hsin Lin

Among the Mazu Festival Celebration images performance, the woodcut prints by Chih-Hsin Lin are the most representative images. Lin completed the print of Mazu Festival Celebration in 20 years, has a total length of 408 ft (124 m) (Fig. 2(a)). It is the longest print in the world by a solo artist. Mazu Festival Celebration has been exhibited at home and abroad for its attention by international museums and many academic and research institutions. Lin’s print expanded the elements of space and time in the picture by using Chinese scroll style expressions. Through elaborate arrangement of 70 scenes which records the process of the Taiwanese people’s most important faith, including the rise of Mazu, the road sign, the lamp, the post, the Mazu head flag, the God of Wealth, White Crane Dance (Fig. 2(b)), rocking boat (peach blossom transition) and the clan array, spring Bulls, Sangu six Po, twelve flower gods, eight will, seven master bye, … and other dance groups.

Fig. 2.
figure 2

(a) Lin with his woodcut print (b) White Crane Dance print by Chih-Hsin Lin

Full size image

2.3 Applications of Virtual Reality (VR)

The evolution of HMD technology has led to the creation and commercialization of devices at different levels of consumption, allowing for an immersive experience at a reasonable cost and the availability of such products to grow in the future. For example, Microsoft’s Oculus Rift, Samsung’s Gear VR emphasize the combination with handsets, HTC’s Vive emphasizes high definition, and Google Cardboard already exists for a solution that is very cheap for converting handsets into the HMD. The application of VR in the field of art and cultural heritage has different purposes. For examples, Gaitatzes et al. [7] put forward the advantages and disadvantages of immersive interactive VR through the application of museums (Fig. 3(a)), Kennedy et al. [8] rebuilt St. Andrews Cathedral (Fig. 3(b)). In the Styliani et al. [9] survey, virtual museums and exhibitions empowered by ARs or authors that mapped different implementations and techniques for such applications investigated VRs. They defined the Virtual Learning Museum as a specific type of museum that provides background, and interest-related content can inspire genuine visits and promote curiosity about the content to better serve the interests of the user. In addition, the RiftArt system developed by Casu et al. [10] improved the learning of art textbooks in a virtual reality manner. As shown in Fig. 3(c), Woodard et al. [11] developed a viable combination of Oculus and Kinect interactive virtual buildings, as shown in Fig. 3(d). In order to overcome the problem of wireless bandwidth transmission, Hao et al. [12] proposed a wireless interactive VR architecture by means of server-side and client-side divisions, as shown in Fig. 3(e).

Fig. 3.
figure 3

Applications of VR: (a) interactive VR through the application of museums [7], (b) Rebuilt St. Andrews Cathedral [8], (c) Rift Art art textbooks in a VR [10], (d) Interactive virtual buildings [11], (e) wireless interactive VR architecture [12].

Full size image

3 Methods

This research is to construct a VR system of White Crane Dance, which is intended to build a prototype of real-time interactive platform after data collection and system integration and development. This system used Unity as integrated platform to import 3D data and connect Kinect motion and output to HTC Vive HMD. The research system architecture is as show in Fig. 4.

Fig. 4.
figure 4

System architecture of White Crane Dance VR

Full size image

1. Integrated original 3D animation

The 3D animation with the texture map imported into Unity, such as the characters, background, skeleton and animation, and Unity was used as the instant interactive platform. The unity animation engine converts the original animation data into real-time interactive data. The original animation made by 3D Max, exported all the data to Unity as FBX file format. In the process we found a loss of the map, so after rebuilt the Material in Unity and re-affixed the model’s texture, then used the Bumped Diffuse shader to match the print-style visual effects. Since 3ds Max and Unity had a slight error in the definition of character skeletons, they needed to go into the skeleton of the model and calibrate to fit the skeleton that Unity can use (Fig. 5).

Fig. 5.
figure 5

White Crane Dance in Unity

Full size image

2. Role Play action design

With Crane Dance is a typical double play, composed of Crane and Fairchild. To achieve the purpose of playing avatar, we designed a role-play action system on the choice of Fairchild or Crane. User can choose to play Crane or Fairchild avatar to imitate the role of practice or interactive with the other.

When choose Crane practice mode, in the HMD would appear two identical virtual Cranes, one is playing Crane animation, and the other Crane as avatar, through the synchronous operation to achieve the purpose of practice, and choose Fairchild practice mode in the same way. On the interactive mode, when choose Crane interactive mode will operate the Crane avatar with the Fairchild animation, and choose Fairchild interactive mode will operate the Fairchild avatar with the Crane animation (Fig. 6).

Fig. 6.
figure 6

White Crane Dance is a typical double play performance

Full size image

3. Play Menu

In order to achieve the above optional play experience, we created a simple menu that allows users to choose from the beginning to play Crane or Fairchild, to practice or interactive play (Fig. 7). Based on the choice of the following needs to establish the following menu: 1. Crane practice mode, 2. Crane interactive mode, 3. Fairchild practice mode, 4. Fairchild interactive mode. The user used HTC Vive Handle controller to select a character and at the same time sees that the 3D animation played by the character appears in the virtual environment. The user can watch the animation action and imitate learning, select the character and operate the avatar. Due to technical bandwidth constraints, this study is one-person experience and has not been designed to provide both experience and co-performance. The process of creating a menu in Unity was made by script. Create an empty object named Select Avatar, and then create a Button object, and then create a Script Prefab can be generated into the virtual scene. After setting up, click Button to set the click event in On Click (), when finished, click the menu button and generate the user’s selected character (Table 1).

Fig. 7.
figure 7

Interface of avatar

Full size image
Table 1. Avatar choice
Full size table

4. Action capture and playback

The research previously used the Kinect module to capture the performance of the folk dancers and animates the motion to the 3D animation characters. Now the Kinect was used to capture user actions. Through the Unity integration platform, the 3D animation characters played and the 3D avatar to be instantly driven were presented to user in the environment, as shown in Fig. 8.

Fig. 8.
figure 8

System framework of Kinect and HTC with Unity

Full size image

Action Capture

  1. 1.

    Positioning correction with the avatar of user in a virtual space: The Kinect must be placed in front of the user in real space, using one of the HTC Vive handle controllers as a positioning reference point, in front of the Kinect, The handle controller must be within the virtual space set by the HTC Vive optical sensor, otherwise the handle controller can not be sensed and the positioning of virtual space and real space is lost.

  2. 2.

    Import the motion of user (the skeleton has been calibrated) into the virtual scene: When the controller of the handlebar is ready, Kinect can accurately capture the user’s motion and then insert the model into the virtual scene. When using a HMD, do some slight adjustment according to visual result.

Playback

  1. 1.

    Animation setup: First create an Animator controller (animation editor) for the White Cane model, and then imported animation data to the model and confirm the numbers of animation is correct or not. Add this Animator controller to the Animator property in Inspector for the Crane Model in the scene.

  2. 2.

    The model in the virtual scene position correction: In order to make the model in the virtual scene to play the animation in the ideal position, in addition to the virtual scene model to do a little coordinate value adjustment, we also set the virtual scene Vive Camera coordinate adjustment, so that the user in the virtual space can feel and sense of space, to match with the sense of reality in the space.

5. Wearing a display technology approach

In this research, the 3D character and avatar calculated by the unity 3D environment is projected onto the eyes of the user through the HMD device so that the user can watch the contents of the space environment at the same time. Experiencing the operated status of the character in 3D of VR. The HMD we used for this project was HTC Vive (Fig. 9), so we had to download a resource bundle named Steam VR from Unity’s official resource store, expand it into the project, and then drag an object called CameraRig into the resource bundle Scene, you can temporarily turn off the preset scene camera, and the scene captured by this object (CameraRig) will be the user to watch the picture in the HMD, if you want another point of view to watch an animation within a virtual scene, then adjust the Rotation or Position of the object.

Fig. 9.
figure 9

White Crane Dance VR system

Full size image

From Table 2 shows the four avatar mode without background which can reduce the interference of background, the user can easy find the avatar and the other character at the beginning, however, there are still some motion error when do the dance action because the limitation of Kinect sensor. Once user knows how to control the avatar then we can add background back to the environment as show in Fig. 10.

Table 2. Avatar mode without background
Full size table
Fig. 10.
figure 10

White Crane Dance in VR

Full size image

4 Conclusion

Our research delivers a balance of art and technology in culture creativity. VR and motion capture have become accessible and affordable which makes new media art a new vision. The valued culture materials like print art may preserve in traditional way and folk dance may varnish day by day. In this research, one is a still art showing in museums and the other is a dynamic performance showing on street of temple affairs, both have weak connection in present time. By integrated folk dance and print art with technology become a digital way to represent culture and art. Transform two art styles into a new style and provide an experience of VR is the mission of this research.

We developed a prototype of Mazu faith with Virtual Reality, in order to extend the digital cultural content under the international attention. The present form of White Crane Dance targeted at interactive avatar experiences the understanding of culture and the original data compiled into a virtual reality playing system to preserve folk art with technology. Taiwan’s creative and industry oriented integration of technology and the arts are still immature. This research’s achievements and experiences will provide and relocate the domestic “Future Museum” builders and hopefully stimulate Taiwan’s digital creative transformation. We will stand on the cutting edge of the digital era and demonstrate the possibility of unlimited creativity and new forms of creative value-added, so as to achieve the cross cutting integration of culture, technology and art that are currently under active development in all fields.