Keywords

1 Introduction

HMI development requires user centered testing and hence the involvement of potential users with HMI prototypes in a realistic context. Typical problems are the acquisition of a representative user group and the setup of a realistic context, e.g. driving scenarios that degrade the HMI function to a secondary task (Alfes et al. 2014).

In our case, it is difficult to get a truck driver to a certain residential test laboratory for testing an HMI. Those who are willing to come are a highly selective sub-group of truck drivers, causing a clear bias to the study. In order to overcome these problems, we adapted the Guerilla Interview technique (Jacobsen and Meyer 2018) and took our HMI prototype to the users.

The original Guerilla technique was proposed for website testing by Martin Belam (2010). The characteristics of the Guerilla approach are to find the users where they are and to get a few minutes in between their regular daily routine in order to test the HMI prototype rather quick and dirty. In order to meet our users, we selected truck stops along German highways and approached truck drivers in their normal work and break routine.

On site Guerilla testing requires portable HMI prototypes that can be used in the given environment where the users are found. In the past we used a portable HMI mockup, consisting of a truck steering wheel and a tablet behind the wheel representing the instrument cluster content. This solution allowed for HMI function tests, but did not provide any context or scenario.

To overcome this limitation VR with head mounted displays (HMD) have been used recently in different studies to assess vehicle HMI (Krupenia et al. 2017; Lottermoser et al. 2018; van der Veer et al. 2019; Bopp-Bertenbreiter et al., in press). In our approach we combined the VR simulation with the Guerilla method, hence a portable VR simulator was set up in the back of a Mercedes-Benz V-Class (Fig. 1). This allowed us to simulate automated driving and integrate the HMI prototype in a mixed-mock-up environment of a full truck cabin (see also Figs. 3 and 4).

Fig. 1.
figure 1

Mercedes-Benz V-Class with integrated VR simulator in the rear seat compartment (Daimler AG 2018)

Full size image

2 Method

The Guerilla method allows to find a representative user group in the field and in their regular work environment. It requires a mobile HMI prototype of which we compared two different versions, a truck steering wheel mockup and a VR simulator.

2.1 Guerilla Interviews

In user centred HMI development a large variety of methods to involve potential users and to gain their feedback on early prototypes have been reported (e.g. Jacobsen und Meyer 2018; Pollmann 2018). Apart from standardized usability tests, these methods also include the remote or Guerrilla testing. According to Lämmler (2017), the advantages of this method are to obtain data fast and efficiently without having to invest lots of time or money (Lämmler 2017).

Especially in the case of truck drivers, the Guerilla interview allows to approach truck drivers efficiently in their break time on truck stops. Driving time regulations mean drivers have 45 min break time a day and hence the total length of the interview in relation to the remaining break time determines whether drivers take part.

By this approach, also truck drivers are included in the test sample who would not come to a test laboratory. The time requirements for the test participants are minimized with this approach and hence it can increase the efficiency of the participant acquisition and the positive response rate when asking for participation.

In addition, Guerilla interviews approach end-users to test the product directly in its usual environment and context and hence the feedback quality provides higher external validity.

For an objective, reliable and valid data analysis back in the office, a structured interview guide helps to retrieve all relevant information (Mayring 2010; Niehaus 2014; Lämmler 2017). The qualitative results are analyzed according to the content analysis process introduced by Mayring and Frenzl (2014). The content analysis is the standard method for analyzing large quantities of qualitative data (Mayring 2010). It assures the classic test quality criteria: objectivity, reliability and validity (Mayring and Brunner 2009).

2.2 Mobile Truck Steering Wheel Mockup and Click Dummy

The first developed UX simulator is a dummy steering wheel with functional keypads from the simulated vehicle (Fig. 2). The rapid-prototyping casing has the geometry of a generic truck steering wheel with integrated series production steering wheel keypads. The HMI prototype is implemented on a Tablet PC positioned behind the steering wheel mockup equivalent to the Instrument Cluster (IC) position in the real truck.

Fig. 2.
figure 2

Click dummy buttons in the steering wheel controls (Daimler AG 2018)

Full size image

The simulator with the setup allows test participants to carry out interactive activities in the represented context. The test participants can look at individual screens as part of the evaluation and interact with the HMI via the steering wheel buttons. Like in the real vehicle, the left keypad is used to control the IC. As the function of the IC is limited, only a few of the interface elements actually create events in the software prototype. To navigate through the individual menus and screens, a user can press the home/back button and the multifunctional touch control button. The home/back button enables the user to navigate in the menu structure and by pressing the touch control button an action is selected. The touch control button also works like a touchpad. It can be used to toggle between elements on the display. Unlike the left keypad, the right keypad does not have any saved functions. The driver uses this keypad in the vehicle to operate the centre console display (IVI), which was not installed in this setup.

2.3 Mobile VR Truck Cockpit Simulator and Click Dummy

The second UX simulator uses VR technology. Apart from the head mounted display (HMD), the simulator requires an avatar tracking device, a steering wheel with optical tracker as well as functional keypads, comparable to the steering wheel mockup described above. A processing unit with the simulation and the SW prototype allow the VR representation of the complete truck cockpit.

During a test run with the simulator, the test participant must put on the HMD. When looking through the lenses of the HMD, the test participant perceives the image shown in Fig. 3. The driver sits in a virtual truck cab and recognizes the changing surroundings. The avatar tracking device tracks the test participant’s hands and provides a model for the simulation to create virtual hands in the relevant position in relation to reality. The visualization of the whole cab and the exterieur, as well as the sounds perceived through the speakers in the HMDs integrated headphones, provide a rich contextual environment for the simulation. The driver has the sensation of actually sitting and driving in the cab.

Fig. 3.
figure 3

View through the HMD - detail from the VR simulation (Daimler AG 2018)

Full size image

The simulator also includes a series production steering wheel as a hardware component and at the same time as a digital representation in the VR (Fig. 4). This allows the test participant to intervene interactively in the simulation and to receive tactile feedback. As a result, the test participant can put themselves into their everyday driving situation, test the available interface and experience the surroundings in a lifelike way. The driver can also turn the real steering wheel during the interview. To display this turning motion reflected in the VR, an optical tracker is installed in the center of the steering wheel.

Fig. 4.
figure 4

Mixed mockup of hardware seat and steering wheel (Daimler AG 2018)

Full size image

In the VR simulator both keypads on the steering wheel include functions. With the left keypad, the buttons can be used to operate the IC, with the right keypad participants control the centre console display (IVI).

To carry out tests flexibly and quickly with the Guerilla method, the simulator is built into a Mercedes-Benz V-Class (Fig. 1). This provides a mobile UX lab. As Fig. 5 shows, the test participant is located during the interview on the rotated simulator seat and wears the HMD with built-in headphones. The HMD require two sensors for positioning in the space, which are shown in Fig. 5 as red boxes.

Fig. 5.
figure 5

Installation of the VR simulator in a Mercedes-Benz V-Class (Daimler AG 2018)

Full size image

The original truck steering wheel with the function keypads and optical tracker in the center are placed straight in front of the test participant. The steering wheel is placed at the same height as in the truck.

During the test phase, the interviewer sits on the rear bench of the V-Class, controls the simulation via the laptop and asks questions associated with the selected scenarios.

The test team also includes an observer and a note taker. The observer takes up the position on the co-driver’s seat and observes the interview scene closely. The observer documents how the test participant interacts with the simulator and the interviewer. An additional note-taker in the driver’s seat documents what the test participant says.

3 Experimental Evaluation

Both HMI mockups, the steering wheel and the VR environment, include the same HMI concept of the new Daimler Trucks Actros Cluster Display. This concept was exemplarily evaluated during the test phase. Additionally, it is important to mention that in this paper we only report on the differences encountered for the two different simulators and not the results of the HMI-evaluation. Both evaluations where carried out with identical procedure and material.

The evaluations took place at truck parking lots. Once there, truck drivers are approached impromptu and recruited for an interview, without offering any payment. The interviewer introduced the team, stated the reason for the study and mentioned that the interview will take around 20 min.

In total 78 participants were approached on truck stops in Germany, resulting in 10 test participants for each HMI mockup. Once a participant has agreed to take part, the driver’s demographic data are collected and their consent to record the interview.

After an introduction of the technology, the structured interview on usability and UX of the HMI prototype started. The interviewer asks the questions from the structured interview guide. The test participant answers, normally resulting in a fluid conversation and interacts parallel with the simulator and the HMI. The note-taker documents the statements. In our particular setup, the observer monitored the interaction between participant and interviewer as well as the interaction with the simulator during the whole interview in order to collect data about the use of the particular mockup.

4 Results

The results compare both HMI testing prototypes in order to identify benefits and disadvantages of each concept.

4.1 Test Participants Data

Among all drivers approached to take part in the study, one quarter accepted to participate. Main hindering were limited time due to soon departure or missing German language skills.

Table 1 shows that our test participant sample mainly consists of male drivers. Just one female truck driver was encountered at the truck stops and she agreed to participate to the test with the steering wheel mockup. Female truck drivers represent only 1.8% of the truck drivers in Germany (Statista 2020) and some apparently prefer to stay in their truck cabin while the male often stand outside of their trucks during break time.

Table 1. Test participants statistics
Full size table

The test participants with the steering wheel mockup have an average age of 54, with the VR simulator an average age of 49. The age is also reflected in the professional experience. On average, the older test participants of the steering wheel mockup have seven years more experience driving a truck. Two test participants already actively used VR. The others had no practical experience with this technology. Furthermore, mainly drivers with German license plates took part in the two evaluations. Given that the interviews require good German skills, this is a consequence of this selection criteria. All participants haul freight nationally and, in some cases in adjacent EU countries. The test participants of both interview phases constitute typical representatives of their vocational group (Statista 2020).

4.2 Interview Duration

Due to the time constraints of the test participants it is important to gain as much information as possible in little time. Table 2 shows that the VR simulator tests require 3.5 min more than the test sessions in the steering wheel mockup.

Table 2. Interview duration
Full size table

4.3 Answer Quality and Variance

The longer test duration with VR is partly a consequence of more relatedness of the test situation to truck drivers’ daily routines. The VR achieved a far higher immersion into the situation, due to the representation of the avatar hands and the steering wheel position and buttons in VR. This could be achieved even though the VR prototype still suffered from some technical insufficiency that resulted in problems in the interaction.

Due to this wider scope of imagination and relating the test situation more to their daily life, the participants address a wider scope of their problems and personal solutions. Not all of them are relevant for the specific HMI test, but some represent situations that are relevant and were not mentioned with the steering wheel mockup. On the other hand, the variance of answers directly related to the HMI was higher with the steering wheel mockup, as the post hoc cluster analysis of qualitative answers indicates. A reason might be, that the HMI can be less intuitive without context. A relevant context often explains HMI functions that are misleading without the context. Based on observation we would say that the rich VR context did not distract the participants’ attention from the relevant HMI function. Hence, their answers may provide a higher external validity and decisions taken based on VR testing may better fit to the real use cases.

4.4 Utility Analysis of Both Simulators

The method utility analysis provides a tool for the specific, detailed assessment of the different demonstrator alternatives (Windolph 2015). This shows the clear, transparent representation of decisions (Müller 2008) and enables a detailed documentation, which allows anyone interested to trace the individual steps (Windolph 2015). As a result, it documents which simulator offers the greater utility.

The first step in the utility analysis includes the determination of comparable criteria. Therefore a cluster analyses is important. The result of the analysis is presented in the column “criteria”. Moreover, a valuation of the criteria is necessary. Windolph (2015) recommends an expert workshop to define the weightings. In such a workshop, an expert team from Daimler Trucks first defined the importance of the different criteria and second evaluated the forms in both simulators.

Next the implementation-rating per simulator of the shown criteria is represented in the column (c) and reaches from 0 to 10, while 10 being the best value.

In Table 3 the weighting factors of the respective subcriteria are multiplied by the target values (d = b * c) and then the sum formed of all part utility values for each superordinate criterion (e = ∑d). This value is then multiplied by the weighting factor of the superordinate criterion (e * a), giving its utility value per superordinate criterion (Haedrich und Tomczak 1996).

Table 3. Utility analyses comparing steering wheel mockup and VR simulator
Full size table

The results are presented in Table 3.

The first criterion in Table 3 is representation of technical components. The VR simulator is the winner by comparison, because it allows the test participants to locate themselves in the future cab design in the relevant driver position and in the context of driving. Thereby the 360° video and the rotatable steering wheel enables an interaction with the simulation.

The second criterion in the technology area is the mobility of the test environment. As it turned out the steering wheel mockup is much more unwieldy than expected. It is difficult to find a suitable place during the test. Either there is no interaction between the test participant and simulator, or the test participant must hold it all the time. This is solved much better by the V-Class with the VR simulator.

The third criterion is human perception, whereby the VR simulator dominated again with its high degree of interaction. It enables the driver to be fully immersed in the fictitious world and perceive themselves and their hands in the VR and point actively to virtual objects. In addition the VR setup provided ambient noises which increase the immersion. However, the ambient noise combined with the questions sometimes lead to sensory overload. None of the test participants reported any discomfort.

When analyzing the main quality criteria it turns out that the objectivity is a distinct shortcoming of the Guerrilla testing method. The objectivity of the result depends substantially on the structured interview guide, and the number and selection of participants. This was kept constant for both simulators. The VR Simulator dominated in the other both subcategories, especially in the validity of the answers.

The last comparable category are the secondary quality criteria, where the steering wheel mockup shows better results. It is much cheaper than the VR setup, requires little effort to create and transport and requires a little less time to conduct quick user tests.

4.5 Technical Readiness of the VR Simulator

The VR simulator prototype was used for the first time in regular HMI testing and already unveiled its benefits regarding representation of the technical components, mobility of the test setup, human perception and immersion as well as the main quality criteria. However several technical limitations were noticed.

During the interaction of the participants with the HMI, they comment on what they see and sometimes point at it. For the interviewer this context was not visible.

The HMD got too warm after prolonged use, which explicitly constitutes a comfort problem when carrying out studies in the summer. The HMD is difficult to clean due to their foam pads and adjustment straps.

The viewing angle of the HMD is modestly restricted and the resolution requires bending forward in order to read text on the cluster display.

Depending on the individual’s stature, a bigger or smaller distance to the steering wheel mockup is required, and hence an adjustable steering wheel is needed. In addition, many test participants noted that the virtual representation of the buttons on the steering wheel do not meet their requirements. A great deal of practice is needed to use these in VR due to imperfect matching for the use of such small buttons.

Also the realization of a real driving task or at least a moving steering wheel with force feedback would be an important improvement, since we found that many participants missed the actual driving as primary task while evaluating the HMI as secondary task. They note the desire for a test in reality, underscoring the limits of the fictitious world.

5 Discussion

The comparison of the steering wheel and the VR simulator show the respective benefits and disadvantages of both approaches. The Guerilla interview method is applicable with both prototypes.

With the VR simulator, interviews take approx. 3 min longer which is a relevant time when quick tests are necessary. These 3 min can be attributed to the time needed to equip the participants and for the tracking calibration. This procedure may become a bit faster in future when more participants have experience with VR and the tracking procedure. Also some technical issues of the VR prototype added some minutes to the test duration in some trials. In the further development, the robustness should be addressed in order to reduce testing time. The VR simulator also provides much more impressions, especially the whole truck cockpit and a driving scenario. For this reason test participants also tell more anecdotes about their work life when they are remembered by the complex driving scenario. This also adds some extra time to the test duration, however improves the external validity of the statements and contributes to higher answer quality.

The higher external validity of the answers lead to a better transfer of the HMI results into the later product. On the other hand the steering wheel mockup created more answers directly related to the specific HMI under investigation. It might be, that the context provided in the VR simulator reduced some doubts resulting from the HMI when it is presented without the context. It may also be, that the richer context in the VR distracted the participants focus on the specific HMI under investigation. Both hypothesis need more attention and data in following studies.

The utility analysis replicated these findings and shows the main benefits of VR simulation over traditional mockups. It also shows the disadvantage of slightly higher effort. This might be the main decision criteria when selecting the best testing environment. The traditional steering wheel is cheaper to produce and to maintain and a bit quicker in application, while the VR simulator provides higher external validity.

There is still a lot of improvement possible in the VR simulator. The technical readiness is already now acceptable and provides relevant results in HMI testing. However it is obvious that technical development is fast and can eliminate many disadvantages encountered in this study. Given the higher benefit of the VR solution it is recommendable to keep on improving this approach.

6 Conclusion

By way of conclusion, the initial question regarding the improvement of the contextual integration can be answered. The VR simulator constitutes a sound tool as part of UX studies. With its aid, test participants can immerse themselves in fictitious worlds and so experience an immersive user experience. Thanks to the realistic context integration, high-quality, verified UX statements are generated.

The VR simulator in comparison with the click-dummy convinces in two categories: utility and generating enthusiasm. The fully-fledged immersion and presence within the VR also leads to a higher degree of acceptance and taking the interviews seriously among the test participants.

This may result in more verified decisions regarding the utility of the presented display functions. To conclude the relevance of this aspect shall be investigated in further studies.

The extent to which the recorded statements can be transferred to reality can also be validated based on the gained insights. The analysis of the feedback shows that the test participants set the HMI prototype in the context of their personal everyday working lives. They not only answer the questions in the interview guide, but also recount related anecdotes from their professional lives.

The integration requires, however, for the method and the presentation tool to have a certain degree of flexibility, efficiency, agility and mobility. The use of the VR simulator in combination with the evaluation methodology implements all the required aspects.

Solely with regard to controlling the displays, the HMI exhibits a shortcoming due to the discrepancy between the real and fictitious steering wheel in relation to the generated hand representations.

Given the lack of implementation of driving tasks in the simulation, no conclusions could be drawn about the control of the displays while driving during this study.

At the same time, high-quality HMI statements are generated with a minimum outlay of time, human resources and money. To this end, the interviews are conducted at truck stops and so in the familiar working environment of the drivers. This additional effort compared with stationary test environments, however, proves worthwhile. The experienced respect by having the field researchers approach the test participants and the specialist interest in their working environment leads to a successful test sequence most of the time.

The qualitative analysis of huge amounts of data is the only shortcoming with the evaluation methodology. It takes a long time to complete and is as of now the most costly element of the process. Development of an alternative method for the time-consuming qualitative analysis should be developed and is in fact the next step in a following study.

The analysis shows many technical shortcoming of the first technical implementation of the VR system. Apart from optimizing the precision of hand & actuator positions, also risk of simulator sickness must be further reduced to allow longer test times. Readability of HMI text should be further enhanced to remove this distraction from the perceived user experience. System resources have to increase drastically to cover up need for higher frame-rates and resolution.

7 Outlook

In the meantime, many conclusions of this study have found its way into implementation:

  • Hand tracking has been improved by hardware & software updates & changes of the tracking strategy

  • Resolution has been elevated by using a new generation of HMD

  • Tracking loss has been reduced using redundant sensors

  • Interview and protocol interfaces are now wireless and separated to the different roles

  • Active driving and steering in an 3D environment is implemented

  • Seating and packaging of the Mercedes-Benz V-Class has been adjusted for the interview situation

  • Computing power has been increased using a fixed tower PC unit in the V-Class separated from the interfaces.

  • Eye tracking & real time tracking of driving behavior and HMI usage indicators has been implemented

Apart from fixing technical difficulties & quality enhancements, the biggest additions to the setup will be an active driving task & adding real time tracking of driving & other behavioral data.

Future evaluations of the simulator will tell us in which extent we can draw conclusions about real driving behavior in respect to HMI usage out of those additions to our VR driving simulation.

Through the further development of optical systems, we are getting closer and closer to actual human perception. With each hardware generation new applications open up that were previously not suitable to be displayed in VR due to their quality requirements. Especially now it is very important to look at applications that we previously excluded from our research, since the possibilities are developing very rapidly. Therefore, simulations will definitely gain importance for user experience and usability testing in the future.

But aside from our “Deep UX” development project, VR has a lot more useful applications in automotive research and development. For example, the technology allows designers to view our vehicles in their original size at a very early stage in the design process. We can implement changes to the virtual vehicle live and assess their design impact directly. Therefore, using virtual reality in simulations is a powerful tool in many use cases—no matter if it is about looking at complex driving scenarios, facing geometric challenges or working on three-dimensional models of future products.

The long-term aim must be to fully implement virtual reality. This should be no different from the real world, allow for any interaction and be deployable worldwide. Implementing all these ascertained aspects will promote even better statements, greater utility and a better fusion between evaluation methodology and simulator.