Applied Ergonomics 104 (2022) 103807

Contents lists available at ScienceDirect

Applied Ergonomics
journal homepage: www.elsevier.com/locate/apergo

Development and evaluation of design guidelines for cognitive ergonomics

in human-robot collaborative assembly systems
Luca Gualtieri a, Federico Fraboni b, Matteo De Marchi a, Erwin Rauch a, *
a
Industrial Engineering and Automation (IEA), Faculty of Science and Technology, Free University of Bozen-Bolzano, Piazza Università 5, 39100, Bolzano, Italy
b
Department of Psychology, Università di Bologna, Via Zamboni 33, 40126, Bologna, Italy

A R T I C L E I N F O A B S T R A C T

Keywords: Industry 4.0 is the concept used to summarize the ongoing fourth industrial revolution, which is profoundly
Collaborative robotics changing the manufacturing systems and business models all over the world. Collaborative robotics is one of the
Collaborative assembly systems most promising technologies of Industry 4.0. Human-robot interaction and human-robot collaboration will be
Cognitive ergonomics
crucial for enhancing the operator’s work conditions and production performance. In this regard, this enabling
Human factors
Industry 4.0
technology opens new possibilities but also new challenges. There is no doubt that safety is of primary impor­
tance when humans and robots interact in industrial settings. Nevertheless, human factors and cognitive ergo­
nomics (i.e. cognitive workload, usability, trust, acceptance, stress, frustration, perceived enjoyment) are crucial,
even if they are often underestimated or ignored. Therefore, this work refers to cognitive ergonomics in the
design of human-robot collaborative assembly systems. A set of design guidelines has been developed according
to the analysis of the scientific literature. Their effectiveness has been evaluated through multiple experiments
based on a laboratory case study where different participants interacted with a low-payload collaborative robotic
system for the joint assembly of a manufacturing product. The main assumption to be tested is that it is possible
to improve the operator’s experience and efficiency by manipulating the system features and interaction patterns
according to the proposed design guidelines. Results confirmed that participants improved their cognitive
response to human-robot interaction as well as the assembly performance with the enhancement of workstation
features and interaction conditions by implementing an increasing number of guidelines.

1. Introduction and production performance by bringing together the individual agent

contributing strengths (e.g. humans’ versatility and dexterity with ro­
1.1. Problem statement bots’ speed and accuracy), which compensate for the weaknesses of the
other, rather than negating the need for one agent or the other (Michalos
New technological advancements are shaping the current transition et al., 2015). Collaborative robotic arms and related integrated devices
of many organizations towards the concept of Industry 4.0, which has (e.g. collaborative end-effectors) present hardware and software solu­
been typified by new levels of sociotechnical interaction between tions that allow the implementation of a collaborative robotic system1
different manufacturing assets across the supply chain (Kadir and Bro­ and, as a consequence, a “collaborative application”. The greatest
berg, 2021). Previous authors connected Industry 4.0 to advanced dig­ innovation with respect to “traditional” industrial robotics is related to
ital technologies such as Autonomous and Collaborative Robots (Kadir the possibility for the operator to voluntarily (functional interaction, i.e
and Broberg, 2021; Rüßmann et al., 2015). The International Federation planned action) or involuntarily (non-functional interaction, i.e unex­
of Robotics (IFR) defines industrial collaborative robots as those able to pected/unwanted contact) interact with the robotic system under
perform tasks in collaboration with workers in industrial settings (In­ certain controlled conditions that have to be carefully evaluated through
ternational Federation of Robotics, 2020). Industrial Human-Robot a risk assessment (International Organization for Standardization,
Interaction (HRI) aims at improving both operator’s work conditions 2010). This allows the operator to interact with the robotic system

* Corresponding author.
E-mail addresses: luca.gualtieri@unibz.it (L. Gualtieri), erwin.rauch@unibz.it (E. Rauch).
1
Note that in this work a “robotic system” is defined as the integrated system composed by the robot (arm), the controller, the end-effector and possible related
devices (e.g. sensors) needed to properly perform production tasks.

https://doi.org/10.1016/j.apergo.2022.103807
Received 16 April 2021; Received in revised form 15 August 2021; Accepted 18 May 2022
Available online 25 June 2022
0003-6870/© 2022 Elsevier Ltd. All rights reserved.
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

physically, flexibly, and safely, without the need to isolate tasks and include guidelines and indications on how to realize effective and state
workspaces due to safety reasons (e.g. by using safety fences) (Gualtieri of the art solutions (e.g. products, systems, services, etc.). At the
et al., 2020a). moment, deliverables (e.g. standards) and related guidelines for the
HRI can generally be considered the broad category of actions that design of human-centered and cognitive-oriented industrial systems
can determine mutual or reciprocal influence between humans and ro­ based on HRI (e.g. CASs) are missing.
bots. Communication, cooperation and collaboration between humans In that regard, the present paper ultimately aims to provide sug­
and robots can be considered sub-categories of HRI (Lasota et al., 2017). gestions and recommendations, in the form of guidelines, for technicians
Human-robot collaboration (HRC) can be considered the most advanced to consider human factors and operators’ cognitive ergonomics when
implementation of HRI in the industry. While HRC has been generally designing and implementing CASs. The guidelines shown in the present
referred to as the use of industrial robotic systems without safety fencing work can be a starting point for future research and investigation related
(Bauer et al., 2016), authors argued that HRC should be described as a to human factors and ergonomics aspects in collaborative robotics.
series of joint actions towards a common goal, in which operators and Neglecting cognitive ergonomics when designing CASs may involve
robotic systems work simultaneously on the same product or component various risks for the operator and lead to a considerable worsening of
by adapting to each other (Kolbeinsson et al., 2019). working conditions (Cascio and Montealegre, 2016; Czerniak et al.,
Industrial HRI introduces multiple advantages but also challenges 2017; Fletcher et al., 2020; Kong, 2019; Adam et al., 2018; Kalakoski
(Gualtieri et al., 2020a). In this regard, one of the most interesting and et al., 2020; Brun and Milczarek, 2007), as mentioned earlier. In addi­
challenging applications is product assembly. This is a very promising tion, addressing cognitive aspects, such as task allocation, could increase
way to make production more flexible and agile by responding to the the system’s overall efficiency and significantly improve potential
ever more demanding requirements of Industry 4.0. Collaborative As­ returns for investing in collaborative robotic technology (Gjeldum et al.,
sembly Systems (CASs) are a real example of semi-automated and 2021). With that said, the target audience will be mainly composed of (i)
human-centered manufacturing systems where operators and machines industrial designers and systems integrators (with no/limited expertise
interact for the assembly of manufacturing products. CASs entails new in human factors but expertise in manufacturing and/or robotics), (ii)
forms of interaction between humans and automation and profound researchers interested in the topic and, (iii) policy-makers bodies (e.g.
changes in work both at the operational level and in its nature (Kadir EUOSHA or ISO).
and Broberg, 2021). Those changes also imply human factors and The article is structured as follows. Section 1 provides the problem
ergonomics-related challenges and risk factors such as stress and statement according to the review of the scientific literature and ex­
burnout (Cascio and Montealegre, 2016), information overload (Czer­ plains the process for the development of the guidelines. Section 2 de­
niak et al., 2017), workers’ safety (Fletcher et al., 2020), increasing scribes the guidelines as well as the materials and methods adopted for
cognitive load (Kong, 2019), frustration and loss of motivation (Adam their experimental evaluation. Section 3 summarises the main qualita­
et al., 2018). Previous authors, addressing cognitive risk factors in work tive and quantitative results. Finally, the discussions and conclusions are
environments, highlighted that excessive levels of cognitive workload, summarised in Section 4 and Section 5, respectively.
as well as low usability, entails risks to workers’ performance and
wellbeing (Kalakoski et al., 2020). In fact, excessive cognitive demands 1.2. Guidelines development
in work environments can lead to cognitive failures that affect overall
performance (Kalakoski et al., 2020). These risks, related to the design of The study of cognitive ergonomics in the field of industrial collabo­
work tasks, technologies and environments may have a detrimental rative robotics is still in its infancy. After preliminary studies on the topic
impact on both the mental and physical health of employees (Brun and (Gualtieri et al., 2021a; Gualtieri et al., 2021b; Fraboni et al., 2021), the
Milczarek, 2007). In that regard, themes like trust (Alarcon et al., 2021; guidelines’ content was identified based on a detailed analysis of the
Kim et al., 2020), acceptability (Zanchettin et al., 2013), and scientific literature. This was performed in spring 2020 according to a
human-robot teaming (Dehais et al., 2011; Teo et al., 2018), have been systematic approach. Scopus was used as the electronic database for
preliminarily studied in the field of social and industrial HRI. keywords search. The authors identified it as being the most relevant for
Previous research thus shows that cognitively straining conditions publications in the area of collaborative robotics. A previous control of
can have direct effects on task performance, as well as indirect, extensive other sources such as ISI Web of Knowledge, Emerald, and Science
effects on work performance and productivity if they expose employees Direct did not show any major changes in relation to adding to the
to cognitive failure and impair occupational safety and health. Occu­ sources. The keywords used in this analysis were identified by prelimi­
pational safety is widely perceived as of primary importance when narily reading different articles on the topic related to the role of
humans and robotic systems have to work together in industrial settings cognitive ergonomics in industrial HRI. In particular, to make the
(Gualtieri et al., 2020a). On the other hand, human factors and cognitive research as complete as possible, the following research keywords have
ergonomics are often underestimated or ignored when designing and been used: “ergonomics” OR “ergonomic” OR “human factors” OR
implementing HRIs and therefore CASs, even if they are crucial for the “human-factors” OR “cognitive ergonomics” OR “psychological risk” OR
operator’s wellbeing and production performances (Cascio and Mon­ “psychological” OR “work stress” OR “work-related stress”) AND
tealegre, 2016). These issues have been little considered even in (“Collaborative Robotics” OR “Human Robot” OR “Collaborative Robot”
academia until recently (Gualtieri et al., 2021a). Human factors and OR “Human Robot” OR “CoBots” OR “Human-Robot” OR “HRI” OR
ergonomics are fundamental in the design of high-tech, automated and, “HRC”) AND (“industrial” OR “industry”). In addition, the following
complex systems (e.g. robotic systems), since they are strictly related to constraints were applied: “article title/abstract/keywords” as search
the operator’s safety, wellbeing, and work-related performance (Thor­ fields, “final” as publication stage, “English only” as language and "≥
vald et al., 2017). 2011′′ as time-period. The year 2011 was selected since it represents the
The present work thus focuses on cognitive ergonomics in industrial beginning of the Industry 4.0 era. Besides, considering the novelty of the
HRI. It refers to the development and experimental evaluation of a set of topic, the author considered this period as large enough for this analysis.
design guidelines related to cognitive ergonomics in CASs. These have This search resulted in 140 documents. To ensure the validity of such
been identified by systematically analyzing the scientific literature. A results, the authors used a coding scheme applied to the results by using
laboratory case study has been used for the evaluation of such a score of 1 or 2. In that regard, 2 denotes high appropriateness, while 1
guidelines. denotes low appropriateness. The screening was carried out in two
In industrial engineering, one of the main tools for helping designers phases by three independent researchers with previous knowledge on
in the fulfillment of design requirements is represented by technical the topic. The first phase evaluated the title and abstract, while the
standards and deliverables (What Is an European Standard, 2020). These second one referred to the read of the whole paper. Then the authors

2
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

computed inter-rater reliability for each paper by considering the dif­ and human trust and found that an industrial appearance leads to better
ference in scoring. In the case the three independent scores came to the collaborative performance without influencing trust levels. In another
same conclusion (i.e., zero differences or the highest interrater reli­ work, Müller et al. (2017b) analyzed the subjective stress level in HRI
ability), the papers were considered. On the other hand, if differences in and found that the robot’s appearance and behavior do not affect the
the scores occurred, related papers were discussed to result in a total subjective stress level. Richert et al. (2016a, 2016b) preliminarily
agreement between the experts. This analysis finally resulted in 32 ar­ introduced and discussed the effects of appearance and behavior on
ticles that have been used as starting content for the development of the human trust and stress level in HRI by using a virtual simulation.
guidelines. Furthermore, the authors categorized each article basing on Johnson et al. (2016) investigated the effectiveness of different light
its content and main results and according to the four main categories indicators for HRI and found that the light system integrated with the
that can characterize a CAS (see Table 1 for details) (Gualtieri et al., robot presented the shortest human reaction time. Schmidtler et al.
2020b): (i) workstation layout and elements, (ii) robot system features, (2015) discussed the effect of robot arm contrast in HRI revealed that
(iii) robot system performance and (iv) organizational measures. Iden­ higher contrasts lead to higher operator distraction. Weistroffer et al.
tified article topics and results are summarised in the following. In (2013) provided a methodology to assess the acceptability of HRI and
Table 1, each paper has also been categorized according to the cognitive showed that a more anthropomorphic robot, both in its appearance and
variables explicitly addressed in the specific study. movements, is not necessarily better accepted by the users in a collab­
oration task.
1.2.1. Workstation Layout and Elements
Bitonneau et al. (2017) presented a simulation-based approach for 1.2.3. Robot System Performance
interactive and inclusive robot system design and observed an Rojas et al. (2019) proposed a more human-like trajectory-planning
improvement in the design process effectiveness. El Makrini et al. (2017) for collaborative robots, which users perceived as less stressful. Kaufeld
developed a collaborative architecture for enhanced HRI during as­ and Nickel (2019) provided evidence for lower mental workload when
sembly, allowing better human-robot communication, providing a per­ robots acted less autonomously while the operator was informed (by
sonal experience, and allowing a more intuitive interaction. Gopinath audio-visual signals) about upcoming HRI. Petruck et al. (2018) pre­
et al. (2017) presented a risk assessment-based design of a collaborative sented an ergonomic concept for collaborative workstations to avoid
assembly cell by analyzing, among other things, how to manage human high mental load by discussing the understandability and predictability
errors through feedback interfaces effectively. of robot actions as well as its acceptance and trust. Koppenborg et al.
(2017) investigated the robot’s motion speed and predictability in HRI,
1.2.2. Robot System Features discovering a decrease in task performance for a lower level of pre­
Tang et al. (2019) analyzed the effect of light-based signaling sys­ dictability, while faster movements resulted in higher values for task
tems for HRI and communication by improving user’s awareness and load and anxiety. Müller et al. (2017b) analyzed the subjective stress
reducing the workload. Changizi et al. (2018) evaluated the use of the level in hybrid collaborations and found that the robot’s appearance and
robot as an assistance system by using the hand-guiding modality, which behavior do not affect the subjective stress level. Richert et al. (2016a,
users perceived as comfortable, controllable, and helpful. Richert et al. 2016b) preliminarily introduced and discussed the effects of appearance
(2018) examined the relationships between robot design and personality and behavior on human trust and stress level in HRI by using a virtual
and discovered that humanoid appearance might be better in the case of simulation. Weistroffer et al. (2013) provided a methodology to assess
close (industrial) collaboration. Kadir et al. (2018) identified the the acceptability of HRI and showed that a more anthropomorphic
emerging opportunities, challenges, and critical design factors in HRI robot, both in its appearance and movements, is not necessarily better
that need to be addressed to maximize the technology’s benefits. Fu and accepted by the users in a collaboration task. Brecher et al. (2013)
Zhang (2018) modeled a robot design scheme from the perspectives of developed methodologies and techniques to transform human move­
emotions and psychology and provided a set of indications to make ment trajectories so that industrial robots can execute them to improve
people feel comfortable and safe. Müller et al. (2017a) examined the operator’s acceptance. Kuz et al. (2013) and Mayer et al. (2013) studied
impact of the robot’s appearance and behavior on team performance anthropomorphism in HRI and found that anthropomorphic

Table 1
Articles classification according to main cognitive variables and related interaction variables.
Main Cognitive Interaction Variable
Variable
Workstation Layout Robot System Features Robot Systems Performance Organizational Measures
and Elements

Trust (Fu and Zhang, 2018; Müller et al., (Petruck et al., 2018; Richert et al., 2016a) (Nelles et al., 2018; Fletcher et al., 2019)
2017a; Richert et al., 2016a)
Usability (Kadir et al., 2018; (Tang et al., 2019) (Nelles et al., 2018; Schleicher and Bullinger,
Gopinath et al., 2018; Rosen et al., 2018)
2017)
Frustration (Kadir et al., 2018)
Perceived (Rosen et al., 2018)
enjoyment
Acceptance (El Makrini et al., (Changizi et al., 2018; Richert (Petruck et al., 2018; Weistroffer et al., 2013; (Nelles et al., 2018; Rosen et al., 2018; Fletcher
2017) et al., 2018; Weistroffer et al., Brecher et al., 2013; Kuz et al., 2013; Mayer et al., 2019; Charalambous et al., 2015, 2016a,
2013) et al., 2013) 2017; Charalambous and Stout, 2016)
Stress (Kadir et al., 2018) ( (Fu and Zhang, 2018; Müller et al., (Rojas et al., 2019; Petruck et al., 2018; (Nelles et al., 2018; Rosen et al., 2018)
Gopinath et al., 2017b; Richert et al., 2016a, Koppenborg et al., 2017; Müller et al., 2017b;
2017) 2016b; Schmidtler et al., 2015) Richert et al., 2016a, 2016b; Bortot et al.,
2013)
Cognitive (Bitonneau et al., (Tang et al., 2019; Johnson et al., (Kaufeld and Nickel, 2019; Petruck et al., (Bragança et al., 2019; Nelles et al., 2018;
workload 2017) 2016) 2018) Rosen et al., 2018)

3
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

Table 2
Guidelines for the design of CASs (workstation) considering cognitive ergonomics requirements.
Interaction Variable Code Guideline Reference

Workstation Layout WE.CE.1 Provide measures to transfer the graphic user interface onto the Schleicher and Bullinger (2018)
and Elements collaborative workspace;
WE.CE.2 Locate the robot arm as distant as possible from the operator’s Nelles et al. (2018), Schmidtler et al. (2015), Weistroffer et al.
position; (2013)
WE.CE.3 Design low-contrast workstation elements with respect to the robotic Schmidtler et al. (2015)
system;
Robot System RF.CE.1 Realize a fluent and smooth robotic system design (avoid bulky joints, Changizi et al. (2018), Charalambous and Stout (2016), Richert
Features wires, external arm components, mechanized shape); et al. (2016b)
RF.CE.2 Design cold-white robot arm; Changizi et al. (2018)
RF.CE.3 Design a low-contrast robot arm with respect to the workstation Schmidtler et al. (2015)
elements;
RF.CE.4 Design robotic system and related devices with industrial appearance Müller et al. (2017b), Charalambous and Stout (2016), Richert
(avoid adding human-like features, e.g. anthropomorphism, and et al. (2016b), Weistroffer et al. (2013)
social appearance for robotic systems to be used in industrial
contexts);
RF.CE.5 Design on-board devices (mounted on the external surface of the Tang et al. (2019), Johnson et al. (2016)
robotic system) for the visual communication of the status of the
robotic system;
RF.CE.6 Demonstrate the operators about the effectiveness and reliability of Tang et al. (2019), Kaufeld and Nickel (2019), Charalambous
safety measures of the robotic system; and Stout (2016)
RF.CE.7 Demonstrate the operators about the efficiency and operational Tang et al. (2019), Kaufeld and Nickel (2019), Charalambous
reliability of the robotic system (e.g. the end-effector); and Stout (2016)
Robot Systems RP.CE.1 Design human-like-inspired/smooth/fluent and non-disruptive Rojas et al. (2019), Charalambous et al. (2017), Gopinath et al.
Performance robotic system actions; (2017), Charalambous and Stout (2016), Weistroffer et al.
(2013), Brecher et al. (2013), Kuz et al. (2013), Mayer et al.
(2013)
RP.CE.2 Provide measures for the implementation of a medium-level robotic Kaufeld and Nickel (2019)
system autonomy;
RP.CE.3 Provide measures for the manual adjustment of robot arm speed Changizi et al. (2018)
according to operator’s needs;
RP.CE.4 Design comprehensible and predictable robotic system actions (avoid Petruck et al. (2018), Charalambous and Stout (2016),
supposedly arbitrary actions of the system); Weistroffer et al. (2013), Kuz et al. (2013), Mayer et al. (2013),
Bortot et al. (2013)
RP.CE.5 Provide measures for the automatic adaptation of robot arm speed to Fletcher et al. (2019)
correspond with an operator’s profile (i.e. expertise, skills,
capabilities, preferences, trust level);
RP.CE.6 Design slow robotic system actions and related motions (related to the Schleicher and Bullinger (2018), Charalambous and Stout
kind of collaborative task); (2016), Schmidtler et al. (2015), Weistroffer et al. (2013)
RP.CE.7 Avoid unreliable/inaccurate performance of the robotic system; Müller et al. (2017b), Richert et al. (2016a), Charalambous and
Stout (2016)
RP.CE.8 Inform the operator about the robotic system speed; Bortot et al. (2013)
RP.CE.9 Inform the operator about the robotic system behavior/state; Bortot et al. (2013)
RP.CE.10 Avoid variations in robot arm velocity; Bortot et al. (2013)
Organizational OM.CE.1 Suggest work breaks to improve performance and concentration Bragança et al. (2019)
Measures (suggestions could be based on age and the monitoring of
psychophysical parameters);
OM.CE.2 Provide information about the workstation systems (including the Bragança et al. (2019)
robotic system) only when relevant and necessary;
OM.CE.3 Provide measures that allow the operators to control the workstation Fletcher et al. (2019)
systems (including the robotic system);
OM.CE.4 Inform operators about the type and functioning of the specific safety Fletcher et al. (2019)
measures implemented in the workstation;
OM.CE.5 Provide functions of the workstation systems (including the robotic Fletcher et al. (2019)
system) that adapt to suit individual operator’s preferred working
methods;
OM.CE.6 Provide workstation systems (including the robotic system) that Fletcher et al. (2019)
adapt safety strategy to suit operator’s preferences and conditions in
the surrounding area;
OM.CE.7 Engage operators in workstation and interaction design (layout, Tang et al. (2019), Kaufeld and Nickel (2019), Nelles et al.
assembly cycle, robotic system performance and motions); (2018), Richert et al. (2016a), Charalambous et al. (2016a),
Charalambous et al. (2015)
OM.CE.8 Demonstrate the operator about the efficiency and reliability of the Tang et al. (2019), Kaufeld and Nickel (2019), Richert et al.
robotic system role; (2018), Charalambous and Stout (2016), Richert et al. (2016b)
OM.CE.9 Inform the operator about upcoming HRI; Kaufeld and Nickel (2019)
OM.CE.10 Provide as much as possible natural and intuitive communication Charalambous et al. (2017)
between the operator and the robotic system;
OM.CE.11 Provide training and empowerment to the operator (understand the Koppenborg et al. (2017), Richert et al. (2016a), Charalambous
abilities, the process complexity, the limitations of the robotic and Stout (2016), Charalambous et al. (2016a), Richert et al.
teammate and the reasons behind the events); (2016b), Charalambous et al. (2015)
OM.CE.12 Visualize alternative decisions to reduce biases in decision-making; Bragança et al. (2019)
OM.CE.13 Minimize the number of feedback interfaces; Müller et al. (2017a)
OM.CE.14 Inform the operator about the collaboration mode change (e.g. from Müller et al. (2017a)
automatic to collaborative);

Legend: WE = Workstation Layout and Elements; RF = Robot System Features; RP = Robot System Performance; OM = Organizational Measures; CE = Cognitive
Ergonomics.

4
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

characteristics embedded into the motion of industrial robots can have designers the possibility to interpret the indications by guiding them
positive effects on the prediction time and accuracy of the human towards a range of possible solutions. The guidelines are classified ac­
co-worker. Bortot et al. (2013) discussed the effects of robot motion cording to the four abovementioned interaction variables and sum­
trajectory on humans and showed that variable (i.e., non-predictable) marised in Table 2.
robot motions reduce human wellbeing and performance.
2.2. Cognitive ergonomic evaluation metrics
1.2.4. Organizational Measures
Bragança et al. (2019) investigated the potential use of collaborative In the following section, the primary cognitive variables and related
robots as assistance systems, also focusing on the cognitive aspects. risk factors to be tested are described according to the scientific litera­
Nelles et al. (2018) reviewed the scientific literature to analyze the ture. These will be used to quantify and assess the effectiveness of the
metrics for the evaluation of different variables associated with human developed guidelines through an experimental case study.
wellbeing and system performance in HRI. Schleicher and Bullinger
(2018) empirically validated a mixed-method framework for the • Trust: can be defined as the willingness to take the risk of being
user-centered design of HRI by developing an assistive surface-finishing vulnerable to the actions of others regardless of the ability to control
robot. Rosen et al. (2018) developed and validated a toolkit enabling the those actions (Mayer et al., 1995). Trust develops dynamically with
evaluation of the quality of HRI in different types of collaborative knowledge and experience and is often addressed as a calibration
workplaces through multiple cognitive variables. Fletcher et al. (2019) process between the actual reliability of the system and the level of
identified the requirements and gaps in ethics and safety standards for trust posed by the person interacting with it. Risk arises following a
HRI, considering the effects of trust and acceptance. Charalambous et al. dysfunctional calibration, which can lead to over-trust or distrusts
(2017) presented a theoretical framework of key organizational human (De Visser et al., 2020).
factors relevant to industrial HRC by developing a readiness level. • Usability: refers to the extent to which a system, product, or service
Charalambous and Stout (2016) discussed the application of HRC in a can be used by specified users to achieve specified goals with effec­
case study from a human factors perspective. In another study, Char­ tiveness, efficiency, and satisfaction in a specified context of use
alambous et al. (2016a) developed a roadmap with key human factors (International Organization for Standardization, 2018; Moorthy
that need to be considered in the design of HRI and, in (Charalambous et al., 2014). Usability has been mentioned to be at the foundation of
et al., 2015) they explored the key organizational human factors to be success before the market introduction or application of a specific
considered for the development of HRC. technology (Pei et al., 2017). It is linked with the acceptance of
Even if different aspects of human factors in industrial HRI have been technology, and authors argued that the introduction of a new sys­
studied in the last years, a set of inclusive and human-centered design tem might be useless unless it is liked and thus used by target users.
principles for the proper integration of cognitive ergonomics in CASs is Lack of usability can imply multiple risk factors for workers such as
missing. In particular, a guide for non-experts in cognitive ergonomics to incorrect task execution, high ratio of failure resulted from human
be applied during the early design stage as well as for the setting of the errors, longer execution time of a task, bad responses from users, loss
interaction conditions could be particularly useful for industrial com­ of information, making the work environment more prone to hazards
panies, especially for Small and Medium-sized Enterprises (SMEs). (Pei et al., 2017).
• Frustration: is a psychological state derived from an unsatisfied need
or unresolved problems. It is linked to increased speed of perfor­
1.3. Research questions
mance and increased error rate, particularly in complex tasks
(Moorthy et al., 2014);
The present work aims to develop and evaluate a set of design
• Perceived enjoyment: it is a feeling of joy or pleasure associated by the
guidelines for cognitive ergonomics in CASs. The main assumption to be
user with the use of the system. It has been studied in relation to
tested is that it is possible to improve the operator’s experience and
robots’ acceptance and intention to use. Lack of perceived enjoyment
assembly performance by manipulating the system features and inter­
can lead to lower levels of acceptance of the system and thus a lower
action patterns according to the design guidelines proposed in this work.
intention to use (Elprama et al., 2016).
According to this hypothesis, the following research questions are
• Acceptance: technology acceptance is the favorable reception of
derived:
technology as a useful and practical tool (Davis, 1989; Bröhl et al.,
RQ1. What are the main design principles to be applied when 2016). A low level of technology acceptance may introduce a risk for
designing or setting a CAS according to cognitive ergonomics? workers as it could compromise the success of interaction between
humans and robots;
RQ2. What is the operator’s response when manipulating the features
• Stress: in general, it is defined as the human body’s response to
and parameters associated with these design principles?
pressures from a situation or life event (Recipe for Stress, 2020).
RQ3. What are the effects on assembly performance? When this pressure exceeds certain limits, stress becomes a risk
factor, as it can have a detrimental effect on performance and can
2. Materials and methods lead to an increased error rate;
• Cognitive workload: refers to the cognitive effort that an individual
2.1. Guidelines development and classification shows during a task or to achieve a particular level of performance
(Hart and Staveland, 1988). Assuming that an individual’s cognitive
The following guidelines were developed according to the content of resources are limited, the more effort is requested by a task, and the
the identified relevant papers founded by systematically analyzing the higher is the cognitive workload. It affects both safety and
scientific literature. Therefore, the indications have been developed by performance;
properly combining and interpreting the conclusions of such validated
works. In the case of possible contradictive suggestions in the analyzed 2.3. Experimental set-up
studies, the authors discussed finding the best interpretation. The rela­
tionship between each guideline and related references is represented in The effectiveness of the guidelines has been evaluated through
the fourth column of Table 2. The authors believe that a “guideline” multiple experiments based on a HRI case study performed in the Smart
should be as general as possible, even if it should contain technical Mini Factory (SMF) laboratory (Gualtieri et al., 2018). This is a labo­
suggestions without being too detailed. This is necessary to leave ratory for teaching and research in the field of the main technologies of

5
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

Industry 4.0, particularly focusing on sustainable manufacturing sys­ were placed on the left side of the human workspace. The buttons array
tems and robotics. The experiment was conducted by using a dedicated and the LCD screen were also located in the same area to facilitate the
workstation (a CAS) for the collaborative assembly of a manufacturing participant’s activities, while the virtual button was located on the right
product. This was a simplified version of a pneumatic cylinder from side of the human workspace. The AI-based 3D vision system was
Kuhnke (see Fig. 1) (diameter of 32 mm and 50 mm stroke) with roughly installed on an aluminum structure in front of the human workspace to
20 different parts. To decrease the complexity of the tasks and to reduce frame the operator’s motion and her/his reachable space. The working
the assembly time, the structure of this product has been simplified to 14 table boundaries were covered with colored adhesive tapes to increase
parts. A description of the overall assembly process can be found in the visibility of the corners. The screwdriver was located on the right
(Gualtieri et al., 2020c). side of the human workspace in line with the operator’s head.
The main components of the workstation are following described Beyond the equipment’s physical arrangement, another design
(see Fig. 2). aspect that is worth to be described is the architectural design. Archi­
tectural design is the representation of the communication infrastruc­
• (1) Collaborative robot model Universal Robots UR3 (Universal Ro­ ture that links all the entities of a system. In this application, there were
bots, 2020) equipped with a (2) Robotiq (2020) collaborative five main actors taking part in the data exchange: the robot’s controller,
gripper; the robot’s arm (and the integrated collaborative gripper), the micro­
• (3) Fixed working table with assembly jigs; controller (that manages the output to the LCD screen) and, the buttons
• (4) Commands (button array) for HRI and emergency stop; array. The controller of the robot was the core component of the system.
• (5) Virtual button (to be activated using the AI-based 3D perception The robot’s arm and the buttons array were physically wired to the ro­
device and vision system); bot’s controller. The AI-based 3D vision system and the microcontroller
• (6) Boxes for the storing and picking of assembly parts; for managing the LCD screen communicated with the robot’s controller
• (7) LCD screen for displaying instructions and other information through the XML-RPC (eXtensible Markup Language – Remote Proced­
about the status of the robotic systems (graphic user interface (GUI)); ure Call) protocol. The software running on the microcontroller (a
• (8) AI-based 3D perception device and vision system (Smart Robots, Raspberry Pi 4) was a Python script that absolved two main tasks: (i)
2020) for HRI (it allows the implementation of the gesture recogni­ receiving commands from the robot’s controller and (ii) transforming
tion) and safety purposes (it allows the implementation of a collision them into meaningful information for the operator. The Python script
avoidance safety measure); was able to generate on-screen popup notifications and manage the in­
• (9) Screwdriver (Fiam (2020)). structions shown on the GUI (provided in OpenOffice slideshow format).
The chosen AI-based 3D vision system is a plug-and-play device and did
Furthermore, the workstation was designed to provide three main not require any peculiar adjustment for the interaction with the robot’s
workspaces: controller.

• The human workspace. It was defined at the right and left sides of the 2.4. Experiment description and scenarios
collaborative workspace (the robotic system cannot reach this area).
This space was intended for the participant for reaching the storage Advertisements about the possibility to participate in the experiment
boxes, using the button array and the virtual button, interact with the were sent through emails channels and social networks of the Free
GUI; University of Bozen-Bolzano. The requisite for inclusion was not having
• The collaborative workspace. It was defined between the participant previous experience in HRI and at least minimal experience in
and the robotic system in front of the participant’s seat (both the manufacturing operations or DIY. The experimenters verified the req­
participant and the robotic system can reach this area). This common uisites through a brief unstructured interview during the first contact
space was intended for the manual assembly of the product as well as with candidates.
for physical HRI; A total of 14 participants were involved in the study: 12 males and
• The robotic system workspace. It was defined as the opposite of the two females. The age of participants ranged from 23 to 57 years old. Due
collaborative workspace (the participant cannot reach this area to constraints related to the COVID-19 pandemic, it was possible to
while sitting). This space was intended for the robotic system to pick enroll only people who had some involvement with the University of
up and handle the parts to be manipulated autonomously. Bozen-Bolzano (primarily for the safety of participants and insurance-
related reasons), thus regarding participant occupation, six of them
According to the different sets of guidelines to be tested, the listed were researchers, one master student, and seven technicians/
features and related interaction modalities have been changed. The administrative.
components were mounted on the top of the working table. The The participants were asked to collaborate with the low-payload
collaborative robot was placed in the center of the working table so that collaborative robotic system to complete the assembly of the simpli­
its end-effector was able to reach both the collaborative and the robot fied pneumatic cylinder. Firstly, a training session (without the robotic
workspace easily. A set of jigs was fixed in front of the participant in the system) was provided in a dedicated training workstation to reduce the
collaborative workspace. There was the option to integrate foam pro­ occurrence of possible errors related to limited and heterogeneous
tections around the jigs to enhance safety and visibility. It was possible knowledge of the product and process. We sought this as a way to reduce
to add highlighting components to the robotic system to increase the the influence of the learning effect on the results. Additionally, in­
visibility of moving parts. The boxes for the storage of the assembly parts structions about the necessary steps to complete the task were displayed
on the workstation LCD screen across the experiments. Later, the par­
ticipants moved to the collaborative workstation and performed the
assembly tasks in collaboration with the robotic system. Table 3 explains
how the guidelines were implemented and manipulated in the experi­
mental case study. Unfortunately, it was not possible to test all the
guidelines in the present experiment because implementing all the
related solutions at the same time was not feasible from a technical
standpoint. In other cases, the achievable results would have resulted
redundant with other applied guidelines (potentially providing a detri­
Fig. 1. Pneumatic cylinder from Kuhnke. mental effect to the experiment). Furthermore, some guidelines referred

6
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

Fig. 2. Experimental set-up.

to generic industrial HRI and were not specifically applicable to HRC, as technology. Participants were asked to rate on a five-point scale (from
proposed in this case study (see i.e. WE.CE.2). Further details about the − 2 to +2) what level of these adjective continuums (e.g., “Effective/­
guidelines that were not implemented in the experiment will be pro­ Superfluous”, “Pleasant/Unpleasant”) they attributed to the robotic
vided in the discussion section (see also Table 8). system. The items were grouped into two sub-scales, one indicating the
The experiment consisted of executing the assembly task in three system’s perceived usefulness and the other the satisfaction resulting
different and sequential scenarios (Scenario1, Scenario 2 and, Scenario from the use of the technology. Cronbach’s alpha reliability coefficient
3, respectively). According to the different scenarios, different guide­ was 0.89.
lines were applied, and therefore the features of the workstation Cognitive Workload. Participants were asked to rate on a 5-point
changed accordingly. The assignment of the scenarios was non- Likert-type scale (1 = very low; 5 = very high) a single item (i.e.,
randomized. The idea was to provide them in such a way as to “How mentally demanding was the task?“) taken from the NASA-Task
improve the interaction conditions by gradually changing the solutions Load Index (NASA-TLX) (Hart, 2006).
implemented by the different guidelines. We assumed that Scenario 1 Frustration. Participants’ perceived frustration in collaborating with
would be the worst one, while Scenario 2 and 3 were supposed to be the robotic system was assessed through a single item (i.e., “Did you
gradually better from a cognitive perspective. A summary of the appli­ experience frustration while performing the collaborative task with the
cation of the guidelines according to different scenarios is presented in robot?“) rated on a 5 point scale (1 = not at all; 5 = a lot).
Table 4. The presumed worst case (Scenario 1) was supposed to simulate Perceived Enjoyment. Participants’ perceived enjoyment related to
a CAS developed without accounting for human factors during the interacting with the robotic system was as well assess through a single
design process. Considering state of the art, Scenario 1 can reasonably item (i.e., “Did you enjoy interacting with the robot?“) rated on a 5 point
represent most of the current real industrial applications. Scenario 2 scale (1 = not at all; 5 = a lot).
would have to represent an intermediate case. Finally, we expected that Perceived stress. Stress was assessed using the Short Stress State
Scenario 3 simulated the best interaction conditions. Questionnaire (SSSQ) (Helton et al., 2005). Participants were asked to
think about how they felt during the execution of the task and rate 5
semantic-differential items (e.g., “At ease/Discomfort”, “Irritated/Calm)
2.5. Measures on a 5 point scale. A Cronbach’s alpha reliability coefficient of 0.67
indicated poor reliability of the scale in the current sample.
A survey was administered to participants to assess how manipu­ Trust. Participants’ trust towards the collaborative robot was
lating features in each scenario would affect their overall experience and assessed using a slightly re-adapted version of the Trust in Industrial
opinions about the collaboration in different experiment phases. Par­ Human-Robot Collaboration Scale (Charalambous et al., 2016b). The
ticipants were asked to complete the survey before starting the experi­ scale consisted of nine items (e.g., “The speed at which the gripper
ment and in between each scenario. In particular, the same questions picked up and released the components made me uneasy”; “I trusted that
were repeated after the conclusion of each scenario. The survey was the robot was safe to cooperate with”; “The gripper seemed like it could
designed to provide results according to the cognitive risk factors pre­ be trusted”) rated on a five-point Likert scale. Cronbach’s alpha reli­
viously identified. The feedback allowed the authors to gain knowledge ability coefficient was 0.73.
directly from the participants’ experiences and perceptions under Usability. Perceived usability has been measured using five items
different experimental conditions. The survey assessed included the taken from the System Usability Scale (Lewis and Sauro, 2017). Items
following sections. were slightly re-adapted to address robotic systems. All items were rated
Acceptance. Users’ acceptance towards the robot was assessed using on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree).
the System Acceptance Scale (Van Der Laan et al., 1997). It includes nine Participants were asked to express their level of agreement with the
semantic differential items representing different attitudes toward the

7
 L. Gualtieri et al.
Table 3
Guidelines implementation (solutions) in the experimental case study.
Code Guideline Solution for the implementation

WE.CE.1 Provide measures to transfer the graphic user interface onto the collaborative workspace; Multiple highlighting-yellow components have been designed and realized through 3D printing techniques. The
aim was to highlight the CAS components, which are crucial for the assembly as well as for the HRI (booth for fixes
and moving parts of the workstation).
RF.CE.1 Realize a fluent and smooth robotic system design (avoid bulky joints, wires, external arm This characteristic was intrinsically present in the design of the robotic system and related devices used in the
components, mechanized shape); experiment.
RF.CE.4 Design robotic system and related devices with industrial appearance (avoid adding human-like The workstation has been realized with industrial components. The environment in which the experiment has been
features, e.g. anthropomorphism, and social appearance for robotic systems to be used in industrial conducted was similar to a manufacturing environment.
contexts);
RF.CE.6 Demonstrate the operators about the effectiveness and reliability of safety measures of the robotic An introductory safety training has been developed. It aimed to explain to the participants the main safety measures
system; of the robotic system through a live demonstration.
RP.CE.1 Design human-like-inspired/smooth/fluent and non-disruptive robotic system actions; A minimum-jerk trajectory (Rojas et al., 2019) has been integrated into the motion of the robotic system. This
trajectory is mainly used for its similarity to human-joint movements and for the possibility to limit robot
vibrations.
RP.CE.2 Provide measures for the implementation of a medium-level robotic system autonomy; This has been implemented through:
- different levels of autonomy in the robot tasks (reached by changing the programming of the robotic system by
considering a level 3 on the LORA scale (Gervasi et al., 2020));
- different control commands that the operator can give to the robotic system.
RP.CE.3 Provide measures for the manual adjustment of robot arm speed according to operator’s needs; This has been implemented by using a button array.
By pressing one of the available buttons, the speed changed accordingly. It was possible to choose: low speed (75%
of the programmed nominal valuea), nominal speed (100% of the programmed nominal valuea) and, high speed
(125% of the programmed nominal valuea);
RP.CE.6 Design slow robotic system actions and related motions (related to the kind of collaborative task); This has been implemented by changing the robot speed (through programming) for each scenario. The program
can be set to: high speed (scenario 1–125% of the programmed nominal valuea); low speed (scenario 2–75% of the
8

nominal valuea) and, nominal speed (scenario 3 - nominal valuea).

RP.CE.7 Avoid unreliable/inaccurate performance of the robotic system; The assembly cycle has been previously tested to avoid possible errors related to the robotic tasks. The robotic
system and related devices were state of the art and reliable.
RP.CE.8 Inform the operator about the robotic system speed; This has been implemented by using the GUI developed by the research team, which also included the possibility of
displaying popup notifications containing information about the status of the robotic system. In particular, the
information graphically explained the selected speed of the tool center point of the robotic system (see RP.CE.3):
slow, nominal, high.
RP.CE.9 Inform the operator about the robotic system behavior/state; This has been implemented by using the GUI, which also included the possibility of displaying popup notifications
containing information about the status of the robotic system: on operation, on position, stopped due to a command,
stopped due to collision avoidance. In particular, the information explained the robotic system’s status according to
the command given by the operator (see OM.CE.3) and according to the interaction conditions.
RP.CE.10 Avoid variations in robot arm velocity; This has been implemented by programming the robot’s motion in such a way as to keep a relatively constant speed
and acceleration of the arm. In case of min. jerk trajectories, this kind of condition was more complicated to be
satisfied.
OM.CE.2 Provide information about the workstation systems (including the robotic system) only when relevant This has been implemented by using the GUI, which also included the possibility of displaying popup notifications
and necessary; containing information about the status of the robotic system. The GUI was designed to avoid overloading the
operator with useless or not meaningful information.
OM.CE.3 Provide measures that allow the operators to control the workstation systems (including the robotic This has been implemented by:

Applied Ergonomics 104 (2022) 103807

system); - allowing the participants to arrest and resume the motion of the robotic system by using the gesture recognition
functionalities provided by the AI-based 3D perception device and vision system;
- allowing the participants to control the robot speed (see RP.CE.3).
OM.CE.4 Inform operators about the type and functioning of the specific safety measures implemented in the This has been implemented by training the participants about the safety measures that operated during the
workstation; upcoming scenario. In addition, a notification about possible collision avoidance was provided by the GUI.
OM.CE.8 Demonstrate the operator about the efficiency and reliability of the robotic system role; This has been implemented by training the participants about the roles and responsibilities of the robotic system
and related devices.
(continued on next page)
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

following statements: (1) “I think I would like to use the robot

the nominal value of the robot speed was set according to a preliminary mechanical risk assessment by following the ISO TS 15066 (International Organization for Standardization, 2016) requirements. This evaluation
This has been implemented by using the GUI, which was designed to avoid overloading the operator with useless or
frequently”; (2) “I found the robot’s behavior to be mostly predictable”;

This has been implemented by training the participants about the assembly cycle (process) and related products.
This has been implemented by properly deign the interfaces that allowed the HRI and communication (buttons,
(3) “I found the various functions in the robot were well-integrated”; (4)
“I found the robot to work appropriately.“; (5) “I found that the robot
could be operated and managed intuitively”. Cronbach’s alpha reli­
ability coefficient was 0.76.
To further integrate the data obtained from the survey, different
collection methods for qualitative data have been used. These are
GUI, gesture recognition based on the AI-based 3D perception device and vision system).

following described.

• Direct observation during the execution of the experiment: the

behavior of the participants was directly observed by the testers
during the experiments. The aim was to collect as much feedback as
possible by noting particular events or situations (errors, near misses,
and participant’s requests);
• Video recording: all the experiments were recorded by using a
camera system. The recordings were used after the conclusion of the
experiments to perform further detailed observations;
• Semi-structured interview: some oral and informal discussions be­
tween the participant and the testers were conducted at the end of
each experiment. The aim was to collect further information not
expressed by the participants during the questionnaire’s fulfillment
(e.g., particular observations made by participants that they want to
Solution for the implementation

share).
not meaningful information.

3. Results

Table 5 and Fig. 3 summarize the main outputs obtained from the
survey. It is relevant to mention that each scenario’s duration was
mainly dependent on the participants’ ability to deal with the specific
assembly situation (i.e., ability to use the available tools, number of
assembly errors, reasoning time according to various events, etc.). On
average, the duration of each scenario was the following:
Provide training and empowerment to the operator (understand the abilities, the process complexity,
Provide as much as possible natural and intuitive communication between the operator and the

• Scenario 1 lasted 228.3 s for each participant (with a standard de­

viation of 17.5 s);
• Scenario 2 lasted 221.2 s for each participant (with a standard de­
viation of 15.7 s);
• Scenario 3 lasted 213.0 s for each participant (with a standard de­
viation of 18.6 s).
the limitations of the robotic teammate and the reasons behind the events);

To assess differences in duration and participants’ cognitive risk

factors scores in each scenario, a multiple repeated measure ANOVA
analysis using SPSS v23 was performed.
Scenario duration was significantly affected by the changes in
workstation features F(2,26) = 14.62, p < .001. Bonferroni post-hoc
has resulted in conservatively estimated speed values of the robot.

tests indicated that the mean scores of duration in Scenario 2 and Sce­
nario 3 were significantly lower than in Scenario 1 (p < .005; p < .005).
Minimize the number of feedback interfaces;

Acceptance was significantly affected by each scenario’s different

features, F(2,26) = 8.20, p < .005. Bonferroni post-hoc tests indicated
that the mean scores on acceptance in Scenario 3 were significantly
higher than in Scenario 1 (p = .05; p = .005).
The perceived cognitive workload was significantly affected by each
scenario’s different features, F(2,26) = 5.02, p = .01. Bonferroni post-
hoc tests indicated that the mean score on perceived cognitive work­
load in Scenario 3 was significantly lower than in Scenario 2 (p = .01).
robotic system;

Perceived enjoyment was significantly affected by each scenario’s

different features F(2,26) = 5.20, p = .01. Bonferroni post-hoc tests
Guideline

showed a tendency to significance for mean score of perceived enjoy­

ment. Perceived enjoyment in Scenario 2 and 3 was higher than in
Table 3 (continued )

Scenario 1 (p = .08).
Participants’ reported stress level was significantly affected by each
scenario’s different features F(2,26) = 13.94, p < .001. Bonferroni post-
OM.CE.10

OM.CE.11

OM.CE.13

hoc tests indicated that the mean stress score in Scenario 2 and Scenario
Code

3 was significantly lower than in Scenario 1 (p < .05; p < .01).

Trust was significantly affected by each Scenario’s different features
a

9
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

Table 4
Application of the guidelines according to different scenarios.
Applied Scenario 1 Scenario 2 Scenario 3 Operator’s
guideline dependencya
The expected worst-case from the The expected intermediate case from the The expected best case from the cognitive
(code)
cognitive point of view cognitive point of view point of view

WE.CE.1 NO (No CAS components were YES (CAS components which are crucial for YES (CAS components which are crucial for NO
highlighted) the assembly as well as for the HRI were the assembly as well as for the HRI were
highlighted) highlighted)
RF.CE.1 YES (intrinsic design of the robotic system YES (intrinsic design of the robotic system YES (intrinsic design of the robotic system and NO
and related devices) and related devices) related devices)
RF.CE.4 YES YES YES NO
RF.CE.6 NO YES YES NO
RP.CE.1 NO NO YES NO
RP.CE.2 NO (the robot autonomy was the lowest; YES (partially: the robot autonomy was the YES (totally: the robot autonomy was YES
there was no possibility to command the highest; there was the possibility to control intermediate; there was the possibility to
robotic system) the robotic system) control the robotic system)
RP.CE.3 NO NO YES YES
RP.CE.6 NO (fast situation: robot speed equal to YES (slow situation: robot speed equal to 75% YES (robot speed equal to the nominal value, NO
125% of the nominal value) of the nominal value) which means a safe speed according to the
application)
RP.CE.7 YES YES YES NO
RP.CE.8 NO NO YES NO
RP.CE.9 NO YES YES NO
RP.CE.10 YES YES YES (min. jerk) NO
OM.CE.2 YES YES YES NO
OM.CE.3 NO YES (partially: participants could control the YES (totally: participants could control the YES
motion of the robotic system) motion of the robotic system)
OM.CE.4 NO YES (partially: provided by training) YES (totally: provided by training and by GUI NO
notifications)
OM.CE.8 YES YES YES NO
OM.CE.10 YES (partially: the robotic system’s YES (partially: the robotic system’s command YES (totally: the participants could choose the YES
command was given by a physical touch was given by using a virtual button) way of interaction they prefer (touch or
on the end-effector) virtual button))
OM.CE.11 YES YES YES NO
OM.CE.13 YES (the GUI provided only the YES (the GUI provided the instructions and YES (the GUI provided the instructions and NO
instructions) the notifications about the robotic system the notifications about the robotic system
status) status and speed)
a
A value equal to “YES” means that the operator has the chance to decide whether or not to activate the technical solution implemented by following the associated
guideline. In other words, such solutions are operator’s dependent.

F(2,26) = 3.43, p < .05. Bonferroni post-hoc tests showed no significant • Near misses related to abnormal behaviors concerning the assembly
difference between estimated marginal means in each scenario. cycle (i.e., anticipating the grip of a component passed by the robotic
The perceived usability was also significantly affected by each sce­ system before the competition of its task);
nario’s different features, F(2,26) = 4.24, p = .03. Bonferroni post-hoc • Requests made from the participants to the tester to understand/
tests indicated that the mean usability score in Scenario 3 was signifi­ clarify certain situations (i.e., request clarification on the robotic
cantly higher than in Scenario 1 (p < .05). system’s status).
Analysis of reported frustration scores showed no significant results.
Table 6 displays the estimated marginal means of each considered var­ 4. Discussion
iable in the three scenarios.
Table 7 and Fig. 4 summarises the main results provided by the video In general, results provide preliminary support for implementing the
analysis (qualitative results). These were obtained by carefully watching identified guidelines and are in accordance with previous literature. In
the records to find possible critical events (for all the participants) particular, according to the survey and interviews, results showed that:
during the experiment (for all the scenarios). The identified events were
then discussed by a team of experts and classified as follow: 1. Participants’ acceptance, perceived enjoyment, and usability signif­
icantly improved as the identified guidelines were implemented in
• Errors related to the assembly cycle (i.e., wrong sequence of tasks); the subsequent scenarios. Analysis of contrasts showed that a

Table 5
Mean and Standard Deviation (SD) of cognitive risk factors according to different scenarios.
Risk Factor Scenario 1 Scenario 2 Scenario 3 Mean values Mean values
difference between difference between
Mean SD Mean SD Mean SD
Scenario 1 and Scenario 2 and
Scenario 2 (%) Scenario 3 (%)

Trust 4.29 0.76 4.56 0.72 4.79 0.35 +6% +5%

Usability 4.09 0.83 4.36 0.77 4.33 0.75 +6% − 1%
Frustration 1.64 1.15 1.36 0.84 1.21 0.58 − 17% − 11%
Perceived Enjoyment 3.93 1.27 4.36 0.84 4.36 0.84 +10% 0%
Acceptance 3.80 0.98 4.18 0.55 4.18 0.70 +9% 0%
Stress 1.80 0.71 1.40 0.71 1.31 0.35 -22% − 7%
Cognitive Workload 0.93 1.14 1.07 1.21 0.57 1.09 +13% − 47%

10
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

Fig. 3. The trend of cognitive risk factors according to different scenarios.

arising in the interaction with the robotic system and in doing the
Table 6
task, contributing to increasing knowledge on the matter. Similar
Estimated marginal means of the dependent variables in each scenario.
studies do, in fact, show inconsistent results (Müller et al., 2017b;
D.V Scenario 1 Scenario 2 Scenario 3 Richert et al., 2016a; Richert et al., 2016b).
Scenario duration 246.93 221.21 213 4. Frustration levels appeared to lower with the enhancement of
Acceptance 3.80 4.18 4.18 workstation features and interaction conditions by shifting from the
Cognitive Workload 0.93 1.07 0.57
various scenarios (− 17% in the first transition and − 11% in the
Perceived enjoyment 3.92 4.36 4.36
Stress 1.80 1.40 1.31 second one). However, it was not possible to highlight a statistically
Trust 4.29 4.56 4.67 significant relationship. Also, trust appears to improve and be
Usability 4.09 4.36 4.33 significantly affected by each scenario’s features, but the analysis of
Frustration 1.64 1.36 1.21 contrasts did not show significant differences.

According to the video analysis, results showed that the overall

Table 7 number of critical events decreased with a complete implementation of
Critical events for each scenario. the guidelines. In particular, they reduced respectively by a value equal
Scenario TOT Errors TOT Near Miss TOT Requests TOT Overall to 25, 21, and 13 in the first transition and by a value equal to 5, 0, and 4
in the second one.
Scenario 1 48 40 21 109
Scenario 2 23 19 8 50 Results show support to the assumption that it is possible to improve
Scenario 3 18 19 4 41 participants’ experience of HRI with the enhancement of workstation
features and interaction conditions, suggesting that the proposed
guidelines can improve cognitive variables and reduce the related risk
significant improvement could be obtained even with a partial factors for operators collaborating with robotic systems. Furthermore,
implementation of the guidelines in Scenario 2, while it was not according to the trend of the critical events, the assembly performance
possible to show a consistent improvement moving from Scenario 2 improved with a complete implementation of the guidelines. As ex­
to Scenario 3. These results are essentially in line with previous pected, the best condition was related to Scenario 3, which was the
literature [e.g., Charalambous et al., 2015; El Makrini et al., 2017; scenario with the highest number of implemented guidelines. Never­
Gopinath et al., 2017; Kadir et al., 2018; Rosen et al., 2018] and theless, according to the quantitative and qualitative data, the largest
indicate which are the cognitive risk factors that can see the most improvement came from the change between Scenario 1 and Scenario 2.
benefits from the implementation of the identified guidelines. Similarly to other human factors and ergonomics frameworks related
2. Perceived Cognitive workload increased slightly by shifting from to HRC, such as the one developed by Kadir (2020), the guidelines
Scenario 1 to Scenario 2 (+13%) but considerably decreased from presented and evaluated in the current study could benefit from the
Scenario 2 to Scenario 3 (− 47%). Analysis of contrast showed that integration with other human factors frameworks and techniques such
the decrease in levels of reported cognitive workload in Scenario 3 as Cognitive Work Analysis (CWT) (Vicente, 1999), Cognitive Tasks
was significant, while the slight increase in Scenario 2 compared to Analysis (CTA) (Crandall et al., 2006), and Hierarchical Task Analysis
Scenario 1 was not significant. Again, this result is in line with pre­ (HTA) (Hollnagel, 2003). As it was suggested by Kadir (2020), the
vious research (Tang et al., 2019; Kaufeld and Nickel, 2019; Rosen developed guidelines could be further improved and refined through
et al., 2018; Bitonneau et al., 2017; Johnson et al., 2016) and high­ consultations and involvement with experts in CTA, CWT and HTA.
lights that a more extensive implementation of the proposed guide­ Similarly, those methods could draw important suggestions in their
lines is needed to show reductions in participants’ cognitive applications by the guidelines here presented.
workload. According to the present results, we could argue that implementing
3. Reported stress levels decreased shifting from Scenario 1 to Scenario human-like trajectories and allowing the operator to set the pace of the
2, and from Scenario 1 to Scenario 3. This result suggests that robotic system as well as choosing the preferred mode of HRI (i.e.,
implementing guidelines can lead to a reduction in reported stress physical pressure vs. a virtual button) contributed to improve

11
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

Table 8
Guidelines that have not been implemented in the experiment.
Code Guideline Reason

WE.CE.2 Locate the robot arm as distant as possible from the operator’s position; According to the (collaborative) assembly cycle, the operator and the robotic
system have to work nearby. Therefore, the position of the robot arm cannot be
chosen arbitrarily due to functional requirements.
WE.CE.3 Design low-contrast workstation elements with respect to the robotic It was not possible to continuously and quickly modify the color of the workstation
system; elements according to the single scenario.
RF.CE.2 Design cold-white robot arm; It was not possible to continuously and quickly modify the color of the robot arm
according to the single scenario.
RF.CE.3 Design a low-contrast robot arm with respect to the workstation elements; As for RF.CE.2.
RF.CE.5 Design on-board devices (mounted on the external surface of the robotic To avoid information redundancy and according to OM.CE.13 (minimize the
system) for the visual communication of the status of the robotic system; number of feedback interfaces), this guideline was not implemented. The visual
status of the robotic system was communicated by using the GUI.
RF.CE.7 Demonstrate the operators about the efficiency and operational reliability To avoid influencing participants’ cognitive responses, this demonstration was not
of the robotic system (e.g. the end-effector); done. Instead, it was done concerning the safety measures of the robotic system
(RF.CE.6) in the second scenario.
RP.CE.4 Design comprehensible and predictable robotic system actions (avoid This guideline has been partially fulfilled by implementing RP.CE.1, RP.CE.6 and
supposedly arbitrary actions of the system); RP.CE.9, especially from the perspective of the robot motion.
RP.CE.5 Provide measures for the automatic adaptation of robot arm speed to The application of such a guideline will be very interesting. Nevertheless, its
correspond with an operator’s profile (i.e. expertise, skills, capabilities, implementation will be too complex and time-consuming for a laboratory case
preferences, trust level); study. In addition, due to the design of the experiment and the profile of the
involved participants, some of the features (e.g. expertise, skills, capabilities) can
be considered the same for all of them (at least by considering the group of
researchers/students and the group of technicians/administrative).
OM.CE.1 Suggest work breaks to improve performance and concentration Since the experiment is based on three sequential scenarios (each of which lasts
(suggestions could be based on age and the monitoring of psychophysical about 5 min), the application of such a guideline was not necessary.
parameters);
OM.CE.5 Provide functions of the workstation systems (including the robotic system) As for RP.CE.5.
that adapt to suit individual operator’s preferred working methods;
OM.CE.6 Provide workstation systems (including the robotic system) that adapt As for RP.CE.5.
safety strategy to suit operator’s preferences and conditions in the
surrounding area;
OM.CE.7 Engage operators in workstation and interaction design (layout, assembly Due to the structure of the experiment, participants did not have to know any
cycle, robotic system performance and motions); details of the system/cycle before their experience. Therefore, this guideline was
not implemented.
OM.CE.9 Inform the operator about upcoming HRI; The experiment referred to HRC. The operator and the robotic system were
supposed to collaborate continuously during the experiment. As a consequence,
there are no other forms of HRI in the case study.
OM.CE.12 Visualize alternative decisions to reduce biases in decision-making; To avoid making the participant’s experience to complex (it was the first time they
collaborated with a robotic system), this guideline has not been implemented.
OM.CE.14 Inform the operator about the collaboration mode change (e.g. from As for OM.CE.9.
automatic to collaborative);

participants experience and reducing related risk factors significantly. implemented, also discussing the main reasons behind these choices.
This is in line with previous literature (Rojas et al., 2019), which showed Finally, the last consideration concerns the relationship between
that the introduction of human-like trajectories in robot movement cognitive and physical ergonomics. As for other assistance systems in
patterns could increase the sense of predictability and familiarity in manufacturing requiring physical interaction between the operator and
participants, reducing levels of perceived stress. Further support can be a device, improper or erroneous use of collaborative robotic systems
found looking at results on perceived usability and cognitive workload could be counter-productive from the point of view of biomechanical
in Scenario 3. Higher predictability and a sense of familiarity mean that loads. As a consequence, the implementation of the proposed guidelines
humans would use less cognitive resources to interact with the robotic should always consider the potential (negative) effects on physical er­
system, thus lowering cognitive workload levels and leading to a lower gonomics. Nevertheless, referring to the experimental case study, the
rate of errors and near misses. Furthermore, the freedom of choosing the new actions that are introduced by implementing the guidelines (e.g. the
speed of the robot and the interaction channel were possibly instru­ rising of the shoulder related to the gesture recognition used for human-
mental features in reducing levels of stress and cognitive workload and robot communication) are neglectable from the perspective of biome­
increasing acceptance, perceived enjoyment, and usability. This could chanical loads.
find a possible explanation in the self-determination theory (Deci and
Ryan, 2000) which states that intrinsic motivation thrives on autonomy, 5. Conclusions
which comprises performing a task based on one’s own volition. Results
suggest that allowing the operator to adjust the system’s features (e.g., 5.1. Conclusions
robot speed) and choosing interaction channels (e.g., type of command)
could lead to increased perceived usability through increased motiva­ This work refers to human factors in industrial HRI. The study aimed
tion. The self-determination theory in HRI has been recently used in the to define and experimentally validate a set of design guidelines related
educational context (van Minkelen et al., 2020), and our study suggests to cognitive ergonomics in CASs. Results confirmed the hypothesis and
that it could have favorable implications in the industrial context as are primarily in accordance with previous literature. Therefore, the
well. proposed research questions have been addressed.
As mentioned in the description of the experiment and related sce­ In particular, multiple design principles (the guidelines) to be
narios, it was not possible to test all the guidelines in the present applied when designing or setting a CAS according to cognitive ergo­
experiment. Table 8 summarises the guidelines that have not been nomics have been developed as generic as possible by analyzing the

12
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

Fig. 4. The trend of critical events according to different scenarios.

scientific literature. A laboratory case study has been used for the them to familiarize themselves with the robotic system and thus influ­
evaluation of such guidelines. Multiple qualitative and quantitative data ence the results. This could affect scenario duration scores in particular.
have been acquired and analyzed. The operator’s (positive or negative) Furthermore, the experiment referred to a collaboration with a low-
response to the manipulation of the features and parameters associated payload collaborative robotic system for the common assembly of a
with these design principles has been tested by using different cognitive small workpiece. This is a specific condition that aims to reproduce a
variables and related risk factors (i.e., trust, usability, frustration, common, but not unique, industrial HRC. Results may change by
perceived enjoyment, acceptance, stress, and cognitive workload). implementing the proposed guidelines in a different CAS (i.e., using a
Finally, the impact of such guidelines on assembly performance has been medium-size collaborative robot or assembling heavy components).
evaluated by analyzing the trend of critical events (i.e., assembly errors, Finally, since it was not possible to test all the developed guidelines,
near misses, clarification requests). future studies should also integrate them to have a more comprehensive
As expected, operators’ experience and assembly performance evaluation. In addition, the mutual relationships between all the pro­
improved with the sequential implementation of the guidelines. Ac­ posed guidelines should be identified and quantified by considering
cording to the experimental outcomes, it is supposed that the following main cognitive variables. In fact, this work did not analyze the hierar­
measures have been appreciated by the participants: a better synchro­ chical relationships between the various guidelines, as well as possible
nization with the robot operations, a higher robot autonomy, greater inconsistencies in their implementation. This is a preliminary work and
control on the system, better awareness about workstation elements, the guidelines will evolve according to the results of further studies. At
better awareness about the robotic systems state. the moment, the choice of the most suitable solution is left to the
These achieved results can support companies (especially SMEs) in designer. Future works should focus more also on these aspects.
implementing human-centered CASs. The proposed guidelines will be According to these limitations, future studies should deepen the
helpful for technicians with expertise in manufacturing systems and/or study of the effectiveness of the proposed guidelines in a wider way. This
collaborative robotics but without knowledge about cognitive risk fac­ will require the use of a larger and more homogeneous focus group and
tors. Their use will help them in overcoming technological barriers and the development and evaluation of multiple case studies related to
in developing more comfortable, safe, and efficient CASs. These various conditions of HRC in assembly (i.e. by using different sizes of
achievements perfectly fit with the larger goals of Industry 4.0 in terms robots and testing different tasks). This will be fundamental to gain more
of social and economic sustainability. Finally, results can be utilized as knowledge about the operator’s wellbeing and assembly performance in
well-structured starting points for further developments of a more the design of CASs from the cognitive perspective.
extensive European-wide technical documentation regarding the psy­
chosocial requirements for industrial collaborative systems.
Declaration of competing interest
5.2. Limitations and future works
The authors declare that they have no known competing financial
Following, the main limitations of this work are presented. interests or personal relationships that could have appeared to influence
Firstly, the number of participants was limited (14 people) and not the work reported in this paper.
homogenous in terms of gender (12 males and 2 females). Even if all the
participants did not have previous experience with collaborative robots References
and had minimal experience in manufacturing operations, their back­
ground was quite different. They can be classified into two groups of the Adam, C., Aringer-Walch, C., Bengler, K., 2018. Digitalization in
manufacturing–employees, do you want to work there?. In: Congress of the
same size: people with industrial backgrounds and people with non-
International Ergonomics Association. Springer, Cham, pp. 267–275. August.
industrial backgrounds. This condition may influence the robustness Alarcon, G.M., Gibson, A.M., Jessup, S.A., Capiola, A., 2021. Exploring the differential
of the results. In addition, it is possible that not randomizing the sce­ effects of trust violations in human-human and human-robot interactions. Appl.
narios would have influenced the results as well. Even though partici­ Ergon. 93, 103350.
Bauer, W., Bender, M., Braun, M., Rally, P., Scholtz, O., 2016. Lightweight Robots in
pants had the opportunity to learn the assembly sequence of the cylinder Manual Assembly–Best to Start Simply. Frauenhofer-Institut für Arbeitswirtschaft
in the training workstation, the unfolding of the experiment would allow und Organisation IAO, Stuttgart.

13
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

Bitonneau, D., Moulières-Seban, T., Dumora, J., Ly, O., Thibault, J.F., Salotti, J.M., Gopinath, V., Ore, F., Johansen, K., 2017. Safe assembly cell layout through risk
Claverie, B., 2017. Design of an industrial human-robot system through participative assessment–an application with hand guided industrial robot. Proced. CIRP 63,
simulations—tank cleaning case study. In: 2017 IEEE/SICE International Symposium 430–435.
on System Integration (SII). IEEE, pp. 1–8. Gualtieri, L., Rojas, R., Carabin, G., Palomba, I., Rauch, E., Vidoni, R., Matt, D.T., 2018.
Bortot, D., Born, M., Bengler, K., 2013. Directly or on detours? How should industrial Advanced automation for SMEs in the I4. 0 revolution: Engineering education and
robots approximate humans?. In: 2013 8th ACM/IEEE International Conference on employees training in the smart mini factory laboratory. In: 2018 IEEE International
Human-Robot Interaction (HRI). IEEE, pp. 89–90. Conference on Industrial Engineering and Engineering Management (IEEM). IEEE,
Bröhl, C., Nelles, J., Brandl, C., Mertens, A., Schlick, C.M., 2016. TAM reloaded: a pp. 1111–1115.
technology acceptance model for human-robot cooperation in production systems. Gualtieri, L., Fraboni, F., De Marchi, M., Rauch, E., 2021b. Evaluation of variables of
In: International Conference on Human-Computer Interaction. Springer, Cham, cognitive ergonomics in industrial human-robot collaborative assembly systems. In:
pp. 97–103. Congress of the International Ergonomics Association. Springer, Cham, pp. 266–273.
Bragança, S., Costa, E., Castellucci, I., Arezes, P.M., 2019. A brief overview of the use of June.
collaborative robots in industry 4.0: human role and safety. In: Occupational and Gualtieri, L., Palomba, I., Wehrle, E.J., Vidoni, R., 2020a. Potential and challenges in
Environmental Safety and Health. Springer, Cham, pp. 641–650. SME manufacturing automation through safety and ergonomics in human-robot
Brecher, C., Müller, S., Kuz, S., Lohse, W., 2013. Towards anthropomorphic movements collaboration. In: Matt, Dominik T., Modrak, Vladimir, Zsifkovits, Helmut (Eds.),
for industrial robots. In: International Conference on Digital Human Modeling and Industry 4.0 for SMEs Challenges, Opportunities and Requirements. Palgrave
Applications in Health, Safety, Ergonomics and Risk Management. Springer, Berlin, Macmillan, Basingstoke, pp. 105–144.
Heidelberg, pp. 10–19. Gualtieri, L., Rauch, E., Vidoni, R., 2021a. Emerging research fields in safety and
Brun, E., Milczarek, M., 2007. European Agency for Safety and Health at Work. Expert ergonomics in industrial collaborative robotics: a systematic literature review.
Forecast on Emerging Psychosocial Risks Related to Occupational Safety and Health. Robot. Comput. Integrated Manuf. 67, 101998.
European Risk Observatory Report. Office for Official Publications of the European Gualtieri, L., Rauch, E., Vidoni, R., Matt, D.T., 2020b. Safety, ergonomics and efficiency
Communities, Luxembourg. in human-robot collaborative assembly: design guidelines and requirements. Proced.
Cascio, W.F., Montealegre, R., 2016. How technology is changing work and CIRP 91, 367–372.
organizations. Ann. Rev. Org. Psychol. Org. Behav. 3, 349–375. Gualtieri, L., Rojas, R.A., Garcia, M.A.R., Rauch, E., Vidoni, R., 2020c. Implementation of
Changizi, A., Dianatfar, M., Lanz, M., 2018. Comfort design in human robot cooperative a laboratory case study for intuitive collaboration between man and machine in SME
tasks. In: International Conference on Human Systems Engineering and Design: assembly. In: Matt, Dominik T., Modrak, Vladimir, Zsifkovits, Helmut (Eds.), Industry
Future Trends and Applications. Springer, Cham, pp. 521–526. 4.0 for SMEs Challenges, Opportunities and Requirements. Palgrave Macmillan,
Charalambous, G., Stout, M., 2016. Optimizing train axle inspection with the Basingstoke, pp. 105–144, 2020.
implementation of human-robot collaboration: a human factors perspective. In: 2016 Hart, S.G., 2006. NASA-task load index (NASA-TLX); 20 years later. In: Proceedings of
IEEE International Conference on Intelligent Rail Transportation (ICIRT). IEEE, the Human Factors and Ergonomics Society Annual Meeting, 50. Sage publications,
pp. 254–258. Sage CA: Los Angeles, CA, pp. 904–908. October 9.
Charalambous, G., Fletcher, S., Webb, P., 2015. Identifying the key organizational Hart, S.G., Staveland, L.E., 1988. Development of NASA-TLX (task load index): results of
human factors for introducing human-robot collaboration in industry: an exploratory empirical and theoretical research. Adv. Psychol. 52, 139–183. North-Holland.
study. Int. J. Adv. Manuf. Technol. 81 (9–12), 2143–2155. Helton, W.S., Fields, D., Thoreson, J.A., 2005. Assessing daily stress with the Short stress
Charalambous, G., Fletcher, S., Webb, P., 2016a. Development of a human factors state questionnaire (SSSQ): relationships with cognitive slips-failures. In:
roadmap for the successful implementation of industrial human-robot collaboration. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 49.
In: Advances in Ergonomics of Manufacturing: Managing the Enterprise of the SAGE Publications, Sage CA: Los Angeles, CA, pp. 886–890, 10.
Future. Springer, Cham, pp. 195–206. Hollnagel, E. (Ed.), 2003. Handbook of Cognitive Task Design. CRC Press.
Charalambous, G., Fletcher, S., Webb, P., 2016b. The development of a scale to evaluate International Federation of Robotics, 2020. IFR Publishes Collaborative Industrial Robot
trust in industrial human-robot collaboration. Int. J. Soc. Robot. 8 (2), 193–209. Definition and Estimates Supply. https://ifr.org/post/international-federation-of-ro
Charalambous, G., Fletcher, S.R., Webb, P., 2017. The development of a Human Factors botics-publishes-collaborative-industrial-rob. Visited on December 2020.
Readiness Level tool for implementing industrial human-robot collaboration. Int. J. International Organization for Standardization, 2010. Safety of Machinery — General
Adv. Manuf. Technol. 91 (5–8), 2465–2475. Principles for Design-Risk Assessment and Risk Reduction. ISO 12100:2010).
Crandall, B., Klein, G., Klein, G.A., Hoffman, R.R., Hoffman, R.R., 2006. Working Minds: https://www.iso.org/standard/51528.html.
A Practitioner’s Guide to Cognitive Task Analysis. Mit Press. International Organization for Standardization, 2016. ISO TS 15066 — Robots and
Czerniak, J.N., Brandl, C., Mertens, A., 2017. Designing human-machine interaction Robotic Devices — Collaborative Robots. ISO/TS 15066:2016). https://www.iso.
concepts for machine tool controls regarding ergonomic requirements. IFAC- org/standard/62996.html.
PapersOnLine 50 (1), 1378–1383. International Organization for Standardization, 2018. ISO 9241-11 — Ergonomics of
Davis, F.D., 1989. Perceived usefulness, perceived ease of use, and user acceptance of Human-System Interaction — Part 11: Usability: Definitions and Concepts, 9241-11.
information technology. MIS Q. 319–340. https://www.iso.org/standard/63500.html, 2018.
De Visser, E.J., Peeters, M.M., Jung, M.F., Kohn, S., Shaw, T.H., Pak, R., Neerincx, M.A., Johnson, T., Tang, G., Fletcher, S.R., Webb, P., 2016. Investigating the effects of signal
2020. Towards a theory of longitudinal trust calibration in human–robot teams. Int. light position on human workload and reaction time in human-robot collaboration
j. soc. robot. 12 (2), 459–478. tasks. In: Advances in Ergonomics of Manufacturing: Managing the Enterprise of the
Deci, E.L., Ryan, R.M., 2000. The” what” and” why” of goal pursuits: human needs and Future. Springer, Cham, pp. 207–215.
the self-determination of behavior. Psychol. Inq. 11 (4), 227–268. Kadir, B.A., 2020. Designing new Ways of Working in Industry 4.0 (Doctoral Thesis).
Dehais, F., Sisbot, E.A., Alami, R., Causse, M., 2011. Physiological and subjective Kadir, B.A., Broberg, O., 2021. Human-centered design of work systems in the transition
evaluation of a human–robot object hand-over task. Appl. Ergon. 42 (6), 785–791. to industry 4.0. Appl. Ergon. 92, 103334.
El Makrini, I., Merckaert, K., Lefeber, D., Vanderborght, B., 2017. Design of a Kadir, B.A., Broberg, O., Souza da Conceição, C., 2018. Designing human-robot
collaborative architecture for human-robot assembly tasks. In: 2017 IEEE/RSJ collaborations in industry 4.0: explorative case studies. In: DS 92: Proceedings of the
International Conference on Intelligent Robots and Systems (IROS). IEEE, DESIGN 2018 15th International Design Conference, pp. 601–610.
pp. 1624–1629. Kalakoski, V., Selinheimo, S., Valtonen, T., Turunen, J., Käpykangas, S., Ylisassi, H.,
Elprama, B.V.S.A., El Makrini, I., Jacobs, A., 2016. Acceptance of collaborative robots by et al., 2020. Effects of a cognitive ergonomics workplace intervention (CogErg) on
factory workers: a pilot study on the importance of social cues of anthropomorphic cognitive strain and wellbeing: a cluster-randomized controlled trial. A study
robots. In: International Symposium on Robot and Human Interactive protocol. BMC psychol. 8 (1), 1–16.
Communication. Kaufeld, M., Nickel, P., 2019. Level of robot autonomy and information aids in human-
Fiam. 2020. https://www.fiamgroup.com/en/products/. Visited on December 2020. robot interaction affect human mental workload–an investigation in virtual reality.
Fletcher, S.R., Johnson, T.L., Larreina, J., 2019. Putting people and robots together in In: International Conference on Human-Computer Interaction. Springer, Cham,
manufacturing: are we ready?. In: Robotics and Well-Being. Springer, Cham, pp. 278–291.
pp. 135–147. Kim, W., Kim, N., Lyons, J.B., Nam, C.S., 2020. Factors affecting trust in high-
Fletcher, S.R., Johnson, T., Adlon, T., Larreina, J., Casla, P., Parigot, L., et al., 2020. vulnerability human-robot interaction contexts: a structural equation modelling
Adaptive automation assembly: identifying system requirements for technical approach. Appl. Ergon. 85, 103056.
efficiency and worker satisfaction. Comput. Ind. Eng. 139, 105772. Kolbeinsson, A., Lagerstedt, E., Lindblom, J., 2019. Foundation for a classification of
Fraboni, F., Gualtieri, L., Millo, F., De Marchi, M., Pietrantoni, L., Rauch, E., 2021. collaboration levels for human-robot cooperation in manufacturing. Prod. Manufac.
Human-robot collaboration during assembly tasks: the cognitive effects of Res. 7 (1), 448–471.
collaborative assembly workstation features. In: Congress of the International Kong, F., 2019. Development of metric method and framework model of integrated
Ergonomics Association. Springer, Cham, pp. 242–249. complexity evaluations of production process for ergonomics workstations. Int. J.
Fu, R., Zhang, Y., 2018. Modeling design of six-freedom-degree collaboration robot. In: Prod. Res. 57 (8), 2429–2445.
International Conference on Human-Computer Interaction. Springer, Cham, Koppenborg, M., Nickel, P., Naber, B., Lungfiel, A., Huelke, M., 2017. Effects of
pp. 448–454. movement speed and predictability in human–robot collaboration. Human Fact.
Gervasi, R., Mastrogiacomo, L., Franceschini, F., 2020. A conceptual framework to Ergon. Manufact. Serv. Ind. 27 (4), 197–209.
evaluate human-robot collaboration. Int. J. Adv. Manuf. Technol. 108, 841–865. Kuz, S., Mayer, M.P., Müller, S., Schlick, C.M., 2013. Using anthropomorphism to
Gjeldum, N., Aljinovic, A., Crnjac Zizic, M., Mladineo, M., 2021. Collaborative robot task improve the human-machine interaction in industrial environments (Part I). In:
allocation on an assembly line using the decision support system. Int. J. Comput. International Conference on Digital Human Modeling and Applications in Health,
Integrated Manuf. 1–17. Safety, Ergonomics and Risk Management. Springer, Berlin, Heidelberg, pp. 76–85.
Lasota, P.A., Fong, T., Shah, J.A., 2017. A Survey of Methods for Safe Human-Robot
Interaction. Now Publishers.

14
L. Gualtieri et al. Applied Ergonomics 104 (2022) 103807

Lewis, J.J.R., Sauro, J., 2017. Revisiting the factor structure of the system usability scale. Richert, A., Müller, S., Schröder, S., Jeschke, S., 2018. Anthropomorphism in social
J. Use. Stud. 12 (4). robotics: empirical results on human–robot interaction in hybrid production
Müller, S.L., Schröder, S., Jeschke, S., Richert, A., 2017a. Design of a robotic workmate. workplaces. AI Soc. 33 (3), 413–424.
In: International Conference on Digital Human Modeling and Applications in Health, Robotiq. 2020. https://robotiq.com/. Visited on December 2020.
Safety, Ergonomics and Risk Management. Springer, Cham, pp. 447–456. Rojas, R.A., Garcia, M.A.R., Wehrle, E., Vidoni, R., 2019. A variational approach to
Müller, S.L., Stiehm, S., Jeschke, S., Richert, A., 2017b. Subjective stress in hybrid minimum-jerk trajectories for psychological safety in collaborative assembly
collaboration. In: International Conference on Social Robotics. Springer, Cham, stations. IEEE Rob. Autom. Lett. 4 (2), 823–829.
pp. 597–606. Rosen, P.H., Sommer, S., Wischniwski, S., 2018. Evaluation of human-robot interaction
Mayer, R.C., Davis, J.H., Schoorman, F.D., 1995. An integrative model of organizational quality: a toolkit for workplace design. In: Congress of the International Ergonomics
trust. Acad. Manag. Rev. 20 (3), 709–734. Association. Springer, Cham, pp. 1649–1662.
Mayer, M.P., Kuz, S., Schlick, C.M., 2013. Using anthropomorphism to improve the Schleicher, T., Bullinger, A.C., 2018. Assistive robots in highly flexible automotive
human-machine interaction in industrial environments (part II). In: International manufacturing processes. In: Congress of the International Ergonomics Association.
Conference on Digital Human Modeling and Applications in Health, Safety, Springer, Cham, pp. 203–215.
Ergonomics and Risk Management. Springer, Berlin, Heidelberg, pp. 93–100. Schmidtler, J., Sezgin, A., Illa, T., Bengler, K., 2015. Black or white? Influence of robot
Michalos, G., Makris, S., Tsarouchi, P., Guasch, T., Kontovrakis, D., Chryssolouris, G., arm contrast on distraction in human-robot interaction. In: International Conference
2015. Design considerations for safe human-robot collaborative workplaces. Proced. on Engineering Psychology and Cognitive Ergonomics. Springer, Cham,
CIRP 37, 248–253. pp. 188–199.
Moorthy, J.T.S., bin Ibrahim, S., Mahrin, M.N.R., 2014. Identifying usability risk: a Smart Robots. 2020. http://smartrobots.it/. Visited on December 2020.
survey study. In: 2014 8th. Malaysian Software Engineering Conference (MySEC), Tang, G., Webb, P., Thrower, J., 2019. The development and evaluation of Robot Light
pp. 148–153. IEEE.Spector, P. E. (1978). Organizational frustration: A model and Skin: a novel robot signalling system to improve communication in industrial
review of the literature. Personnel Psychology, 31(4), 815-829. human–robot collaboration. Robot. Comput. Integrated Manuf. 56, 85–94.
Nelles, J., Kwee-Meier, S.T., Mertens, A., 2018. Evaluation metrics regarding human Teo, G., Reinerman-Jones, L., Matthews, G., Szalma, J., Jentsch, F., Hancock, P., 2018.
well-being and system performance in human-robot interaction–A literature review. Enhancing the effectiveness of human-robot teaming with a closed-loop system.
In: Congress of the International Ergonomics Association. Springer, Cham, Appl. Ergon. 67, 91–103.
pp. 124–135. Thorvald, P., Lindblom, J., Andreasson, R., 2017. CLAM–A method for cognitive load
Pei, Y.C., Chen, J.L., Wong, A.M., Tseng, K.C., 2017. An evaluation of the design and assessment in manufacturing. Adv. Manufact. Technol. XXXI, 114–119.
usability of a novel robotic bilateral arm rehabilitation device for patients with Universal Robots. 2020. https://www.universal-robots.com/. Visited on December 2020.
stroke. Front. Neurorob. 11, 36. Van Der Laan, J.D., Heino, A., De Waard, D., 1997. A simple procedure for the
Petruck, H., Faber, M., Giese, H., Geibel, M., Mostert, S., Usai, M., et al., 2018. Human- assessment of acceptance of advanced transport telematics. Transport. Res. C Emerg.
robot collaboration in manual assembly–A collaborative workplace. In: Congress of Technol. 5 (1), 1–10.
the International Ergonomics Association. Springer, Cham, pp. 21–28. van Minkelen, P., Gruson, C., van Hees, P., Willems, M., de Wit, J., Aarts, R., et al., 2020.
Rüßmann, M., Lorenz, M., Gerbert, P., Waldner, M., Justus, J., Engel, P., Harnisch, M., Using self-determination theory in social robots to increase motivation in L2 word
2015. Industry 4.0: the future of productivity and growth in manufacturing learning. In: Proceedings of the 2020 ACM/IEEE International Conference on
industries. Boston Consult. Grp. 9 (1), 54–89. Human-Robot Interaction, pp. 369–377. March.
Recipe for Stress, 2020. http://humanstress.ca/stress/understand-your-stress/sources- Vicente, K.J., 1999. Cognitive Work Analysis: toward Safe, Productive, and Healthy
of-stress. Visited on December 2020. Computer-Based Work. CRC press.
Richert, A., Shehadeh, M.A., Müller, S.L., Schröder, S., Jeschke, S., 2016a. Socialising Weistroffer, V., Paljic, A., Callebert, L., Fuchs, P., 2013. A methodology to assess the
with robots: human-robot interactions within a virtual environment. In: 2016 IEEE acceptability of human-robot collaboration using virtual reality. In: Proceedings of
Workshop on Advanced Robotics and its Social Impacts (ARSO). IEEE, pp. 49–54. the 19th ACM Symposium on Virtual Reality Software and Technology, pp. 39–48.
Richert, A., Shehadeh, M., Müller, S., Schröder, S., Jeschke, S., 2016b. Robotic What Is an European Standard?, 2020. https://www.cencenelec.eu/standards/DefEN/.
workmates: hybrid human-robot-teams in the industry 4.0. In: International Visited on July 2021.
Conference on E-Learning. Academic Conferences International Limited, p. 127. Zanchettin, A.M., Bascetta, L., Rocco, P., 2013. Acceptability of robotic manipulators in
shared working environments through human-like redundancy resolution. Appl.
Ergon. 44 (6), 982–989.

15