ORIGINAL RESEARCH

published: 17 December 2020

doi: 10.3389/fmech.2020.578379

Intelligent Predictive Maintenance and

Remote Monitoring Framework for
Industrial Equipment Based on
Mixed Reality
Dimitris Mourtzis *, John Angelopoulos and Nikos Panopoulos
Laboratory for Manufacturing Systems and Automation, Department of Mechanical Engineering and Aeronautics, University of
Patras, Patras, Greece

The currently applied maintenance strategies, including Reactive and Preventive

maintenance can be considered obsolete. The constant improvements in Information
and Communication Technologies as well as in Digital Technologies along with the
increase of computational power, have facilitated the development of new Artificial
Intelligence algorithms to integrate cognition in computational systems. This trend is
posing a great challenge for engineers, as such developments will enable the creation
of robust systems that can monitor the current status of the machines and by extension to
predict unforeseeable situations. Furthermore, Smart Computers will be capable of
Edited by:
Dimitris Kiritsis, examining all possible scenarios and suggest viable solutions in a fraction of time
École Polytechnique Fédérale de compared to humans. Therefore, in this paper, the modelling, design and development
Lausanne, Switzerland
of a Predictive Maintenance and Remote Monitoring system are proposed, based on the
Reviewed by:
Somnath Chattopadhyaya,
utilization of Artificial Intelligence algorithms for data acquisition, fusion, and post-
Indian Institute of Technology processing. In addition to that, the proposed framework will integrate a Mixed Reality
Dhanbad, India application for the intuitive visualization of the data, that will ultimately facilitate production
Marco Macchi,
Politecnico di Milano, Italy and maintenance engineers to monitor the condition of the machines, and most
*Correspondence: importantly to get an accurate prediction of the oncoming failures.
Dimitris Mourtzis
mourtzis@lms.mech.upatras.gr Keywords: artificial intelligence, predictive maintenance, remote monitoring, augmented reality, machine learning

Specialty section:
This article was submitted to INTRODUCTION
Digital Manufacturing,
a section of the journal Maintenance of industrial equipment as a part of the manufacturing lifecycle, approaches 60–70% of
Frontiers in Mechanical Engineering the overall cost of production. Therefore, being able to predict and perform machine maintenance
Received: 30 June 2020 operations in a short period of time can lead to successful troubleshooting, and at the same time
Accepted: 30 October 2020 increase the availability of machine tools (Mourtzis et al., 2015). Currently, inadequate maintenance
Published: 17 December 2020
techniques can reduce the total productive ability of the plant by between 5 and 20% (Wollenhaupt,
Citation: 2016). Traditionally, maintenance professionals have combined several techniques, both quantitative
Mourtzis D, Angelopoulos J and and qualitative, with the aim to anticipate potential problems and alleviating downtime in their
Panopoulos N (2020) Intelligent
production plants. Predictive maintenance gives them the potential to optimize maintenance tasks in
Predictive Maintenance and Remote
Monitoring Framework for Industrial
real time, maximizing the useful life of their equipment while avoiding disruption of operation.
Equipment Based on Mixed Reality. Recent studies also show that unplanned downtime is costly, with an estimation of $50 billion per
Front. Mech. Eng. 6:578379. year for global producers (Deloitte, 2017a). In the Industry 4.0 environment, maintenance should do
doi: 10.3389/fmech.2020.578379 much more than simply prevent the downtime of individual assets. Predictive maintenance increases

Frontiers in Mechanical Engineering | www.frontiersin.org 1 December 2020 | Volume 6 | Article 578379

Mourtzis et al. Intelligent Predictive Maintenance Framework

uptime by 10–20%, while reducing overall maintenance costs by State of the Art the most pertinent literature is reviewed, and
5–10% and maintenance planning time by 20–50%. Furthermore, commercial devices are compared. In Proposed System
due to increased interconnectivity and new opportunities for Architecture, the proposed system architecture is presented. In
collecting, processing, and analyzing information, predictive System Implementation the practical implementation steps are
maintenance can be a very powerful strategy (Deloitte, 2017b). discussed. Then in Case Study, an experimental case study that
In addition to failure prediction, a significant challenge is the has derived from Industry is presented and the results are discussed.
implementation of reliable and error-free maintenance operations Finally, in Concluding Remarks and Outlook, the paper is
and, as a result, the constant validation of fully working equipment concluded, and future research points are discussed by the authors.
as soon as possible. To that end, a significant amount of
development work has been made to design and improve real-
time technical service systems and software focused on mobile apps STATE OF THE ART
to prevent unwanted errors and malfunctions (Masoni et al., 2017).
Moreover, the handling of complex cases of smart factories, Machine Learning
intelligent maintenance, self-organized adaptive logistics, Among the latest trends in the modern manufacturing world, is
customer-integrated engineering and smart factory architectures the so-called AI. An also well-known subset of the above-
require the integration of production data into modeling that can mentioned concept is Machine Learning (ML). Concretely ML
only be achieved with the use of advanced simulation and algorithms are defined as computer-based algorithms that
Information Technology (IT) (Mourtzis, 2020). Moreover, the improve their efficiency through experience, i.e. data
value of products will eventually be focused on their software processing (Mitchell, 1995). Globally the AI adoption is
parts not on their specification or implementation functions under surging at enormous rates, as it becomes apparent in the
the Product Service Systems (PSS) paradigm (Mourtzis et al., 2018). report presented in (Hupfer, 2019), from where it can be
With the advancement of Information and Communication concluded that AI adoption marked a surprising 270%
Technology (ICT) and cutting-edge technologies such as Mixed- increase in a timespan of 4 years along with an increase in
Reality (MR), Augmented Reality (AR) and Virtual Reality (VR), global spending of around 80 billion dollars Figure 1.
the academic domain is expanding this strategy by leveraging the Additionally, Deep Learning (DL) techniques have been
advantages of AR for data visualization during maintenance applied for the integration of systems in edge computing,
operations (Mourtzis et al., 2017; Mourtzis et al., 2018; setting edge nodes in edge services and terminal devices, using
Palmarini et al., 2018). Emerging technologies such as the DL architectures for predictive analysis with quick preprocessing
Internet of Things (ΙoΤ), cyber-physical networks and cloud and accurate performance classification to assess the life
computing have enabled the processing of vast volumes of expectancy of components. A classification of the most
tracking data, which is intended to significantly increase common DL frameworks is as follows:
manufacturing productivity (Tao et al., 2018; Fantini et al.,
2020). However, as a core topic in prognostics and health • Neural Networks (NN) (Chryssolouris, 2006; Chen et al.,
management, the remaining useful life (RUL) prediction based 2019)
on monitoring data ca be used to prevent a failure triggered (Lei • Deep Neural Networks (DNN) (Zhao et al., 2017)
et al., 2018). RUL prediction is thus a hot topic that has drawn more • Convolutional Neural Networks (CNN) (Li et al., 2018;
and more interest in recent years (Yang et al., 2019). Most research Mourtzis et al., 2020a)
studies on intelligent prognosis and health management (PHM) • Recurrent Neural Networks (RNN) and Long Short-Term
analysis using data-driven approaches by deducing correlations Memory (LSTM) (Zhao et al., 2017)
between data from different sensors ( e.g. accelerometers, acoustic • Gated Recurrent Units (GRU) (Chen et al., 2019)
energy emission) to determine the remaining useful life (e.g. • Recurrent Neural Network (CNN-RNN) (Banerjee et al.,
accelerometers, acoustic energy sounders, etc.). In order to limit 2019)
the complexity inherent in the dynamic updating of online data, • Convolutional Neural Network and Long Short-Term
Machine Learning has arisen as a way of analyzing vast volumes of Memory (CNN-LSTM) (Kong et al., 2019)
data for statistical purposes. Especially in the implementation of • Convolutional Neural Network and Gated Recurrent Unit
neural network-based techniques, complex multidimensional non- (CNN-GRU) (Lei et al., 2018)
linearities can be used for automated learning, allowing for efficient
processing of data features in an attempt to provide optimized As Industry 4.0 continues to evolve, many companies are
solutions (Vogl et al., 2019). struggling with the realities of AI implementation. Indeed, the
Having identified the above-mentioned challenges, this research benefits of PdM such as helping determine the condition of
work presents the design and development of a predictive equipment and predicting when maintenance should be
maintenance framework for industrial equipment. Further to performed, are extremely strategic. The implementation of ML
that the contribution of this research work extends to the frameworks can lead to major cost savings, higher predictability,
presentation of a custom Data Acquisition (DAQ) device and a and increased availability of the systems. Therefore engineers
framework for processing the data via the Digital Twin of the have focused their efforts on the development of new technologies
equipment for the calculation of Remaining Useful Life of critical and techniques for facilitating the prediction of manufacturing
components. The remainder of the paper is structured as follows. In equipment malfunctions and therefore to further optimize

Frontiers in Mechanical Engineering | www.frontiersin.org 2 December 2020 | Volume 6 | Article 578379

Mourtzis et al. Intelligent Predictive Maintenance Framework

FIGURE 1 | Machine Learning Types & algorithms (Adapted from Ahmed and Khan, 2019).

existing maintenance policies as well as to introduce more (2020), the authors have developed a framework based on
adaptive maintenance policies. PdM can be defined as a series Convolution Auto Encoder and Long-Short Term Memory
of processes, where data is collected over time in order to monitor (LSTM) in an attempt to estimate RUL more accurately in
the state of equipment, in a manufacturing system. Ultimately, comparison to conventional methods. For the recognition of
the goal is to identify/recognize patterns that in turn will facilitate patterns, which facilitates the process of building the
engineers to predict and ultimately prevent failures predictive model, data exploration techniques must be utilized
(Rezaeianjouybari and Shang, 2020). Some of the most so that the engineer can determine whether the data includes
common problems that can be addressed with PdM include, degradation or failure patterns (Erfani et al., 2016; Li et al., 2019).
the calculation of Remaining Useful Life (RUL), which aims at the
suitable scheduling of future Maintenance and Repair Operations Remaining Useful Life
(MRO), Flagging Irregular Behavior, which is based on anomaly As the name indicates, Remaining Useful Life, also referred to as
detection by the utilization of time series analysis, and Failure RUL, describes a wide variety of algorithms which aim to predict
Diagnosis and Recommendation of Mitigation after failure (Lei the remaining life of assets and/or their components, ultimately
et al., 2018; Mourtzis et al., 2020a). While certain Facility developed under a predictive maintenance framework. According
Managers perform PdM, this has been done traditionally by to Baru (2018) there can be identified three basic techniques
using Supervisory Control and Data Acquisition (SCADA) regarding the calculation of RUL based on the data that are
systems set up with human-coded/hard-coded thresholds, alert available, namely lifetime data, run-to-failure data, and known
rules, and configurations. However, this is a semi-manual threshold data. An interesting approach in presented in (Loutas
approach that does not take into consideration the more et al., 2013) for the calculation of RUL for rolling bearings based
complex dynamic behavioral patterns of the machinery, or the on the utilization of ε-Support Vector Regression (ε-SVR),
contextual data relating to the manufacturing process, thus concluding that linear models cannot provide accurate results
lacking adaptability relative to the current status of the since there is non-linear between the features extracted by the
industrial equipment (Nicholson et al., 2012). What is more, data spectral analysis and the RUL prediction. Another aspect of
in recent research works SCADA systems are integrated with ML the usefulness of RUL estimation is presented by (Sun et al.,
algorithms, in order to extend their usability as well as to shift 2020). The authors have implemented a framework for the RUL
towards prognostics (Pang et al., 2020; Ruiming et al., 2020; calculation of cutting tools, thus managing to increase the
Zhang and Lang, 2020). In the research work of Wang et al. environmental sustainability of the cutting tools by 8.39% per

Frontiers in Mechanical Engineering | www.frontiersin.org 3 December 2020 | Volume 6 | Article 578379

Mourtzis et al. Intelligent Predictive Maintenance Framework

flute. From the investigation of the available literature it can be implementations of AR are based on manual solutions and
concluded that the estimation of RUL is a challenging topic, the use of Predictive Analytics/Prognostics has not been yet
requiring exhaustive data processing. Further to that in the faced, thus indicating that there is fertile ground for further
majority of the publications it is implied that linear research in that field.
approximations regarding the degradation of the physical
system are not sufficient in terms of accuracy, as presented in
(Yang et al., 2021), where the authors investigated the prediction PROPOSED SYSTEM ARCHITECTURE
of RUL in induction motors. The authors in (Wen et al., 2021) has
proposed a generalized methodology for the prediction of RUL In the following paragraphs the proposed system architecture will
based on the fusion of multiple signals. It is worth noting that they be discussed in detail. The key aspects of the proposed
achieved an increase in terms of accuracy of approximately 10 methodology are the DAQ device, which conforms to the
percent. Kozjek et al. (2020) have also presented an interesting latest IoT standards. However, in order to efficiently monitor
research work on the prediction of RUL with the utilization the status of professional refrigeration systems, they have to be
reinforcement learning, which is compared with two other analyzed into two subsystems, namely the cooling chamber of the
algorithms, indicating promising results. In the research work refrigerator and the compressor compartment. These two
of (Liu et al., 2019), a RUL prediction framework is proposed subsystems often are not located in the same room/building,
based on Health Index comparison, making it suitable for cases thus require different DAQ devices to be installed. By the virtue of
where there is limited amount historical data. It is stressed out the diversity of installed sensors, crosschecking the measured
that the topic of RUL prediction is still challenging for engineers values, is enabled and therefore more accurate predictive models
and by extension there is plenty of room for improvement. In can be trained. The general architecture of the proposed is
addition to that, the use of Digital Twin could compensate the depicted in Figure 2.
lack of raw data from machines, with the generation of fault
datasets. Data Acquisition Device
In this section the architecture of the framework for the DAQ
Extended Reality device will be discussed. For the DAQ module, two main aspects
Among the latest developments of the current industrial will have to be investigated, namely the DAQ device and the
revolution, advances in high-end digital technologies are communication interfaces as well. The development of the DAQ
entailed, including Extended Reality (XR). In its essence, XR device is based on the design of a custom circuit board in
is an umbrella term, often used by engineers and researchers combination with an Arduino micro-controller which
around the world, in order to describe technologies such as incorporates all the required modules for the data acquisition
Augmented Reality (AR), Mixed Reality (MR), and Virtual from the sensors attached to the board, the pre-processing of the
Reality (VR) (Mourtzis et al., 2020b). The two former data, an interface for user interaction and a wireless network
technologies are very close, since they are based on the module for the data transmission to the Cloud Database. In order
partial immersion of the user to a virtual environment, while to make the DAQ device adaptive to the customer needs, and
the latter, implies the total user immersion in a virtual, subsequently to support a wide variety of configurations, the
computer generated environment. In addition to that, what sensor modules are not hardwired/soldered on the main PCB of
differentiates AR from MR is the fact that MR is based on the DAQ device. More specifically, the PCB supports wired
the user interaction with the digital information, also known as connectivity, through 3.5 mm jack ports for the sensors.
holograms in that case (Fast-Berglund et al., 2018). The use of However, in order to enable the communication between the
AR in maintenance is an important aspect that has to be further DAQ device (Fig. 3A) and the Cloud Database an RF-based
researched under the Industry 4.0 framework. Since new Wireless Sensor Network (WSN) is utilized. For the setup of the
technologies are constantly becoming available, existing WSN, XBee modules are utilized. More specifically, an XBee
techniques could be leveraged so as to increase the efficiency module (Figure 3B) is installed on each of the DAQ devices and
of maintenance tasks, minimize the errors and the risks imposed another one is installed on a computer which acts as the network
in such operations. As presented in the research work of coordinator. For the correct communication of the DAQ devices
Vorraber et al. (2020), both maintenance technicians and to the computer, each RF module is tagged. Furthermore, during
experts are keen on integrating AR and MR solutions in their the data transmission, the data packets are also including the tag
line of job, in order to achieve better communication and most of each RF tag, so that the received data can be correlated to the
importantly to limit the complexity of the maintenance corresponding machine.
procedures. Although the maturity level of AR applications As far as the sampling rate is concerned, the DAQ device
has increased during the last decades (Mourtzis et al., 2020c), collects feedback from the installed sensors on a varying rate. The
there are constantly arising new challenges, such as the sampling rate for the accelerometer sensor is set to one (1) second
integration of Predictive Maintenance and AR/MR so that or 1 Hz. As far as the sampling rate for the temperature sensor is
digitalization of the manufacturing processes becomes a set to 5 s and for the pressure sensor is set to 10 ms (milli-second)
reality (Wolfartsberger et al., 2020). Further to that in two as soon as a surge event is detected. However, if the customers
recent systematic literature reviews, presented by Palmarini require a different resolution regarding the data collection and by
et al. (2018) and Egger and Masood (2020) the current extension the estimations made by the framework, then they

Frontiers in Mechanical Engineering | www.frontiersin.org 4 December 2020 | Volume 6 | Article 578379

Mourtzis et al. Intelligent Predictive Maintenance Framework

FIGURE 2 | Proposed System Architecture.

FIGURE 3 | Prototype DAQ device.

adjust the sampling rates. In Figure 3, the prototype board for the the condenser, and the refrigerator compartment. In the “Data
DAQ device is presented. Input” block, the data from the DAQ device are imported to the
model. Then the standard refrigeration cycle for refrigeration is
Digital Twin Development run and the results are plotted. Through the plots, crucial
In order to create a suitable framework for the predictive parameters of the refrigerator, such as temperature, power
maintenance functionality of the equipment, the design and consumption and pressures within the refrigerant distribution
development of a Digital Twin is required. The aim is to network can be observed. For the simulation, the fluid properties
analyze the data gathered from the DAQ devices and based on of the R134a refrigerant were also imported in the model. As a
the simulation model to predict future equipment malfunctions. result with the proposed methodology, it is possible to predict
Therefore, in the case of the refrigerators, the physical model is future asset malfunctions based on the simulation of the
created in the Simulink programming environment. For the refrigeration cycle and plan accordingly their production
simulation of the model, MATLAB is also utilized for schedule so that the equipment downtime is further
handling the imported data as well as setting up the minimized. In addition to that, the simulation results are also
simulation parameters. The physical model of the refrigerator combined/fused with the data gathered from the physical
is fully parametrically designed so that it can be adapted to the machine so as to predict the RUL of specific components of
technical specifications of the physical system. In Figure 4, the the equipment. The usability of the Simulink model extends also
developed model within the Simulink environment is presented. to the generation of fault data. For instance, in the refrigerators a
It is stressed out that the model consists of several subsystems, common failure is the loss of pressure in the refrigerant
or else functional blocks, in an attempt to increase the resolution distribution network. Therefore, in the existing model the
of the simulation model, such as the compressor, the evaporator, “fault” is simulated with the addition of an array of blocks,

Frontiers in Mechanical Engineering | www.frontiersin.org 5 December 2020 | Volume 6 | Article 578379

Mourtzis et al. Intelligent Predictive Maintenance Framework

FIGURE 4 | Refrigerator model in MATLAB Simulink.

based on which the differential pressure in specific subsystems, the simulation runs, the outputs were transformed via Fourier
such as the compressor pressure differential is offset to a fault Transformation, in order to represent the events in the
value. Then after the corresponding datasets for the healthy state frequency domain. Then, with the use of spectrograms,
and the fault state have been generated, the model automatically useful features were extracted and based on these features,
recalls the RUL algorithm. with the use of a Support Vector Machine, the faults could be
In its essence the RUL algorithm utilizes data from both the classified.
digital twin and the physical model for the prediction of the
time, in hours, before maintenance is required. The first step in Augmented Reality Module
the RUL algorithm is the Fourier transformation of the signals An AR module is provisioned in order to facilitate the monitoring
to the frequency domain. The next step is the creation of the process of the industrial equipment. This module can be realized
spectrograms for each of the under-examination parameters, as a multi-platform application, from which the customers can
e.g. pressure inlet and outlet in the compressor, vibration either remotely or on-site visualize crucial information about
signal from rotational components. In this step, two their equipment and interact with it, easily and intuitively with
spectrograms are created, one for the fault data and one for the use of this cutting-edge digital technology. Concretely, the
the healthy data. Based on the spectrograms of the faulty and current implementation of the framework supports handheld
the healthy datasets, features can be extracted and classified for Android-based platforms, e.g. mobile phones and tablets, as well
future use via the use of a Support Vector Machine (SVM). as Head Mounted Displays (HMD), such as the Microsoft
Therefore, boundary conditions can be formed for the under- HoloLens.
examination parameters. As soon as the above-mentioned User tracking and pose estimation for the Android-based
model is trained, then the model is constantly running and devices is based on the recognition of a feature-rich image
gets updated at a regular basis, given that there are new data target, as in the one presented in Figure 5. Further to that, in
posted on the Cloud database. In an attempt to generate a fault Figure 5A the physical form of the image target, whereas in
dataset, modification of the Simulink model is required. The Figure 5B, the features recognized by the device are overlaid on
modification involved the creation of additional subsystems the image target.
which are used for the simulation of faults. Ultimately, the goal As soon as the image target is recognized by the device,
of this experiment series is to generate fault datasets, i.e. through the integrated camera, then the transformation
datasets containing measurements of the physical model matrix, denoted as T, between the camera and the marker is
operating under malfunction. For the generation of the Eq. 1:
fault datasets, a pressure drop in the refrigerant network/ xc
T p X (1)
piping was simulated and increased humidity within the
cooling chamber. In order to process the data derived from Where:

Frontiers in Mechanical Engineering | www.frontiersin.org 6 December 2020 | Volume 6 | Article 578379

Mourtzis et al. Intelligent Predictive Maintenance Framework

FIGURE 5 | Example of Image Target used in Android-based AR applications.

xc is the projection of a point in ideal image coordinates. Therefore the depth value into real, 3D world coordinates can
T is the pose matrix. be derived from Eq. 5.
X expresses the points in world coordinates.
Therefore for the calculation of xc a 3 × 3 rotation matrix is x U
⎜
⎛
⎝y⎞ ⎟ ⎜
⎠
D⎛
⎝V ⎞⎟
⎠ (5)
utilized, denoted by R, as per the Eq. 2.
z 1
X
⎢
x
⎥ ⎢
r1,1 r1,2 r1,3 tx ⎢
⎥⎥⎥⎤⎡
⎢
⎢ Y ⎤⎥⎥⎥⎥⎥
xc
[R|t] p X0⎢
⎢ ⎡
⎣ y ⎥⎥⎤⎦
⎢
⎡ ⎢
⎣ r2,1 ⎢
⎢
⎦⎢
Consequently, for the user pose estimation in the case of
r2,2 r2,3 ty ⎣ Z ⎥⎥⎦
⎢ (2)
Microsoft HoloLens, the developed application initially prompts
z r3,1 r3,2 r3,3 tz
1 the user to select/setup an initial point of reference. This reference
point is then automatically translated into a 3D world anchor
Afterwards, in order to translate the result in pixel coordinates, based on which the AR visualizations are positioned and rendered
denoted by xpix , i.e. as a 2D representation, calibration matrix is in the physical environment.
used, denoted by C. Consequently, xpix based on Eq. 3 becomes: One of the main aspects of the AR module/application, is the
xpix f 0 px 0 xc condition monitoring of the assets. This can be done in two ways.
⎢
⎠
⎡ ⎥⎝
⎜
xpix
K p xc 0⎛ ⎟
⎝ ypix ⎞ ⎢
⎢
⎣0 f py 0 ⎤⎥⎥⎦⎜ ⎟
⎛ yc ⎞
⎠ (3) The first solution is for remote monitoring, where the responsible
1 0 0 1 0 zc engineer uses either device to visualize a scaled 3D model of the
refrigerator, upon which important information are displayed.
Based on the pose estimation steps described in the previous The second solution is for monitoring the condition of the asset
paragraph, for the registration of the AR content using Android- while inspecting it physically. In the case of the refrigerators,
based platforms, a fiducial image target is required. As soon as it is which is presented in the following paragraphs, it is impossible
recognized by the device camera, then by calculating the user’s for the responsible engineer to monitor the current status/health
position and pose, the augmentations are overlaid on the physical of the refrigerator group without having to physically inspect the
environment. What is more, in order to enhance the user experience, compressor unit, which is located away from the cooling
the application supports the functionality called “Extended chamber. However, with the proposed framework, it is
Tracking”, based on which, the handheld device can continue possible to recognize the refrigerator, by utilizing the image
overlaying the augmentations in the physical environment in the targets, discussed previously and retrieve data for the
event of the camera loosing direct contact with the image target. corresponding refrigerator group from the Cloud database. As
However, for the implementation of the AR module in the a result, the equipment/asset inspection can be performed in
Microsoft HoloLens HMD, the user pose estimation is near-real-time.
approached in a different way, as the HMD is integrated with Another aspect of the AR application is the provision of a
four (4) greyscale tracking cameras. As a result the Depth, communication tool, which enables the communication between
denoted by D, is calculated with the use of Eq. 4. the OEM and customer, in order to inspect the equipment in real
R time, and in addition to that to create basic AR instructions also in
D
√ (4) real time, by performing common “drag and drop” operation in
U + V2 + 1
2
the field of view of the user. Further to that, this functionality
Where: enables both the OEM and the client to communicate via a video
D Is the Depth call session, where the OEM can visualize the field of view of the
R is the Range, which is measured from the integrated user and with the use of basic 3D tool representation, the client
HoloLens ToF (Time-of-Flight) camera. can perform maintenance tasks in real time. The above-
U and V are the distance values of a certain point. mentioned functionalities are based on the adaptation of the

Frontiers in Mechanical Engineering | www.frontiersin.org 7 December 2020 | Volume 6 | Article 578379

Mourtzis et al. Intelligent Predictive Maintenance Framework

FIGURE 6 | (A) AR visualization of compressor and its working parameters; (B) Cooling chamber cluster and the current working conditions.

methodology presented in the research work of Mourtzis et al. User Interfaces (GUI), a Universal Windows Platform (UWP)
(2020b) Figure 6. (Microsoft, 2018) application has been developed. The benefits of
using UWP is the multi-platform implementation, the ease of
configurability, ease of implementing security protocols,
SYSTEM IMPLEMENTATION serviceability of the framework and updates’ distribution. In
the following paragraphs the functionalities and the GUIs
The proposed architecture can be realized as a multi-sided designed and developed so far will be discussed in detail. In
application. The first aspect of the application is a desktop- order to do so, the Unity 3D game engine is utilized (Unity, 2020).
based application, which communicates with the server in As regards the code scripts, the Microsoft Visual Studio IDE is
order to retrieve the data from the server and process them used (Microsoft, 2020). More specifically, for the development of
through the predictive algorithm. The predictive algorithm is the main functionalities of the application, the code scripts are
responsible for the identification of patterns within the processed written in C# programming language. Since the application
data. Each of these patterns represents a classification of the supports two user groups, one for the OEM supplier/service
possible situations of the under-examination machine, or cluster provider and one for the customers, a common login/register
of machines. A predictive algorithm will have to analyze the data system is implemented. Upon installation of the application on
gathered from the sensors so that a prediction of unforeseeable the end-user’s desired platform, the application prompts the user
machine malfunctions can be identified. However, since the data to register an account which automatically saved in the Cloud
are available on the server, it is of great importance to create an database. Therefore each time the user is connected, after getting
application for monitoring the current situation of the machines. authorized, their user group is automatically retrieved by the
From a software point of view, for the setup of the DAQ Cloud database and the suitable GUIs are loaded. It is stressed out
device, the Arduino IDE (Arduino, 2020) was used. Moreover, for that although the development of a UWP application enables
the setup of the WSN the X-CTU (X-CTU, 2020) application multi-platform support, the AR functionalities are only available
from Digi has been utilized. For the development of the Graphical for handheld devices, such as Android-based mobile phones and

Frontiers in Mechanical Engineering | www.frontiersin.org 8 December 2020 | Volume 6 | Article 578379

Mourtzis et al. Intelligent Predictive Maintenance Framework

FIGURE 7 | (A) Settings for a new refrigerator line; (B) Settings for new/old sensors.

tablets, and Head Mounted Displays (HMD) such as Microsoft Transfer Protocol Secure). Secondly, DELETE requests are not
HoloLens. allowed for anyone trying to connect to the Cloud Platform.
As long as the login is successful, if the user is listed as a client, Therefore, in order to delete any records from the database, this
then the available options are to create a new cluster of process has to be undertaken manually by the authorized system
refrigerators or process/monitor an existing cluster. The “Set administrator. In an attempt to make the proposed framework
Up New Line” functionality is targeted for new customers, or more general, a custom editor has been developed for supporting
customers that acquired new equipment, i.e. new refrigerator the functionalities of the framework itself. More specifically, the
group(s) or new DAQ device(s), as presented in Figure 7A. development team assisted by the editor can create virtually any
The core functionality of the developed framework lies within configuration of systems and functionalities, so that the
the monitoring functionalities. In the corresponding GUIs, the framework can be adapted to the actual needs of the
customer can visualize in 3D a scaled version of their refrigerator corresponding company.
group and upon request to visualize the available information, From a hardware point of view, a desktop PC has been utilized
which are automatically fetched by the Cloud database. Then, in for the development of the application as well as the Cloud
continuation, if requested, the data can be transformed into Database and its services. For the implementation of the
statistical figures, so that the client can visualize the current developed AR-based application, an Android-based tablet and
status of their equipment. All of the above-mentioned data a Microsoft HoloLens HMD are used. As regards, the DAQ
can also be visualized in the form of augmentations in case device, an Arduino Mega 2560 microcontroller is paired with the
the customer is close to the refrigerator. In order to further custom board presented in the previous paragraphs. In addition
notify the customer about an upcoming maintenance action or if to that for the XBee RF modules, the Arduino XBee shield and the
any piece of equipment requires special attention, certain alerts Adafruit explorer shield were used.
have been implemented as presented in Figures 6 and 7.
Regarding the communication interfaces between the DAQ
device, the end-user application and the Cloud Platform, CASE STUDY
RESTful API services have been developed. As regards the
communication interface between the DAQ device and the In manufacturing systems the profit is derived by the subtraction
Cloud Database, the DAQ device as a client can send POST, of operating costs from total income. Therefore, the profit
and PUT requests to the Cloud Platform, so as to enable the data becomes a problem with two possible solutions, either the
to be posted on the database if they are not existent, or updated as minimization of operating costs, or the maximization of
needed. On the contrary, the majority of the services income. As regards, operating costs, industrial equipment
implemented on the end-user application send only GET maintenance costs are also included. Production equipment
requests, in order to fetch data to the end-user’s device, with affects operating costs with machine deterioration and failures
the exception of user registration, where a POST request is sent in as well.
order to record the user’s data on the database. However, due to The applicability of the developed framework has been tested
the large volumes of sensitive data that are circulated within the and validated in a real-life industrial scenario,derived from an
proposed framework, a set of security measures are taken. OEM supplier of professional refrigerator systems. The OEM is
Initially, all the HTTP requests are of type HTTPS (Hypertext looking forward to transforming their business model based on

Frontiers in Mechanical Engineering | www.frontiersin.org 9 December 2020 | Volume 6 | Article 578379

Mourtzis et al. Intelligent Predictive Maintenance Framework

FIGURE 8 | Time distribution per task, as-is situation vs proposed

the PSS paradigm, by providing the proposed framework as a refrigerator in such case, the inspection of the equipment from an
service, in an attempt to improve their after-sales policy. What is OEM technician would require travelling and inspection which
more, it is estimated that based on the monitoring and analysis according to the OEM it would account for approximately 28 h,
from the AI algorithms, the engineering department of the OEM the order and acquisition of the replacement compressor would
will be able to gather insightful feedback, aiming at the require approximately 24 h and the installation on the
improvement of the design process and the quality of their refrigerator group would require approximately 5 h. However,
products. The main benefit of the proposed methodology, is with the adoption of the proposed framework, and based on the
that it can provide time estimations about future equipment calculation of the compressor RUL, the client is capable to order
malfunctions, which by extension can enable both the OEM and the failing part timely, thus minimizing the waiting time. Further
the client to act proactively and in time, in order to further to that, with the utilization of the AR application the inspection of
minimize the equipment downtime. Further to that, with the the equipment can be facilitated, thus eliminating the need for an
provision of the AR application knowing beforehand the OEM maintenance technician to visit the client. The time for
upcoming equipment failures, can facilitate maintenance inspection was calculated to be approximately 5 h. In the
engineers to prepare the AR content timely and communicate following figure, the time estimations for the current situation
it to the client. Therefore, the need for an external maintenance as well as the corresponding times with the adoption of the
technician is further minimized, thus reducing both the overall proposed methodology are presented Figure 8.
time and cost of maintenance.
In order to test and validate the proposed framework, the
DAQ device was installed, on an experimental refrigerator group CONCLUDING REMARKS AND OUTLOOK
located at the premises of the OEM, used for test purposes. It is
stressed out that the OEM has already integrated sensing systems The scope of this research work was to present the latest trends
on the majority of their products for monitoring purposes. regarding the fields of predictive maintenance and XR and
However, the existing solution is wired and requires a furthermore to propose a novel framework that will facilitate
computer and an engineer close to the refrigerator engineers to constantly monitor the status of the manufacturing
compartment, in order to monitor their status. Therefore, the equipment and in advance to predict the forthcoming
first step was the installation of the DAQ device, presented in the maintenance activities. By extension, the prediction of
previous paragraphs as well as the setup of the required WSN malfunctions will enable companies to schedule their
network. The WSN follows the star topology, meaning that the production more efficiently, whilst it makes them more
one XBee is connected to a PC and acts as the WSN coordinator. adaptive to any disturbances caused within the company
Then, each DAQ connects to the coordinator and transmits the limits. From the practical implementation of the developed
data at the defined rate. Afterwards, in order to handle the data framework in the industrial partner, it became evident that the
arrived at the WSN coordinator, the corresponding COM port is refrigerator downtimes can be reduced by approximately 20%,
listened by the PC via a Python script and the data are uploaded to since both the clients and the OEM were capable to monitor the
the database and saved within the corresponding CSV file. Then status of their equipment and by extension, with the use of the AI
the data saved in the Cloud database are automatically input to algorithm, the RUL prediction for crucial components of the
the Digital Twin of the refrigerator in order to simulate its refrigerator system, the client got a trustworthy estimation of
condition and calculate the RUL for the critical components. when their equipment should be maintained. In addition to that,
Based on this setup the monitoring and simulation runs were since the customer, could get an estimation of the upcoming
executed at the premises of the case study provider. For the failure, they are able to schedule a maintenance session with the
purposes of the experiments, a scenario of compressor OEM much faster and fitted to both ends’ schedule without
malfunction has been examined. For the maintenance of the creating great disturbances. An equally important finding is that

Frontiers in Mechanical Engineering | www.frontiersin.org 10 December 2020 | Volume 6 | Article 578379

Mourtzis et al. Intelligent Predictive Maintenance Framework

the maintenance costs can be reduced by approximately 10% framework in similar equipment, i.e. a fleet of assets, will enable
since the OEM can order and acquire the needed components the creation of data ensembles. The idea behind this is to utilize
beforehand, thus eliminating overnight delivery costs. Along with similar datasets in order to improve the predicting accuracy of the
that, by predicting and scheduling timely the maintenance Digital Twin.
session, the equipment is not left to run until failure, which
could affect the operation of other components, thus leading to
increased maintenance cost, due to additional technician labor DATA AVAILABILITY STATEMENT
and extra replacement parts.
Although the development and the implementation of the The raw data supporting the conclusions of this article will be
proposed framework have yielded promising results, there are made available by the authors, without undue reservation.
several implications that must be addressed before such solutions
reach an acceptable maturity level and by extension, become
commercially available. The most important implication faced, is AUTHOR CONTRIBUTIONS
that the calculation of the RUL cannot be performed in real-time
thus inducing a certain amount of latency in the AR MD conceived the idea and supervised the writing and the
visualizations. The amount of latency is affected by two major experimentation of the research work, AJ is responsible for the
issues, first the network speed and second the computational conceptualization of the project and the writing of the paper, PN
power of the system handling the Digital Twin. Another conducted the research for similar works and contributed in the
implication is the authoring of the AR visualizations in the results as well as in the writing of the paper.
field of view of the end-users. Currently, this is a manual task,
which in the future must be addressed, so that AR applications
can become useful tools in the modern manufacturing FUNDING
environment rather than increasing the complexity of the
systems, by increasing the time and effort to prepare the so- This work has been partially supported by the H2020 EC funded
called “AR scenes”. project “SPARE – Sensor-based Product monitoring system to
In the future, the Digital Twin will also be improved. It is support Augmented Reality remote maintenancE” (GA No:
estimated that following the implementation of the proposed 80765 ΔΕΡ5-0020271).

Erfani, S. M., Rajasegarar, S., Karunasekera, S., and Leckie, C. (2016). High-
REFERENCES dimensional and large-scale anomaly detection using a linear one-class SVM
with deep learning. Pattern Recogn. 58, 121–134. doi:10.1016/j.patcog.2016.03.028
Ahmed, M., and Khan, A.-S.P. (2019). Data analytics: concepts, techniques, and Fantini, P., Pinzone, M., and Taisch, M. (2020). Placing the operator at the centre of
applications. Boca Raton, FL: CRC Press Taylor & Francis Group. Industry 4.0 design: modelling and assessing human activities within cyber-
Arduino. (2020). Available at: https://www.arduino.cc/ (Accessed October 2,2020) physical systems. Comput. Ind. Eng. 139, 105058. doi:10.1016/j.cie.2018.01.025
Banerjee, I., Ling, Y., Chen, C. M., Hasan, A. S., Langlotz, P. C., Moradzadeh, N., Fast-Berglund, Å., Gong, L., and Li, D. (2018). Testing and validating Extended
et al. (2019). Comparative effectiveness of convolutional neural network (CNN) Reality (xR) technologies in manufacturing. Procedia Manufacturing. 25,
and recurrent neural network (RNN) architectures for radiology text report 31–38. doi:10.1016/j.promfg.2018.06.054
classification. Artif. Intell. Med., 97:79–88. doi:10.1016/j.artmed.2018.11.004 Hupfer, S. (2019). Capitalizing on the promise of artificial intelligence. Perspectives
Baru, A. (2018). Three ways to estimate remaining useful life for predictive on AI adoption from around the world, Deloitte Insights. Available at: https://
maintenance. Technical Articles and Newsletters, MathWorks. Available at: www2.deloitte.com/us/en/insights/focus/cognitive-technologies/global-perspectives-
https://www.mathworks.com/company/newsletters/articles/three-ways-to- ai-adoption.html (Accessed June 20, 2020)
estimate-remaining-useful-life-for-predictive-maintenance.html (Accessed Kong, Ζ., Cui, Y., Xia, Z., and He, L. (2019). Convolution and long short-term
June 20, 2020) memory hybrid deep neural networks for remaining useful life prognostics.
Chen, J., Jing, H., Chang, Y., and Liu, Q. (2019). Gated recurrent unit based Appl. Sci. 9(19), 4156–4174. doi:10.3390/app9194156
recurrent neural network for remaining useful life prediction of nonlinear Kozjek, D., Malus, A., and Vrabi”, R. (2020). Multi-objective adjustment of
deterioration process. Reliab. Eng. Syst. Saf. 185:372–382. doi:10.1016/j.ress. remaining useful life predictions based on reinforcement learning. Procedia
2019.01.006 CIRP. 93, 425–430. doi:10.1016/j.procir.2020.03.051
Chryssolouris, G. (2006). Manufacturing systems: theory and practice, 2nd ed. New Lei, Y., Li, N., Guo, L., Li, N., Yan, T., and Lin, J. (2018). Machinery health
York, NY: Springer-Verlag prognostics: a systematic review from data acquisition to RUL prediction. Mech.
Deloitte, (2017a). Predictive maintenance and the smart factory, Available at: Syst. Signal Process. 104, 799–834. doi:10.1016/j.ymssp.2017.11.016
https://www2.deloitte.com/content/dam/Deloitte/us/Documents/process- Li, X., Zhang, W., and Ding, Q. (2019). Deep learning-based remaining useful life
and-operations/us-cons-predictive-maintenance.pdf (Accessed June 20, estimation of bearings using multi-scale feature extraction. Reliab. Eng. Syst.
2020) Saf. 182, 208–218. doi:10.1016/j.ress.2018.11.011
Deloitte, (2017b). Predictive Maintenance. Taking pro-active measures based on Li, Χ., Ding, Q., and Sun, J. Q. (2018). Remaining useful life estimation in
advanced data analytics to predict and avoid machine failure, Analytics prognostics using deep convolution neural networks. Reliab. Eng. Syst. Saf.
Institute. Available at: https://www2.deloitte.com/content/dam/Deloitte/de/ 172, 1–11. doi:10.1016/j.ress.2017.11.021
Documents/deloitte-analytics/Deloitte_Predictive-Maintenance_PositionPaper.pdf Liu, Y., Hu, X., and Zhang, W. (2019). Remaining useful life prediction based on
(Accessed June 20, 2020) health index similarity, Reliab. Eng. Syst. Saf. 185, 502–510. doi:10.1016/j.ress.
Egger, J., and Masood, T. (2020). Augmented reality in support of intelligent 2019.02.002
manufacturing – a systematic literature review. Comput. Ind. Eng. 140, 106195. Loutas, T. H., Roulias, D., and Georgoulas, G. (2013). Remaining useful life
doi:10.1016/j.cie.2019.106195 estimation in rolling bearings utilizing data-driven probabilistic E-support

Frontiers in Mechanical Engineering | www.frontiersin.org 11 December 2020 | Volume 6 | Article 578379

Mourtzis et al. Intelligent Predictive Maintenance Framework

vectors regression. IEEE Trans. Reliab. 62(4), 821–832. doi:10.1109/TR.2013. dynamical network marker. Renew. Energy 154, 625–635. doi:10.1016/j.
2285318 renene.2020.03.036
Masoni, R., Ferrise, F., Bordegoni, M., Gattullo, M., Uva, A. E., Fiorentino, M., et al. Sun, H., Liu, Y., Pan, J., Zhang, J., and Ji, W. (2020). Enhancing cutting tool
(2017). Supporting remote maintenance in industry 4.0 through augmented sustainability based on remaining useful life prediction, J. Clean. Prod. 244,
reality. Procedia Manufacturing. 11, 1296–1302. doi:10.1016/j.promfg.2017.07. 118794. doi:10.1016/j.jclepro.2019.118794
257 Tao, F., Qi, Q., Liu, A., and Kusiak, A. (2018). Data-driven smart manufacturing. J.
Microsoft. (2020). Visual Studio. Best-in-class tools for any developer. Available at: Manuf. Syst. 48, 157–169. doi:10.1016/j.jms y.2018.01.006
https://visualstudio.microsoft.com/ (Accessed June 20, 2020) Unity. (2020). Available at: https://unity.com/ (Accessed June 20, 2020)
Microsoft. (2018). What’s a universal Windows platform (UWP) app? Available at: Vogl, G. W., Weiss, B. A., and Helu, M. (2019). A review of diagnostic and
https://docs.microsoft.com/en-us/windows/uwp/get-started/universal-application- prognostic capabilities and best practices for manufacturing. J. Intell. Manuf.
platform-guide (Accessed June 20, 2020) 30(1), 79–95. doi:10.1007/s10845-016-1228-8
Mitchell, R. J. (1995). “How computers process data.” In: Microprocessor systems. Vorraber, W., Gasser, J., Webb, H., Neubacher, D., and Url, P. (2020). Assessing
London: Palgrave. augmented reality in production: remote-assisted maintenance with HoloLens.
Mourtzis, D. (2020). Simulation in the design and operation of manufacturing Procedia CIRP. 88, 139–144. doi:10.1016/j.procir.2020.05.025
systems: state of the art and new trends. Int. J. Prod. Res. 58(7), 1927–1949. Wang, H., Peng, M. J., Miao, Z., Liu, Y. K., Ayodeji, A., and Hao, C. (2020).
doi:10.1080/00207543.2019.1636321 Remaining useful life prediction techniques for electric valves based on
Mourtzis, D., Angelopoulos, J., and Panopoulos, N. (2020a). A framework for convolution auto encoder and long short term memory. ISA (Instrum. Soc.
automatic generation of augmented reality maintenance & Repair instructions Am.) Trans. doi:10.1016/j.isatra.2020.08.031
based on convolutional neural networks, 53rd CIRP conference on Wen, P., Zhao, S., Chen, S., and Li, Y. (2021). A generalized remaining useful life
manufacturing systems (CMS 2020). Procedia CIRP. 93, 977–982. doi:10. prediction method for complex systems based on composite health indicator.
1016/j.procir.2020.04.130 Reliability Engineering & System Safety. 205, 107241. doi:10.1016/j.ress.2020.
Mourtzis, D., Siatras, V., Angelopoulos, J., and Panopoulos, N. (2020c). An 107241
augmented reality collaborative product design cloud-based platform in the Wolfartsberger, J., Zenisek, J., and Wild, N. (2020). Data-driven maintenance:
context of learning factory. Procedia Manufacturing. 45, 546–551. doi:10.1016/j. combining predictive maintenance and mixed reality-supported remote
promfg.2020.04.076 assistance. Procedia Manufacturing. 45, 307–312, doi:10.1016/j.promfg.2020.
Mourtzis, D., Siatras, V., and Angelopoulos, J. (2020b). Real-time remote 04.022
maintenance support based on augmented reality (AR). Appl. Sci. 10(5). X-CTU, (2020). Available at: https://www.digi.com/products/embedded-systems/
doi:10.3390/app10051855 digi-xbee/digi-xbee-tools/xctu (Accessed October 2, 2020)
Mourtzis, D., Vlachou, A., and Zogopoulos, V. (2017). Cloud-based augmented Yang, F., Habibullah, S. M., and Shen, Y. (2021). Remaining useful life
reality remote maintenance through shop-floor monitoring: a product- prediction of induction motors using nonlinear degradation of health
service system approach. J. Manuf. Sci. Eng. 139(6), 061011. doi:10.1115/ index. Mech. Syst. Signal Process. 148, 107183. doi:10.1016/j.ymssp.2020.
1.4035721 107183
Mourtzis, D., Vlachou, E., Milas, N., and Xanthopoulos, N. (2015). A cloud-based Yang, H., Zhao, F., Jiang, G., Sun, Z., and Mei, X. (2019). A novel deep learning
approach for maintenance of machine tools and equipment based on shop-floor approach for machinery prognostics based on time Windows. Appl. Sci. 9, 4813.
monitoring. Procedia CIRP 41, 655–660. doi:10.1016/j.procir.2015.12.069 doi:10.3390/app 9224813
Mourtzis, D., Zogopoulos, V., Katagis, I., and Lagios, P. (2018). Augmented reality Zhang, S., and Lang, Z. Q. (2020). SCADA-data-based wind turbine fault detection:
based visualization of CAM instructions towards industry 4.0 paradigm: a CNC a dynamic model sensor method. Contr. Eng. Pract. 102, 104546. doi:10.1016/j.
bending machine case study. Procedia CIRP 70, 368–373. doi:10.1016/j.procir. conengprac.2020.104546
2018.02.045 Zhao, Ζ., Liang, B., Wang, X., and Lu, W. (2017). Remaining useful life prediction
Nicholson, A., Webber, S., Dyer, S., Patel, T., and Janicke, H. (2012). SCADA of aircraft engine based on degradation pattern learning, Reliab. Eng. Syst. Saf.
security in the light of cyber-warfare. Comput. Secur. 31(4), 418–436. doi:10. 164, 74–83. doi:10.1016/j.ress.2017.02.007
1016/j.cose.2012.02.009
Palmarini, R., Erkoyuncu, A. J., Roy, R., and Torabmostaedi, H. (2018). A Conflict of Interest: The authors declare that the research was conducted in the
systematic review of augmented reality applications in maintenance. Robot. absence of any commercial or financial relationships that could be construed as a
Comput. Integrated Manuf. 49, 215–228. doi:10.1016/j.rcim.2017.06.002 potential conflict of interest.
Pang, Y., He, Q., Jiang, G., and Xie, P. (2020). Spatio-temporal fusion neural
network for multi-class fault diagnosis of wind turbines based on SCADA data. Copyright © 2020 Mourtzis, Angelopoulos and Panopoulos. This is an open-access
Renew. Energy. 161, 510–524. doi:10.1016/j.renene.2020.06.154 article distributed under the terms of the Creative Commons Attribution License (CC
Rezaeianjouybari, B., and Shang, Y. (2020). Deep learning for prognostics and BY). The use, distribution or reproduction in other forums is permitted, provided the
health management: state of the art, challenges, and opportunities. original author(s) and the copyright owner(s) are credited and that the original
Measurement. 163, 107929. doi:10.1016/j.measurement.2020.107929 publication in this journal is cited, in accordance with accepted academic practice.
Ruiming, F., Minling, W., Xinhua, G., Rongyan, S., and Pengfei, S. (2020). No use, distribution or reproduction is permitted which does not comply with
Identifying early defects of wind turbine based on SCADA data and these terms.

Frontiers in Mechanical Engineering | www.frontiersin.org 12 December 2020 | Volume 6 | Article 578379