PROCEEDINGS OF SPIE

SPIEDigitalLibrary.org/conference-proceedings-of-spie

Machine learning for structural health

monitoring: challenges and
opportunities

Fuh-Gwo Yuan, Sakib Ashraf Zargar, Qiuyi Chen,

Shaohan Wang

Fuh-Gwo Yuan, Sakib Ashraf Zargar, Qiuyi Chen, Shaohan Wang, "Machine
learning for structural health monitoring: challenges and opportunities," Proc.
SPIE 11379, Sensors and Smart Structures Technologies for Civil,
Mechanical, and Aerospace Systems 2020, 1137903 (23 April 2020); doi:
10.1117/12.2561610

Event: SPIE Smart Structures + Nondestructive Evaluation, 2020, Online Only

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024 Terms of Use: https://www.spiedigitallibrary.org/terms-of-use

 Keynote Paper

Machine Learning for Structural Health Monitoring: Challenges and

Opportunities
Fuh-Gwo Yuan*, Sakib Ashraf Zargar*,
Qiuyi Chen, Shaohan Wang

Department of Mechanical and Aerospace Engineering, North Carolina State University, Raleigh,
North Carolina 27606, USA

*
Corresponding authors, email: yuan@ncsu.edu, szargar@ncsu.edu

ABSTRACT

A physics-based approach to structural health monitoring (SHM) has practical shortcomings which restrict its suitability to
simple structures under well controlled environments. With the advances in information and sensing technology (sensors and
sensor networks), it has become feasible to monitor large/diverse number of parameters in complex real-world structures either
continuously or intermittently by employing large in-situ (wireless) sensor networks. The availability of this historical data has
engendered a lot of interest in a data-driven approach as a natural and more viable option for realizing the goal of SHM in
such structures. However, the lack of sensor data corresponding to different damage scenarios continues to remain a
challenge. Most of the supervised machine-learning/deep-learning techniques, when trained using this inherently limited
data, lack robustness and generalizability. Physics-informed learning, which involves the integration of domain knowledge
into the learning process, is presented here as a potential remedy to this challenge. As a step towards the goal of automated
damage detection (mathematically an inverse problem), preliminary results are presented from dynamic modelling of beam
structures using physics-informed artificial neural networks. Forward and inverse problems involving partial differential
equations are solved and comparisons reveal a clear superiority of physics-informed approach over one that is purely data-
driven vis-à-vis overfitting/generalization. Other ways of incorporating domain knowledge into the machine learning
pipeline are then presented through case-studies on various aspects of NDI/SHM (visual inspection, impact diagnosis).
Lastly, as the final attribute of an optimal SHM approach, a sensing paradigm for non-contact full-field measurements for
damage diagnosis is presented.

Keywords: Machine learning, artificial neural networks, physics-informed learning, visual inspection, augmented reality,
impact diagnosis, damage diagnosis, structural health monitoring.

Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2020
edited by Haiying Huang, Hoon Sohn, Daniele Zonta, Proc. of SPIE Vol. 11379, 1137903
© 2020 SPIE CCC code: 0277-786X/20/$21 · doi: 10.1117/12.2561610

Proc. of SPIE Vol. 11379 1137903-1

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 1. Introduction

Structural health monitoring (SHM) is a transdisciplinary field of engineering devoted to ensuring the structural integrity
and operational safety of a component or structure. This is achieved by facilitating the (in-service) detection and
characterization of damage in a structure that may adversely affect its ability to fully and safely perform its intended
function. As such, the problem of damage detection lies at the heart of SHM and the goal is to identify the damage at the
earliest possible stage (in near real-time). This is indispensable for ensuring timely corrective action in order to minimize
the system downtime, overall operational and maintenance costs, and to reduce the risk of catastrophic failure. Broadly
speaking, as can be seen in Fig. 1, two approaches have traditionally been adopted for automated damage detection in SHM:
physics-based and data-driven.
Physics-based methods, in some form or the other, rely primarily on the physical laws governing the structural behavior in
order to extract meaningful information about the damage and its evolution from the measured sensor data. However,
difficulty in the modelling of complex real-world structures, considerations of multiple sensing modalities, material and/or
geometric non-linearity, and uncertainty in material properties, boundary conditions, environmental/operational variations
are some of the factors that make this exclusive reliance on the system physics rather impractical in case of complex real-
world structures. This limits the application of such methods to the health monitoring of rather simple structures with pre-
defined boundary conditions and well-controlled environments [1-3]. As the underlying system complexity increases, such
an approach becomes much less dependable. Owing to the advances in information and sensing technologies in recent times,
it has now become feasible to monitor a large number of parameters in-situ in large/complex real-world structures on either a
continuous or sporadic basis. This motivates the use of a data-driven approach for SHM where-in damage assessment is
dealt with, at least at a lower level, primarily as being a type of statistical pattern recognition problem thus circumventing
some of the major challenges associated with the physics-based approach.
During the last few decades, machine learning (ML) techniques have been extensively employed by researchers both for
vibration based and ultrasonic guided wave based damage detection [4-14]. Machine learning in SHM aims at building
models or representations for mapping input patterns in measured sensor data to output targets for damage assessment at
different levels, Rytter [15]. Conventional machine learning techniques are however limited in their ability to process the
large amounts of measured sensor data in their raw form. As such, careful engineering and considerable domain knowledge
are required to extract damage-sensitive features from the raw data which are then fed into a suitable ML model.
Historically, the choice of these hand-crafted damage-sensitive features has stemmed from the wealth of well-developed
literature on physics-based SHM techniques including but not limited to modal assurance criterion (MAC) and Coordinate
MAC [16, 17], modal strain energy (MSE) [18, 19], modal curvature (MC) [20, 21], modal flexibility (MF) [22-24], damage
locating vector (DLV) [25], wavelet transform [26-30], Hilbert-Huang transform [31], probabilistic reconstruction
algorithm (PRA) [32], ZLCC [33] etc,. Multi-layer feedforward artificial neural networks (ANNs), also referred to as multi-
layer perceptrons (MLPs), have been the most well-known and widely used ML technique for automated damage detection
[34-37]. Apart from the algorithm itself, the overall performance of such a damage detection system depends on the choice
of the damage-sensitive features used. The problem with hand-crafted features is that while they may be sub-optimal even
for the structure under consideration, there is no guarantee that the same set of features can be adopted for other structures.
In order to overcome this shortcoming, deep learning (DL) methods have attracted much attention in recent years. These
allow data to be used in their raw form and features are learned automatically from data using a general-purpose learning
procedure. Deep learning automatically discovers intricate features in high-dimensional data which has led to its widespread
adoption across many application domains [38-41]. It is certain that DL will continue to flourish in the future and the
progress will only be accelerated with advances in computational capabilities through the development of high-tech central
processing units (CPUs)/graphics processing units (GPUs), availability of large amounts of data (big data), and development
of new learning algorithms.
While supervised learning is the most common and well developed form of learning, a fundamental challenge in many
SHM applications is that damage detection must often be performed in an unsupervised manner. This is because, for real-
world structures, it is highly unlikely to have data corresponding to different damage scenarios. Among the four levels of

Proc. of SPIE Vol. 11379 1137903-2

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Figure 1: The big picture of structural health monitoring (SHM) indicating that as the complexity of the structure increases, exclusive
reliance on the system physics for analyzing the measured sensor data is reduced, and an approach that is primarily data-driven is
adopted. An optimal approach for large/complex real-world structures must be data-driven with the system-physics/domain knowledge
embedded in some form.

damage assessment identified by Rytter [15], the lowest level (i.e., establishing the presence of damage) has been achieved
in the past through unsupervised learning by what is referred to as novelty (outlier) detection [42-44]. Higher levels of
damage assessment in real-world structures require either a mechanism for augmenting the insufficient/incomplete training
data by incorporating some form of prior knowledge into the learning/training process and/or the development of more
comprehensive data acquisition systems. These will henceforth be referred to as the two aspects of an optimal SHM system.
This paper introduces these two aspects through a discussion on physics-informed learning in Sections 2, 3 and 4 and non-
contact full-field measurements using state-of-the-art high-speed digital cameras for vision-based SHM in Section 5.
Summary, conclusions and future prospects are finally presented in Section 6.

2. Physics-informed Machine Learning

Mathematically speaking, the problem of automated damage detection from measured sensor data is an inverse problem. In
conventional machine learning/deep learning, it is usually formulated as a minimization problem with a purely data-based
loss function. However, in most SHM applications, not only is the cost of data acquisition prohibitive, but it is highly
unlikely to have sufficient data capturing different damage scenarios, especially for structures that have been in service for
relatively short periods of time. When trained using this inherently limited data, the models tend to overfit the given data
which eventually leads to poor generalization. This gives rise to the need for some form of model regularization which
primarily entails guiding the training process towards an optimal solution in one way or the other. Accomplishing this can
result in what may be thought of as data-efficient function approximators.

Proc. of SPIE Vol. 11379 1137903-3

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 The requisite domain knowledge for model regularization (guiding the training process) can originate from various sources,
can be represented in different forms, and can be integrated into the model in different ways [45]. From the literature on
informed machine learning, in case of ANNs, knowledge has been integrated at all three stages–input (pattern) layer [46-
53], hidden layers [54-57], output (target) layer [58-60]. Specifically within the context of SHM, domain knowledge can:
(1) Aid in the extraction of damage-sensitive features from raw sensor data in cases where conventional machine learning
is used and data cannot be used in their raw form.
(II) Augment the measured sensor data through transfer learning or by providing secondary data sets from simulations,
historical NDI/SHM records or data from similar structures. The diversity of the available historical data usually leads to
the need for some form of data fusion as a pre-processing step.
(III) Dictate the network architecture by defining local connectivity of nodes within the network, choice of activation
functions, learning algorithm, etc.
(IV) Impose constraints on the output through proper loss function design or knowledge about damage types and
characteristics.
The list is by no means exhaustive as knowledge-based learning in SHM is still in its infancy and is envisioned to bloom in
the coming years. In this paper, two classes of problems are discussed based on the type of domain knowledge utilized:
1. When the prior knowledge is in the form of physical laws governing the structural behavior and the same is representable
in the form of compact mathematical models governed by partial differential equations (PDEs). The integration of this
knowledge into the model is done directly at the loss function level i.e., apart from having a data-based component, the loss
function is designed to have a component that is physics-based as well. This is discussed in more detail in Section 3 with
the help of forward and inverse problems for dynamic modelling of beams which can be thought of as a promising first step
towards the eventual goal of damage detection.
2. When the prior knowledge is available in other less succinct forms like expert human knowledge about a particular
system/structure, results from computer simulations which mimic the real-world situation, general information about an
underlying physical phenomenon etc,. This form of knowledge can be integrated into the model through strategies like
transfer learning, network architecture customization etc,. Section 4 discusses this in more detail with case studies directly
related to NDI/SHM.

3. Prior knowledge in the form of Compact Mathematical Models (PDEs)

Figure 2: Conventional and ANN-based approaches for solving differential equations

Proc. of SPIE Vol. 11379 1137903-4

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Differential equations is the lingua franca long developed to formulate physical phenomena with both accuracy and
succinctness. A purely analytical solution to these equations is feasible only when the underlying structure/phenomenon
being modeled is rather simple and straight-forward. Also, the Green’s function approach is usually limited to linear time-
invariant cases for unbounded domains. Often times, owing to the inherent structural complexity and/or the nature of the
initial/boundary conditions, such approaches no longer remain tractable. It is in these situations that numerical methods for
solving differential equations come to the rescue. Some commonly used methods include Runge-Kutta methods, collocation
method, weighted residual method etc. The recently popular methods such as finite-element, finite-volume, finite-difference
etc., which involve discretization of the domain and weakly solving the differential equations over this discretization yield
solutions that are discrete and have limited differentiability.
All these numerical methods have been studied for a long time and are well developed, however, when the source functions
and/or initial/boundary conditions change, the results need to be computed afresh from scratch in each case. The experience
gained from each computation is lost as there is no fundamental learning of the underlying mechanics. As the size of the
structure increases, each simulation becomes computationally expensive owing to the curse of dimensionality and as a result
these methods become prohibitively slow. If however a mapping is learned for the structure/phenomenon being modelled,
slight variations in the input can be handled rather quickly. ANNs are comprised of a densely interconnected set of very
simple processing units (nodes) that can learn highly challenging non-linear mappings with high robustness, adaptability,
and fault tolerance due to its distributed representation [61]. Researchers have long been interested in developing
representations based on ANNs (which act as universal function approximators [62]) for approximate solutions to these
differential equations [63-71]. Research in this direction blossomed especially after 2010 with the advent of GPUs
(monstrous computational power) and TensorFlow (a deep learning framework with embedded automatic differentiation).
Raissi et al. [72, 73] solved a variety of non-linear PDEs including Brown equations, Schrodinger equations etc., using
TensorFlow. Then they combined this purely numerical method for solving differential equations with traditional data-
driven approach to function approximation and actualized the idea of physics-informed learning. By doing so, they were
able to not only use sparse datasets to reliably train neural networks, but also to inversely determine unknown parameters
of PDEs [74]. Around the same time, several other researchers started investigating physics-informed learning and some of
that work can be found in [75-80]. In this paper, the solution of a beam vibration problem using an ANN is presented first
followed by a direct demonstration of system physics coming to the rescue when training neural nets with limited data
within the context of SHM. The latter is achieved by solving an inverse problem using the same ANN architecture used for
the forward problem, for reconstructing the full displacement field using measured sensor data from a sparsely distributed
array mounted on the beam. Comparison with the exact analytical solutions is made to highlight the accuracy and
generalization capabilities of the physical-informed learning paradigm.

3.1 Beam Vibration Modeling using ANNs (Forward Problem)

Based on Euler-Bernoulli beam theory, the vibration of a simply supported beam under transverse load q(x,t) is governed
by Eq. (1) with boundary conditions (BCs) described by Eq. (2)

Figure 3: Setup of a simply supported beam

Proc. of SPIE Vol. 11379 1137903-5

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 4w 2w
   q, x    (0, l ), 0, l   (1)
x 4 t 2

w(0, t )  w(l , t )  0, wxx (0, t )  wxx (l , t )  0 (2)

For free vibration of the beam (q=0), let us suppose the initial conditions (ICs) are given by

w( x,0)  w0 ( x), wt ( x,0)  w0 ( x) (3)

where EI/A, E is the elastic modulus of the beam’s material, I is the area moment of inertia, is the density of the
material, A is the cross sectional area, l is its length, and w is the transverse displacement. Collocation, which entails
choosing a number of points (called collocation points) in the spatial and temporal domains, is adopted and the loss function
 for solving the PDE using an ANN consists of three terms and can be expressed as follows:

 MSEI  MSEB  MSEG

  wˆ ( x ,0)  w ( x ) 
NI
1
 I  wˆ t ( xi ,0)  w0 ( xi )
2 2
i 0 i
NI xi 

  wˆ (0, t ) 
NB
1 2 2 2 2
 B j  wˆ (l , t j )  wˆ xx (0, t j )  wˆ xx (l , t j ) (4)
NB t j T

NG
1
 G   wˆ ( xk , tk )  wˆ tt ( xk , tk )
2
xxxx
NG xk 
tk T

where wˆ ( x, t ) is the predicted displacement and the first term in is the mean squared error (MSE) term associated with
the initial conditions, the second term corresponds to the boundary conditions and the third term to the governing equation.
NI, NB, and NG stand for the number of collocation points selected from T,  , and Ω respectively. The weights I, B,
and G for the three terms are determined heuristically to be 1, 1, and 2×10-4 respectively. For free vibration of the simple
supported beam with =1, l=1 and
w0 ( x)  sin(3 x), w0 ( x)  0,   [0,1], T  [0,0.15]

the exact solution for transverse displacement can be easily obtained as w(x,t)= sin(3x).cos(9t). The steady-state solution
can be considered as a superposition of two transient waves with identical amplitude and propagating in opposite direction
as w(x,t)= ½ {sin[(3x–3)]+ sin[(3x+3)]}.
The collocation method implemented via the ANN shows good convergence as long as the time is short, however, for long
time durations, convergence problems are encountered and the solutions often converge to the trivial solution. To
circumvent this problem, the entire time duration is segmented into a finite number of short time-windows and solved
sequentially. By invoking Huygen’s principle, which states that every point on a wavefront is itself the source of secondary
wavelets, the solution at the end of a particular time-window can be treated as the initial condition for the subsequent time-
window. For this example, the time domain is divided into six time windows (to satisfy the Nyquist criterion). For each
window, an ANN with eight hidden layers each of which consists of sixty four neurons is used to approximate the full
displacement field. For expediting the process and ensuring convergence, the training process in each case is divided into

Proc. of SPIE Vol. 11379 1137903-6

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Figure 4: Comparison of the MLP predicted solution with the exact solution for the simply-supported beam with given initial
conditions

two phases. The first phase primarily focuses on fitting the ICs and BCs on a data-driven basis, thus the last term in the loss
function  is excluded. A gradient descent method is performed first (number of collocation points NI = NB = 50) followed
by quasi-Newton algorithm (a limited-memory quasi-Newton code for bound-constrained optimization, L-BFGS-B) (NI =
NB = 150) to guide the solution closer to the global minimum in a rather coarse but speedy manner. In the second phase,
the term corresponding to the governing equation is added back to the loss function, then L-BFGS-B is employed to
finely optimize the parameters of the ANN. The collocation points are selected from a uniform 150 × 150 grid such that
NI = NB = 150 and NG = 22500. The results displayed in Fig. 4 show good agreement between the exact solution w(x,t)
and the predicted solution wˆ ( x, t ) .

3.2 Displacement field Reconstruction using Physics-informed ANNs (Inverse Problem)

Figure 5: Setup of a simply supported beam with a sparse sensor array

Proc. of SPIE Vol. 11379 1137903-7

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 After demonstrating the applicability of ANNs for solving PDEs in the previous example as a forward problem, the power
of physics-informed learning for the inverse problem is now demonstrated vis-à-vis training with limited samples. This is
done via a problem involving displacement field reconstruction from a sparse sensor array. Fig. 5 shows the schematic of a
simply supported beam with a surface mounted sparse sensor array used in the study. The sparse sensor data w( xi , t ) is
synthetically obtained such that w( xi , t )  w( xi , t )  n( xi , t ) where xi is the location of the ith sensor, w( xi , t ) is the
exact displacement at xi sampled from the analytical solution and n( xi , t ) is the Gaussian noise with zero mean. The goal
is to reconstruct the full displacement field from the sparse measurements and highlight the advantage of embedding the
governing equation into the loss function. The loss function for solving it using an ANN consisting of four terms can be
defined as follows:

 MSED  MSEB1  MSEG  MSEB 2

  wˆ  0, t  
ND NB
1 1
 wˆ  xi , ti   w  xi , ti    B1  wˆ 1, ti 
2 2 2
 D i
ND xi ,ti  NB ti T

  wˆ  0, t  
NG NB (5)
1 1
  wˆ xxxx  xi , ti   wˆ tt  xi , ti    B  wˆ xx 1, ti 
2 2 2
 G xx i
NG xk 
2
NB ti T
tk T

where the first term is the purely data-driven loss, the second term is for the Dirichlet boundary conditions, the third term
is for the governing equation, and the fourth term is for high order boundary conditions. The weights D, B1, G and B2
for the four terms are determined heuristically to be 5000, 25000, 1 and 1 respectively. For the simple supported beam
shown in Fig. 5 with w(x,t)= sin(x).cos(2t) and =4, in the first phase, the parameters of the ANN are trained with Adam
optimizer with a learning rate of 0.01 for 10000 iterations. During each iteration, while all the D measurements are used,
the collocation points are selected from the defined domain. To be specific, B = 50 points are sampled for the boundary
conditions and G = 2500 points are sampled for the governing equation. After that, the parameters are further optimized
with L-BFGS-B on a larger number of static collocation points until the loss converges. The collocation points for L-BFGS-
B optimizer are taken from an equally spaced 150×150 grid over the entire domain, so that B = 150 andG = 22500 . Fig.
6(a) shows the comparison between the results obtained with physics-informed learning and those obtained using the sparse
data only and Fig. 6(b) shows the error for the two approaches as the sparsity of the data and SNR changes. Clearly physics-
informed learning outperforms traditional data-driven learning vis-à-vis overfitting when data is sparse and/or noisy. Table
1 presents the comparison in a tabulated manner. The maximum relative error metric used for comparison is defined as
(max |𝑤 ̂ − 𝑤|⁄max |𝑤|).

Table 1: Physics-informed learning vs data-driven learning

SNR=20 SNR=10 SNR=15 SNR=20 SNR=10

20 Sensors 20 Sensors 13 Sensors 5 Sensors 5 Sensors
Training
Physics 2293.33 1808.92 1864.43 1846.19 1398.11
Time (s)
Informed
Max. Error 2.7% 2.4% 0.72% 0.95% 1.9%
Training
Data- 48.25 48.24 42.88 52.35 37.21
Time (s)
Driven
Max. Error 3.9% 7.2% 6.4% 29.7% 30.3%

Proc. of SPIE Vol. 11379 1137903-8

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 (a) (b)

Figure 6: (a) Comparison of the results for the displacement field reconstruction using physics-informed approach and purely data-
driven approach (b) Variation of the error for physics-informed learning and data-driven learning as SNR value and data sparsity
changes

If the system parameter µ is unknown and is to be identified, it can be set as a trainable parameter and inversely determined
6
through training. For instance, a displacement field w( x, t )   n sin n x  (cos n 2 2t / 8) is constructed by linearly
n 1
combining six mode shapes where the coefficients n are random sampled from [-1, 1]. The system parameter for generating
this displacement field is =0.0156 but assumed unknown. Noisy measurements extracted from this field are then provided
to an MLP for it to reconstruct this displacement field and identify the unknown parameter at the same time. Same training
strategy is used as above and the result is shown in Fig. 7.  is determined to be 0.01555, with a relative error of only 0.32%.
To summarize, in this section, initial promise has been demonstrated for the use of ANNs for solving PDEs in forward
problems and simple inverse dynamic problems for beams. This can be regarded as a promising first step towards the
ultimate goal of solving more complicated inverse problems for automated damage detection in SHM. This logic is
analogous to the historical evolution of conventional numerical methods for inverse problems through the development of
different regularization schemes after methods for solving forward problems were well matured. Next, two case studies are
presented to demonstrate how knowledge in other forms could be utilized for augmenting the ML/DL models. This is done
through cases directly falling within the context of NDI/SHM. It is worth mentioning that while the discussion in Section 3
dealt primarily with discrete sensor array data, Sections 4 shifts the focus to vision based health monitoring with Section
4.1 being a static problem (still images as input) and Section 4.2 being a dynamic problem (wavefield videos as input).

Proc. of SPIE Vol. 11379 1137903-9

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Figure 7: Results for the full displacement field reconstruction for the given beam when the parameter 𝜇 is not provided

4. Examples on Physics-informed Deep Learning in NDI/SHM

In this Section, a physics-informed deep learning (DL) approach for enhanced visual inspection via augmented reality is
first presented as a demonstration of how domain knowledge in the form of expert human feedback and historical NDI/SHM
records can be used to augment the training data. This is followed by a demonstration of impact diagnosis using a physics-
informed deep learning model.

4.1 Enhanced Visual Inspection via Augmented Reality (AR)

The first direct demonstration of the role of domain knowledge in augmenting the ML/DL models for tasks directly related
to NDI/SHM is done through what is referred to as enhanced visual inspection. In the aerospace industry, for safety critical
structures, visual inspection (VI) is usually the first line of defense and given its various advantages, it accounts for at least
80% of all aircraft inspections according to FAA [81]. Apart from being considerably flexible, it is the most direct, intuitive,
straightforward, and economical method for accessing the condition of a structure [82]. However, when performed in its
traditional form (by a human inspector without any external aids), VI has certain inherent shortcomings like, being error-
prone, labor-intensive, tedious, subjective/inconsistent, etc [83-85]. An inspector is required to first acquire relevant training
over a period of time and then to bridge the gap between the acquired knowledge and the physical systems in the field
mentally which limits his/her ability to make the best use of the wealth of information available about the inspection task
being performed.
The use of augmented reality (AR) as an information delivery paradigm has been proposed in order to overcome some of
the major shortcomings of traditional VI. AR encompasses a set of technologies that superimpose digital data and images on
the physical world thereby enhancing the user’s perception of reality [86]. It has the potential to bridge the disconnect between
the physical and digital worlds, thus enabling humans to make efficient use of the information available and consequently make

Proc. of SPIE Vol. 11379 1137903-10

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Figure 8: Overview of the type of domain knowledge available and its integration into the machine learning (ML) pipeline for
accomplishing enhanced visual inspection

more informed decisions. As such, it can act as a tool to enhance the traditional visual inspection process. A proof-of-concept
for such an enhanced visual inspection system using a head mounted AR device (AR glasses) is presented here. The primary
focus in the study was the detection of most commonly occurring defects in metallic structures– corrosion, fatigue cracks,
and/or a combination of the two. This was achieved by deploying a deep learning (DL) based computer vision model on
the AR device.
Oftentimes, considerations of system downtime and accessibility prohibit a detailed inspection of the entire system. Under
such circumstances, while traditional VI may give rise to high rates of false-positives (false alarms) and/or false-negatives
(misdetections), using a purely data-driven approach to search for damages/anomalies over the entire region of interest
(ROI) can be extremely expensive computationally making real-time detection almost impractical with the current state-of-
the-art AR devices. Also, depending on the distance and/or angle between the inspector and the area being inspected, a
damage may or may not be easily discernible. As such, with the aim of improving the probability of detection (POD)
enhancing the overall efficiency of the detection process, prior domain knowledge was incorporated into the ML pipeline.
Fig 8 gives an overview of the type of domain knowledge available and its integration into the machine learning pipeline.
The details are explained next.
Rather than searching only for damages (in this case cracks/corrosion) in the actual ROI, regions that are prone to damage
are also categorized and grouped. The information about damage prone areas can come either from expert knowledge about
the system or from historical data. As an example, areas prone to cracking may include regions with similar types of
fasteners like bolts, rivets, etc. In this way, a preliminary analysis of the ROI may lead to the detection of prominent damages
and/or damage prone regions. If a damage prone region is detected, the inspector is prompted to zoom into the region in
order to carry out a more detailed analysis with the aim of detecting less prominent damages i.e., damages that are otherwise
below the detectability threshold of the algorithm at the original scale of view. This can be achieved either by using the
zoom-in function in the AR device camera or by simply walking closer to the region if there are no accessibility issues. In
this way, analyzing the ROI at multiple scales (prompted by expert/domain knowledge about the system) has the potential
to greatly increase the POD while reducing false alarms. It should also be noted that since a detailed analysis is carried out
only of the highlighted damage prone regions, this region-based detection paradigm has the potential to make the overall
inspection process much more efficient than carrying out a detailed inspection of the entire system. A schematic of the
proposed physics-informed methodology is shown in Fig. 9 and some representative results are shown in Fig 10. The results
were obtained using Epsom BT-300 smart glasses (AR/Developer edition) with a pre-trained MobileNet (SSD MobileNet
V2) and k-means clustering was used to group the detected fasteners together. More details about the DL algorithms used
in the study and the details of the dataset used for training can be found in [87].
While the identification of visible damage is an important aspect of monitoring the health of structures, the real challenge
in most SHM applications is posed by hidden damage. This is discussed in the next section.

Proc. of SPIE Vol. 11379 1137903-11

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Figure 9: Multi-scale analysis of the region-of-interest (ROI) prompted by the domain knowledge

Figure 10: (a) Original ROI (b) Fasteners detected in the ROI, no cracks detected (c) Regions identified based on the fasteners after
zooming-in into the identified regions, the cracks become more prominent thereby increasing the overall POD. Each detection in
the figure has the associated confidence level with it.

Proc. of SPIE Vol. 11379 1137903-12

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 4.2 A Physics-informed Deep Learning (DL) Approach for Impact Diagnosis

Effective, reliable, and robust identification of impact events that have the potential to damage a structure is an aspect of
SHM that is of prime importance in the aerospace industry. This is commonly referred to as impact diagnosis and its
importance stems from the fact that low-velocity impact events (like tool drops, runway debris, bird strikes etc.) can lead
to what is called barely-visible-impact-damage (BVID) in composites which has been found to be the most prominent cause
of in-service damage to aerospace structures. The existence and/or extent of this damage can be correlated with the impact
energy (which is the area underneath the impact force time-history curve), with no damage being assumed below a certain
energy threshold. Impact diagnosis, which entails the identification of the impact location and reconstruction of the impact
force time-history, can quantify this impact energy and in this way can serve as an indirect method for the detection of an
otherwise invisible damage right at its inception [88].
Low-velocity impact on a structure emanates an elastic wave that propagates through the structure carrying a wealth of
information about the impact event. Most of the classical impact diagnosis methods (both physics-based [89, 90] and data-
driven [91-94]) capture information about the wave propagation phenomenon only at discrete locations by employing a
limited number of sensors spatially distributed on the structure. Both of these approaches are limited in their practicality as
only a limited amount of information about the scattered wavefield can be captured by the discrete sensor array. This
becomes especially problematic in complex real-world structures with geometric features like joints, stiffeners, etc., where
the discrete sensor signals fail to provide sufficient insights as they are often corrupted by high levels of coherent noise
from multiple wave reflections, wave mode conversions etc. In order to address this issue of limited wavefield data, it was
proposed to use the full wavefield for impact diagnosis. A proof-of-concept for a DL based approach for analyzing full
wavefields for impact diagnosis is presented here. For this study, while simulated wavefields were employed, the feasibility
of an integrated high-speed camera system for non-contact full-field wavefield capture is discussed in the next section.
The high spatio-temporal dimensionality of the wavefield mandates the use deep learning for analysis. Also, the nature of
the impact diagnosis problem requires the capturing of context from the wavefield evolution which necessitates learning
across multiple time-frames of the wavefield simultaneously rather than focusing independently on each frame. Fig 11
shows an end-to-end trainable CNN (convolutional neural network)–RNN (recurrent neural network) network architecture
employed for the

Figure 11: A unified CNN-RNN network architecture for spatio-temporal analysis of the impact generated wavefield

Proc. of SPIE Vol. 11379 1137903-13

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Figure 12: Overview of the type of domain knowledge available and its integration into the machine learning (ML) pipeline for
accomplishing impact diagnosis

spatio-temporal analysis of the scattered wavefield. The model was trained on simulated wavefield data and care was
exercised to ensure that the training data was not significantly deviated from the data the model is expected to encounter in
the real world. This was assured by incorporating different noise levels and location biases in the training data and by testing
the model on simulated wavefields in response to real impact force profiles generated using an impact hammer. Overall,
this lies under the paradigm of letting DL models learn in a simulated environment before transferring their knowledge to
the real world.
Impact diagnosis clearly represents a case which is extremely data-intensive and where collecting sufficient real data is
hard, as such, an ideal candidate for incorporating prior domain knowledge into the ML pipeline. Fig 12 gives an overview
of the

Figure 13: Results for impact diagnosis including impact location and impact force reconstruction using the physics-informed
DL-based methodology

Proc. of SPIE Vol. 11379 1137903-14

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 type of domain knowledge available and its integration into the machine learning pipeline. A wealth of information available
about the dynamics of the wave propagation phenomenon is available in this case. In particular, Huygens’ principle, which
states that different time-frames of the wavefield are not independent of each other and an inherent temporal structure exists
between them, dictated the overall network architecture. This formed the basis for treating the wavefield frames/feature
vectors extracted from the frames by the CNN part of the network as a sequence with the order of the individual feature
vectors being meaningful/important. Also, the classical time-reversal (T-R) technique pioneered by Fink [95-97] which is
usually employed for source identification in many disciplines as an optimal focusing technique formed the basis for the
so-called reversed-time learning in the recurrent part of the network. Fig 13 shows some of the results for impact diagnosis
and the effect of incorporating domain knowledge into the model. As can be seen, reasonable improvement, especially in
impact force reconstruction accuracy is obtained by incorporating the concept of time-reversal with accuracies in the range
of (89-93)% being obtained for SNR values of 20 and 10. The results obtained for SNR value 5 are expectedly poor because
of the very extreme noise level.
So far, in Sections 2-4, the focus has been on augmenting the incomplete/insufficient training data by incorporating domain
knowledge into the model during training. While this is indispensable for realizing the goal of real-time health monitoring
in large/complex real-world structure, effort should also be made to develop more efficient data acquisition systems for
SHM. To this end, the next Section presents a non-contact sensing paradigm for full-field measurements for vision-based
SHM.

5. A Non-contact Full-field Sensing Paradigm for Vision-based SHM

Computer vision techniques for automated damage assessment have been engendering a lot of interest in recent times in
civil infrastructures, especially during post-disaster inspections, when the number of structures to be inspected is far beyond
the capability of available inspectors [98-103]. In the field of civil infrastructure assessment, safety-critical damage is
usually big enough to be visible to the naked eye, as such, static images captured using ordinary digital cameras suffice for
the purpose of automated damage assessment using state-of-the-art computer vision algorithms. For instance, a fusion CNN
was proposed by Dr. Li’s group [104-105] to successfully detect distributed cracks on steel girders of actual bridges.
However, for aerospace structures, the most critical type of damage is barely visible to the naked eye. In such (plate-like)
structures, hidden details of the structure, including information about the location of the damage, if present, and its
characteristics can be unearthed using ultrasonic guided wave based techniques, which entail investigating the scattered
guided waves in the structure using appropriate signal/image processing algorithms. These scattered waves can be sensed
using either contact (e.g., a network of piezoelectric (PZT) sensor arrays) [106] or fully non-contact means (e.g., air-coupled
transducers, laser Doppler vibrometer (LDV)) [107-112]. Because of the multitude of advantages associated with non-
contact full-field sensing, the scanning LDV has attracted a lot of interest in recent times, however, it is limited to point-
by-point measurement/sensing (i.e., one point at a time) [113].
In order to overcome these limitations, a sensing paradigm capable of taking all the measurements simultaneously is
desirable. Digital image correlation (DIC), which is a non-interferometric optical technique, extracts surface deformations
by comparing digital images of the structure’s surface captured before and after deformation [114-118]. Advances in
computers and digital imaging technology have already enabled 2-D/3-D DIC methods to extend from static or quasi-static
to dynamic applications, such as low-frequency vibration (in the range of kHz) and full-field modal analysis for
characterizing the behavior of structures [119-121]. In order to expand DIC applications into the ultrasonic range (> 20
kHz), for example, in the area of ultrasonic guided-wave based damage imaging, image acquisition systems with extremely
high spatial/temporal resolution are required.
High-speed digital cameras, even though not there yet, provide the most viable means for accomplishing this. Not only do
they enable simultaneous data acquisition at all the points in the region-of-interest (ROI) but the reconstructed wavefield
using the sensed data provides the most spatially continuous information about the wave propagation phenomena as each
pixel is considered to effectively act as a sensor. The requirement of extremely high spatial/temporal resolution, however,
continues to remain a bottleneck in the adoption of this technology for ultrasonic guided wavefield reconstruction. However,
the problem is purely related to the processing speed and will improve with the development of computer technology.

Proc. of SPIE Vol. 11379 1137903-15

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 Figure 14: (a) Schematic of the experimental setup for non-contact full-field sensing (b) Damage image of a BVID generated from
the captured guided wavefield using modified wavenumber index (WI) imaging condition

Furthermore, as IC technology rapidly advances following Moore’s law, new high-speed cameras promise to have high
potential for capturing the propagating guided wave on the surface of the structures.
Fig. 14 shows the schematic of a proof-of-concept for an integrated high-speed camera system for non-contact full-field
sensing of the scattered wavefield in structures [122] and the damage image generated from the captured wavefield using
modified wavenumber index (WI) imaging condition [123]. It should be noted that since only a single high-speed camera
was used, it was possible to capture only the in-plane scattered wavefield. In order to circumvent the current hardware
limitations associated with using a single high-speed camera for the acquisition of ultrasonic guided in-plane wavefields,
three strategies were adopted. First, for improving the spatial/temporal resolution, sample interleaving and image stitching
techniques were used to get an effective data acquisition rate that was nearly 250 times that of the original rate specified by
the camera. Second, the concept was demonstrated on a thin, flat, low-modulus, high-density polyethylene (HDPE) plate,
which is a flexible polymer sheet with less than one percent of the stiffness of regular carbon fiber reinforced polymers
(CFRPs). The low-modulus and high-density plate enables large deformations and low wave speeds, which make it possible
to successfully detect the in-plane wavefields using the affordable in-house high-speed camera (Photron Fastcam Mini
AX200). Third, a large PZT actuator was used in conjunction with a power amplifier to generate sufficient energy to excite
measurable guided waves in the plate, and the excitation frequency was carefully chosen to be a continuous harmonic wave
such that not only the fundamental A0 and S0 modes exist, but a strong fundamental SH0 wave mode also persists (due to
multiple reflections from the damage and plate boundaries).
While currently the concept could only be demonstrated on the HDPE sheet due to hardware limitations associated with
using a single high-speed camera, the study shows significant initial promise for the adoption of this technology for non-
contact full-field measurements with the ultimate goal of visualizing hidden damage such as barely visible impact damage
(BVID) in complex flat/curved composite structures. In order to continue development of this concept past this successful
feasibility study and begin optimizing it for practical applications, the following tasks are planned. (1) Multiple high-speed

Proc. of SPIE Vol. 11379 1137903-16

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 cameras will be employed to enable the capture of three-dimensional displacement/strain scattered wavefields with larger
fields of view. It is envisioned that out-of-plane anti-symmetric wave modes (such as the fundamental A0 wave mode) will
provide significantly higher SNR values to yield better damage images. (2) Physics-based image stitching techniques will
be explored in order to handle more-complicated geometries and damage scenarios. (3) For using the captured full
wavefields in conjunction with physics-informed learning for realizing the eventual goal of developing an end-to-end non-
contact vision based damage detection system in SHM, other transfer learning models like the one developed in Section 4.2
will be developed.

6. Summary, Conclusions and Future Prospects

When the structures being monitored (either continuously or intermittently) are significantly large/complex and/or when
the variations in environmental/operational conditions cannot be neglected, a physics-based approach for assessing the
structural health becomes highly unreliable. In recent times, the number of parameters being monitored in-situ in real-world
structures has been steadily increasing through the use of miniaturized wireless sensor networks and that has led to an
abundance of historical data for these structures. The heterogeneous nature of this data (due to multiple sensing modalities)
and the emphasis on developing long-term SHM systems has engendered a lot of interest in a data-driven approach for
SHM. However, for these real-world structures, it is highly impractical to collect sufficient data (corresponding to different
damage scenarios) to train a robust and generalizable model for automated damage detection and characterization.
Consequently, an optimal approach to SHM must encompass, on one hand, some means of augmenting the collected sensor
data in order to compensate for the lack of completeness in the data and on the other hand comprehensive sensing
mechanisms with a framework for subsequent data fusion and processing of the large volumes of collected data.
In this paper, these aspects of such an optimal SHM paradigm were addressed individually through: (1) The use of physics-
informed machine learning for the analysis of the measured sensor data. (2) The use of high-speed digital cameras for
capturing ultrasonic guided wavefields in structures as a means of providing the most spatially continuous information
about the propagating ultrasonic guided waves within the structure. Owing to the attention it has been engendering in recent
times, physics-informed machine learning was discussed in some detail with the eventual goal of SHM in mind. By
demonstrating the applicability of ANNs for solving forward and inverse problems involving PDEs, initial promise was
demonstrated towards solving more complicated inverse problems for the task of automated damage detection in SHM. It
was shown how the overfitting/generalization issue generally associated with the insufficient nature of the training data can
be mitigated by incorporating physical laws/domain knowledge to guide the back-propagation during the training process.
Other ways of incorporating domain knowledge into the ML/DL pipeline were also discussed through the case studies
related to different aspects of SHM i.e., enhanced visual inspection and impact diagnosis.
Internet of Things (IoT) has recently attracted great attention due to its promising potential and capacity to be integrated
into complex systems [124, 125]. With rapid advances in information and sensing technologies such as wireless
communication, cloud computing, and wireless sensor networks (WSNs), IoT is emerging as an enabling technology for
the development of low cost, efficient, reliable, and scalable SHM systems. Recent advances in deep learning (both
supervised and unsupervised) and breakthroughs in the integration of system physics/domain knowledge into the ML/DL
models have had a catalytic effect on the adoption of a physics-informed data-driven approach to learning in many
application domains. It is envisioned that as robust non-contact full-field sensing mechanisms are developed and more
success in physics-informed learning is achieved, especially for solving complex inverse problems in SHM, the goal of in-
situ real-time vision based health monitoring can be achieved in near future.

Acknowledgement

One of the authors, Mr. Sakib A. Zargar would like to express their sincere thanks to Dr. Jeremy Yagle of NASA Langley
Research Center for the support through National Institute of Aerospace (NIA).

Proc. of SPIE Vol. 11379 1137903-17

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 References

[1] Giurgiutiu, V. (2007). Structural health monitoring: with piezoelectric wafer active sensors. Elsevier.
[2] Yuan, F. G. (Ed.). (2016). Structural health monitoring (SHM) in aerospace structures. Woodhead Publishing.
[3] Cawley, P. (2018). Structural health monitoring: Closing the gap between research and industrial deployment. Structural
Health Monitoring, 17(5), 1225-1244.
[4] Doebling, S. W., Farrar, C. R., & Prime, M. B. (1998). A summary review of vibration-based damage identification
methods. Shock and vibration digest, 30(2), 91-105.
[5] Farrar, C. R., Doebling, S. W., & Nix, D. A. (2001). Vibration–based structural damage identification. Philosophical
Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, 359(1778), 131-
149.
[6] Alvandi, A., & Cremona, C. (2006). Assessment of vibration-based damage identification techniques. Journal of sound
and vibration, 292(1-2), 179-202.
[7] Montalvao, D., Maia, N. M. M., & Ribeiro, A. M. R. (2006). A review of vibration-based structural health monitoring
with special emphasis on composite materials. Shock and vibration digest, 38(4), 295-324.
[8] Fan, W., & Qiao, P. (2011). Vibration-based damage identification methods: a review and comparative study. Structural
health monitoring, 10(1), 83-111.
[9] Raghavan, A., & Cesnik C. E. S. (2007). Review of guided-wave structural health monitoring. The Shock and Vibration
Digest, 39(2), 91-114.
[10] Su, Z., & Ye, L. (2009). Identification of damage using Lamb waves: from fundamentals to applications (Vol. 48).
Springer Science & Business Media.
[11] Mitra, M., & Gopalakrishnan, S. (2016). Guided wave based structural health monitoring: A review. Smart Materials
and Structures, 25(5), 053001.
[12] Wu, X., Ghaboussi, J., & Garrett Jr, J. H. (1992). Use of neural networks in detection of structural damage. Computers
& structures, 42(4), 649-659.
[13] Worden, K., & Manson, G. (2007). The application of machine learning to structural health monitoring. Philosophical
Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 365(1851), 515-537.
[14] Farrar, C. R., & Worden, K. (2012). Structural health monitoring: a machine learning perspective. John Wiley & Sons.
[15] Rytter, A. (1993). Vibrational based inspection of civil engineering structures (Doctoral dissertation, Dept. of Building
Technology and Structural Engineering, Aalborg University).
[16] Allemang, R. J., & Brown, D. L. (1982). A correlation coefficient for modal vector analysis. In Proceedings of the 1st
international modal analysis conference (Vol. 1, pp. 110-116). Orlando, FL: SEM.
[17] Allemang, R. J. (2003). The modal assurance criterion–twenty years of use and abuse. Sound and vibration, 37(8), 14-
23.
[18] Kim, J. T., Ryu, Y. S., Cho, H. M., & Stubbs, N. (2003). Damage identification in beam-type structures: frequency-
based method vs mode-shape-based method. Engineering structures, 25(1), 57-67.
[19] Shi, Z., Law, S. S., & Zhang, L. (2000). Structural damage detection from modal strain energy change. Journal of
engineering mechanics, 126(12), 1216-1223.
[20] Pandey, A. K., Biswas, M., & Samman, M. M. (1991). Damage detection from changes in curvature mode
shapes. Journal of sound and vibration, 145(2), 321-332.
[21] Wahab, M. A., & De Roeck, G. (1999). Damage detection in bridges using modal curvatures: application to a real
damage scenario. Journal of Sound and vibration, 226(2), 217-235.
[22] Pandey, A. K., & Biswas, M. (1994). Damage detection in structures using changes in flexibility. Journal of sound
and vibration, 169(1), 3-17.
[23] Pandey, A. K., & Biswas, M. (1995). Damage diagnosis of truss structures by estimation of flexibility change. Modal
Analysis-the International Journal of Analytical and Experimental Modal Analysis, 10(2), 104-117.
[24] Jaishi, B., & Ren, W. X. (2006). Damage detection by finite element model updating using modal flexibility
residual. Journal of sound and vibration, 290(1-2), 369-387.
[25] Bernal, D. (2002). Load vectors for damage localization. Journal of Engineering Mechanics, 128(1), 7-14.

Proc. of SPIE Vol. 11379 1137903-18

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 [26] Staszewski, W. J., Worden, K., & Tomlinson, G. R. (1997). Time–frequency analysis in gearbox fault detection using
the Wigner–Ville distribution and pattern recognition. Mechanical systems and signal processing, 11(5), 673-692.
[27] Sohn, H., Park, G., Wait, J. R., Limback, N. P., & Farrar, C. R. (2003). Wavelet-based active sensing for delamination
detection in composite structures. Smart Materials and structures, 13(1), 153.
[28] Ovanesova, A. V., & Suarez, L. E. (2004). Applications of wavelet transforms to damage detection in frame
structures. Engineering structures, 26(1), 39-49.
[29] Kim, H., & Melhem. H. (2004). Damage detection of structures by wavelet analysis, Engineering structures, 26(3),
347-362.
[30] Taha, M. R., Noureldin, A., Lucero, J. L., & Baca, T. J. (2006). Wavelet transform for structural health monitoring: a
compendium of uses and features. Structural health monitoring, 5(3), 267-295.
[31] Yang, J. N., Lei, Y., Lin, S., & Huang, N. (2004). Hilbert-Huang based approach for structural damage
detection. Journal of engineering mechanics, 130(1), 85-95.
[32] Hay, T. R., Royer, R. L., Gao, H., Zhao, X., & Rose, J. L. (2006). A comparison of embedded sensor Lamb wave
ultrasonic tomography approaches for material loss detection. Smart materials and structures, 15(4), 946.
[33] He, J., & Yuan, F. G. (2017). Lamb-wave-based two-dimensional areal scan damage imaging using reverse-time
migration with a normalized zero-lag cross-correlation imaging condition. Structural Health Monitoring, 16(4), 444-457.
[34] Kudva, J. N., Munir, N., & Tan, P. W. (1992). Damage detection in smart structures using neural networks and finite-
element analyses. Smart Materials and Structures, 1(2), 108.
[35] Teboub, Y., & Hajela, P. (1992, September). A neural network based damage analysis of smart composite beams.
In 4th Symposium on Multidisciplinary Analysis and Optimization (p. 4685).
[36] Mehrjoo, M., Khaji, N., Moharrami, H., & Bahreininejad, A. (2008). Damage detection of truss bridge joints using
Artificial Neural Networks. Expert Systems with Applications, 35(3), 1122-1131.
[37] Liu, Y. Y., Ju, Y. F., Duan, C. D., & Zhao, X. F. (2011). Structure damage diagnosis using neural network and feature
fusion. Engineering applications of artificial intelligence, 24(1), 87-92.
[38] Hinton, G. E., Osindero, S., & Teh, Y. W. (2006). A fast learning algorithm for deep belief nets. Neural computation,
18(7), 1527-1554.
[39] LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.
[40] Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural networks, 61, 85-117.
[41] Sze, V., Chen, Y. H., Yang, T. Y., & Emer, J. S. (2017). Efficient processing of deep neural networks: A tutorial and
survey. Proceedings of the IEEE, 105, (12), 2295-2329.
[42] Diez, A., Khoa, N. L. D., Alamdari, M. M., Wang, Y., Chen, F., & Runcie, P. (2016). A clustering approach for
structural health monitoring on bridges. Journal of Civil Structural Health Monitoring, 6(3), 429-445.
[43] Alves, V., Cury, A., Roitman, N., Magluta, C., & Cremona, C. (2015). Novelty detection for SHM using raw
acceleration measurements. Structural Control and Health Monitoring, 22(9), 1193-1207.
[44] Toivola, J., Prada, M. A., & Hollmén, J. (2010). Novelty detection in projected spaces for structural health monitoring.
In International Symposium on Intelligent Data Analysis (pp. 208-219). Springer, Berlin, Heidelberg.
[45] Von Rueden, L., Mayer, S., Garcke, J., Bauckhage, C., & Schuecker, J. (2019). Informed machine learning–towards a
taxonomy of explicit integration of knowledge into machine learning. Learning, 18(19), 20.
[46] Kramer, M. A. (1991). Nonlinear principal component analysis using autoassociative neural networks. AIChE journal,
37(2), 233-243.
[47] Mavrovouniotis, M. L., & Chang, S. (1992). Hierarchical neural networks. Computers & chemical engineering, 16(4),
347-369.
[48] Tan, S., & Mayrovouniotis, M. L. (1995). Reducing data dimensionality through optimizing neural network
inputs. AIChE Journal, 41(6), 1471-1480.
[49] Yun, C. B., & Bahng, E. Y. (2000). Substructural identification using neural networks. Computers & Structures, 77(1),
41-52.
[50] Marwala, T. (2000). Damage identification using committee of neural networks. Journal of Engineering
Mechanics, 126(1), 43-50.
[51] Chen, S. L., & Jen, Y. W. (2000). Data fusion neural network for tool condition monitoring in CNC milling
machining. International journal of machine tools and manufacture, 40(3), 381-400.

Proc. of SPIE Vol. 11379 1137903-19

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 [52] Zang, C., & Imregun, M. (2001). Structural damage detection using artificial neural networks and measured FRF data
reduced via principal component projection. Journal of sound and vibration, 242(5), 813-827.
[53] Ghosh, N., Ravi, Y. B., Patra, A., Mukhopadhyay, S., Paul, S., Mohanty, A. R., & Chattopadhyay, A. B. (2007).
Estimation of tool wear during CNC milling using neural network-based sensor fusion. Mechanical Systems and Signal
Processing, 21(1), 466-479.
[54] Kramer, M. A. (1992). Autoassociative neural networks. Computers & chemical engineering, 16(4), 313-328.
[55] Zhang, J., Walter, G. G., Miao, Y., & Lee, W. N. W. (1995). Wavelet neural networks for function learning. IEEE
transactions on Signal Processing, 43(6), 1485-1497.
[56] Wang, F., & Zhang, Q. J. (1997). Knowledge-based neural models for microwave design. IEEE Transactions on
Microwave Theory and Techniques, 45(12), 2333-2343.
[57] Jiang, X., & Adeli, H. (2007). Pseudospectra, MUSIC, and dynamic wavelet neural network for damage detection of
highrise buildings. International Journal for Numerical Methods in Engineering, 71(5), 606-629.
[58] Psichogios, D. C., & Ungar, L. H. (1992). A hybrid neural network‐first principles approach to process modeling.
AIChE Journal, 38(10), 1499-1511.
[59] Kramer, M. A., Thompson, M. L., & Bhagat, P. M. (1992). Embedding theoretical models in neural networks. In 1992
American Control Conference (pp. 475-479). IEEE.
[60] Thompson, M. L., & Kramer, M. A. (1994). Modeling chemical processes using prior knowledge and neural networks.
AIChE Journal, 40(8), 1328-1340.
[61] Jain, A. K., Mao, J., & Mohiuddin, K. M. (1996). Artificial neural networks: A tutorial. Computer, 29(3), 31-44.
[62] Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal
approximators. Neural networks, 2(5), 359-366.
[63] Dissanayake, M. W. M. G., & Phan‐Thien, N. (1994). Neural‐network‐based approximations for solving partial
differential equations. communications in Numerical Methods in Engineering, 10(3), 195-201.
[64] Lagaris, I. E., Likas, A., & Fotiadis, D. I. (1998). Artificial neural networks for solving ordinary and partial differential
equations. IEEE transactions on neural networks, 9(5), 987-1000.
[65] Lagaris, I. E., Likas, A. C., & Papageorgiou, D. G. (2000). Neural-network methods for boundary value problems with
irregular boundaries. IEEE Transactions on Neural Networks, 11(5), 1041-1049.
[66] He, S., Reif, K., & Unbehauen, R. (2000). Multilayer neural networks for solving a class of partial differential
equations. Neural networks, 13(3), 385-396.
[67] Aarts, L. P., & Van Der Veer, P. (2001). Neural network method for solving partial differential equations. Neural
Processing Letters, 14(3), 261-271.
[68] Alli, H., Uçar, A., & Demir, Y. (2003). The solutions of vibration control problems using artificial neural
networks. Journal of the Franklin Institute, 340(5), 307-325.
[69] Malek, A., & Beidokhti, R. S. (2006). Numerical solution for high order differential equations using a hybrid neural
network—optimization method. Applied Mathematics and Computation, 183(1), 260-271.
[70] Beidokhti, R. S., & Malek, A. (2009). Solving initial-boundary value problems for systems of partial differential
equations using neural networks and optimization techniques. Journal of the Franklin Institute, 346(9), 898-913.
[71] Shirvany, Y., Hayati, M., & Moradian, R. (2009). Multilayer perceptron neural networks with novel unsupervised
training method for numerical solution of the partial differential equations. Applied Soft Computing, 9(1), 20-29.
[72] Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2017). Physics informed deep learning (part i): Data-driven solutions
of nonlinear partial differential equations. arXiv preprint arXiv:1711.10561.
[73] Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2017). Physics informed deep learning (part II): Data-driven discovery
of nonlinear partial differential equations. arXiv. arXiv preprint arXiv:1711.10561.
[74] Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks: A deep learning framework
for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational
Physics, 378, 686-707.
[75] Sirignano, J., & Spiliopoulos, K. (2018). DGM: A deep learning algorithm for solving partial differential
equations. Journal of Computational Physics, 375, 1339-1364.
[76] Anitescu, C., Atroshchenko, E., Alajlan, N., & Rabczuk, T. (2019). Artificial neural network methods for the solution
of second order boundary value problems. Computers, Materials & Continua, 59(1), 345-359.

Proc. of SPIE Vol. 11379 1137903-20

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 [77] Berg, J., & Nyström, K. (2018). A unified deep artificial neural network approach to partial differential equations in
complex geometries. Neurocomputing, 317, 28-41.
[78] Meng, X., & Karniadakis, G. E. (2020). A composite neural network that learns from multi-fidelity data: Application
to function approximation and inverse PDE problems. Journal of Computational Physics, 401, 109020.
[79] Zhu, Y., Zabaras, N., Koutsourelakis, P. S., & Perdikaris, P. (2019). Physics-constrained deep learning for high-
dimensional surrogate modeling and uncertainty quantification without labeled data. Journal of Computational
Physics, 394, 56-81.
[80] Lutter, M., Ritter, C., & Peters, J. (2019). Deep Lagrangian networks: Using physics as model prior for deep
learning. arXiv preprint arXiv:1907.04490.
[81] Federal Aviation Administration, Visual inspection for aircraft, U.S. Department of Transportation, Advisory circular
[online] https://www.faa.gov/documentLibrary/media/Advisory_Circular/43-204.pdf.
[82] Drury, C. G., & Watson, J. (2002). Good practices in visual inspection. Human factors in aviation maintenance-
phase nine, progress report, FAA/Human Factors in Aviation Maintenance.@ URL: http://hfskyway. faa. gov.
[83] Swain, A. D., & Guttmann, H. E. (1983). Handbook of human-reliability analysis with emphasis on nuclear power
plant applications. Final report (No. NUREG/CR--1278). Sandia National Labs.
[84] See, J. E. (2015). Visual inspection reliability for precision manufactured parts. Human factors, 57(8), 1427-1442.
[85] See, J. E. (2012). Visual inspection: a review of the literature. Sandia National Laboratories, Albuquerque, New
Mexico.
[86] Azuma, R. T. (1997). A survey of augmented reality. Presence: Teleoperators & Virtual Environments, 6(4), 355-385.
[87] Wang, S., Zargar, S. A., Xu, C. & Yuan, F. G. (2019). An Efficient Augmented Reality (AR) System for Enhanced
Visual Inspection. The 12th International Workshop on Structural Health Monitoring, Stanford, California, USA,
September 10-12, 2019. Lancaster, PA, USA:DEStech Publications, Inc., Electronic product (pp. 1575-1582)
[88] Zargar, S. A. & Yuan, F. G. (2020). Impact Diagnosis in Stiffened Structural Panels Using a Deep Learning Approach.
Structural Health Monitoring Journal, Accepted.
[89] Inoue, H., Harrigan, J. J., & Reid, S. R. (2001). Review of inverse analysis for indirect measurement of impact
force. Applied Mechanics Reviews, 54(6), 503-524.
[90] Chen, C., Li, Y., & Yuan, F. G. (2012). Impact source identification in finite isotropic plates using a time-reversal
method: experimental study. Smart Materials and Structures, 21(10), 105025.
[91] Worden, K., & Staszewski, W. J. (2000). Impact location and quantification on a composite panel using neural
networks and a genetic algorithm. Strain, 36(2), 61-68.
[92] Haywood, J., Coverley, P. T., Staszewski, W. J., & Worden, K. (2004). An automatic impact monitor for a composite
panel employing smart sensor technology. Smart Materials and Structures, 14(1), 265.
[93] Ghajari, M., Sharif-Khodaei, Z., Aliabadi, M. H., & Apicella, A. (2013). Identification of impact force for smart
composite stiffened panels. Smart Materials and Structures, 22(8), 085014.
[94] Seno, A. H., & Aliabadi, M. H. (2019). Impact localisation in composite plates of different stiffness impactors under
simulated environmental and operational conditions. Sensors, 19(17), 3659.
[95] Fink, M. (1992). Time reversal of ultrasonic fields. I. Basic principles. IEEE transactions on ultrasonics, ferroelectrics,
and frequency control, 39(5), 555-566.
[96] Wu, F., Thomas, J. L., & Fink, M. (1992). Time reversal of ultrasonic fields. Il. Experimental results. IEEE transactions
on ultrasonics, ferroelectrics, and frequency control, 39(5), 567-578.
[97] Cassereau, D., & Fink, M. (1992). Time-reversal of ultrasonic fields. III. Theory of the closed time-reversal
cavity. IEEE transactions on ultrasonics, ferroelectrics, and frequency control, 39(5), 579-592.
[98] Yeum, C. M., & Dyke, S. J. (2015). Vision‐based automated crack detection for bridge inspection. Computer‐Aided
Civil and Infrastructure Engineering, 30(10), 759-770.
[99] Feng, D., & Feng, M. Q. (2017). Experimental validation of cost-effective vision-based structural health monitoring.
Mechanical Systems and Signal Processing, 88, 199-211.
[100] Hoskere, V., Narazaki, Y., Hoang, T., & Spencer Jr, B. (2018). Vision-based structural inspection using multiscale
deep convolutional neural networks. arXiv preprint arXiv:1805.01055.
[101] Narazaki, Y., Hoskere, V., Hoang, T. A., & Spencer Jr, B. F. (2018). Automated vision-based bridge component
extraction using multiscale convolutional neural networks. arXiv preprint arXiv:1805.06042.

Proc. of SPIE Vol. 11379 1137903-21

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 [102] Spencer Jr, B. F., Hoskere, V., & Narazaki, Y. (2019). Advances in computer vision-based civil infrastructure
inspection and monitoring. Engineering.
[103] Bao, Y., Chen, Z., Wei, S., Xu, Y., Tang, Z., & Li, H. (2019). The State of the Art of Data Science and Engineering
in Structural Health Monitoring. Engineering, 5(2), 234-242.
[104] Xu, Y., Bao, Y., Chen, J., Zuo, W., & Li, H. (2019). Surface fatigue crack identification in steel box girder of bridges
by a deep fusion convolutional neural network based on consumer-grade camera images. Structural Health Monitoring,
18(3), 653-674.
[105] Wei, S., Bao, Y., & Li, H. (2020). Optimal policy for structure maintenance: A deep reinforcement learning
framework. Structural Safety, 83, 101906.
[106] V. Giurgiutiu, "Tuned Lamb Wave Excitation and Detection with Piezoelectric Wafer Active Sensors for Structural
Health Monitoring," Journal of Intelligent Material Systems and Structures, Vol. 16, No. 4, pp. 291-305, 2005.
[107] Z. Tian, L. Yu, and C. Leckey, "Delamination Detection and Quantification on Laminated Composite Structures with
Lamb Waves and Wavenumber Analysis," Journal of Intelligent Material Systems and Structures, Vol. 26, No. 13, pp.
1723-1738, 2015.
[108] B. Park, Y. K. An, and H. Sohn. "Visualization of Hidden Delamination and Debonding in Composites through
Noncontact Laser Ultrasonic Scanning," Composites Science and Technology, Vol. 100, pp. 10-18, 2014.
[109] J. He and F. G. Yuan. "Lamb Wave-based Subwavelength Damage Imaging using the DORT-MUSIC Technique in
Metallic Plates," Structural Health Monitoring, Vol. 15, No. 1, pp. 65-80, 2016.
[110] J. He and F. G. Yuan. "Damage Identification for Composite Structures using a Cross-correlation Reverse-time
Migration Technique," Structural Health Monitoring, Vol. 14, No. 6, pp. 558-570, 2015.
[111] M. S. Harb and F. G. Yuan, "Damage Imaging using Non-contact Air-coupled Transducer/Laser Doppler Vibrometer
System," Structural Health Monitoring, Vol. 15, No. 2, pp. 193-203, 2016.
[112] E. B. Flynn, S. Y. Chong, G. J. Jarmer, and J. R. Lee, "Structural Imaging through Local Wavenumber Estimation
of Guided Waves," Ndt & E International, Vol. 59, pp. 1-10, 2013.
[113] Reu, P. L., Rohe, D. P., & Jacobs, L. D. (2017). Comparison of DIC and LDV for practical vibration and modal
measurements. Mechanical Systems and Signal Processing, 86, 2-16.
[114] Chu, T. C., Ranson, W. F., & Sutton, M. A. (1985). Applications of digital-image-correlation techniques to
experimental mechanics. Experimental mechanics, 25(3), 232-244.
[115] Luo, P. F., Chao, Y. J., Sutton, M. A., & Peters, W. H. (1993). Accurate measurement of three-dimensional
deformations in deformable and rigid bodies using computer vision. Experimental mechanics, 33(2), 123-132.
[116] Luo, P. F., Chao, Y. J., & Sutton, M. A. (1994). Application of stereo vision to three-dimensional deformation
analyses in fracture experiments. Optical Engineering, 33(3), 981-991.
[117] Sutton, M. A., Matta, F., Rizos, D., Ghorbani, R., Rajan, S., Mollenhauer, D. H., ... & Lasprilla, A. O. (2017). Recent
progress in digital image correlation: background and developments since the 2013 WM Murray Lecture. Experimental
Mechanics, 57(1), 1-30.
[118] Pan, B. (2018). Digital image correlation for surface deformation measurement: historical developments, recent
advances and future goals. Measurement Science and Technology, 29(8), 082001.
[119] Helfrick, M. N., Niezrecki, C., Avitabile, P., & Schmidt, T. (2011). 3D digital image correlation methods for full-
field vibration measurement. Mechanical systems and signal processing, 25(3), 917-927.
[120] Trebuňa, F., & Hagara, M. (2014). Experimental modal analysis performed by high-speed digital image correlation
system. Measurement, 50, 78-85.
[121] Beberniss, T. J., & Ehrhardt, D. A. (2017). High-speed 3D digital image correlation vibration measurement: Recent
advancements and noted limitations. Mechanical Systems and Signal Processing, 86, 35-48.
[122] Chang, H. Y. (2019). Damage Visualization of Scattered Ultrasonic Wavefield via Integrated High-speed Camera
System. Ph. D. Thesis, Department of Mechanical and Aerospace Engineering, North Carolina State University.
[123] H. Y. Chang and F. G. Yuan. "Damage Imaging in a Stiffened Curved Composite Sandwich Panel with Wavenumber
Index via Riesz Transform," Structural Health Monitoring, 1475921719858432, 2019.
[124] Tokognon, C. A., Gao, B., Tian, G. Y., & Yan, Y. (2017). Structural health monitoring framework based on Internet
of Things: A survey. IEEE Internet of Things Journal, 4(3), 619-635.

Proc. of SPIE Vol. 11379 1137903-22

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use
 [125] Abdelgawad, A., & Yelamarthi, K. (2017). Internet of things (IoT) platform for structure health monitoring. Wireless
Communications and Mobile Computing, 2017.

Proc. of SPIE Vol. 11379 1137903-23

Downloaded From: https://www.spiedigitallibrary.org/conference-proceedings-of-spie on 02 Jan 2024
Terms of Use: https://www.spiedigitallibrary.org/terms-of-use