biosensors

Article
Mental Stress Assessment Using Ultra Short Term HRV
Analysis Based on Non-Linear Method
Seungjae Lee 1 , Ho Bin Hwang 1 , Seongryul Park 1 , Sanghag Kim 2 , Jung Hee Ha 3 , Yoojin Jang 3 ,
Sejin Hwang 4 , Hoon-Ki Park 5 , Jongshill Lee 1, * and In Young Kim 1, *

1 Department of Biomedical Engineering, Hanyang University, Seoul 04763, Korea;

seungjaelee@hanyang.ac.kr (S.L.); hobin0215@hanyang.ac.kr (H.B.H.); seongryul@hanyang.ac.kr (S.P.)
2 Department of Sociology, Hanyang University, Seoul 04763, Korea; sanghag@hanyang.ac.kr
3 Graduate School of Counseling Psychology, Hanyang University, Seoul 04763, Korea;
hajung366@hanyang.ac.kr (J.H.H.); yoojinjang@hanyang.ac.kr (Y.J.)
4 Department Anatomy and Cell Biology, College of Medicine, Hanyang University, Seoul 04763, Korea;
hwangsj@hanyang.ac.kr
5 Department of Family Medicine, Hanyang University, Seoul 04763, Korea; hoonkp@hanyang.ac.kr
* Correspondence: netlee@hanyang.ac.kr (J.L.); iykim@hanyang.ac.kr (I.Y.K.)

Abstract: Mental stress is on the rise as one of the major health problems in modern society. It is
important to detect and manage mental stress to prevent various diseases caused by stress and to
maintain a healthy life. The purpose of this paper is to present new heart rate variability (HRV)
features based on empirical mode decomposition and to detect acute mental stress through short-term
HRV (5 min) and ultra-short-term HRV (under 5 min) analysis. HRV signals were acquired from
74 young police officers using acute stressors, including the Trier Social Stress Test and horror movie
viewing, and a total of 26 features, including the proposed IMF energy features and general HRV
features, were extracted. A support vector machine (SVM) classification model is used to classify
the stress and non-stress states through leave-one-subject-out cross-validation. The classification
accuracies of short-term HRV and ultra-short-term HRV analysis are 86.5% and 90.5%, respectively.
Citation: Lee, S.; Hwang, H.B.; Park, In the results of ultra-short-term HRV analysis using various time lengths, we suggest the optimal
S.; Kim, S.; Ha, J.H.; Jang, Y.; Hwang, duration to detect mental stress, which can be applied to wearable devices or healthcare systems.
S.; Park, H.-K.; Lee, J.; Kim, I.Y.
Mental Stress Assessment Using
Keywords: heart rate variability (HRV); ultra-short-term HRV analysis; empirical mode decomposition
Ultra Short Term HRV Analysis
(EMD); non-linear features; stress assessment
Based on Non-Linear Method.
Biosensors 2022, 12, 465. https://
doi.org/10.3390/bios12070465

Received: 6 May 2022 1. Introduction

Accepted: 24 June 2022
As the number of people with health problems due to stress increases every year,
Published: 27 June 2022
mental stress is emerging as one of the major human health problems in modern society.
Publisher’s Note: MDPI stays neutral Mental stress not only harms mental and physical health, but also causes various diseases,
with regard to jurisdictional claims in such as diabetes, cardiovascular and respiratory diseases, depression, and cancers [1–4].
published maps and institutional affil- Mental stress is largely classified into chronic stress and acute stress. When one faces an
iations. acute stressor, the human body triggers a “fight-or-flight” response, a survival mechanism
triggered by external stimuli to maintain homeostasis. This response activates the sym-
pathetic nervous system within the body’s autonomic nervous system (ANS) and causes
changes in the body through the endocrine system [5]. If an acute stressor continuously
Copyright: © 2022 by the authors.
affects the body, it can be transformed into chronic stress, in which the ANS becomes
Licensee MDPI, Basel, Switzerland.
unbalanced. Therefore, it is necessary to quantitatively measure and manage acute stress in
This article is an open access article
daily life to remain healthy.
distributed under the terms and
New methods for measuring stress are required in various fields. First, questionnaire
conditions of the Creative Commons
Attribution (CC BY) license (https://
methods [6–8] are often used in the medical field, but such methods can be considered
creativecommons.org/licenses/by/
a subjective indicator of each individual. Another stress measurement method utilizes
4.0/).
biomarkers, such as salivary alpha-amylase and cortisol [9,10]. Although the biomarker

Biosensors 2022, 12, 465. https://doi.org/10.3390/bios12070465 https://www.mdpi.com/journal/biosensors

Biosensors 2022, 12, 465 2 of 16

method is an objective stress measurement method, it has disadvantages in that continuous

stress monitoring is impossible, and measurement is inconvenient. Stress measurement
using physiological signals can compensate for the shortcomings of other methods and
has been actively studied in recent years to detect the physiological responses that change
due to the presence of a stressor. Stress measurement studies are mainly conducted on
physiological signals, such as ECG [11,12], GSR [13,14], and EEG [15,16], and some studies
have described the combined use of several physiological signals [17,18].
Well-described methods of inducing acute stress in humans include the Stroop color-
word test [19], the Trier Social Stressor Test (TSST) [20], mental arithmetic tasks [21], public
speaking [22], Montreal Imaging Stress Task [23], and horror movie viewing [24]. According
to a review of acute stressors [25], the most effective method for inducing stress among
the various acute mental stressors is the TSST including public speaking and cognitive
tasks, and there are TSST studies that conducted stress classification based on a single
or multiple physiological signals. The research to classify stress and non-stressed states
based on a single physiological signal is as follows. A study was conducted using HRV
signals and human stress assessments, resulting in an accuracy of 84.38% [11]. In a study
using EEG signals, features including absolute power, relative power, coherence, phase
lag, and amplitude asymmetry were extracted for each frequency band using a 4-channel
EEG wearable device. When non-stress and stress groups were classified, the highest
reported accuracy was 98.76% [15]. Another stress classification study using GSR signals
obtained an average classification accuracy of 94.62% based on the cvxEDA method, which
is a rigorous and robust model [14]. Most previous studies were conducted using a
single physiological signal, such as EEG or GSR; have a small number of subjects or data
imbalance; and their application might not be suitable for real-life scenarios. In studies
considering multiple physiological signals, stress classification results using EEG, GSR,
and PPG achieved 79% accuracy; classification accuracy of 94% was achieved when the
aforementioned measurements were fused with sociometric sensors [17]. Another study
using EEG, GSR, and PPG achieved accuracies of 87.5% and 96.25% when EEG only and all
three physiological signals were considered, respectively [18]. Although the performance
of the previous stress classification studies has higher accuracy when using multiple signals
rather than a single signal, it would be difficult to assess mental stress using multiple
physiological signals in real-world situations.
HRV, which is used as a marker related to the ANS, is mainly used in studies of
stress assessment. Much of the mental stress assessment research analyzes HRV signals
recorded at 5 min intervals, defined as short-term HRV analysis [26]. As the demand
for real-time stress monitoring increases with the development of wearable devices [27],
recent healthcare systems require analysis of HRV signals shorter than 5 min, which is
defined as ultra-short-term HRV analysis [26]. Recently, stress classification studies have
been conducted using ultra-short-term HRV analysis [28–30]. These studies use HRV
features related to the ANS. Since short-term and ultra-short-term HRV analyses have
different lengths of data, and it is necessary to validate the ultra-short-term HRV features
as short-term features. Among stress-related HRV features, frequency domain features
are greatly affected by data length. In order to measure high-frequency (HF) and low-
frequency (LF) spectrum power, HRV data is required for a duration of at least 60 and
250 s, respectively [26]. Moreover, approximate entropy, one of the non-linear methods,
cannot be used for data representing less than 3 min [31]. Castaldo et al. [32] suggested
that ultra-short-term HRV features are surrogates of short-term features at different time
lengths during mental stress. Since the HRV signal is non-linear and non-stationary due
to the dynamics of the complex cardiac system, it is appropriate to use Empirical Mode
Decomposition (EMD), a time-frequency domain signal method with non-stationary and
non-linear characteristics. There are studies that have conducted stress analysis of HRV
signals using the EMD method [33–35]. There are also studies that analyzed HRV signal
using entropy methods with non-linear properties, including approximate entropy [36],
sample entropy [37], and permutation entropy [38].
 with non-linear properties, including approximate entropy [36], sample entropy [37], and
permutation entropy [38].
From this point of view, we analyzed intrinsic mode function (IMF) energy features
extracted from EMDs that can provide frequency domain information using less than 5
min 2022,
Biosensors of data.
12, 465 Furthermore, we evaluated the stress classification accuracy of short-term 3 of 16

and ultra-short-term HRV data using time domain features, including general HRV fea-
tures, entropy features, and proposed
From this EMD-based
point of view, we analyzedfeatures, andfunction
intrinsic mode suggested
(IMF) the
energyoptimal
features
time length for ultra-short-term
extracted from HRV data
EMDs that cancollection in acutely
provide frequency domainstressful situations.
information In5ad-
using less than min
of data. Furthermore, we evaluated the stress classification
dition, the classification method used a linear SVM classifier, and the appropriate evalua-accuracy of short-term and
ultra-short-term HRV data using time domain features, including general HRV features,
tion was performed entropywith the leave-one-subject-out cross-validation (LOSOCV) method,
features, and proposed EMD-based features, and suggested the optimal time length
which is most similarfortoultra-short-term
real-life application.
HRV data collection in acutely stressful situations. In addition, the
classification method used a linear SVM classifier, and the appropriate evaluation was
performed with the leave-one-subject-out cross-validation (LOSOCV) method, which is
2. Methods most similar to real-life application.
The stress classification method proposed in this study is shown in Figure 1. Subject
2. Methods
selection and the experimental protocol for HRV signal measurement are described in
The stress classification method proposed in this study is shown in Figure 1. Subject
Sections 2.1 and 2.2, respectively;
selection and thepreprocessing to remove
experimental protocol for HRV ectopic heartbeatsare
signal measurement is described
described in
in Section 2.3; the description of the
Sections 2.1 and proposed preprocessing
2.2, respectively; HRV features and general
to remove features
ectopic heartbeats is out-
is described
in Section 2.3; the description of the proposed HRV features and general features is outlined
lined in Section 2.4; and ranked feature processing to increase accuracy and a classification
in Section 2.4; and ranked feature processing to increase accuracy and a classification
method using leave-one-subject-out cross-validation
method using leave-one-subject-out are outlined
cross-validation in Section
are outlined 2.5.2.5.
in Section

Figure 1. Flowchart for Figure

the proposed stress
1. Flowchart for theclassification method. method.
proposed stress classification

2.1. Dataset
2.1. Dataset The stress database [39,40] provided by PhysioNet assumes that the subject is stressful
The stress database
in the[39,40] provided
given stress byand
situations, PhysioNet assumes
this approach may be that thein subject
limited is stress.
classifying stress-In
addition, the PhysioNet stress database is difficult to generalize because it consists of a
ful in the given stress situations, and this approach may be limited in classifying stress. In
small number of subjects. In order to supplement the above limitations, the following
addition, the PhysioNet stresswas
experiment database
conducted.is difficult to generalize because it consists of a
small number of subjects. In order to supplement the above limitations, the following ex-
2.1.1. Subjects
periment was conducted.
The dataset consists of physiological signals and questionnaires from 80 participants
(78 males and 2 females). All participants were third-year police officers without heart
2.1.1. Subjects
The dataset consists of physiological signals and questionnaires from 80 participants
(78 males and 2 females). All participants were third-year police officers without heart
sensors 2022, 12, x FOR PEER REVIEW 4 of

Biosensors 2022, 12, 465 4 of 16

disease and were in good physical condition. The data for six subjects (all male) were
cluded due to sensor problems. This study was approved by the Institutional Revi
Board of Hanyang
disease and wereUniversity Hospital
in good physical and was
condition. The conducted according
data for six subjects to thewere
(all male) guidelines
the Declaration
excluded dueof
toHelsinki, and allThis
sensor problems. subjects provided
study was informed
approved consent before
by the Institutional Reviewthe exp
Board of Hanyang University
iment (HYUIRB-202009-032-3) Hospital and was conducted according to the guidelines
of the Declaration of Helsinki, and all subjects provided informed consent before the
experiment (HYUIRB-202009-032-3)
2.1.2. Experimental Protocol
2.1.2.participants
All Experimental performed
Protocol the stress-inducing experiment in a laboratory enviro
All participants performed
ment according to the experimental the stress-inducing experiment
protocol shown in a laboratory
in Figure 2. Theenvironment
sequence of the
according to the experimental protocol shown in Figure 2. The sequence of the experiment
periment protocol was sensor attachment, resting state (pre) for 5 min, exposure to ea
protocol was sensor attachment, resting state (pre) for 5 min, exposure to each of the
of the3 3stressors
stressorsfor for 5 min,
5 min, resting
resting statestate
(post)(post) for 5and
for 5 min, min, and self-reported
self-reported subjectivesubjective
stress str
intensity. WeWe
intensity. used
used various
various methods
methods toto induce
induce acute
acute mental
mental stress,stress, including
including TSST (pub
TSST (public
speaking and arithmetic task) and horror movie
speaking and arithmetic task) and horror movie viewing. viewing.

Figure 2. Experimental protocol for inducing stress.

Figure 2. Experimental protocol for inducing stress.
Before the experiment, participants wore a Polar H10 HR monitor with a Polar Pro
Before the (Polar
Chest Strap experiment, participants
Electro Oy, Kempele, Finland) wore atoPolar
measureH10 HR monitor
respective with
heart rate (HR)a Polar P
and heart rate variability (HRV), which represent autonomic responses.
Chest Strap (Polar Electro Oy, Kempele, Finland) to measure respective heart rate (H The RR interval
data were transferred to Samsung Galaxy Tab A (Samsung Electronics, Co., Ltd., Seoul,
and heart rate variability (HRV), which represent autonomic responses. The RR inter
South Korea) and saved in Polar’s app. Based on findings that alpha-amylase increases
data when
werehumans
transferred to Samsung
are exposed to a stressful Galaxy Tab[41,42],
situation A (Samsung Electronics,was
salivary alpha-amylase Co., Ltd., Seo
also
Southmeasured
Korea) using
and asaved
COCORO in Polar’s
Meter (Niproapp. Co,Based
Osaka,onJapan).
findings that alpha-amylase increa
when humansAs shownareinexposed
Figure 2, to
theastress-inducing
stressful situation [41,42],
experiment salivary
protocol alpha-amylase
consists of a non-stress was a
section, which is comprised of rest (pre
measured using a COCORO Meter (Nipro Co, Osaka, Japan).and post), and a stress section, comprised of TSST
(public speaking, arithmetic task) and horror movie viewing. In the resting section, the
As shown in Figure 2, the stress-inducing experiment protocol consists of a non-str
participants were seated on a comfortable chair, and the HRV signal was measured for
section,
5 minwhich is comprised
of relaxation. Duringofthe rest (pre1 and
Stress post),
section, andspeaking
public a stressincluding
section, job-related
comprised of TS
(public speaking,
questions and aarithmetic
simulated job task) and horror
interview that was movie viewing.
conducted In theInresting
for 5 min. section,
the Stress
2 section,were
participants mental arithmetic
seated on atasks (MATs) were
comfortable performed.
chair, and theThe HRV MAT was was
signal to continue
measured fo
subtracting 17 from 2023. If an answer was wrong, subtraction started again from 2023,
min of relaxation. During the Stress 1 section, public speaking including job-related qu
and the MAT was repeated for 5 min. During the horror movie viewing within the Stress
tions3and a simulated
section, participants jobwatched
interview horror that wasclips
movie conducted
for 5 min. forAfter
5 min.the In the Stressa 2 secti
experiment,
mental arithmeticwas
questionnaire tasks (MATs)
conducted on were performed.
the ranking The stress
of subjective MATintensity.
was to continue
The ranking subtracti
17 from 2023. If an answer was wrong, subtraction started again from 2023, and the M
of subjective stress intensity was set to 5 for the most stressful situations and 1 for the
least stressful
was repeated forsituations.
5 min. During59 of 74the subjects
horror answered
moviethat mentalwithin
viewing arithmetic
thewas the most
Stress 3 section, p
stressful, while the remaining eight and seven subjects said horror movie clips and public
ticipants watched horror movie clips
speaking were the most stressful, respectively.
for 5 min. After the experiment, a questionnaire w
conducted on the ranking of subjective stress intensity. The ranking of subjective str
2.2. Data
intensity wasPreprocessing
set to 5 for the most stressful situations and 1 for the least stressful situatio
2.2.1. Noise
59 of 74 subjects Removal
answeredand Interpolation
that mental of HRV Signal was the most stressful, while the
arithmetic
maining Prior
eighttoand
HRVseven
analysis, preprocessing
subjects said is essential
horror to remove
movie clipstheand
outliers in RR
public intervals were
speaking
caused by noise, such as movement. Outliers in RR interval data were removed and defined
mostasstressful, respectively.
data outside 3 standard deviations (SD) from the mean [43]. Considering the non-linear
characteristics of HRV signals, cubic spline interpolation was performed [44]. The HRV
2.2. Data
signalPreprocessing
used to obtain EMD-based features was interpolated with cubic spline and was
resampled at 8 Hz.
2.2.1. Noise Removal and Interpolation of HRV Signal
Prior to HRV analysis, preprocessing is essential to remove the outliers in RR int
vals caused by noise, such as movement. Outliers in RR interval data were removed a
defined as data outside 3 standard deviations (SD) from the mean [43]. Considering
Biosensors 2022, 12, x FOR PEER REVIEW 5 of 17

Biosensors 2022, 12, 465 2.2.2. Comparison of Short-Term HRV and Ultra-Short-Term HRV 5 of 16
For comparison with ultra-short-term HRV, we analyzed short-term HRV using data
collected over 5 min, which is the data length commonly used in previous studies. We
2.2.2. the
selected Comparison of Short-Term
resting state and stressHRV stateand
forUltra-Short-Term
each participantHRV using the rankings of subjec-
tive stress intensity. The resting state was selected as the
For comparison with ultra-short-term HRV, we analyzed short-term lowest subjective
HRVstress
using intensity
data
collected
ranking over 5Resting
between min, which is the data
1, measured lengththe
before commonly used in previous
stress experiment, studies.2,We
and Resting meas-
selected the resting state and stress state for each participant using the rankings
ured after the stress experiment. The stress state was selected as the highest stress ranking of subjective
stress
among intensity.
Stress The2,resting
1, Stress state was
and Stress 3. selected as the lowest subjective stress intensity
ranking between Resting 1, measured before the stress experiment, and Resting 2, measured
According to Thomas Wyss et al. [45], the HR, an ANS indicator, is initially high
after the stress experiment. The stress state was selected as the highest stress ranking among
during acute mental stress situations and decreases as the stress situation continues. These
Stress 1, Stress 2, and Stress 3.
results indicate
According thattothe response
Thomas Wyss ofetthe
al.sympathetic
[45], the HR,nervous
an ANS system to acute
indicator, mental
is initially highstress
not during
only responds rapidly, but also adapts to a stress stimulus due to
acute mental stress situations and decreases as the stress situation continues. TheseANS homeostasis.
As shown in Figure
results indicate that3,theultra-short-term HRV analysis
response of the sympathetic nervouswassystem
performed
to acuteby dividing
mental stresstime
not only
segments responds
into rapidly, but
first, middle, andalsolastadapts
segments to a stress
over stimulus
1, 2, anddue to ANS
3 min, homeostasis.Each
respectively.
As shown
resting in Figure
state was paired 3, with
ultra-short-term
stress state, HRV andanalysis was performed
the segment with theby dividing
lowest time HR
average
segments
was used. into first, middle, and last segments over 1, 2, and 3 min, respectively. Each
resting state was paired with stress state, and the segment with the lowest average HR
was used.

Figure 3. Ultra-short-termHRV
3. Ultra-short-term
Figure HRVexplanation
explanation for
foreach
eachtime segment
time during
segment the experimental
during protocol.
the experimental proto-
col.
2.3. Feature Extraction
2.3.1. Time Domain Features
2.3. Feature Extraction
Time domain HRV features frequently used to evaluate acute mental stress were
2.3.1. Time Domain
extracted. Features
The mean RR interval (RR), standard deviation of RR interval (SDNN), square
root
Timeof the meanHRV
domain squared difference
features between
frequently successive
used RR intervals
to evaluate (RMSSD),
acute mental andwere
stress the ex-
proportion
tracted. of successive
The mean differences
RR interval between RR
(RR), standard intervalsof
deviation greater than x msec
RR interval (pNN30,
(SDNN), square
pNN50) were extracted. Expressions for these features are as follows.
root of the mean squared difference between successive RR intervals (RMSSD), and the
proportion of successive differences v between RR intervals greater than x msec (pNN30,
N −1
u
u 1
pNN50) were extracted. Expressions
SDNN = t for these features are as)follows.
(
N−1 ∑
(RRi − meanRR 2
) (1)
i =1

v1
SDNN = u 1 ( N (RR𝑖 − 𝑚𝑒𝑎𝑛RR) ) (1)
u
𝑁 − 1 ∑ (RRi+1 − RRi )2
RMSSD = t (2)
N − 1 i =1

NNx
pNNx =1 ∗ 100 (3)
N − 1 (RR
RMSSD = − RR ) (2)
Additionally, we extracted G-pNNx𝑁− 1
(Grouped-pNNx), a new feature that can be
applied to a group of young and healthy people. Based on a prior study [46], using a
parameter between pNN10-40 instead of pNN50𝑁𝑁𝑥
is more suitable for stress assessment. We
pNNx = ∗ 100 (3)
𝑁−1
Biosensors 2022, 12, 465 6 of 16

developed G-pNNx as a pNNx-based stress feature optimized for young people using data
collected from young police officers at rest. The procedure for calculating the x-value of
G-pNNx is as follows.
1. On the basis of subjectively ranked self-reported stress intensity in the experimental
protocol, select the lowest stress rank between Resting 1 and Resting 2.
2. If the distribution of data satisfies normality, obtain G-pNNx using the mean value of
the distribution; otherwise, obtain it using the median value.
Since the data do not satisfy normality (Kolmogorov–Smirnov, p < 0.05) in our study,
the x value of G-pNNx was obtained using the median value (18.5).

2.3.2. EMD-Based Features

EMD
EMD is an adaptive time-series analysis method suitable for processing non-stationary
and non-linear series [47]. Since the HRV signal has non-stationary and non-linear char-
acteristics [48], EMD is suitable for our research. EMD can decompose any signal with
an IMF. As the x-value in an IMF increases, the low-frequency component of the original
signal is included. For example, IMF1 represents the highest local frequency component of
the signal. In order to extract IMFs from the original signal using the EMD method, the
following two basic conditions are essential [49].
1. The numbers of extrema and zero crossings must be the same or different at most
by one within the entire dataset.
2. The mean value of the envelope defined by local maxima and minima must be zero
at any point.
If the two conditions are satisfied, the IMF can be continuously decomposed by the
EMD method for x(t) and mathematically expressed as follows.
n
x (t) = ∑ IMFk + rn (4)
k =1

where x(t) is the HRV signal, we used decomposed IMF of 1~3, and the residual r was not used.

Entropy Features
The entropy method is suitable for HRV signals as a non-linear method similar to EMD,
and the entropy methods used in our study are permutation entropy and sample entropy.
Permutation entropy (PE) is useful to represent the complexity of dynamic time-series
signals and has the advantages of simple calculation and robustness to noise [50]. PE is
calculated by the following equation [51].

m!
PEn = − ∑ Ai log( Ai ) (5)
i =1

The factorial calculated from sequence length m (dimension) is the number of possi-
ble permutation patterns, and Ai is the probability of the i-th permutation pattern. The
setting values in the permutation entropy method are time delay tau and dimension m.
Since the sampling frequency of the resampled HRV signal is 8 Hz, the maximum value
of tau was set to 4 by the Nyquist–Shannon sampling theorem [52]. Since the minimum
length of data used for ultra-short-term HRV analysis was N (data length) = 480 (sam-
pling frequency = 8 Hz, 60 s), m was set to 4, suitable for values under the condition of
N > 5m! [53].
Sample entropy compensates for the disadvantage of approximate entropy [54] and
measures the irregularity and complexity of HRV signal and IMF components. The setting
values used in sample entropy consist of embedding dimension m and tolerance r. In a
previous study [55], the embedding dimension value was set to 2, and the tolerance value
was set using the standard deviation of the HRV data (r = 0.2 × SD).
Biosensors 2022, 12, 465 7 of 16

Energy Features
Previous stress assessment studies based on HRV signals frequently use frequency
domain features, including the VLF component (~0.04 Hz), LF component (0.04~0.15 Hz),
HF component (0.15~0.4 Hz), and LF/HF ratio. A major drawback of frequency domain
features is the inaccuracy when using ultra-short-term HRV signals. According to [56], LF
band (0.04~0.15 Hz) power requires HRV data to span at least 250 s from a theoretical point
of view, so ultra-short-term HRV analysis using HRV data lasting about 4 min or less has
poor reliability regarding several HRV features based on LF band power.
Based on Parseval’s theorem [57] that the total energy of a signal can be calculated by
summing power across time or spectral power across frequency, as shown in Equation (6),
this paper presents a methodology that can replace frequency domain features with the
energy of IMF components. The methodology compares the HF band power and energy
of the high-frequency component IMF1 and the LF band power and energy of the rel-
atively low-frequency components (IMF2 and IMF3). In addition, IMF energy features
corresponding to the LF/HF ratio, normalized-LF, and normalized-HF were extracted.
∞
1
Z π
∑ | x [n]|2 =
2π −π
| X2π (φ)|2 dφ (6)
n=−∞

SD-IMF and RMSSD-IMF Features

Three IMF components obtained through the EMD method were extracted using
SDNN Equations (1) and RMSSD (2). Instead of RRi in Equation (1), three SD-IMF features
were used for the IMF components, and three RMSSD-IMF features were extracted using
Equation (2).

2.4. Feature Ranking Method

The number of extracted HRV features was 26, including 6 general time domain
features, 3 entropy/energy features of HRV signal not decomposed through EMD (permu-
tation entropy, sample entropy, and energy), 6 EMD-based entropy features, 5 EMD-based
entropy features, and 6 EMD-based time domain features.
The Relief-F algorithm was applied to remove features that did not contribute to stress
classification performance and to improve computational efficiency. The Relief-F algorithm
is a method to evaluate the contribution of each feature based on k-nearest neighbors by
increasing the inter-class difference and decreasing the intra-class difference [58]. The
algorithm finds a sample with the same class label and a sample with a different class
label closest to a randomly selected sample among the k samples in the closest order and
calculates a weight using the difference between the selected sample and the closest sample.
In this study, the feature with the lowest weight value between the resting state and the
stress state was removed sequentially to achieve good performance.

2.5. Classification Method

2.5.1. Support Vector Machine (SVM) Classifier
In order to classify the resting and stress states, the SVM classifier widely used in
stress research was used. The SVM classifier is a method of finding the optimal hyperplane
to classify a class, and a vector contributing to creating the hyperplane is called a support
vector [59]. Mathematically, an SVM is represented as follows [60]:
"
p
#
H ( x ) = sign ∑ αk tk P(z, zk ) + b (7)
k =1

where P(z,zk ) is the kernel function, zk is the D-dimension k-input vector (feature), tk is the
target class vector, ak is the LaGrangian multiplier, and b represents the bias term. We used
SVM with a linear classifier and classified stress and non-stress states using short-term
HRV and ultra-short-term HRV data.
 Biosensors 2022, 12, 465 8 of 16

2.5.2. Leave-One-Subject-Out Cross Validation (LOSOCV)

The LOSOCV method is a variation of the k-fold cross-validation approach that vali-
dates as many folds as there are subjects included in the data set [61]. LOSOCV evaluates
the accuracy on new subjects that have not been seen by the model, suggesting whether the
developed classifier model can achieve classification performance when applied in real situ-
ations. In this study, the performance of the training model was evaluated using LOSOCV.

2.5.3. Performance Evaluation

We calculated evaluation indicators, such as accuracy, precision, recall, and F1-score,
to explain the results of LOSOCV. Each evaluation indicator is expressed as follows [62]:

TP + TN
Accuracy = (8)
TP + TN + FP + FN
TP
Precision = (9)
TP + FP
TP
Recall = (10)
TP + FN
Precision ∗ Recall
F1 Score = 2 ∗ (11)
Precision + Recall

3. Results
3.1. Relationships between Frequency Domain Features and IMF Energy Features
In order to perceive the trends in 5-min HRV signal and IMF components in the
frequency domain, we represented each component using fast Fourier transforms (FFTs),
as shown in Figure 4. The HF band (0.15–0.4 Hz) of the HRV and IMF1 spectral areas (blue)
are similar, and the LF band (0.04–0.15 Hz) of the HRV spectrum and the sum of the IMF2
(red) and IMF3 spectral areas (pink) are similar. In addition, the mean frequency for each
osensors 2022, 12, x FOR PEER REVIEW IMF using the entire data including the resting and stress state is shown in Figure 5. The 9 o
mean frequency of IMF1 is included in the HF band, and IMF2 and IMF3 are included in
the LF band.

FFT Spectrum of resting state FFT Spectrum of stress state

Figure 4. FFT
Figure spectra
4. FFT ofofresampled
spectra HRV
resampled HRV and
and IMFIMF components:
components: restingresting state
state (left), (left),
stress statestress
(right).state (rig
Biosensors 2022, 12, 465 9 of 16
Figure 4. FFT spectra of resampled HRV and IMF components: resting state (left), stres

Figure 5. Mean frequency range of each IMF components in short-term HRV.

Figure 5. Mean frequency range of each IMF components in short-term HRV.
The Pearson correlation method was used to describe the proposed IMF energy fea-
turesThe Pearsonofcorrelation
as surrogates method
frequency domain wasTable
features. used to describe
1 presents the proposed
the correlation coeffi- IMF
cients of the relationships between IMF energy (IMF1, IMF2+IMF3, and ratio) and frequency
tures as surrogates of frequency domain features. Table 1 presents the corre
domain features (HF, LF, and HF/LF ratio). Using the scale of Hopkins [63], the results
cients
in Tableof1 were
the relationships between
qualitatively analyzed with theIMF energy
Pearson (IMF1,
correlation IMF2+IMF3,
r, described as trivial and ra
quency
(0.0–0.1),domain features
small (0.1–0.3), (HF,
moderate LF, and
(0.3–0.5), largeHF/LF ratio).
(0.5–0.7), Using
very large the or
(0.7–0.9), scale of Hopk
nearly
perfect (0.9–1.0).
results in Table 1 were qualitatively analyzed with the Pearson correlation r,
trivial
Table 1.(0.0–0.1), small (0.1–0.3),
Correlation coefficients moderate
between frequency domain(0.3–0.5),
features. large (0.5–0.7), very large
nearly perfect (0.9–1.0). HF LF LF/HF Ratio
Using all data, including
EnergyIMF1 0.93
both the resting 0.79
state and the −stress
0.29
state, IMF1
tures hadIMF2
Energy a nearly
+IMF3 perfect0.77 correlation value0.92 (r = 0.93) with HF −0.03and a relative
Energy
IMF2+IMF3 − 0.43 − 0.09 0.86
lation value
Energy with LF (r = 0.77). Contrary to IMF1-energy features, the IMF2+I
IMF1

features had a nearly perfect correlation value (r = 0.92) with LF and a relativ
very Using
large all data, including
correlation valueboth(rthe restingin
= 0.79) state
HF.and The theIMF-energy
stress state, IMF1-energy
ratio was high
features had a nearly perfect correlation value (r = 0.93) with HF and a relatively low
with the LF/HF
correlation ratio
value with LF(r(r = 0.86).Contrary to IMF1-energy features, the IMF2+IMF3-
= 0.77).
energy features had a nearly perfect correlation value (r = 0.92) with LF and a relatively
low and
Table very large correlation
1. Correlation value
coefficients (r = 0.79)frequency
between in HF. The domain
IMF-energy ratio was highly
features.
correlated with the LF/HF ratio (r = 0.86).
HF LF LF/H
3.2. Comparison of Feature Ranks between Resting and Stress States
𝐄𝐧𝐞𝐫𝐠𝐲
When 𝐈𝐌𝐅𝟏
classifying the resting state0.93 0.79algorithm using
and stress state with the Relief-F −
𝐄𝐧𝐞𝐫𝐠𝐲HRV
short-term 0.77 in descending order of 0.92
data, the results are listed
𝐈𝐌𝐅𝟐 𝐈𝐌𝐅𝟑
weight, as shown in −
𝐄𝐧𝐞𝐫𝐠𝐲𝐈𝐌𝐅𝟐 𝐈𝐌𝐅𝟑
Table 2. Linear SVM classifier training was repeated by adding features in ascending order
from the ranked features one at a time,−0.43and the optimal number of −0.09
features with the best 0
𝐄𝐧𝐞𝐫𝐠𝐲
classification performance
𝐈𝐌𝐅𝟏 was selected. From these results, the proposed energy-related
features, such as ranks 1, 3, and 6, have high weights.
 1 Energy 14 SpE
2 SDNN 15 G-pN
3 Energy_(IMF2+IMF3) 16 PmEn_
Biosensors 2022, 12, 465 4 SpEn_IMF3 17 10 of 16 pNN
5 RMSSD_IMF3 18 pNN
6 Energy_IMF1 19
Table 2. HRV features in descending order of significance according to the Relief-F algorithm.
SpEn_
7 SDNN_IMF3 20 Energy_IM
Rank Feature Name Rank Feature Name
8 1 HR
Energy 14
21 SpEn
Energy_IM
9 2 SDNN_IMF2
SDNN 15 22 G-pNNxEnergy_IMF
3 Energy_(IMF2+IMF3) 16 PmEn_IMF3
10 4 RMSSD_IMF2
SpEn_IMF3 17 23 pNN50 Pm
11 56 RMSSD_IMF3
RMSSD
Energy_IMF1
18
19
pNN30
24 SpEn_IMF2 PmEn_
12 7 RMSSD_IMF1
SDNN_IMF3 20 25Energy_IMF23/IMF1 PmEn_
8 HR 21 Energy_IMF1/IMF123
13 9 SDNN_IMF1
SDNN_IMF2 22 26Energy_IMF23/IMF123 SpEn_
10 RMSSD_IMF2 23 PmEn
11 RMSSD 24 PmEn_IMF1
3.3. Short-Term
12 HRV RMSSD_IMF1
Classification and Performance
25 Evaluation
PmEn_IMF2
13 SDNN_IMF1 26 SpEn_IMF1
The results of LOSOCV performance evaluation on short-term HRV d
selection areHRV
3.3. Short-Term shown in Figure
Classification 6. TheEvaluation
and Performance best classification performance was
the classifier model was developed using
The results of LOSOCV performance evaluation the top HRV
on short-term 17 highest
data after ranked
feature featu
selection are shown in Figure 6. The best classification performance was obtained when
comparing the stress state to the resting state were 86.5% accurate, with a
the classifier model was developed using the top 17 highest ranked features. The results
precision
comparing the ofstress
87.5%,
stateand
to theF1-score ofwere
resting state 86.3%.
86.5% accurate, with a recall of 85.1%,
precision of 87.5%, and F1-score of 86.3%.

Figure 6. Classification performance using short-term HRV data.

Figure 6. Classification
3.4. Ultra-Short-Term performance
Classification using
and Performance short-term HRV data.
Evaluation
The classification results based on ultra-short-term HRV data by dividing the time
segments into each time length, designated as the first, middle, and last time lengths
(1, 2, and 3 min), are shown in Figure 3. The linear-SVM classifier model was used in the
same way as the classification method based on the short-term HRV data, and LOSOCV
performance evaluation was performed using the ranked features listed in Table 2. The
results of ultra-short-term HRV classification were compared for time segment and time
 Biosensors 2022, 12, 465 11 of 16
osensors 2022, 12, x FOR PEER REVIEW 11 of

length (Table 3). When compared in the time segments between the first, middle, and last
time lengths, the classification accuracy of the first length is higher than that of the middle
3.4. Ultra-Short-Term Classification
and last lengths, as shown in Figure and Performance
7. The Evaluation
accuracy of the first segment of 2-min and 3-min
lengths is higher than the classification accuracy using
The classification results based on ultra-short-term HRV short-term HRV data.by
data Segments
dividingof the tim
1-min length achieved the highest accuracy in the middle part, and this result indicated a
segments into each time length, designated as the first, middle, and last time lengths (1
different tendency than those of segments of 2-min and 3-min length. All segments of 1-min
and 3length
min), ledare shown
to less in Figure
accurate 3. The
predictions thanlinear-SVM classifier
the classification model
performance wasshort-term
using used in the sam
way HRV
as the classification method based on the short-term HRV data, and LOSOCV perfo
data.
mance evaluation was performed using the ranked features listed in Table 2. The resu
Table 3. Classification performance using ultra-short-term HRV data according to time segments and
of ultra-short-term
time lengths.
HRV classification were compared for time segment and time leng
(Table 3). When compared in the time segments between the first, middle, and last tim
Classification Performance (%)
lengths, the classification accuracy of the first length is higher than that of the middle a
last lengths, as shown in Figure 7. The accuracy of theMiddle
Front first segment of Last
2-min and 3-m
3-min
lengths is higher than the Accuracy 90.5
classification accuracy 84.5
using short-term HRV84.5data. Segments
segments F1 Score 90.3 83.7 84.6
1-min length achieved the highest accuracy in the middle part, and this result indicated
2-min Accuracy 87.2 81.8 82.4
differentsegments
tendency than F1 those
Score of segments 86.7of 2-min and82.13-min length. 81.9 All segments of
min length1-min led to less accurate
Accuracypredictions than the classification
82.4 85.1 performance
79.7 using sho
term HRV data.
segments F1 Score 82.4 84.5 79.2

Figure 7. Comparison of classification accuracy using ultra-short-term HRV data.

3.5. Salivary Cortisol of Different States

Figure 7. Comparison of classification accuracy using ultra-short-term HRV data.
According to the experiment protocol in Figure 2, sAA measurements were performed
a total of four times during the experiment (sAA1: After public speaking sAA, sAA2: After
Tablemental
3. Classification performance
arithmetic sAA, sAA3: Afterusing
horrorultra-short-term
movie sAA, sAA4: HRV data
After according
resting to time
sAA). Since not segme
and time lengths.
all sAA data satisfied the normality test, the resting state (sAA4) and stress states (sAA1~3)
analyses were performed using the non-parametric Mann–Whitney U test. Compared with
sAA4, sAA1 (p = 0.056), sAA2 (p = 0.797),Classification
and sAA3 (p =Performance
0.082) were not (%)
significant in all
stress states. Front Middle Last
3-min Accuracy 90.5 84.5 84.5
segments F1 Score 90.3 83.7 84.6
2-min Accuracy 87.2 81.8 82.4
segments F1 Score 86.7 82.1 81.9
Biosensors 2022, 12, 465 12 of 16

4. Discussion
This study suggests that the proposed IMF energy features obtained using EMD are
good surrogates of the frequency domain features. Although there is previous research that
showed similarity among HRV features, including time domain, frequency domain, and
non-linear features [64], this is the first study to suggest that frequency domain features
such as HF, LF, and ratio can be replaced with other HRV features.
Ultra-short-term HRV analysis, which evaluates mental stress using data less than
5 min long, is a topic of increasing interest [56]. Castaldo et al. [32] presented time segments
and HRV features in ultra-short-term HRV data that can be replaced with short-term HRV
analysis as a result of correlation analysis between ultra-short-term and short-term HRV
features. It was suggested that features including HR, RMSSD, pNN50, and sample entropy
can be substituted for ultra-short-term HRV analysis with a 2-min length compared to
short-term HRV analysis (r > 0.7). Using the same method from the research above, 4 time
lengths (5 min, 3 min, 2 min, and 1 min) were compared, as in Table 4. In the resting state,
all features were significantly correlated in each time scale (r > 0.7), and all features except
the 1 min time scale were significantly correlated in the stress state (r > 0.7). Accordingly,
our proposed IMF energy features show that ultra-short-term HRV analysis can replace
short-term HRV analysis with a time length of at least 2 min.

Table 4. Correlation analysis of ultra-short-term vs. short-term HRV features.

Rest State Stress State

HRV Features 3 vs. 5 min 2 vs. 5 min 1 vs. 5 min 3 vs. 5 min 2 vs. 5 min 1 vs. 5 min
EnergyIMF1 0.978 0.941 0.901 0.959 0.948 0.920
EnergyIMF2+IMF3 0.935 0.881 0.813 0.943 0.889 0.836
EnergyIMF1
EnergyIMF2+IMF3
0.986 0.968 0.883 0.809 0.742 0.674

The proposed stress classification method was compared with some of the latest
studies that selected TSST including public speaking and cognitive tasks as mental stressors
and conducted acute stress classification with short-term HRV analysis. Table 5 compares
the number of subjects, type of measured physiological signals, classifier model, validation
method, and accuracy of our and previous studies. Multi-physiological signals, including
ECG, PPG, and GSR, were measured using public speaking as a stressor, and the result was
achieved with 79% accuracy using the AdaBoost classifier and four-fold cross-validation to
classify between stress and non-stress states [17]. The results of this study have relatively
low accuracy compared to other studies. The stress classification accuracy of another
study [18] was 96.3% using leave-one-out cross-validation with the SVM-RBF classifier
considering PPG, GSR, and EEG. However, the result considering a single signal (PPG)
was 80.0%, which is lower than our accuracy of 86.5%. Another study [12] selected TSST
and Stroop color test as stressors and used a random forest classifier and three-fold cross-
validation. The results for overlapping and not overlapping HRV signals over 5 min are
accuracies of 96.0% and 85.9%, respectively. The disadvantage of the aforementioned
study is that the number of subjects is small, and leave-one-subject-out cross-validation,
similar to that applied in real-world, was not used. The study most similar to the present
study [11] selected TSST as the stressor, and the classification accuracy results of five-fold
cross-validation and leave-one-subject-out cross-validation with an ANN classifier were
91% and 84.4%, respectively. When comparing based on LOSOCV, the results of our study
achieved a higher accuracy of 86.5%.
The dataset of our study, collected from 74 people, is the largest study that selected
TSST as a stressor. Compared with precedent studies using a single physiological signal
and LOSOCV, the result of the proposed stress detection method presents the highest
performance. For various stress-inducing tasks (public speaking, arithmetic, horror movie
viewing), the stress state and resting state were selected as individual stress intensities
Biosensors 2022, 12, 465 13 of 16

through a questionnaire to rank subjective stress intensity. The reason for dividing the
stress and resting states in this way is that the stress intensity for each task differs by person,
which is also one of the advantages compared to previous studies.

Table 5. Performance comparison of the proposed method with the state-of-the-art methods for
short-term HRV analysis.

Physiological
Number of
Paper Signals Classifier Validation Accuracy (Classes)
Subjects
(Modalities)
ECG (HRV), PPG,
[17], 2017 18 AdaBoost 4-fold 79.0% (2)
GSR
PPG 80.0% (2)
[18], 2021 40 PPG + GSR SVM-RBF LOSOCV 86.3% (2)
PPG + GSR + EEG 96.3% (2)
Non-overlapping:
[12], 2021 12 ECG (HRV) Random Forest 3-fold 85.9% (2)
overlapping: 96.0% (2)
5-fold 91.0% (2)
[11], 2020 57 ECG (HRV) ANN
LOSOCV 84.4% (2)
Proposed 74 ECG (HRV) Linear SVM LOSOCV 86.5% (2)

As HRV analysis of data spanning less than 5 min becomes important in several
healthcare applications, ultra-short-term HRV analysis has been actively studied in previous
studies of HRV stress assessment using various methods included in mental arithmetic
tasks [28], examination stress [11,32], and the Stroop color-word task [19,65]. However, in
all previous studies, the HRV data used for ultra-short-term HRV analysis was divided
according to the required duration or the central position of the 5 min HRV data. The
proposed study is the first considering stress adaptation due to ANS homeostasis during
stress-induced tasks. By comparing the results of ultra-short-term HRV stress classification
consisting of three time segments (first, middle, and last) and three durations (1 min,
2 min, and 3 min) with general short-term HRV stress classification, stress adaptation was
confirmed by deriving higher accuracy in the first segment than in the last segment. In
addition, the accuracy of ultra-short-term HRV stress classification (Table 3) shows higher
accuracy in the first segments of 2-min and 3-min lengths compared to the accuracy of
short-term HRV stress classification (86.5%). The optimal time length to be used for stress
assessment using ultra-short-term HRV data is 3 min, but a minimum length of 2 min
is suggested.

5. Conclusions
In our study, we presented IMF energy features based on EMD as a surrogate of
general frequency domain features in HRV analysis and suggested the optimal duration
for ultra-short-term HRV analysis as a result of stress classification accuracy comparison
between short-term and ultra-short-term HRV analysis. Performance evaluation was
conducted using LOSOCV, most similar to real-world situations. The proposed study has
high accuracy compared to previous studies using similar classification methods based on
a single physiological signal.
This study has three contributions. First, we provide frequency domain information
with data of less than 5 min. Through EMD-based features, it is possible to replace infor-
mation about frequency domain that are difficult to use in ultra-short-term. Second, we
identified stress adaptability over time. In this way, the ultra-short-term analysis would
be more appropriate when evaluating stress than the commonly used short-term analysis.
Finally, we propose an optimal time length for ultra-short-term HRV data collection in
Biosensors 2022, 12, 465 14 of 16

acutely stressful situations. It is possible to provide a comfortable stress analysis service for
several healthcare applications through the proposed length.

Author Contributions: Conceptualization, I.Y.K. and J.L.; methodology, J.H.H. and Y.J.; software,
H.B.H. and S.P.; formal analysis, S.L.; investigation, S.K.; resources, S.L.; data curation, H.B.H. and S.P.;
writing—original draft preparation, S.L.; writing—review and editing, I.Y.K. and J.L.; visualization,
S.H. and H.-K.P.; supervision, I.Y.K. and J.L.; project administration, H.-K.P. and J.L.; funding
acquisition, H.-K.P. All authors have read and agreed to the published version of the manuscript.
Funding: This work was supported by the National Research Foundation of Korea (NRF) grant
funded by the Korea government (MSIT) (No. 2018R1A5A7025522).
Institutional Review Board Statement: The study was conducted in accordance with the Declaration
of Helsinki, and approved by the Institutional Review Board of the Hanyang University (HYUIRB-
202009-032-3).
Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.
Data Availability Statement: Data sharing is not applicable to this article.
Conflicts of Interest: The authors declare no conflict of interest.

References
1. Dimsdale, J.E. Psychological Stress and Cardiovascular Disease. J. Am. Coll. Cardiol. 2008, 51, 1237–1246. [CrossRef] [PubMed]
2. Duman, R.S. Neurobiology of Stress, Depression, and Rapid Acting Antidepressants: Remodeling Synaptic Connections. Depress.
Anxiety 2014, 31, 291–296. [CrossRef] [PubMed]
3. Heraclides, A.M.; Chandola, T.; Witte, D.R.; Brunner, E.J. Work Stress, Obesity and the Risk of Type 2 Diabetes: Gender-Specific
Bidirectional Effect in the Whitehall II Study. Obesity 2012, 20, 428–433. [CrossRef] [PubMed]
4. Yaribeygi, H.; Panahi, Y.; Sahraei, H.; Johnston, T.P.; Sahebkar, A. The Impact of Stress on Body Function: A Review. EXCLI J.
2017, 16, 1057–1072. [PubMed]
5. Takahashi, T.; Ikeda, K.; Ishikawa, M.; Kitamura, N.; Tsukasaki, T.; Nakama, D.; Kameda, T. Anxiety, Reactivity, and Social
Stress-Induced Cortisol Elevation in Humans. Neuroendocrinol. Lett. 2005, 26, 351–354. [PubMed]
6. Goyal, A.; Singh, S.; Vir, D.; Pershad, D. Automation of Stress Recognition Using Subjective or Objective Measures. Psychol. Stud.
2016, 61, 348–364. [CrossRef]
7. Cohen, S.; Kamarck, T.; Mermelstein, R. A Global Measure of Perceived Stress. J. Health Soc. Behav. 1983, 24, 385–396. [CrossRef]
8. Spielberger, C.D. State-Trait Anxiety Inventory; Mind Garden: Redwood City, CA, USA, 1981. [CrossRef]
9. Khoulji, S.; García, E.; Aguiló, S.; Arza, A.; Garzón-Rey, J.M.; Aguilóa, J. Psychological and Physiological Profiles in Oncology
Caregivers: A Multivariable Cross-Sectional Study. Trans. Mach. Learn. Artif. Intell. 2017, 5, 4. [CrossRef]
10. Urwyler, S.A.; Schuetz, P.; Sailer, C.; Christ-Crain, M. Copeptin as a Stress Marker Prior and after a Written Examination-the
CoEXAM Study. Stress 2015, 18, 134–137. [CrossRef]
11. Kalatzis, A.; Stanley, L.; Karthikeyan, R.; Mehta, R.K. Mental Stress Classification during a Motor Task in Older Adults Using
an Artificial Neural Network. In Proceedings of the Adjunct Proceedings of the 2020 ACM International Joint Conference on
Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers,
Online Conference, 12–17 September 2020; pp. 244–248.
12. Szakonyi, B.; Vassányi, I.; Schumacher, E.; Kósa, I. Efficient Methods for Acute Stress Detection Using Heart Rate Variability Data
from Ambient Assisted Living Sensors. Biomed. Eng. Online 2021, 20, 73. [CrossRef]
13. Villarejo, M.V.; Zapirain, B.G.; Zorrilla, A.M. A Stress Sensor Based on Galvanic Skin Response (GSR) Controlled by ZigBee.
Sensors 2012, 12, 6075–6101. [CrossRef]
14. Greco, A.; Valenza, G.; Lazaro, J.; Garzon-Rey, J.M.; Aguilo, J.; De-la-Camara, C.; Bailon, R.; Scilingo, E.P. Acute Stress State
Classification Based on Electrodermal Activity Modeling. IEEE Trans. Affect. Comput. 2021, 3045. [CrossRef]
15. Asif, A.; Majid, M.; Anwar, S.M. Human Stress Classification Using EEG Signals in Response to Music Tracks. Comput. Biol. Med.
2019, 107, 182–196. [CrossRef]
16. Arsalan, A.; Majid, M.; Butt, A.R.; Anwar, S.M. Classification of Perceived Mental Stress Using a Commercially Available EEG
Headband. IEEE J. Biomed. Health Inform. 2019, 23, 2257–2264. [CrossRef]
17. Mozos, O.M.; Sandulescu, V.; Andrews, S.; Ellis, D.; Bellotto, N.; Dobrescu, R.; Ferrandez, J.M. Stress Detection Using Wearable
Physiological and Sociometric Sensors. Int. J. Neural Syst. 2017, 27, 1650041. [CrossRef]
18. Arsalan, A.; Majid, M. Human Stress Classification during Public Speaking Using Physiological Signals. Comput. Biol. Med. 2021,
133, 104377. [CrossRef]
19. Karthikeyan, P.; Murugappan, M.; Yaacob, S. Analysis of Stroop Colorword Test-Based Human Stress Detection Using Electrocar-
diography and Heart Rate Variability Signals. Arab. J. Sci. Eng. 2012, 39, 1835–1847. [CrossRef]
Biosensors 2022, 12, 465 15 of 16

20. Kirschbaum, C.; Pirke, K.-M.; Hellhammer, D.H. The ‘Trier Social Stress Test’—A Tool for Investigating Psychobiological Stress
Responses in a Laboratory Setting. Neuropsychobiology 1993, 28, 76–81. [CrossRef]
21. Subhani, A.R.; Mumtaz, W.; Saad, M.N.B.M.; Kamel, N.; Malik, A.S. Machine Learning Framework for the Detection of Mental
Stress at Multiple Levels. IEEE Access 2017, 5, 13545–13556. [CrossRef]
22. Slavich, G.M.; Taylor, S.; Picard, R.W. Stress Measurement Using Speech: Recent Advancements, Validation Issues, and Ethical
and Privacy Considerations. Stress 2019, 22, 408–413. [CrossRef]
23. Bornoiu, I.V.; Grigore, O. A Study about Feature Extraction for Stress Detection. In Proceedings of the 2013 8th International
Symposium on Advanced Topics in Electrical Engineering (ATEE), Bucharest, Romania, 23–25 May 2013; pp. 1–4. [CrossRef]
24. Fan, J.; Li, H.; Zhan, Y.; Yu, Y. An Electrocardiogram Acquisition and Analysis System for Detection of Human Stress. In
Proceedings of the 12th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics:
CISP-BMEI 2019, Huaqiao, China, 19–21 October 2019. [CrossRef]
25. Dickerson, S.S.; Kemeny, M.E. Acute Stressors and Cortisol Responses: A Theoretical Integration and Synthesis of Laboratory
Research. Psychol. Bull. 2004, 130, 355–391. [CrossRef]
26. Force, T. Heart Rate Variability Guidelines: Standards of Measurement, Physiological Interpretation, and Clinical Use. Eur. Hear.
J. 1996, 17, 354–381.
27. Athavale, Y.; Krishnan, S. Biosignal Monitoring Using Wearables: Observations and Opportunities. Biomed. Signal Process. Control.
2017, 38, 22–33. [CrossRef]
28. Zubair, M.; Yoon, C. Multilevel Mental Stress Detection Using Ultra-Short Pulse Rate Variability Series. Biomed. Signal Process.
Control. 2020, 57, 101736. [CrossRef]
29. Zhang, C.; Starner, T.; Inan, O.; Abowd, G.D.; Xue, Q.; Waghmare, A.; Meng, R.; Jain, S.; Han, Y.; Li, X.; et al. FingerPing:
Recognizing Fine-Grained Hand Poses Using Active Acoustic On-Body Sensing. In Proceedings of the 2018 CHI Conference on
Human Factors in Computing Systems—CHI ’18, Montreal, QC, Canada, 21–26 April 2018; ACM Press: New York, NY, USA,
2018; pp. 1–10.
30. Castaldo, R.; Xu, W.; Melillo, P.; Pecchia, L.; Santamaria, L.; James, C. Detection of Mental Stress Due to Oral Academic
Examination via Ultra-Short-Term HRV Analysis. In Proceedings of the 38th Annual International Conference of the IEEE
Engineering in Medicine and Biology Society, Orlando, FL, USA, 16–20 August 2016; pp. 3805–3808. [CrossRef]
31. Yentes, J.M.; Hunt, N.; Schmid, K.K.; Kaipust, J.P.; McGrath, D.; Stergiou, N. The Appropriate Use of Approximate Entropy and
Sample Entropy with Short Data Sets. Ann. Biomed. Eng. 2013, 41, 349–365. [CrossRef]
32. Castaldo, R.; Montesinos, L.; Melillo, P.; James, C.; Pecchia, L. Ultra-Short Term HRV Features as Surrogates of Short Term HRV: A
Case Study on Mental Stress Detection in Real Life. BMC Med. Inform. Decis. Mak. 2019, 19, 12. [CrossRef]
33. Cheema, A.; Singh, M. Psychological Stress Detection Using Phonocardiography Signal: An Empirical Mode Decomposition
Approach. Biomed. Signal Process. Control. 2019, 49, 493–505. [CrossRef]
34. Cheema, A.; Singh, M. An Application of Phonocardiography Signals for Psychological Stress Detection Using Non-Linear
Entropy Based Features in Empirical Mode Decomposition Domain. Appl. Soft. Comput. J. 2019, 77, 24–33. [CrossRef]
35. Deka, D.; Deka, B. Characterization of Heart Rate Variability Signal for Distinction of Meditative and Pre-Meditative States.
Biomed. Signal Process. Control. 2021, 66, 102414. [CrossRef]
36. Porta, A.; Gnecchi-Ruscone, T.; Tobaldini, E.; Guzzetti, S.; Furlan, R.; Montano, N. Progressive Decrease of Heart Period Variability
Entropy-Based Complexity during Graded Head-up Tilt. J. Appl. Physiol. 2007, 103, 1143–1149. [CrossRef]
37. Li, Y.; Pan, W.; Li, K.; Jiang, Q.; Liu, G. Sliding Trend Fuzzy Approximate Entropy as a Novel Descriptor of Heart Rate Variability
in Obstructive Sleep Apnea. IEEE J. Biomed. Health Inform. 2019, 23, 175–183. [CrossRef] [PubMed]
38. Kumar, M.; Pachori, R.B.; Rajendra Acharya, U. Automated Diagnosis of Atrial Fibrillation ECG Signals Using Entropy Features
Extracted from Flexible Analytic Wavelet Transform. Biocybern. Biomed. Eng. 2018, 38, 564–573. [CrossRef]
39. Amin, M.R.; Wickramasuriya, D.S.; Faghih, R.T. A Wearable Exam Stress Dataset for Predicting Grades Using Physiological
Signals. In Proceedings of the 2022 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT), Houston, TX, USA,
10–11 March 2022; pp. 30–36.
40. Healey, J.A.; Picard, R.W. Detecting Stress during Real-World Driving Tasks Using Physiological Sensors. IEEE Trans. Intell.
Transp. Syst. 2005, 6, 156–166. [CrossRef]
41. Skosnik, P.D.; Chatterton, R.T.; Swisher, T.; Park, S. Modulation of Attentional Inhibition by Norepinephrine and Cortisol after
Psychological Stress. Int. J. Psychophysiol. 2000, 36, 59–68. [CrossRef]
42. Bosch, J.A. Psychological Stress as a Determinant of Protein Levels and Salivary-Induced Aggregation of Streptococcus Gordonii in
Human Whole Saliva. Psychosom. Med. 1996, 382, 374–382. [CrossRef]
43. Olewuezi, N.P. Note on the Comparison of Some Outlier Labeling Techniques. J. Math. Stat. 2011, 7, 353–355. [CrossRef]
44. Choi, A.; Shin, H. Quantitative Analysis of the Effect of an Ectopic Beat on the Heart Rate Variability in the Resting Condition.
Front. Physiol. 2018, 9, 922. [CrossRef]
45. Wyss, T.; Boesch, M.; Roos, L.; Tschopp, C.; Frei, K.M.; Annen, H.; La Marca, R. Aerobic Fitness Level Affects Cardiovascular and
Salivary Alpha Amylase Responses to Acute Psychosocial Stress. Sport. Med. Open 2016, 2, 33. [CrossRef]
46. Trimmel, M. Relationship of Heart Rate Variability (HRV) Parameters Including PNNxx With the Subjective Experience of Stress,
Depression, Well-Being, and Every-Day Trait Moods (TRIM-T): A Pilot Study. Ergon. Open J. 2015, 8, 32–37. [CrossRef]
47. Zhaohua Wu, N.E.H. Ensemble Empirical Mode Decomposition: A Noise-Assited. Biomed. Tech. 2010, 55, 193–201.
Biosensors 2022, 12, 465 16 of 16

48. Voss, A.; Kurths, J.; Kleiner, H.J.; Witt, A.; Wessel, N.; Saparin, P.; Osterziel, K.J.; Schurath, R.; Dietz, R. The Application of
Methods of Non-Linear Dynamics for the Improved and Predictive Recognition of Patients Threatened by Sudden Cardiac Death.
Cardiovasc. Res. 1996, 31, 419–433. [CrossRef]
49. Huang, N.E.; Shen, Z.; Long, S.R.; Wu, M.C.; Snin, H.H.; Zheng, Q.; Yen, N.C.; Tung, C.C.; Liu, H.H. The Empirical Mode
Decomposition and the Hilbert Spectrum for Nonlinear and Non-Stationary Time Series Analysis Analysis. Proc. R. Soc. A Math.
Phys. Eng. Sci. 1998, 454, 903–995. [CrossRef]
50. Zunino, L.; Olivares, F.; Scholkmann, F.; Rosso, O.A. Permutation Entropy Based Time Series Analysis: Equalities in the Input
Signal Can Lead to False Conclusions. Phys. Lett. Sect. A Gen. At. Solid State Phys. 2017, 381, 1883–1892. [CrossRef]
51. Bandt, C.; Pompe, B. Permutation Entropy: A Natural Complexity Measure for Time Series. Phys. Rev. Lett. 2002, 88, 4. [CrossRef]
52. Berger, S.; Schneider, G.; Kochs, E.F.; Jordan, D. Permutation Entropy: Too Complex a Measure for EEG Time Series? Entropy 2017,
19, 692. [CrossRef]
53. Riedl, M.; Müller, A.; Wessel, N. Practical Considerations of Permutation Entropy: A Tutorial Review. Eur. Phys. J. Spec. Top. 2013,
222, 249–262. [CrossRef]
54. Richman, J.S.; Moorman, J.R. Physiological Time-Series Analysis Using Approximate Entropy and Sample Entropy Maturity in
Premature Infants Physiological Time-Series Analysis Using Approximate Entropy and Sample Entropy. Am. J. Physiol. Hear. Circ.
Physiol. 2000, 278, H2039–H2049. [CrossRef]
55. Brugnera, A.; Zarbo, C.; Tarvainen, M.P.; Marchettini, P.; Adorni, R.; Compare, A. Heart Rate Variability during Acute Psychosocial
Stress: A Randomized Cross-over Trial of Verbal and Non-Verbal Laboratory Stressors. Int. J. Psychophysiol. 2018, 127, 17–25.
[CrossRef]
56. Pecchia, L.; Castaldo, R.; Montesinos, L.; Melillo, P. Are Ultra-Short Heart Rate Variability Features Good Surrogates of Short-Term
Ones? State-of-the-Art Review and Recommendations. Healthc. Technol. Lett. 2018, 5, 94–100. [CrossRef]
57. Parseval, M.-A. Mémoire Sur Les Séries et Sur L’Intégration Complète d’une Équation Aux Différences Partielles Linéaires
Du Second Ordre, à Coefficients Constants. In Mémoires Présentés Par Divers Savants À L’académie Des Sciences De L’institut De
France/Sciences Mathématiques Et Physiques; Nabu Press: Charleston, SC, USA, 1806; Volume 1, pp. 638–648.
58. Kononenko, I. Estimating Attributes: Analysis and Extensions of RELIEF. In European Conference on Machine Learning; Springer:
Berlin/Heidelberg, Germany, 1994; pp. 171–182. [CrossRef]
59. Cortes, C.; Vapnik, V. Support-Vector Networks. Mach. Learn. 1995, 20, 273–297. [CrossRef]
60. Suykens, J.A.K.; Vandewalle, J. Least Squares Support Vector Machine Classifiers. Neural Process. Lett. 1999, 9, 293–300. [CrossRef]
61. Gholamiangonabadi, D.; Kiselov, N.; Grolinger, K. Deep Neural Networks for Human Activity Recognition with Wearable
Sensors: Leave-One-Subject-Out Cross-Validation for Model Selection. IEEE Access 2020, 8, 133982–133994. [CrossRef]
62. Sokolova, M.; Japkowicz, N.; Szpakowicz, S. Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for
Performance Evaluation. In Australasian Joint Conference on Artificial Intelligence; Springer: Berlin/Heidelberg, Germany, 2006;
pp. 1015–1021. ISBN 3540307370.
63. Hopkins, W.G. A Scale of Magnitudes for Effect Statistics. A New View Stat. 2002, 502, 411.
64. Esco, M.R.; Williford, H.N.; Flatt, A.A.; Freeborn, T.J.; Nakamura, F.Y. Ultra-Shortened Time-Domain HRV Parameters at Rest and
Following Exercise in Athletes: An Alternative to Frequency Computation of Sympathovagal Balance. Eur. J. Appl. Physiol. 2018,
118, 175–184. [CrossRef]
65. Salahuddin, L.; Cho, J.; Jeong, M.G.; Kim, D. Ultra Short Term Analysis of Heart Rate Variability for Monitoring Mental Stress in
Mobile Settings. In Proceedings of the 29th Annual International Conference of the IEEE Engineering in Medicine and Biology
Society, Lyon, France, 23–26 August 2007; IEEE: Piscataway, NJ, USA, 2007; pp. 4656–4659.