Article

Depression, Anxiety and Stress Scales (DASS-21):

Construct Validity Problem in Hispanics
Juan Aníbal González-Rivera * , Orlando M. Pagán-Torres and Emily M. Pérez-Torres
School of Behavioral and Brain Sciences, Ponce Health Sciences University, 388 Zona Industrial Reparada 2,
Ponce, PR 00716, USA; orlando.m.pagan.torres@gmail.com (O.M.P.-T.); emilyperez@psm.edu (E.M.P.-T.)
* Correspondence: jagonzalez@psm.edu




Received: 30 October 2019; Accepted: 4 January 2020; Published: 8 January 2020


Abstract: The main purpose of this research was to examine the construct validity of the Depression,
Anxiety and Stress Scales (DASS-21) in order to determine whether it is able to adequately discriminate
between symptoms of depression and anxiety in the Hispanic population in Puerto Rico. This study
has an instrumental design. A total of 1073 Hispanics participated in this psychometric study.
The results showed that the DASS-21 has serious psychometric deficiencies, especially related to the
construct validity, as well as convergent and discriminatory validity. In addition, it was shown that
DASS-21 do not replicate the three-dimensional structure of the original instrument in the Hispanic
community. Finally, it was confirmed that the DASS-21 have difficulty in properly identifying and
discriminating between symptoms associated with depression and anxiety in a Hispanic population.

Keywords: anxiety; DASS-21; depression; stress; psychometric properties; validity

1. Introduction
Mood disorders have the highest prevalence worldwide when compared to all other mental
disorders, followed by anxiety disorders [1–3]. The scientific literature has thoroughly documented
the comorbidity between anxiety and depression disorders in a variety of studies [4–6]. In fact,
between 40% and 70% of people diagnosed with depression have been shown to simultaneously
meet the diagnostic criteria for an anxiety disorder [7]. Similar studies have empirically shown the
significant association between anxiety and depression, which is why it is often difficult to identify,
treat, and distinguish both diagnoses [8].
For this reason, it is essential that at the beginning of any treatment, mental health professionals
evaluate the presence and severity of symptoms associated with depression and anxiety in their
patients [6]. Valid and reliable clinical measurement instruments are required to facilitate the diagnosis
and treatment of people with simultaneous symptoms. This is only possible with instruments that
properly discriminate between symptoms of anxiety and depression. In the case of Puerto Rico,
very few researchers have conducted studies to analyze the psychometric properties of the most
commonly used depression and anxiety measures in clinical scenarios [9–11]. Therefore, we do not
find instrumental studies that evaluate the psychometric properties of the Depression, Anxiety and
Stress Scales (DASS-21) in Puerto Rico, which is an instrument that in recent years has taken on become
much more widely used for Spanish-speaking populations [12–15].
Given this lack of studies and with the certainty that DASS-21 are being used in Puerto Rico [16–18],
several questions arise: Will the DASS-21 have adequate psychometric properties for the Puerto Rican
population? Will they have the ability to adequately discriminate between clinical symptoms associated
with depression and anxiety? To answer these questions, our study will analyze the construct validity
of the Hispanic version of DASS-21 in a Puerto Rican adult sample.

Eur. J. Investig. Health Psychol. Educ. 2020, 10, 375–389; doi:10.3390/ejihpe10010028 www.mdpi.com/journal/ejihpe
Eur. J. Investig. Health Psychol. Educ. 2020, 10 376

1.1. Prevalence of Anxiety and Depression in Puerto Rico

The Center for Disease Control [19] found that the overall prevalence of depression in Puerto Rico
was 16.8% in 2011, 16.3% in 2012, 18.8% in 2013, and 18.5% in 2014. More recent findings revealed
that about 10.4% of the Puerto Rican adult population suffers from some form of mood disorder,
while 12.5% of that population experiences some form of anxiety disorder [20]. On the other hand,
comparison analyses between Puerto Ricans living on the island and Puerto Ricans living in the United
States (USA) showed that Puerto Ricans from the island had similar rates of psychiatric disorders
to Puerto Ricans living in the U.S. [21]. Puerto Ricans in the USA had specifically higher levels of
anxiety and depression, but not psychiatric disorders in general, compared to those on the island.
Furthermore, among young Puerto Ricans, 13.4% of adolescents were found to suffer from major
depression, 8.3% from suicidal ideation, and 6.9% from behavioral disorders [22].

1.2. Instruments that Measure Depression and Anxiety in Puerto Rico

The most widely used measuring instrument in Puerto Rico to examine symptoms of depression
in adults is the Beck Depression Inventory (BDI; BDI-II) [23]. This instrument was validated in Puerto
Rico with 300 participants and obtained an internal Cronbach alpha consistency of 0.89 [9]. A revised
version of the BDI was later developed in Puerto Rico according to the DSM-IV diagnostic criteria [24].
The psychometric properties of the instrument were examined with a sample of 351 Puerto Rican
university students, and revealed an internal consistency of 0.88. Later, other Puerto Rican researchers
assessed the reliability of the BDI-II and revealed an internal consistency of 0.91 [10]. In addition,
the Rodríguez-Gómez Hispanic Depression Questionnaire was developed with a sample of Puerto
Rican elders in 2003 [25]. In 2005, it was administered in children and adolescents, and obtained a
Cronbach’s alpha for internal consistency of 0.82 [26]. That same year, the applicability of the Spanish
version of the Zulf Self-Rating Depression Scale was studied in a sample of 258 Puerto Rican adults,
and obtained an alpha coefficient of 0.85 [27].
The efforts made in Puerto Rico to developing and validating instruments to measure depressive
symptomatology in the adolescent population have been remarkable. For example, in 2008 the
Self-Efficacy Scale for Depression in Adolescents was developed and validated with 116 participants,
and obtained a Cronbach’s alpha for internal consistency of 0.90 [28]. Ten years later, it was validated
using a sample of 51 Puerto Rican teens with Type 1 Diabetes, and obtained a Cronbach’s alpha
for internal consistency rate of 0.93 for the total scale and between 0.71 to 0.85 for sub-scales [29].
Other Puerto Rican authors validated the Inventory for the Spectrum Assessment of Depressive
Symptomatology with 201 adolescents by revealing a Cronbach’s alpha for internal consistency of
0.98 [30]. Finally, preliminary data from the Children Depression Inventory-2 on 51 Puerto Rican teens
revealed a reliability of 0.84 for the total scale [31].
Regarding instruments measuring anxiety symptomatology in Puerto Rico, the internal consistency
of the Beck Anxiety Inventory (BAI) was evaluated in 2001, which obtained a Cronbach alpha of
0.94 [11]. Later in 2016, the psychometric properties of the BAI were evaluated, and revealed an internal
consistency of 0.95 [10]. Finally, the Generalized Anxiety Scale-7(GAD-7) revealed a Cronbach’s alpha
for internal consistency between 0.91 to 0.92 in studies with Hispanics in Puerto Rico [32,33].

1.3. Description of Depression, Anxiety, and Stress Scales

Lovibond and Lovibond [34], DASS-21 authors, intended to develop an instrument capable of
measuring symptomatology associated with depression and anxiety, and that could simultaneously
discriminate between these constructs. For the development of the instrument, the authors included
clinical and diagnostic symptoms of depression and anxiety, and excluded symptoms that may be
present in both disorders, such as changes in appetite. However, factorial analyses in their first
validations yielded a third factor (stress) that, according to the authors, gathers symptoms associated
with difficulty relaxing, nervous tension, irritability, and agitation [34]. In this way, the first version of
Eur. J. Investig. Health Psychol. Educ. 2020, 10 377

the instrument, the DASS-42 (long version of 42 items) was born, which is one of the most widely used
tools in the world to measure affective symptoms.
This instrument comprises three scales: (1) the depression scale, which measures hopelessness,
low self-esteem, and low positive affection; (2) the anxiety scale, which evaluates autonomic arousal,
musculoskeletal symptoms, situational anxiety, and the subjective experience of anxious arousal; and
(3) the stress scale, which measures tension, agitation, and negative affection. The instrument items refer to
the previous week and each item is classified into four Likert responses from 0, which means “nothing” to
3, which means “Most of the time”. A short version of the DASS was later developed, which has now been
recognized as the DASS-21 [34]. This instrument contains seven selected items from each of the scales.

1.4. Psychometric Properties in Other Countries and Populations

DASS-21 has been translated into Spanish and validated into Hispanic populations, revealing
adequate psychometric properties [12–15]. The short version has shown adequate psychometric properties
in validation studies of adults, the general-population, and in clinical samples. Similarly, previous research
indicates that DASS-21 has a strong internal consistency and provides an adequate distinction between
anxiety and depression, relative to other existing measures [12]. The authors of the instrument reported
that the scales had an adequate convergent and discriminatory validity [34].
Regarding the factorial structure of the instrument, the literature is broad and diverse, showing
that factorial structures can fluctuate depending on the sociocultural context and the type of population
in which the instrument is being administered. For example, in Latin American countries, DASS-21 has
revealed to fit well with a three-factor structure [12,35,36]. This is consistent with findings found
in 2,630 Asian participants from Malaysia, Indonesia, Singapore, Taiwan, and Thailand who, after
removing three items, identified a three-factor structure (DASS-18) [37]. The study authors argue that
the original DASS-21 may not be applicable in Asian samples and therefore suggest that the instrument
could be presented differently. Also, in eastern regions such as China, Iran, and Greece, a three-factor
model was best suited in a sample of non-clinical Greeks [38–40].
Moreover, in a study with American teenagers, a four-factor structure was found, involving a new
factor called negative affection [41]. This is consistent with findings in a Portuguese community [42]
and in a sample of Vietnamese adolescents [43], where they found an additional factor of psychological
distress (negative affectivity). Also, a sample of patients with brain trauma found a four-factor
structure, with distress overall being the fourth factor [44]. The authors of these studies concluded that
psychological distress is a factor underlying depression, anxiety, and stress. Other studies with clinical
and non-clinical samples in the USA [45], Brazil [46], and Italy [47] have found two-factor models in
DASS-21. In fact, a recent systematic review underpins the plausibility of an overall factor underlying
depression and anxiety [48]. Also, the authors support the potential usefulness of a two-factorial model.
As documented, the factorial structure of DASS has evolved into a variety of structures depending on
the context. Examples of this are DASS-14 [49] and DASS-9 [40].
It is notable that DASS-21 has not been used in experimental studies with Latinos in which
the effectiveness of Cognitive Behavioral Therapy in anxiety disorders has been evaluated [50].
Nor was the use of DASS-21 identified in randomized studies conducted in Puerto Rico until 2015 [51].
However, the instrument was used in a pilot intervention study showing a Cronbach’s alpha for
internal consistency of 0.94 for the total scale with 32 adults [18]. In relation to non-experimental studies
conducted in Puerto Rico, DASS has revealed an internal consistency between 0.87 and 0.90 [16,17,52,53].
However, our literature review revealed that, so far, the factorial structure and construct validity of
DASS-21 in Puerto Rico Hispanics have not been examined.

1.5. Purpose of the Study

The main purpose of this study was to examine the construct validity of the DASS-21 in order to
determine whether it is able to adequately discriminate between symptoms of depression and anxiety
Eur. J. Investig. Health Psychol. Educ. 2020, 10 378

in the Hispanic population in Puerto Rico. To achieve our goal, several construct validity analyses
were performed using advanced statistics. Specifically, this study had three main objectives:

1. Analyze the factorial structure of the DASS-21 by using an exploratory factor analysis to identify
the dimensions behind the 21 items.
2. Perform a confirmatory factor analysis to examine whether the original three-factor model
has a good fit in Puerto Rico Hispanics and analyze whether the factors maintain adequate
independence between them.
3. Analyze the convergent and divergent validity of the three DASS-21 scales using the extracted
mean variance analysis.

2. Methods

2.1. Research Design

This study employed an instrumental design [54] by using exploratory and confirmatory factor
analyses with structural equations to examine the construct validity of DASS-21. This study was approved
by the Institutional Ethics for Research Committee of the Carlos Albizu University, San Juan Campus,
Puerto Rico. The data compilation was carried out by using online questionnaires through the PsychData
platform and posting a paid ad in the main social networks as a recruitment method: FB, Twitter, Google+,
and WhatsApp, among other platforms. This ad redirected the participants to the online survey, where
they read the informed consent, which notified them of the following: (a) the purpose of the study,
(b) inclusion criteria, (c) the voluntary nature of the study, (d) possible risks and benefits, and (e) their
right to withdraw from the study at any time. To guarantee the privacy and confidentiality of the
participants, the questionnaires were completed anonymously, and they were able to print a copy of the
informed consent.

2.2. Participants
The process for selecting participants was by non-probabilistic availability. The sample of
this research consisted of 1,073 participants recruited electronically. In Table 1 we present the full
sociodemographic distribution. The age of the participants ranged from 21 to 77 years of age with an
average of 37.68 years and a standard deviation of 11.69.

Table 1. Sociodemographic data of the sample.

n %
Sex
Female 818 76.2%
Male 225 23.8%
Academic Preparation
High school or less 66 6.2%
Associate degree/technical 210 19.6%
Bachelor’s degree 462 43.1%
Master’s degree 253 23.6%
Doctoral degree 82 7.6%
Civil Status
Marriage
566 52.7%
Single
187 17.4%
Cohabiting (free union)
320 29.8%

Annual Income
$0–25,000 588 54.8%
$26,000–50,000 320 29.8%
$51,000–75,000 108 10.1%
$76,000–100,000 36 3.4%
$101,000 or more 21 2.0%
Note: N = 1073.
Eur. J. Investig. Health Psychol. Educ. 2020, 10 379

2.3. Measurement
To identify the sociodemographic characteristics of the sample, we developed a general data
questionnaire composed of relevant data such as age, sex, academic preparation, civil status,
and annual income.
An abbreviated version of the Depression, Anxiety and Stress Scales (DASS-21) was used. This scale
was developed by Lovibond and Lovibond [34]. The three-dimensional self-reporting scales assess
the presence and intensity of affective states of depression, anxiety, and stress. Each item is answered
according to the presence and intensity of each symptom in the last week on a 4-point Likert response
scale, the limits of which are the answer nothing and the answer most of the time. Each scale has
seven items and its total score is calculated with the sum of the items belonging to that scale and varies
between 0 and 21 points. A higher score indicates a higher participant symptomatology. Items 1, 6, 8,
11, 12, 14, and 18 belong to the stress scale, items 3, 5, 10, 13, 16, 17, and 21 to the depression scale,
and items 2, 4, 7, 9, 15, 19, and 20 to the anxiety scale.

2.4. Data Analysis

Once the data was collected, we analyzed it using the IBM SPSS version 24.0 statistical analysis
system. Specifically, descriptive sample analysis, exploratory factor analysis, reliability analysis,
and factor correlation analysis were performed. For exploratory factor analysis, the method of
extraction of main axes with oblique rotation was used to identify the latent variables underlying the
items. This adjustment procedure was used for two main reasons: (1) the main axis extraction method
is not based on the normality scenario [55], and (2) oblique rotation is more accurate and provides more
information than rotation octagonal [56]. For factor identification, we used two criteria: (a) each factor
must explain 5% or more of the variance [57]; and (b) each item must have a factorial load greater than
0.30 in a single factor [58].
The STATA version 14.1 statistical program was used for confirmatory factor analysis, with the
maximum likelihood estimation method and the corrections of Satorra and Bentler [59]. To evaluate
the adjustment of the models, we used the following adjustment indexes: Chi-square test (χ2 ), root
mean square error of approximation (RMSEA), Tucker–Lewis Index (TLI), Comparative Fit Index (CFI),
and Akaike Information Criterion (AIC). RMSEA values less than 0.05 indicate an adequate adjustment
of the model [60]. Likewise, CFI and TLI values greater than 0.90 represent an adequate adjustment of
the model [60]. AIC was used to examine the parsimony and compare the models, where the model
with the lower index shows a lower adjustment [61]. Meanwhile, the regression coefficients of each
item on its respective factor should exceed 0.50 to be considered adequate [62]. The correlation between
the instrument factors was calculated using Pearson’s product-moment coefficient (r). Values less than
0.35 were considered to be weak or low correlations, values between 0.36 and 0.67 were considered
moderate correlations, values between 0.68 and 0.89 were seen as high correlations and, finally, values
from 0.90 onwards were considered to be very high correlations [63].
In addition, following the recommendations of Fornell and Larcker [64], we examined the
convergent and discriminatory validity of DASS-21 through the Average Variance Extracted (AVE).
To support convergent validity, the AVE must be equal to or greater than 0.50, thus establishing that
more than 50% of the construct’s variance is due to its indicators [65]. For its part, in order to determine
the discriminatory validity of each dimension, the Maximum Shared Variance (MSV) and the Average
Shared Variance (ASV) must be less than the value obtained from the individual AVE of each factor.
Finally, confirmatory factor analyses were computed to compare the following five factor models of
the DASS-21: (a) a one-factor model; (b) a one-factor model with depression and anxiety loading on
the same factor; (c) a two-correlated-factor model with depression and stress items loading on the
same factor; (d) a two-correlated-factor model with anxiety and stress loading on the same factor; (e) a
three-correlated-factor model (original model).
Eur. J. Investig. Health Psychol. Educ. 2020, 10 380

3. Results

3.1. Structure Validity: Exploratory Factor Analysis

To determine the factorial structure of DASS-21 and identify the underlying dimensions behind its
21 items, several exploratory factorial analyses were performed. The first analysis showed a three-factor
structure that explained 63% of the variance of the original data. The Kaiser-Meyer-Olkin (KMO) test
supported the adequacy of sampling data for the analysis, KMO = 0.962. Bartlett’s sphericity test
was significant, X2 (210) = 15,217.489, p < 0.001, indicating that the correlations between the reagents
were significantly different from zero, thereby providing an additional indicator of the adequacy
for factor analysis. However, when the distribution of items by factor was reviewed, we identified
that most depression and stress items were grouped into the first factor, three depression items were
accommodated in factor 2, and the seven anxiety items were grouped into factor 3 (see Table 2). In the
item retention process, item 2 was removed by loading more than 0.30 into two factors. Items 10, 17,
and 21 were also removed for negative charges on factor 2.

Table 2. Distribution of items in the four exploratory factorial analyses.

1st Analysis 2nd Analysis 3rd Analysis 4th Analysis

1 2 3 1 2 1 2 1
Depression
Item 3 0.62 0.65 0.64 0.68
Item 5 0.41 0.42 0.41 0.61
Item 10 −0.79
Item 13 0.66 0.72 0.70 0.75
Item 16 0.48 0.52 0.50 0.69
Item 17 −0.58
Item 21 −0.85
Anxiety
Item 2 0.31 0.31
Item 4 0.67 0.61 0.63 0.66
Item 7 0.82 0.84 0.79 0.65
Item 9 0.64 0.73 0.70 0.76
Item 15 0.58 0.70 0.71 0.82
Item 19 0.60 0.63 0.65 0.72
Item 20 0.53 0.72 0.73 0.78
Stress
Item 1 0.76 0.76 0.77
Item 6 0.62 0.64 0.64
Item 8 0.67 0.67
Item 11 0.56 0.60 0.59
Item 12 0.86 0.89 0.90
Item 14 0.68 0.70 0.69
Item 18 0.79 0.82 0.80

With the remaining 17 items, the second exploratory factor analysis was performed showing
a two-factor structure that explained 59% of the variance of the original data, KMO = 0.961;
χ2 (136) = 11,553.959, p < 0.001. Most of the depression and stress items loaded back into the first
factor and six of the anxiety items loaded on the second factor. Item 8 was removed for loading in a
non-corresponding factor. The remaining 16 items underwent a third exploratory factor analysis that
again showed a two-factor structure that explained 61% of the variance of the original data, KMO = 0.958;
χ2 (120) = 10569.397, p < 0.001. Finally, an exploratory factor analysis was performed where we eliminated
all stress items with the intention of examining whether the anxiety and depression items discriminate
against each other. The results showed a one-dimensional structure that explained 59% of the variance of
the original data, KMO = 0.944; χ2 (136) = 6275.645, p < 0.001. This analysis provides evidence of the lack
of discrimination between depression and anxiety items in Puerto Rico’s Hispanic community. None of the
analyses replicated the factorial distribution of the original instrument version.
Eur. J. Investig. Health Psychol. Educ. 2020, 10 381

3.2. Structure Validity: Confirmatory Factor Analysis

We then performed a confirmatory factor analysis using the maximum likelihood estimation
method and the Satorra and Bentler corrections [59] to verify whether the instrument maintains its
theoretical and three-dimensional structure in Hispanics of Puerto Rico. The model examined was
composed of three latent factors (anxiety, depression, and stress) with each item. The results showed
an inadequate adjustment: χ2 = 2119.229 (186) p < 0.001, RMSEA = 0.10, CFI = 0.87, TLI = 0.86, χ2
sb = 1392.490 (186) p < 0.001, RMSEA sb = 0.08, CFI sb = 0.88, TLI sb = 0.86. These indices did not meet
acceptable levels of adjustment [59]. In turn, we examined the regression coefficients of each item,
which ranged from 0.51 to 0.83 (see Table 3).

Table 3. Regression coefficients (β) on its respective scales, and confidence intervals.

Items β 95% C.I.

Depression
3. I could not seem to experience any positive feeling at all. 0.70 [0.67, 0.74]
5. I found it difficult to work up the initiative to do things. 0.60 [0.55, 0.64]
10. I felt that I had nothing to look forward to. 0.80 [0.76, 0.83]
13. I felt down-hearted and blue. 0.79 [0.76, 0.81]
16. I was unable to become enthusiastic about anything. 0.72 [0.68, 0.76]
17. I felt I was not worth much as a person. 0.80 [0.76, 0.83]
21. I felt that life was meaningless. 0.80 [0.76, 0.83]
Anxiety
2. I was aware of dryness of my mouth. 0.51 [0.46, 0.57]
4. I experienced breathing difficulty. 0.67 [0.62, 0.72]
7. I experienced trembling (e.g., in the hands). 0.67 [0.61, 0.72]
9. I was worried about situations in which I might panic and make a fool of myself. 0.78 [0.74, 0.81]
15. I felt I was close to panic. 0.83 [0.80, 0.86]
19. I was aware of the action of my heart in the absence of physicalexertion. 0.74 [0.70, 0.77]
20. I felt scared without any good reason. 0.79 [0.75, 0.82]
Stress
1. I found it hard to wind down. 0.64 [0.60, 0.68]
6. I tended to over-react to situations. 0.70 [0.66, 0.73]
8. I felt that I was using a lot of nervous energy. 0.72 [0.69, 0.76]
11. I found myself getting agitated. 0.82 [0.79, 0.85]
12. I found it difficult to relax. 0.80 [0.78, 0.83]
14. I was intolerant of anything that kept me from getting on with what I was doing. 0.75 [0.71, 0.78]
18. I felt that I was rather touchy. 0.80 [0.78, 0.83]
Note: β = standardized regression coefficients; p = significance; 95% C.I. = 95% confidence intervals of
regression coefficients.

3.3. Convergent and Discriminant Validity

Both discriminant and convergent validity were examined through the average variance extracted
(AVE). This method indicates the variance explained by the construct in the items. The higher the value
of the AVE, the lower the error variance. The AVE values obtained for the factors ranged between
0.52–0.56 (see Table 4). For the AVE to be considered as acceptable, the scores must be equal to or
greater than 0.50 [64,65]. On the other hand, for there to be evidence of discriminatory validity, the MSV
and the ASV must be less than the value obtained from the AVE. However, the results showed that
the MSV and ASV values exceeded the AVE of all three scales (see Table 4). This means that the three
scales share a substantial amount of variance with each other. This is confirmed by observing high
correlations between latent variables (ranged between 0.86–0.88), as well as correlations of the direct
scores (ranged between 0.77–0.80).
Eur. J. Investig. Health Psychol. Educ. 2020, 10 382

Table 4. Means, standard deviations, alphas, omega coefficient, average variance extracted,
and correlations.

M SD α ω AVE MSV ASV 1 2 3

1.
4.78 4.87 0.89 0.89 0.54 0.77 0.77 - 0.88 ** 0.87 **
Depression
2. Anxiety 3.81 4.54 0.88 0.88 0.52 0.77 0.76 0.79 ** - 0.86 **
3. Stress 6.98 5.32 0.90 0.90 0.56 0.76 0.75 0.80 ** 0.77 ** -
Note. M = Mean; SD = standard deviation; α = Cronbach’s alpha coefficient; ω = omega coefficient; AVE = average
variance extracted; MSV = maximum shared variance; ASV = average shared variance; ** = significant correlations
p < 0.001. The values on the diagonal represent the correlations between the latent factors, while the values below
the diagonal represent the correlations of the direct scores.

3.4. DASS-21 Alternative Models

Since the three-dimensional model of the DASS-21 did not obtain an adequate fit, other competitive
model were examined. These alternative models were based on exploratory factorial analyses
conducted in our study and previous research conducted in Spanish-American contexts [12,35,36].
Specifically, six competitive models were evaluated: the original model of three factors (M1); a
unifactorial model where the 21 original items were loaded to one factor (M2); a one-factor model
(10 items) with depression and anxiety loading on the same factor (M3; obtained from the 4th
exploratory factor analysis of this study) (see Figure 1); a one-factor model (14 items) with depression
and anxiety loading on the same factor (M4); a two-correlated-factor model with depression and stress
items loading on the same factor (M5); and a two-correlated-factor model with anxiety and stress
loading on the same factor (M6). The M1, M2, M4, M5, and M6 did not show an adequate adjustment
to the data (see Table 5). The only model with adequate adjustment rates was the M3. The comparative
analysis provides additional evidence on the lack of discrimination between depression and anxiety
items in Puerto Rico’s Hispanic community.
Eur. J. Investig. Health Psychol. Educ. 2020, 10 383
Eur. J. Investig. Health Psychol. Educ. 2019, 1, x FOR PEER REVIEW 9

Figure 1. Model 3: One-factor model with depression and anxiety loading on the same factor.
Figure 1. Model 3: One-factor model with depression and anxiety loading on the same factor.
Table 5. Goodness-of-fit test for analyzed models.
Table 5. Goodness-of-fit test for analyzed models.
Model χ2 χ2 sb GL RMSEA RMSEA sb CFI CFI sb TLI TLI sb AIC
Model χ2 χ2sb GL RMSEA RMSEAsb CFI CFIsb TLI TLIsb AIC
M1 2119.229 1392.49 186 0.10 0.08 0.87 0.88 0.86 0.86 46,023.094
M2 M1 2748.703
2119.2291800.22
1392.49189 186 0.11 0.10 0.09 0.08 0.83 0.87 0.840.88 0.810.86 0.820.86 46,646.568
46,023.094
M3 *M2 791.755
2748.703 1800.2276
482.499 189 0.09 0.11 0.07 0.09 0.92 0.93 0.91 0.91
0.83 0.84 0.81 0.82 46,646.568 30,085.567
M4 1323.558 796.913 77 0.12 0.09 0.86 0.84 0.87 0.85 30,615.370
M5 M3 * 2442.991
791.7551602.66
482.499 188 76 0.11 0.09 0.08 0.07 0.85 0.92 0.860.93 0.830.91 0.840.91 46,342.856
30,085.567
M6 M4 2484.373 1631.30
1323.558 796.913 77 188 0.11 0.12 0.09 0.09 0.85 0.86 0.83 0.84
0.86 0.84 0.87 0.85 30,615.370 46,384.237
Note.
M5 * = adequate adjustment; sb
2442.991 1602.66 188 = Satorra–Bentler
0.11adjustments;
0.08χ 2 = Chi-square test; χ2 = Corrected Chi-square
0.85 0.86 sb0.83 0.84 46,342.856
test; GL = degrees of freedom; RMSEA = root mean square error of approximation; RMSEAsb = corrected RMSEA;
M6
CFI = Comparative
2484.373Fit Index; CFIsb = Corrected
1631.30 188 0.11
CFI; TLI = Tucker–Lewis
0.09 sb = Corrected
0.85 TLI0.86
Index; AIC = Akaike
0.83 TLI;0.84 46,384.237
Information Criterion; All statistics χ2 and χ2 sb are significant, p < 0.001.
Note. * = adequate adjustment; sb = Satorra–Bentler adjustments; χ = Chi-square test; χ sb= Corrected
2 2

Chi-square test; GL = degrees of freedom; RMSEA = root mean square error of approximation; RMSEAsb =
4. Discussion
corrected RMSEA; CFI = Comparative Fit Index; CFIsb = Corrected CFI; TLI = Tucker–Lewis Index; TLIsb =
The high prevalence of mood and anxiety disorders in Latin America shows the need for a certain
Corrected TLI; AIC = Akaike Information Criterion; All statistics χ2 and χ2sb are significant, p < 0.001.
number of validated instruments in our cultural context, which allow the detection and treatment of
these
4. symptoms.
Discussion However, the high comorbidity between depression and anxiety makes it difficult
to develop instruments that properly discriminate between symptoms. For this reason, the main
objectiveThe high
of this prevalence
study of mood the
was to examine andconstruct
anxiety disorders
validity ofinthe
Latin America
DASS-21 shows
in order to the need for a
determine
whether it is able to adequately discriminate between symptoms of depression and anxiety in theand
certain number of validated instruments in our cultural context, which allow the detection
treatment
Hispanic of these in
population symptoms. However,
Puerto Rico. the high
The results of comorbidity between depression
this study demonstrated that theand anxietyhas
DASS-21 makes
it difficult
serious to develop
psychometric instruments
deficiencies, that properly
especially discriminate
in matters related between symptoms.
to the construct For this
validity, reason,
as well as the
main objective of this study was to examine the construct validity of the DASS-21 in order to
determine whether it is able to adequately discriminate between symptoms of depression and anxiety
in the Hispanic population in Puerto Rico. The results of this study demonstrated that the DASS-21
Eur. J. Investig. Health Psychol. Educ. 2020, 10 384

convergent and discriminatory validity. These results will help us to offer a recommendation on its
possible use in clinical and research scenarios with Hispanic populations.
However, our study has some limitations. First, the sample gathered was a convenience one, so it
was not random. Second, it was not possible to establish the reliability of the instrument over time,
as it could only be done through its components. However, the advanced techniques that were used in
the study provided empirical strength to our results. Third, the procedure to collect the data was not
standardized, and this may have affected the study means and increased the standard error. As for the
strengths of the study, it is important to note that this study was the first in Puerto Rico to analyze the
psychometric properties of DASS-21 in such a broad and heterogeneous sample. Finally, performing a
confirmatory factor analysis with structural equations added value to our study. In a broader area,
our study strengthens the importance of continuously and repeatedly reviewing the performance
and psychometric properties of measurement instruments. In turn, it discredits the practice of some
researchers in assuming that the psychometric characteristics of scales in social and behavioral sciences
are consistent across time and culture. This implies the need to carefully and deeply analyze the
psychometric properties of a measuring instrument in each population, culture, and/or country used.
Regarding our results, exploratory and confirmatory factorial analyses showed that DASS-21 does
not replicate the three-dimensional structure or factorial distribution found in other research in
international contexts [13,34,66,67]. This confirms that the internal structure of DASS-21 fluctuates
depending on the socio-cultural context where the instrument is administered. Even the results of
other studies with samples of Hispanics advocating for a three-dimensional model are questionable
and inconclusive [12,35]. For example, in the relevant study of Chile, they carried out an exploratory
factor analysis with a solution restricted to three factors, and found that five items presented significant
or relevant loads in two factors and an item with a factorial weight that was below the expected
level [12]. These items had to be removed in the validation process to perform further exploratory
factor analyses only with the remaining items. For their part, in a study of Mexico, researchers
performed an exploratory factorial analysis with varimax rotation that revealed six factors [35].
However, the authors decided to maintain the first three factors to preserve the theoretical consistency
of the instrument. Of the 21 items, only 14 were grouped into three factors. The authors had to perform
a new exploratory factorial analysis with these 14 items to confirm whether the three-dimensional
structure was maintained. For these reasons, we suggest taking the conclusions of these two studies
that preliminary state that the DASS-21 is reliable and valid for measuring anxiety and depression with
caution. In contrast, our findings argue that the original version of DASS-21 may not be applicable
in Hispanic samples and its three-dimensionality is highly questionable, so it cannot be said that the
scales are three distinct measures in Hispanic populations.
On the other hand, we find high correlations between the three instrument scales, both in the
correlation rates between the latent factors and between the direct scores of each scale. This finding is
not surprising given that studies using clinical scales of anxiety and depression often reflect moderately
high or high correlations between the two constructs [14,15,34,68]. This precisely indicates that the
impossibility of adequately discriminating between symptoms associated with depression and anxiety
is the biggest problem when using DASS-21 in Hispanic populations. This is confirmed by our analysis
of discriminatory validity, which showed that the three scales share a substantial amount of variance
with each other, subtracting evidence from the instrument’s construct validity. Also, despite the high
comorbidity between symptoms of anxiety and depression in clinical populations, our findings should
be interpreted as empirical evidence in favor of a one-dimensional structure in DASS-21. In fact,
our analysis of competitive models showed that the only model that was properly adjusted was the
one-dimensional model that collected all items from the anxiety and depression scales.
In theoretical terms, depression and anxiety manifest in a distinct way, but in empirical terms they
are very difficult to distinguish by using of self-reports. Hence there is a need to develop valid and
reliable clinical measurement instruments that facilitate the diagnosis and treatment of people who
have simultaneous symptoms of anxiety and depression. In this case, it appears that the DASS-21 does
Eur. J. Investig. Health Psychol. Educ. 2020, 10 385

not meet the requirements of such a need. So, if the one-dimensional structure was the most appropriate
one to explain the data, what model can construct measures or examine this one-factor dimension?
Some authors have suggested that depression and anxiety scales predominantly measure the common
factor of negative affectivity [35,69]. Negative affectivity, referred to in some studies as psychological
distress [42], reflects dispositional dimensions, where high negative affectivity is characterized by
subjective affliction and displaced feelings, and low negative affectivity is characterized by the absence
of these feelings [69]. In this sense, the use of the one-dimensional version of DASS could be justified
by researchers or clinicians to identify the presence of negative affectivity in individuals. Notably, some
studies in Puerto Rico have used it for this purpose [52,53]. However, it should not be used to
discriminate or differentiate between symptoms of anxiety and depression in Hispanics, at least until
there is greater psychometric evidence supporting this function.
In practical terms, the use of DASS-21 as an evaluation method in clinical scenarios is discouraged.
The lack of construct validity demonstrated in this study highlights the difficulty of DASS-21 identifying
symptomatology associated with different treatable mental disorders in psychotherapy. That is,
the scores obtained in the original version of the instrument would not represent weight indicators
that favor the psychological evaluation process. Regarding the internal consistency of the scales,
our results reflected acceptable values of reliability, all of which were above what the literature
suggested. However, the reliability of the DASS-21 should not be interpreted as evidence of construct
validity. Reliability is about certainty and not truthfulness [58]. In this sense, the reliability rates of
the three scales only allow us to know if the items measure the same phenomenon, which in this case
could be some aspect of negative affectivity and not necessarily depression, stress, or anxiety.
We recommend administering the DASS-21 to another sample of participants to perform the
cross-validation process again and test the factorial invariance of the instrument, as well as to evaluate
the concurrent validity of the instrument using other scales that measure depression, anxiety, and stress.
For example, BDI-II [70], BAI [71], or the Patient Health Questionnaire (PHQ-9) [72] could be used for
the validity process. Likewise, we advise examining the properties of DASS-21 in a clinical population,
as well as in an adolescent population in Puerto Rico. Finally, it would be interesting for future research
to examine whether the Hispanic population of Puerto Rico has a particular psychosocial condition
that explains the results of DASS-21 in Puerto Rico.

5. Conclusions
This study demonstrated that the DASS-21 has serious psychometric deficiencies, especially in
matters related to construct validity, as well as convergent and discriminatory validity. The findings
empirically demonstrated that the DASS-21 does not replicate the three-dimensional structure of the
original instrument in Puerto Rico’s Hispanic community. The results also suggest that the internal
structure of the instrument fluctuates depending on the socio-cultural context where the instrument is
administered. In Hispanics, it seems that the instrument is better suited to a one-dimensional model
that examines negative affectivity. Finally, this research confirmed the difficulty of the DASS-21 in
properly identifying and discriminating between symptoms associated with depression and anxiety in
Hispanic populations. It is recommended not to use DASS-21 with Hispanics in clinical and research
contexts, at least until there is greater psychometric evidence.

Author Contributions: Conceptualization, J.A.G.-R.; methodology, J.A.G.-R.; validation, J.A.G.-R.; formal analysis,
J.A.G.-R; investigation, J.A.G.-R., and O.M.P.-T.; resources, J.A.G.-R., O.M.P.-T. and E.M.P.-T.; data curation,
J.A.G.-R.; writing—original draft preparation, J.A.G.-R. and O.M.P.-T. writing—review and editing, J.A.G.-R.,
O.M.P.-T., and E.M.P.-T. visualization, J.A.G.-R. and O.M.P.-T.; supervision, J.A.G.-R.; project administration,
J.A.G.-R. All authors have read and agreed to the published version of the manuscript.
Funding: This research received no external funding.
Conflicts of Interest: The authors declare no conflict of interest.
Eur. J. Investig. Health Psychol. Educ. 2020, 10 386

References
1. American Psychiatric Association, DSM-5 Task Force. Diagnostic and Statistical Manual of Mental Disorders:
DSM-5™, 5th ed.; American Psychiatric Publishing: Arlington, VA, USA, 2013.
2. Sadock, J.E.; Sadock, V.A.; Ruiz, P. Synopsis of Psychiatry: Behavioral Sciences/Clinical Psychiatry, 11th ed.;
Wolters Kluwer Health: Philadelphia, PA, USA, 2014.
3. World Health Organization. Depression and Other Common Mental Disorders Global Health Estimates; WHO:
Geneva, Switzerland, 2017; Available online: https://apps.who.int/iris/bitstream/handle/10665/254610/WHO-
MSD-MER-2017.2-eng.pdf (accessed on 11 September 2019).
4. Braam, A.W.; Copeland, J.R.; Delespaul, P.A.; Beekman, A.T.; Como, A.; Dewey, M.; Skoog, I.
Depression, subthreshold depression and comorbid anxiety symptoms in older Europeans. Results from the
EURODEP concerted action. J. Affect. Disord. 2014, 155, 266–272. [CrossRef] [PubMed]
5. Johansson, R.; Carlbring, P.; Heedman, A.; Paxling, B.; Andersson, G. Depression, anxiety and their
comorbidity in the Swedish general population: Point prevalence and the effect on health-related quality of
life. PeerJ 2013, 1, e98. [CrossRef] [PubMed]
6. Wu, Z.; Fang, Y. Comorbidity of depressive and anxiety disorders: Challenges in diagnosis and assessment.
Shanghai Arch. Psychiatry 2014, 26, 227–231. [PubMed]
7. Lamers, F.; Van-Oppen, P.; Comijs, H.C.; Smit, J.H.; Spinhoven, P.; Van-Balkom, A.J.; Penninx, B.W.
Comorbidity patterns of anxiety and depressive disorders in a large cohort study: The Netherlands Study of
Depression and Anxiety (NESDA). J. Clin. Psychiat. 2011, 72, 341–348. [CrossRef]
8. Duc-Tran, T.; Tran, T.; Fisher, J. Validation of the depression anxiety stress scales (DASS) 21 as a screening
instrument for depression and anxiety in a rural community-based cohort of northern Vietnamese women.
BMC Psychiatry 2013, 13, 1–7.
9. Bernal, G.; Bonilla, J.; Santiago, J. Confiabilidad interna y validez de construcción lógica de dos instrumentos
para medir sintomatología psicológica en una muestra clínica: El inventario de depresión de Beck y la lista
de cotejo de Sintomas-361. Rev. Lat. Psicol. 1995, 27, 207–229.
10. González-Barrios, P.; Morales-Rodríguez, C.; Merced-Morales, K.; Lampón, A.; González, R.; Martínez, K.
Dimensional assessment of anxiety in Puerto Rican patients: Evaluating applicability of psychological
questionnaires. P. R. Health Sci. J. 2016, 35, 134–141.
11. Rodríguez-Reynaldo, M.; Martínez-Lugo, M.; Rodríguez-Gómez, J. Estudio de las características psicométricas
del Inventario de Ansiedad Beck (en castellano) en una muestra de envejecidos puertorriqueños. Rev. Esp.
Geriatr. Gerontol. 2001, 36, 353–360. [CrossRef]
12. Antúnez, Z.; Vinet, E.V. Escalas de Depresión, Ansiedad y Estrés (DASS-21): Validación de la versión
abreviada en estudiantes universitarios chilenos. Ter. Psicol. 2012, 30, 49–55. [CrossRef]
13. Daza, P.; Novy, D.; Stanley, M.; Averill, P. The depression anxiety stress scale-21: Spanish translation and
validation with a hispanic sample. J. Psychopathol. Behav. Assess. 2002, 24, 195–205. [CrossRef]
14. Román, F.; Santibáñez, P.; Vinet, E. Uso de las Escalas de Depresión Ansiedad Estrés (DASS-21) como
instrumento de tamizaje en jóvenes con problemas clínicos. Acta Investig. Psicol. 2016, 6, 2325–2336.
[CrossRef]
15. Román, F.; Vinet, E.; Alarcón, A. Escalas de Depresión, Ansiedad y Estrés (DASS-21): Adaptación y
propiedades psicométricas en estudiantes secundarios de Temuco. Rev. Argent. Clin. Psic. 2014, 23, 179–190.
16. González-Rivera, J.A. Propiedades psicométricas de la Escala de Florecimiento en Puerto Rico. Rev. Evaluar
2018, 18, 30–43. [CrossRef]
17. Pagán-Torres, O.M.; González-Rivera, J.A. The association between religious coping and depressive
symptomatology in Puerto Rico: A cross-sectional study. J. Relig. Theol. 2019, 3, 1–19.
18. Rosselló, J.; Zayas, G.; Lora, V. Impacto de un adiestramiento en meditación en consciencia plena
(mindfulness) en medidas de ansiedad, depresión, ira y estrés y consciencia plena: Un estudio piloto.
Rev. Puertorriquena Psicol. 2016, 27, 62–78.
19. Center for Disease Control. Behavioral Risk Factor Surveillance System (BRFSS); Center for Disease Control:
Washington, DC, USA, 2014. Available online: https://www.cdc.gov/brfss/annual_data/2014/pdf/overview_
2014.pdf (accessed on 8 August 2019).
Eur. J. Investig. Health Psychol. Educ. 2020, 10 387

20. Canino, G.; Vila, D.; Santiago-Batista, K.; García, P.; Vélez-Báez, G.; Moreda-Alegría, A. Need Assessment
Study of Mental Health and Substance Use Disorders and Service Utilization Among Adult Population of
Puerto Rico. Behavioral Sciences Research Institute. 2016. Available online: http://www.assmca.pr.gov/
(accessed on 8 August 2019).
21. Canino, G.; Shrout, P.E.; NeMoyer, A.; Vila, D.; Santiago, K.M.; García, P.; Quiñones, A.; Cruz, V.; Alegría, M.
A comparison of the prevalence of psychiatric disorders in Puerto Rico with the United States and the Puerto
Rican population of the United States. Soc. Psychiatry Psychiatr Epidemiol. 2019, 54, 369–378. [CrossRef]
22. Moscoso-Álvarez, M.R.; Rodríguez-Figueroa, L.; Reyes-Pulliza, J.C.; Colón, H.M. Adolescentes de Puerto
Rico: Una mirada a su salud mental y su asociación con el entorno familiar y escolar. Rev. Puertorriquena Psicol.
2016, 27, 320–332.
23. Beck, A.T. Depression: Causes and Treatment; University of Pennsylvania Press: Philadelphia, PA, USA, 1970.
24. Bonilla, J.; Bernal, G.; Santos, A.; Santos, D. A revised spanish version of the Beck Depression Inventory:
Psychometric properties with a Puerto Rican sample of college students. J. Clin. Psychol. 2004, 60, 119–130.
[CrossRef]
25. Rodríguez-Gómez, J.R.; Nogueras, J.; Pérez, E. Cernimiento de depresión en las personas de edad avanzada en
Puerto Rico: Primer estudio piloto con el Cuestionario de Depresión Hispano Rodríguez-Gómez. Hospitales
2003, 16, 17–21.
26. Díaz-Díaz, V.; Rodríguez-Gómez, J.R.; Sayers, S.K. Estudio preliminar de las propiedades psicométricas del
Cuestionario de Depresión Hispano Rodríguez-Gómez. Cienc. De La Conducta 2005, 20, 143–162.
27. Martínez, K.; Guiot, H.M.; Casas-Dolz, I.; González-Tejera, G.; Colón-De-Martí, L.N. Applicability of the
spanish translation of the Zung Self Rating Depression Scale in a general Puerto Rican population. P. R.
Health Sci. J. 2003, 22, 179–185. [PubMed]
28. Díaz-Santos, M.; Cumba-Avilés, E.; Bernal, G.; Rivera-Medina, C. Desarrollo y propiedades psicométricas de
la Escala de Autoeficacia para la Depresión en Adolescentes. Interam. J. Psychol. 2008, 42, 218–427. [PubMed]
29. Pagán-Torres, O.M.; Cumba-Avilés, E.; Matos-Melo, A.L. Psychometric properties of the Escala de Autoeficacia
para la Depresión en Adolescentes (EADA) among Latino youth with type 1 diabetes. Diabetol. Int. 2018, 10,
126–137. [CrossRef] [PubMed]
30. Feliciano-López, V.; Cumba-Avilés, E. Propiedades psicométricas del inventario para la evaluación del
espectro de la sintomatología depresiva en adolescentes. Rev. Puertorriquena Psicol. 2014, 25, 260–278.
31. Camacho-Acevedo, Z.; Piñero-Meléndez, M.; Crespo-Ramos, G.; López-Robledo, Y.; Cumba-Avilés, E.;
Torres-Aponte, L. Estudio piloto para validar el Children’s Depression Inventory-2 en adolescentes de Puerto
Rico: Hallazgos cuantitativos. In Proceedings of the 2nd Behavioral Health Student Research Symposium in
Southern Puerto Rico, Ponce, Puerto Rico, 17 September 2015.
32. Morales-Cruz, J.; Bernal, G.; Amaral-Figueroa, M. Impacto y viabilidad de una intervención con activación
conductual mediante actividad física para sobrevivientes de cáncer de mama. Rev. Puertorriquena Psicol.
2017, 28, 116–132.
33. Scaramutti, C.; Salas-Wright, C.P.; Vos, S.R.; Schwartz, S.J. The mental health impact of Hurricane Maria on
Puerto Ricans in Puerto Rico and Florida. Disaster Med. Public Health Prep. 2019, 13, 24–27. [CrossRef]
34. Lovibond, P.F.; Lovibond, S.H. The structure of negative emotional states: Comparison of the Depression
Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories. Behav. Res. Ther. 1995, 33,
335–343. [CrossRef]
35. Gurrola-Peña, G.M.; Balcázar-Nava, P.; Bonilla-Muños, M.P.; Virseda-Heras, J.A. Estructura factorial y
consistencia interna de la escala Depresión Ansiedad y Estrés en una muestra no clínica. Psicol. Y Cienc. Soc.
2006, 8, 3–7.
36. Ruiz, F.J.; García Martín, B.; Suárez Falcón, J.C.; Odriozola González, P. The hierarchical factor structure of
the spanish version of Depression Anxiety and Stress Scale-21. Int. J. Psychol. Psychol. Ther. 2017, 17, 97–105.
37. Oei, T.P.; Sawang, S.; Goh, Y.W.; Mukhtar, F. Using the depression anxiety stress scale 21 (DASS-21) across
cultures. Int. J. Psychol. 2013, 48, 1018–1029. [CrossRef]
38. Chan, R.C.; Xu, T.; Huang, J.; Wang, Y.; Zhao, Q.; Shum, D.H.; Potangaroa, R. Extending the utility of the
Depression Anxiety Stress scale by examining its psychometric properties in Chinese settings. Psychiatry Res.
2012, 200, 879–883. [CrossRef] [PubMed]
39. Jafari, P.; Nozari, F.; Ahrari, F.; Bagheri, Z. Measurement invariance of the Depression Anxiety Stress
Scales-21 across medical student genders. Int. J. Med. Educ. 2017, 8, 116–122. [CrossRef] [PubMed]
Eur. J. Investig. Health Psychol. Educ. 2020, 10 388

40. Kyriazos, T.A.; Stalikas, A.; Prassa, K.; Yotsidi, V. Can the Depression Anxiety Stress Scales Short be shorter?
Factor structure and measurement invariance of DASS-21 and DASS-9 in a Greek, non-clinical sample.
Psychology 2018, 9, 1095–1127. [CrossRef]
41. Szabó, M. The short version of the Depression Anxiety Stress Scales (DASS-21): Factor structure in a young
adolescent sample. J. Adolesc. 2010, 33, 1–8. [CrossRef] [PubMed]
42. Vasconcelos-Raposo, J.; Fernandes, H.M.; Teixeira, C.M. Factor structure and reliability of the depression,
anxiety and stress scales in a large Portuguese community sample. Span. J. Psychol. 2013, 16, 1–10. [CrossRef]
[PubMed]
43. Le, M.T.H.; Tran, T.D.; Holton, S.; Nguyen, H.T.; Wolfe, R.; Fisher, J. Reliability, convergent validity and factor
structure of the DASS-21 in a sample of Vietnamese adolescents. PLoS ONE 2017, 12, e0180557. [CrossRef]
44. Randall, D.; Thomas, M.; Whiting, D.; McGrath, A. Depression Anxiety Stress Scales (DASS-21): Factor
structure in Traumatic Brain Injury rehabilitation. J. Head Trauma Rehab. 2017, 32, 134–144. [CrossRef]
45. Osman, A.; Wong, J.L.; Bagge, C.L.; Freedenthal, S.; Gutierrez, P.M.; Lozano, G. The depression anxiety stress
Scales-21 (DASS-21): Further examination of dimensions, scale reliability, and correlates. J. Clin. Psychol.
2012, 68, 1322–1338. [CrossRef]
46. Silva, H.A.; Passos, M.H.; Oliveira, V.M.; Palmeira, A.C.; Pitangui, A.C.; Araújo, R.C. Short version of the
Depression Anxiety Stress Scale-21: Is it valid for Brazilian adolescents? Einstein 2016, 14, 486–493. [CrossRef]
47. Bottesi, G.; Ghisi, M.; Altoè, G.; Conforti, E.; Melli, G.; Sica, C. The Italian version of the Depression
Anxiety Stress Scales-21: Factor structure and psychometric properties on community and clinical samples.
Compr. Psychiatry 2015, 60, 170–181. [CrossRef]
48. Yeung, A.Y.; Yuliawati, L.; Cheung, S.-H. A Systematic Review and Meta-Analytic Factor Analysis of the
Depression Anxiety Stress Scales. 2019. Available online: Psyarxiv.com/bzhgk (accessed on 8 August 2019).
49. Wise, F.M.; Harris, D.W.; Olver, J.H. The DASS-14: Improving the construct validity and reliability of the
Depression, Anxiety, and Stress Scale in a cohort of health professionals. J. Allied Health 2017, 46, 85–90.
50. Bernal, G.; Adames, C.; Mariani, K.; Morales, J. Cognitive behavioral models, measures, and treatments
for anxiety disorders in latinos: A systematic review. In Treating Depression, Anxiety, and Stress in Ethnic
and Racial Groups: Cognitive Behavioral Approaches; Chang, E.C., Downey, C.A., Hirsch, J.K., Yu, E.A., Eds.;
American Psychological Association: Washington, DC, USA, 2018; pp. 149–177.
51. Bernal, G.; Adames, C. El estado de la ciencia de la psicoterapia en Puerto Rico: ¿Una copa media llena o
media vacía? Cienc. De La Conducta 2016, 30, 68–115.
52. González-Rivera, J.A.; Hernández, I.; Martínez, R.; Matos, J.; Galindo, M.; García, S. Interferencia de
la tecnología en las relaciones de pareja y su impacto en la salud mental de la mujer puertorriqueña.
Rev. Puertorriquena Psicol. 2018, 29, 56–71.
53. González-Rivera, J.A.; Segura, L.; Urbistondo, V. Phubbing en las relaciones románticas: Uso del celular,
satisfacción en la pareja, bienestar psicológico y salud mental. Interacciones 2018, 4, 81–91. [CrossRef]
54. Montero, I.; León, O.G. A guide for naming research studies in psychology. Int. J. Clin. Health Psychol. 2007,
7, 847–862.
55. Fabrigar, L.R.; Wegener, D.T.; MacCallum, R.C.; Strahan, E.J. Evaluating the use of exploratory factor analysis
in psychological research. Psychol. Methods 1999, 4, 272–299. [CrossRef]
56. Schmitt, T.A. Current methodological considerations in exploratory and confirmatory factor analysis.
J. Psychoeduc. Assess. 2011, 29, 304–321. [CrossRef]
57. Hatcher, L. A Step-by-Step Approach to Using the SAS System for Factor Analysis and Structural Equation Modeling;
SAS Institute Inc.: Cary, NC, USA, 1994.
58. DeVellis, R.F. Scale Development: Theory and Applications; Sage: California, CA, USA, 2017.
59. Satorra, A.; Bentler, P.M. A scaled difference chi-square test statistic for moment structure analysis.
Psychometrika 2001, 66, 507–514. [CrossRef]
60. Byrne, B.M. Structural Equation Modeling with AMOS: Basic Concepts, Applications, and Programming; Psychology
Press: New York, NY, USA, 2010.
61. Schumacker, R.E.; Lomax, R.G. A Beginner’s Guide to Structural Equation Modeling, 3rd ed.; Erlbaum: Mahwah,
NJ, USA, 2010.
62. Hair, J.F.; Black, W.C.; Babin, B.J.; Anderson, R.E.; Tatham, R.L. Multivariate Data Analysis, 6th ed.; Prentice-Hall
International: Upper Saddle River, NJ, USA, 2006.
Eur. J. Investig. Health Psychol. Educ. 2020, 10 389

63. Taylor, R. Interpretation of the correlation coefficient: A basic review. J. Diagn Med. Sonogr. 1990, 6, 35–39.
[CrossRef]
64. Fornell, C.; Larcker, D.F. Evaluating structural equation models with unobservable variables and measurement
error. J. Mark. Res. 1981, 18, 39–50. [CrossRef]
65. Fornell, C.; Bookstein, F.L. Two structural equation models: LISREL and PLS applied to consumer exit-voice
theory. J. Mark. Res. 1982, 19, 440–452. [CrossRef]
66. Bados, A.; Solanas, A.; Andrés, R. Psychometric properties of the Spanish version of depression, anxiety and
stress scales (DASS). Psicothema 2005, 17, 679–683.
67. Tully, P.J.; Zajac, I.T.; Venning, A.J. The structure of anxiety and depression in a normative sample of younger
and older Australian adolescents. J. Abnorm. Child Psychol. 2009, 37, 717–726. [CrossRef] [PubMed]
68. Lovibond, S.H.; Lovibond, P.F. Manual for the Depression Anxiety Stress Scales; Psychology Foundation: Sydney,
Australia, 1995.
69. Watson, D.; Clark, L. Negative affectivity. The disposition to experience aversive emotional states. Psychol. Bull.
1984, 96, 465–490. [CrossRef]
70. Beck, A.T.; Steer, R.A.; Brown, G.K. BDI-II. Beck Depression Inventory: Manual, 2nd ed.; The Psychological
Corporation: San Antonio, TX, USA, 1996.
71. Beck, A.T.; Epstein, N.; Brown, G.; Steer, R.A. An inventory for measuring clinical anxiety: Psychometric
properties. J. Consult. Clin. Psych. 1988, 56, 893–897. [CrossRef]
72. Kroenke, K.; Spitzer, R.L. The PHQ-9: A new depression and diagnostic severity measure. Psychiatr. Ann.
2002, 32, 509–521. [CrossRef]

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access
article distributed under the terms and conditions of the Creative Commons Attribution
(CC BY) license (http://creativecommons.org/licenses/by/4.0/).