Confirmatory Factor Analysis

& Structural Equation Models Course overview

Lecture 1: Overview & Path Analysis

Course notes & other materials will be avaiable at:

Michael Friendly http://datavis.ca/courses/CFA-SEM
Lecture 1: Setting the stage: EFA, CFA, SEM, Path analysis
Goal: Understand relations among a large number of observed variables
Goal: Extend regression methods to (a) multiple outcomes, (b) latent
SCS Short Course, May, 2019 variables, (c) accounting for measurement error or unreliability
Thinking: Equations → Path diagram → estimate, test, visualize
Lecture 2: Measurement models & CFA
Effects of measurement error
Testing equivalence of measures with CFA
Multi-factor, higher-order models
Lecture 3: SEM with latent variables

2 / 67

Overview Overview

EFA, CFA, SEM? EFA, CFA, SEM?

Exploratory Factor Analysis (EFA) Confirmatory Factor Analysis (CFA)
Method for “explaining” correlations of observed variables in terms of a Method for testing hypotheses about relationships among observed
small number of “common factors” variables
Primary Q: How many factors are needed? Does this by imposing restrictions on an EFA model
Secondary Q: How to interpret the factors? Q: Do the variables have a given factor structure?
Q: How to compare competing models?

Three-factor EFA model. Each variable

Two-factor CFA model with non-
loads on all factors.
overlapping factors
The factors are assumed to be uncor-
The factors are allowed to be corre-
related
lated, as are two unique factors

3 / 67 4 / 67
 Overview Overview Basic EFA ideas

EFA, CFA, SEM? Recall basic EFA ideas

Observed variables, x1 , x2 , . . . , xp is considered to arise as a set of

regressions on some unobserved, latent variables called common
Structural Equation Models (SEM)
factors, ξ1 , ξ2 , . . . , ξk .
Generalizes EFA, CFA to include
That is, each variable can be expressed as a regression on the common
Simple and multiple regression
General linear model (Anova, multivariate regression, ...)
factors. For three variables and one common factor, ξ, the model is:
Path analysis — several simultaneous regression models
Higher-order CFA models
Multi-sample CFA models (“factorial invariance”)
x1 = λ1 ξ + z1
Latent growth/trajectory models
Many more ... x2 = λ2 ξ + z2
A general framework for describing, estimating and testing linear x3 = λ3 ξ + z3
statistical models
The common factors account for correlations among the xs.
The zi are error terms, or unique factors

5 / 67 6 / 67

Overview Basic EFA ideas Overview Basic EFA ideas

The EFA model The EFA model

However, the EFA model implies a particular form for the

For k common factors, the common factor model is
variance-covariance matrix, Σ, which is testable

where:
Λp×k = factor pattern (“loadings”)
Φk ×k = matrix of correlations among factors.
Ψ = diagonal matrix of unique variances of observed variables.
Typically, it is initially assumed that factors are uncorrelated (Φ = I, the
This looks like a set of multiple regression models for the xs, but it is not
identity matrix)
testable, because the factors, ξ, are unobserved
Can use an oblique rotation to allow correlated factors

7 / 67 8 / 67
 Overview Basic EFA ideas Overview EFA to CFA

Limitations of EFA Historical development: EFA → CFA

ML estimation for the EFA model finds estimates that minimize the
The only true statistical tests in EFA are tests for the number of common difference between the observed covariance matrix, S, and that
reproduced by the model, Σb =ΛbΦbΛ bT + Ψ
factors (when estimated by ML)
b
Requires imposing k 2 restrictions for a unique solution
H0 : k = k0 k0 factors are sufficient Gives a χ2 test for goodness of fit
Ha : k > k0 > k0 factors are necessary b ∼ χ2
(N − 1)Fmin (S, Σ) with df = [(p − k )2 − p − k ]/2

Substantive questions about the nature of factors can only be addressed Joreskog (1969) proposed that a factor hypothesis could be tested by
approximately through factor rotation methods imposing restrictions on the EFA model— fixed elements in Λ, Ψ, usually
Varimax & friends attempt rotation to simple structure 0
Oblique rotation methods allow factors to be correlated Needs more than k 2 restrictions
Procrustes rotation allows rotation to a “target” (hypothesized) loading matrix The ML solution is then found for the remaining free parameters
The χ2 for the restricted solution gives a test for how well the hypothesized
factor structure fits.

9 / 67 10 / 67

Overview EFA to CFA Overview CFA to SEM

CFA: Restricted EFA Historical development: CFA → SEM

The pattern below specifies two non-overlapping oblique factors. The x’s are Higher-order factor analysis: The ACOVS model
the only free parameters. With more than a few factors, allowed to be correlated (Φ 6= I), can we
 x factor the factor correlations?
0 
In EFA, this was done by another EFA of the estimated factor correlations
 x 0    from an oblique rotation
 x 0  1
Λ=
 0 x

 Φ=
x 1 The second stage of development of CFA/SEM models combined these

0 x
 steps into a single model, and allowed different hypotheses to be
0 x compared

This CFA model has only 7 free parameters and df = 15 − 7 = 8.

A k = 2-factor EFA model would have all parameters free and
df = 15 − 11 = 4 degrees of freedom.
If this restricted model fits (has a small χ2 /df ), it is strong evidence for
two non-overlapping oblique factors.
That hypothesis cannot be tested by EFA + rotation.

11 / 67 12 / 67
 Overview CFA to SEM Overview CFA to SEM

LISREL/SEM Model LISREL/SEM Model

Jöreskog (1973) further generalized the ACOVS model to include SEM model for measures of Math Self-Concept and MATH achievement:
structural equation models along with CFA.
Two parts:
Measurement model — How the latent variables are measured in terms of
the observed variables; measurement properties (reliability, validity) of
observed variables. [Traditional factor analysis models]
Structural equation model — Specifies causal relations among observed
and latent variables.
Endogenous variables - determined within the model (y)
Exogenous variables - determined outside the model (x)
This model has:
Measurement models 3 observed indicators in a measurement model for MSC (x)
x = Λx ξ + δ 2 observed indicators in a measurement model for MATH achievement
for observed variables
y = Λy η +  (y)
Structural eqn. for latent A structural equation predicting MATH achievement from MSC
variables η = Bη + Γξ + ζ

13 / 67 14 / 67

Overview CFA to SEM Overview software

LISREL/SEM Model CFA/SEM software: LISREL

Measurement sub-models for x and y LISREL (http://www.ssicentral.com/) [student edition available]
Originally designed as stand-alone program with matrix syntax
LISREL 8.5+ for Windows/Mac: includes
interactive, menu-driven version;
PRELIS (pre-processing, correlations and models for categorical variables);
SIMPLIS (simplified, linear equation syntax)
path diagrams from the fitted model

Structural model, relating ξ to η

15 / 67 16 / 67
 Overview software Overview software

CFA/SEM software: Amos SAS: PROC CALIS

Amos (www.ibm.com/software/products/en/spss-amos): Linear equation
syntax + path diagram model description
import data from SPSS, Excel, etc; works well with SPSS
Create the model by drawing a path diagram
simple facilities for multi-sample analyses SAS 9.3+: PROC CALIS
nice comparative displays of multiple models MATRIX (à la LISREL), LINEQS (à la EQS), RAM, ... syntax
Now handles multi-sample analyses
Multiple-model analysis syntax, e.g., Model 2 is like Model 1 except ...
Enhanced output controls
customizable fit summary table
SAS macros http://datavis.ca/sasmac/:
caliscmp macro: compare model fits from PROC CALIS à la Amos
csmpower macro: power estimation for covariance structure models

17 / 67 18 / 67

Overview software Overview software

R: sem, lavaan and others Mplus

sem package (John Fox)

flexible ways to specify models: cfa(), linearEquations(), and Mplus https://www.statmodel.com/ [$$$, but cheaper student price]
multigroupModel() Handles the widest range of models: CFA, SEM, multi-group, multi-level,
bootSem() provides bootstrap analysis of SEM models latent group
miSem() provides multiple imputation Variables: continuous, censored, binary, ordered categorical (ordinal),
path diagrams using pathDiagram() → graphviz
unordered categorical (nominal), counts, or combinations of these
polychor package for polychoric correlations
variable types
lavaan package (Yves Rossell)
For binary and categorical outcomes: probit, logistic regression, or
Functions lavaan(), cfa(), sem(), growth() (growth curve models) multinomial logistic regression models.
Handles multiple groups models
For count outcomes: Poisson and negative binomial regression models.
semTools provides tests of measurement invariance, multiple imputation,
bootstrap analysis, power analysis for RMSEA, ...
Extensive facilities for simulation studies.
semPlot package — path diagrams for sem, lavaan, Mplus, ... models

19 / 67 20 / 67
 Overview software Path diagrams

Caveats Path diagrams: Symbols

Visual representation of a set of simultaneous equations for EFA, CFA, SEM
models (idea from Sewell Wright, 1920s)
CFA and SEM models are fit using the covariance matrix (S)
The raw data is often not analyzed
Graphs that can reveal potential problems often not made
Typically, this assumes all variables are complete, continuous,
multivariate normal. Implies:
S is a sufficient statistical summary
Relations assumed to be linear are in fact linear
Goodness-of-fit (χ2 ) and other tests based on asymptotic theory (N → ∞)
Missing data, skewed or long-tailed variables must be handled first
Topics not covered here:
Using polychoric correlations for categorical indicators
Distribution-free estimation methods (still asymptotic)
Bootstrap methods to correct for some of the above
Multiple imputation to handle missing data

21 / 67 22 / 67

Path diagrams Path diagrams

Path diagrams Path diagrams

Substantive example: Path analysis model for union sentiment (McDonald &
Schematic Examples: Clelland, 1984)

No latent variables— all variables are observed indicators

x1 , x2 are exogenous variables— they are not explained within the model
Correlation between x1 , x2 is shown as a double-headed arrow
y1 , y2 , y3 are endogenous variables— they are explained within the model
CFA, 1-factor model SEM, two latent variables, each with two indicators Causal relations are shown among the variables by single-headed arrows
(correlated errors) Causal relation betweeen ξ (Xs) and η (Ys) Residual (error) terms, ζ1 , ζ2 , ζ3 are shown as single-headed arrows to
the y variables

23 / 67 24 / 67
 Path diagrams Path analysis

Path diagrams Path Analysis

Substantive example: SEM with multiple indicators, path model for latent
variables (error terms not shown)
Path analysis is a simple special case of SEM
These models contain only observed (manifest) variables,
No latent variables
Assumes that all variables are measured without error
The only error terms are residuals for y (endogenous) variables
They are comprised of a set of linear regression models, estimated
simultaneously
Traditional approaches using MRA fit a collection of separate models
Multivariate MRA (MMRA) usually has all y variables predicted by all x
variables
In contrast, SEM path models allow a more general approach, in a single
model

25 / 67 26 / 67

Path analysis Simple examples Path analysis Simple examples

Path Analysis: Simple examples Path Analysis: Simple examples

Simple linear regression
Multivariate multiple regression
yi = γxi + ζi

y1i = γ11 x1i + γ12 x2i + ζ1i

γ is the slope coefficient; ζ is the residual (error term)
Means and regression intercepts usually not of interest, and suppressed y2i = γ21 x2i + γ22 x2i + ζ2i

Multiple regression Now need two equations to specify the model

Note subscripts: γ12 is coeff of y1 on x2 ; γ21 is coeff of y2 on x1
With more equations and more variables, easier with vectors/matrices
yi = γ1 x1i + γ2 x2i + ζi       
y1 γ11 γ12 x1 ζ1
= + or y = Γx + ζ
y2 γ21 γ22 x2 ζ2

Double-headed arrow signifies the assumed correlation between x1 & x2

In univariate MRA (y ∼ x1 + . . . ), there can be any number of xs
27 / 67 28 / 67
 Path analysis Simple examples Path analysis Exogenous and Endogenous

Path Analysis: Simple examples Exogenous and Endogenous Variables

Simple mediation model Exogenous variables

Are only independent (x) variables in the linear equations
Never have arrows pointing at them from other variables
y1i = γ11 xi + ζ1i They are determined outside (“ex”) the model
y2i = γ21 xi + β21 y1i + ζ2i In path analysis models they are considered measured w/o error

Endogenous variables
Something new: y1 is a dependent variable in the first equation, but a Serves as a dependent variable (outcome) in at least one equation
predictor in the second If a variable has at least one arrow pointing to it, it is endogenous
This cannot be done simultaneously via standard MRA or MMRA models They are determined inside (“en”) the model
In path analysis models they always have error terms
        
y1 0 0 y1 γ11 ζ1 In the simple mediation model, x is exogenous,
= + x+ or y = By + Γx + ζ
y2 β21 0 y2 γ21 ζ2 and y1 , y2 are endogenous

29 / 67 30 / 67

Path analysis Exogenous and Endogenous Path analysis General path model

Example: Union sentiment The general path analysis model

Norma Rae example— Union sentiment among non-union Southern textile
The general form of a SEM path analysis model is expressed in the matrix
workers (McDonald & Clelland (1984); Bollen (1986))
equation
Exogenous variables: x1 (years of work); x2 (age) y = By + Γx + ζ
Endogenous variables: y1 (deference to managers); y2 (support for labor
activism); y3 (support for unions) where:
The hypothesized model is comprised of three linear regressions y is a p × 1 vector of endogenous variables
x is a q × 1 vector of exogenous variables
Bp×p (“Beta”) gives the regression coefficients of endogenous (y)
y1 = γ12 x2 + ζ1 variables on other endogenous variables
y2 = β21 y1 + γ22 x2 + ζ2 Γp×q (“Gamma”) gives the regression coefficients of endogenous
variables on the exogenous variables (x)
y3 = β31 y1 + β32 y2 + γ31 x1 + ζ3
ζp×1 is the vector of errors in the equations (i.e., regression residuals)
These can be expressed as a single matrix equation for the y variables: However, some parameters in B and Γ are typically fixed to 0
   
         0 0 0 0 γ12
y1 0 0 0 y1 0 γ12   ζ1 B =  β21 0 0  Γ =  0 γ22 
 y2  =  β21 x1
0 0   y2  +  0 γ22  +  ζ2  β31 β32 0 γ31 0
x2
y3 β31 β32 0 y3 γ31 0 ζ3

31 / 67 32 / 67
 Path analysis General path model Path analysis General path model

The general path analysis model Union sentiment: using the sem package
Other parameters pertain to variances and covariances of the exogenous Read the variance-covariance matrix of the variables using readMoments()
variables and the error terms library(sem)
Φq×q (“Phi”)— variance-covariance matrix of the exogenous variables. union <- readMoments(diag=TRUE,
names=c('y1', 'y2', 'y3', 'x1', 'x2'),
Typically, these are all free parameters. text="
For the union sentiment example, Φ is a 2 × 2 matrix: 14.610
-5.250 11.017
  -8.057 11.087 31.971
var(x1 ) -0.482 0.677 1.559 1.021
Φ=
cov(x1 , x2 ) var(x2 ) -18.857 17.861 28.250 7.139 215.662
")
Ψp×p (“Psi”)— variance-covariance matrix of the error terms (ζ).
Typically, the error variances are free parameters, but their covariances The model can be specified in different, equivalent notations, but the simplest
are fixed to 0 (models can allow correlated errors) is often linear equations format, with specifyEquations()
For the union sentiment example, Ψ is a 3 × 3 diagonal matrix:
union.mod <- specifyEquations(covs="x1, x2", text="
  y1 = gam12*x2
var(ζ1 )
y2 = beta21*y1 + gam22*x2
Ψ= 0 var(ζ2 ) 
y3 = beta31*y1 + beta32*y2 + gam31*x1
0 0 var(ζ2 ) ")

33 / 67 34 / 67

Path analysis General path model Path analysis General path model

Union sentiment: using the sem package Union sentiment: Goodness-of-fit statistics
Internally, sem expresses the model using “RAM” path notation (same as The summary() method prints a collection of goodness-of-fit statistics:
used by specifyModel()):
opt <- options(fit.indices = c("GFI", "AGFI", "RMSEA", "NNFI",
union.mod "CFI", "AIC", "BIC"))
summary(union.sem)
## Path Parameter
## 1 x2 -> y1 gam12
## 2 y1 -> y2 beta21 ##
## 3 x2 -> y2 gam22 ## Model Chisquare = 1.25 Df = 3 Pr(>Chisq) = 0.741
## 4 y1 -> y3 beta31 ## Goodness-of-fit index = 0.997
## 5 y2 -> y3 beta32 ## Adjusted goodness-of-fit index = 0.986
## 6 x1 -> y3 gam31 ## RMSEA index = 0 90% CI: (NA, 0.0904)
## 7 x1 <-> x1 V[x1] ## Tucker-Lewis NNFI = 1.0311
## 8 x1 <-> x2 C[x1,x2] ## Bentler CFI = 1
## 9 x2 <-> x2 V[x2] ## AIC = 25.3
## 10 y1 <-> y1 V[y1] ## BIC = -14.2
## 11 y2 <-> y2 V[y2] ##
## 12 y3 <-> y3 V[y3] ## ...
##
## R-square for Endogenous Variables
Fit the model using sem(): ## y1 y2 y3
## 0.113 0.230 0.390
union.sem <- sem(union.mod, union, N=173) ##
## ...

35 / 67 36 / 67
 Path analysis General path model Path analysis General path model

Union sentiment: Parameter estimates Union sentiment: Path diagrams

## Parameter Estimates
## Estimate Std Error z value Pr(>|z|) Path diagrams for a sem() model can be produced using
## gam12 -0.0874 0.0187 -4.68 2.90e-06 y1 <--- x2
## beta21 -0.2846 0.0617 -4.61 3.99e-06 y2 <--- y1 pathDiagram(model)
## gam22 0.0579 0.0161 3.61 3.09e-04 y2 <--- x2 This uses the graphvis program (dot), that must be installed first
## beta31 -0.2177 0.0971 -2.24 2.50e-02 y3 <--- y1
## beta32 0.8497 0.1121 7.58 3.52e-14 y3 <--- y2
(http://www.graphviz.org/)
## gam31 0.8607 0.3398 2.53 1.13e-02 y3 <--- x1 The latest version (sem 3.1-6) uses the DiagrammeR package instead
## V[x1] 1.0210 0.1101 9.27 1.80e-20 x1 <--> x1 Edges can be labeled with parameter names, values, or both
## C[x1,x2] 7.1390 1.2556 5.69 1.30e-08 x2 <--> x1
## V[x2] 215.6620 23.2554 9.27 1.80e-20 x2 <--> x2
## V[y1] 12.9612 1.3976 9.27 1.80e-20 y1 <--> y1
## V[y2] 8.4882 0.9153 9.27 1.80e-20 y2 <--> y2 x1 0.86
## V[y3] 19.4542 2.0978 9.27 1.80e-20 y3 <--> y3 pathDiagram(union.sem,
edge.labels="values", -0.22 y3
file="union-sem1",
-0.09 y1
The fitted model is: x2 -0.28
0.85
min.rank=c("x1", "x2")) 0.06
y2
−0.087x2
 
yb1 = 12.96
yb2 = −0.285y1 + 0.058x2 b = 0
Ψ 8.49 
0 0 19.45
yb3 = −0.218y1 + 0.850y2 + 0.861x1

37 / 67 38 / 67

Path analysis General path model Identification Fundamental SEM hypothesis

Union sentiment: Path diagrams Fundamental hypothesis of CFA & SEM

dot produces a text file describing the path diagram
This can easily be (hand) edited to produce a nicer diagram The covariance matrix (Σ) of the observed variables is a function of the
Using color or linestyle for + vs. − edges facilitates interpretation parameters (θ) of the model

Σ = Σ(θ)

Years That is, if

0.86
Σ is the population covariance matrix of the observed variables, and
7.14 -0.22 Sentiment θ is a vector of all unique free parameters to be estimated,
-0.09 Deference 0.85 then, Σ(θ) is the model implied or predicted covariance matrix, expressed in
-0.28
Age terms of the parameters.
0.06
Activism If the model is correct, and we knew the values of the parameters, then

Σ = Σ(θ)

says that the population covariance matrix would be exactly reproduced

The coefficients shown are unstandardized— on the scale of the by the model parameters
variables
Can also display standardized coefficients, easier to compare
39 / 67 40 / 67
 Identification Fundamental SEM hypothesis Identification Fundamental SEM hypothesis

Fundamental hypothesis of CFA & SEM Fundamental hypothesis of CFA & SEM
Example: Consider the simple linear regression model,

yi = γxi + ζi

If this model is true, then the variance and covariance of (y, x) are This general hypothesis forms the basis for several important ideas in CFA
and SEM
var(yi ) = var(γxi + ζi )
Model identification: How to know if you can find a unique solution?
= γ 2 var(xi ) + var(ζi )
Model estimation: How to fit a model to an observed covariance matrix
cov(yi , xi ) = γvar(xi ) (S)?
The hypothesis Σ = Σ(θ) means that Σ can be expressed in terms of the Goodness-of-fit statistics: How to assess the discrepancy between S
model-implied parameters, γ (regression slope), var(ζ) (error variance) and and Σ(θ)?
var(x):
 
     2  γ
y var(y ) γ var(x) + var(ζ)
Σ = = = Σ  var(ζ) 
x cov(y , x) var(y ) γvar(x) var(x)
var(x)

41 / 67 42 / 67

Identification Identification rules Identification Identification rules

Model identification Model identification: t-rule and degrees of freedom

The simplest rule, the t-rule says:
The number of unknown parameters to be estimated (t) cannot exceed
A model is identified if it is possible to find a unique estimate for each the number of non-redundant variances and covariances of the observed
parameter variables
A non-identified model has an infinite number of solutions— not too This is a necessary condition for identification, but it is not sufficient
useful For path analysis models, let P = p + q be the total numbr of endogenous (y )
Such models may be made identified by: and exogenous (x) variables in Σ, and let t be the number of free parameters
Setting some parameters to fixed constants (like β12 = 0 or var(ζ1 ) = 1) in θ. The t-rule is
Constraining some parameters to be equal (like β12 = β13 ) P(P + 1)/2 ≥ t
Identification can be stated as follows:
The difference gives the number of degrees of freedom for the model:
An unknown parameter θ is identified if it can be expressed as a function of
one or more element of Σ
df = P(P + 1)/2 − t
The whole model is identified if all parameters in θ are identified
Complex models can often lead to identification problems, but there are a
few simple helpul rules If df < 0, the model is under-identified (no unique solution)
If df = 0, the model is just-identified (can’t calculate goodness-of-fit)
If df > 0, the model is over-identified (can calculate goodness-of-fit)
=⇒ Useful SEM models should be over-identified!!
43 / 67 44 / 67
 Identification Identification rules Identification Identification rules

Example: Union sentiment B rules: B = 0

For the Union sentiment model, the model parameters were:
   
0 0 0 0 γ12 Another simple rule applies if no endogenous y variable affects any other
B =  β21 0 0  Γ =  0 γ22  endogenous variable, so B = 0
β31 β32 0 γ31 0 For example:
and y1 = γ11 x1 + γ12 x2 + ζ1
  y2 = γ21 x1 + γ23 x3 + ζ2
  var(ζ1 )
var(x1 ) y3 = γ31 x1 + γ33 x3 + γ34 x4 + ζ3
Φ= Ψ= 0 var(ζ2 ) 
cov(x1 , x2 ) var(x2 )
0 0 var(ζ2 )
B = 0 because no y appears on the RHS of an equation
Observed covariance matrix: p = 3 endogenous y s +q = 2 exogenous xs Such models are always identified
=⇒ Σ5×5 has 5 × 6/2 = 15 variances and covariances. This is a sufficient, but not a necessary condition
12 free parameters in the model: Residuals ζi in such models need not be uncorrelated, i.e., Ψ can be
6 regression coefficients (3 non-zero in B, 3 non-zero in Γ) non-diagonal (“seemingly unrelated regressions”)
3 variances/covariances in Φ
3 residual variances in diagonal of Ψ
The model df = 15 − 12 = 3 > 0, so this model is over-identified
45 / 67 46 / 67

Identification Identification rules Identification Identification rules

B rules: recursive rule B rules: recursive rule

Non-recursive because B is not lower-triangular:

The recursive rule applies if
the only free elements in B are on its lower (or upper) triangle, and
Ψ is diagonal (no correlations amongst residuals)    
This basically means that there are no reciprocal relations among the y s 0 β12 var(ζ1 )
B= Ψ=
and no feedback loops β21 0 0 var(ζ2 )
This also is a sufficient condition for model identification.
The union sentiment mode is recursive because B is lower-triangular and Ψ is
diagonal Non-recursive because Γ is not diagonal:
   
0 0 0 var(ζ1 )
B =  β21 0 0  Ψ =  0 var(ζ2 )     
0 0 var(ζ1 )
β31 β32 0 0 0 var(ζ2 ) B= Ψ=
β21 0 cov(ζ1 , ζ2 ) var(ζ2 )

47 / 67 48 / 67
 Estimation Estimation

Model estimation Model estimation: Maximum likelihood

How to fit the model to your data? Maximum likelihood estimation is designed to maximize the likelihood
In ordinary regression analysis, the method of least squares is used to (“probability”) of obtaining the observed data (Σ) over all choices of
find values of the parameters (regression slopes) that minimize the sum parameters (θ) in the model
of squared residuals, (yi − ybi )2 .
P

This is fitting the model to the individual observations L = Pr(data | model) = Pr(S | Σθ )
In constrast, SEM methods find parameter estimates that fit the model to
the observed covariance matrix, S. This assumes that the observed data are multivariate normally distributed
They are designed to minimize a function of the residual covariances, ML estimation is equivalent to minimizing the following function:
S − Σθ
If the model is correct, then Σθ = Σ and as N → ∞, S = Σ. FML = log |Σθ | − log |S| + tr(SΣ−1
θ )−p
There is a variety of estimation methods for SEM, but all attempt to choose
the values of parameters in θ to minimize a function F (•) of the difference All SEM software obtains some initial estimates (“start values”) and uses
between S and Σθ an iterative algorithm to minimize FML

49 / 67 50 / 67

Estimation Model evaluation

Model estimation: Maximum likelihood Model fit

SEM provides R 2 values for each endogenous variable — the same as in
separate regressions for each equation
ML estimates have optimal properties ## R-square for Endogenous Variables
Unbiased: E(θ)b =θ ## y1 y2 y3
## 0.113 0.230 0.390
Asymptotically consistent: as N → ∞, θb → θ
Maximally efficient: smallest standard errors More importantly, it provides overall measures of fit for the entire model.
As N → ∞, parameter estimates θbi are normally distributed, The model for union sentiment fits very well, even though the R 2 s are
N (θbi , var(θi )), providing z (Wald) tests and confidence intervals rather modest
## Model Chisquare = 1.25 Df = 3 Pr(>Chisq) = 0.741
## Goodness-of-fit index = 0.997
θb ## Adjusted goodness-of-fit index = 0.986
z= CI1−α : θb ± z1−α/2 se(θ)
b
## RMSEA index = 0 90% CI: (NA, 0.0904)
s.e.(θ)
b
## Bentler CFI = 1
## AIC = 25.3
As N → ∞, the value (N − 1)FML has a χ2 distribution with ## BIC = -14.2
df = P(P + 1)/2 − t degrees of freedom, giving an overall test of model
A just-identified model will always fit perfectly— but that doesn’t mean it is
fit.
a good model: there might be unnecessary or trivial parameters.
An over-identified model that fits badly might have too many fixed or
constrained parameters

51 / 67 52 / 67
 Model evaluation chi-square test Model evaluation chi-square test

Model fit: χ2 test Model fit: χ2 test— problems

b used to minimize the discrepancy between S The test statistic, X 2 = (N − 1)Fmin is a function of sample size.
The fitting function F (S, Σ)
and the model estimate Σ b gives a chi-square test of model fit
b = Σ(θ) With large N, trivial discrepancies will give a significant chi-square
If the model is correct, then the minimized value, Fmin , has an asympotic Worse, it tests an unrealistic hypothesis that the model fits perfectly
chi-square distribution, the specified model is exactly correct in all details
any lack-of-fit is due only to sampling error
X 2 = (N − 1)Fmin ∼ χ2df it relies on asymptotic theory (X 2 ∼ χ2 as N → ∞) and an assumption of
multivariate normality
with df = P(P + 1)/2 − t degrees of freedom Another problem is parsimony— a model with additional free parameters
This gives a test of the hypothesis that the model fits the data will always fit better, but smaller models are simpler to interpret
If you fit several nested models, M1 ⊃ M2 ⊃ M3 . . . , chi-square tests for
H0 : Σ = Σ(θ) the difference between models are less affected by these problems
a large (significant) X 2 indicates that the model does not fit the data. ∆X 2 = X 2 (M1 ) − X 2 (M2 ) ∼ χ2 with df = df1 − df2

53 / 67 54 / 67

Model evaluation RMSEA Model evaluation RMSEA

Model fit: RMSEA Model fit: RMSEA

The measure of root mean square error of approximation (RMSEA) attempts
to solve these problems (Browne & Cudeck, 1993)
s In addition, the RMSEA statistic has known sampling distribution properties
X 2 − df (McCallum et al., 1996). This means that:
RMSEA = You can calculate confidence intervals for RMSEA
(N − 1)df
It allows to test a null hypothesis of “close fit” or “poor fit”, rather than
“perfect fit”
Relatively insensitive to sample size
Parsimony adjusted— denominator adjusts for greater df H0 : RMSEA < 0.05
Common labels for RMSEA values: H0 : RMSEA > 0.10
RMSEA interpretation
0 perfect fit
≤ .05 close fit It allows for power analysis to find the sample size (N) required to reject a
.05 − .08 acceptable fit hypothesis of “close fit” (RMSEA ≤ 0.05)
.08 − .10 mediocre fit
> .10 poor fit

55 / 67 56 / 67
 Model evaluation Other fit indices Model evaluation Other fit indices

Incremental fit indices Incremental fit indices

Creating new indices of goodness-of-fit for CFA/SEM models was a

“growth industry” for many years— there are many possibilities Parsimony-adjusted indices also adjust for model df
Incremental fit indices compare the existing model with a null or baseline Bentler’s comparative fit index (CFI) is often widely used
model
The null model, M0 assumes all variables are uncorrelated— the worst XM2 − dfM
possible model. CFI = 1 −
X02 − df0
Incremental fit indices compare the XM2 for model M with X02 for the null
model Tucker-Lewis Index (TLI), also called “non-normed fit index” (NNFI) are
All of these are designed to range from 0 to 1, with larger values (e.g., also popularly reported
> 0.95) indicating better fit.
The generic idea is to calculate an R 2 -like statistic, of the form X02 /df0 − XM2 /dfM
TLI ≡ NNFI =
f (null model) − f (my model) X02 /df0 − 1
f (null model) − f (best model)

for some function f (•) of X 2 and df , and where the “best” model fits perfectly.

57 / 67 58 / 67

Model evaluation Other fit indices Model evaluation Model modification

Information criteria: AIC, BIC Model modification

What to do when your model fits badly?

First, note that a model might fit badly due to data problems:
Other widely used criteria, particularly when you have fit a collection of
outliers, missing data problems
potential models are the “information criteria”, AIC and BIC non-normality (highly skewed, excessive kurtosis)
Unlike the likelihood ratio tests these can be used to compare non-nested non-linearity, omitted interactions, ...
models Otherwise, bad model fit usually indicates that some important paths
Each of these uses a penalty for model complexity; BIC expresses a have been omitted, so some variances or covariances in S are poorly
greater preference for simpler models as the sample size increases. reproduced by the model
Some regression effects among (x, y) omitted (fixed to 0)?
AIC = X 2 − 2df Covariances among exogenous variables omitted? (all should be included)
BIC = X 2 − log(N)df Covariances among residuals might need to be included as free parameters
Actions:
Smaller is better Examine residuals, S − Σ(θ)b to see which variances/covariances are badly fit
Modification indices provide a way to test the impact of freeing each fixed
parameter

59 / 67 60 / 67
 Model evaluation Model modification Model evaluation Model modification

Example: Union sentiment

To illusrate, consider what would have happened if we omitted the important
As expected, this model fits very badly
path of y3 (sentiment) on y2 (activism) in the Union sentiment example
summary(union.sem.bad, fit.indices=c("RMSEA", "NNFI", "CFI"))

x1 gam31 ##
mod.bad <- specifyEquations(covs="x1, x2", text=' y3 ## Model Chisquare = 50.235 Df = 4 Pr(>Chisq) = 3.2251e-10
y1 = gam12*x2 beta31
y2 = beta21*y1 + gam22*x2 ## RMSEA index = 0.25923 90% CI: (0.19808, 0.32556)
y3 = beta31*y1 + gam31*x1
gam12 y1 beta21 ## Tucker-Lewis NNFI = 0.38328
')
x2 y2 ## Bentler CFI = 0.75331
gam22
##
## Normalized Residuals
## Min. 1st Qu. Median Mean 3rd Qu. Max.
Fit the model: ## -0.159 0.000 0.000 0.594 0.330 5.247
##
union.sem.bad <- sem(mod.bad, union, N=173) ## R-square for Endogenous Variables
union.sem.bad ## y1 y2 y3
## 0.1129 0.2295 0.1957
## ##
## Model Chisquare = 50.235 Df = 4 ...
##
## gam12 beta21 gam22 beta31 gam31 V[x1]
## -0.087438 -0.284563 0.057938 -0.509024 1.286631 1.021000
## C[x1,x2] V[x2] V[y1] V[y2] V[y3]
## 7.139000 215.662000 12.961186 8.488216 25.863934
##
## Iterations = 0 61 / 67 62 / 67

Model evaluation Model modification Model evaluation Model modification

Modification indices
Normalized residuals show the differences S − Σ(θ)b as approximate z-scores,
so values outside of ±2 can be considered significantly large. Modification indices provide test statistics for fixed parameters
The statistics estimate the decrease in X 2 if each fixed parameter was
round(normalizedResiduals(union.sem.bad), 3)
allowed to be freely estimated
## y1 y2 y3 x1 x2 These are χ2 (1) values, so values > 4 can be considered “significantly”
## y1 0.000 0.000 0.103 0.477 0.000 large.
## y2 0.000 0.000 5.246 0.330 0.000
## y3 0.103 5.246 -0.054 -0.159 1.454
## x1 0.477 0.330 -0.159 0.000 0.000 modIndices(union.sem.bad)
## x2 0.000 0.000 1.454 0.000 0.000
##
## 5 largest modification indices, A matrix (regression coefficients):
This points to the one very large residual for the y2 -> y3 (or y3 -> ## y3<-y2 y2<-y3 x2<-y3 y3<-x2 y1<-y3
## 42.071 38.217 4.240 3.947 3.763
y2 ) path ##
In this example Union sentiment (y3) is the main outcome, so it would ## 5 largest modification indices, P matrix (variances/covariances):
make sense here to free the y2 -> y3 path ## y3<->y2 y3<->y1 x2<->y3 x1<->y3 x1<->y2
## 38.3362 3.9468 3.9468 3.9468 0.4114

Once again, we see large values associated with the y2 -> y3 path

63 / 67 64 / 67
 Model evaluation Model modification Summary

Modification indices: Caveats Summary I

Structural equation models are an historical development of EFA and

Using modification indices to improve model fit is called specification CFA methods and path analysis
search EFA and CFA attempt to explain correlations among observed variables in
terms of latent variables (“factors”)
This is often deprecated, unless there are good substantive reasons for
EFA used factor rotation to obtain an interpretable solution
introducing new free parameters
CFA imposes restrictions on a solution, and allows specific hypothesis tests
New paths or covariances in the model should make sense theoretically Higher-order CFA further generalized CFA to the ACOVS model
Large modification indices could just reflect sample-specific effects Meanwhile, path analysis developed methods for analyzing systems of
equations together
The result, was SEM, in the form of the LISREL model

65 / 67 66 / 67

Summary

Summary II
Path diagrams provide a convenient way to portray or visualize a SEM
Direct translation from/to a system of linear equations
Some software (AMOS graphics) allows construction of the model via a path
diagram
Most SEM software provides for output of models and results as path
diagrams
Path analysis models provide a basic introduction to SEM
No latent variables— only observed (“manifest”) ones
Does not allow for errors of measurement in observed variables
exogenous variables (xs)— only predictors in the linear equations
endogenous variables (ys)— a dependent variable in one or more equations
Error terms reflect errors-in-equations— unmodeled predictors, wrong
functional form, etc.
An important question in SEM models is model identification— can the
parameters be uniquely estimated?
Another important question is how to evaluate model fit?

67 / 67