STA302/1001 - Methods of Data Analysis I

(Week 09 lecture notes)

Wei (Becky) Lin

Nov. 07, 2016

1/39
Last Week

• Review on matrices
• Simple linear regression model in matrix form.

Yi = —0 + —1 Xi + ‘i , i = 1, . . . , n

Yn◊1 = Xn◊2— 2◊1 + ‘ n◊1

That is S T S T S T
Y1 1 X1 ‘1
WY2 X W1 X2 X 5 6 W‘ X
W X W X —0 W 2X
W .. X = W .. X +W.X
U . V U. V —1 U .. V
Yn 1 Xn ‘n
5 6
b0
• b= = (XÕ X)≠1 XÕ Y
b1
symmetric
idempotent
• Fitted: Ŷ =Xb=HY, where H = X (X Õ X )≠1 X Õ , (HH = H, H Õ = H)
H).
• Ŷ
Residuals: e=Y- =(I-H)Y

2/39
Last Week (contd..)
• If ‘ ≥ N(0, ‡ 2 I), then we have
• Y≥ N(X—X—
X—,‡ ‡2 I)
b≥ N(— ‡2 (X Õ X )≠1 )
— ,‡
fit
•
• X—
Ŷ ≥ N(X—
X—,‡ ‡2 H)
• e≥ N(00,‡ 2
‡ (I-H))
• XhÕ —
Ŷh ≥ N(X —,‡‡2 XhÕ (X Õ X )≠1 Xh )
• ANOVA in matrix form

ÿ ÿ ÿ 1
SSTO = (Yi ≠ Ȳ )2 = Yi2 ≠ ( Yi )/n = YÕ (I ≠ J)Y
n
SSE = e Õ e = (Y ≠ Xb)Õ (Y ≠ Xb) = YÕ Y ≠ bÕ XÕ Y = YÕ (I ≠ H)Y
ÿ 1 1
SSR = (Ŷi ≠ Ȳ )2 = bÕ XÕ Y ≠ YÕ JY = YÕ [H ≠ J]Y
n n
• Mean response and prediction of new observation
1 (Xh ≠ X̄ )2
V (Ŷh ) = ‡ 2 [ + q ],
n (Xi ≠ X̄ )2
∆ s 2 (Ŷh ) = MSE (XhÕ (X Õ X )≠1 Xh ), spred
2
= MSE (1 + XhÕ (X Õ X )≠1 Xh ).
3/39
Week 09- Learning objectives & Outcomes

• First order model with two predictor variables.

• Multiple Linear regression (MLR).
• MLR with dummy variable.
• MLR with regressor f(X).
• Polynomial regression.
• With interaction effect.
• With transformed predictor.

4/39
 Chapter 6:
Multiple Linear Regression

5/39
Multiple Regression Models

EMI pot A K
pzxzt Bfa ) Py Dit Ppt Xp
= + t -
-

• Different options for regressor variables (X’s) lead to different model

interpretations.
• Distinct X’s.
• Functions of the X’s.
• Indicator variables. linear in X 's
)
-
• Distinct variables (first order model)
• E.g. Salary (Y) versus educational level (X1 ) and years of experience
(X2 ).
• X1 , X2 measure different quantitles that can affect Y .

6/39
First-order Models with Two Predictors Variables

• Two predictor variables X1 , X2 :

Yi = —0 + —1 Xi1 + —2 Xi2 + ‘i , i = 1, . . . , n

• first-order model with two predictor variables: linear in predictor

variables.
• X1 , X2 : additive effect or not to interact (no Xi1 Xi2 term in model).
• Assume E (‘i )=0:

E (Y) = —0 + —1X1 + —2X2

• Mean response is on regression surface or a response surface

• —0 : intercept in the regression plane: X1 = 0, X2 = 0.
• —1 : the change in the mean response with —X1 = 1 and X2 =
constant; ˆE (Y )/ˆX1 = —1
• —2 : the change in the mean response with X1 = constant and
—X2 = 1; ˆE (Y )/ˆX2 = —2
• —1 , —2 : partial regression coefficients: they reflect the partial effect.
7/39
First-order Models with Two Predictor Variables (contd.)
Model :
Etlk ftp.xitpaxz
Regression surface with two predictors
fixed X .
,
OEMFPZ

- -
.
#R×
.
.
'
' for fixed xz

: '

X Xz=kz

8/39
First-order Models with Two Predictor Variables (contd.)

Example: Regression surface with two predictors

9/39
Multiple linear regression (MLR) model

• MLR with p ≠ 1 predictor variables (X1 , . . . , Xp≠1 )

Yi = —0 + —1 Xi1 + —2 Xi2 + . . . + —p≠1 Xi(p≠1) + ‘i (1)

p≠1
ÿ
= —0 + —i Xik + ‘i (2)
k=1
p≠1
ÿ
= —i Xik + ‘i , where Xi0 = 1 (3)
k=0

• Regression coefficient parameters: —0 , —1 , . . . —p≠1 . dimcpkp

• Known constants: Xi0 , Xi1 , Xi2 , . . . Xi(p≠1)
• ‘i ≥ N(0, ‡ 2 ) and being independent.
• Since E (‘i ) = 0, the response mean function, E (Y) is a linear
combination of (p-1) predictors which defines a hyper-plane.

E (Y) = —0 + —1X1 + —2X2 + . . . + —p≠1Xp≠1

10/39
Multiple linear regression (MLR) model

E (Y) = —0 + —1X1 + —2X2 + . . . + —p≠1Xp≠1

• A hyperplane: in more than two dimensions; no long possible to have

a picture.
• Additive and do not interact.
• Interpretation of regression coefficients
• —0 : intercept in the regression hyper-plane:
X1 = 0, X2 = 0, . . . , Xp≠1 = 0.
• —k : the change in the mean response with —Xk = 1 and all other
predictor variables are held constant.
-

mm

11/39
MLR Model with More than One Predictor Variables

• We have more than one predictor variable in the MLR model.

• MLR with factor variable (dummy variable)
• factor variable is a qualitative explanatory variable (or dummy
variable), with categories (also called levels).
• MLR with regressor f(X): a function of X’s.
• Polynomial regression
2
E (Yi ) = —0 + —1 Xi1 + —2 Xi2 + —3 Xi1
• Interaction effect

E (Yi ) = —0 + —1 Xi1 + —2 Xi2 + —3 Xi1 Xi2

• Transformed X’s

E (Yi ) = —0 + —1 Xi1 + —2 Xi2 + —3 exp{Xi1 }

12/39
MLR model with dummy variable of binary levels.

• Response: Y be the income.

• Predictors: X1 be the total education years, X2 be the gender
information I
1, X2 = male
Di =
0, X2 = female
• SLR model:
Yi = —0 + —2 Di + ‘i , ‘i ≥ N(0, ‡ 2 )

• —0 = E (Y |D = 0), —1 = E (Y |D = 1) ≠ E (Y |D = 0)
• Reg. Coef. estimators:
b0 = —ˆ0 = Ȳ (D = 0), b1 = —ˆ1 = Ȳ (D = 1) ≠ Ȳ (D = 0)
• MLR model:

Yi = —0 + —1 Xi1 + —2 Di + ‘i , ‘i ≥ N(0, ‡ 2 )

• —0 = E (Y |X1 = 0, D = 0),
• —1 = E (Y |X1 = x + 1, D = d0 ) ≠ E (Y |X1 = x , D = d0 ), d0 = 0/1
• —2 = E (Y |X1 = x0 , D = 1) ≠ E (Y |X1 = x0 , D = 0)
13/39
MLR model with dummy variable of binary levels
Do :
Yiipotpixiitei
two sub models {
p

Xit
Del lie Potpa ) t P
:
'
, Ei
Yi = —0 + —1 Xi1 + —2 Di + ‘i , ‘i ≥ N(0, ‡ 2 )

:
Yitpotf) TAXIHEI

XIHEI
:
Yiipotp ,

fixed Debi
→
.tk#whenoX=1

~
vertical distance between
two sub models
The .

14/39
MLR model with factor variable of multiple levels.

• Response: Y be the income.

• Predictors: X1 be the total education years. X2 be the total years of
experience, X3 be the occupations which can be classified into 3
levels: (1) professonal; (2) white-collar; (3) blue-collar.
I I
1, X3 = white-collar 1, X3 = professional
Di2 = Di3 =
0, X3 = otherwise 0, X3 = otherwise

• SLR model with only X3 :

Yi = – + “2 Di2 + “3 Di3 + ‘i , ‘i ≥ N(0, ‡ 2 )

• – = E (Yblue≠collar ),
• “2 = E (Ywhite≠collar ) ≠ E (Yblue≠collar )
• “3 = E (Yprofessional ) ≠ E (Yblue≠collar )
• Regression Coefficient estimators:
• b0 = –
ˆ = Ȳ (blue-collar),
• b1 = “
ˆ2 = Ȳ (white-collar)-Ȳ (blue-collar),
• b2 = “
ˆ3 = Ȳ (professional)-Ȳ (blue-collar)
15/39
MLR model with factor variable of multiple levels (contd.)

Yi = – + —1 Xi1 + —2 Xi2 + “2 Di2 + “3 Di3 + ‘i , ‘i ≥ N(0, ‡ 2 )

⇐
a • This model describes three parallel regression planes, which can differ
in their intercepts
• • Blue collar: Yi = – + —1 Xi1 + —2 Xi2 + ‘i
{• White collar: Yi = (– + “2 ) + —1 Xi1 + —2 Xi2 + ‘i
• Professional collar: Yi = (– + “3 ) + —1 Xi1 + —2 Xi2 + ‘i
• Interpreation of regression coefficients
• – = E (Y |X1 = 0, X2 = 0, X3 =blue-collar): gives the intercept for
blue-collar model.
• “2 = E (Y |D2 = 1,other fixed)-E (Y |D2 = 0,other fixed): represents
the constant vertical distance between the parallel regression planes
for white-collar and blue-collar occupations (fixing the values of
education and income).
• “3 = E (Y |D3 = 1,other fixed)-E (Y |D3 = 0,other fixed): represents
the constant vertical distance between the parallel regression planes
for professional and blue-collar occupations (fixing the values of
education and income).
16/39
MLR model with factor variable of multiple levels (contd.)

Yi = – + —1 Xi1 + —2 Xi2 + “2 Di2 + “3 Di3 + ‘i , ‘i ≥ N(0, ‡ 2 )

• Interpreation of regression coefficients (contd.)

• —1 = E (Y |X1 = x1 + 1, X2 = x2 , X3 = x3 ) ≠ E (Y |X1 = x1 , X2 =
x2 , X3 = x3 ): holding all other predictors fixed, the change in the
mean income for one more year of education.
• —2 = E (Y |X1 = x1 , X2 = x2 + 1, X3 = x3 ) ≠ E (Y |X1 = x1 , X2 =
x2 , X3 = x3 ): holding all other predictors fixed, the change in the
mean income for one more year of experience

17/39
MLR model with factor variable of multiple levels (contd.)

Yi = – + —1 Xi1 + —2 Xi2 + “2 Di2 + “3 Di3 + ‘i , ‘i ≥ N(0, ‡ 2 )

• The additive dummy-regression model showing three parallel

regression planes. model
0 Blue collar
.

Etlkdtpiktfxz
white collar model
@
-

3
EHK @)tfXHfXr
th

3 professional model

2 EGKHTBHAX,tfXz
@

18/39
MLR model: Polynomial Regression (Ch8)

• Special cases of the general linear regression model or MLR model.

• Squared and higher-order terms of the predictors ∆ curvilinear.
• Polynomial regression is useful for describing curvilinear relationships.
• Quadratic (2nd order)

E (Y ) = —0 + —1 X + —2 X 2

• Cubic (3rd order)

E (Y ) = —0 + —1 X + —2 X 2 + —3 X 3

• Higher order

E (Y ) = —0 + —1 X + —2 X 2 + —3 X 3 + . . .

19/39
Example: Polynomial Regression
• Data
• Y: female steroid level.
• X: age.
• Fitted regression model

Ŷ = ≠6.21 + 3.08age ≠ 0.06age2

20/39
MLR with interaction term
• Two-way interaction

Yi = —0 + —1 Xi1 + —2 Xi2 + —3 Xi1 Xi2 + ‘i (4)

= —0 + —1 Xi1 + —2 Xi2 + —3 Xi3 + ‘i , Xi3 = Xi1 Xi2 (5)

• Two predictor variables interact in determining a response variable

when the partial effect of one depends on the value of the other.
• ˆE (Y )/ˆX1 = —1 + —3 X2 . The partial effect of X1 depends on the
value of X2 . Or, the effect of X1 varies by X2 .
• Interaction is a symmetric concept — the effect of X1 varies by X2 ,
and the effect of X2 by X1 ( ˆE (Y )/ˆX2 = —2 + —3 X1 )
• If the regressions in different categories of a dummy variable are not
parallel, then the dummy variable interacts with one or more of the
quantitative predictor variables. Move to slides 23 24 ¥
-

• The dummy-regression model can be modified to reflect interactions.

• Three-way interaction

E (Y ) = —0 + —1 X1 + —2 X2 + —3 X3 + —3 X1 X2 X3
21/39
Visualization: two-way interaction effect

E (Y ) = 3 + 2X1 + 2X2 ≠ 3X1 X2

22/39
Example: two-way interaction effect
• Response (Y): income.
• Predictors: X1 denotes the eduation years; X2 denotes the gender
(D=1 for male,D=0 for female).

parallel
§ t.no parallel

(a) ,Lb)⇒

interaction
effect exists

• In (a), gender and education are independent. In (b), women on

average have more education than men.
• In both cases, gender and education interact in determing income.
The within-gender regressions of income on education are not parallel:
the slope for men is larger than the slope for women.
• the effect of education varies by gender, or the gender effect varies by
education. 23/39
Example: two-way interaction effect (contd.)

Yi = – + —Xi + “Di + ”(Xi Di ) + ‘i

• Model for female: Yi = – + —Xi + ‘i

• Model for male: Yi = (– + “) + (— + ”)Xi + ‘i
• “ gives the difference in intercepts between the male and female
groups.
• ” gives the difference in slopes between the male and female groups.

:
Etlk Htr )t ( ptdlx

ECYK 2tpX

24/39
MLR with Transformed Variables

• Include regressors that are functions of one or several X’s

• E.g. E (Y ) = —0 + —1 X1 + —2 X2 + —3 X12 + —4 log(X2 /X1 )
• Apply some transformation to Y.
• Curvilinear response mean functions
• log(Y ) = —0 + —1 X1 + —2 X2 + —3 X3 + ‘, Y Õ = log(Y )
• Y = 1/(—0 + —1 X1 + —2 X2 + ‘), Y Õ = 1/Y .

25/39
MLR with Interaction Effects and Combination of Cases

• An example of non-additive regression model

Yi = —0 + —1 Xi1 + —2 Xi2 + —3 Xi1 Xi2 + ‘i

= —0 + —1 Xi1 + —2 Xi2 + —3 Xi3 + ‘i

• By cross-product term:

Yi = —0 + —1 Xi1 + —2 Xi2 + —3 Xi12 + —4 Xi22 + —5 Xi1 Xi2 + ‘i

= —0 + —1 Zi1 + —2 Zi2 + —3 Zi3 + —4 Zi4 + —5 Zi5 + ‘i

26/39
MLR with Interaction Effects and Combination of Cases
(contd.)

Additional Example of Mean Response Functions

27/39
Meaning of Linear in MLR model

• MLR refers to Multiple Linear Regression.

• MLR is also called General Linear Regression Model (GLRM).
• A regression is linear in parameters:

Yi = —0 Ci0 + —1 Ci1 + —2 Ci2 + . . . + —p≠1 Ci(p≠1) + ‘i

• where Cik , k=0, 1, . . . p1 are coefficients involving the predictor

variables, X’s.
• Ilustration: nonlinear

Yi = —0 exp (—1 Xi ) + ‘i

28/39
Matrix approach to MLR

29/39
MLR in Matrix Form

• The MLR model

Yi = —0 Xi0 + —1 Xi1 + —2 Xi2 + . . . + —p≠1 Xi(p≠1) + ‘i , i = 1, . . . , n

• In matrix form Y = X — + ‘
n◊1 n◊p p◊1 n◊1
S ST TS T S T
Y1 1 X11 X12 ... X1(p≠1) —0 ‘1
WY2 X W1 X21 X22 ... X2(p≠1) X W —1 X W‘2 X
W X W XW X W X
Y = W . X = W. .. .. .. .. X W .. X + W .. X
n◊1 U .. V U .. . . . . VU . V U.V
Yn 1 Xn1 Xn2 ... Xn(p≠1) —p≠1 ‘n

30/39
MLR in Matrix Form (contd.)

Y = X — + ‘
n◊1 n◊p p◊1 n◊1

• Y: vector of responses.
• — : vector of parameters.
• X: matrix of constants (design matrix).
• ‘ : vector of independent normal random variables.

‘ ≥ N(0, ‡ 2 In◊n )

• Expectation and variance-covariance matrix of Y:

E(Y) = X—
—; Var (Y) = ‡ 2 In◊n
n◊1

• — , ‡ 2 In◊n ).
That is, Y ≥ N(X—

31/39
Estimation of Regression Coefficients in MLR model
• The MLR model
Yi = —0 Xi0 + —1 Xi1 + —2 Xi2 + . . . + —p≠1 Xi(p≠1) + ‘i , i = 1, . . . , n
• The Least squares method: the values of —0 , . . . , —p≠1 minimizes Q
n
ÿ T
Q= (Yi ≠ Ŷi )2 = (Y ≠ Xbb )(Y ≠ Xbb )T
h

i=1

ˆQ
∆ = ≠2Y Õ X + 2b T X T X
ˆb
• Setting this equation to zero: ≠2Y Õ X + 2b T X T X = 0
• Transpose both sides of the equation, we have
X ÕX b = X ÕY
p◊p p◊1 p◊1

ˆ Assume X Õ X is invertible
• Solving it for —.
∆ LSE : b = (X Õ X )≠1 X Õ Y
p◊1 p◊p p◊1
32/39
Estimation of Regression Coefficients in MLR model
(contd.)

• The method of MLE leads to the same estimators of normal error

regression model.
• The likelihood function

1 Ó
L(—, ‡ 2 ) = exp
(2fi‡ )
2 n/2
n Ô
1 ÿ
≠ 2 (Yi ≠ —0 ≠ —1 Xi1 ≠ . . . ≠ —p≠1 Xi(p≠1) )2
2‡ 1

Maximizing the likelihood function w.r.t —0 , —1 , . . . , —p≠1 leads to the

the same LS estmators b.

33/39
Estimation of Regression Coefficients in MLR model
(contd.) @ Xxi symmetric is matrix
* { '

Khat
'
2
( 1×1×5 ) =

varcbkvarllxlxitxy )

xtxixvarlyl.li#xTT=rzlxTx5'xTxInh-*=rixxi
=
(

↳
b = (X Õ X )≠1 X Õ Y
'

p◊1 p◊p p◊1

(X Õ X )≠1 X Õ Y )=(X
• E(b)=E((X
p◊p p◊1
(X Õ X )≠1 X Õ E (Y) = — HIT
• Var(b)=Var((X (X X )≠1 X Õ
(X X ) X Y)=‡ 2(X Õ X )≠1 ,
Õ

(X Õ X )≠1
• Var(b) is estimated by s 2 (b)=MSE(X
2
• b≥ N(—— , ‡ (X X ) )
Õ ≠1

34/39
Fitted Values and Residuals ⇐
iii. =kiIaIi¥nH"
it
II hi ,
. hin Fi hijtj
=

• The vector of Ŷi .

(
hi ) IYLI ⇒
=

,
. .

Ŷ = Xbb = HY, H = X (X Õ X )≠1 X Õ

n◊1 n◊1n

• E(Ŷ )=E(HY)=HX— X—
— =X
• Var(Ŷ )=Var(HY)=‡ 2 H.
• Ŷ ≥ N(X —, ‡ 2 H)
X—
• The vector of residuals

e = Y ≠ Ŷ = (I-H)Y
n◊1

• E(e)=E((I-H)Y)=Xb b ≠ X—
—=0
• Var(e)=Var((I-H)Y)=‡ 2 (I-H),estimated by s 2 (e)=MSE(I-H).
• e ≥ N(0, ‡ 2 (I-H))

35/39
Analysis of Variance Results (ANOVA)

The sums of squares for ANOVA

ÿ ÿ ÿ 1
SSTO = (Yi ≠ Ȳ )2 = Yi2 ≠ ( Yi )/n = YÕ (I ≠ J)Y
n
SSE = e Õ e = (Y ≠ Xb)Õ (Y ≠ Xb) = YÕ Y ≠ bÕ XÕ Y = YÕ (I ≠ H)Y
ÿ 1 1
SSR = (Ŷi ≠ Ȳ )2 = bÕ XÕ Y ≠ YÕ JY = YÕ [H ≠ J]Y
n n
SSR
MSR =
p≠1
.se#.YLrsIIFsnD
sin :p
For
SSE
MSE =
n≠p

36/39
Analysis of Variance Results (ANOVA) (contd.)

Table: ANOVA table for MLR model

Source of Variation SS d.f. MS
Regression SSR=b’X’Y- n1 Y’JY p-1 SSR
MSR= p≠1
SSE
Error SSE=Y’Y-b’X’Y n-p MSE= n≠p
Total SSTO= Y’Y- n1 Y’JY n-1

• E{MSE}=‡ 2
• p-1=2 =) Etfkpotp Xitpzxz
,

1 ÿ ÿ
E {MSR} = ‡ 2 + [—12 (Xi1 ≠ X̄1 )2 + —22 (Xi2 ≠ X̄2 )2
2
ÿ
+ 2—1 —2 (Xi1 ≠ X̄1 )(Xi2 ≠ X̄2 )]

If —1 = —2 = 0 ∆ E{MSR}=‡ 2 , otherwise, E{MSR}Ø ‡ 2 ,

37/39
Practice problems after Week 09 lectures

• For MLR model, if ‘ ≥ N(0, ‡ 2 I), show that

• Y≥ N(X—X—
X—,‡ ‡ 2 I)
2
• b≥ N(— — ,‡
‡ (X Õ X )≠1 )
• X—
Ŷ ≥ N(X—
X—,‡ ‡2 H)
• e≥ N(00,‡ 2
‡ (I-H))
• XhÕ —
Ŷh ≥ N(X —,‡‡2 XhÕ (X Õ X )≠1 Xh )
• Practice problem in Chapter 6: 6.2, 6.3, 6.5 (a-d), 6.6, 6.7 (a), 6.8,
6.18 (b-e), 6.19, 6.20, 6.21, 6.22, 6.25, 6.27.

38/39
Upcoming topics

• Chapter 6:
• F test for regression coefficients.
• Coefficient of Multiple Determination.
• Inferences about Regression Parameters.
• Interval Estimation of —k , E(Yh ).
• F test of lack of fit
• Chapter 7:
• The extra sum of squares

39/39