SPC 2308 Modeling and Simulation

Chapter 10: Statistical Analysis

Tools
 Goals of Today

• Know how to compare between

two distributions
• Know how to evaluate the
relationship between two random
variable
 Outline
• Comparing Distributions: Tests for Goodness-of-Fit
– Chi-Square Distribution (for discrete models: PMF)
– Kolmogorov-Smirnov Test (for continuous models:
CDF)
• Evaluating the relationship
– Linear Regression
– Correlation
 GOODNESS-OF-FIT ‫جودة المطابقة‬
Statistical Tests enables to compare between two distributions,
also known as Goodness-of-Fit.

 The goodness-of-fit of a statistical model describes how well it

fits a set of observations.

Measures of goodness of fit typically summarize the discrepancy

between observed values and the values expected under the model
in question

Goodness-of-fit means how well a statistical model fits a set of

observations
 PEARSON’S C²-TESTS
CHI-SQUARE TESTS FOR DISCRETE MODELS

The Pearson's chi-square test enables to compare two

probability mass functions of two distribution.

If the difference value (Error) is greater than the critical

value, the two distribution are said to be different or the
first distribution does not fit (well) the second distribution.

If the difference if smaller that the critical value, the first

distribution fits well the second distribution
 (Pearson's ) Chi-Square test
• Pearson's chi-square is used to assess two types of comparison:
– tests of goodness of fit: it establishes whether or not an
observed frequency distribution differs from a theoretical
distribution.
– tests of independence. it assesses whether paired observations
on two variables are independent of each other.
• For example, whether people from different regions differ in the
frequency with which they report that they support a political candidate.
• If the chi-square probability is less or equal to 0.05 then we say
that
– both distributions are equal (goodness-of-fit) or that
– the row variable is unrelated (that is, only randomly related) to the
column variable (test of independence).
Chi-Square Distribution
http://en.wikipedia.org/wiki/Chi-square_distribution
Chi-Square Distribution
http://en.wikipedia.org/wiki/Chi-square_distribution
 (Pearson's ) Chi-Square test
• The chi-square test, in general, can be used to check whether an empirical
distribution follows a specific theoretical distribution.
• Chi-square is calculated by finding the difference between each observed (O)
and theoretical or expected (E) frequency for each possible outcome, squaring
them, dividing each by the theoretical frequency, and taking the sum of the
results.
• For n data outcomes (observations), the chi-square statistic is defined as:
n
Oi  E i  2
 n21  
i 1
Ei

– Oi = an observed frequency for a given outcome;

– Ei = an expected (theoretical) frequency for a given outcome;
– n = the number of possible outcomes of each event;
 (Pearson's ) Chi-Square test

A chi-square probability of
0.05 or less is the criteria to
accept or reject the test of
difference between the
empirical and theoretical
distributions.
 Chi-Square test: General Algorithm
http://en.wikipedia.org/wiki/Inverse-chi-square_distribution

 We say that the observed distribution (empricial) fits well the

expected distribution (theoretical) if:
 02  a2*,k 1c
which means p   02  a2*, k 1c   1  a
 
where
critical
2*
 a2*,k 1c  idfChiSquare  k  1  c , a 

• (k – 1 – c) is the degree of freedom, where k is the number of

possible outcome and c is the number of estimated parameters
• 1-a is the confidence level (basically, we use a = 0.05)
 Chi-Square test: Example
Uniform distribution in [0 .. 9]

PASS
 (KS-TEST) KOLMOGOROV – SMIRNOV TEST
FOR CONTINUOUS MODELS
 In statistics, the Kolmogorov–Smirnov test (K–S test) quantifies a
distance between the empirical distribution function of the sample
and the cumulative distribution function of the expected
distribution, or between the empirical distribution functions of two
samples.

 It can be used for both continuous and discrete models

 Basic idea: compute the maximum distance between two

cumulative distribution functions and compare it to critical value.
If the maximum distance is smaller than the critical value, the first
distribution fits the second distribution
If the maximum distance is greater than the critical value, the first
distribution does not fit the second distribution
 Kolmogorov – Smirnov test
 In statistics, the Kolmogorov – Smirnov test is used to determine
whether two one-dimensional probability distributions differ, or
whether an probability distribution differs from a hypothesized
distribution,
in either case based on finite samples.
 The Kolmogorov-Smirnov test statistic measures the largest vertical
distance between an empirical cdf calculated from a data set and a
theoretical cdf.
 The one-sample KS-test compares the empirical distribution
function with a cumulative distribution function.
 The main applications are testing goodness-of-fit with the normal
and uniform distributions.
 Kolmogorov–Smirnov Statistic
• Let X1, X2, …, Xn be iid random variables in with the CDF equal to F(x).
• The empirical distribution function Fn(x) based on sample
X1, X2, …, Xn is a step function defined by:

n
number of element in the sample  x
 I X
1
Fn  x    i x
n n i 1

where I(A) is the indicator of event A. I  X i  x   1 if  X i  x 


0 otherwise

• The Kolmogorov-Smirnov test statistic for a given function F(x) is

D n  sup Fn  x   F  x 
x
 Kolmogorov–Smirnov Statistic
• The Kolmogorov-Smirnov test statistic for a given function F(x) is

D n  sup Fn  x   F  x 
x

Facts
• By the Glivenko-Cantelli theorem, if the sample comes from a
distribution F(x), then Dn converges to 0 almost surely.
• In other words, If X1, X2, …, Xn really come from the distribution
with CDF F(X), the distance Dn should be small
Example

Dmax
 Example: Grade Distribution?
• We would like to know the distribution of the
Grades of students.
– First, determine the empirical distribution
– Second, compare to Normal and Poisson
distributions
• Data Sample: 50 Grades in a course and
computed the empirical distribution
– Mean = 63
– Standard Deviation = 15
 Example: Grade Distribution?
D n  sup Fn  x   F  x 
x

 
Frequency X grade = Number of grades  X grade


Frequency X grade 
   
Empirical Distribution = F X grade =p X  X grade =
Sample Size
Example: Grade Distribution?

Dmax,Poisson= 0.153

Dmax,Normal= 0.119

D n  sup Fn  x   F  x 
x
 Kolmogorov–Smirnov Acceptance Criteria

 Rejection Criteria: We consider that the two distributions

are not equal if the empirical CDF is too far from the
theoritical CDF of the proposed distribution
 This means: We reject if Dn is too large.
 But the question is: What does large mean?

For which values of Dn

should we accept the distribution?
Kolmogorov–Smirnov test
http://en.wikipedia.org/wiki/Kolmogorov-Smirnov_test

In the 1930’s, Kolmogorov and Smirnov showed that


  
2
t2
lim P n Dn  t  1- 2 (-1)i -1 e -2i
n 
i 1

So, for large sample sizes, you could assume Critical


value
  
2
t2
P n D n  t  1- 2 (-1)i -1 e -2i
i 1

alevel test: find the value of t such a  2  (-1)i -1 e -2i .t
2 2

i 1
ta
So, the test is accepted if Dn 
n
Kolmogorov–Smirnov test

 For small samples, people have worked out and

tabulated critical values, but there is no nice closed form
solution.
• J. Pomeranz (1973)
• J . Durbin (1968)

 For Large Samples: Good approximations for n>40:

a 0.20 0.10 0.05 0.02 0.01
1.0730 1.2239 1.3581 1.5174 1.6276
critical value
n n n n n
Example: Grade Distribution?

• For our example, we have n = 50

• The critical value for a a = 0.05
1.3581
Dcritical   0.192
50

Dmax,Normal  0.119  Dcritical  0.192 ACCEPT

Dmax,Poisson  0.153  Dcritical  0.192 ACCEPT

Example: Grade Distribution?

• If we get the same distribution for n = 100

• The critical value for a a = 0.05
1.3581
Dcritical   0.1358
100

Dmax,Normal  0.119  Dcritical  0.1358 ACCEPT

Dmax,Poisson  0.153  Dcritical  0.1358 REJECT

 LINEAR REGRESSION: LEAST SQUARE
METHOD
http://en.wikipedia.org/wiki/Linear_regression

In statistics, linear regression is a form of regression analysis in

which the relationship between one or more independent
variables and another variable, called dependent variable, is
modeled by a least squares function, called linear regression
equation. This function is a linear combination of one or more
model parameters, called regression coefficients.
A linear regression equation with one independent variable
represents a straight line. The results are subject to statistical
analysis.
 The Method of Least Squares
• The equation of the best-
fitting line is calculated
using a set of n pairs (xi, yi).
• We choose our estimates a and
b to estimate a and b so that
the vertical distances of the
points from the line, are
minimized.
Bestfitting line :yˆ  a  bx
Choosea and b to minimize
SSE  ( y  yˆ ) 2  ( y  a  bx) 2 SSE: Sum of Square of Errors
Least Squares Estimators
Calculatethe sumsof squares:
( x)
2
( y ) 2
Sxx   x 
2
Syy   y 
2

n n
( x)( y )
Sxy   xy 
n
Bestfitting line : yˆ  a  bx where
S xy
b and a  y  bx
S xx
 Example
The table shows the math achievement test scores for a
random sample of n = 10 college freshmen, along with
their final calculus grades.
Student 1 2 3 4 5 6 7 8 9 10
Math test, x 39 43 21 64 57 47 28 75 34 52
Calculus grade, y 65 78 52 82 92 89 73 98 56 75

Use your calculator to  x  460  y  760

find the sums and
sums of squares.  x 2  23634  y 2  59816
 xy  36854
x  46 y  76
 Example
(460) 2
Sxx  23634   2474
10
(760) 2
Syy  59816   2056
10
(460)(760)
Sxy  36854   1894
10
1894
b  .76556 and a  76  .76556(46)  40.78
2474
Bestfitting line : yˆ  40.78  .77 x
 Correlation Analysis
• The strength of the relationship between x and y
is measured using the coefficient of correlation:
S xy
Correlation coefficient : r 
S xx S yy
• The sign of r indicates the direction of the
relationship;
• r near 0 indicates no linear relationship,
• r near 1 or -1 indicates a strong linear
relationship.
• A test of the significance of the correlation
coefficient is identical to the test of the slope b.
 Example
The table shows the heights and weights of
n = 10 randomly selected college football players.

Player 1 2 3 4 5 6 7 8 9 10
Height, x 73 71 75 72 72 75 67 69 71 69
Weight, y 185 175 200 210 190 195 150 170 180 175

Use your calculator to S xy  328 S xx  60.4 S yy  2610

find the sums and
sums of squares. 328
r  .8261
(60.4)(2610)
 Football Players
Scatterplot of Weight vs Height

210

200

190
Weight

180 r = .8261
170 Strong positive
160 correlation
150 As the player’s height
66 67 68 69 70 71
Height
72 73 74 75 increases, so does his
weight.
Some Correlation Patterns
r = 0; No correlation r = .931; Strong positive
correlation

r = 1; Linear
relationship r = -.67; Weaker negative
correlation