Statistical Modeling and Computation Full Access Download
Uploaded by
ja.xonmit.chelldef1445Statistical Modeling and Computation Full Access Download
Uploaded by
ja.xonmit.chelldef1445Statistical Modeling and Computation
Visit the link below to download the full version of this book:
https://medidownload.com/product/statistical-modeling-and-computation/
Click Download Now
Dirk P. Kroese
School of Mathematics and Physics, The University of Queensland,
Brisbane, Australia
Joshua C. C. Chan
Department of Economics, Australian National University, Canberra,
Australia
ISBN 978-1-4614-8774-6 e-ISBN 978-1-4614-8775-3
Springer New York Heidelberg Dordrecht London
Library of Congress Control Number: 2013948920
© The Author(s) 2014
This work is subject to copyright. All rights are reserved by the Publisher,
whether the whole or part of the material is concerned, specifically the
rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and
transmission or information storage and retrieval, electronic adaptation,
computer software, or by similar or dissimilar methodology now known or
hereafter developed. Exempted from this legal reservation are brief excerpts
in connection with reviews or scholarly analysis or material supplied
specifically for the purpose of being entered and executed on a computer
system, for exclusive use by the purchaser of the work. Duplication of this
publication or parts thereof is permitted only under the provisions of the
Copyright Law of the Publisher’s location, in its current version, and
permission for use must always be obtained from Springer. Permissions for
use may be obtained through RightsLink at the Copyright Clearance Center.
Violations are liable to prosecution under the respective Copyright Law.
The use of general descriptive names, registered names, trademarks, service
marks, etc. in this publication does not imply, even in the absence of a
specific statement, that such names are exempt from the relevant protective
laws and regulations and therefore free for general use.
While the advice and information in this book are believed to be true and
accurate at the date of publication, neither the authors nor the editors nor
the publisher can accept any legal responsibility for any errors or omissions
that may be made. The publisher makes no warranty, express or implied,
with respect to the material contained herein.
Printed on acid-free paper
Springer is part of Springer Science+Business Media (www.springer.com)
In memory of Reuven Rubinstein, my Friend and Mentor
Dirk Kroese
To Raquel
Joshua Chan
Preface
Statistics provides one of the few principled means to extract information
from random data and has perhaps more interdisciplinary connections than
any other field of science. However, for a beginning student of statistics, the
abundance of mathematical concepts, statistical philosophies, and numerical
techniques can seem overwhelming. The purpose of this book is to provide
a comprehensive and accessible introduction to modern statistics,
illuminating its many facets, from both classical (frequentist) and Bayesian
points of view. The book offers an integrated treatment of mathematical
statistics and modern statistical computation.
The book is aimed at beginning students of statistics and practitioners
who would like to fully understand the theory and key numerical techniques
of statistics. It is based on a progression of undergraduate statistics courses
at The University of Queensland and the Australian National University.
Parts of the book have also been successfully tested at the University of
New South Wales. Emphasis is laid on the mathematical and computational
aspects of statistics. No prior knowledge of statistics is required, but we
assume that the reader has a basic knowledge of mathematics, which forms
an essential basis for the development of the statistical theory. Starting from
scratch, the book gradually builds up to an advanced undergraduate level,
providing a solid basis for possible postgraduate research. Throughout the
text we illustrate the theory by providing working code in Matlab , rather
than relying on black-box statistical packages. We make frequent use of the
symbol ☞ in the margin to facilitate cross-referencing between related
pages. The book is accompanied by the web site www.statmodcomp.org
from which the Matlab code and data files can be downloaded. In addition,
we provide an R equivalent for each Matlab program.
The book is structured into three parts. In Part I we introduce the
fundamentals of probability theory. We discuss models for random
experiments, conditional probability and independence, random variables,
and probability distributions. Moreover, we explain how to carry out
random experiments on a computer.
In Part II we introduce the general framework for statistical modeling
and inference, from both classical and Bayesian perspectives. We discuss a
variety of common models for data, such as independent random samples,
linear regression, and ANOVA models. Once a model for the data is
determined one can carry out a mathematical analysis of the model on the
basis of the available data. We discuss a wide range of concepts and
techniques for statistical inference, including likelihood-based estimation
and hypothesis testing, sufficiency, confidence intervals, and kernel density
estimation. We encompass both classical and Bayesian approaches and also
highlight popular Monte Carlo sampling techniques.
In Part III we address the statistical analysis and computation of a
variety of advanced models, such as generalized linear models,
autoregressive and moving average models, Gaussian models, and state
space models. Particular attention is paid to fast numerical techniques for
classical and Bayesian inference on these models. Throughout the book our
leading principle is that the mathematical formulation of a statistical model
goes hand in hand with the specification of its simulation counterpart.
The book contains a large number of illustrative examples and problem
sets (with solutions). To keep the book fully self-contained, we include the
more technical proofs and mathematical theory in Appendix B. Appendix A
features a concise introduction to Matlab .
Dirk Kroese
Joshua Chan
Brisbane, Australia Canberra, Australia
Mathematical Notation
Throughout this book we use notation in which different fonts and letter
cases signify different types of mathematical objects. For example, vectors
are written in lowercase boldface font and matrices A , B , X in
uppercase normal font. Sans serif fonts indicate probability distributions,
such as , , and . Probability and expectation symbols are written
in blackboard bold font: and . Matlab code and functions will always
be written in typewriter font.
Traditionally, classical and Bayesian statistics use a different notation
system for random variables and their probability density functions. In
classical statistics and probability theory random variables usually are
denoted by uppercase letters X , Y , Z , … and their outcomes by lowercase
letters x , y , z , … . Bayesian statisticians typically use lowercase letters for
both. More importantly, in the Bayesian notation system, it is common to
use the same letter f (or p ) for different probability densities, as in f ( x , y )
= f ( x ) f ( y ). Classical statisticians and probabilists would prefer a
different symbol for each function, as in . We will
predominantly use the classical notation, especially in the first part of the
book. However, when dealing with Bayesian models and inference, such as
in Chaps. 8 and 11 , it will be convenient to switch to the Bayesian
notation system. Here is a list of frequently used symbols:
≈ Is approximately
∝ Is proportional to
∞ Infinity
⊗ Kronecker product
Is defined as
∼ Is distributed as
, Are independent and identically distributed as
Is approximately distributed as
Maps to
Union of sets A and B
Intersection of sets A and B
A c Complement of set A
A ⊂ B A is a subset of B
∅ Empty set
Euclidean norm of vector
∇ f Gradient of f
∇ 2 f Hessian of f
, Transpose of matrix A or vector
Diagonal matrix with diagonal entries defined by
Trace of matrix A
det( A ) Determinant of matrix A
| A | Absolute value of the determinant of matrix A . Also, number of
elements in set A or absolute value of real number A
is a value x ∗ for which f ( x ∗ ) ≥ f ( x ) for all x
Differential symbol
Expectation
Euler’s constant
Indicator function: equal to 1 if the condition/event A holds
and 0 otherwise.
ln (Natural) logarithm
Set of natural numbers {0,1, … }
Pdf of the standard normal distribution
Φ Cdf of the standard normal distribution
Probability measure
Big-O order symbol: if | f ( x )|⩽ α g ( x ) for some
constant α as x → a
o Little-o order symbol: f ( x ) = o ( g ( x )) if f ( x )∕ g ( x ) → 0 as x
→a
The real line = one-dimensional Euclidean space
Positive real line: [0, ∞ )
n -Dimensional Euclidean space
Estimate/estimator
Vectors
Random vectors
Set of integers { … ,−1,0,1, … }
Probability Distributions
Bernoulli distribution
Beta distribution
Binomial distribution
Cauchy distribution
χ2 Chi-squared distribution
Dirichlet distribution
Discrete uniform distribution
Exponential distribution
F distribution
Gamma distribution
Geometric distribution
Inverse-gamma distribution
Multinomial distribution
Normal or Gaussian distribution
Poisson distribution
Student’s t distribution
TN Truncated normal distribution
Uniform distribution
Weibull distribution
Acknowledgements
This book has benefited from the input of many people. We thank Zdravko
Botev, Tim Brereton, Hyun Choi, Eric Eisenstat, Eunice Foo, Catherine
Forbes, Patricia Galvan, Ivan Jeliazkov, Ross McVinish, Gary Koop,
Rongrong Qu, Ad Ridder, Leonardo Rojas–Nandayapa, John Stachurski,
Rodney Strachan, Mingzhu Sun, Thomas Taimre, Justin Tobias, Elisse
Yulian, and Bo Zhang for their valuable comments and suggestions on
previous drafts of the book.
Abbreviations
ANOVA Analysis of variance
AR Autoregressive
ARMA Autoregressive-moving average
cdf Cumulative distribution function
EM Expectation–maximization
iid Independent and identically distributed
pdf Probability density function (discrete or continuous)
PGF Probability generating function
KDE Kernel density estimate/estimator
MA Moving average
MCMC Markov chain Monte Carlo
MGF Moment generating function
ML(E) Maximum likelihood (estimate/estimator)
PRESS Predicted residual sum of squares
Contents
Part I Fundamentals of Probability
1 ProbabilityModels
1.1 Random Experiments
1.2 Sample Space
1.3 Events
1.4 Probability
1.5 Conditional Probability and Independence
1.5.1 Product Rule
1.5.2 Law of Total Probability and Bayes’ Rule
1.5.3 Independence
1.6 Problems
2 Random Variables and Probability Distributions
2.1 Random Variables
2.2 Probability Distribution
2.2.1 Discrete Distributions
2.2.2 Continuous Distributions
2.3 Expectation
2.4 Transforms
2.5 Common Discrete Distributions
2.5.1 Bernoulli Distribution
2.5.2 Binomial Distribution
2.5.3 Geometric Distribution
2.5.4 Poisson Distribution
2.6 Common Continuous Distributions
2.6.1 Uniform Distribution
2.6.2 Exponential Distribution
2.6.3 Normal (Gaussian) Distribution
2.6.4 Gamma and χ 2 Distribution
2.6.5 F Distribution
2.6.6 Student’s t Distribution
2.7 Generating Random Variables
2.7.1 Generating Uniform Random Variables
2.7.2 Inverse-Transform Method
2.7.3 Acceptance–RejectionMethod
2.8 Problems
3 Joint Distributions
3.1 Discrete Joint Distributions
3.1.1 Multinomial Distribution
3.2 Continuous Joint Distributions
3.3 Mixed Joint Distributions
3.4 Expectations for Joint Distributions
3.5 Functions of Random Variables
3.5.1 Linear Transformations
3.5.2 General Transformations
3.6 Multivariate Normal Distribution
3.7 Limit Theorems
3.8 Problems
Part II Statistical Modeling and Classical and Bayesian Inference
4 Common Statistical Models
4.1 Independent Sampling from a Fixed Distribution
4.2 Multiple Independent Samples
4.3 Regression Models
4.3.1 Simple Linear Regression
4.3.2 Multiple Linear Regression
4.3.3 Regression in General
4.4 Analysis of Variance Models
4.4.1 Single-Factor ANOVA
4.4.2 Two-Factor ANOVA
4.5 Normal Linear Model
4.6 Problems
5 Statistical Inference
5.1 Estimation
5.1.1 Method of Moments
5.1.2 Least-Squares Estimation
5.2 Confidence Intervals
5.2.1 Iid Data: Approximate Confidence Interval for μ
5.2.2 Normal Data: Confidence Intervals for μ and σ 2
5.2.3 Two Normal Samples: Confidence Intervals for μ X − μ Y and
5.2.4 Binomial Data:Approximate Confidence Intervals for
Proportions
5.2.5 Confidence Intervals for the Normal Linear Model
5.3 Hypothesis Testing
5.3.1 ANOVA for the Normal Linear Model
5.4 Cross-Validation
5.5 Sufficiency and Exponential Families
5.6 Problems
6 Likelihood
6.1 Log-Likelihood and Score Functions
6.2 Fisher Information and Cramér–Rao Inequality
6.3 Likelihood Methods for Estimation
6.3.1 Score Intervals
6.3.2 Properties of the ML Estimator
6.4 Likelihood Methods in Statistical Tests
6.5 Newton–Raphson Method
6.6 Expectation–Maximization (EM) Algorithm
6.7 Problems
7 Monte Carlo Sampling
7.1 Empirical Cdf
7.2 Density Estimation
7.3 Resampling and the Bootstrap Method
7.4 Markov Chain Monte Carlo
7.5 Metropolis–Hastings Algorithm
7.6 Gibbs Sampler
7.7 Problems
8 Bayesian Inference