ECMT2150 Properties of the OLS Estimator
Properties of the OLS Estimator
Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: THEend8_
ECMT2150 – Lecture 3
Topics Today
Week 3
Properties of the OLS Estimator
- Focus on Multiple Linear Regression
Reference: 2.3; 2.5; 3.3; 3.4; 3.5; 4.1
A quick reminder of where we are at…
• Discussed some basic properties
• Incorporating non-linearities (in variables => log(x); x^2)
• Handy algebraic properties
• Goodness of fit (be careful with using it – only a guide)
• Discussed the key assumptions for simple linear
regression
• Started to look at the properties for the OLS
estimator under these assumptions
4
Standard assumptions for linear regression
• Assumption SLR.1 - Linear in parameters
• Assumption SLR.2 - Random sampling
• Assumption SLR.3 - Sample variation in explanatory variable (i.e. x)
• Assumption SLR.4 - Zero conditional mean
=> Theorem (Unbiasedness of OLS)
Under assumptions SLR.1 - SLR.4,
8
Property: Unbiasedness
• Theorem (Unbiasedness of OLS)
Under assumptions SLR.1 - SLR.4:
• Interpretation of unbiasedness
• The estimated coefficients may be smaller or larger, depending on the sample
that is the result of a random draw
• However, on average, they will be equal to the values that characterize the
true relationship between y and x in the population
• ``On average‘‘ means if the sampling were repeated (i.e., draw the random
sample and do the estimation over and over again)
• In a given sample, estimates may differ considerably from true values
9
2nd Important Property:
Variance of the OLS estimators
• Depending on the sample, the estimates will be nearer or
further away from the true population values
• How far can we expect our estimates to be from the true
population values on average? => Sampling variability
Sampling variability is measured by the variances
of the estimators:
WHY IS THIS IMPORTANT?
Sampling Variability Standard errors Inference
10
Add another assumption: Homoskedasticity
• Assumption SLR.5 (Homoskedasticity)
The value of the explanatory variable must
contain no information about the
variability of the unobserved factors
11
Property: Sampling Variance of OLS Estimator
• Theorem (Variances of OLS estimators)
Under assumptions SLR.1 – SLR.5
Conclusion: The sampling variability of the estimated regression coefficients
will be:
1. larger for a higher variability in unobserved influences (numerator)
2. smaller for a higher variability in explanatory variable (denominator)
14
Property: Unbiasedness of the error variance
• Theorem (Unbiasedness of the error variance)
We need this for calculation of standard errors for regression coefficients
• the estimated standard deviations of the regression coefficients are
called standard errors
• They measure how precisely the regression coefficients are estimated
Plug-in for
the unknown
15
…And where we going…
Discuss:
• Key assumptions for multiple linear regression
• Properties for the OLS estimator under these
assumptions
• Expected values/unbiasedness under MLR.1 – MLR.4
• Variance formulas under MLR.1 – MLR.5
• Gauss-Markov Theorem under MLR.1 – MLR.5
• Exact sampling distributions/tests under MLR.1 – MLR.6
16
Properties of OLS in Multiple
Linear Regression
Multiple Linear Regression:
Explains variable y in terms of variables x1, x2, …, xk
Standard assumptions for the MLR model
• Assumption MLR.1 (Linear in parameters)
• Assumption MLR.2 (Random sampling)
The population relationship between y and the β’s is linear
The data are randomly drawn
from the population
Each data point follows
the population equation
18
Standard assumptions cont.
• Assumption MLR.3 (No perfect collinearity)
• in the sample (and therefore in the population) ...
none of the independent variables is constant and
there are no exact relationships among the independent variables
• Assumption MLR.4 (Zero conditional mean)
• in a multiple regression model, the zero conditional mean assumption is more
likely to hold because fewer `features‘ end up in the error
19
The value of the explanatory variables
must contain no information about the
mean of the unobserved factors
Property: Unbiasedness
•Theorem (Unbiasedness of OLS)
Under assumptions MLR.1 - MLR.4,
20
Property: Unbiasedness
Remarks:
• Unbiasedness is an average property in repeated samples; in a
given sample, the estimates may still be far away from the true
values
• When we say that OLS is unbiased under MLR.1 – MLR.4 we mean
that the procedure by which the OLS estimates are obtained is
unbiased when that procedure is applied across all possible
random samples
• There is no reason to believe that any particular estimate is more
likely to be too big vs. more likely to be too small
21
Discussion of MLR.3: No Perfect Collinearity
Remarks:
• This only rules out perfect collinearity (i.e. perfect correlation)
between explanatory variables; imperfect correlation is
allowed
• If an explanatory variable is a perfect linear combination of
other explanatory variables, it is redundant and may be
eliminated
• Constant variables are also ruled out (collinear with intercept)
22
Discussion of MLR.3: Examples of Perfect Collinearity
• Perfect collinearity: relationships between regressors
• Either shareA or shareB will have to be dropped from the
regression because there is an exact linear relationship
between them: shareA + shareB = 1
• Perfect collinearity: small sample
• In a small sample, avginc may accidentally be an exact
multiple of expend; it will not be possible to disentangle their
separate effects because there is exact covariation
23
Discussion of MLR.4 – Zero Conditional Mean
• Example: average test scores
• Example: wage return to education
if avginc were not included in the regression, it would end up in
the error term; it would then be hard to defend that expend is
uncorrelated with the error
24
The omission of a key variable leads
to a violation of MLR.4 and is
known as Omitted Variable Bias
(OVB). Without MLR.4, the OLS is
biased.
• Explanatory variables that are correlated with the error term
are called endogenous
o endogeneity is a violation of assumption MLR.4
• Explanatory variables that are uncorrelated with the error
term are called exogenous
o MLR.4 holds if all explanatory variables are exogenous
• Exogeneity is the key assumption for a causal interpretation
of the regression, and for unbiasedness of the OLS
estimators
• MLR.4 is violated and OLS is biased if there are any omitted
variables
25
Discussion of MLR.4 – Zero Conditional Mean
We will return to this later in the semester
Standard assumptions for the MLR model
• Assumption MLR.1 - Linear in parameters
• Assumption MLR.2 - Random sampling
• Assumption MLR.3 - No perfect collinearity
• Assumption MLR.4 - Zero Conditional Mean
• Assumption MLR.5 (Homoskedasticity)
• Assumptions MLR.1-MLR.5:
⇒Theorem: Sampling Variances of the OLS slope estimators
⇒Theorem: Unbiased estimator of the error variance
⇒Theorem: Gauss Markov Therorem – OLS estimators are BLUE
26
The value of the explanatory variables
must contain no information about the
variance of the unobserved factors
Unbiasedness
of OLS
• Example: wage equation
• If the variance changes with any of the 3 explanatory variables, then the
assumption does not hold and we are in the case of heteroskedastic errors.
We will come back to this in Week 9
• NB: short-hand notation
27
This assumption may also be
hard to justify in many cases
with
All explanatory
variables are collected
in a random vector
Discussion of MLR.5 – Homoskedasticity
Multiple Linear Regression
Sampling Variance
Assumptions MLR.1-MLR.5 =>
Theorem: Variances of the OLS slope estimators
Under assumptions MLR.1 – MLR.5:
Variance of the error term
Total sample variation in
explanatory variable xj:
R2 from a regression of explanatory
variable xj on all the other
independent variables (including a
constant)
29
Why do we care? The size of is important – a larger variance
means a less precise estimator and that means less accurate
hypothesis tests – see Weeks 4 & 5
Components of OLS variances
1) the error variance,
• a high error variance increases the sampling variance
because there is more noise in the equation
• a large error variance necessarily makes estimates
imprecise
• error variance does not decrease with sample size
30
Components of OLS variances
2) the total sample variation in the explanatory variable
• more sample variation leads to more precise estimates
• total sample variation automatically increases with the
sample size
• increasing the sample size is a way to get more precise
estimates
31
Components of OLS variances
3) linear relationships among the independent variables,
• Regress on all the other independent variables (including a
constant)
• R2 of this regression will be higher the better can be linearly
explained by the other independent variables
• sampling variance of ̂ will be higher the more correlated the
explanatory variables are with one another – i.e. the more
can be linearly explained by the other independent variables
• problem of almost linearly dependent explanatory variables is
called multicollinearity (i.e. 2 → 1, for some j)
32
Estimating the error variance:
• an unbiased estimate of the error variance is obtained by subtracting
the number of estimated regression coefficients (k + 1) from n
• (n – k – 1) is also called the degrees of freedom
• Why? The n estimated squared residuals in the sum are not completely
independent. They are related through the (k + 1) equations that
define the first order conditions of the minimisation problem
33
Assumptions MLR.1-MLR.5 =>
Theorem: Unbiased estimator of the error variance
Estimation of the sampling variances of the OLS
estimators
Note that these formulas are only valid under assumptions
MLR.1-MLR.5 (in particular, homoscedasticity)
Plug-in for the unknown
The true sampling
variation of the
estimated
The estimated
sampling variation of
the estimated
34
Efficiency of OLS
• Under assumptions MLR.1 - MLR.5 we know OLS is unbiased
• but there may be many other estimators that are unbiased
• Ideally we like the unbiased estimator with the smallest
variance (the most efficient estimator)
• In order to answer this question restrict the analysis to
comparing all estimators that are linear in the dependent
variable:
May be an arbitrary function of the sample
values of all the explanatory variables; the OLS
estimator is of this form
35
Assumptions MLR.1-MLR.5 =>
Theorem: Gauss Markov Theorem
The Gauss-Markov Theorem
• Under assumptions MLR.1 - MLR.5, the OLS estimators
are the Best Linear Unbiased Estimators (BLUEs) of the
regression coefficients, i.e.
OLS is the best estimator if MLR.1 – MLR.5 hold
If there is heteroskedasticity, then there are better (i.e.,
more efficient) estimators
for all for which
36
Let’s pause and take stock
• Properties of the OLS Estimator – SLR and MLR
• Key assumptions SLR.1-SLR.5, MLR.1-MLR.5
• When Assumptions 1-4 hold, then we have:
• OLS estimator is unbiased
• When Assumptions 1-5 hold, then we have:
• OLS estimator is unbiased
• OLS estimator is BLUE
• Examined the variance of the OLS slope estimator
37
What’s next?
• Now that we can derive our estimators (last week), have
studied their properties, and now have expressions for the
variance of our estimators, we are almost ready to conduct
inference
• Inference:
⇒Hypothesis testing
⇒Confidence intervals
• We need one more ingredient:
A Normality Assumption
⇒Assumption MLR.6 - Normality of error terms
• Alternatively, we need to appeal to large sample results – more
on this in Week 5
38
Multiple Linear Regression
Normality
Reminder: Normal Distribution
Normal distribution
• most widely-used and well-known
distribution
• also called the `bell curve’
• proposed by Carl Friedrich Gauss (1777-1855)
Arguably the most important distribution in statistics due to
the Central Limit Theorem (CLT)
Useful for the modelling of many variables in the natural
world (as we shall see later) …. e.g., height, weight, IQ