đŸȘŽ Anil's Garden

Derivation of the Least Squares Estimator for Beta in Matrix Notation Economic Theory Blog

02 Apr 20253 min read

Derivation of the Least Squares Estimator for Beta in Matrix Notation | Economic Theory Blog

Excerpt

The following post is going to derive the least squares estimator for , which we will denote as . In general start by mathematically formalizing relationships we think are pre


The following post is going to derive the least squares estimator for \beta, which we will denote as b. In general start by mathematically formalizing relationships we think are present in the real world and write it down in a formula.

(1) y= X\beta +\epsilon

Formula (1) depicts such a model, where \beta represents the true relationship between variables in our population. However, it is rare that we observe the whole population and with it the true relationship \beta.  Most times we observe just a small fraction of what is really going on in the world. Nevertheless, even if you just observe a faction, it is our job to estimate the true value \beta as good as possible. One way to estimate the value of \beta is done by using Ordinary Least Squares Estimator (OLS). In the following we we are going to derive an estimator for \beta. The estimated values for \beta will be called b.

Assume we collected some data and have a dataset which represents a sample of the real world. Let the following equation (2) represent the mathematical model of relationships we presume to exist in the real world and consequently in our sample.

(2) y= Xb +\epsilon

Equation (3) is supposed to present equation (2) in a more intuitively accessible way for those of you who still need some routine in reading matrix notation, however it is really just the same as equation (2).

(3) \begin{bmatrix} y_{1} \ y_{2} \ \vdots \ y_{N} \end{bmatrix} = \begin{bmatrix} x_{11} & x_{12} & \cdots & x_{1K} \ x_{21} & x_{22} & \cdots & x_{2K} \ \vdots & \ddots & \ddots & \vdots \ x_{N1} & x_{N2} & \cdots & x_{NK} \end{bmatrix} * \begin{bmatrix} b_{1} \ b_{2} \ \vdots \ b_{K} \end{bmatrix} + \begin{bmatrix} \epsilon_{1} \ \epsilon_{2} \ \vdots \ \epsilon_{N} \end{bmatrix}

The idea of the ordinary least squares estimator (OLS) consists in choosing b_{i} in such a way that, the sum of squared residual (i.e. \sum_{i=1}^{N} \epsilon_{i}) in the sample is as small as possible. Mathematically this means that in order to estimate the b we have to minimize \sum_{i=1}^{N} \epsilon_{i} which in matrix notation is nothing else than e'e.

(4) \epsilon'\epsilon = \begin{bmatrix} e_{1} & e_{2} & \cdots & e_{N} \ \end{bmatrix} \begin{bmatrix} e_{1} \ e_{2} \ \vdots \ e_{N} \end{bmatrix} = \sum_{i=1}^{N}e_{i}^{2}

In order to estimate b we need to minimize e'e. This is what we are going to do. Per definition we know that e = y - Xb which follows directly from formula (2). Consequently we can write e'e as (y-Xb)'(y-Xb) by simply plugging in the expression e = y - Xb into e'e. This leaves us with the following minimization problem:

(5) min_{b} e'e = (y-Xb)'(y-Xb)

(6) min_{b} e'e = (y'-b'X')(y-Xb)

(7) min_{b} e'e = y'y - b'X'y - y'Xb + b'X'Xb

(8) min_{b} e'e = y'y - 2b'X'y + b'X'Xb

It is important to understand that b'X'y=(b'X'y)'=y'Xb. As both terms are are scalars, meaning of dimension 1×1, the transposition of the term is the same term.

In order to minimize the expression in (8), we have to differentiate the expression in (8) with respect to b and set the derivative equal zero. In order to be able to do that we make use of the following mathematical statements:

  1. \frac{\partial b'X'y}{\partial b}=X'y
  2. \frac{\partial b'X'Xb}{\partial b} =2X'Xb (proof)

Using the two statements allows us to minimize expression (8).

(8) min_{b} e'e = y'y - 2b'X'y + b'X'Xb

(9) \frac{\partial(e'e)}{\partial b} = -2X'y + 2X'Xb \stackrel{!}{=} 0

(10) X'Xb=X'y

Finally to solve expression (9) for b it is necessary to pre-multiply expression (10) with (X'X)^{-1}. This gives us the least squares estimator for \beta.

(11) b=(X'X)^{-1}X'y

One last mathematical thing, the second order condition for a minimum requires that the matrix X'X is positive definite. This requirement is fulfilled in case X has full rank.

Congratulation you just derived the least squares estimator b.

Post navigation

“In God we trust; all others must bring data.” W. Edwards Deming

Graph View

Backlinks

  • No backlinks found