Review: Linear algebra
Goals
- Review linear algebra.
- Matrices and vectors
- Basic operations: multiplication, transpose, inverse
- Geometry: length, orthogonality, and projections
- Eigenvalues and eigenvectors
Reading
This lecture supplements the reading
- Freedman (2009) Chapter 3
Additional linear algebra review materials are described in this page.
Linear algebra review
These lecture notes won’t replace the readings. The topics I’ll review are:
- Dimensions of matrices and vectors
- Multiplication, transposes, and inner products
- Linear systems and inverses of matrices
- Inner products and lengths
- Inner products and orthogonality
- Basis vectors and orthogonal matrices
- Eigenvalues and eigenvectors
Generic random vectors
In general, we can consider a random vector — or a random matrix — as a collection of potentially non-independent random values. For such a vector \({\boldsymbol{z}} \in \mathbb{R}^{P}\), we can speak about its expectation and covariance. Specifically,
\[ \mathbb{E}\left[{\boldsymbol{z}}\right] = \begin{pmatrix} \mathbb{E}\left[{z}_1\right] \\ \vdots \\ \mathbb{E}\left[{z}_P\right] \\ \end{pmatrix} \quad\textrm{and}\quad \mathrm{Cov}\left({\boldsymbol{z}}\right) = \mathbb{E}\left[\left({\boldsymbol{z}} - \mathbb{E}\left[{\boldsymbol{z}}\right] \right) \left({\boldsymbol{z}} - \mathbb{E}\left[{\boldsymbol{z}}\right] \right)^\intercal\right] = \begin{pmatrix} \mathrm{Var}\left({z_1}\right) & \mathrm{Cov}\left({z_1}, {z_2}\right) & \ldots & \mathrm{Cov}\left({z_1}, {z_P}\right) \\ \mathrm{Cov}\left({z_2}, {z_1}\right) & \ldots & \ldots & \mathrm{Cov}\left({z_2}, {z_P}\right) \\ \vdots & & & \vdots \\ \mathrm{Cov}\left({z_P}, {z_1}\right) & \ldots & \ldots & \mathrm{Var}\left({z_P}\right) \\ \end{pmatrix}. \]
For these expressions to exist, it suffices for \(\mathbb{E}\left[{z}_p^2\right] < \infty\) for all \(p \in \{1,\ldots,P\}\).
We will at times talk about the expectation of a random matrix, \({\boldsymbol{Z}}\), which is simply the matrix of expectations. The notation for covariances of course doesn’t make sense for matrices, but it won’t be needed. (In fact, in cases where the covariance of a random matrix is needed in statistics, the matrix is typically stacked into a vector first.)
The law of large numbers for vectors
The law of large numbers is particularly simple for vectors — as long as the dimension stays fixed as \(N \rightarrow \infty\), you can simply apply the LLN to each component separately. Suppose you’re given a sequence of vector-valued random variables, \({\boldsymbol{z}}_n \in \mathbb{R}^{P}\). Write \(\mathbb{E}\left[{\boldsymbol{z}}_n\right] = \boldsymbol{\mu}_n\). Then, as long as we can apply the LLN to each component, we get
\[ \overline{{\boldsymbol{z}}} := \frac{1}{N} \sum_{n=1}^N{\boldsymbol{z}}_n \rightarrow \overline{\boldsymbol{\mu}} \quad\textrm{as }N\rightarrow \infty. \]
Extensions
Since we see that the shape of the vector doesn’t matter, we can also apply the LLN to matrices. For example, if \({z}_n\) is a sequence of random matrices, with \(\mathbb{E}\left[{z}_n\right] = \boldsymbol{A}\), then
\[ \frac{1}{N} \sum_{n=1}^N{z}_n \rightarrow \boldsymbol{A}. \]
We will also use (without proof) a theorem called the continuous mapping theorem, which says that, for a continous function \(f(\cdot)\), then
\[ f\left(\frac{1}{N} \sum_{n=1}^N{\boldsymbol{z}}_n\right) \rightarrow f(\overline{\mu}). \]
Note that the preceding statment is different than saying
\[ \frac{1}{N} \sum_{n=1}^Nf\left( {\boldsymbol{z}}_n \right) \rightarrow \mathbb{E}\left[f\left( {\boldsymbol{z}}_n \right)\right], \]
which may also be true, but which applies the LLN to the random variables \(f\left( {\boldsymbol{z}}_n \right)\).