$$ \newcommand{\mybold}[1]{\boldsymbol{#1}} \newcommand{\trans}{\intercal} \newcommand{\norm}[1]{\left\Vert#1\right\Vert} \newcommand{\abs}[1]{\left|#1\right|} \newcommand{\bbr}{\mathbb{R}} \newcommand{\bbz}{\mathbb{Z}} \newcommand{\bbc}{\mathbb{C}} \newcommand{\gauss}[1]{\mathcal{N}\left(#1\right)} \newcommand{\chisq}[1]{\mathcal{\chi}^2_{#1}} \newcommand{\studentt}[1]{\mathrm{StudentT}_{#1}} \newcommand{\fdist}[2]{\mathrm{FDist}_{#1,#2}} \newcommand{\iid}{\overset{\mathrm{IID}}{\sim}} \newcommand{\argmin}[1]{\underset{#1}{\mathrm{argmin}}\,} \newcommand{\projop}[1]{\underset{#1}{\mathrm{Proj}}\,} \newcommand{\proj}[1]{\underset{#1}{\mybold{P}}} \newcommand{\expect}[1]{\mathbb{E}\left[#1\right]} \newcommand{\prob}[1]{\mathbb{P}\left(#1\right)} \newcommand{\dens}[1]{\mathit{p}\left(#1\right)} \newcommand{\var}[1]{\mathrm{Var}\left(#1\right)} \newcommand{\cov}[1]{\mathrm{Cov}\left(#1\right)} \newcommand{\sumn}{\sum_{n=1}^N} \newcommand{\meann}{\frac{1}{N} \sumn} \newcommand{\cltn}{\frac{1}{\sqrt{N}} \sumn} \newcommand{\trace}[1]{\mathrm{trace}\left(#1\right)} \newcommand{\diag}[1]{\mathrm{Diag}\left(#1\right)} \newcommand{\grad}[2]{\nabla_{#1} \left. #2 \right.} \newcommand{\gradat}[3]{\nabla_{#1} \left. #2 \right|_{#3}} \newcommand{\fracat}[3]{\left. \frac{#1}{#2} \right|_{#3}} \newcommand{\W}{\mybold{W}} \newcommand{\w}{w} \newcommand{\wbar}{\bar{w}} \newcommand{\wv}{\mybold{w}} \newcommand{\X}{\mybold{X}} \newcommand{\x}{x} \newcommand{\xbar}{\bar{x}} \newcommand{\xv}{\mybold{x}} \newcommand{\Xcov}{\mybold{M}_{\X}} \newcommand{\Xcovhat}{\hat{\mybold{M}}_{\X}} \newcommand{\Covsand}{\Sigmam_{\mathrm{sand}}} \newcommand{\Covsandhat}{\hat{\Sigmam}_{\mathrm{sand}}} \newcommand{\Z}{\mybold{Z}} \newcommand{\z}{z} \newcommand{\zv}{\mybold{z}} \newcommand{\zbar}{\bar{z}} \newcommand{\Y}{\mybold{Y}} \newcommand{\Yhat}{\hat{\Y}} \newcommand{\y}{y} \newcommand{\yv}{\mybold{y}} \newcommand{\yhat}{\hat{\y}} \newcommand{\ybar}{\bar{y}} \newcommand{\res}{\varepsilon} \newcommand{\resv}{\mybold{\res}} \newcommand{\resvhat}{\hat{\mybold{\res}}} \newcommand{\reshat}{\hat{\res}} \newcommand{\betav}{\mybold{\beta}} \newcommand{\betavhat}{\hat{\betav}} \newcommand{\betahat}{\hat{\beta}} \newcommand{\betastar}{{\beta^{*}}} \newcommand{\betavstar}{{\betav^{*}}} \newcommand{\loss}{\mathscr{L}} \newcommand{\losshat}{\hat{\loss}} \newcommand{\f}{f} \newcommand{\fhat}{\hat{f}} \newcommand{\bv}{\mybold{\b}} \newcommand{\bvhat}{\hat{\bv}} \newcommand{\alphav}{\mybold{\alpha}} \newcommand{\alphavhat}{\hat{\av}} \newcommand{\alphahat}{\hat{\alpha}} \newcommand{\omegav}{\mybold{\omega}} \newcommand{\gv}{\mybold{\gamma}} \newcommand{\gvhat}{\hat{\gv}} \newcommand{\ghat}{\hat{\gamma}} \newcommand{\hv}{\mybold{\h}} \newcommand{\hvhat}{\hat{\hv}} \newcommand{\hhat}{\hat{\h}} \newcommand{\gammav}{\mybold{\gamma}} \newcommand{\gammavhat}{\hat{\gammav}} \newcommand{\gammahat}{\hat{\gamma}} \newcommand{\new}{\mathrm{new}} \newcommand{\zerov}{\mybold{0}} \newcommand{\onev}{\mybold{1}} \newcommand{\id}{\mybold{I}} \newcommand{\sigmahat}{\hat{\sigma}} \newcommand{\etav}{\mybold{\eta}} \newcommand{\muv}{\mybold{\mu}} \newcommand{\Sigmam}{\mybold{\Sigma}} \newcommand{\rdom}[1]{\mathbb{R}^{#1}} \newcommand{\RV}[1]{{#1}} \def\A{\mybold{A}} \def\A{\mybold{A}} \def\av{\mybold{a}} \def\a{a} \def\B{\mybold{B}} \def\b{b} \def\S{\mybold{S}} \def\sv{\mybold{s}} \def\s{s} \def\R{\mybold{R}} \def\rv{\mybold{r}} \def\r{r} \def\V{\mybold{V}} \def\vv{\mybold{v}} \def\v{v} \def\vhat{\hat{v}} \def\U{\mybold{U}} \def\uv{\mybold{u}} \def\u{u} \def\W{\mybold{W}} \def\wv{\mybold{w}} \def\w{w} \def\tv{\mybold{t}} \def\t{t} \def\Sc{\mathcal{S}} \def\ev{\mybold{e}} \def\Lammat{\mybold{\Lambda}} \def\Q{\mybold{Q}} \def\eps{\varepsilon} $$

STAT151A Quiz 2

Please write your full name and email address here:

\[\\[2in]\]

Also, please put your intials on each page in case the pages get separated.

\[\\[1in]\]

You have 30 minutes for this quiz.

There are three questions, (1), (2), and (3), each weighted equally..

There are extra pages at the end if you need more space for solutions.

Question 1

Consider the regression $\y_n \sim \beta_1 + \beta_2 \z_n + \beta_3 r_n$, where $n=1,\ldots,N$. Let $\betav = (\beta_1, \beta_2, \beta_3)^\trans$.

(a) Write the regression in the form $\Y \sim \X \betav$. Be precise about the dimensions and entries of the matrix $\X$.

(b) In terms of $N$ and the quantites defined in Equation 1, write expressions for $\X^\trans \X$ and $\X^\trans \Y$. \[ \begin{aligned} \ybar =& \meann \y_n & \overline{yz} =& \meann \y_n \z_n & \overline{yr} :=& \meann \y_n r_n \\ \zbar =& \meann \z_n & \overline{zr} =& \meann \z_n r_n & \overline{r} :=& \meann r_n \\ \overline{zz} =& \meann \z_n^2 & \overline{rr} =& \meann r_n^2 \\ \end{aligned} \tag{1}\]

(c) Suppose now that:

$\z_n$ and $r_n$ are both random and IID. So $\z_n$ is independent of $r_n$, and both $z_n$ and $r_n$ are independent of $\z_m$ and $r_m$ for $m \ne n$,
$\expect{\z_n} = \expect{r_n} = 0$, and
$\var{\z_n} = \sigma_z^2$, and $\var{r_n} = \sigma_r^2$.

What is $\lim_{N \rightarrow \infty}\frac{1}{N} \X^\trans \X$?

Question 2

Consider two categorical variables, $\z_{n1}$ and $\z_{n2}$, where $\z_{n1}$ is either “good” or “bad”, and $\z_{n2}$ is either “red” or “yellow”. Note, for example, that an observation can by “good” and “red” at the same time.
However, an observation cannot be “good” and “bad” at the same time, nor can it be “red” and “yellow” at the same time.

Define the one-hot encodings \[ \begin{aligned} \x_{ng} ={}& 1 \textrm{ when }\z_{n1}\textrm{ is ``good'' and }0\textrm{ otherwise} \\ \x_{nb} ={}& 1 \textrm{ when }\z_{n1}\textrm{ is ``bad'' and }0\textrm{ otherwise} \\ \textrm{and}\\ \x_{nr} ={}& 1 \textrm{ when }\z_{n2}\textrm{ is ``red'' and }0\textrm{ otherwise} \\ \x_{ny} ={}& 1 \textrm{ when }\z_{n2}\textrm{ is ``yellow'' and }0\textrm{ otherwise}. \end{aligned} \]

Consider the regression $\y_n \sim \beta_1 \x_{ng} + \beta_2 \x_{nr}$, where $n=1,\ldots,N$. Let $\betav = (\beta_1, \beta_2)^\trans$. That is, we are regressing only on the one–hot encodings for “good” and for “red”.

Let $N_g$ denote the number of rows with $\z_{n1} =$ “good”, $N_r$ denote the number of rows with $\z_{n2} =$“red”, and so on. Similarly, let $N_{gr}$ denote the number of rows with both $\z_{n1} =$“good” and $\z_{n2} =$“red”.

(a) Write $\X^\trans \X$ in terms of $N_g$, $N_r$, and $N_{gr}$.

(b) Write a formula in terms of $N_g$, $N_r$, and $N_{gr}$ that tells when $\X^\trans \X$ is invertible.

Hint: recall that the determinant of the $2\times 2$ matrix $\begin{pmatrix} a & b \\ c & d \end{pmatrix}$ is given by $ad - bc$.

(c) Suppose that every “good” row is also “red”, and every “red” row is “good”. Is $\X^\trans \X$ invertible? Justify your answer.

Question 3

In the setting of Question 2, consider the regression $\y_n \sim \beta_0 + \beta_1 \x_{ng} + \beta_2 \x_{nb}$. Note that a row is either “good” or “bad”, so that exactly one of $\x_{ng}$ or $\x_{nb}$ is equal to $1$ for any particular observation $n$. Let $\betav = (\beta_0, \beta_1, \beta_2)^\trans$, and $\X$ the corresponding regressor matrix.

(a) Let $N_g$ denote the number of rows with $\z_{n1} =$ “good” and $N_b$ denote the number of rows with $\z_{n1} =$ “bad”. In terms of $N$, $N_g$, and $N_b$, write an expression for $\X^\trans \X$.

(b) Suppose that $\ybar_g = \frac{1}{N_g} \sum_{\textrm{good } n} \y_n$ and $\ybar_b = \frac{1}{N_b} \sum_{\textrm{bad } n} \y_n$ denote the average of $\y_n$ in “good” and “bad” rows, respectively. Find at least one $\betavhat$ that satisfies $\X^\trans \X \betavhat = \X^\trans \Y$.

(c) Find another value $\betahat'$, different than the answer you gave in (b), such that $\betavhat'$ also satisfies $\X^\trans \X \betavhat' = \X^\trans \Y$.

Extra space for answers (indicate clearly which problem you are working on)