Chapter 40 : Regression and Linear regression

Let $(X_{i},Y_{i})$ be i.i.d random variables. For example, we could pick people at random from a population and measure their height ($X$) and weight ($Y$). One question of interest is to predict the value of $Y$ from the value of $X$. This may be useful if $Y$ is difficult to measure directly. For instance, $X$ could be the height of a person and $Y$ could be the xxx In other words, we assume that there is an underlying relationship $Y=f(X)$ for an unknown function $f$ which we want to find. From a random sample $(X_{1},Y_{1}),\ldots ,(X_{n},Y_{n})$ we try to guess the function $f$. If we allow all possible functions, it is easy to find one that fits all the data points, i.e., there exists a function $f:\mathbb{R}\rightarrow \mathbb{R}$ (in fact we may take $f$ to be a polynomials of degree $n$) such that $f(X_{i})=Y_{i}$ for each $i\le n$ (this is true only if we assume that all $X_{i}$ are distinct which happens if $X$ has a continuous distribution). This is not a good predictor, because the next data point $(U,V)$ will fall way off the curve. We have found a function that ''predicts'' well all the data we have, but not for a future observation! Instead, we fix a class of functions, for example the collection of all linear functions $y=mx+c$ where $m,c\in \mathbb{R}$ and within this class, find the best fitting function.

 

Finding the best linear fit : We need a criterion for deciding the ''best''. A basic one is the method of least squares which recommends finding $\alpha,\beta$ such that the error sum of squares $R^{2}:=\sum_{k=1}^{n}(Y_{k}-\alpha-\beta X_{k})^{2}$ is minimized. For fixed $X_{i},Y_{i}$ this is a simple problem in calculus. We get $$ \hat{\beta}=\frac{\sum_{k=1}^{n}(X_{k}-\bar{X}_{n})(Y_{k}-\bar{Y}_{n})}{\sum_{k=1}^{n}(X_{k}-\bar{X}_{n})^{2} }=\frac{s_{X,Y} }{s_{X}^{2} }, \qquad \hat{\alpha}=\bar{Y}_{n}-\hat{\beta}\bar{X}_{n} $$ where $s_{X,Y}$ is the sample covariance of $X,Y$ and $s_{X}$ is the sample variance of $X$. We leave the derivation of the least squares estimators by calculus to you. Instead we present another approach. For a given choice of $\beta$, we know that the choice of $\alpha$ which minimizes $R^{2}$ is the sample mean of $Y_{i}-\beta X_{i}$ which is $\bar{Y}-\beta \bar{X}$. Thus, we only need to find $\hat{\beta}$ that minimizes $$\sum_{k=1}^{n}\left((Y_{k}-\bar{Y})-\beta(X_{k}-\bar{X})\right)^{2}$$ and then we simply set $\hat{\alpha}=\bar{Y}-\beta\bar{X}$. Let ¹ We are dividing by $X_{k}-\bar{X}$. What if it is zero for some $k$? But note that in the expression $\sum \left((Y_{k}-\bar{Y})-\beta(X_{k}-\bar{X})\right)^{2}$, all such terms do not involve $\beta$ and hence can be safely left out of the summation. We leave the details for you to work out (the expressions at the end should involve all $X_{k},Y_{k}$). $Z_{k}=\frac{Y_{k}-\bar{Y} }{X_{k}-\bar{X} }$ and $w_{k}=(X_{k}-\bar{X})^{2}/s_{X}^{2}$. Then, $$ \sum_{k=1}^{n}\left((Y_{k}-\bar{Y})-\beta(X_{k}-\bar{X})\right)^{2}=s_{X}^{2}\sum_{k=1}^{n}w_{k}\left(Z_{k}-\beta \right)^{2}. $$ Since $w_{k}$ are non-negative numbers that add to $1$, we can intepret it as a probability mass function and hence we see that the minimizing $\beta$ is given by the expectation with respect to this mass function. In other words, $$ \hat{\beta}=\sum_{k=1}^{n}w_{k}Z_{k} = \frac{s_{X,Y} }{s_{X}^{2} }. $$ Another way to write it is $\hat{\beta}=\frac{s_{Y} }{s_{X} }r_{X,Y}$ where $r_{X,Y}$ is the sample correlation coefficient. A motivation for the least squares criterion : Suppose we make more detailed model assumptions as follows. Let $X$ be a control variable (i.e., not random but we can tune it to any value, like temperature) and assume that $Y_{i}=\alpha+\beta X_{i}+\epsilon_{i}$ where $\epsilon_{i}$ are i.i.d. $N(0,{\sigma}^{2})$ ''errors''. Then, the data is essential $Y_{i}$ that are independent $N(\alpha+\beta X_{i},{\sigma}^{2})$ random variables. Now we can extimate $\alpha,\beta$ by the maximum likelihood method.

 

 

How good is the fit? For the same data $(X_{1},Y_{1}),\ldots ,(X_{n},Y_{n})$, suppose we have two candidates (a) $Y=f(X)$ and (b) $Y=g(X)$. How to decide which is better? Or how to say if a fit is good at all? By the least-squares criterion, the answer is the one with smaller residual sum of squares $SS:=\sum_{k=1}^{n}(Y_{k}-f(X_{k}))^{2}$. Usually one presents a closely related quantity $R^{2}=1-\frac{SS}{SS_{0} }$ (where $\mbox{SS}_{0}=\sum_{k=1}^{n}(Y_{k}-\bar{Y})^{2}=(n-1)s_{Y}^{2}$). Since $SS_{0}$ is (a multiple of) the total variance in $Y$, $R^{2}$ measures how much of it is ''explained'' by a particular fit. Note that $0\le R^{2}\le 1$. And higher (i.e., closer to $1$) the $R^{2}$ is, the better the fit. Thus, the first naive answer to the above question is to compute $R^{2}$ in the two situations (fitting by $f$ and fitting by $g$) and see which is higher. But a more nuanced approach is preferable. Consider the same data and three situations.

1. Fit a constant function. This means, choose $\alpha$ to minimize $\sum_{k=1}^{n}(Y_{k}-\alpha)^{2}$. The solution is $\hat{\alpha}=\bar{Y}$ and the residual sum of squares is $\mbox{SS}_{0}$ itself. Then, $R_{0}^{2}=0$.
2. Fit a linear function. Then $\alpha,\beta$ are chosen as discussed earlier and the residual sum of squares is $\mbox{SS}_{1}=\sum_{k=1}^{n}(Y_{k}-\hat{\alpha}-\hat{\beta}X_{k})^{2}$. Then, $R_{1}^{2}=1-\frac{SS_{1} }{SS_{0} }$.
3. Fit a quadratic function. The the residual sum of squares is $\mbox{SS}_{2}=\sum_{k=1}^{n}(Y_{k}-\hat{\alpha}-\hat{\beta}X_{k}-\hat{\gamma}X_{k}^{2})^{2}$ where $\hat{\alpha},\hat{\beta},\hat{\gamma}$ are chosen so as to minimize $\sum_{k=1}^{n}(Y_{k}-\alpha-\beta X_{k}-\gamma X_{k}^{2})^{2}$. Then $R_{2}^{2}=1-\frac{SS_{2} }{SS_{0} }$.

Obviously we will have $R_{2}^{2}\ge R_{1}^{2}\ge R_{0}^{2}$ (since linear functions include constants and quadratic functions include linear ones). Does that mean that the third is better? If that were the conclusion, then we can continue to introduce more parameters as that will always reduce the residual sum of squares! But that comes at the cost of making the model more complicated (and having too many parameters means that it will fit the current data well, but not future data!). When to stop adding more parameters? Qualitatively, a new parameter is desirable if it leads to a significant increase of the $R^{2}$. The question is, how big an increase is significant. For this, one introduces the notion of adjusted $R^{2}$, which is defined as follows: If the model has $p$ parameters, then define $\bar{SS}=SS/(n-1-p)$. In particular, $\bar{SS}_{0}=\frac{SS_{0} }{n-1}=s_{Y}^{2}$. Then define the adjusted $R^{2}$ as $\bar{R}^{2}=1-\frac{\bar{SS} }{\bar{SS}_{0} }$. In particular, $\bar{R}_{0}^{2}=R_{0}^{2}$ as before. But $R_{1}^{2}=1-\frac{SS_{1}/(n-2)}{SS_{0}/(n-1)}$. Note that $\bar{R}^{2}$ does not necessarily increase upon adding an extra parameter. If we want a polynomial fit, then a rule of thumb is to keep adding more powers as long as $\bar{R}^{2}$ continues to increase and stop the moment it decreases.