讀書筆記，Hastie and Tibshirani's An Introduction to Statistical Learning

• p.34, (2.7):
  $$\text{E}(y_0-\hat{f}(x_0))^2 =\text{Var}(\hat{f}(x_0))+[\text{Bias}(\hat{f}(x_0))]^2+\text{Var}(\epsilon).$$
  
  Proof: I don't know why. The authors don't prove it in the book The Elements of Statistical Learning (3.22).


• p.79, line -14:
  Recall that in simple regression, $R^2$ is the square of the correlation of the response and the variable. In multiple linear regression, it turns out that it equals $\text{Cor}(Y, \hat{Y})^2$,
  
  Proof: I don't know why.


• p.98, (3.37):
  $$h_i=\frac{1}{n}+\frac{(x_i-\bar{x})^2}{\sum_{i'=1}^{n}(x_{i'}-\bar{x})^2}.$$
  
  Proof: See Anderson, p.707, (14.33) and Casella, p.557, subsec. 11.3.5.


• p.133, sec.4.3.2:
  Estimating the regression coefficients $\beta_0$ and $\beta_1$ in $p(X)=\frac{e^{\beta_0+\beta_1 X}}{1+e^{\beta_0+\beta_1 X}}$
  
  Proof: See Casella, p.593, subsec.12.3.2.


• p.138, sec.4.4:
  Linear Discriminant Analysis
  
  Proof: The Elements of Statistical Learning p.116, subsec.4.3.3 有直觀解釋。


• p.140, (4.13):
  $$\delta_k(x)=x\cdot \frac{\mu_k}{\sigma^2}-\frac{\mu_k^2}{2\sigma^2}+\log{(\pi_k)}$$
  
  Proof: 在(4.12) $$p_k(x)=\frac{\pi_k \frac{1}{\sqrt{2\pi}\sigma}\exp{\left(-\frac{1}{2\sigma^2}(x-\mu_k)^2\right)}}{\sum_{l=1}^{K}\pi_l \frac{1}{\sqrt{2\pi}\sigma}\exp{\left(-\frac{1}{2\sigma^2}(x-\mu_l)^2\right)}}$$ 中，是 $x$ 固定，$k$ 在變動。因為分母不會隨著 $k$ 變動而變動，所以不用管，分子的 $\frac{1}{\sqrt{2\pi}\sigma}$ 也是一樣不用管。將分子的 $(x-\mu_k)^2$ 展開後的 $\frac{-x^2}{2\sigma^2}$ 也不用管。


• p.143, (4.19):
  $$\delta_k(x)=x^T\mathbf{\Sigma}\mu_k-\frac{1}{2}\mu_k^T \mathbf{\Sigma}^{-1}\mu_k+\log{\pi_k}$$
  
  Proof: 類似(4.13)的證明，但這裡比較不一樣的地方是，因為 $\mathbf{x}^T\mathbf{\Sigma}^{-1}\boldsymbol{\mu}_k$ 是一個scalar，所以 $$\boldsymbol{\mu}_k^T\mathbf{\Sigma}^{-1}\mathbf{x} = \mathbf{x}^T\mathbf{\Sigma}^{-1}\boldsymbol{\mu}_k$$


• p.148, fig.4.8:
  The true positive rate is the sensitivity: the fraction of defaulters that are correctly identified, using a given threshold value. The false positive rate is 1-specificity: the fraction of non-defaulters that we classify incorrectly as defaulters, using that that same threshold value.
  
  Proof: 想想我們生活中的用語“偽陽性”，就比較容易理解這段話。


• p.151, (4.24):
  $$\log{\left(\frac{p_1(x)}{1-p_1(x)}\right)}=\log{\left(\frac{p_1(x)}{p_2(x)}\right)}=c_0+c_1x$$
  
  Proof: $c_0=\ln{\frac{\pi_1}{\pi_2}}-\frac{\mu_1^2-\mu_2^2}{2\sigma^2}$, $c_1=\frac{(\mu_1-\mu_2)}{\sigma^2}$.


• p.187, (5.6):
  $$\alpha=\frac{\sigma_Y^2-\sigma_{XY}}{\sigma_X^2+\sigma_Y^2-2\sigma_{XY}}$$
  
  Proof: By [Casella, p.171, thm.4.5.6], $$\text{Var}(\alpha X+(1-\alpha)Y)=\alpha^2\text{Var}(X)+(1-\alpha)^2\text{Var}(Y)+2\alpha(1-\alpha)\text{Cov}(X, Y).$$ Express it in terms of $\alpha$. We have $$f(\alpha)=2[\text{Var}(X)+\text{Var}(Y)-2\text{Cov}(X, Y)]\alpha^2+[-2\text{Var}(Y)+2\text{Cov}(X, Y)]\alpha+\text{Var}(Y).$$ Solve $f'(\alpha)=0$. Then we have $$\alpha=\frac{\sigma^2_Y-\sigma_{XY}}{\sigma^2_X+\sigma^2_Y-2\sigma_{XY}}.$$


• p.213, line 20
  As an alternative to the approaches just discussed, we can directly estimate the test error using the validation set and cross-validation methods discussed in Chapter 5.
  
  Proof: 這裡並沒有說明清楚該怎麼用cross-validation，下面說明使用方法，主要是參考p.275, line 4。
  
  $K$-fold cross-validation
  
  在一個模型 $M$ 中，有個值 $n$ 要決定，$n_1, n_2, ...$ 是候選的值。
  • 在sec.6.1是要決定predictor的個數 $p=1\text{ or }2\text{ or }3\text{ or }\cdots$
  • 在sec.7.4是要決定knot的個數 $k=1\text{ or }2\text{ or }3\text{ or }\cdots$（原文用大寫 $K$ ，但這裡避免符號重複，改用小寫 $k$）。
  我們會按照下面的程序來決定 $n$ 的值。
  • 假設 $n=n_1$，把資料分成 $K$ 份 $g_1, g_2, g_3, ..., g_K$。
    • 取出 $g_1$ 放旁邊，用 $g_2, g_3, ..., g_K$ train出模型 $M$（記住，這時候模型是假設 $n=n_1$），然後用 $g_1$ test出誤差 $e_1$。
    • 取出 $g_2$ 放旁邊，用 $g_1, g_3, ..., g_K$ train出模型 $M$（記住，這時候模型是假設 $n=n_1$），然後用 $g_2$ test出誤差 $e_2$。
    • 反覆此步驟
    得到 $e_1, e_2, ..., e_K$，並計算 $\frac{e_1+e_2+\cdots+e_K}{K}=c_1$。
  • 假設 $n=n_2$，把資料分成 $K$ 份 $g_1, g_2, g_3, ..., g_K$。
    • 取出 $g_1$ 放旁邊，用 $g_2, g_3, ..., g_K$ train出模型 $M$（記住，這時候模型是假設 $n=n_2$），然後用 $g_1$ test出誤差 $e_1$。
    • 取出 $g_2$ 放旁邊，用 $g_1, g_3, ..., g_K$ train出模型 $M$（記住，這時候模型是假設 $n=n_2$），然後用 $g_2$ test出誤差 $e_2$。
    • 反覆此步驟
    得到 $e_1, e_2, ..., e_K$，並計算 $\frac{e_1+e_2+\cdots+e_K}{K}=c_2$。
  • 反覆此步驟
  得到 $c_1, c_2, ...$，看哪個 $c_i$ 最小，就選擇 $n_i$。


• p.214, line -1
  It may not be immediately obvious why such a constraint should improve the fit, but it turns out that shrinking the coefficient estimates can significantly reduce their variance.
  
  Proof: 參考Bishop的Pattern Recognition and Machine Learning，p.8, table 1.1，當有 $M=9$ 個predictors的時候，estimated coefficients的值變得很大，所以把p.5, (1.2) $$\text{E}(\mathbf{w})=\frac{1}{2}\sum_{n=1}^{N}\{y(x_n, \mathbf{w})-t_n\}^2$$ 改成p.10, (1.4) $$\tilde{\text{E}}(\mathbf{w})=\frac{1}{2}\sum_{n=1}^{N}\{y(x_n, \mathbf{w})-t_n\}^2(\mathbf{w})+\frac{\lambda}{2}||\mathbf{w}||^2$$ 這樣就可以限制那些estimated coefficient的大小。


• p.220, (6.8)
  One can show that the lasso and ridge regression coefficient estimates solve the problems $$\underset{\beta}{\text{minimize}}\left\{\sum_{i=1}^{n}\left(y_i-\beta_0-\sum_{j=1}^{p}\beta_j x_{ij}\right)^2\right\} \text{ subject to }\sum_{j=1}^{p}|\beta_j|\leq s$$
  
  Proof: I don't know why.


• p.231, line 24.
  $\text{Var}(\phi_{11}\times(\text{pop}-\overline{\text{pop}})+\phi_{21}\times(\text{ad}-\overline{\text{ad}})$
  
  Proof: See Jolliffe's Principal Component Analysis Sections 1.1, p.5，但是Hastie是考慮 $$\mathbf{X}-\boldsymbol{\mu} = \begin{pmatrix} \text{pop}-\mu_{\text{pop}}\\ \text{ad}-\mu_{\text{ad}} \end{pmatrix}$$ 的covariance matrix而不是 $\mathbf{X}$ 的covariance matrix。這並沒有差別，因為兩個是一樣的，參考Hogg, IMS, p.141, (2.6.13)及p.143, thm.2.6.3, (2.6.15)。 $$\begin{array}{lll} \text{Cov}(\mathbf{X}-\boldsymbol{\mu}) &=& \text{E}((\mathbf{X}-\boldsymbol{\mu})(\mathbf{X}-\boldsymbol{\mu})^T) \\ &=& \text{E}((\mathbf{X}-\boldsymbol{\mu})(\mathbf{X}^T-\boldsymbol{\mu}^T)) \\ &=& \text{E}(\mathbf{X}\mathbf{X}^T-\boldsymbol{\mu}\mathbf{X}^T-\mathbf{X}\boldsymbol{\mu}^T+\boldsymbol{\mu}\boldsymbol{\mu}^T) \\ &=& \text{E}(\mathbf{X}\mathbf{X}^T)-\boldsymbol{\mu}\text{E}(\mathbf{X}^T)-\text{E}(\mathbf{X})\boldsymbol{\mu}^T+\boldsymbol{\mu}\boldsymbol{\mu}^T \\ &=& \text{E}(\mathbf{X}\mathbf{X}^T)-\boldsymbol{\mu}\boldsymbol{\mu}^T \\ &=& \text{Cov}(\mathbf{X}). \end{array}$$


• p.232, fig.6.15, left panel.
  
  
  Proof: 圖中直線的求法可以參考Casella, p.581, subsec.12.2.2或是Jolliffe's Principal Component Analysis p.34, prop.G3.


• p.267, line -5
  What is the variance of the fit, i.e. $\text{Var}(\hat{f}(x_0))$?
  
  Proof: See Montgomery's Introduction to Linear Regression Analysis, ch.3.


• p.278, subsec.7.5.2
  Choosing the Smoothing Parameter $\lambda$
  
  Proof: See Wang's Smoothing Splines Methods and Applications, ch.3.


• p.
  bla
  
  Proof:


• p.
  bla
  
  Proof:


• p.
  bla
  
  Proof:


• p.
  bla
  
  Proof:


• p.
  bla
  
  Proof: