23考研一元线性回归笔记

1. Random Variables

$$Y_i=\beta_0+\beta_1X_i+\varepsilon_i$$ $$df=n-2$$

1-1. $\varepsilon_i$

Distribution

$$\varepsilon_i\sim N(0,\sigma^2)\ \text{i.i.d}$$

Properties

$$\frac{\varepsilon_i}{\sigma}\sim {\rm Std}N$$ $$\frac{\sum \varepsilon_i^2}{\sigma^2}\sim \chi^2(n-2)$$

Point Estimation

$$s^2_\varepsilon:=s^2={\rm MSE}_{\rm OBS}=\frac{{\rm SSE}_{\rm OBS}}{n-2}=\frac{\sum e_i^2}{n-2}$$

1-2. ${\rm SSE}/\sigma^2$

Definition

$${\rm SSE}:=\sum \varepsilon_i^2$$

Distribution

$$\frac{{\rm SSE}}{\sigma^2}=\frac{\sum \varepsilon_i^2}{\sigma^2}\sim \chi^2(n-2)$$

1-3. ${\rm MSE}/\sigma^2$

Definition

$${\rm MSE}:=\frac{{\rm SSE}}{n-2}$$

Distribution

$$\frac{{\rm MSE}}{\sigma^2}=\frac{{\rm SSE}/n-2}{\sigma^2}\sim \frac{\chi^2(n-2)}{n-2}$$

1-4. $Y_i$

Definition

$$Y_i:=\beta_0+\beta_1X_i+\varepsilon_i,\quad \beta_0,\beta_1,X_i\in \mathbb R$$

$\hat Y_i\in \mathbb R$

Definition

$$\hat Y_i=\beta_0+\beta_1X_i$$

Properties

$$Y_i=\hat Y_i+\varepsilon_i$$ $$\bar Y=\frac{1}{n}\sum \hat Y_i=\frac{1}{n}\sum (\beta_0+\beta_1X_i)=\beta_0+\beta_1\bar X$$

Distribution

$$Y_i\sim \hat Y_i+N(0,\sigma^2)=N(\hat Y_i,\sigma^2)\ \text{i.i.d}$$

1-5. $b_1$

Definition

$$b_1:=\frac{\ell_{xy}}{\ell_{xx}}=\frac{\sum (X_i-\bar X)(Y_i-\bar Y)}{\sum (X_i-\bar X)^2}=\sum \frac{X_i-\bar X}{\sum (X_i-\bar X)^2}Y_i$$

$k_i\in \mathbb R$

Definition

$$k_i:=\frac{\tilde X_i}{\ell_{xx}}=\frac{X_i-\bar X}{\sum (X_i-\bar X)^2}$$

Properties

$$b_1=\sum k_iY_i$$ $$\sum k_i=\frac{\sum(X_i-\bar X)}{\sum (X_i-\bar X)^2}=0$$ $$\sum k_iX_i=\frac{\sum(X_i-\bar X)X_i}{\sum (X_i-\bar X)^2}=\frac{\sum(X_i-\bar X)^2}{\sum (X_i-\bar X)^2}+\frac{\bar X\sum(X_i-\bar X)}{\sum (X_i-\bar X)^2}=1$$ $$\sum k_i^2=\frac{\sum (X_i-\bar X)^2}{\left(\sum (X_i-\bar X)^2\right)^2}=\frac{1}{\sum (X_i-\bar X)^2}=\frac{1}{\ell_{xx}}$$

Distribution

$$b_1=\sum k_iY_i\sim \sum k_iN(\hat Y_i,\sigma^2)=N(\sum k_i\hat Y_i,\sum k_i^2\sigma^2)$$

$$\sum k_i\hat Y_i=\sum k_i(\beta_0+\beta_1X_i)=\beta_0\sum k_i+\beta_1\sum k_iX_i=\beta_1$$ $$\sum k_i^2\sigma^2=\sigma^2\sum k_i^2=\frac{\sigma^2}{\ell_{xx}}$$

$$b_1\sim N\left(\beta_1,\frac{\sigma^2}{\ell_{xx}}\right)$$

Point Estimation

$$\hat b_1=\beta_1$$ $$s^2_{b_1}=\frac{{\rm MSE}}{\ell_{xx}}=\frac{{\rm MSE}}{\sum (X_i-\bar X)^2}$$

$(Z-\mu)/s\sim t(df)$

Distribution

$$\frac{Z-\mu}{\sigma}\sim {\rm Std}N,\qquad\frac{s}{\sigma}=\sqrt{\frac{{\rm MSE}}{\sigma^2}}\sim\sqrt{\frac{\chi^2(df)}{df}}$$ $$\frac{Z-\mu}{s}=\frac{Z-\mu}{\sigma}\Big/\frac{s}{\sigma}\sim \frac{{\rm Std}N}{\sqrt{\chi^2(df)/df}}=t(df)$$

1-6. $b_0$

Definition

$$b_0:=\bar Y-b_1\bar X$$

Distribution

$\bar Y\sim N(\beta_0+\beta_1\bar X,\sigma^2/n)$

Definition

$$\bar Y:=\frac{1}{n}\sum Y_i\sim \frac{1}{n}\sum N(\hat Y_i,\sigma^2)=N\left(\frac{\sum \hat Y_i}{n},\frac{\sigma^2}{n}\right)=N\left(\beta_0+\beta_1\bar X,\frac{\sigma^2}{n}\right)$$

Properties

$$Cov(\bar Y,b_1)=Cov\left(\frac{1}{n}\sum Y_i,\sum k_iY_i\right)=\frac{\sum k_i}{n}\sigma^2=0$$

$$\begin{aligned}b_0&=\bar Y-b_1\bar X\\&\sim N\left(\beta_0+\beta_1\bar X,\frac{\sigma^2}{n}\right)-\bar XN\left(\beta_1,\frac{\sigma^2}{\ell_{xx}}\right)\\&=N\left(\beta_0,\sigma^2\left(\frac{1}{n}+\frac{\bar X^2}{\ell_{xx}}\right)\right)\end{aligned}$$

1-7. $Y_h$

Definition

$$Y_h:=b_0+b_1X_h$$

Distribution

$$\begin{aligned}Y_h&=\bar Y+b_1\tilde X_h\\&\sim N\left(\beta_0+\beta_1\bar X,\frac{\sigma^2}{n}\right)+\tilde X_hN\left(\beta_1,\frac{\sigma^2}{\ell_{xx}}\right)\\&=N\left(\beta_0+\beta_1X_h,\sigma^2\left(\frac{1}{n}+\frac{\tilde X_h^2}{\ell_{xx}}\right)\right)\end{aligned}$$

Confidence Interval

$$y_h\in[\hat Y_h\pm t_{1-\frac{\alpha}{2}}(n-2)s_h]$$

Working-Hotelling Confidence Bend

$$y_h\in[\hat Y_h\pm Ws_h]$$ $$W^2=2F_{1-\alpha}(2,n-2)$$

Remark

Average response of $\infty$ predictions

1-8. $Y_{\rm pred}$

Definition

$$Y_{\rm pred}=Y_h+\varepsilon_{\rm pred},\qquad \varepsilon_{\rm pred}\sim N(0,\sigma^2)$$

Distribution

$$Y_{\rm pred}\sim N\left(\beta_0+\beta_1X_h,\sigma^2\left(1+\frac{1}{n}+\frac{\tilde X_h^2}{\ell_{xx}}\right)\right)$$

Remark

Response of single prediction

1-9. $Y_{\rm predmean}$

Definition

$$Y_{\rm predmean}=Y_h+\frac{1}{m}\sum_{j=1}^m\varepsilon_{\rm pred_j},\qquad \varepsilon_{\rm pred_j}\sim N(0,\sigma^2)\ \text{i.i.d}$$

Distribution

$$Y_{\rm predmean}\sim N\left(\beta_0+\beta_1X_h,\sigma^2\left(\frac{1}{m}+\frac{1}{n}+\frac{\tilde X_h^2}{\ell_{xx}}\right)\right)$$

Remark

Average response of $m$ predictions

Relations Between $Y_h,\ Y_{\rm pred},\ Y_{\rm predmean}$

$$Y_h=Y_{\rm predmean}(m=\infty)$$ $$Y_{\rm pred}=Y_{\rm predmean}(m=1)$$

1-10. $e_i$

Definition

$$e_i=Y_i-b_0-b_1X_i$$

Distribution

$$e_i\sim N\left(0,\sigma^2\left(1-\frac{1}{n}-\frac{\tilde X_i^2}{\ell_{xx}}\right)\right)$$ $${\rm Cov}(e_i,e_j)=\sigma^2\left(-\frac{1}{n}-\frac{\tilde X_i\tilde X_j}{\ell_{xx}}\right)$$

1-11. Summary

Variable	Distribution	Mean	Variance	Covariance
$\varepsilon_i$	$N$	$0$	$\sigma^2$	$0$
${\rm SSR}/\sigma^2$	$\chi^2(n-2)$
${\rm MSR}/\sigma^2$	$\frac{\chi^2(n-2)}{n-2}$
$Y_i$	$N$	$\hat Y_i$	$\sigma^2$	$0$
$b_1$	$N$	$\beta_1$	$\frac{\sigma^2}{\ell_{xx}}$
$b_0$	$N$	$\beta_0$	$\sigma^2\left(\frac{1}{n}+\frac{\bar X^2}{\ell_{xx}}\right)$
$Y_{\rm predmean}$	$N$	$\beta_0+\beta_1X_h$	$\sigma^2\left(\frac{1}{m}+\frac{1}{n}+\frac{\tilde X_h^2}{\ell_{xx}}\right)$
$e_i$	$N$	$0$	$\sigma^2\left(1-\frac{1}{n}-\frac{\tilde X_i^2}{\ell_{xx}}\right)$	$\sigma^2\left(-\frac{1}{n}-\frac{\tilde X_i\tilde X_j}{\ell_{xx}}\right)$

1-12. ANOVA Table

ANOVA	${\rm SS}$	$df$	${\rm MS}$	$E({\rm MS})$
Regression	${\rm SSR}=\sum(\hat Y_i-\bar Y)^2$	$1$	${\rm MSR}={\rm SSR}$	$\sigma^2+\beta_1^2\ell_{xx}^2$
Error	${\rm SSE}=\sum(Y_i-\hat Y_i)^2$	$n-2$	${\rm MSE}=\frac{{\rm SSE}}{n-2}$	$\sigma^2$
Total	${\rm SSTO}=\sum(Y_i-\bar Y)^2$	$n-1$

• Coefficient of Determination $R^2:={\rm SSR}/{\rm SSTO}$
• Coefficient of Correlation $r:=\mathop{\mathrm {sgn}}b_1\cdot \sqrt{R^2}=\ell_{xy}/\sqrt{\ell_{xx}\ell_{yy}}$

2. Estimations & Tests

2-1. $\beta_1$

Interval Estimation

$$t_{\frac{\alpha}{2}}(n-2)\leq \frac{b_1-\beta_1}{s_{b_1}}\leq t_{1-\frac{\alpha}{2}}(n-2)$$ $$\beta_1\in\left[b_1\pm t_{1-\frac{\alpha}{2}}(n-2)s_{b_1}\right]$$

Tests

$$H_0:\beta_1=0\quad\text{vs}\quad H_1:\beta_1\neq 0$$

• $t$-Test

$$t:=\frac{b_1}{s_{b_1}},\qquad W=\Big\{|t|>t_{1-\frac{\alpha}{2}}(n-2)\Big\}$$

• $F$-Test

$$F:=\frac{{\rm MSR}}{{\rm MSE}}\sim F(1,n-2),\qquad W=\Big\{|F|>F_{1-\alpha}(1,n-2)\Big\}$$

• General Linear Test

$$R:\text{Reduced Model},\qquad F:\text{Full Model}$$ $$F:=\frac{{\rm SSE}_R-{\rm SSE}_F}{df_R-df_F}\Big/\frac{{\rm SSE}_F}{df_F},\qquad W=\Big\{|F|>F_{1-\alpha}(df_R-df_F,df_F)\Big\}$$

$$R:Y_i=\beta_0+\varepsilon_i,\qquad F:Y_i=\beta_1X_i+\beta_0+\varepsilon_i$$ $${\rm SSE}_R={\rm SST},\qquad {\rm SSE}_F={\rm SSE}$$ $$df_R=n-1,\qquad df_F=n-2$$ $$F={\rm MSR}/{\rm MSE}$$

2-2. $\beta_0$

Interval Estimation

$$t_{\frac{\alpha}{2}}(n-2)\leq \frac{b_0-\beta_0}{s_{b_0}}\leq t_{1-\frac{\alpha}{2}}(n-2)$$ $$\beta_0\in\left[b_0\pm t_{1-\frac{\alpha}{2}}(n-2)s_{b_0}\right]$$

3. Variance Normality & Constancy

3-1. Studentized and Semi-studentized Residuals

$$s_{e_i}=\sigma_{e_i}\Big|_{\sigma=\sqrt{\rm MSE}},\qquad s_{e_i}^{(\rm stu)}=\frac{e_i}{s_{e_i}},\qquad s_{e_i}^{(\rm semistu)}=\frac{e_i}{\sqrt{\rm MSE}}$$

3-2. Normal Q-Q Plot of Residuals

$$E(\varepsilon_i\mid \text{$e_i$ is the $k$-th smallest among $e$})\approx u_{\frac{k-3/8}{n+1/4}}\sqrt{{\rm MSE}}$$

• Plot $y=E(\varepsilon_i\mid k),\ x=e_i$
• Normal residuals if scatters near $y=x$

3-3. Brown-Forsythe Test for Variance Constancy

$$H_0:\varepsilon={\rm const}\quad\text{vs}\quad H_1:\varepsilon\neq {\rm const}$$ $$\mathcal X_{\rm OBS}=\mathcal X^{(1)}\cup \mathcal X^{(2)},\qquad \mathcal X^{(1)}=\{x\in\mathcal X_{\rm OBS}\mid x<\bar x\}$$ $$d^{(1)}_i=|e^{(1)}_i-e^{(1)}_{\rm mid}|,\qquad s^2=\frac{\sum \tilde d^{(1)}_i+\sum \tilde d^{(2)}_i}{n-2}$$ $$t_{\rm BF}=\frac{\bar d_1-\bar d_2}{s\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}},\qquad W=\Big\{|t_{\rm BF}|>t_{1-\frac{\alpha}{2}}(n-2)\Big\}$$

3-4. Breusch-Pagan Test for Variance Constancy of Large Sample

$$\ln \sigma_i^2=\gamma_0+\gamma_1X_i$$ $$H_0:\varepsilon={\rm const}\iff \gamma_1=0\quad\text{vs}\quad H_1:\varepsilon\neq {\rm const}$$ $$\chi^2_{\rm BP}=\frac{{\rm SSR}_\gamma}{2}\Big/\left(\frac{{\rm SSE}_\beta}{n}\right)^2\ \dot\sim\ \chi^2(1)\qquad W=\Big\{|\chi^2_{\rm BP}|>\chi^2_{1-\alpha}(1)\Big\}$$

4. Samples of Repeat $X$

4-1. Dividing of $\mathcal X_{\rm OBS}$

Divide $\mathcal X_{\rm OBS}$ into groups by same $X$

$$\mathcal X_{\rm OBS}=\bigcup \mathcal X^{(j)},\quad j=1,\cdots,c,\qquad {\rm SSE}={\rm SSPE}+{\rm SSLF}$$ $${\rm SSPE}=\sum_j\sum_i(Y_i^{(j)}-\bar Y^{(j)})^2,\qquad {\rm SSLF}=\sum_j\sum_i(\bar Y^{(j)}-\hat Y_i^{(j)})^2$$

ANOVA	${\rm SS}$	$df$	${\rm MS}$
Regression	${\rm SSR}=\sum\sum(\hat Y_i^{(j)}-\bar Y)^2$	$1$	${\rm MSR}={\rm SSR}$
Error	${\rm SSE}=\sum\sum(Y_i^{(j)}-\hat Y_i^{(j)})^2$	$n-2$	${\rm MSE}=\frac{{\rm SSE}}{n-2}$
Lack of Fit	${\rm SSLF}=\sum\sum(\bar Y^{(j)}-\hat Y_i^{(j)})^2$	$c-2$	${\rm MSLF}=\frac{{\rm SSLF}}{c-2}$
Pure Error	${\rm SSPE}=\sum\sum(Y_i^{(j)}-\bar Y^{(j)})^2$	$n-c$	${\rm MSPE}=\frac{{\rm SSPE}}{n-c}$
Total	${\rm SSTO}=\sum\sum(Y_i^{(j)}-\bar Y)^2$	$n-1$

$$E({\rm SSPE})=\sigma^2,\quad E({\rm SSLF})=\sigma^2+\frac{\sum n_j(\mu_j-\hat Y_j)^2}{c-2}$$

4-2. $F$-Test of Lack-of-fit with repeat $X$

$$H_0:EY=\beta_0+\beta_1X\quad\text{v.s}\quad H_1:EY\neq \beta_0+\beta_1X$$ $$F:X_i^{(j)}=\mu^{(j)}+\varepsilon_i^{(j)},\qquad R:Y_i=\beta_1X_i+\beta_0+\varepsilon_i$$ $${\rm SSE}_F={\rm SSPE},\qquad {\rm SSE}_R={\rm SSE}$$ $$df_F=n-c,\qquad df_R=n-2$$ $$F={\rm MSLF}/{\rm MSPE}$$

4-3. Box-Cox Transformations of Regression to $Y^\lambda$ or $\ln Y$

$$Y^\lambda=\left\{\begin{aligned} &Y^\lambda,\qquad&&\lambda\neq 0,\\ &\ln Y,\qquad&&\lambda=0 \end{aligned}\right.$$ $$Y^\lambda_i=\beta_0+\beta_1X_i+\varepsilon_i$$

5. Simultaneous PI & CI

$${\rm CI}_0=b_0\pm t_{1-\frac{\alpha}{2}}(n-2)s_{b_0},\qquad {\rm CI}_1=b_1\pm t_{1-\frac{\alpha}{2}}(n-2)s_{b_1}$$ $$\Pr(\beta_0\notin{\rm CI}_0)=\Pr(\beta_1\notin{\rm CI}_1)=\alpha$$ $$\Pr(\beta_0\in{\rm CI}_0\land \beta_1\in{\rm CI}_1)=1-2\alpha$$

5-1. Joint CI of $\beta_0,\beta_1$

$$B:=t_{1-\frac{\alpha}{4}}(n-2)$$ $${\rm BonfCI}_0=b_0\pm Bs_{b_0},\qquad{\rm BonfCI}_1=b_1\pm Bs_{b_1}$$ $$\Pr(\beta_0\in{\rm BonfCI}_0\land \beta_1\in{\rm BonfCI}_1)=1-\alpha$$

5-2. Simultaneous $Y_h$ CI of $\{X_{h_i}\}_{i=1}^g$

$$\hat Y_h\pm Us_{Y_h}$$

Bonferroni CI

$$U=B_\alpha(g)=t_{1-\frac{\alpha}{2g}}(n-2)$$

Working-Hotelling CI

$$U=W_\alpha=\sqrt{2F_{1-\alpha}(2,n-2)}$$

5-3. Simultaneous $Y_{\rm pred}$ PI of $\{X_{h_i}\}_{i=1}^g$

$$\hat Y_{\rm pred}\pm Us_{Y_{\rm pred}}$$

Bonferroni PI

$$U=B_\alpha(g)=t_{1-\frac{\alpha}{2g}}(n-2)$$

Scheffe PI

$$U=S_\alpha(g)=\sqrt{gF_{1-\alpha}(g,n-2)}$$

6. Regression Assuming $\beta_0=0$

$$Y_i=\beta_1X_i+\varepsilon_i$$

6-1. $b_1$

$$b_1=\frac{\sum X_iY_i}{\sum X_i^2}=\frac{\ell_{xy}}{\ell_{xx}}\Big|_{\bar X=\bar Y=0}\sim N\left(\beta_1,\frac{\sigma^2}{\ell_{xx}}\right)\Big|_{\bar X=0},\qquad df=n-1$$

6-2. $e_i$

$$e_i\sim N\left(0,\sigma^2\left(1-\frac{X_i^2}{\ell_{xx}}\right)\right)\Big|_{\bar X=0},\qquad {\rm Cov}(e_i,e_j)=\sigma^2\left(-\frac{X_iX_j}{\ell_{xx}}\right)\Big|_{\bar X=0}$$

6-3. $Y_{\rm pred}$

$$Y_{\rm pred}\sim N\left(\beta_1X_h,\sigma^2\left(1+\frac{X_h^2}{\ell_{xx}}\right)\right)\Big|_{\bar X=0}$$

6-4. ANOVA Table

ANOVA	${\rm SS}$	$df$	${\rm MS}$	$E({\rm MS})$
Regression	${\rm SSRU}=\sum\hat Y_i^2$	$1$	${\rm MSRU}={\rm SSRU}$	$\sigma^2+\beta_1^2\ell_{xx}^2\mid_{\bar X=0}$
Error	${\rm SSE}=\sum(Y_i-\hat Y_i)^2$	$n-1$	${\rm MSE}=\frac{{\rm SSE}}{n-1}$	$\sigma^2$
Total	${\rm SSTOU}=\sum Y_i^2$	$n$

6-5. Test of $\beta_1=0$

$$H_0:\beta_1=0\quad\text{vs}\quad H_1:\beta_1\neq 0$$ $$F:=\frac{{\rm MSRU}}{{\rm MSE}}\sim F(1,n-1),\qquad W=\Big\{|F|>F_{1-\alpha}(1,n-1)\Big\}$$