Random Design Analysis of Ridge Regression

Abstract

This work gives a simultaneous analysis of both the ordinary least squares estimator and the ridge regression estimator in the random design setting under mild assumptions on the covariate/response distributions. In particular, the analysis provides sharp results on the “out-of-sample” prediction error, as opposed to the “in-sample” (fixed design) error. The analysis also reveals the effect of errors in the estimated covariance structure, as well as the effect of modeling errors, neither of which effects are present in the fixed design setting. The proofs of the main results are based on a simple decomposition lemma combined with concentration inequalities for random vectors and matrices.

Appendix: Probability Tail Inequalities

The following probability tail inequalities are used in our analysis. These specific inequalities were chosen in order to satisfy the general conditions set up in Sect. 2.4; however, our analysis can specialize or generalize with the availability of other tail inequalities of these sorts.

The first tail inequality is for positive semidefinite quadratic forms of a sub-Gaussian random vector. It generalizes a standard tail inequality for Gaussian random vectors based on linear combinations of $\chi ^2$ random variables [15].

Lemma 8

(Quadratic forms of a sub-Gaussian random vector [11]) Let $\xi$ be a random vector taking values in $\mathbb {R}^n$ such that for some $c \ge 0$,

$$\begin{aligned} \mathbb {E}[\exp (\langle u,\xi \rangle )] \le \exp (c \Vert u\Vert ^2 / 2), \quad \forall u \in \mathbb {R}^n. \end{aligned}$$

For all symmetric positive semidefinite matrices $K \succeq 0$, and all $t > 0$,

$$\begin{aligned} \Pr \biggl [ \xi ^{\scriptscriptstyle \top }K \xi > c \Bigl ( {{\mathrm{tr}}}(K) + 2\sqrt{{{\mathrm{tr}}}(K^2)t} + 2\Vert K\Vert t \Bigr ) \biggr ] \le \mathrm {e}^{-t}. \end{aligned}$$

The next lemma is a tail inequality for sums of bounded random vectors; it is a standard application of Bernstein’s inequality.

Lemma 9

(Vector Bernstein bound; see, e.g., [11]) Let $x_1,x_2,\cdots ,x_n$ be independent random vectors such that

$$\begin{aligned} \sum _{i=1}^n \mathbb {E}[ \Vert x_i\Vert ^2 ] \le v \quad \text {and} \quad \Vert x_i\Vert \le r \end{aligned}$$

for all $i=1,2,\cdots ,n$, almost surely. Let $s := x_1 + x_2 + \cdots + x_n$. For all $t > 0$,

$$\begin{aligned} \Pr \left[ \Vert s\Vert > \sqrt{v} (1 + \sqrt{8t}) + (4/3) r t \right] \le \mathrm {e}^{-t}. \end{aligned}$$

The last tail inequality concerns the spectral accuracy of an empirical second moment matrix.

Lemma 10

(Matrix Bernstein bound [12]) Let $X$ be a random matrix, and $r > 0$, $v > 0$, and $k > 0$ be such that, almost surely,

$$\begin{aligned} \mathbb {E}[X] = 0 , \quad \lambda _{\max }[X] \le r , \quad \lambda _{\max }[\mathbb {E}[X^2]] \le v , \quad {{\mathrm{tr}}}(\mathbb {E}[X^2]) \le v k. \end{aligned}$$

If $X_1,X_2,\cdots ,X_n$ are independent copies of $X$, then for any $t > 0$,

$$\begin{aligned} \Pr \left[ \lambda _{\max }\left[ \frac{1}{n} \sum _{i=1}^n X_i \right] > \sqrt{\frac{2vt}{n}} + \frac{rt}{3n} \right] \le k t (\mathrm {e}^t - t - 1)^{-1}. \end{aligned}$$

If $t \ge 2.6$, then $t (\mathrm {e}^t - t - 1)^{-1} \le \mathrm {e}^{-t/2}$.