Diagnosis and quantification of the non-essential collinearity

Abstract

Marquandt and Snee (Am Stat 29(1):3–20, 1975), Marquandt (J Am Stat Assoc 75(369):87–91, 1980) and Snee and Marquardt (Am Stat 38(2):83–87, 1984) refer to non-essential multicollinearity as that caused by the relation with the independent term. Although it is clear that the solution is to center the independent variables in the regression model, it is unclear when this kind of collinearity exists. The goal of this study is to diagnose the non-essential collinearity parting from a simple linear model. The collinearity indices $k_{j}$, traditionally misinterpreted as variance inflation factors, are reinterpreted in this paper where they will be used to distinguish and quantify the essential and non-essential collinearity. The results can be immediately extended to the multiple linear model. The study also has some recommendations for statistical software such as SPSS, Stata, GRETL or R for improving the diagnosis of non-essential collinearity.

Appendix A: Stewart indices

Given matrix $\mathbf {A}$ with dimensions $n \times p$ partitioned as $\mathbf {A} = [ \mathbf {A}_{1}, \ldots , \mathbf {A}_{i}, \ldots , \mathbf {A}_{p} ] = [ \mathbf {A}_{i}, \mathbf {A}_{-i}]$ where $\vert \mathbf {A}\vert$ is the determinant of A and $\mathbf {A}_{-i}$ is equal to $\mathbf {A}$ after eliminating column i, Stewart (1987) defined the following index to measure the relation between $\mathbf {A}_{i}$ and the rest of the columns of $\mathbf {A}$:

$$\begin{aligned} k_{i}^{2} = \frac{\vert \mathbf {A}_{-i}^{t} \mathbf {A}_{-i} \vert \cdot \mathbf {A}_{i}^{t} \mathbf {A}_{i}}{\vert \mathbf {A}^{t} \mathbf {A} \vert }, \quad i=1,\ldots ,p. \end{aligned}$$

(20)

Since $\vert \mathbf {A}^{t} \mathbf {A} \vert = \vert \mathbf {A}_{-i}^{t} \mathbf {A}_{-i} \vert \cdot \vert \mathbf {A}_{i}^{t} \mathbf {A}_{i} - \mathbf {A}_{i}^{t} \mathbf {A}_{-i} \cdot \left( \mathbf {A}_{-i}^{t} \mathbf {A}_{-i} \right) ^{-1} \cdot \mathbf {A}_{-i}^{t} \mathbf {A}_{i} \vert$, is clear that:

$$\begin{aligned} k_{i}^{2} = \frac{\mathbf {A}_{i}^{t} \mathbf {A}_{i}}{\mathbf {A}_{i}^{t} \mathbf {A}_{i} - \mathbf {A}_{i}^{t} \mathbf {A}_{-i} \cdot \left( \mathbf {A}_{-i}^{t} \mathbf {A}_{-i} \right) ^{-1} \cdot \mathbf {A}_{-i}^{t} \mathbf {A}_{i}}, \quad i=1,\ldots ,p. \end{aligned}$$

(21)

Then, it is verified that:

$$\begin{aligned} k_{i}^{2}= & {} 1, \quad \text{ if } \mathbf {A}_{i}^{t} \mathbf {A}_{-i} = \mathbf {0},\\ k_{i}^{2}&\not =&1, \quad \text{ if } \mathbf {A}_{i}^{t} \mathbf {A}_{-i} \not = \mathbf {0}, \end{aligned}$$

where $\mathbf {0}$ is a vector composed of zeros with appropriate dimensions. In addition, when $i=1,\ldots ,p$, it is verified that:

• $k_{i}^{2} > 1$    if $\mathbf {A}_{-i}^{t} \mathbf {A}_{-i}$ is positive defined.

• $k_{i}^{2} < 1$    if $\mathbf {A}_{-i}^{t} \mathbf {A}_{-i}$ is negative defined.

Thus, this index can capture the orthogonality between $\mathbf {A}_{i}$ and the rest of the columns of matrix $\mathbf {A}$. However, note that orthogonality does not imply that there is no correlation:

$$\begin{aligned} \mathbf {A}_{i}^{t} \mathbf {A}_{j}= & {} \sum \limits _{k=1}^{n} A_{ik} A_{jk} = 0 \nRightarrow corr(\mathbf {A}_{i},\mathbf {A}_{j})\\= & {} - \frac{\overline{\mathbf {A}}_{i} \cdot \overline{\mathbf {A}}_{i}}{\sqrt{\sum \nolimits _{k=1}^{n} \left( A_{ik} - \overline{\mathbf {A}}_{i} \right) ^{2}} \cdot \sqrt{\sum \nolimits _{k=1}^{n} \left( A_{jk} - \overline{\mathbf {A}}_{j} \right) ^{2}}} = 0, \end{aligned}$$

for $i,j = 1,\ldots ,p, \ i \not = j$, unless the columns have zero mean.