Robust generalized canonical correlation analysis

Abstract

Generalized canonical correlation analysis (GCCA) has been widely used for classification and regression problems. The key idea of GCCA is to map the data from different views into a common space with the minimum reconstruction error. However, GCCA employs the squared Frobenius norm as a distance metric to find a latent correlated space without a specific strategy to cope with outliers, thus misguiding the GCCA’s training task in real-world applications and leading to suboptimal performance. This inspires us to propose a novel robust formulation for GCCA, namely, GCCA with the p-order ($0<p\le 2$) of Frobenius norm minimization (called RGCCA). It is difficult to solve the RGCCA involving the nonsmooth and nonconvex p-order of F-norm terms. Therefore, an efficient iterative algorithm is developed to solve RGCCA, theoretically analyzing its convergence property. In addition, the parameters of RGCCA nicely trade-off between accuracy and training time, a property especially useful for larger samples. Empirical experiments and theoretical analysis prove the effectiveness and robustness of RGCCA on both noiseless and noisy datasets.

Appendix

1.1 Convergence Analysis

This section proves that Algorithm 1 monotonically decreases the objective function value of (8) and has a local optimal solution. Next, we introduce Lemma 1.

Lemma 1: [61]: For any nonzero matrices ${V}^{\left(t+1\right)},{V}^{t}$, when $0<p\le 2$, the inequality holds:

$${\Vert {V}^{\left(t+1\right)}\Vert }_{F}^{p}-\frac{p}{2}{\Vert {V}^{t}\Vert }_{F}^{p-2}{\Vert {V}^{\left(t+1\right)}\Vert }_{F}^{2}\le {\Vert {V}^{t}\Vert }_{F}^{p}-\frac{p}{2}{\Vert {V}^{t}\Vert }_{F}^{p-2}{\Vert {V}^{t}\Vert }_{F}^{2}.$$

(A.1)

Theorem 1: In each iteration of Algorithm 1, we have

$$\sum_{j=1}^{J}{\Vert {G}^{\left(t+1\right)}-{U}_{j}^{T}{X}_{j}\Vert }_{F}^{2}{d}_{j}^{\left(t+1\right)}\le \sum_{j=1}^{J}{\Vert {G}^{\left(t\right)}-{U}_{j}^{T}{X}_{j}\Vert }_{F}^{2}{d}_{j}^{t}.$$

(A.2)

It means that Algorithm 1 monotonically decreases the objective function value of (8).

Proof: According to step 3 in Algorithm 1, in the $\left(t+1\right)$ ^th iteration, we have the following inequality:

$$tr\left(\sum_{j=1}^{J}{G}^{\left(t\text{+}1\right)}M{\left({G}^{\left(t\text{+}1\right)}\right)}^{T}{d}_{j}^{t}\right)\ge tr\left(\sum_{j=1}^{J}{G}^{t}M{\left({G}^{t}\right)}^{T}{d}_{j}^{t}\right).$$

(A.3)

The inequality (A.3) multiplies -1 and adds $\sum_{j=1}^{J}tr\left({I}_{N}{d}_{j}^{t}\right)$ on both sides, we have

$$\sum_{j=1}^{J}tr\left({I}_{N}{d}_{j}^{t}\right)-\sum_{j=1}^{J}tr\left({G}^{\left(t\text{+}1\right)}M{\left({G}^{\left(t\text{+}1\right)}\right)}^{T}{d}_{j}^{t}\right)\le \sum_{j=1}^{J}tr\left({I}_{N}{d}_{j}^{t}\right)-\sum_{j=1}^{J}tr\left({G}^{t}M{\left({G}^{t}\right)}^{T}{d}_{j}^{t}\right).$$

(A.4)

By simple algebra, (A.4) becomes,

$$\sum_{j=1}^{J}tr\left({G}^{\left(t\text{+}1\right)}\left({I}_{N}-{P}_{j}\right){\left({G}^{\left(t\text{+}1\right)}\right)}^{T}{d}_{j}^{t}\right)\le \sum_{j=1}^{J}tr\left({G}^{t}\left({I}_{N}-{P}_{j}\right){\left({G}^{t}\right)}^{T}{d}_{j}^{t}\right).$$

(A.5)

Since ${\Vert G\left({I}_{N}-{P}_{j}\right)\Vert }_{F}^{2}=tr\left(G\left({I}_{N}-{P}_{j}\right){G}^{T}\right)$ holds for each j, then the (A.5) is transformed to

$$\sum_{j=1}^{J}{\Vert {G}^{\left(t\text{+}1\right)}\left({I}_{N}-{P}_{j}\right)\Vert }_{F}^{2}{d}_{j}^{t}\le \sum_{j=1}^{J}{\Vert {G}^{t}\left({I}_{N}-{P}_{j}\right)\Vert }_{F}^{2}{d}_{j}^{t}.$$

(A.6)

From the definitions of ${U}_{j}$ and ${P}_{j}$, (A.6) becomes,

$$\sum_{j=1}^{J}{\Vert {G}^{\left(t\text{+}1\right)}-{U}_{j}^{T}{X}_{j}\Vert }_{F}^{2}{d}_{j}^{t}\le \sum_{j=1}^{J}{\Vert {G}^{t}-{U}_{j}^{T}{X}_{j}\Vert }_{F}^{2}{d}_{j}^{t}.$$

(A.7)

From the definition of ${d}_{j}$, and denote by ${V}_{j}^{\left(t+1\right)}\text{=}{G}^{\left(t\text{+}1\right)}-{U}_{j}^{T}{X}_{j}$ and ${V}_{j}^{t}\text{=}{G}^{t}-{U}_{j}^{T}{X}_{j}$. Substituting ${V}_{j}^{\left(t+1\right)}$ and ${V}_{j}^{t}$ into the (A.7), by simple algebra, we rewrite (A.7) as follows,

$$\sum_{j=1}^{J}\frac{{\Vert {V}_{j}^{\left(t+1\right)}\Vert }_{F}^{2}}{{\Vert {V}_{j}^{t}\Vert }_{F}^{2}}{\Vert {V}_{j}^{t}\Vert }_{F}^{p}\le \sum_{j=1}^{J}{\Vert {V}_{j}^{t}\Vert }_{F}^{p}.$$

(A.8)

According to Lemma 1, we have

$$\frac{p}{2}\frac{{\Vert {V}_{j}^{\left(t+1\right)}\Vert }_{F}^{2}}{{\Vert {V}_{j}^{t}\Vert }_{F}^{2}}{\Vert {V}_{j}^{t}\Vert }_{F}^{p}\ge {\Vert {V}_{j}^{\left(t+1\right)}\Vert }_{F}^{p}-\left(1-\frac{p}{2}\right){\Vert {V}_{j}^{t}\Vert }_{F}^{p}.$$

(A.9)

Inequality (A.9) holds for each index j, thus we have

$$\frac{p}{2}\sum_{j=1}^{J}\frac{{\Vert {V}_{j}^{\left(t+1\right)}\Vert }_{F}^{2}}{{\Vert {V}_{j}^{t}\Vert }_{F}^{2}}{\Vert {V}_{j}^{t}\Vert }_{F}^{p}\ge \sum_{j=1}^{J}{\Vert {V}_{j}^{\left(t+1\right)}\Vert }_{F}^{p}-\sum_{j=1}^{J}\left(1-\frac{p}{2}\right){\Vert {V}_{j}^{t}\Vert }_{F}^{p}.$$

(A.10)

Combing inequalities (A.8) and (A.10), by simple algebra, we have

$$\sum_{j=1}^{J}{\Vert {V}_{j}^{\left(t+1\right)}\Vert }_{F}^{p}\le \sum_{j=1}^{J}{\Vert {V}_{j}^{t}\Vert }_{F}^{p}.$$

(A.11)

According to the definitions of ${V}_{j}^{\left(t+1\right)}$ and ${V}_{j}^{t}$, the (A.11) is rewritten as follows,

$$\sum_{j=1}^{J}{\Vert {G}^{\left(t\text{+}1\right)}-{U}_{j}^{T}{X}_{j}\Vert }_{F}^{p}\le \sum_{j=1}^{J}{\Vert {G}^{t}-{U}_{j}^{T}{X}_{j}\Vert }_{F}^{p}.$$

(A.12)

The (A.12) holds indicates that Algorithm 1 monotonically decreases the objective function value of (8) in each iteration. This suggests that the ${G}^{*}$ moves towards the optimal solution in each iteration. □

Theorem 2: Algorithm 1 converges to a local optimal solution of the objective function (8).

Proof: The (8) is a conditional extremum problem. We transform (8) to an unconditional extremum problem by utilizing the Lagrange multiplier method. The constructed Lagrange auxiliary function of (8) is shown as follows:

$${L}_{1}\left(G,\alpha \right)=\sum_{j=1}^{J}{\Vert G-{U}_{j}^{T}{X}_{j}\Vert }_{F}^{p}-tr\left(\alpha \left(G{G}^{T}-{I}_{r}\right)\right),$$

(A.13)

where $\alpha$ is the Lagrange multiplier. To satisfy the constraint condition $G{G}^{T}\text{=}{I}_{r}$, we set $\alpha$ as the diagonal matrix. According to KKT condition for the optimal solution, we take $\left(\partial {L}_{1}/\partial G\right)=0$, then,

$$\frac{\partial {L}_{1}}{\partial G}=p\sum_{j=1}^{J}\left({\Vert G-{U}_{j}^{T}{X}_{j}\Vert }_{F}^{p-2}\right)G\overline{M }-2G\alpha =0,$$

(A.14)

where $\overline{M }=\left({I}_{N}-2M\right)$. By simple algebra, we obtain the optimal solution

$$\sum_{j=1}^{J}\left({\Vert G-{U}_{j}^{T}{X}_{j}\Vert }_{F}^{p-2}\right)G\overline{M }=G\left(2\alpha /p\right).$$

(A.15)

According to the aforementioned analysis, the optimal solution of the objective function (12) can be obtained in step 3 of Algorithm 1. Therefore, the converged solution (${G}^{*}$) of Algorithm 1 satisfies the KKT condition of the (12). The Lagrange auxiliary function of the objective function (12) is shown as follows:

$${L}_{2}\left(G,\overline{\alpha }\right)=tr\left(GDM{G}^{T}\right)-tr\left(\overline{\alpha }\left(G{G}^{T}-{I}_{r}\right)\right),$$

(A.16)

where $\overline{\alpha }$ is the Lagrangian multiplier. Taking $\left(\partial {L}_{2}/\partial G\right)=0$, we obtain the KKT condition of (12) as follows,

$$\frac{\partial {L}_{2}}{\partial G}=GMD-G\overline{\alpha }=0.$$

(A.17)

The (A.17) is formally similar to the (A.15). The key difference between them is that the diagonal matrix $D$ is known in each iteration. In Algorithm 1, assume that obtaining the optimal solution ${G}^{*}$ in the $\left(t+1\right)$ ^th iteration. Then, we have ${G}^{\left(t\text{+}1\right)}\text{=}{G}^{*}\text{=}{G}^{t}$. In addition, according to definition of $D$, we can see that the (A.17) is the same as (A.15) in this case. This suggests that the converged solution of Algorithm 1 satisfies the KKT condition of the objective function (8). That is $\left(\partial {L}_{1}/\partial G\right)\left|{}_{G\text{=}{G}^{*}}\right.\text{=}{0}$. Given this, the converged solution of Algorithm 1 is a local optimal solution of the objective function (8). □

According to Theorem 1, an iterative algorithm is developed to search for the local optimal solution of the objective function (8). However, the developed algorithm needs to solve the eigen-decomposition of the matrix $DM$ iteratively. The iteration procedure makes the RGCCA higher computational cost than the GCCA family methods. This defect exists in iterative algorithms [47, 48, 58, 61, 62]. We will explore a better strategy with the theoretical guarantee to reduce the cost of RGCCA. Fortunately, we can adjust RGCCA’s parameters to balance the robustness and cost, especially for larger samples. In summary, Algorithm 1 scales well to large-scale datasets, which indicates that the developed iterative algorithm is useful for practical applications.