Graph Diffusion & PCA Framework for Semi-supervised Learning

Abstract

A novel framework called Graph diffusion & PCA (GDPCA) is proposed in the context of semi-supervised learning on graph structured data. It combines a modified Principal Component Analysis with the classical supervised loss and Laplacian regularization, thus handling the case where the adjacency matrix is Sparse and avoiding the Curse of dimensionality. Our framework can be applied to non-graph datasets as well, such as images by constructing similarity graph. GDPCA improves node classification by enriching the local graph structure by node covariance. We demonstrate the performance of GDPCA in experiments on citation networks and images, and we show that GDPCA compares favourably with the best state-of-the-art algorithms and has significantly lower computational complexity.

Supported by MyDataModels company.

Appendices

A Proof of Proposition 1

Proof

This proof uses the same strategy as the proof of Proposition 2 in [1]. Rewriting Problem (4) in matrix form with the standard Laplacian $L = D - A$ and with $Z_{.i} , Y_{.i} \in \mathbb {R}^{n \times 1}$:

$$\begin{aligned} \begin{aligned} Q(Z)&= 2 \sum _{i=1}^{k} Z^T_{.i} D^{\sigma - 1} L D^{\sigma - 1} Z_{.i} \\ {}&+ \mu \sum _{i=1}^{k} (Z_{.i} - Y_{.i})^T D^{2\sigma - 1}(Z_{.i} - Y_{.i}) - \delta \sum _{i=1}^{k} Z_{.i}S Z_{.i}^T \end{aligned} \end{aligned}$$

where $S = \bar{X}^T\bar{X}/(d-1) \in \mathbb {R}^{n \times n}$. Considering $\frac{Q(Z)}{\partial Z} = 0$:

$$\begin{aligned} \begin{aligned}&2 Z^T (D^{\sigma - 1} L D^{\sigma - 1} + D^{\sigma - 1} L^T D^{\sigma - 1} ) + 2\mu (Z - Y )^TD^{2\sigma - 1} - \delta Z ^T(S + S^T) = 0 \end{aligned} \end{aligned}$$

Multiplying by $D^{-2\sigma + 1}$ and replacing $L = D - A$ results in:

$$\begin{aligned} \begin{aligned} Z^T (&2I - 2D^{\sigma - 1} AD^{-\sigma } + \mu I - 2 \delta SD^{-2\sigma + 1}) - \mu Y^T = 0 \end{aligned} \end{aligned}$$

Taking out the $\mu$ over the parentheses and transposing the equation:

$$\begin{aligned} \begin{aligned} Z&= \frac{\mu }{(2 + \mu ) } (I - \frac{2}{(2 + \mu )} ( D^{\sigma -1} AD^{-\sigma } + \delta SD^{-2\sigma + 1} ))^{-1}Y \end{aligned} \end{aligned}$$

Finally, the desired result is obtained with $\alpha = 2/(2+\mu )$.   $\square$

B Proof of Proposition 2

Proof

Apply Theorem 1 of sums of spectral radii [35] for the following inequality:

$$\begin{aligned} \rho ( D^{\sigma - 1}A D^{-\sigma }+ \delta SD^{-2\sigma + 1} ) \le \rho \left( D^{\sigma - 1}A D^{-\sigma }\right) + \rho (\delta SD^{-2\sigma + 1})< 1/\alpha \end{aligned}$$

based on the fact that spectral radius of a matrix similar to the stochastic matrix is equal to 1 (Gershgorin bounds):

$$\begin{aligned} 1 + \delta \rho (SD^{-2\sigma + 1})< 1/\alpha \end{aligned}$$

apply the Theorem 7 [26] for replacing $\rho (SD^{-2\sigma + 1})$ by the $\gamma$ maximum singular value of $SD^{-2\sigma + 1}$ we obtain the desired result in (6).    $\square$

C Generation of Synthetic Adjacency Matrix

For selecting the best synthetic adjacency matrix for GDPCA, we have considered three standard distances, such as Cosine, Minkowski, Dice and the number of neighbours from 1 till 14 for KNN algorithm. The accuracy of GDPCA on above parameters on the validation set for MNIST and CCT datasets are shown in Fig. 4. Figure 4 shows that the best GDPCA accuracy on the validation set is obtained with the use of 7 neighbours and Dice distance for the CCT dataset is obtained with the use of 7 neighbours and Cosine distance for the MNIST dataset.

Fig. 4.

Estimate different adjacency matrix for GDPCA.