Toward Fast Transform Learning

Abstract

This paper introduces a new dictionary learning strategy based on atoms obtained by translating the composition of $K$ convolutions with $S$-sparse kernels of known support. The dictionary update step associated with this strategy is a non-convex optimization problem. We propose a practical formulation of this problem and introduce a Gauss–Seidel type algorithm referred to as alternative least square algorithm for its resolution. The search space of the proposed algorithm is of dimension $KS$, which is typically smaller than the size of the target atom and much smaller than the size of the image. Moreover, the complexity of this algorithm is linear with respect to the image size, allowing larger atoms to be learned (as opposed to small patches). The conducted experiments show that we are able to accurately approximate atoms such as wavelets, curvelets, sinc functions or cosines for large values of K. The proposed experiments also indicate that the algorithm generally converges to a global minimum for large values of $K$ and $S$.

Appendix

1.1 Proof of Proposition 1

First notice that ${\mathcal D}$ is a compact set. Moreover, when (7) holds, the objective function of $(P_1)$ is coercive in $\lambda$. Thus, for any threshold $\mu$, it is possible to build a compact set such that the objective function evaluated at any $(\lambda ,\mathbf h)$ outside this compact set is larger than $\mu$. As a consequence, we can extract a converging subsequence from any minimizing sequence. Since the objective function of $(P_1)$ is continuous in a closed domain, any limit point of this subsequence is a minimizer of $(P_1)$.

1.2 Proof of Proposition 2

The proof of 1 hinges on formulating the expression of a stationary point of $(P_1)$, then showing that the Lagrange multipliers associated with the norm-to-one constraint for the $(h^k)_{1 \le k \le K}$ are all equal to $0$. First, considering the partial differential of the objective function of $(P_1)$ with respect to $\lambda$ and a Lagrange multiplier $\gamma _\lambda \ge 0$ for the constraint $\lambda \ge 0$, we obtain

$$\begin{aligned} \lambda \Vert \alpha *h^1 * \dots * h^K \Vert _{2}^2 - \left\langle \alpha * h^1 * \dots * h^K,u \right\rangle =\frac{\gamma _\lambda }{2}, \end{aligned}$$

(19)

and

$$\begin{aligned} \lambda \gamma _\lambda = 0. \end{aligned}$$

(20)

Then, considering Lagrange multipliers $\gamma _k\in \mathbb R$ associated with each constraint $\Vert h^k \Vert _{2}=1$, we have for all $k \in \{1,\dots ,K\}$

$$\begin{aligned} \lambda \tilde{H}^k * (\lambda \alpha *h^1 * \dots * h^K-u) = \gamma _k h^k, \end{aligned}$$

(21)

where $H^k$ is defined by (5). Taking the scalar product of (21) with $h^k$ and using both $\Vert h^k\Vert _2=1$ and (19), we obtain

$$\begin{aligned} \gamma _k = \lambda \frac{\gamma _\lambda }{2} =0, \quad \forall k \in \{1,\dots ,k\}. \end{aligned}$$

Hence, (21) takes the form, for all $k \in \{1,\dots ,K\}$

$$\begin{aligned} \lambda \tilde{H}^k * (\lambda \alpha *h^1 * \dots * h^K-u) = 0. \end{aligned}$$

(22)

When $\lambda >0$, this immediately implies that the kernels $\mathbf g$ defined by (8) satisfy

$$\begin{aligned}\frac{\partial E}{\partial h^k} \left( \mathbf h \right) = 0 ,\quad \forall k\in \{1,\ldots K\}, \end{aligned}$$

i.e., the kernels $\mathbf g \in (\mathbb {R}^{\mathcal {P}})^K$ form a stationary point of $(P_0)$.

The proof of the item 2 is straightforward since for any $(f^k)_{1 \le k \le K} \in (\mathbb {R}^{\mathcal {P}})^K$ satisfying the constraints of $(P_0)$ ⁸, we have

$$\begin{aligned}&\Vert \alpha *g^1*\ldots *g^K - u \Vert _{2}^2 \\&= \Vert \lambda \alpha * h^1*\ldots *h^K - u \Vert _{2}^2 \\&\le \left\| \left( \prod _{k=1}^K \Vert f^k \Vert _{2}\right) ~~\alpha * \frac{f^1}{\Vert f^1 \Vert _{2}}*\ldots *\frac{f^K}{\Vert f^K \Vert _{2}}- u\right\| _2^2 \\&\le \Vert \alpha *f^1*\ldots *f^K - u \Vert _{2}^2. \end{aligned}$$

As a consequence, the kernels $(g^k)_{1\le k\le K}$ defined by (8) form a solution of $(P_0)$.

1.3 Proof of Proposition 3

The first item of proposition 3 can be obtained directly since 1) the sequence of kernels generated by the algorithm belongs to ${\mathcal D}$ and ${\mathcal D}$ is compact, 2) the objective function of $(P_1)$ is coercive with respect to $\lambda$ when (13) holds, and 3) the objective function is continuous and decreases during the iterative process.

To prove the second item of proposition 3, we consider a limit point $(\lambda ^*,\mathbf h^*) \in \mathbb R\times {\mathcal D}$. We denote by $F$ the objective function of $(P_1)$ and denote by $(\lambda ^o,\mathbf h^o)_{o\in \mathbb N}$ a subsequence of $(\lambda ^n,\mathbf h^n)_{n\in \mathbb N}$ which converges to $(\lambda ^*,\mathbf h^*)$. The following statements are trivially true, since $F$ is continuous and $\left( F(\lambda ^n,\mathbf h^n)\right) _{n\in \mathbb N}$ decreases:

$$\begin{aligned} \lim _{o\rightarrow \infty } F\left( T(\mathbf h^o) \right) = \lim _{o\rightarrow \infty } F(\lambda ^o,\mathbf h^o) = F(\lambda ^*,\mathbf h^*) \end{aligned}$$

(23)

However, if for any $k$ inside $\{1,\ldots , K\}$, we have $C_k^Tu\ne 0$ and the matrix $C_k$ generated using $T_k(\mathbf h^*)$ is full column rank, then there exist an open neighborhood of $T_k(\mathbf h^*)$ such that these conditions remain true for the matrices $C_k$ generated from kernels $\mathbf h$ in this neighborhood. As a consequence, the $k$th iteration of the for loop is a continuous mapping on this neighborhood. Finally, we deduce that there is a neighborhood of $\mathbf h^*$ in which $T$ is continuous.

Since $T$ is continuous in the vicinity of $\mathbf h^*$ and $(\mathbf h^o)_{o\in \mathbb N}$ converges to $(\mathbf h^*)$, the sequence $(T(\mathbf h^o))_{o\in \mathbb N}$ converges to $T(\mathbf h^*)$ and (23) guarantees that

$$\begin{aligned} F\left( T(\mathbf h^*) \right) =F(\lambda ^*,\mathbf h^*) . \end{aligned}$$

As a consequence, denoting $\mathbf h^*=(h^{*,k})_{1\le k\le K}$, for every $k\in \{1,\ldots ,K\}$, $F(\lambda ^*, h^{*,k})$ is equal to the minimal value of $(P_k)$. Since $C_k$ is full column rank, we know that this minimizer is unique (see the end of Sect. 3.2) and therefore $(\lambda ^*, h^{*,k})$ is the unique minimizer of $(P_k)$. We can then deduce that $(\lambda ^*,\mathbf h^*)=T(\mathbf h^*)$.

Finally, we also know that $(\lambda ^*,\mathbf h^*)$ is a stationary point of $(P_k)$. Combining all the equations stating that, for any $k$, $(\lambda ^*, \mathbf h^{*,k})$ is a stationary point of $(P_k)$, we can find that $(\lambda ^*,\mathbf h^*)$ is a stationary point of $(P_1)$.