A matrix-free approach to efficient affine-linear image registration on CPU and GPU

Abstract

This paper presents a generic approach to highly efficient image registration in two and three dimensions. Both monomodal and multimodal registration problems are considered. We focus on the important class of affine-linear transformations in a derivative-based optimization framework. Our main contribution is an explicit formulation of the objective function gradient and Hessian approximation that allows for very efficient, parallel derivative calculation with virtually no memory requirements. The flexible parallelism of our concept allows for direct implementation on various hardware platforms. Derivative calculations are fully matrix free and operate directly on the input data, thereby reducing the auxiliary space requirements from ${\mathcal {O}}(n)$ to ${\mathcal {O}}(1)$. The proposed approach is implemented on multicore CPU and GPU. Our GPU code outperforms a conventional matrix-based CPU implementation by more than two orders of magnitude, thus enabling usage in real-time scenarios. The computational properties of our approach are extensively evaluated, thereby demonstrating the performance gain for a variety of real-life medical applications.

Appendices

Appendix

Extension to the three-dimensional case

In this appendix, explicit matrix-free calculation rules will be derived for affine-linear registration of three-dimensional images with the SSD and NGF distance measures. Most definitions of the occuring functions are briefly repeated here to improve readability.

2.1 Sum of squared differences (SSD)

For any $y:\Omega _{{\mathcal {R}}}\rightarrow {\mathbb {R}}^{3}$, the sum of squared differences (SSD) distance measure [28] is given by

$$\begin{aligned} {\mathcal {D}}_{\text {SSD}}({\mathcal {R}},{\mathcal {T}};y) := \frac{1}{2} \int _{\Omega _{\mathcal {R}}} \left( {\mathcal {T}}(y({\mathbf {x}})) - {\mathcal {R}}({\mathbf {x}}) \right) ^2 {\mathrm {d}}{\mathbf {x}}. \end{aligned}$$

Let $y_w:{\mathbb {R}^{3}}\rightarrow {\mathbb {R}^{3}}, \quad x\,\mapsto \,Ax+b$ denote a three-dimensional affine-linear transformation with $w=(w_1,\ldots ,w_{12})$ and

$$ A: = \left( {\begin{array}{*{20}c} {w_{1} } & {w_{2} } & {w_{3} } \\ {w_{5} } & {w_{6} } & {w_{7} } \\ {w_{9} } & {w_{{10}} } & {w_{{11}} } \\ \end{array} } \right),\;b: = \left( {\begin{array}{*{20}c} {w_{4} } \\ {w_{8} } \\ {w_{{12}} } \\ \end{array} } \right). $$

Setting ${\mathcal {D}}_{\text {SSD}}(w):={\mathcal {D}}_{\text {SSD}}({\mathcal {R}},{\mathcal {T}};y_w)$ yields the formulation of affine-linear image registration with SSD as minimization problem

$$\begin{aligned} \min _w \ {\mathcal {D}}_{\text {SSD}}(w) \end{aligned}$$

(25)

with ${\mathcal {D}}_{\text {SSD}}:{\mathbb {R}^{12}}\rightarrow {\mathbb {R}}$. For discretization, the domain $\Omega _{{\mathcal {R}}}$ is assumed to be cuboid and decomposed into n cells of equal size with center points ${\mathbf {x}}_{i},\, i=1,\ldots ,n$, arranged in lexicographical ordering. Using the midpoint quadrature rule for numerical integration, a discretized version of (25) reads

$$\begin{aligned} \min _w \ D_{\mathrm {SSD}}(w) :=\frac{\bar{h}}{2} \displaystyle \sum _{i=1}^{n} \left( {\mathcal {T}}(y_w({\mathbf {x}}_i)) - {\mathcal {R}}({\mathbf {x}}_i) \right) ^2, \end{aligned}$$

where $\bar{h}$ denotes the volume of each cell. Multilinear interpolation with Dirichlet zero boundary conditions is used to evaluate the discrete template image at arbitrary coordinates.

Let $({\mathbf {x}}_{i})_j$ denote the j-th component of ${\mathbf {x}}_{i}\in {\mathbb {R}}^{3}$. For transformation parameters $w\in {\mathbb {R}^{12}}$, we define the vector

$$ v_{i} : = \left( {\begin{array}{*{20}c} {\left( {A{\mathbf{x}}_{1} + b} \right)_{i} } \\ {\left( {A{\mathbf{x}}_{2} + b} \right)_{i} } \\ \vdots \\ {\left( {A{\mathbf{x}}_{n} + b} \right)_{i} } \\ \end{array} } \right) \in {\mathbb{R}^{n}} , \quad i = 1,2,3, $$

to construct the function

$$ y:{\mathbb{R}^{{12}}} \to {\mathbb{R}^{{3n}}} ,\quad w \; \mapsto \left( {\begin{array}{*{20}c} {v_{1} } \\ {v_{2} } \\ {v_{3} } \\ \end{array} } \right). $$

(26)

Using ${{\mathbf {y}}_{i}} = (y_i,y_{i+n},y_{i+2n})^\top$, we define

$$ T:{\mathbb{R}^{{3n}}} \to {\mathbb{R}^{n}} ,\;\left( {\begin{array}{*{20}c} {y_{1} } \\ \vdots \\ {y_{{3n}} } \\ \end{array} } \right) \mapsto \left( {\begin{array}{*{20}c} {{\mathcal{T}}({\mathbf{y}}_{1} )} \\ \vdots \\ {{\mathcal{T}}({\mathbf{y}}_{n} )} \\ \end{array} } \right). $$

(27)

With $R_i := {{\mathcal {R}}}({{\mathbf {x}}}_{i})$, we set

$$ r:{\mathbb{R}^{n}} \to {\mathbb{R}^{n}} ,\;\left( {\begin{array}{*{20}c} {T_{1} } \\ \vdots \\ {T_{n} } \\ \end{array} } \right) \mapsto \left( {\begin{array}{*{20}c} {T_{1} - R_{1} } \\ \vdots \\ {T_{n} - R_{n} } \\ \end{array} } \right) $$

as residual function and finally

$$ \psi :{\mathbb{R}^{n}} \to {\mathbb{R}},\;\left( {\begin{array}{*{20}c} {r_{1} } \\ \vdots \\ {r_{n} } \\ \end{array} } \right) \mapsto \frac{{\bar{h}}}{2}\sum\limits_{{i = 1}}^{n} {r_{i}^{2} } $$

as the sum of all squared residual elements. Now, $D_{\mathrm {SSD}}$ can be written as a concatenation of four functions:

$$\begin{aligned} D_{\text {SSD}}: {{\mathbb {R}}^{12}\xrightarrow {y}}{{\mathbb {R}}^{3n}\xrightarrow {T}}{{\mathbb {R}}^{n}\xrightarrow {r}}{{\mathbb {R}}^{n}\xrightarrow {\psi }}{\mathbb {R}}. \end{aligned}$$

(28)

2.1.1 Matrix-based differentiation

The differentiation of (28) is performed with the chain rule as

$$\begin{aligned} \nabla D_{\text {SSD}}(w) = \frac{\partial \psi }{\partial r}\frac{\partial r}{\partial T}\frac{\partial T}{\partial y} \frac{\partial y}{\partial w} \end{aligned}$$

(29)

just as in the two-dimensional case. Again, we define the gradient as a row vector. The first two individual derivatives in (29) are given by

$$\begin{aligned} \frac{\partial \psi }{\partial r}[r]&= {\bar{h}}(r_1,\ldots ,r_{n}) \ \text {and} \nonumber \\ \frac{\partial r}{\partial T}[T]&= I_{n}, \end{aligned}$$

(30)

with $I_{n} \in {\mathbb {R}^{n\;\times\;n}}$ as the identity matrix. Denoting the partial derivative with respect to the i-th component by $\partial _i$ and defining $\partial _i {\mathcal {T[y]}}$ as

$$ \partial _{i} {\mathcal{T}}[y]: = \left( {\begin{array}{*{20}c} {\partial _{i} {\mathcal{T}}({\mathbf{y}}_{1} )} & {} & {} \\ {} & \ddots & {} \\ {} & {} & {\partial _{i} {\mathcal{T}}({\mathbf{y}}_{n} )} \\ \end{array} } \right),\quad i = 1,2,3, $$

it holds that

$$ \frac{{\partial T}}{{\partial y}}[y] = \left( {\begin{array}{*{20}c} {\partial _{1} {\mathcal{T}}} & {\partial _{2} {\mathcal{T}}} & {\partial _{3} {\mathcal{T}}} \\ \end{array} } \right) \in {\mathbb{R}}^{{n\; \times \;3n}} . $$

(31)

Finally, the derivative of the function y is given by

$$\begin{aligned} \frac{\partial y}{\partial w}[w] = I_3 \otimes {\mathbf {X}} \in {\mathbb {R}^{3n\;\times\;12}} \end{aligned}$$

(32)

with the Kronecker product $\otimes$ and the grid matrix X as

$$ {\mathbf{X}}: = \left( {\begin{array}{*{20}c} {({\mathbf{x}}_{1} )_{1} } & {({\mathbf{x}}_{1} )_{2} } & {({\mathbf{x}}_{1} )_{3} } & 1 \\ {({\mathbf{x}}_{2} )_{1} } & {({\mathbf{x}}_{2} )_{2} } & {({\mathbf{x}}_{2} )_{3} } & 1 \\ \vdots & \vdots & \vdots & \vdots \\ {({\mathbf{x}}_{n} )_{1} } & {({\mathbf{x}}_{n} )_{2} } & {({\mathbf{x}}_{n} )_{3} } & 1 \\ \end{array} } \right) \in {\mathbb{R}^{{n\; \times \;4}}} , $$

thus completing the analysis of the gradient components from (29). With

$$\begin{aligned} {\mathrm {d}} r := \frac{\partial r}{\partial T}\frac{\partial T}{\partial y}\frac{\partial y}{\partial w}\in {\mathbb {R}}^{n\;\times\;12}, \end{aligned}$$

(33)

the Gauss–Newton approximation $H_{\text {SSD}}$ of the Hessian matrix is given by

$$\begin{aligned} H_{\text {SSD}}(w) := {\mathrm {d}} r^\top {\mathrm {d}}_2\psi {\mathrm {d}} r \end{aligned}$$

with ${\mathrm {d}}_2\psi =\bar{h}$. Again, note $\frac{\partial r}{\partial T}=I_n$.

2.1.2 Matrix-free derivative calculation

With (31) and (32), it follows that

$$\left( \frac{\partial T}{\partial y}\frac{\partial y}{\partial w}\right) _{i,j} \,= \, \left\{ \begin{array}{ll} \partial _1 {\mathcal{T}}({\mathbf{y}}_i) {\mathbf{X}}_{i,j}&{}1\le j \le 4\\ \partial _2 {\mathcal{T}}({\mathbf{y}}_{i}){\mathbf{X}}_{i,j-4}&{}5\le j \le 8\\ \partial _3 {\mathcal{T}}({\mathbf{y}}_{i}){\mathbf{X}}_{i,j-8}&{}9\le j \le 12 \end{array}\right. . $$

(34)

Using (30), it holds that

$$\begin{aligned} \left( \frac{\partial \psi }{\partial r}\right) _i = {\mathcal {T}_{w}}({{\mathbf {x}}}_i) - {\mathcal R}({{\mathbf {x}}}_i) \end{aligned}$$

with ${\mathcal {T}_{w}}({{\mathbf {x}}}_i)\,\,{:=}\,\,{{\mathcal {T}}}(y_w({{\mathbf {x}}}_i))$. The explicit calculation rule for the objective function gradient in the three-dimensional case is therefore given by

$$ \nabla D_{{{\text{SSD}}}} (w) = \bar{h}\sum\limits_{{i = 1}}^{n} {\left( {{\mathcal{T}}_{w} ({\mathbf{x}}_{i} ) - {\mathcal{R}}({\mathbf{x}}_{i} )} \right)} \left( {\begin{array}{*{20}c} {\partial _{1} {\mathcal{T}}_{w} ({\mathbf{x}}_{i} )({\mathbf{x}}_{i} )_{1} } \\ {\partial _{1} {\mathcal{T}}_{w} ({\mathbf{x}}_{i} )({\mathbf{x}}_{i} )_{2} } \\ {\partial _{1} {\mathcal{T}}_{w} ({\mathbf{x}}_{i} )({\mathbf{x}}_{i} )_{3} } \\ {\partial _{1} {\mathcal{T}}_{w} ({\mathbf{x}}_{i} )} \\ {\partial _{2} {\mathcal{T}}_{w} ({\mathbf{x}}_{i} )({\mathbf{x}}_{i} )_{1} } \\ {\partial _{2} {\mathcal{T}}_{w} ({\mathbf{x}}_{i} )({\mathbf{x}}_{i} )_{2} } \\ {\partial _{2} {\mathcal{T}}_{w} ({\mathbf{x}}_{i} )({\mathbf{x}}_{i} )_{3} } \\ {\partial _{2} {\mathcal{T}}_{w} ({\mathbf{x}}_{i} )} \\ {\partial _{3} {\mathcal{T}}_{w} ({\mathbf{x}}_{i} )({\mathbf{x}}_{i} )_{1} } \\ {\partial _{3} {\mathcal{T}}_{w} ({\mathbf{x}}_{i} )({\mathbf{x}}_{i} )_{2} } \\ {\partial _{3} {\mathcal{T}}_{w} ({\mathbf{x}}_{i} )({\mathbf{x}}_{i} )_{3} } \\ {\partial _{3} {\mathcal{T}}_{w} ({\mathbf{x}}_{i} )} \\ \end{array} } \right)^{{ \top }}. $$

(35)

The Gauss–Newton approximation to the Hessian for the SSD distance measure is defined as

$$\begin{aligned} H_{\mathrm {SSD}}&= {\mathrm {d}}r^\top {\mathrm {d}}_2\psi {\mathrm {d}}r\\&= {\bar{h}}\left( \frac{\partial T}{\partial y}\frac{\partial y}{\partial w}\right) ^\top \left( \frac{\partial T}{\partial y}\frac{\partial y}{\partial w}\right) \in {\mathbb {R}}^{12\;\times\;12}. \end{aligned}$$

By utilizing (34) and setting

$$ l_{k} : = \left( {\begin{array}{*{20}c} {} \\ {\left( {\frac{{\partial T}}{{\partial y}}\frac{{\partial y}}{{\partial w}}} \right)_{{k,i}} \cdot\left( {\frac{{\partial T}}{{\partial y}}\frac{{\partial y}}{{\partial w}}} \right)_{{k,j}} } \\ {} \\ \end{array} } \right)_{{1 \le i,j \le 12}} , $$

(36)

it directly follows that

$$\begin{aligned} H_{\mathrm {SSD}}(w) = \bar{h} \displaystyle \sum _{k=1}^{n} l_k. \end{aligned}$$

2.2 Normalized gradient fields (NGF)

We consider the NGF distance measure [16]

$$ {\mathcal{D}}_{{{\text{NGF}}}} : = \frac{1}{2}\int\limits_{{\Omega _{{\mathcal{R}}} }} 1 - \left( {\frac{{\langle \nabla{\mathcal{R}}({\mathbf{x}}),\nabla {\mathcal{T}}(y({\mathbf{x}}))\rangle _{{{\varrho },\tau }} }}{{|| \nabla {\mathcal{R}}({\mathbf{x}})|| _{{\varrho }} \, || \nabla{\mathcal{T}}(y({\mathbf{x}}))|| _{\tau } }}} \right)^{2} \;{\text{d}}{\mathbf{x}}, $$

$\langle a,b \rangle _{\alpha ,\beta }:=\sum _{i=1}^{3}a_ib_i+\alpha \beta ,\ a,b\in {\mathbb {R}}^{3}$, $\Vert a\Vert _\varepsilon :=\sqrt{\sum _{i=1}^3 a_i^2+\varepsilon ^2}$, with separate edge parameters for reference and template image, cf. [35]. Setting ${\mathcal {D}}_{\text {NGF}}(w) := {\mathcal {D}}_{\text {NGF}}({\mathcal {R}},{\mathcal {T}};y_w)$, affine-linear image registration with NGF translates to

$$\begin{aligned} \min _w \ {\mathcal {D}}_{\text {NGF}}(w). \end{aligned}$$

(37)

For numerical optimization, the continuous formulation in (37) is discretized. For a reference image of size $n_1\;\times\;n_2\;\times\;n_3$ and an index $i,\ i=1,\ldots ,n$, let $i', j',k'\in {\mathbb {N}},1\le i'\le n_1,\ 1\le j'\le n_2,\ 1\le k'\le n_3$ such that $i = i' + j'n_1 + k'n_1n_2$. The indices of neighboring points with Neumann zero boundary conditions are given by

$$\begin{aligned} i_{-x}&= \max (i'-1,1)+ j'n_1 + k'n_1n_2, \nonumber \\ i_{+x}&= \min (i'+1,n_1)+j'n_1+ k'n_1n_2, \nonumber \\ i_{-y}&= i'+\max (j'-1,1)n_1+ k'n_1n_2, \nonumber \\ i_{+y}&= i'+\min (j'+1,n_2)n_1+ k'n_1n_2, \nonumber \\ i_{-z}&= i'+j'n_1+ \max (k'-1,1)n_1n_2, \nonumber \\ i_{+z}&= i'+j'n_1+ \min (k'+1,n_3)n_1n_2. \end{aligned}$$

(38)

We define functions

$$ g_{i} :{\mathbb{R}^{n}} \to {\mathbb{R}^{3}} , \quad T \mapsto \left( {\begin{array}{*{20}c} {\frac{1}{{2h_{1} }}( - T_{{i_{{ - x}} }} + T_{{i_{{ + x}} }} )} \\ {\frac{1}{{2h_{2} }}( - T_{{i_{{ - y}} }} + T_{{i_{{ + y}} }} )} \\ {\frac{1}{{2h_{3} }}( - T_{{i_{{ - z}} }} + T_{{i_{{ + z}} }} )} \\ \end{array} } \right) $$

and

$$\begin{aligned} s_i:&{\mathbb {R}^{3}} \rightarrow {\mathbb {R}}, \quad a \; \mapsto \; \langle g_i(R),a\rangle + \varrho \tau \end{aligned}$$

for gradient and scalar product type operations at the position i, respectively. Further setting

$$\begin{aligned} n_\varepsilon :&{\mathbb {R}^{3}} \rightarrow {\mathbb {R}}, \quad a \; \mapsto\; \sqrt{a_1^2 + a_2^2 + a_3^2 + \varepsilon ^{2}}, \end{aligned}$$

the discretized version of (37) is given by

$$\begin{aligned} \min _w \ D_{\mathrm {NGF}}(w) :=\frac{\bar{h}}{2} \displaystyle \sum _{i=1}^{n} 1 - \left( \frac{s_i(g_i(T_w))}{n_\varrho (g_i(R)) \ n_\tau (g_i(T_w))} \right) ^2 \end{aligned}$$

with $(T_w)_i = {\mathcal {T}}(y_w({{\mathbf {x}}}_i))$.

2.2.1 Matrix-based differentiation

Let y and T as in (26) and (27). We define the residual function $r:{\mathbb {R}^{n}}\rightarrow {\mathbb {R}^{n}}$ by setting the i-th component function $r_i:{\mathbb {R}^{n}}\rightarrow {\mathbb {R}}$ to

$$\begin{aligned} r_i:T \; \mapsto \; \frac{s_i(g_i(T))}{ n_\varrho ( g_i(R))\ n_\tau ( g_i(T))}. \end{aligned}$$

The reduction function $\psi :{\mathbb {R}^{n}}\rightarrow {\mathbb {R}}$ is given by

$$\begin{aligned} \psi (r)&= \frac{\bar{h}}{2} \sum _{i=1}^{n} 1 - r_i^2, \end{aligned}$$

yielding the function chain

$$\begin{aligned} D_{\text {NGF}}: {{\mathbb {R}}^{12}\xrightarrow {y}}{{\mathbb {R}}^{3n}\xrightarrow {T}}{{\mathbb {R}}^{n}\xrightarrow {r}}{{\mathbb {R}}^{n}\xrightarrow {\psi }}{\mathbb {R}}. \end{aligned}$$

The derivatives of T and y have already been computed in (31) and (32). For the reduction function $\psi$, it holds that

$$\begin{aligned} \frac{\partial \psi }{\partial r}&= -\bar{h} r^\top \in {\mathbb {R}^{1\;\times\;n}}. \end{aligned}$$

(39)

The calculation of $\frac{\partial r}{\partial T}$ is performed by differentiating the component functions $r_i,\; i=1,\ldots ,n$. The functions $r_i$ are composed of $s_i$, $g_i$ and $n_\varepsilon$ whose derivatives are given by

$$\begin{aligned} \frac{\partial s_i}{\partial a} = g_i(R)^\top \in {\mathbb {R}^{1\;\times\;3}}, \end{aligned}$$

and

$$\begin{aligned} \frac{\partial n_\varepsilon }{\partial a} = \frac{1}{n_\varepsilon (a)}a^\top \in {\mathbb {R}^{1\;\times\;3}} \end{aligned}$$

with $\frac{\partial g_i}{\partial T}\in {\mathbb {R}^{3\;\times\;n}}$. Applying the chain rule in both numerator and denominator of $r_i$ yields

$$ \frac{{\partial r_{i} }}{{\partial T}} = \left( {\begin{array}{*{20}c} \vdots \\ {\frac{1}{{2h_{3} }}\left[ {\frac{{ - g_{i} (R)_{3} }}{{n_{{\varrho }} (g_{i} (R))n_{\tau } (g_{i} (T))}} + \frac{{s_{i} (g_{i} (T))g_{i} (T)_{3} }}{{n_{{\varrho }} (g_{i} (R))n_{\tau } (g_{i} (T))^{3} }}} \right]} \\ \vdots \\ {\frac{1}{{2h_{2} }}\left[ {\frac{{ - g_{i} (R)_{2} }}{{n_{{\varrho }} (g_{i} (R))n_{\tau } (g_{i} (T))}} + \frac{{s_{i} (g_{i} (T))g_{i} (T)_{2} }}{{n_{{\varrho }} (g_{i} (R))n_{\tau } (g_{i} (T))^{3} }}} \right]} \\ \vdots \\ {\frac{1}{{2h_{1} }}\left[ {\frac{{ - g_{i} (R)_{1} }}{{n_{{\varrho }} (g_{i} (R))n_{\tau } (g_{i} (T))}} + \frac{{s_{i} (g_{i} (T))g_{i} (T)_{1} }}{{n_{{\varrho }} (g_{i} (R))n_{\tau } (g_{i} (T))^{3} }}} \right]} \\ 0 \\ {\frac{1}{{2h_{1} }}\left[ {\frac{{g_{i} (R)_{1} }}{{n_{{\varrho }} (g_{i} (R))n_{\tau } (g_{i} (T))}} - \frac{{s_{i} (g_{i} (T))g_{i} (T)_{1} }}{{n_{{\varrho }} (g_{i} (R))n_{\tau } (g_{i} (T))^{3} }}} \right]} \\ \vdots \\ {\frac{1}{{2h_{2} }}\left[ {\frac{{g_{i} (R)_{2} }}{{n_{{\varrho }} (g_{i} (R))n_{\tau } (g_{i} (T))}} - \frac{{s_{i} (g_{i} (T))g_{i} (T)_{2} }}{{n_{{\varrho }} (g_{i} (R))n_{\tau } (g_{i} (T))^{3} }}} \right]} \\ \vdots \\ {\frac{1}{{2h_{3} }}\left[ {\frac{{g_{i} (R)_{3} }}{{n_{{\varrho }} (g_{i} (R))n_{\tau } (g_{i} (T))}} - \frac{{s_{i} (g_{i} (T))g_{i} (T)_{3} }}{{n_{{\varrho }} (g_{i} (R))n_{\tau } (g_{i} (T))^{3} }}} \right]} \\ \vdots \\ \end{array} } \right)^{\top} $$

with the vector entries at positions $i_{-z},i_{-y},i_{-x},i_{+x},i_{+y},$ and $i_{+z}$ (in that order) as defined in (38). Note that these positions may coincide, in which case the values are added.

The Gauss–Newton approximation $H_{\text{NGF}}$ to the Hessian is given by

$$\begin{aligned} H_{\text {NGF}}(w) := {\mathrm {d}} r^\top {\mathrm {d}}_2\psi {\mathrm {d}}r \approx \nabla ^2 D_{\text {NGF}}(w) \end{aligned}$$

with ${\mathrm {d}}r$ defined as in (33) and ${\mathrm {d}}_2\psi =-{\bar{h}}$.

2.2.2 Matrix-free derivative calculation

Setting $r_i:=\frac{s_i(g_i(T))}{ n_\varrho ( g_i(R))\ n_\tau ( g_i(T))}$ and ${\mathrm {d}}r_i:=\frac{\partial r_i}{\partial T}\frac{\partial T}{\partial y}\frac{\partial y}{\partial w}$, it holds with (39) that

$$\begin{aligned} \nabla D_{\text {NGF}}(w) = -\bar{h} \sum _{i=1}^{n}r_i{\mathrm {d}}r_i. \end{aligned}$$

(40)

As $r_i\in {\mathbb {R}}$ are scalars, it suffices to derive a matrix-free description of the vectors ${\mathrm {d}}r_i\in {\mathbb {R}}^{12}$ to achieve a fully matrix-free formulation of the objective function gradient. Let $1\le i \le n$ and define indices $i_{-z},i_{-y},i_{-x},i_{+x},i_{+y},i_{+z}$ as in (38). With the definition

$$ \begin{aligned} d_{i}^{{j,k}}& \,\,{:=}\,\,\,\, \partial r_{i} [i_{{ - z}} ]\partial _{j} {\mathcal{T}}({\mathbf{y}}_{{i_{{ - z}} }} ){\mathbf{X}}_{{i_{{ - z}} ,k}} \\ &\,\, + \,\partial r_{i} [i_{{ - y}} ]\partial _{j} {\mathcal{T}}({\mathbf{y}}_{{i_{{ - y}} }} ){\mathbf{X}}_{{i_{{ - y}} ,k}} \\ & \,\,+ \,\partial r_{i} [i_{{ - x}} ]\partial _{j} {\mathcal{T}}({\mathbf{y}}_{{i_{{ - x}} }} ){\mathbf{X}}_{{i_{{ - x}} ,k}} \\ & \,\, + \, \partial r_{i} [i_{{ + x}} ]\partial _{j} {\mathcal{T}}({\mathbf{y}}_{{i_{{ + x}} }} ){\mathbf{X}}_{{i_{{ + x}} ,k}} \; \\ & \,\, + \, \partial r_{i} [i_{{ + y}} ]\partial _{j} {\mathcal{T}}({\mathbf{y}}_{{i_{{ + y}} }} ){\mathbf{X}}_{{i_{{ + y}} ,k}} \; \\ & \,\, + \, \partial r_{i} [i_{{ + z}} ]\partial _{j} {\mathcal{T}}({\mathbf{y}}_{{i_{{ + z}} }} ){\mathbf{X}}_{{i_{{ + z}} ,k}} \\ \end{aligned} $$

for $i=1,\ldots ,n$, $j=1,2,3$, $k=1,\dots ,4$ and

$$ d_{i}^{j} : = \left( {\begin{array}{*{20}c} {d_{i}^{{j,1}} ,d_{i}^{{j,2}} ,d_{i}^{{j,3}} ,d_{i}^{{j,4}} } \\ \end{array} } \right), $$

it follows that

$$ {\text{d}}r_{i} = \left( {\begin{array}{*{20}c} {d_{i}^{1} } & {d_{i}^{2} } & {d_{i}^{3} } \\ \end{array} } \right)^{{ \top }} \in {\mathbb{R}^{{12}}} , $$

(41)

which according to (40) yields

$$ \nabla D_{{{\text{NGF}}}} (w) = - \bar{h}\sum\limits_{{i = 1}}^{n} {\frac{{s_{i} (g_{i} (T))}}{{n_{{\varrho }} (g_{i} (R))\;n_{\tau } (g_{i} (T))}}} \left( {\begin{array}{*{20}c} {{\text{d}}r_{i} [1]} \\ {{\text{d}}r_{i} [2]} \\ \vdots \\ {{\text{d}}r_{i} [12]} \\ \end{array} } \right)^{{ \top }} , $$

(42)

completing the gradient calculation for the three-dimensional case. Since

$$ H_{{{\text{NGF}}}} (w) = \left( {\frac{{\partial r}}{{\partial T}}\frac{{\partial T}}{{\partial y}}\frac{{\partial y}}{{\partial w}}} \right)^{{ \top }} {\text{d}}_{2} \psi \left( {\frac{{\partial r}}{{\partial T}}\frac{{\partial T}}{{\partial y}}\frac{{\partial y}}{{\partial w}}} \right) = \left( {\begin{array}{*{20}c} {{\text{d}}r_{1}^{{ \top }} } & \ldots & {{\text{d}}r_{n}^{{ \top }} } \\ \end{array} } \right){\text{d}}_{2} \psi \left( {\begin{array}{*{20}c} {{\text{d}}r_{1} } \\ \vdots \\ {{\text{d}}r_{n} } \\ \end{array} } \right), $$

the calculation of the Hessian approximation can directly be performed with the help of the matrix-free formulation of ${\mathrm {d}}r_i$ from (41). By defining the matrices $l_k\in {\mathbb {R}^{12}\;\times\;12}$ as

$$ l_{k} : = \left( {\begin{array}{*{20}c} {} \\ {{\text{d}}r_{k} [i]\cdot{\text{d}}r_{k} [j]} \\ {} \\ \end{array} } \right)_{{1 \le i,j \le 12}} $$

(43)

analog to the case of SSD, the matrix-free formulation for the Gauss–Newton approximation to the Hessian is given by

$$\begin{aligned} H_{\mathrm {NGF}}(w) = \bar{h} \displaystyle \sum _{k=1}^{n} l_k. \end{aligned}$$

This finalizes the derivation of matrix-free calculation rules for objective function gradient and Gauss–Newton approximation to the Hessian also for the Normalized Gradient Fields distance measure with three-dimensional images.