Page 1 of 9 Bioinformatics

1
2
Bioinformatics, YYYY, 0–0
3
doi: 10.1093/bioinformatics/xxxxx
4
Advance Access Publication Date: DD Month YYYY
5
Manuscript Category
6
7

Downloaded from https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btac816/6960919 by guest on 27 December 2022

8
9 Subject Section
10
11 Tracing weak neuron fibers
12
13
Yufeng Liu, Ye Zhong, Xuan Zhao, Lijuan Liu, Liya Ding, and Hanchuan
14 Peng*
15 SEU-ALLEN Joint Center, Institute for Brain and Intelligence, Southeast University, Nanjing, Jiangsu,
16 210096, China
17
18 *To whom correspondence should be addressed.
19 Associate Editor: XXXXXXX
20
Received on XXXXX; revised on XXXXX; accepted on XXXXX
21
22
23 Abstract
Motivation: Precise reconstruction of neuronal arbors is important for circuitry mapping. Many auto-tracing
24
algorithms have been developed toward full reconstruction. However, it is still challenging to trace the weak
25 signals of neurite fibers that often correspond to axons.
26 Results: We proposed a method, named the NeuMiner, for tracing weak fibers by combining two strategies: an
27 online sample mining strategy and a modified gamma transformation. NeuMiner improved the recall of weak
28 signals (voxel values less than 20) by a large margin, from 5.1% to 27.8%. This is prominent for axons, which
29 increased by 6.4 times, compared to 2.0 times for dendrites. Both strategies were shown to be beneficial for weak
30 fiber recognition, and they reduced the average axonal spatial distances to gold standards by 46% and 13%
31 respectively. The improvement was observed on two prevalent automatic tracing algorithms and can be applied
32 to any other tracers and image types.
33 Availability: Source codes of NeuMiner are freely available on GitHub
(https://github.com/crazylyf/neuronet/tree/semantic_fnm). Image visualization, preprocessing, and tracing are
34
conducted on the Vaa3D platform, which is accessible at the Vaa3D GitHub repository
35
(https://github.com/Vaa3D). All training and testing images are cropped from high-resolution fMOST mouse
36 brains downloaded from the Brain Image Library (https://www.brainimagelibrary.org/), and the corresponding
37 gold standards are available at (https://doi.brainimagelibrary.org/doi/10.35077/g.25).
38 Contact: h@braintell.org
39 Supplementary information: Supplementary data are available at Bioinformatics online.
40
41
42
43 Xiao and Peng, 2013; Quan et al., 2016; Zhou et al., 2021; Chen et al.,
44 2015; Peng et al., 2011, 2017; Yang et al., 2019; Zhou et al., 2016; Liu et
1 Introduction
45 al., 2016, 2017), as well as many pre-tracing and post-tracing algorithms,
Understanding the morphology and connectivity of neurons is crucial for
46 including image enhancement (Liang et al., 2017), segmentation (R. Li et
cell typing, wiring characterization, and simulation. Amount of neuronal al., 2017; Liu et al., 2018; Zhou et al., 2018; Wang et al., 2019; Huang et
47
reconstructions have been collected and archived on open-sharing al., 2020; Li and Shen, 2020; Wang et al., 2021), post-processing (S. Li et
48 websites such as NeuroMorpho.org (Ascoli et al., 2007), FlyCircuits al., 2017; S. Li, Quan, Xu, et al., 2019), critical points detection (Liu et
49 (Chiang et al., 2011), and FlyLight (Jenett et al., 2012), and Initiatives al., 2019; Tan et al., 2020; Chen et al., 2021; Shen et al., 2021), and
50 including DIADEM challenge (Brown et al., 2011), and BigNeuron ensembling (Wang et al., 2017). Nevertheless, most of these algorithms
51 project (Peng et al., 2015). However, the production of high-quality single are designed for well-selected images, and the tracing quality for whole
52 neuron reconstructions in high throughput is still challenging. brain light microscopic images could not meet the requirement for
53 A promising way to fill the gap between primary data (i.e., microscopic morphological and physiological analyses (Manubens-Gil et al., 2022).
54 images) and secondary data (i.e., morphological reconstructions) is Many obstacles make neuron reconstruction challenging (Basu et al.,
55 automatic neuron tracing. To address this problem, researchers developed 2013). Although current microscopes, e.g. fluorescence micro-optical
56 numerous automatic or semi-automatic algorithms (Garvey et al., 1973; sectioning tomography (fMOST) (Gong et al., 2016), generate whole-
Coleman et al., 1977; Glaser and Glaser, 1990; Al-Kofahi et al., 2002;
57 brain images in submicron resolution with high quality, considerable
58
© The Author(s) 2022. Published by Oxford University Press.
59 This is an Open Access article distributed under the terms of the Creative Commons Attribution License
60 (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium,
provided the original work is properly cited.
 Bioinformatics Page 2 of 9

Liu et al.
1
2
fibers are often weakly imaged (S. Li, Quan, Zhou, et al., 2019). These backgrounds.
3 weak fibersare difficult to identify when surrounded by noisy
4
5
6
7

Downloaded from https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btac816/6960919 by guest on 27 December 2022

8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33 Figure 1. Schematic illustration of NeuMiner. (a) An example of weak fibers (fibers within the ellipses) consisted of low-intensity voxels. (b) Gold standard
34 swc overlaid on the image, with red, magenta, and blue are basal dendrite, apical dendrite, and axon. (c) Relation between voxel intensity and recall of
35 APP2 reconstructions. (d-f) Intensity distributions between traced and untraced neurites, for all neurites, axons, and dendrites. The weak fibers (voxel
36 intensity < 20) are not well recognized, and untraced fibers have significantly weaker fibers. (g) Workflow of NeuMiner tracing. The original image is
firstly enhanced with a modified gamma transformation (Derivative Truncated Gamma Transformation, DTGT), and subsequently segmented with a U-
37
Net. The segmentation network is trained with False Negative Mining (FNM) to improve the weak fiber recognition. Training with partial labels can
38
identify most fibers. The segmentation is then fused with the original image and traced using existing tracing algorithms, like APP2, to get the final
39 reconstruction.
40
41 In this paper, we proposed a neuron tracing module called NeuMiner brains and over 1700 neurons manually annotated from 34 brains by
42 with a focus on tracing weak fibers (Fig. 1). To this end, we introduced a experts. We downloaded 1726 neurons from 31 brains (3 brains are failed
to download. ID: 191807, 182724, 210254). Resolutions of images vary
43 False Negative Mining (FNM) strategy into the segmentation network by
from (0.2 × 0.2 × 1.0 µm) to (0.35 × 0.35 × 1.0 µm). Every neuron is
44 adjusting the losses of false-negative voxels in training. We also applied a cropped to constant size (1024 × 1024 × 512) centered at the soma position
45 modified gamma transformation, Derivative Truncated Gamma and then down-sampled to (512 × 512 × 256). The crop size is chosen so
Transformation (DTGT), to the input images by truncating the derivative that it contains the majority of dendrites and considerable axons while
46
of the gamma function to no less than 1. We also demonstrated neuron keeping the computation workload affordable. 1676 neurons from 29
47 brains are randomly split into train, validation, and test set, containing
segmentation with partially annotated images can identify 93% more
48 1357, 168, and 151 neurons, respectively. The other 50 neurons from 2
signals on average. Overall, NeuMiner empowers base tracers with much new independent brains are used for validation of unseen data. Images
49
higher recall for weak fibers, based on extensive evaluations of the newly with more than one somata are removed in validation and test sets to get
50 released single neuron dataset. rid of densely packed neurons, which is out of the scope of this work.
51 Therefore, 61 and 59 samples are used for validation and testing, and 29
52 neurons are used for benchmarking of new brains. The validation set is
2 Methods used for parameter searching, and the final results are evaluated on the test
53
set. Image resolutions and sizes are in (𝑥, 𝑦, 𝑧) order unless explicitly
54 2.1 Dataset
55 stated.
The recently released brain-wide single neuron dataset is adopted in this
56 work (Peng et al., 2021). The dataset contains 53 high-quality imaged
57
58
59
60
Page 3 of 9 Bioinformatics

1
2
3 a b c d
4
5
6
7

Downloaded from https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btac816/6960919 by guest on 27 December 2022

8
9
10
11
12
13
14
15
16
17
18
19 Figure 2. Comparison of gamma transformation and derivative truncated gamma transformation. (a) Original image. (b) Gamma transformation of original
20 image. (c) Derivative truncated gamma transformation of the original image. (d) Illustration of gamma and derivative truncated gamma function, with 𝛾
21 =0.4 for both functions. The high-intensity fibers highlighted by red rectangles exemplify the halo effect after gamma transformation. These regions are
22 zoomed in for better visualization. While both transformations enhance the weak fibers significantly, the proposed DTGT suppresses the diffusive halo
23 artifact around the bright fibers.
24 strategy is introduced to alleviate the problem. The detailed
25 implementation of it is elaborated as follows.
26 2.2 Segmentation network
For per-voxel loss, it can be formulated as
27 A vanilla 3D UNet (Ronneberger, 2017) is utilized. The model is trained 1
𝐵 𝐶 𝑍 𝑌 𝑋

28 with randomly cropped (160 × 160 × 128) volumes. Diverse 𝐿= ∑∑ ∑ ∑ ∑

𝑁𝑖 = 1𝑗 = 1𝑘 = 1𝑙 = 1𝑚 = 1
𝑤𝑖,𝑗,𝑘,𝑙,𝑚 ∙ 𝐿𝑖,𝑗,𝑘,𝑙,𝑚
29 augmentations, including cropping, gamma transformation, Gaussian
noises, flipping, and resizing, are randomly sampled for each input image. where 𝐵, 𝐶, 𝑍, 𝑌, 𝑋 are the batch size, channel number, image shape in
30 𝑧, 𝑦, 𝑥-axes, respectively. 𝑁 is the number of voxels calculated as
Instance normalization (Ulyanov et al., 2017) is applied before every
31 𝐵 ∙ 𝐶 ∙ 𝑍 ∙ 𝑌 ∙ 𝑋. 𝑤𝑖,𝑗,𝑘,𝑙,𝑚 and 𝐿𝑖,𝑗,𝑘,𝑙,𝑚 are weight and loss for the voxel (𝑖, 𝑗,
nonlinear activation layer. Networks are trained with Nesterov-based
32 SGD, with an initial learning rate of 0.01, weight decay of 3e-5, and 𝑘, 𝑙, 𝑚). The loss is universal to all per-voxel losses.
33 momentum of 0.99. Dice loss (Milletari et al., 2016) and cross-entropy Firstly, the on-the-fly false-negative sample set 𝐹𝑁 is extracted via
34 loss are applied to the last layers of the last two blocks. All models are 𝐹𝑁 = 𝐹 ∩ (𝑃 < 0.5)
35 trained for 15,200 iterations using Pytorch (version 1.8) with Automatic where 𝐹 is the foreground voxel set generated according to the labels,
𝑃 is the segmentation.
36 Mixed Precision acceleration enabled.
Once 𝐹𝑁 is estimated, the loss weights for the false-negative voxels can
37
be adjusted as
38
{
2.3 Label generation 𝑤𝐹𝑁, 𝑝𝑖,𝑗,𝑘,𝑙,𝑚 ∈ 𝐹𝑁
39 𝑤𝑖,𝑗,𝑘,𝑙,𝑚 =
Segmentation labels are generated according to manual annotated gold 1.0, 𝑝𝑖,𝑗,𝑘,𝑙,𝑚 ∉ 𝐹𝑁
40
standards (i.e., swc files), by resampling annotated nodes to connective In this work, false-negative weight 𝑤𝐹𝑁 is set as 1.5 by line-search.
41
points in 3D space. Each point is represented by a uniform width of (3 × Notably, only false-negative weights are increased, as there are
42
3 × 1) voxels. The somata are overlaid with a fixed size (18 × 18 × 6) considerable numbers of unlabeled fibers.
43 cuboid for both input images and label images. Ellipsoids with different
44 radii are also evaluated on the validation set, they share similar
45 performances. Only part of the neurons is annotated, and there may be 2.5 Derivative Truncated Gamma Transformation
46 considerable numbers of unlabeled fibers in each image volume. Gamma transformation is a nonlinear image transformation that follows
47 gamma function (i.e., power function) expressed as
48 𝑔(𝑥) = 𝑥𝛾, 𝑥 ∈ [0,1]
2.4 False Negative Mining 𝑥
49 where image voxel are normalized in range [0,1] in prior. To amplify
50 Segmentation is a per-pixel/voxel classification problem (Zhou et al., the weak fibers, 𝛾 should be smaller than 1.0.
2018), and is usually trained with pixel/voxel-wise cross-entropy loss or One drawback of the gamma function is that its derivatives of higher
51
dice loss. Although the foreground in both input image and label image intensity region (voxel value 𝑥 near 1.0) are smaller than 1 (Fig. 2d), which
52
shows co-occurrence, their relationship is not guaranteed while inference. can be inferred from the derivative function
53 Moreover, the fibers are very diverse in intensity, many of them are 𝑔′(𝑥) = 𝛾 ∙ 𝑥𝛾 ― 1, 𝑥 ∈ [0,1]
54 discontinuous and weak (fibers in red ellipses in Fig. 1a). Therefore, the contrast of brighter fibers will be attenuated after
55 Those weak fibers are challenging for a segmentation network, applying the gamma function. This may cause a diffusive effect which we
56 resulting in false-negative classification. A False Negative Mining (FNM) call the halo effect, as illustrated in Fig. 2b.
57
58
59
60
 Bioinformatics Page 4 of 9

Liu et al.
1
2
We propose to replace the higher intensity region of the gamma (yellow arrow in Fig. 4). This is caused by the lack of soma segmentation
3 function with the identity function so that the derivatives are no less than labels, and the artificial uniform labeling is ambiguous for determining the
4 1. The new function is formulated as highly diverse soma bodies. The disconnection can be rescued by fusing
5
6
𝑔(𝑥) = {𝑥𝛾 , 𝑥 ∈ [0,𝛿]
𝑥 ― 𝛿 + 𝛿𝛾, 𝑥 ∈ [𝛿,1]
with the original image.
Except for training with partial labels adopted in this work, weakly-
7 where 𝛿 is truncation point where the derivative equals 1, calculated as supervised learning methods that leverage automatic tracing

Downloaded from https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btac816/6960919 by guest on 27 December 2022

𝑙𝑛𝛾
8 reconstructions as pseudo-labels are proved to be effective in many large-
𝛿 = 𝑒1 ― 𝛾
9 Voxel values are standardized after the transformation to keep the range
scale datasets (Huang et al., 2020). For comparison, we applied automatic
10 tracer APP2 to all neurons in the train set and used the reconstructions as
unchanged.
11 training labels. It is comparable for the majority of fibers, but most weak
We call the new function 𝑔(𝑥) derivative truncated gamma function,
fibers are missed (Fig. 3c). Statistical analysis on the tracing results
12 and transformation based on it as Derivative Truncated Gamma
confirmed the poor recognition on axons (average 𝑆𝐷12 12.53, compared
13 Transformation (DTGT). The truncated function is illustrated in Fig. 2d.
to our method 1.87).
14 As expected, the halo effect is restrained as shown in Fig. 2c.
15
16
a Original Image b NeuMiner
2.6 Tracing
17
Segmentation confidences are fused with the original image through a
18
simple linear combination
19 𝑥𝑖 = 𝛼 ∙ 𝑥𝑖 + (1 ― 𝛼) ∙ 𝑝𝑖
20 where 𝑥𝑖 and 𝑝𝑖 are corresponding values on input image and
21 segmentation, and the value of 𝛼 is 0.8, which is the same as (R. Li et al.,
22 2017).
23 The fused image is subsequently traced with existing tracers. In this
24 paper, APP2 (Xiao and Peng, 2013) and SmartTracing (Chen et al., 2015)
25 are adopted to evaluate our methods, similar to (R. Li et al., 2017). APP2
26 is a faster and more accurate variant of APP1 (Peng et al., 2011), and it is c Pseudo-Labeling d w/o FNM

27 still one of the most robust and popular methods ever since its release. By
28 integrating SVM-based segmentation into the APP2 baseline,
SmartTracing is a prevalent alternative. Another critical reason for
29
choosing these methods is that these two methods are relatively tuning-
30
free, and tracing with default parameters will get sufficient good results.
31
32
33 3 Results
34 3.1 NeuMiner improves the segmentation of weak fibers
35 The recall of weak fibers (5.1%) is much lower than other fibers (~36%,
36 Fig. 1c) and they are more likely to be untraced, especially for axons (Fig. Figure 3. Segmentation results under different conditions. (a) An
37 1d-f). Global contrast normalization, e.g., histogram equalization, exemplar image volume from brain 18455. (b) Corresponding
segmentation generated by NeuMiner. (c) Segmentation generated by
38 improves the contrast at the expense of over-exposure (Fig. S1). Adaptive
pseudo-labelling experiment, whose label for segmentation network is
39 enhancement, e.g., Contrast Limited Adaptive Histogram Equalization generated from APP2. (d) Segmentation without False Negative Mining
40 (CLAHE), overcomes this problem and successfully enhances the weak (FNM) applied. While most fibers are well segmented for all methods,
41 fibers (Fig. S1). Nevertheless, the patch-based contrast normalization weak fibers pointed out by the red arrows are better recognized by
CLAHE is highly correlated to nearby signals. If the fibers are close to NeuMiner.
42
high-intensity voxels, they will be suppressed after transformation (yellow
43
arrows in Fig. S1). Our proposed DTGT enhances the weak fibers
44 a
comparably to CLAHE, and the enhancement is independent of nearby a
45 signals (Fig. S1).
46 The FNM strategy is critical for the higher recall of weak fibers.
47 Theoretically, FNM increases the recall of all fibers indiscriminately.
48 Since the strong fibers (fibers of high intensity) are easily classified, the
49 weak fibers benefit preferentially. As expected, the strong fibers are
50 comparably recognized regardless of whether FNM is used, while the very
b
51 weak fibers are better segmented when FNM is enabled (Fig. 3b).
52 NeuMiner successfully recognized most fibers, including unlabeled
53 ones (Fig. 4) and very weak fibers (Fig. 4, pointed out by red arrows). The
average ratio of foreground voxel number in segmentation to that of label
54
images is 1.93±0.69, i.e., 93% more voxels are identified. Meanwhile,
55
both discrete Gaussian Noises (Fig. 4a) and oval-shaped plaques (Fig. 4b)
56 are suppressed. Soma body is occasionally disconnected from other fibers Original Image Label Image Output Segmentation
57
58
59
60
Page 5 of 9 Bioinformatics

Article short title

1
2
3 Figure 4. Examples of segmentation with partial labels. (a-b) are We also tested a recently proposed neuronal image enhancing method,
examples of two different neurons. The first columns are original image
4 volumes, the second and last columns are label images and segmentations.
imPreProcess (Guo et al., 2022), which was demonstrated to enhance
5 signal-background contrast and improve reconstruction quality by a large
Red arrows highlight the very weak fibers that identified by our method,
6 and the dark yellow arrow points out the disconnected prediction near margin. Results on our dataset show it optimized signal recognition
soma. Despite large numbers of fibers are not annotated, NeuMiner considerably (average 𝑆𝐷12is 6.11), but is inferior to our method in weak
7
correctly recognized most unlabeled fibers, as well as many very weak fiber recognition.

Downloaded from https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btac816/6960919 by guest on 27 December 2022

8 fibers.
9
10
11
12 Table 1 Quantitative comparison between base tracers and corresponding NeuMiner enabled versions. The best values in each category are highlighted
13
in bold. Down arrows indicate that the smaller the metrics, the better the results. Up arrows are the opposite.
14
15 𝑺𝑫𝟏𝟐/𝑺𝑫𝟐𝟏/𝑺𝑫↓ 𝑺𝑺𝑫𝟏𝟐/𝑺𝑺𝑫𝟐𝟏/S𝑺𝑫↓ 𝑷𝑫𝑺𝟏𝟐/𝑷𝑫𝑺𝟐𝟏/PDS↓ Topology-based metrics↑
16 All neurites Dendrite1 Axon 𝑶𝑷𝑻 ― 𝑱 𝑶𝑷𝑻 ― 𝑷 𝑶𝑷𝑻 ― 𝑮
17
APP2 11.82/5.47/8.64 4.42/-/- 17.86/-/- 23.04/15.67/19.35 0.36/0.15/0.26 0.80 ± 0.12 0.69 ± 0.17 0.65 ± 0.23
18
+NeuMiner 1.47/7.31/4.39 0.93/-/- 1.87/-/- 6.36/22.40/14.38 0.10/0.20/0.15 0.88 ± 0.07 0.82 ± 0.11 0.80 ± 0.15
19
20 ST2 23.93/16.60/20.26 18.39/-/- 29.12/-/- 31.54/27.30/29.42 0.46/0.35/0.41 0.75 ± 0.16 0.62 ± 0.19 0.55 ± 0.25
21 +NeuMiner 1.06/14.08/7.57 0.78/-/- 1.24/-/- 5.49/32.08/18.78 0.09/0.31/0.20 0.82 ± 0.10 0.76 ± 0.13 0.73 ± 0.18
22 1 For distance metrics of separate axon or dendrite, 𝐵21 and 𝐵 is meaningless as a considerable proportion of neurites are removed, so they are represented as -/-.
23 2 43 reconstructions are used for evaluation of SmartTracing (ST) based methods, as the others are not successfully reconstructed by SmartTracing.
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
Figure 5. Tracing improvement of NeuMiner. (a1-c1) Neuron image volumes from different brains (brain id: 17302, 18869 and 18463). (a2-c2) Neuron
54
55 image volumes with gold standards overlaid. Blue, magenta, and red are basal dendrites, apical dendrites, and axons. (a3-c3,a4-c4,a5-c5,a6-c6)
56 Corresponding reconstructions by APP2, APP2 with NeuMiner enabled, SmartTracing (ST) and SmartTracing with NeuMiner enabled. Considerable
57
58
59
60
 Bioinformatics Page 6 of 9

Liu et al.
1
2
weak fibers are missed by both APP2 and SmartTracing, while most of them are identified when NeuMiner is enabled. (d) Relation between voxel
3
4 intensity and recall of NeuMiner enhanced APP2 reconstructions. (d-f) Intensity distributions between traced and untraced neurites, for all neurites, axons
5 and dendrites. The recall of weak fibers (27.8%) is significantly improved, compared to vanilla APP2 reconstruction (5.1%) as shown in Fig. 1. The
6 intensity distributions for untraced and traced fibers are similar when NeuMiner is applied.
7

Downloaded from https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btac816/6960919 by guest on 27 December 2022

8
9
10
dramatically (from 5.47 to 19.00), resulting in an increased average
11 distance discrepancy (𝑆𝐷12, from 8.64 to 12.24). Therefore, a smaller
12 3.2 NeuMiner improves the tracing of weak fibers
background threshold does not solve the tracing of weak fibers.
13 While both APP2 and SmartTracing perform well on most neurons, they For a comprehensive assessment, three topology-based metrics,
14 are not good at tracing weak fibers by design (Fig. 1c). Weak fibers are Junction-based metric 𝑂𝑃𝑇 ― 𝐽 Path-based metric 𝑂𝑃𝑇 ― 𝑃, and
15 frequently missed by both APP2 and SmartTracing. It is not only observed Subgraph-based metric 𝑂𝑃𝑇 ― 𝐺 (Citraro et al., 2020) are also calculated.
16 on axon tracing (Fig. 5(a3,a5)), but also on dendrites (Fig. 5(b3,b5,c3,c5)). By fixing the weaknesses of existing metrics, the above three metrics are
The problem is alleviated when combined with NeuMiner, and most fibers
17 demonstrated to be superior on road segmentation. When combined with
are successfully identified (Fig. 5(a4,a6,b4,b6,c4,c6)).
18 NeuMiner, both APP2 and SmartTracing show significant improvement
Quantitative analysis is conducted for a more comprehensive for all those three metrics, especially for 𝑂𝑃𝑇 ― 𝑃 and 𝑂𝑃𝑇 ― 𝐺 (Table
19 evaluation. We first evaluated with three mostly cited metrics: Spatial 1).
20 Distance (𝑆𝐷), Substantial Spatial Distance (𝑆𝑆𝐷), and Percentage of We also applied our method to another 26 neurons from two new brains,
21 Different Structure (𝑃𝐷𝑆) (Peng et al., 2011) on the test set. 𝑆𝐷 is the and the improvements of NeuMiner are again confirmed based on the
22 average of minimal distances of points in the comparing neuron for all performance on most metrics (Table. S2).
23 voxels in the estimating neuron, 𝑆𝑆𝐷 is a modified 𝑆𝐷 metric that takes
24 into consideration only distances larger than 2 voxels, and 𝑃𝐷𝑆 means
25 percentage of different structures, which is defined as minimal reciprocal 3.3 Ablation study
26 spatial distance larger than 2 voxels. These metrics are asymmetric, thus Ablation studies were carried out to reveal the contribution of each part.
27 they are represented by label-to-prediction (𝐵12), prediction-to-label (𝐵21 Since all three metrics share similar patterns (Table. 1), this time we only
), and the average of these two (𝐵), where 𝐵 represents any kind of above
28 use 𝑆𝐷 for simplicity. Quantitative analyses for these studies are
metrics (𝑆𝐷, 𝑆𝑆𝐷, 𝑃𝐷𝑆). 𝐵12 is highly correlated to recall, and 𝐵21 to summarized in Table. S1. The False Negative Mining (FNM) strategy is
29
precision. Lower is better. the principal contributor among all tricks. When it is removed, some weak
30 All metrics show consistent conclusions. The recall of NeuMiner- fibers are not well identified (Fig. 3d), and the overall 𝑆𝐷12 distance
31 enabled versions is significantly higher than vanilla APP2 and increases from 1.47 to 2.67. The deterioration is especially profound for
32 SmartTracing, with 𝑆𝐷12 distances decreased by 87.6% and 95.6% axon, (increases from 1.87 to 3.45), highlighting the necessity of the FNM
33 respectively (Table. 1). Similar advantages are also observed on 𝑆𝑆𝐷12and strategy for achieving a high fiber recognition.
34 𝑃𝐷𝑆12. On the other hand, the precision (𝐵21) is comparable. Specifically, The Derivative Truncated Gamma Transformation (DTGT) also
35 all 𝐵21 distances are marginally increased compared to APP2, while improves weak fiber recognition. The overall 𝑆𝐷12 is also lower than
36 NeuMiner has a minor advantage to SmartTracing baseline as two out of baseline. Note that the increase of 𝑆𝐷12 is mainly on the axons, which
37 three metrics are smaller. Overall, the averaged distances of NeuMiner are increased by 0.29 voxels from 1.87 to 2.16, while the dendrite distance
38 much better, with 49.2%, 25.7%, and 42.3% distances reduction for APP2, keeps unchanged. This is reasonable as DTGT has a larger slope at a small
and 62.6%, 36.2%, and 51.2% for SmartTracing.
39 value region, and only enhances the contrast for weak fibers.
The much lower 𝐵12 distances indicate higher weak fiber tracing of Without truncation, i.e., vanilla gamma transformation, the recall of
40
NeuMiner. This is verified by the smaller average voxel intensity of extra- both dendrite and axon keeps unchanged, but the precision is slightly
41 traced fibers (0.79±0.68, ratio to the average intensity of all APP2 traced worse. Finally, if the image is not fused with segmentation, the 𝑆𝐷12
42 voxels). Visual inspection confirms this assumption. Large numbers of distances for dendrite, axon, and all neurites are similar to our method, but
43 weak fibers are recognized by NeuMiner, but not by APP2 and the 𝑆𝐷12 distance is 12 times larger than our baseline, highlighting the
44 SmartTracing (Fig. 5). The improvement varies for different neurite types: great noise discrimination power of the segmentation network, consistent
45 the average distance 𝑆𝐷12 of the axons is decreased by 89.5% (from 17.86 with previous studies. When all those tricks are turned off, i.e., the original
46 to 1.87, Table. 1), compared to 79.0% (from 4.42 to 0.93, Table. 1) for APP2, the performance is much worse as previously discussed.
47 dendrites. Overall, the recall of weak fibers is increased from 5.1% to
48 27.8% (Fig. 1c and Fig.5d). The improvement is more prominent for
49 axons, which increased by 6.4 times, compared to 2.0 times for dendrites. 3.4 Single neuron tracing
The average 𝑆𝐷12 distances for axons are 1.87 and 1.24 voxels, which are
50 Single neuron tracing is highly challenging for long projection neurons
even smaller than dendrite distances of the original APP2 and across multiple brain regions. We tested the performance of NeuMiner on
51
SmartTracing (Table. 1). single neuron tracing, using UltraTracer (Peng et al., 2017) plugin in
52 A straightforward alternative to increase the recall of weak fibers is Vaa3D (Peng et al., 2010).
53 tracing with a lower background threshold. To evaluate the solution, we Results show NeuMiner improves the single neuron tracing on most
54 decreased the background threshold by half of the standard deviation of metrics and categories. The average distances (the last values of All
55 the input image, and re-run APP2. The average 𝑆𝐷12distance decreased neurites item in Table. S3) of the NeuMiner-enabled version for all
56 (from 11.82 to 5.48), while the average 𝑆𝐷12 distance increased neurites are lower, and it is more prominent on axon recognition. While
57
58
59
60
Page 7 of 9 Bioinformatics

Article short title

1
2
3 reconstructions of UltraTracer is significantly shorter than gold standards
a b
4 (path length ratio to gold standards is 0.348±0.279), NeuMiner enabled
5 UltraTracer to get reconstructions with path length similar to gold
6 standards (path length ratio to gold standards is 0.804±0.823).
Although most of the dendritic fibers are identified, many axonal fibers
7
are missed. Only 7% of dendritic voxels in gold standards are not

Downloaded from https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btac816/6960919 by guest on 27 December 2022

8
identified, and the average spatial distance 𝑆𝐷12 is small (4.26, Table. S3).
9 On the other hand, the average 𝑆𝐷12 and percentage of different structures
10 (𝑃𝐷𝑆12) for axonal voxels are much higher (721.57 and 0.668, Table. S3).
11 The reason for the poor axon recognition is there are large numbers of
12 inter-neural fiber crossings (Fig. 6b), and UltraTracer early stops to avoid
13 over-tracing caused by crossing. c d
14
15
3.5 Over-tracing
16
17 Over-tracing is a common problem for all tested methods, from the
baseline methods, i.e., APP2 and SmartTracing, to our enhanced versions
18
(Table. 1). This is mainly caused by inter-neuronal fiber crossing. We
19
computed self-crossing (crossing between fibers of the same neuron) as a
20 coarse estimation of crossing. To this end, we calculated the pairwise
21 distances between the nodes within the central (512 × 512 × 256) crop of
22 all gold standards, after excluding nodes less than 50 voxels from the soma
23 center and those sharing ancestors or offspring within 10 generations. The Figure 6. Fiber crossing and over-tracing. (a) Image volume with gold
24 average number of pairs with distances less than 1 voxel is 3.7, showing standard overlaid. (b) Image volume with reconstruction overlaid. Inter-
25 considerable self-crossing between fibers. Assuming inter-neuronal
crossing is pointed out by yellow arrows. (c) Violin plot for distributions
26 distances share similar distribution, the inter-crossing caused over-tracing
should not be uncommon. of total path length. GS, APP2, NeuMiner and GS_trace are total path
27
We also conducted an interesting experiment, in which the tracing length distributions for gold standards, APP2 reconstructions,
28
29 image is fused with the label image instead of segmentation. Ideally, the reconstructions of APP2 with NeuMiner enabled, and reconstructions for
average 𝑆𝐷21 distance should be 0. However, the average 𝑆𝐷21 for APP2 gold standards fused images. (d) Correlation between 𝑆𝐷21 and the
30
tracing is 3.43. These results again highlight the existence of inter-
31 number of nodes in reconstructions. Triangular blue points are sampled
neuronal fiber crossing and self-crossing problems under low-resolution
32 conditions. tests by randomly deleting a certain number of nodes and all downstream
33 The problem can be further confirmed by the total path length statistics nodes. Blue line is the fitted line for those tests. The circular red point and
34 (Fig. 6c). Morphologies traced by APP2 undergo remarkable under- triangular red point are APP2 and our baseline. The precision (𝑆𝐷21) is
35 tracing, with very few cases greatly over-traced. After integrating with our
linearly correlated to the number of nodes reconstructed. Therefore, the
36 method, there is a little bit of over-tracing, and the upper tail is supposed
relatively higher 𝑆𝐷21 distances should be a byproduct of higher neurite
37 to be caused by the inter-neuronal crossing. As for the morphologies of
the label fused images, the total length is slightly larger than the gold recognition, but not errors from our method.
38
39 standards.
We also conducted additional analysis to pursue the reason why 𝑆𝐷21
40 of our method is larger than APP2 baseline. Results show the 𝑆𝐷21 values
41 are linearly correlated with the numbers of average traced nodes (Fig. 6d),
42 indicating that over-tracing stems from other reasons but not NeuMiner.
43
44
45 4 Discussion
46 In this work, we proposed a module for tracing weak fibers by introducing
47 false-negative mining and derivative truncated gamma transformation
strategies. Both strategies are demonstrated to improve the weak fibers
48
recognition by a large margin, based on the extensive evaluation of the test
49
set. Our study also illustrates that neuron segmentation with partial
50 labeling can identify most fibers, which greatly relieves the annotation
51 burden.
52 One limitation of NeuMiner is the use of fixed width (3 × 3 × 1 voxels)
53 fibers, which is inappropriate for axonal fibers, as the radii of axonal fibers
54 are small and many of them are only 1 pixel. In this case, the (3 × 3 × 1)
55 width ground truth may lead to artificial connections between signals,
56 especially in the densely packed axonal arbors. The reason for choosing a
57 fixed-width fiber is that there are small shifts between the gold standards
58
59
60
 Bioinformatics Page 8 of 9

Liu et al.
1
2
annotated at low resolution and the high-resolution signals. We are Gong,H. et al. (2016) High-throughput dual-colour precision imaging for brain-
3 designing strategies to remove the shifts and will then improve the wide connectome with cytoarchitectonic landmarks at the cellular level. Nat
4 segmentation accuracy of slim fibers. Commun, 7, 12142.
5 Another problem for NeuMiner and all other tracing algorithms is the Guo,S. et al. (2022) Image enhancement to leverage the 3D morphological
6 over-tracing caused by fibers crossing. Ideally, the over-tracing can be
reconstruction of single-cell neurons. Bioinformatics, 38, 503–512.
7 addressed with higher-resolution imaging systems, including optical
Huang,Q. et al. (2020) Weakly Supervised Learning of 3D Deep Network for

Downloaded from https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btac816/6960919 by guest on 27 December 2022

8 microscopy and electric microscopy. Nevertheless, there are many
Neuron Reconstruction. Front. Neuroanat., 14, 38.
9 difficulties for both techniques, e.g., high-resolution imaging and high-
throughput data processing. An alternative solution is post-tracing pruning Isensee,F. et al. (2021) nnU-Net: a self-configuring method for deep learning-based
10
of over-traced neurites. There are several works on this topic (Quan et al., biomedical image segmentation. Nat Methods, 18, 203–211.
11
2016; R. Li et al., 2019), most of them are mainly based on angular Jenett,A. et al. (2012) A GAL4-Driver Line Resource for Drosophila
12
preference, more analysis and attribution should be conducted to solve the Neurobiology. Cell Reports, 2, 991–1001.
13
problem. Li,Q. and Shen,L. (2020) 3D Neuron Reconstruction in Tangled Neuronal Image
14
With Deep Networks. IEEE Trans Med Imaging, 39, 425–435.
15
Li,R. et al. (2017) Deep Learning Segmentation of Optical Microscopy Images
16 Funding
Improves 3-D Neuron Reconstruction. IEEE Trans. Med. Imaging, 36, 1533–1541.
17 This work was funded by Provincial subsidy for the "Double First-Class" Initiative
Li,R. et al. (2019) Precise segmentation of densely interweaving neuron clusters
18 of Emerging Interdisciplinary of Brain Science (4050272031) and a MOST (China)
using G-Cut. Nat Commun, 10, 1549.
19 Brain Research Project, "Mammalian Whole Brain Mesoscopic Stereotaxic 3D
Li,S., Quan,T., Zhou,H., et al. (2019) Identifying Weak Signals in Inhomogeneous
20 Atlas" (2022ZD0205200, 2022ZD0205204) and also partially supported by NSFC-
Neuronal Images for Large-Scale Tracing of Sparsely Distributed Neurites.
21 Guangdong Joint Fund-U20A6005 and Key-Area Research and Development
Program of Guangdong Province (2018B030331001). Neuroinform, 17, 497–514.
22
Li,S., Quan,T., Xu,C., et al. (2019) Optimization of Traced Neuron Skeleton Using
23
Conflict of Interest: none declared. Lasso-Based Model. Front. Neuroanat., 13, 18.
24
Li,S. et al. (2017) SparseTracer: the Reconstruction of Discontinuous Neuronal
25
Morphology in Noisy Images. Neuroinform, 15, 133–149.
26 References
Liang,H. et al. (2017) Content-aware neuron image enhancement. In, 2017 IEEE
27 Al-Kofahi,K.A. et al. (2002) Rapid automated three-dimensional tracing of neurons
International Conference on Image Processing (ICIP). IEEE, Beijing, pp. 3510–
28 from confocal image stacks. IEEE Trans. Inform. Technol. Biomed., 6, 171–187.
3514.
29 Ascoli,G.A. et al. (2007) NeuroMorpho.Org: A Central Resource for Neuronal
Liu,M. et al. (2019) A Multiscale Ray-Shooting Model for Termination Detection
30 Morphologies. Journal of Neuroscience, 27, 9247–9251.
of Tree-Like Structures in Biomedical Images. IEEE Trans. Med. Imaging, 38,
31 Basu,S. et al. (2013) Segmentation and Tracing of Single Neurons from 3D
1923–1934.
32 Confocal Microscope Images. IEEE J. Biomed. Health Inform., 17, 319–335.
Liu,M. et al. (2018) Improved V-Net Based Image Segmentation for 3D Neuron
33 Brown,K.M. et al. (2011) The DIADEM data sets: Representative light microscopy
Reconstruction. In, 2018 IEEE International Conference on Bioinformatics and
34 images of neuronal morphology to advance automation of digital reconstructions.
Biomedicine (BIBM). IEEE, Madrid, Spain, pp. 443–448.
35 Neuroinform, 9, 143–157.
Liu,S. et al. (2017) Automated 3D Neuron Tracing with Precise Branch Erasing
36 Chen,H. et al. (2015) SmartTracing: self-learning-based Neuron reconstruction.
and Confidence Controlled Back-Tracking Neuroscience.
37 Brain Inf., 2, 135–144.
Liu,Siqi et al. (2016) Rivulet: 3D Neuron Morphology Tracing with Iterative Back-
38 Chen,W. et al. (2021) Spherical-Patches Extraction for Deep-Learning-Based
Tracking. Neuroinform, 14, 387–401.
39 Critical Points Detection in 3D Neuron Microscopy Images. IEEE Trans. Med.
Manubens-Gil,L. et al. (2022) BigNeuron : A resource to benchmark and predict
40 Imaging, 40, 527–538.
best-performing algorithms for automated reconstruction of neuronal morphology
41 Chiang,A.-S. et al. (2011) Three-dimensional reconstruction of brain-wide wiring
Bioinformatics.
42 networks in drosophila at single-cell resolution. Current Biology, 21, 1–11.
Milletari,F. et al. (2016) V-Net: Fully Convolutional Neural Networks for
43 Citraro,L. et al. (2020) Towards Reliable Evaluation of Algorithms for Road
Volumetric Medical Image Segmentation. In, 2016 Fourth International
44 Network Reconstruction from Aerial Images. In, Vedaldi,A. et al. (eds), Computer
Conference on 3D Vision (3DV). IEEE, Stanford, CA, USA, pp. 565–571.
45 Vision – ECCV 2020, Lecture Notes in Computer Science. Springer International
Peng,H. et al. (2011) Automatic 3D neuron tracing using all-path pruning.
46 Publishing, Cham, pp. 703–719.
Bioinformatics, 27, i239–i247.
47 Coleman,P.D. et al. (1977) Semiautomatic Tracking of Neuronal Processes. In,
Peng,H. et al. (2017) Automatic tracing of ultra-volumes of neuronal images. Nat
48 Lindsay,R.D. (ed), Computer Analysis of Neuronal Structures. Springer US,
Methods, 14, 332–333.
49 Boston, MA, pp. 91–109.
Peng,H. et al. (2015) BigNeuron: Large-Scale 3D Neuron Reconstruction from
50 Garvey,C.F. et al. (1973) Automated three-dimensional dendrite tracking system.
Optical Microscopy Images. Neuron, 87, 252–256.
51 Electroencephalography and Clinical Neurophysiology, 35, 199–204.
Peng,H. et al. (2021) Morphological diversity of single neurons in molecularly
52 Glaser,J.R. and Glaser,E.M. (1990) Neuron imaging with neurolucida — A PC-
defined cell types. Nature, 598, 174–181.
53 based system for image combining microscopy. Computerized Medical Imaging
Peng,H. et al. (2010) V3D enables real-time 3D visualization and quantitative
54 and Graphics, 14, 307–317.
analysis of large-scale biological image data sets. Nat Biotechnol, 28, 348–353.
55
Quan,T. et al. (2016) NeuroGPS-Tree: automatic reconstruction of large-scale
56
neuronal populations with dense neurites. Nat Methods, 13, 51–54.
57
58
59
60
Page 9 of 9 Bioinformatics

Article short title

1
2
3 Ronneberger,O. (2017) Invited Talk: U-Net Convolutional Networks for Wang,H. et al. (2021) Single Neuron Segmentation using Graph-based Global
4 Biomedical Image Segmentation. In, Maier-Hein, geb. Fritzsche,K.H. et al. (eds), Reasoning with Auxiliary Skeleton Loss from 3D Optical Microscope Images.
5 Bildverarbeitung für die Medizin 2017, Informatik aktuell. Springer Berlin arXiv:2101.08910 [cs, eess].
6 Heidelberg, Berlin, Heidelberg, pp. 3–3. Xiao,H. and Peng,H. (2013) APP2: automatic tracing of 3D neuron morphology
7 Shen,L. et al. (2021) Efficient Critical Point Detection for Curvilinear Structures based on hierarchical pruning of a gray-weighted image distance-tree.

Downloaded from https://academic.oup.com/bioinformatics/advance-article/doi/10.1093/bioinformatics/btac816/6960919 by guest on 27 December 2022

8 Using a Ring-Like Ray-Shooting Model. IEEE Trans. Instrum. Meas., 70, 1–11. Bioinformatics, 29, 1448–1454.
9 Tan,Y. et al. (2020) DeepBranch: Deep Neural Networks for Branch Point Yang,J. et al. (2019) FMST: an Automatic Neuron Tracing Method Based on Fast
10 Detection in Biomedical Images. IEEE Trans. Med. Imaging, 39, 1195–1205. Marching and Minimum Spanning Tree. Neuroinform, 17, 185–196.
11 Ulyanov,D. et al. (2017) Improved Texture Networks: Maximizing Quality and Zhou,H. et al. (2021) GTree: an Open-source Tool for Dense Reconstruction of
12 Diversity in Feed-Forward Stylization and Texture Synthesis. In, 2017 IEEE Brain-wide Neuronal Population. Neuroinform, 19, 305–317.
13 Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Zhou,Z. et al. (2018) DeepNeuron: an open deep learning toolbox for neuron
14 Honolulu, HI, pp. 4105–4113. tracing. Brain Inf., 5, 3.
15 Wang,C.-W. et al. (2017) Ensemble Neuron Tracer for 3D Neuron Reconstruction. Zhou,Z. et al. (2016) TReMAP: Automatic 3D Neuron Reconstruction Based on
16 Neuroinform, 15, 185–198. Tracing, Reverse Mapping and Assembling of 2D Projections. Neuroinform, 14,
17 Wang,H. et al. (2019) Multiscale Kernels for Enhanced U-Shaped Network to 41–50.
18 Improve 3D Neuron Tracing. In, 2019 IEEE/CVF Conference on Computer Vision
19 and Pattern Recognition Workshops (CVPRW). IEEE, Long Beach, CA, USA, pp.
20 1105–1113.
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60