To quantitatively characterize neuronal morphology, various hand-crafted morphometrics (Uylings and Van Pelt, 2002; Costa et al., 2010; Wan et al., 2015; Bird and Cuntz, 2019) are designed and widely utilized. These morphometrics can be primarily divided into four categories. They are distance-related metrics (e.g., branch length and Euclidean distance), angle-related metrics (such as bifurcation angles), topology-related metrics (including the number of bifurcations and branch order), and size-related metrics (like branch radius and surface area) (Chen et al., 2022). Typically, these features are summarized using statistics, such as mean or standard deviation. They are easily acquired using toolboxes (e.g., L-Measure Scorcioni et al., 2008, Neurolucida Glaser and Glaser, 1990, and nGauge Walker et al., 2021).

Traditional methods for neuronal morphology classification mainly rely on these hand-crafted morphometrics (Vasques et al., 2016; Laturnus et al., 2020; Cervantes et al., 2019). For example, 43 morphological measurements computed by the L-Measure toolbox (Scorcioni et al., 2008) are utilized to describe the characteristics of rat neurons (Vasques et al., 2016). Although these methods effectively identify neuronal morphology, they would fail to describe neurons with complex dendrites (Scorcioni et al., 2008). Other methods utilize the Sholl analysis (Sarkar et al., 2013; Khalil et al., 2022) and TMD (Kanari et al., 2018) to classify or cluster neurons. For example, Kanari et al. (2019) use TMD to classify cortical pyramidal cells in rat somatosensory cortex. Khalil et al. (2022) treat the multiple topological or spatial features of neurons as a function of distance from the soma and builds new Sholl descriptors. Their single descriptors perform better than traditional hand-crafted morphometrics in separating specific neuron types. More importantly, their results prove that combined descriptors can produce better classification or clustering performance than single descriptors. This motivates us to explore the fusion of multiple features for better classification or clustering for neuronal morphology.

Inspired by the advancements in deep learning techniques, some studies utilize CNNs to extract deep features from the 3D neuron data or 2D images (Lin and Zheng, 2018; Li et al., 2018; Zhang et al., 2021; Sun et al., 2023; Li et al., 2021; Zhao et al., 2022; Kanari et al., 2024). These methods directly feed 3D neuron data or 2D images into CNNs to automatically learn deep feature representation. Importantly, compared to the hand-crafted morphometrics, deep features obtained through CNNs are good at characterizing holistic structures of neurons (Li et al., 2018; Zhang et al., 2021). However, there are many difficulties in extracting deep features via CNNs directly from 3D neuron data. Specifically, the number of points in 3D space for neuron data varies greatly among various neuron samples. Consequently, creating a general neural network to extract deep characteristics from 3D neuron data is challenging. Additionally, if 3D neuron data is projected into 2D images, neuronal morphology information inevitably is lost during the projection process. As a result, the performance of these 2D projection-based approaches is constrained. Recent researches (Li et al., 2018; Zhang et al., 2021) attempt to create a feature vector by directly concatenating different features. While their results demonstrate that the fusion of diverse feature representations leads to performance gains, these improvements are constrained. This is because they do not consider the complementary and redundant information among the different features. Therefore, our method aims to leverage the strengths of these features and effectively fuse them to provide a precise characterization of neuronal morphology.

Attention mechanisms discern and emphasize pivotal local regions, building more discriminative and pertinent features. Attention mechanisms have showcased remarkable success across various applications, such as image classification (Hu et al., 2018; Woo et al., 2018; Park et al., 2018; Hassanin et al., 2024), machine translation (Gehring et al., 2017; Vaswani et al., 2017), and visual question answering (Yu et al., 2017; Gao et al., 2018). For instance, in the domain of image classification, SENet (Hu et al., 2018) introduces a channel attention block to augment the representational capacity of neural networks. Woo et al. (2018) further extend these advancements by integrating channel and spatial attention modules into a unified block structure. Later, Vaswani et al. (2017) achieve another significant advancement in attention-based models based on the self-attention mechanism. The self-attention captures long-range dependencies effectively and provides interpretability by highlighting the relevance of different input components (Vaswani et al., 2017; Lin et al., 2017). It is highly scalable, parallelizable, and can be easily stacked in layers for more complex modeling. Here, our approach leverages a variant of self-attention, referred to as the cross-attention module, to effectively harness the dependencies and relationships of different features. This aims to facilitate the interaction of features extracted from different branches, promoting a more cohesive and integrated feature representation.

In our work, the hand-crafted morphometrics and 2D images are taken into account simultaneously to learn a more distinguishing representation of neuronal morphology. Hand-crafted morphometrics almost measure the local geometry information of neurons, while deep features extracted from 2D projected images better characterize the holistic structures of neurons (Li et al., 2018).

There are many predefined feature measurements to measure neuronal dendrites and axons (Laturnus et al., 2020; Uylings and Van Pelt, 2002). Based on the neuron data stored in the SWC file, morphological features computed by the L-Measure toolbox (Scorcioni et al., 2008) are widely used to characterize morphologies of neurons quantitatively. They consist of total branch length, average branch diameter, average depth, number of bifurcations, maximum branch order, etc. As done in (Lin et al., 2018; Lin and Zheng, 2019; Li et al., 2018), we calculate 43 hand-crafted morphometrics (Figure 1D) using the L-Measure toolbox (Scorcioni et al., 2008) to measure the morphological characteristics of neurons comprehensively.

Designing a generalized 3D network to directly process 3D neuron data is challenging. This is due to the inherent characteristic of neuron reconstructions (e.g., relative sparse in 3D space, dramatic difference in the number of points). Therefore, we first project the 3D neuron data into 2D images as done in Li et al. (2018), Li et al. (2021), and Sun et al. (2023) and then utilize a 2D CNN to extract morphological features. We first extract the coordinates of neuron points and employ principal component analysis to normalize neurons into a normalized axis. Such an operation ensures the axis is consistent across neurons, without considering the origin axis provided by multiple laboratories. Then, to reduce the loss of morphological information, the 3D neuron data is projected orthogonally into three angles of view (i.e., x-y, y-z, and x-z planes). Thus, three projected images are obtained to fully describe different neuronal morphology (shown in Figure 4). Subsequently, these three projected images are stacked together along the channel and fed into CNN.

Given a 3D digital neuron reconstruction, we first preprocess it to obtain the hand-crafted morphometrics and 2D projected images. Subsequently, an FCN branch and a CNN branch are applied to extract neuronal morphological features from these two data formats. When the information flows through the two branches, the MLFMs we developed are embedded into the feature extraction block for information enhancement and communication. Besides, each branch can utilize the information from the other to facilitate their representation learning.

For the FCN branch, considering that the hand-crafted morphometrics are statistical summary values and the number of them is limited, we opt to employ a shallow FCN to extract features. It consists of several FCN blocks. Note that the number of FCN blocks is consistent with the number of blocks in the CNN for better information communication. Besides, each FCN block includes three Fully Connected (FC) layers, two Batch Normalization (BN) layers, and two ReLU (Nair and Hinton, 2010) layers. The dimensions of the output of the first two FC layers are half and a quarter of the dimension of the input, respectively. To better perform information exchange channel-wise between two networks in the MLFM, the dimension of the output of the third FC layer is set to be the same as the number of channels of the corresponding convolution block in the CNN.

For the CNN branch, we first stack three projected images together along the channel and then feed them into CNN to extract deep features. Here, we set the CNN as the ResNet-50 (He et al., 2016). Considering that the output of the first block is relatively specific, information communication with the FCN occurs starting from the second convolution block. Therefore, three MLFMs are added to bridge the information communication between the FCN and CNN. Each MLFM receives the features extracted from the FCN and CNN. Then, it enhances each feature through the channel attention-based FEM. Next, it discovers and transmits the interacted information between the FCN and CNN branches via cross-attention-based FIM. In this way, these two branches can comprehensively learn the morphological features of neurons with the help of the other branch.

To achieve a more comprehensive characterization of neuronal morphology, we further fuse features extracted from the FCN and CNN branches to generate a more informative vector as the final descriptor (denoted as F_j). Subsequently, a classifier, consisting of two FC layers, is employed to categorize neurons based on this final descriptor. The dimensions of the output of these FC layers are the dimension of the lasted fused feature and the number of neuron types, respectively.

To fully capture the neuronal morphology, some approaches concatenate different feature vectors into a vector directly (Li et al., 2018; Zhang et al., 2021). However, these methods do not thoroughly consider the complementarity of different features. To capture informative features for each branch and the complementarity between different branches, we introduce the MLFM. It includes a channel attention-based FEM and a cross-attention-based FIM.

As presented in Figure 5A, the MLFM takes the output (denoted as F and C) of the feature extraction block in the FCN and CNN branch as the input. It first utilizes the channel attention-based FEM in each branch to capture the salient features (F′ and C′). Then, the salient features are fed to cross-attention-based FIM to mine the complementary features between F′ and C′. Finally, the outputs of these two modules are concatenated to generate a more informative feature (i.e., f_l and f_c for FCN and CNN branches, respectively). The fused feature is then fed into the next feature extraction block to facilitate the presentation learning of each branch. To keep the dimension of the concatenated features as same as the input (F), an FC layer in the FCN branch is employed to reduce the dimension of the fused features. Similarly, a convolution layer with a kernel size of 1 × 1 in the CNN branch is utilized to reduce the feature dimension. Therefore, the concatenated features can be directly fed back into the FCN and CNN branches to further convey information in the next feature extraction block.

Feature Enhancement Module (FEM). The FEM mines salient features within a single branch to dynamically enhance and update reliable information. Here, we illustrate the FEM using the FCN branch as a representative example (as shown in Figure 5B).

To enhance the discriminability of features, especially given the sparseness of neurons, we develop a channel attention-based FEM. As depicted in the green box of Figure 5B, the FEM first calculates the weights (W_f) for F. This is achieved through two FC layers and a sigmoid function as follows,

where δ is the ReLU function (Nair and Hinton, 2010), fc is the FC layer, and σ is the sigmoid function. The feature dimensions of FC layers are half and full of the dimension of F, respectively. The sigmoid layer σ enables our module to learn non-linear interactions between channel features. It allows for the emphasis of multiple channel features. The enhanced features F_e are then defined by applying the calculated weights to the normalized features as,

Such operations ensure that the FEM effectively improves the discriminative power of features by leveraging channel attention mechanisms.

Furthermore, to address the morphological diversity of neuronal arbors, we implement a group-wise refinement process for the global features F. Specifically, we divide the global features (F) into N_s parts along the channels, resulting in grouped features (F_s). As illustrated in Figure 5B, the green, orange, and light blue rectangles represent different grouped features. Drawing from the optimal outcomes of our previous research, we set N_s as 3. For each group, we apply a feature learning module M to enhance the grouped feature. The M consists of two FC layers and a BN layer. The enhanced grouped features F_s1 are then constructed by concatenating the outputs of the module M applied to each group as follows:

where [·] is the concatenation operation. Subsequently, the final enhanced global features (F′) are generated by concatenating the previously enhanced features F_e with the newly enhanced grouped features F_s1. Besides, an FC layer is utilized to reduce the dimension of the F′ as the same as F.

For the CNN branch, the same operation is conducted to capture the final enhanced features (C′). Note that the feature (C) is first fed to one average pooling layer to get global features and then fed into the FEM to extract enhanced features. Moreover, a 1 × 1 convolution layer is utilized to reduce the dimension of the C′ as the same as C.

Feature Interaction Module (FIM). The proposed FIM (as shown in Figure 5C) is designed to facilitate dynamic interaction between the two representations. It facilitates our model to effectively extract more distinctive and comprehensive morphological representations for neuronal morphology.

We obtain the enhanced features F′ and C′ through FEM for the FCN and CNN branches, respectively. The FIM first transforms them into one-dimensional feature vectors with the same dimension. This can be achieved through average pooling or reshaping operations. We leverage the principles of the cross-attention mechanism (Galassi et al., 2020; Woo et al., 2018; Park et al., 2018) to uncover the relationships and shared information between different branches, thereby enhancing the feature learning for each branch. In this process, features from one branch act as the query (Q), while those from the other branch serve as the key (K) and value (V). The shared features between the FCN and CNN branches are obtained through a scaled dot-product operation, as delineated as:

where d is the channel dimension of features. The captured shared features are crucial for promoting the mutual refinement of different branches.

To amplify the distinctive traits of the FCN branch, we fully utilize the input Q (i.e., F′) to generate the final interactive feature F″. This involves a multi-step process, including the application of an addition layer, an FC layer, and a BN layer, as expressed as:

where fc represents the FC layer, and BN signifies batch normalization. Similarly, we can obtain C″ for the CNN branch through the same operations. In this way, the FIM is meticulously designed to fortify the branch's distinctiveness by integrating the adaptive attention features and fostering collaborative learning between the two branches.

The designed multi-level feature fusion network comprehensively characterizes neuronal morphology by fully improving the feature learning of FCN or CNN and tactfully integrating different features. Therefore, our learning goal is to train the CNN and FCN to extract distinguishable characteristics and promote the accurate classification of neurons. The objective function is:

where N is the number of samples, c is the number of classes, and y_ij is the true neuron type provided by experts. As shown in Figure 2, the p_ij is the prediction given by the classifier based on the fused features F_j, where F_j are derived from the features F_l and F_c extracted from the FCN and CNN branches.

The proposed multi-level feature fusion network mines the salient information from 2D images and hand-crafted morphometrics from multiple levels. It conveys fused and complementary features to CNN and FCN to enhance their representation learning. We first verified the performance of the CNN and FCN based on the images and hand-crafted morphometrics, respectively. As illustrated in Table 1, the classification accuracy of CNN with images as input can reach 85.44%. The FCN with geometric statistics as input can identify 10 types of neurons with an accuracy of 82.40%. The proposed multi-level feature fusion network makes full use of features obtained from these two inputs. It enhances the distinguishing features from one network and propagates the distinguishing information across these two networks on multiple levels. Therefore, our method reaches an accuracy of 95.18%.

As shown in Figure 7, our method can accurately and reliably identify each type of neuron (as shown in Figures 7A, B). Although different types of neurons share similarities in their shape and size, the proposed multi-level fusion network is still able to effectively capture the differences between types and represent them accurately. Our method achieves excellent AUC and F1-score for each type of neuron. Specifically, our method achieves satisfactory AUC values for each type, where AUC values exceed 0.999 for most classes. Furthermore, our method achieves the AUC values of 1.000 for C4 and C8, indicating our method can be used to precisely analyze the neurons from C4 and C8 types. Moreover, our method yields an average F1-score of 0.94 and the F1-scores of all classes exceed 0.89. This demonstrates that our method effectively represents the differences among neuron classes and accurately identifies neurons. Most of the misclassified samples by our method are from C6 (i.e., mouse pyramidal cells) and C10 types (i.e., mouse pyramidal cells) (as shown in Figure 7C). Specifically, 6% of neurons from the C6 type are misclassified as C10 type, while 6% of neurons from the C10 type are misclassified as C6 type (as shown in Figure 7C). This is because there are some morphological similarities between these two types. For example, the average contraction of neurons from C6 and C10 are 0.8949 ± 0.0571 and 0.8925 ± 0.0611, respectively. The ratio between the diameter of a daughter and its father of neurons from C6 and C10 are 0.8642 ± 0.1733 and 0.8502 ± 0.0965, respectively. This indicates that our method should be further modified to effectively learn and identify these subtle morphological nuances. Therefore, our future work will focus on improving the ability of our method to represent subtle differences so that it can accurately describe the subtle morphological properties of neurons.

Besides, the t-SNE tool (Van der Maaten and Hinton, 2008) is exploited to visualize the feature distributions of the last layer of our network. As seen from Figure 7D, each category is grouped into a cluster and neurons in each cluster are relatively compact. This indicates that our method can effectively learn and characterize the morphological features of each type of neuron. Besides, we observe that the neurons in the C4 cluster (i.e., various principal medium spiny) distribute relatively sparse compared to other clusters. We randomly select the neurons of the C4 type without considering their species and brain regions. Moreover, neurons in the C4 type have more dendritic spines than other cells. These factors may cause relatively large morphological differences among neurons in the C4 type. Nonetheless, our method accurately characterizes the morphological differences of neurons in the C4 type and groups them. Furthermore, there is a great deal of distance between different clusters. These results further prove that our method can precisely different types of neurons and fully using different features is conducive to recognizing neuron types.

These neuron classification methods based on hand-crafted morphometrics and traditional classifiers are re-implemented to make a comparison with our method, including TMD (Kanari et al., 2018), novel Sholl analysis (Khalil et al., 2022), Support Vector Machine (SVM; Lee et al., 2004), Naïve Bayes (NB; Mihaljević et al., 2015), Linear Regression (LR; Alavi et al., 2009), and k-Nearest Neighbors (KNN; Wu et al., 2008). The recent methods that take hand-crafted morphometrics as input and neural networks as feature extractors are also compared, including Deep Residual Neural Networks (DRNN; Lin et al., 2018) and Locally Cumulative Connected Deep Neural Networks (LCCDNN; Lin and Zheng, 2019). The MorphoGNN (Zhu et al., 2022) method employs Graph Neural Networks (GNN) to learn the spatial structure information between the nodes of reconstructed neuron fibers. Besides, the SCAEs and TRNN+CNN extract features from 2D neuron images and hand-crafted morphometrics or SWC data are taken as a comparison (Zhang et al., 2021; Li et al., 2018). They concatenate different features directly to generate a feature descriptor. Additionally, we compare our method to the methods (Lin and Zheng, 2018; Chen et al., 2022) directly taking the 3D neuron data as input. Table 2 reports the classification performance of different methods.

As seen from Table 2, the classification performance of these methods that only use hand-crafted morphometrics to characterize neurons is barely satisfactory, and the highest accuracy is only 81.28% achieved by the KNN method (Wu et al., 2008). Since hand-crafted morphometrics are a set of statistical measurements and have 43 dimensions, they are not good at describing the holistic structures of neurons with complex morphological structures. Therefore, these methods perform relatively weaker. In contrast, our method characterizes neurons by integrating hand-crafted morphometrics with deep features extracted from 2D view images. This allows our method to capture a comprehensive descriptor for neuronal local and holistic structures. Consequently, our approach effectively identified neuronal types. The novel Sholl method built by Khail et al. achieves satisfactory performance with an accuracy of 90.00%. However, creating a tree structure to obtain the Sholl descriptors requires higher computation resources and time, especially for neurons with a larger number of points (such as those >8,000). While MorphoGNN achieves an accuracy of 85.89%, it requires processing the number of neuron points to a unified input size, such as 1,024 or 2,048. The larger the input size, the higher the computation resources. Note that the number of neuron points ranges from 10 to 10,000. Therefore, their method risks losing significant morphological information for neurons with greater points and introducing noise for neurons with fewer points. Although our method utilizes the same input as the SCAEs (Li et al., 2018), our method utilizes the MLFM to facilitate the information interaction between different inputs at multiple levels. Conversely, SCAEs directly connect deep features and hand-crafted morphometrics. Our method can obtain complementary information between different features and effectively use their advantages. Therefore, the performance of SCAEs is relatively lower than that of our method. Besides, the TRNN+CNN (Zhang et al., 2021) method utilizes CNN and tree-based RNN to capture morphological features from SWC format data and 2D slice format data. However, it directly fuses different features only before the classifier to build the morphological descriptors. Compared to the SCAEs (Li et al., 2018) and TRNN+CNN (Zhang et al., 2021), our method fully uses two different features and tactfully integrates them at multiple scales during feature extraction. Furthermore, it mines and shares complementary information between two features via the MLFM. Consequently, our method yields an accuracy of 95.18% and an F1-score of 0.952, outperforming other methods.

Here, we verify the generalization ability of our method from many aspects. In this section, we utilize the same data preprocessing described in Section 3.1 and experimental setting reported in Section 4.1.2 on different datasets to conduct the evaluation experiments. Researchers analyzing neuronal morphology can conveniently find the datasets on our Google drive (see text footnote ¹).

In the work of Zhang et al. (2021), a tree-based RNN is used to process the SWC format data, and a CNN is employed to extract features from 2D slice images. To verify their method, they download 35,000 digital neuron reconstructions from NeuroMorpho.org (Ascoli et al., 2007). Moreover, they augment the training images to 98,700 and SWC-format data to 99,996. The model concatenates the features extracted by CNN and RNN and achieves an accuracy of 91.90%. The same dataset as that in their work is exploited to evaluate our proposed method (denoted as NeuroMorpho-12). NeuroMorpho-12 includes the 12 types of cells (as shown in Figures 8A, B). We use only 5,752 cells provided in their study (Zhang et al., 2021) and do not utilize data augmentation techniques. Figure 8C shows the ROC curves of our method for each type. Our method identifies 12 types of neurons with satisfying AUC values. See from Table 3, our method achieves a classification accuracy of 92.00%. Note that, our method makes comparable results with Zhang et al. (2021), but the number of neurons used in our method only is 1/19 of that in Zhang et al. (2021). Our multi-level fusion method not only enhances the individual feature representations but conveys the interacted information between different feature representations. Consequently, our method achieves comparable results with smaller neurons compared to that (Zhang et al., 2021) directly concatenating feature presentations.

Besides, a dataset (denoted as NeuroMorpho-17) consisting of 17 classes of cells is utilized to verify the performance of the proposed method. The NeuroMorpho-17 consists of 47,460 cells downloaded from the NeuroMorpho.org (Ascoli et al., 2007). As shown in Figures 8D, E, this dataset includes more types compared to the NeuroMorpho-10 and NeuroMorpho-12 datasets. The number of neurons in each type ranges from 1,000 to 3,000, making neuron identification challenging. As shown in Figure 8F, our method yields excellent AUC values for each type. This indicates that our method fits well in identifying more types of neurons. Moreover, as shown in Table 3, our method yields an accuracy of 90.10% and an F1-score of 0.902, which is comparable to that of the other two datasets. Our method extracts morphological features of neurons from the hand-crafted morphometrics and 2D images and tactfully fuses them at the multi-level. It can more accurately characterize the structure of neurons and the differences between different neurons. Therefore, although neurons in this dataset have more complex arbors and exhibit more morphological diversity, our method still effectively characterizes their morphologies and accurately identifies them. More importantly, these results show the capability of our method to discern a broader range of neuron types and open up a novel avenue for the analysis of large-scale neuronal morphology.