To demonstrate the applicability of our presented system for ACL diagnosis, we evaluate the system on two MRI datasets, one is the in-house MRI-ACL dataset and the other is the publicly available MRNet-v1.0 dataset ¹ (Bien et al., 2018). The type, the total number, the number of views and ACL patients, and the division of training and test sets for each dataset are summarized in Table 1. The details of the two datasets are as follows:

As shown in Figure 1, our system is based on a multi-view CNN to achieve ACL diagnosis by fusing features from different MRI views. The detailed architecture of the 3D CNN consists of residual-connected convolutional layers, pooling layers, activation function, and fully connected layer, as shown in Figure 2. Overall, our system consists of a feature extraction module, a feature weighted fusion module, and a lesion diagnosis module. The feature extraction module automatically maps the original knee joint MRI scans into features containing semantic information. The feature weighted fusion module weights and fuses all features according to the proportion of information provided by each view. The lesion diagnosis module maps the fused features to a final diagnostic score by a supervised classifier. We provide details of these modules in the following sections.

The feature extraction module is a 3D CNN without fully connected layers for extracting MRI representations, as shown in Figure 1A. In our framework, all views share the weights of the model, and the advantages of shared weights are as follows: 1) Improving the training speed and efficiency, especially for 3D networks with more parameters and memory; 2) Improving the generalization ability of the model; 3) Improving interpretability, as it can learn complementary information from different views; 4) Reducing heterogeneity among different views.

Given a 3D MRI view v ∈ R D × H × × W , where D, H, W represent the depth (number of slices), height, and width. We extract features by using residual blocks in the module, which is mathematically formulated as follows: z = F v , W i + v , (1) where v and z are the input and output vectors of a residual block. The function F ( x , { W i } ) represents the residual mapping. If the residual block has two layers, F = W 2 σ ( W 1 x ) , where σ denotes ReLU activation function (Nair and Hinton, 2010). Finally, we can formulate the feature extraction process after all residual blocks as h = f x , (2) where f and h represent the feature extraction network (the stacked of all residual blocks) and learned view feature, respectively.

We simultaneously forward all MRI views of the same patient through a feature extraction encoder with shared weights to obtain their individual representations. To obtain the overall characteristic of the knee joint, we further employ a weighting mechanism to fuse these individual representations and map them into the same embedding space, as shown in Figure 1B. The purpose of this step is to embed the representations of all views into the same embedding space and reduce the gap between views. Suppose there are three views in an MRI sequence, namely, v = {v ₁, v ₂, v ₃}. Therefore, we can obtain the individual features h = {h ₁, h ₂, h ₃} after feature extraction module. To get the weight of each view, we first feed the features of all views separately into a multi-layer perceptron (MLP), thus, the MLP will output three weight values. Next, we apply a softmax function to normalize these weights. Finally, the representations of each view are weighted and concatenated to form a unified visual feature.

We can regard the final lesion diagnosis task as a binary classification task, i.e., separating ACL patients from all patients. Therefore, we use a binary classifier to map the fused representations to the final diagnostic score, as shown in Figure 1C. The classifier is based on a two-layer MLP. Assuming that there are n views and the feature dimension of each view obtained by the feature extraction module is 512, and the dimension of the concatenated feature is n × 512. The first layer of the MLP maps the feature to the dimension of 512, and the second layer will output the binary classification result.

During the training process, we use the cross-entropy loss to measure the training effect of the model. The cross-entropy loss formula is as in Eq. 3. L y , y ˆ = − y l o g y ˆ + 1 − y l o g 1 − y ˆ , (3) where y is the ground truth, y = 1 if a patient suffers from an ACL tear, otherwise y = 0. The yˆ represents the prediction of the model.

In this study, we use 3D CNN (He et al., 2016) as the baselines for all experiments. We describe the architecture and details of 3D CNN in Figure 2. The method is trained using an Adam optimizer. Meanwhile, we train the model for 100 epochs with an initial learning rate of 0.0001 and a batch size of 32. The momentum and weight decay coefficient are set to 0.9 and 0.0001, respectively. Throughout the training phase, the model that achieves the best performance on the validation set is used for the evaluation. We implement our method with PyTorch using NVIDIA Tesla A100 40 GB GPUs. We use the area under the receiver operating characteristic curve (AUC) to evaluate all methods.

In this section, we compare the diagnosis performance of our presented system with several existing methods on the in-house MRI-ACL dataset and the publicly available MRNet-v1.0 dataset, as shown in Table 2. We divide the existing methods into two categories, one is single-view methods and the other is multi-view methods. The former uses only one view to train and evaluate the model, represented by the method of training 3D VGG16 (Simonyan and Zisserman, 2014) and 3D ResNet-10 (He et al., 2016) from scratch, respectively. The latter achieves ACL diagnosis by fusing different views, including MVCNN (Su et al., 2015) and MRNet (Bien et al., 2018). To perform a fair comparison, we use the same initial learning rate and image size as our model for all the above models. The AUC is used to evaluate the performance of ACL diagnosis comprehensively. As can be seen from Table 2, our system outperforms 3D VGG16, 3D ResNet-10, MVCNN, and MRNet on the MRI-ACL dataset by 3.00%, 1.00%, 2.00%, and 1.00%, respectively. Our system also achieves the best AUC of 92.86% on the MRNet-v1.0 dataset, which exceeds 3D VGG16, ResNet-10, MVCNN, and MRNet by 2.61%, 1.08%, 0.92%, and 0.28%, respectively. The results show that our method outperforms existing single-view and multi-view classification models and can solve the ACL diagnosis task well.

We further conduct ablation experiments on the MRI-ACL dataset and MRNet-v1.0 dataset to evaluate the effectiveness of the presented system with different numbers of views, as shown in Figure 3 and Figure 4, respectively. If we only use one view, the best AUC is 96.00%, which is 1.00% lower than the system based on four views on the MRI-ACL dataset. Meanwhile, the system achieves the best AUC of 91.78% using a single view, which is 1.08% lower than the system using three views on the MRNet-v1.0 dataset. From Figures 3, 4, It can be seen that the AUC improves as the number of views increases. Therefore, we can observe that multiple views contribute positively to ACL diagnosis.

To demonstrate the effectiveness of the fusion approach, we compare two typical fusion approaches, including label fusion (contains a class average method and a class probability weighting method) and feature fusion (i.e., feature concatenation and our presented method), as shown in Figure 5. The label fusion occurs on the prediction results, which are calculated by averaging or weighted average over all the results. Feature fusion happens at the representation generation stage. First, each view generates its own representation through the backbone. Later, these representations are concatenated or weighted concatenated into the embedding space. The concatenated representation is fed into a classifier to obtain the final ACL diagnosis scores. From Figure 5, We can observe three aspects: 1) Compared with the two fusion methods, the result of the feature fusion is better than label fusion; 2) The weighted fusion is better than the direct fusion method; 3) Our method achieves the best results with an AUC of 97.00% (MRI-ACL dataset) and 92.86% (MRNet-v1.0 dataset), respectively.