The LUS datasets used were fully anonymised and were collected at the Royal Melbourne Hospital. The study was approved by the Melbourne Health Human Research Ethics Committee (HREC/18/MH/269) (27). The US dataset comprised 125 patients for a total of 1,034 LUS clips. At least six unique lung scanning zones were evaluated and labelled (Figure 1) by our clinical experts (DC and XC) following the protocol shown in Figure 2. The initial dataset included clips from 54 unidentified patients, 27 healthy, and 27 with IS labelled as “non-healthy.” The total number of LUS clips included were 108, comprising 16,962 LUS frames (8,481 frames each for healthy and IS patients) (Figure 3). Two LUS examples of IS and healthy frames are demonstrated in Figures 4A, B, respectively.

The filtering techniques applied in this study are as follows: Scenario 1 involves the thorough inclusion of all LUS frames from all clips in the training datasets (Dataset 1). In contrast, Scenarios 2 and 3 utilise a selective filtering technique to refine the training dataset (Dataset 2) by excluding LUS frames that do not exhibit the main IS features (i.e., absence of B-lines), thereby prioritising clinically relevant features. The two datasets are illustrated in Figure 3. Based on predefined clinical criteria, all LUS clips were labelled as healthy or non-healthy (IS cases) (Figure 5). No other lung pathologies (e.g., pleural effusion, consolidation, or atelectasis) were included in the dataset. These criteria were adapted from international evidence-based recommendations for point-of-care ultrasound (POCUS) (28). It is crucial to note that, in the clip-based labelling method used in this study, a clip classified as IS may include individual frames that do not exhibit IS features and could be deemed healthy. This observation highlights the natural variation and complexity of LUS, emphasising that not all frames from IS clips will consistently show the exact features associated with IS.

The LUS dataset was divided into three subsets for effective model development and performance assessment: training (70% ≈ 76 LUS clips), validation (15% ≈ 16 LUS clips), and test sets (15% ≈ 16 LUS clips). Three different approaches were followed, referred to as Scenarios 1, 2, and 3, with distinct pre-processing techniques and pre-trained models. In Scenario 1, the Xception pre-trained model was employed (29). All LUS frames were included in the training and validation process, incorporating the total number of healthy and non-healthy frames from all clips (14,902 frames, referred to as Dataset 1). Scenario 2 entailed the utilisation of Xception and Inceptionresnetv2 pre-trained models. A more selective strategy was adopted in Scenario 2 (10,546 frames; referred to as Dataset 2), where non-healthy frames were re-evaluated, and any frames exhibiting characteristics not conducive to the main features characterising IS (i.e., absence of B-lines) were excluded. This selection criterion was applied to ensure that only relevant features were included in the training dataset. In Scenario 3, an Xception model without pre-existing features was trained (non-pre-trained model). It learnt features and weights exclusively from the LUS data without any TL from general image knowledge. Furthermore, similar to Scenario 2, data filtering was applied to the non-pre-trained model, with an evaluation of only non-healthy frames and the exclusion of any frames exhibiting characteristics not conducive to the main features characterising IS (absence of B-lines) within the training dataset. The pre-trained models, Xception and InceptionResNetV2, were selected for their proficiency in medical image classification gained through training on the ImageNet dataset with over 14 million natural images across more than 20,000 classes (29, 30).

To adapt the DL models to our IS detection task, with two classes, IS and healthy class, the architectures of all models (Xception, Inceptionresnetv2, and non-pre-trained model) were customised. The top layer of the models, known as the classifier, which was originally designed to classify 1,000 different classes (such as animals or household items), was replaced with a two-class classifier. The LUS images were also downsized from 720 × 920 pixels to 299 × 299 pixels to align with the input dimensions specified by the models' architectures. The resizing was performed using bilinear interpolation in MATLAB software (Version R2023b). The aspect ratio of the LUS images was preserved during resizing to avoid distortion of anatomical features.

The Xception model has about 170 layers and 22.9 million trainable parameters. It uses depth-wise separable convolutions across 14 modules to improve feature extraction. The InceptionResnetV2 model, with a more complex structure, includes 843 layers and 55.8 million trainable parameters, combining the Inception and ResNet architectures. As the non-pre-trained model, we employed a modified version of the Xception model that was devoid of its pre-trained weights. The MATLAB software was used to run the models and monitor the training and testing process. The models were trained on a graphics processing unit with an NVIDIA TITAN RTX and 25 GB RAM, running Ubuntu 20.04.6 LTS. The Adam optimiser was used during training. The configuration of training code and model customisation steps was guided by the available code from the GitHub repository by Alzubaidi et al. (31). After training, the models were tested on the test subset, and their performance was evaluated using multiple performance metrics. Figure 6 illustrates the workflow followed to train and test the models, which includes data processing, model customisation, and model performance evaluation tools. In addition, Table 1 outlines the model hyperparameters.

The Grad-CAM visualisation technique was used in our model evaluation to enhance the explainability of model predictions (26). Grad-CAM provides a visual explanation in the form of a heatmap overlay on the image, highlighting the region of interest (ROI) in the output image (26), which refers to a specific area within an LUS image used to identify particular pathologies or diseases. For instance, in cases of non-healthy frames (IS) (Figure 7. Grad-CAM), the ROI may be defined as an area containing B-lines. This technique generates heatmaps, overlaying the original image to highlight areas influencing the model's decision and aiding in identifying related features. In addition, LIME was used to explain predictions by estimating the decision boundary in a specific input image, focusing on the intended ROI in the LUS image, and generating a heatmap scale (Figure 7, LIME).

LIME highlights influential regions that contribute to a specific prediction, aiding understanding of the model's decision-making process (24). It approximates the boundary that defines the ROI by creating a new scale for a heatmap. This scale highlights the regions of the input image that have the most impact on the model's prediction. LIME divides the image into identifiable portions and evaluates the impact of each part of the LUS image. In Figure 7, the LIME visualisations show the most important features, which are represented by scores and colours. Higher scores, indicated by more intense colours (from blue to red), correspond to features that contributed more significantly to the model's decision.

A smaller, representative subset of frames was randomly selected from the test set (2,060 LUS frames) for evaluation. Specifically, 20% of the total dataset (412 out of 2,060) was sampled by selecting one frame every five frames using a simple MATLAB script. Grad-CAM and LIME were generated to analyse which areas of the images the model focused on, producing corresponding cropped images for each frame. Each sampled frame was manually reviewed to determine whether the Grad-CAM or the LIME visualisations correctly highlighted the intended ROI. For IS frames, correct localisation was defined as “accurate emphasis on B-lines or pleural irregularities”; for healthy frames, correct localisation was defined as “emphasis on normal pleural and A-line features.” Grad-CAM and LIME accuracies were then computed as the proportion of frames correctly localised with respect to the expert-defined ROI.

Along with Grad-CAM and LIME plots, the confidence score was used. The confidence value, or the probability score, quantifies the model's level of confidence in its predictions (25). Higher probabilities or confidence values generally indicate higher confidence in the model (i.e., a confidence score of 100% indicates that the model has absolute certainty in its prediction). In comparison, lower probabilities suggest lower confidence in the model (25). A confidence score of 50% means that the model is equally likely to be correct or incorrect. All developed models were tested on the unseen test dataset to generate Grad-CAM and LIME plots, along with the corresponding confidence scores (Figure 7).

Nevertheless, a high level of confidence does not guarantee the accuracy of the prognosis. The model's indication solely reflects its confidence level derived from the knowledge acquired during training. The model's confidence may be significantly high. Yet, it can produce an inaccurate clinical diagnosis, particularly if it has been trained on biased data or encounters data that significantly deviate from its training set (32).

The feature fusion process in artificial intelligence (AI) combines information from multiple AI models trained on the same dataset using different ML classifiers (33). This strategy is a powerful technique employed to enhance overall performance by incorporating features from different DL models. Its objective is to acquire and merge additional knowledge from multiple models in order to improve the representation of the features extracted from them (34). During the learning stage, the initial layers of each DL model acquire low-level features such as colours, edges, and forms, while the last layers acquire the high-level features of an object. Consequently, the model's final output features result from this hierarchical learning process, in which complex, high-level features are built upon more fundamental ones. Features are extracted from the bottleneck layers, which are the layers prior to the output layer. These layers are rich in complex features that have been analysed through the network and are considered highly informative for the classification task (35). Feature fusion is then utilised, where features from the bottleneck layers learnt from different models are combined or “fused.”

After the feature extraction phase, the extracted features undergo a process of normalisation to ensure that they are on a comparable scale, followed by concatenation to form a unified feature vector for each image. This combination offers an improved depiction of the features and enables a more thorough representation of the underlying patterns and features in the data. These fused features are then used as inputs to train a machine learning (ML) classifier, which adjusts its parameters by comparing predictions to ground truth (GT) labels to minimise prediction errors. This method enables ML classifiers to leverage the capabilities and distinctive attributes of each DL model, thereby improving understanding of the target tasks (36). The integration of features from different models provides numerous benefits for ML classifiers. In the present study, the built-in Classification Learner in MATLAB 2023b was utilised to develop ML classifiers, which include linear discriminant analysis, neural networks, coarse KNN, cubic SVM, the boosted tree, and the coarse tree, to determine the most efficient classifier for this detection task.

The study utilised multiple models for this task, beginning with a comprehensive fusion (F1) involving all models mentioned in Scenarios 2 and 3, namely Xception, InceptionResnetV2, and the non-pre-trained model. A feature fusion (F2) was also performed with the two best models identified in Scenario 2. Furthermore, two separate feature fusion processes were performed: one between the non-pre-trained model and Xception (F3) and another between the non-pre-trained model and InceptionResnetV2 (F4), each done individually. Each feature fusion process generated fused features, which were then used as input to the ML classifiers for training and performance evaluation, as shown in Figure 8. The resulting classifiers are named C1, C2, C3, and C4.

To assess the robustness of our developed models for IS detection, a comparative analysis was conducted against clinical diagnoses made on individual clips. A randomly selected test set of 16 video clips representing our dataset was assessed. The GT labels were established during data acquisition by two senior clinicians (DC and XC) and ensured expert consensus. Each LUS clip was reviewed and labelled as healthy or interstitial syndrome (IS) according to predefined diagnostic criteria described earlier (Section 2.3, Figure 5). These LUS labels were documented and served as the GT for model training and evaluation. A blinded review was further conducted by two additional experts (MS and CE), each with 15 years of experience and holding senior positions at the Queensland University of Technology. They independently labelled the clips as 0 (healthy) or 1 (IS). Their diagnoses served as a reference point for evaluating the performance of our pre-trained models across three scenarios, allowing a direct comparison with human expert evaluations. Each clip within our test set was assigned a distinct numerical identifier. These identifiers and the corresponding ground-truth (GT) labels were documented in an Excel spreadsheet for recording outputs and analysis. To ensure the integrity of our diagnostic assessment, each expert conducted their evaluations independently.

The algorithm performance was assessed on each LUS clip to simulate clinical evaluation practices, similar to how clinicians assess entire videos during diagnosis. For algorithmic assessment, the conversion from frames to video clip analysis adopts a Simple Majority Voting scheme (SVE) to aggregate individual frame predictions into a singular diagnosis for each video clip (37). This transition to video clip analysis compiles predictions from individual frames into a single diagnosis for each video clip. The class with the highest number of predictions is taken as the output prediction for the entire clip. To qualify as a single-video diagnosis, the model must identify healthy or IS frames that comprise more than 50% of the video's total frames, ensuring that they represent a significant portion of the video frames. Upon completion of these assessments, the diagnostic results from each expert were cross-referenced with the GT labels. This comparative analysis enabled us to determine the accuracy of each expert's predictions by identifying correct and false predictions. All developed DL models were evaluated and compared with our clinical experts, using accuracy, sensitivity, and specificity as performance metrics. In addition, a receiver operating characteristic (ROC) curve, which is created by plotting the true positive rate (TPR) against the false positive rate (FPR), was used in the comparative analysis to discern the strengths and limitations of DL models in IS detection in comparison with our medical experts.

To assess training stability of the DL models, learning curves for training and validation accuracies were analysed across all models and scenarios (Figure 9). The results showed a consistent upward trend in training accuracy with increasing epochs, confirming effective learning progression. Models in Scenario 2 (Xception and InceptionResNetV2) converged rapidly within the first few epochs and achieved stable performance by the final epoch, reaching approximately 99% and 95% rates for training and validation accuracies, respectively. In comparison, the models in Scenarios 1 and 3 exhibited a slower convergence pattern and slightly lower final accuracy rates, plateauing near 90%. The minimal gap between the training and the validation curves demonstrates stable learning behaviour and suggests limited overfitting.

The performance of the trained models across the various scenarios was evaluated on the test set using accuracy, precision, recall, and F-1 score. The test set consisted of 2,060 LUS frames extracted from 16 LUS clips (eight healthy and eight non-healthy, with two clips per patient). The GT label used to assess the performance of the DL models is the label corresponding to the whole video. This means that a single label representing the overall LUS video classification is assigned to each video frame, against which the model's predictions for individual frames are evaluated. Table 2 shows that both models of Scenario 2 (i.e., pre-trained models with filtered training data) outperformed those of Scenario 1 and 3 in terms of accuracy, precision, recall, and F1 score. The Xception model in Scenario 2 achieved a higher accuracy rate (95.9%) and higher precision and recall rate (95.8%) than the model in Scenario 1. It also had a higher F1 score of 96.0%. On the other hand, the InceptionResnetV2 model in Scenario 2 achieved an accuracy rate of 95.73% and a specificity, precision, recall, and F-1 score of 95.7%. Lastly, the non-pre-trained model in Scenario 3 achieved a specificity and precision rate of 90.6%, a recall rate of 90.4%, and an F1 score of 90.5%.

A subset of the testing dataset was selected, consisting of 412 frames: 189 healthy (H) frames and 223 non-healthy (IS) frames. A randomly selected subset was used to evaluate the model's performance along with the accuracy of its corresponding Grad-CAM and LIME visualisations, as shown in Figure 11 (with additional details in Supplementary Appendix B). The accuracy of Grad-CAM and LIME was manually assessed by examining how well the highlighted regions aligned with the expected ROI in both IS and healthy cases. For non-healthy cases, the assessment focused on whether the visualisations correctly highlighted existing pathological features (e.g., B-lines). For healthy cases, the evaluation examined whether the visualisation instead emphasised normal lung features, such as A-lines, while avoiding false highlighting of non-existent pathologies such as the background.

Figures 11, 12 summarise model performance and Grad-CAM visualisations for healthy and SIS LUS frames. In the heatmaps, warmer colours (red/orange) represent areas of stronger model activation, whereas cooler colours (green/blue) denote weaker activation. The red box marks the predefined ROI used for evaluation. In healthy examples, the apparent shift in the ROI relative to SIS frames reflects the model's attention to normal pleural lines or A-lines rather than to vertical B-lines.

The Xception model in Scenario 1 achieved an overall accuracy rate of 84.71%, correctly classifying 79.89% of healthy frames and 88.79% of non-healthy frames. Grad-CAM, as shown in Figure 12, highlighted relevant areas in 83.44% of healthy and 73.74% of SIS frames, while LIME, as shown in Figure 13, provided a better accuracy rate for healthy frames (100%) but only a rate of 57.07% for SIS frames. In Scenario 2, Xception performed better with a 95.39% accuracy rate, but Grad-CAM, as shown in Figure 12, failed to identify ROIs in healthy frames (0%), while excelling in SIS frames (100%). As shown in Figure 12, LIME demonstrated more consistency, with a 98.31% accuracy rate for healthy and 100% rate for SIS frames. The InceptionResNetV2 model in Scenario 2 achieved the highest accuracy rate (95.96%), effectively identifying healthy (95.24%) and SIS (95.96%) frames. Grad-CAM, as shown in Figure 12, performed well for healthy frames (92.22%) but failed for SIS (0%), while LIME, as shown in Figure 13, correctly identified all frames (100%). Scenario 3 (non-pre-trained) reached an 89.56% accuracy rate with Grad-CAM, as shown in Figure 12, inconsistently localising healthy frames (59.76%) but excelling in SIS frames (100%). In contrast, as shown in Figure 13, LIME remained highly reliable for both classes (98.22% and 100%). More details can be found in Supplementary Appendix B.

All the false predictions of the Xception model in Scenario 2 were re-evaluated because of its superior performance in terms of accuracy among the four models evaluated in Scenario 1, Scenario 2, and Scenario 3. This thorough evaluation aimed not only to capture errors in the model but also to rigorously assess whether the identified predictions were truly false, thereby enhancing our understanding of the model's diagnostic reliability. A total of 84 frames were reviewed by our expert (MS), including 33 false positives and 51 false negatives identified by the model.

For the false positive frames (33 out of 1,115 = 2.96%) where the Xception model in Scenario 2 predicted “healthy,” the expert re-evaluated 33 frames. The clinical expert categorised these frames into three classes, as shown in Figure 14. In the first class, 22 of the 33 frames (77.67%) emerged. Our clinicians classified the frames as healthy because of either the absence of B-lines or limited visibility, which matches the Xception model's prediction in Scenario 2. Only 1 of the 33 frames (3.03%) exhibited potential B-lines in the second class. In the third class, 10 of the 33 frames (30.3%) were classified as non-diagnostic or marked with limited visibility because of the shadowing caused by ribs.

In the case of false negatives, where the model incorrectly identified frames as “IS,” the expert re-evaluated 51 frames (51 out of 945 frames = 5.4%). For the first class, the clinicians determined 11 of the 51 frames (21.57%) to be IS, indicating that the model's predictions were incorrect. The remaining 40 of the 51 frames (78.43%) were considered healthy, as there was no evidence of B-lines. Figure 15 shows examples along with each of the two classes.

Upon re-evaluating the false prediction frames (84 frames) of the Xception model in Scenario 2, the model correctly predicted false positives, with a 22% (out of 33) success rate. However, the model correctly predicted only 11 out of 51 false negatives. This demonstrates the importance of analysing videos at the frame level to better understand how the algorithm performs. Also, not all frames in a video labelled as IS necessarily show IS features, which should be considered in the evaluation.

Multiple ML classifiers were trained using features extracted from the three models (second and third scenarios) previously mentioned.

All developed AI and ML models were evaluated in terms of true and false positives and negatives, as shown in Figure 17, following the methodology described in Section 2.6. The GT label used to assess the performance of the DL models compared with our experts is the label that matches the entire LUS clip label. Our first expert (MS) identified 75% (12 clips) of the labelled cases within the sample test subset (16 clips) as correct predictions and 25% (four clips) as false predictions of the labelled cases. In contrast, our second expert (CE) identified 88% (14 clips) of the labelled cases within the sample test subset (16 clips) as correct predictions and 12% (two clips) as false predictions of the labelled cases. Among all developed models, both Scenario 2 and fusion models F1, F2, F3, and F4 predicted 100% of the LUS clips with the best accuracy rates. They identified 100% (16 clips) of the labelled cases within the sample test subset (16 clips) as correct predictions with no false predictions. This gives these models an F1 score of 100%, a precision rate of 100%, and a recall rate of 100%.

The DL models in Scenario 1 and Scenario 3 identified 88% (14 clips) of the labelled cases within the sample test subset (16 clips) as correct predictions and two as false predictions, as shown in Figure 18. Upon reviewing the two LUS clips in Figure 18, it was found that both experts faced challenges in identifying IS cases, and these have been highlighted. In Clip (a), experts classified the clip as a solid organ with peripheral B-lines but did not confirm the presence of three B-lines. Clip (b) also shows a mixed pattern with scattered B-lines, complicating the verification of the presence of three B-lines. Both cases show subjectivity in B-line interpretation and quantification, as experts struggled to distinguish artefacts from true pathology. Figure 19 shows a spreadsheet capturing the evaluation process and outcomes of LUS clip predictions performed by our experts and all developed models on the set of 16 clips. Each column under the experts and models represents their predictions for the clips, with “1” indicating an IS prediction and “0” representing a healthy clip. The cells highlighted in pink mark the instances where a false prediction was recorded by the corresponding expert or developed model. Overall, both ML classifiers (F1 and F2) and models in Scenario 2 outperformed our expert performance and other developed AI and ML models in terms of accuracy, precision, recall, and F1 score. A more detailed display of the performance of AI and ML models is provided in Table 4.

Furthermore, Figure 20 illustrates the ROC curve, which shows the performance of our experts and developed models via TPR and FPR. The ROC curve shows that the fused F1, F2, F3, F4, and Scenario 2 models significantly outperform our experts and other developed models. However, it is important to note that this evaluation is based on a small subset of videos.