This retrospective study used a prospective registry of consecutive LVO ischemic stroke patients subjected to MT between April 2016 and January 2023 at La Fe University and Polytechnic Hospital in Valencia. The study protocol was approved by the research ethics committee (CEIm, approval #2021–577-1). Informed consent was obtained from all participating patients or their legal representatives. This study follows the Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research (21).

Vascular neurologists retrieved the demographic and clinical characteristics from the medical history (age, sex, active smoking, and occurrence of diabetes, dyslipidemia, or arterial hypertension). The biological material was cerebral clots retrieved by neurointerventionalists during MT. Depending on the decision of the neurointerventionalist, based on the characteristics of the LVO, different extraction devices were used: (a) distal aspiration catheter with manual syringe, (b) balloon guide catheter plus stent-retriever, or (c) combined stent-retriever with distal aspiration catheter. The retrieved clot was either detached from the extraction device by gentle flushing with saline solution in most cases, or in some cases found in the aspiration syringe, and then preserved in saline solution at 4°C. No distinction was made between the head and tail of the thrombus, or between its shell and core. In the case of fragmented thrombi, the entire set of fragments was collected and treated as a single sample. Digital images of the thrombi were acquired once within 24 h of MT.

All the thrombus images were captured with the same camera (OLYMPUS^® CAMEDIA C-5050, Olympus Optical Co., Ltd., Tokyo, Japan), format (TIF) and resolution (640 × 480 pixels) to avoid internal variability. As the camera captured images containing thrombi and non-relevant information, a segmentation process was necessary to focus exclusively on the thrombi for image analysis. The result of this segmentation is a new binary image called mask that contains the relevant information of the original image (i.e., the thrombus; see Figure 1 for representative images of cardioembolic and atherothrombotic thrombi). The manually segmented masks were obtained using the software ITK-SNAP v3.4.0 (22). This set of images/masks was used as a ground truth to train the segmentation neural network.

Convolutional neural network was the modeling technique selected in order to combine thrombus images and patient features as input for DL categorical predictive fused models.

Choosing the architecture of a neural network is crucial when developing a model in DL. We used one of the most widely used networks in biomedical image segmentation, the U-NET network, first developed by Ronneberger et al. (23). The U-NET consists of two parts: an encoder and a decoder which are interconnected with skip connections. The detailed architecture of the segmentation network can be seen in Figure 2. During the encoder phase, 4 blocks of convolutions layers are used, and in each block, different operations are applied such as convolution layers, batch normalization (24) and dropout (25) followed by pooling layers, effectively capturing the image’s contextual information. Batch normalization layers are used to normalize data during training, and dropout was used after pooling layers to avoid overfitting. In our U-NET, the kernel size utilized was 3×3 pixels, the starting number of filters was 64, and it was doubled in each down sampling step. Subsequently, the decoder comprises up-convolution layers and concatenations of the cropped feature maps extracted from the encoder part. Conversely to the encoder, the number of filters is divided in the “up-convolutions” of the decoder (23). All layers were activated with the ReLU function except the output layer, which was activated with the softmax function.

The classification network adopts an architecture reminiscent of the LeNet model (26). The input to the classification network results from multiplying the original image by the mask generated in the segmentation network. This approach selectively retains only the pertinent information for thrombus classification, discarding non-relevant details and emphasizing the macroscopic structure. The detailed architecture of the classification network can be seen in Figure 3. The network started with 64 filters, each with 3×3 pixel kernels, and this number of filters was doubled after each pooling layer, culminating in a total of 1,024 filters. Dropout and Convolutional Block Attention Module (CBAM) (27) layers were applied following each fixed resolution block to enhance accuracy. Subsequent to the feature extraction layers, a global average pooling layer consolidates all features, yielding 1,024 values per image. These values are then utilized to input a fully connected subnetwork comprising of dense layers with 128, 32 and lastly 1 neuron (all with ReLU activation, exception for the last layer which used sigmoid activation). The model’s output is a number between 0 and 1, and the image was classified as atherothrombotic or cardioembolic according to a set threshold.

Several experiments were conducted to improve the accuracy of the model. The first model exclusively used as input data the thrombus image and the mask. After the creation of this model, the concatenation of the six demographic and clinical characteristics was done. Statistical analysis was performed to find significant differences between atherothrombotic and cardioembolic etiologies regarding patient characteristics. Then, different models were created in order to find the best layer concatenation to the fully connected layer. Once the best model with the best concatenation was obtained, data augmentation techniques such as blurring or sharpening were applied to ameliorate the model’s accuracy.

A k-fold cross-validation strategy was used to account for the rather small sample size. This involves dividing the dataset into k folds. Each fold will have a training set formed by k-1 folds and a test group which contains the remaining one. This process will be iterated k times so all data will be used as a train and test set (28). We used 5-fold cross-validation in both the segmentation and classification networks. As atherothrombotic and cardioembolic classes were unbalanced, a generator function was used to provide balanced samples to the classification network by randomly sampling the corresponding datasets.

The segmentation network was first trained on a subset of original images of the thrombi and their manually generated masks. Then, all the images included in the study were used. The loss function chosen was Dice loss, which employs the DICE coefficient (29). The DICE coefficient measures the overlap between two images. The optimizer utilized was Adam (30), and the number of epochs was established to 500 with 100 steps per epoch.

The classification network was trained with images whose etiology was known (i.e., atherothrombotic or cardioembolic). The loss function chosen in this network was the Binary Cross Entropy because, unlike the segmentation network, the classification operates as a binary classifier. Adamax was the optimizer employed in this network (30). The number of epochs was 1,000 with 100 steps per epoch.

DL algorithms require a substantial amount of training data, yet the disposition of annotated medical images is limited. Data augmentation was used, where some modifications are applied to the original images and improve the network’s performance (31). In our case, the modifications applied were random rotations/flips along the horizontal and vertical axes, and the application of blurring and sharpening, which affect the image edges. This process of data augmentation was applied in both segmentation and classification networks to improve their generalizability and minimize overfitting.

The DICE coefficient was employed to assess the performance of the segmentation network. The mask predicted by the model was generated using Test Time Data Augmentation (TTDA) (32), a technique consisting in predicting the output with different input transformations (horizontal and vertical flip in our case) to generate several predictions to later average them (after inverting the transformation). This technique is a simple way to use auto-assembling that generally improves accuracy and robustness of the network at test time.

The classification network underwent the evaluation using also the TTDA approach. The model’s input was the image multiplied by the mask generated in the segmentation network, and the output was the model’s prediction. The image was classified as either atherothrombotic or cardioembolic depending on the average numerical value returned by TTDA. A threshold was set to determine if images were classified as cardioembolic or atherothrombotic. The accuracy estimated from the training data of the model was used to determine the threshold which was stablished to 0.2. The models were evaluated three times and the results represent their average.

Once the classification predictions were obtained, which is the index test of this study, accuracy, precision, sensitivity, specificity and AUC score were used for the evaluation of the model. We used a bootstrapping approach in order to estimate the 95% confident intervals of our predictions. Bootstrap used 1,000 samples of the test set. In order to calculate these metrics, the reference standard was the TOAST classifications provided by clinicians. Performance accuracy was used to select the best model.

Statistical analysis was performed with the demographic and clinical characteristics obtained from the patients to observe the existence of significant differences between patients who suffered a stroke of atherothrombotic or cardioembolic etiology. The χ² test was used for the categorical variables (sex, smoking, hypertension, dyslipidemia, and diabetes) and the Wilcoxon test for the non-normally distributed continuous variables (age and DICE coefficients). R software¹ in version 4.3 was used. p-values <0.05 were considered statistically significant.

A flowchart of the study is shown in Figure 4. Out of 290 LVO stroke cases subjected to MT, 166 patients were included (age [mean ± SD] 69.11 ± 13.39 years; 67 [40.36%] female). TOAST etiologies were: 87 cardioembolic, 31 atherothrombotic, and 48 cryptogenic. Demographic and clinical characteristics are detailed in Table 1.

In the segmentation network, the images of all 166 patients were used, regardless of stroke etiology. The proposed segmentation network had a total of 1,963,202 parameters (1,959,938 trainable) and obtained a DICE coefficient of 0.95551 ± 0.12996 without TTDA and 0.95553 ± 0.13037 using it. As can be noticed, TTDA improved results (although not statistically significant, p = 0.744, W = 0.310, Wilcoxon test). The average processing time of the segmentation network was 0.5 s per image.

In the classification network, the images of 118 patients with known etiology were used (31 atherothrombotic and 87 cardioembolic). Three consecutive models were developed to optimize accuracy, precision, sensitivity, specificity, and area under the curve (AUC). Cardioembolic etiology was considered the positive class. Table 2 summarizes input data, data augmentation, and performance metrics (mean [95% CI]) for each model.

Model 1 exclusively used the original image multiplied by the mask as input. The accuracy of the model was 0.944, and the precisions were 0.910 and 0.955 for atherothrombotic and cardioembolic etiologies, respectively.

Given the possibility of accessing demographic and clinical data associated with the thrombus image, Model 2 concatenated all the six characteristics (age, sex, active smoking, diabetes, dyslipidemia, and arterial hypertension occurrence) in the dense layer with 128 neurons to the fully connected layer of the classification model. In addition, image sharpening and blurring operations were applied in the data augmentation process. The accuracy of the model increased to 0.958, and the precisions were 1 and 0.946 for atherothrombotic and cardioembolic etiologies, respectively.

Given that the addition of demographic and clinical characteristics improved the accuracy of Model 2, we decided to perform a statistical analysis of these characteristics to find out statistically significant differences between patients who suffered from a stroke of cardioembolic or atherothrombotic etiology (Table 1, characteristics with a p-value <0.05 have been highlighted). In the light of the statistical analysis results, only the 3 significantly different characteristics (age, sex, and active smoking) were included in Model 3, and they were concatenated in the same layer that Model 2. The model increased its accuracy from 0.958 to 0.960. The precisions were 0.964 and 0.959 for atherothrombotic and cardioembolic etiologies, respectively.

Finally, we tested Model 3 with a different TTDA configuration consisting of blurring the images to see its effect not only in the training process, but also in the test. The accuracy of this optimal Model 3^* increased to 0.968, and the precisions for atherothrombotic and cardioembolic etiologies were 1 and 0.959, respectively. AUC for cardioembolic (positive) prediction was 0.947. As can be noticed, the blurring at TTDA improved the accuracy metrics, thus suggesting that color information was more relevant for classification that texture information. The concatenation of the 3 patient characteristics in Model 3^* was performed in the dense layer with 32 neurons. The parameters for this model were 4,874,922 (4,872,938 of them were trainable).

Using Model 3^*, the best performing classification network for atherothrombotic and cardioembolic thrombi, we conducted a preliminary experiment to evaluate the 48 images of thrombi with cryptogenic etiology (together with age, sex, and active smoking data) and predict their classification. Cryptogenic thrombi were evaluated by the 5 folds of the model and the final result was the vote of the majority of the folds. The network classified 46 images as cardioembolic and 2 images as atherothrombotic (Figure 5). This suggests that cryptogenic thrombi are more similar to cardioembolic thrombi (95.83%) than atherothrombotic thrombi (4.17%), which is in good agreement with the higher incidence of cardioembolic etiology. In 36 out of 46 cases (78.26%) the prediction of cardioembolic origin was unanimous. After further diagnostic workup, 5 out of the 36 unanimous predictions were clinically confirmed as cardioembolic etiology (Figure 5).