This study was conducted in accordance with the Declaration of Helsinki. This study was reviewed and approved by the Institutional Review Board of Jinshan Hospital (JIEC 2023-S85). Written informed consent was obtained from all participants prior to enrollment for publication of any potentially identifiable data or images.

This prospective study aimed to assess the effectiveness of DL model in classifying AK and SK. Participants with histopathologically confirmed AK or SK were included. The datasets included: a cohort of 2,307 patients from the international skin imaging collaboration dataset (ISIC, https://www.isic-archive.com), a cohort of 386 patients from Center A, a cohort of 196 patients from Center B and Center C, and a cohort of 215 patients from DermNet (https://dermnetnz.org/images).

From November 1, 2023, to April 1, 2024, adult patients undergoing surgical resection for skin neoplasm were enrolled from three centers (Center A, Center B and Center C). The inclusion criteria were as follows: (1) Histopathologically confirmed AK or SK; (2) Age ≥ 18 years. Exclusion criteria included: (1) Presence of systemic infection; (2) Incomplete clinical data; (3) Images with motion blur or artifacts.

The data from ISIC with histopathologically confirmed AK and SK were used as a training cohort for training the DL model. The data from Center A was used as a validation dataset1; the data from Center B and Center C were combined and used as a validation dataset2; the data from DermNet (histopathologically confirmed clinical photographs) was used as a validation dataset3. These dataset were used for validating the DL model.

Image preprocessing steps are consistent with what we previously reported (12). Briefly, the images were captured by dermoscopy or devices with a minimum camera resolution of 12 megapixels. Adequate natural daylight or bright artificial light was used for clear visibility of the skin lesions. All images were resized to a standard size suitable for the input layer of DL models and converted into tensor format. The preprocessing stage included data augmentation operations such as random cropping, rotation, flipping, and color transformations. The normalization process was performed by subtracting the mean value of the entire dataset and dividing by the standard deviation to normalize pixel values to a standard range.

The DL model was based on a contrastive language-image pre-training (CLIP) model with ViT-B/16 architecture (14, 15). Images were first divided into a set of fixed-size patches, each patch was then flattened and embedded into a vector. These vectors were then passed as input to the Transformer encoder to model the image globally. Finally, these representations went through several fully connected layers to produce the final classification or regression outputs. To benchmark CLIP-ViT against widely used CNNs like ResNet or EfficientNet, we also conducted a head-to-head evaluations of CLIP-ViT and conventional CNNs (ResNet-50 fine-tuned on the same data sets) to clarify transformer-based architectures’ advantages.

The input images were divided into fixed-size image patches, and each image patch was mapped to a low-dimensional space through a learnable linear projection, forming an embedding vector for the image patch. Position encodings were added to the embedding vectors of each image patch to represent the relative positional relationships between the image patches. A Transformer encoder was then employed to encode the sequence of embedding vectors, which included self-attention mechanisms and fully connected feed-forward networks to capture semantic information and contextual relationships between the image patches. A fixed-length vector representation was obtained through feature pooling, which was then projected through a fully connected layer to the same embedding space as the text features.

The model was trained using the stochastic gradient descent (SGD) optimizer with momentum. The initial learning rate, momentum, and weight decay were set to 0.002, 0.9, and 0.005, respectively. We adopt the ViT-B/16 variant of the CLIP model and set the number of the learnable prompt vectors to 16. The number of training epochs is set to 100. The input images of the dataset were all resized to 224 × 224 pixels. To ensure reproducibility, we set the random seed to 0 and used a batch size of 32 to maintain training efficiency.

Cross-entropy loss was utilized to measure the disparity between the predicted results and the true labels. The outputs of the model were first processed through a softmax function to transform them into a probability distribution. Then, these probabilities were compared with the true labels to compute the cross-entropy loss between the model’s predictions and the true labels. For each sample, the cross-entropy loss was the negative log-likelihood of the predicted label at the corresponding position (16).

Our server was equipped with two NVIDIA RTX 4090 GPUs, each with 24GB of memory. The central processing unit (CPU) was an Intel Core i9-13900K, featuring 24 cores and 36 threads. 256 GB of DDR4 RAM was used for memory. Data storage was managed with a 1 TB SSD, enabling fast data read and write operations. The operating system was Ubuntu 20.04, and the software environment included tools and libraries such as PyTorch 1.10, CUDA 11.2, Anaconda 23.3.1, CUDA 12.0, cuDNN v8.8.1, PyTorch 1.13.1, and Python 3.7.16. For the public ISIC dataset, the total training time for 100 epochs was approximately 1.1 h, while the average inference time per instance was 20.50 milliseconds. During the inference phase, the average inference time per instance was 2.56 milliseconds, with a throughput of 390 samples per second.

Attention mechanisms were employed to visualize the model’s regions of interest during image recognition. Specifically, attention maps were generated to illustrate the model’s focus on different regions of the images. The attention maps revealed the image features that the model primarily relied on for predictions, such as color, shape, and texture. Grad-CAM was utilized to highlight the regions in the images considered by the model as crucial for predicting the corresponding labels. The area under the receiver operating characteristic (ROC) curves (AUC) were used to evaluate the clinical application of the DL model in assisting dermatologists.

First, two dermatologists (dermatologist 1 with 15 years and dermatologist 2 with 3 years of experience, both blinded to histopathological data) reviewed the images to identify AK or SK. Second, each dermatologist was shown the classification result of the DL model. The dermatologists were allowed to reclassify the diagnosis if needed according to the DL results. Any changes of the dermatologist in reclassification were recorded. Net reclassification index (NRI) and total integrated discrimination index (IDI) were calculated to compare the discrimination performances of the dermatologists before and after considering the DL model’s results (17).

Statistical analysis was performed using R software (version 4.3.2; https://www.r-project.org/). Data normality and homogeneity of variance were assessed using appropriate tests. For continuous variables, independent-samples t-tests (met the assumptions of normality) or non-parametric Mann–Whitney U test (not met the assumptions of normality) was performed. Categorical variables were compared using the chi-squared test or Fisher’s exact test. A p-value less than 0.05 was considered statistically significant.

The training cohort (ISIC dataset) included a total of 2,307 patients (1,004 females and 1,303 males, aged 64 ± 13, ranged from 20 to 85), with 1,348 diagnosed with SK and 959 diagnosed with AK. The validation dataset1 included a total of 386 patients (211 females and 175 males, aged 60 ± 15, ranged from 21 to 95). The validation dataset2 included a total of 195 patients (138 females and 57 males, aged 59 ± 16, ranged from 23 to 91). The workflow of this study is shown in Figure 1. The clinical characteristics of patients in training and validation cohorts are shown in Table 1. Two case examples of AK and SK is shown in Figure 2.

The architecture of the DL model is shown in Supplementary Figure 1. For the training cohort (ISIC dataset), the model demonstrated an AUC of 0.85 with sensitivity of 0.87 and a specificity of 0.84, with PPV and NPV of 0.79 and 0.90, respectively.

For the validation cohort1 (Center A), the initial performance metrics of the model in the training cohort revealed an AUC of 0.89, indicating a moderate discrimination ability. The model demonstrated a sensitivity of 0.84 and a specificity of 0.94, with positive predictive value (PPV) and negative predictive value (NPV) of 0.86 and 0.93, respectively.

For the validation cohort2 (Center B and C), the model’s accuracy and discrimination capabilities were further affirmed, with an AUC of 0.84 showcasing an excellent ability to differentiate between AK and SK. The model achieved a sensitivity of 0.71 and a specificity of 0.98. The PPV and NPV were noted at 0.93 and 0.89, respectively (Table 2).

For the validation cohort3 (DermNet with 165 AK and 50 SK), the model achieved an AUC of 0.89 in differentiating AK from SK with a sensitivity of 0.81 and a specificity of 0.98. The PPV and NPV were of 0.99 and 0.60, respectively (Table 2). The comparison of the performance of resnet50 fine-tuned and CLIP on the same data sets is shown in Supplementary Table 1.

Without the DL model’s assistance, dermatologist 1 achieved AUCs of 0.77 and 0.69 in diagnosing SK from AK with SEN, SPE, PPV and NPV of 0.66, 0.89, 0.71, and 0.86 and 0.47, 0.92, 0.71, and 0.80 for the validation cohort1 and 2. Dermatologist 2 achieved AUCs of 0.79 and 0.77 in diagnosing SK from AK with SEN, SPE, PPV and NPV of 0.77, 0.82, 0.64, and 0.90 and 0.86 and 0.79, 0.74, 0.57, and 0.89 for the validation cohort1 and 2 (Figure 3).

After giving the predict results of the model, the dermatologist 1 achieved AUCs of 0.80 and 0.80 in diagnosing SK from AK with SEN, SPE, PPV and NPV of 0.67, 0.93, 0.81, and 0.87 and 0.60, 0.97, 0.90 and 0.85 for the validation cohort1 and 2. The dermatologist 2 achieved AUCs of 0.89 and 0.87 in diagnosing SK from AK with SEN, SPE, PPV and NPV of 0.89, 0.88, 0.76, and 0.95 and 0.78, 0.96, 0.90, and0.91 for the validation cohort1 and 2.

The categorical NRI was 0.10 (p = 0.006) and 0.19 (p < 0.001) for dermatologist 1 and dermatologist 2, indicating a significant improvement with the DL model’s assistance. The IDI was 0.14 (p < 0.001) and 0.27 (p < 0.001), confirming statistically significant betterment in discrimination between AK and SK with 14 and 27% improvement for dermatologist 1 and dermatologist 2, respectively (data from a merged data set of Center A–C, Figure 4).

The results showed that when correctly predicted, the model often relied on the color and shape features of the target objects. For both AK and SK the DL model focus on the background color of the images. Additionally, features of objects within the image such as papules or patches on the surface were also factors considered by the model. Furthermore, the smoothness of the skin surface may also be a factor considered by the model. The surfaces of SK were smooth, whereas that of AK were rough (Supplementary Figure 2).

We found that when predictions were incorrect, model failures fell into the following categories: Instances of color blending, Impact of blurry images, Interference from similar colors, and specific features of small sample sizes (Supplementary Figure 3).