The CLE images used in this study were collected during clinical studies conducted at Barrow Neurological Institute and approved by the Institutional Review Board of Human Research at St. Joseph's Hospital and Medical Center (16, 17). We used AI techniques to analyze CLE images acquired intraoperatively from patients who underwent brain tumor surgery at Barrow Neurological Institute, St. Joseph's Hospital and Medical Center, Phoenix, Arizona, between 2010 and 2023. All participants provided written informed consent prior to their participation.

Only CLE images from nonenhancing brain mass lesions identified on preoperative MRI, along with confirmed histopathology and assigned WHO CNS grade, were included in the analysis. Images were acquired using 1 of 2 CLE systems: a clinical system (CONVIVO, Carl Zeiss Meditec AG, Jena, Germany) or a preclinical system (Five1, Optiscan Imaging Ltd., Mulgrave, Australia). Both systems use similar technology developed by Optiscan Imaging Ltd. and generate comparable image quality, differing mainly in the user interface, image dimensions, and functional features. For all cases, FNa was administered intravenously at a dose of 5 mg/kg within minutes before CLE image acquisition. CLE images with H&E-stained histologic sections from the same imaging spot were collected for analysis.

All CLE images were reviewed by a single neuropathologist (J.M.E.) with 15 years of experience and expertise in interpreting CLE images. Each CLE region of interest (ROI) was evaluated, and an overall high- or low-grade interpretation was assigned to the entire sequence. The neuropathologist remained blinded to the final pathological diagnosis during the evaluation. Atypical features (e.g., hypercellularity, cellular pleomorphism, necrosis, and microvascular proliferation) were documented and used as indicators of higher tumor grade and malignancy. In contrast, normal or mildly increased cellularity without evident cellular pleomorphism suggested lower-grade tumors. CLE interpretations were subsequently compared with the corresponding histopathological diagnosis and H&E-stained sections, which served as the ground truth for the neuropathologist's interpretation and the AI model's classification.

After manually excluding CLE image sequences entirely affected by blood or motion artifacts, the remaining images underwent standardized preprocessing, which consisted of 2 main steps: (1) normalizing image dimensions to the [0,1] range to ensure numerical stability, maintain dataset consistency, and prevent gradient explosion during the training process; and (2) downsizing the image to meet the input requirements of the vision backbone architectures.

Two approaches were employed for the AI-based interpretation of CLE images: a conventional single-frame approach (14, 15, 18) and a newly proposed sequence-based approach. A new neural network architecture composed of multiple extraction components was developed for the sequence-based approach (Figure 1).

We randomly divided the dataset of 105 CLE image sequences into a training set (78 sequences, 74.3%) and a testing set (27 sequences, 25.7%). The neuropathologist's interpretation of the matching H&E section as high- or low-grade tumor was used as the ground truth label. To ensure robustness and minimize bias from randomization, we conducted experiments on 3 different random splits. In the training stage, we used the cross-entropy loss function to compute classification loss by comparing the predicted high and low tumor grades with the ground truth label. Model parameters were optimized using the adaptive moment estimator optimizer, which facilitated efficient backpropagation and convergence. During testing, the model processed CLE image sequences following the same procedure as in the training stage. Performance was evaluated using top-1 accuracy, calculated on a per-examination basis and reported for each test set in addition to sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV). All experiments were conducted using a desktop NVIDIA GeForce RTX 3090 graphics processing unit (GPU) (24 GB memory).

In total, 105 CLE image sequences containing 3,173 CLE images from 40 ROIs across 16 cases were included in the analysis (Table 1). These comprised 2 astrocytomas (12%), 10 oligodendrogliomas (63%), 1 glioblastoma (6%), 2 glioneuronal and neuronal tumors (12%), and 1 miscellaneous mass lesion (6%). Eight (50%) were pathology-confirmed low-grade (WHO CNS grade 1 and 2) lesions, and 8 (50%) were pathology-confirmed high-grade (WHO CNS grade 3 and 4) lesions.

The neuropathologist successfully interpreted CLE images from 38 of 40 ROIs. Concurrent hypercellularity and pleomorphism were observed in 23 ROIs, with 18 classified as high-grade based on histology; 15 ROIs did not exhibit hypercellularity or pleomorphism, all corresponding to low-grade pathology (Figure 2A). The overall accuracy of the neuropathologist's interpretation was 87%.

To evaluate the effectiveness and clinical relevance of our proposed sequence-based CLE image classification framework, we conducted a thorough assessment focusing on 2 main aspects: (1) the influence of the 4 visual backbone models, and (2) the classification accuracy of our sequence-based method compared to traditional frame-based image classification.

The 4 visual backbone models process images in different ways, as evidenced by the varying high-activation regions (Figures 3–5). For Inception-ResNet-V2, ResNet50, and VGG16, the neural networks appeared to be activated to different degrees by the areas of high cellular density (Figures 3C–E, 4C–E, 5C–E), whereas the specific activation patterns did not always align with the regions that human experts focus on. Therefore, such AI model applications offer valuable dimensions to image analysis that can identify, concentrate on, and interpret features that might be overlooked by human interpretation or involve subtleties with significant implications.

VGG16 is a classic deep learning model composed of many simple layers that use small convolutional filters to extract detailed visual features progressively. Its straightforward and uniform architecture allows the model to focus on local textures and visually distinct patterns, which is advantageous for analyzing CLE images in which dense cellular regions are often key indicators. In our study, VGG16 achieved an accuracy of 88% in the sequence-based classification task, closely following vision transformer. Although its reliance on localized features can be a limitation in more subtle or heterogeneous cases, the activation maps reveal strong and focused attention on hypercellular areas, which aligns with the model's tendency to respond to clear localized signals.

ResNet50 introduces shortcut or “residual” connections that allow the model to preserve information across layers, enabling it to learn more abstract patterns. Despite this innovative design, ResNet50 achieved a relatively low accuracy (73%) in our study. Although it excels at detecting obvious features, it is less effective at integrating global context, which is crucial in interpreting CLE images with diffuse or ambiguous findings. In activation maps, ResNet50's narrower focus was associated with misclassification errors, suggesting that it missed more subtle contextual cues, corresponding to prominent histoarchitectural features, in the overall frame (Figures 3D, 4D, 5D).

Inception-ResNet-V2 combines 2 advanced design concepts. The Inception module allows the model to analyze features at multiple scales simultaneously, and the residual connections help preserve information as it passes through many layers. This architecture can be overly sensitive to complex textures and subtle variations. The activation maps demonstrated excessive attention to what appears to resemble hypercellularity, whereas other parts of the image that showed low-grade features were ignored, indicating that the model tended to interpret benign structural complexity as malignancy (Figures 3C, 4C, 5C). This led to the obvious tendency to overclassify low-grade pathology as high-grade pathology, indicated by the confusion matrix (Figure 2B).

Vision transformer divides an image into small, fixed-size patches and converts each patch into a token composed of numbers. It then processes these tokens using self-attention mechanisms to learn how they are related to each other and build a global understanding of the image, without relying on traditional convolutional operations. Vision transformer produces more dispersed activation patterns due to its patch-based processing and lack of spatial inductive bias, making the high-activation areas less intuitive to human visual inspection and less evident in terms of histoarchitectural characteristics in the image (Figures 3F, 4F, 5F).

In a multinodular and vacuolating neuronal tumor, regions of hypercellularity were observed in the image, but there was no significant cellular atypia or heterogeneity. ResNet50, VGG16, and vision transformer accurately classified this sample as a low-grade tumor (Figures 3D–F, Supplementary Video S1), whereas Inception-ResNet-V2 appeared to focus on the hypercellularity and mistakenly classified the tumor as a high-grade pathology (Figure 3C). In another WHO grade 3 oligodendroglioma sample, both ResNet50 and VGG16 were activated by the hypercellular regions and incorrectly classified the tumor as low-grade (Figures 4D,E, Supplementary Video S2), whereas the activation of Inception-ResNet-V2 and vision transformer resulted in the correct classification (Figures 4C,F). All 4 visual models correctly classified the sample from a WHO grade 3 astrocytoma as high-grade, although the activation pattern varied significantly (Figure 5, Supplementary Video S3).

For our experiments, before preprocessing, CLE image sequences that were completely affected by red blood cell or motion artifacts were manually removed. We did not use convolutional neural network-based diagnostic frame detection model (27) to filter single artifactual images to avoid disrupting the entirety of the image sequences.

To extract temporal features from the CLE image sequences, we selected the transformer encoder. It is equipped with 2 multihead self-attention layers that enable it to capture long-range temporal correlations and the sequential progression of CLE image content. Recurrent neural network–based models, such as long short-term memory (LSTM) networks (28), are theoretically suitable for this task. However, these models operate in an autoregressive manner, predicting each output step based on the previous one, which can lead to recurrent error propagation over long sequences. In contrast, the transformer encoder allows for at-once inference across the entire sequence, mitigating this issue through the use of self-attention encoding in parallel.

We compared our sequence-based classification model against a conventional frame-based classification model as the baseline. Although simple, the frame-based model overlooks temporal progression and contextual continuity, often resulting in erratic or inconsistent predictions due to frame-level noise or artifacts. In contrast, our sequence-based classification model mimics how a human expert evaluates the entire CLE examination, leveraging temporal modeling with a transformer encoder and local pattern extraction embedded in temporal convolutions to create a robust, temporally coherent representation of the full examination. This led to more stable and clinically aligned predictions, especially in borderline cases or instances with consecutive images affected by blood or motion artifacts.

Importantly, our method mimics how human experts interpret CLE imaging in a real-world setting. Human experts not only assess single frames in isolation but also gather information from sequential frames. In the first clinical feasibility study with the CLE system, it was apparent that our CLE-trained neuropathologist preferred examining whole sequences to uncover histoarchitectural features that individual static images do not reveal (29). This distinction is crucial when differentiating between blood contamination and densely packed tumor cells. Small, uniformly sized, disc-shaped cells that flow across the field of view in consecutive frames are more likely to be red blood cells entering the CLE imaging site and can thus be differentiated from relatively stable, motionless tumor cells. Additionally, the human brain's ability to parse information from unaffected areas when examining a series of images partially affected by motion artifacts allows for better utilization of these otherwise unusable images.

By explicitly modeling CLE as a temporal sequence and predicting at the examination level, our system emulates these human reasoning processes. This alignment between model behavior and clinical workflow enhances both the interpretability and trustworthiness of AI-assisted diagnostics, making it a better candidate for real-world intraoperative support. In turn, all visual backbone models benefited from being embedded in our sequence-aware architecture, showing clear improvements over their standalone, frame-based counterparts.

There are several features that greatly enhance the integration of CLE imaging into current intraoperative neurosurgical workflow. Martirosyan et al. previously reported successful co-registration of the CLE probe with the neuronavigation system to localize the precise CLE imaging site (17). Recently, Muscas et al. demonstrated the feasibility of connecting the CLE system (Zeiss CONVIVO), the neuronavigation system (Medtronic StealthStation S8), and the surgical microscope (Zeiss Kinevo 900) to enabled picture-in-picture display of CLE images in the surgical microscope visual field. This configuration reduced CLE imaging time and increased the proportion of usable CLE images by allowing the surgeon to assess image quality without diverting their gaze from the operative field (30). In addition, a built-in telepathology software platform allows for real-time transfer of CLE images from the operating room to the neuropathologist's device, along with voice communication between the neurosurgeon and neuropathologist (31).

Coupling our sequence-based AI model with an integrated surgical visualization ecosystem could provide neurosurgeons and neuropathologists with additional diagnostic information. This model can potentially be generalized to other CLE and intraoperative imaging systems that generate sequential images. Importantly, our approach is not limited by computing power. We used a consumer-grade GPU for model training and testing. For the 4 visual backbones, average inference times per sequence were as follows: Inception-ResNet-V2, 0.17 s; ResNet50, 0.11 s; VGG16, 0.19 s; and vision transformer, 0.33 s. With cloud-based deployment, our model can be pretrained and executed remotely on professional AI-optimized GPUs to provide near real-time feedback after image acquisition, enabling seamless integration with intraoperative CLE workflow.

Combined with in vivo CLE imaging, this tool would allow immediate intraoperative sampling of multiple spots within a nonenhancing tumor. Regions suspicious for higher tumor grade can be identified and biopsy performed to further confirm tumor grade and potentially avoid histological undergrading. Integration of such an AI-assisted system into the intraoperative workflow may ultimately enhance surgical precision and improve diagnostic yield while streamlining decision-making and improving workflow efficiency during brain tumor surgeries. However, prior to being implemented in clinical practice, our AI model needs to be refined. A more comprehensive dataset collected from multiple institutions and annotated uniformly should be used to further train and validate the model. A more balanced distribution of tumor histology type and WHO grade would be highly desirable. Prospective clinical studies should be considered to determine the reliability, clinical benefit, and workflow impact of the model.

The relatively small sample size and single-institution nature limited this study. However, the point of this study was a proof of principle, procedure, and exploration of appropriate and applicable AI image assessment programs to engage with the modes of CLE image acquisition and interpretation. All CLE images were acquired at a single neurosurgical center using standardized imaging protocols and interpreted by a single neuropathologist, which may not fully represent the variability of intraoperative imaging conditions, tumor pathologies, and inter-rater variability across other institutions. To mitigate potential bias and reduce randomization errors from assigning CLE image sequences to training and testing sets, we repeated all experiments across 3 different random splits and reported the average top-1 accuracy. This approach aimed to ensure a more robust and reliable performance evaluation despite the dataset size constraint. The results of this study will inform sample size calculations for future, larger-scale experiments aimed at validating the conclusions of this preliminary work. Furthermore, more than half of the cases in our dataset were oligodendrogliomas, whereas astrocytomas were significantly underrepresented. This may limit the generalizability of our results; nevertheless, we consider this feasibility study to be an important first step. Future expansion of the dataset with a more balanced representation of tumor types will likely enhance the robustness and performance of our model.