The dataset we used to conduct experiments was initially collected and published by the In-vivo HS Human Brain database (40) comprising 36 in-vivo brain surface images from 22 unique patients. This labeled dataset includes tumor and normal tissue, blood vessels, and other irrelevant materials within the surgical scene (referred to as background). Tumor types are differentiated in the dataset, encompassing primary (grade IV glioblastoma and grade III and II anaplastic oligodendrogliomas) and secondary tumors (lung and breast). Additionally, RGB representations of hyperspectral cubes within the in-vivo hyperspectral human brain image database are presented in Figure 1. The dataset designates the approximate tumor area using a yellow line, aiding in identifying the rubber ring marker corresponding to pathological analyses of the tumor tissue. Patient ID and Image ID details in Table 1 offer a comprehensive overview, including image characteristics and the pathological diagnosis of each image. The total number of labeled pixels for each class and image is specified, addressing cases where certain images were diagnosed as specific tumor types without labeled tumor samples due to procedural challenges.

The authors (40) mention the inherent challenges in acquiring in-vivo HI during neurosurgical procedures; the dataset primarily captures common tumor types over two years. The customized hyperspectral acquisition system, a preliminary demonstrator, is designed to capture tumor images on the surface or in easily focused deeper layers. The authors utilize a push broom camera for spatial scanning; the system's limitations include increased acquisition time and potential spatial coherence issues due to patient brain movement and procedural artifacts. As snapshot cameras offer real-time image acquisition but have fewer spectral bands than push-broom cameras, future investigations using high spectral resolution push-broom cameras are warranted. The dataset creation process by authors (40) addresses challenges from limited patient availability, presenting a preliminary database for exploring HI applications in tissue and tumor identification, tumor boundary delineation, and providing pertinent information for neurosurgeons. Their methodology leverages spectral characteristics guided by intraoperative MRI, surgeon expertise, and pathological analysis results. Subsequent data acquisition efforts are anticipated to broaden the database, encompassing more tumor types with detailed pathological descriptions.

Our innovative methodology for tumor tissue classification within an HHC Unit unfolds with the patient's arrival at the facility. The initial phase involves a hyperspectral sensor scan, capturing intricate details of the patient's internal composition. This technology provides a comprehensive overview, laying the foundation for precise diagnosis. Following the hyperspectral scan, the acquired data undergoes processing through Factor Analysis. This step is crucial for dimension reduction, ensuring that the hyperspectral cube retains only relevant features essential for accurate classification. The processed data then traverses through the layers of our SCS model. As a breakthrough in tumor classification, the SCS model enhances precision, even when trained with limited data. This stage is pivotal for predicting and classifying tumor tissues, contributing to superior performance compared to existing models.

Once the classification is complete, the results are securely stored within the hospital's private records, ensuring data confidentiality. This stored information becomes a valuable resource for future reference and analysis. Integrated into the HHC Unit is a seamless access mechanism through Healthcare APIs. Healthcare professionals can leverage these APIs to access detailed reports and results related to tumor tissues. This integration streamlines the diagnostic process, providing a user-friendly interface for medical interpretation.

The final phases of our methodology involve the medical interpreter within the Healthcare API, aiding healthcare professionals in interpreting results and making informed recommendations. These recommendations extend to surgical interventions and ongoing medical care, all tailored to the specific classification of tumor tissues and their respective locations. Figure 2 presents a comprehensive and patient-centric approach to tumor tissue classification within the HHC Unit. By seamlessly integrating hyperspectral imaging, Factor Analysis, and the innovative SCS model, we aim to revolutionize healthcare diagnostics and enhance the overall patient experience.

HI represented as X∈ℝ^{(M×N) × B}, where the dimensions (M×N) correspond to a specific area on the tissue surface and B denotes the total number of spectral bands in the HI. Each pixel within X, indicated as x_ij where i = 1, 2, …, M and j = 1, 2, …, N, is grouped into C unique tissue types, collectively expressed as Y = (y₁, y₂, …, y_n). Moreover, every x_ij∈X describes a tissue pixel through a spectral vector x_ij = [x_{i, j, 1}, x_{i, j, 2}, …, x_{i, j, B}]∈X, containing a series of B spectral data points.

In the initial processing phase, spatial characteristics are emphasized by implementing a patch extraction method. This preliminary step involves the creation of a hyperspectral cube, xi,j∈ℝ(s×s)×D, encapsulating the area surrounding the focal pixel (i, j) over a region of dimensions s×s. This approach is instrumental in enhancing the model's ability to distinguish between different features by integrating spectral and spatial attributes. As such, the spectral-spatial cubes x_{i, j}, drawn from the primary data and conforming to the dimensionality ℝ^{(s×s) × D}, are consolidated into the dataset X in preparation for subsequent feature extraction processes. The concluding step involves the selection of training and testing samples across each distinct class.

In neural networks, the convolution operation involves a sliding dot product operation, symbolized as w·x_ij, between an image patch x_ij and a filter w, which might miss crucial information due to its basic similarity measure. Enhancing this with normalization transforms the operation into cosine similarity, defined as w·xij||w||||xij||. This is similar to calculating the cosine of the angle between vectors, utilizing Euclidean distance.

To address these limitations, Strided Cosine Similarity (SCS) was developed as expressed in Equation (1). It operates similarly to convolution but includes key differences. In standard convolution, the operation is a dot product w·x_ij, while SCS involves normalizing the vectors. The normalization in SCS ensures the magnitude of vectors is unity before the dot product, leading to an expression like w·xij||w+q||||xij+q||, where q is a small value to avoid numerical instability.

The similarity measure in SCS ranges between –1 and 1, indicating complete opposition or perfect alignment of the kernel and image patch, respectively. To mitigate the issue of small magnitudes, which can lead to noise inclusion, additional parameters are introduced in SCS, formulated as;

Similar to conventional convolution in deep learning, SCS is a striding operation that extracts features from an image patch. However, it includes an additional step of magnitude normalization before the dot product, leading to what some literature refers to as Sharpened Cosine normalization. The effectiveness of SCS surpasses traditional convolutional processes in terms of speed due to fewer required parameters and the absence of normalization or activation functions.

In contrast to standard pooling, absolute max-pooling is employed in SCS for backpropagation filter updates, selecting the highest magnitude irrespective of the sign. The overall model with SCS is trained over 50 epochs, a batch size of 256, and a learning rate of 0.001. The learning rate significantly influences the model's learning rate, while momentum aids accuracy and speed. An root mean square prop and momentum-based optimizer, specifically the Adam optimizer, is utilized for its efficiency and computational advantages.

The results presented in this study are evaluated using the following metrics:

Kappa statistic: This statistical measure assesses the level of agreement between predicted classifications and ground-truth maps, as defined by Equation (2). In this equation, A_o represents the observed agreement, calculated using Equation (3), while A_e denotes the expected agreement, computed using Equation (4).

where,

and,

Here, TP and FP denote true positives and false positives, respectively, while TN and FN represent true negatives and false negatives.

Average accuracy (AA): AA signifies the average classification performance across different classes, as depicted in Equation (5).

Overall accuracy (OA): OA is computed as the ratio of correctly classified examples to the total number of test examples, as defined by Equation (6).

In the equations above, TP represents true positives, FP represents false positives, TN represents true negatives, and FN represents false negatives.

For our empirical assessment, we utilized in-vivo HS Human Brain database which is already discussed in Section 3.1 and accessible on request on this https://hsibraindatabase.iuma.ulpgc.es/. This experiment used a Jupyter notebook running on an Intel 11th Gen processor and 32 GB of RAM. For all experiments, the training, validation, and test samples distribution was set at 15%, 15%, and 70%, respectively. To ensure an equitable comparison, all models, including the RNN, 2D CNN, LeeNet, Xception, and proposed SCS models, were executed simultaneously with a single, randomly chosen set of samples. The reported results were achieved using a patch size of 3 × 3, and the three most informative bands were identified through Factor Analysis (FA). Regarding training parameters, the models began with randomly initialized weights, which were subsequently optimized via backpropagation using the Adam optimizer and a softmax loss function. Figure 3 present a detailed analysis of the validation loss and accuracy for all models under consideration. In this study, we adhere to this principle by keeping these parameters uniform across all compared methods, including our SCS pipeline, within a single execution run.

We present a comprehensive performance analysis based on the F1-Score, comparing SOTA, including 2D CNN, RNN, LeeNet, Xception, and our proposed SCS across different patients. The objective of this comparative experiment class level is to evaluate and compare the performance of these models in accurately classifying different tissue types in HI as results discussed in Table 2. Across different patients, our SCS consistently achieves high accuracy in predicting tissue classes, as presented in the Table 3. Notably, for Patient IDs 005, 008, 022, 028, and 029, SCS achieves exceptional accuracy close to or at 100% in classifying normal tissue, hypervascularized tissue, and background classes. This demonstrates the model's robustness in handling diverse cases. In cases where tumor tissue is present, the SCS model also successfully achieves accurate predictions comparison with other models (2D CNN, RNN, LeeNet, and Xception 2D). The model's effectiveness in leveraging SCS-enhanced features for accurate tissue classification irrespective of body location are shown in classification map outputs (Figure 4). These sub-figures correspond to different tumor tissue types: Normal Brain, Renal Carcinoma, Lung Carcinoma, Meningioma, and Lung Adenocarcinoma. The model's ability to handle various tissue classes and consistent accuracy across different patients and images highlight its potential as a valuable tool in medical diagnostics, particularly for tumor tissue classification.

A detailed analysis of the performance of SOTA models on key metrics, including Kappa Accuracy, Overall Accuracy, Average Accuracy, F1-Score, Training Time, Testing Time, and Memory Consumption. Table 3 summarizes the performance metrics for each patient and their corresponding image IDs across various tissue classes. From the patient-level experiment, the SCS model consistently outperformed both models across multiple performance metrics. For instance, in Patient ID 004, the SCS model achieved a Kappa Accuracy of 80.53, surpassing 2D CNN (49.93), RNN (47.66), LeeNet (71.67) and Xception (30.02). Similar trends were observed regarding Overall Accuracy, Average Accuracy, and F1 score, where the SCS model consistently demonstrated superior performance across all patient IDs. Notably, in Patient ID 021, the SCS model achieved a Kappa Accuracy of 98.55, significantly surpassing 2D CNN (85.12), RNN (45.74), LeeNet (97.83), and Xception (38.61). SCS model's ability to consistently attain high accuracy, coupled with efficient training times and memory consumption, underscores its potential for accurate tissue classification in HI data, highlighting its value in practical medical applications. Although other models such as 2D CNN, RNN, and LeeNet have less training time, their accuracy is low compared to the SCS model; as we know, in deep learning, there is a trade-off between speed and accuracy. Figure 5 shows results underscore the superior performance of the SCS model across various metrics, indicating its efficacy in accurately classifying tissue types in HI data. The consistent out performance of the SCS model reaffirms its potential to enhance medical diagnostics and contribute to real-world applications.