Classification of images of RLPM is multifaceted. In one problem, one would like to distinguish pores of different sizes and classify them into micro-, meso-, and macropores. Note, however, that there are multiple definitions of the three length scales, or respective ranges of pore sizes, so that those working on imaging of biological systems and geologists may not be entirely consistent in their interpretation of what the pore sizes represent. Many porous media contain microfractures, and therefore, another problem is detection, i.e., identifying microfractures and distinguishing them from large pores. One issue in both cases is a lack of extensive data for training the ML algorithm. Addressing this problem is relatively straightforward and is usually accomplished through transfer learning (TL), i.e., using NNs that have already been trained on other types of data related to porous materials that are relatively abundant and then fine-tuning them for the properties of interest. For example, [24] used CT images together with five well-known pre-trained ML models, namely, VGG-16, ResNet-50, InceptionV3 [25] (introduced by Google), DenseNet121 [26], and MobileNet [27], all of which represent deep convolutional NNs (CNNs), to generate accurate synthetic data for pore sizes and then used the NNs to classify images of soils based on their pore-size distribution, classifying the pores into various types mentioned above.

Precisely, the same principles used for classifying images of RLPM can also be applied to MIs, for which two types of classification are of interest, as follows.

The use of ML algorithms in problems involving porous media can be limited by the lack of large datasets required for training the algorithms. One way to address this problem is through the enhancement of images of porous media, which allows for the expansion of the dataset and more accurate estimation of the physical properties of the media. However, the application of ML algorithms to problems involving porous materials and media, particularly their images, initially encountered challenges. For example, pixel values in an image must first be transformed into a feature vector that can be detected by the ML method. After deep-learning methods were developed in early 2006, trained using multiple levels of representation, progress began to accelerate. Raw data can then be used as input to accomplish important tasks such as the detection or classification of certain features in datasets and image enhancement.

Enhancement of images to higher resolutions, given images with lower resolution, in order to obtain more data with good accuracy, is an important problem in medical physics that has been studied for a long time. The ML-based algorithms that were described above for enhancing images of RLPM can also be used in the context of MIs. [91] utilized multi-stream networks to reconstruct high-resolution cardiac MRI from one or more low-resolution 3D MRI images, precisely the same problem that was studied by [92] in the context of RLPM, which is described in the next section. Enhancement of medical images also has other applications. For example, one can not only recover missing spatial information but also utilize the enhanced image to infer advanced MRI diffusion parameters from limited data, as demonstrated by [93]. Comprehensive recent reviews are provided by [94] and [95].

One ML-based approach to the reconstruction of porous media uses generative adversarial networks (GANs), developed by [96, 97]. Assume that a digital image DI is a sample of a real probability distribution function (PDF) of the images f d , of which we have a number of sub-domains that are extracted from DI without any overlap. Every GAN consists of a pair of NNs, a generator G θ , defined by its parameter θ that generates realizations of the unknown PDF f d ( D I ) , and a discriminator D ω , with ω being a parameter that tries to distinguish between the realizations of the training set and the synthetic ones that are generated by G θ . The latter learns to classify inputs as either real or false, whereas the former generates sufficiently realistic fake inputs to the discriminator so that they are classified as real. For example, one feeds random noise into a generator that is trained to transform the noise into data that closely mimic the real data from the training set. The discriminator is trained to distinguish the actual training dataset from the generated data. As the training of the two NNs progresses, the discriminator becomes more accurate at identifying the fake data, whereas the generator improves its ability to generate fake data for deceiving the discriminator. Thus, GANs are capable of generating realizations from an arbitrary PDF without any restriction on its form, which is the feature that makes them a powerful method for reconstruction. This is particularly true when the input for reconstruction is an image. The generator performs a mapping from a random prior Z onto the domain of the image, defined as follows. Z ∼ N 0,1 d × 1 × 1 × 1 , (4) G θ :    Z → R 1 × n × n × n , (5) In Equation 5 N represents the normal distribution, while d in Equation 6 is the dimension of Z, with n being the linear size of a subdomain extracted from the original image. D ω ( D I ) assigns a probability to an image DI from a sample drawn from f d , the distribution of the true data: D ω : R 1 × n × n × n . Values close to 1 represent a high probability of being a sample of D I ∼ f d ( D I ) . Both the generator and discriminator must be optimized, which can be carried out by a variety of methods. One can, for example, represent both G θ ( Z ) and D ω ( D I ) using differentiable NNs with parameters θ and ω . The optimization process is carried out in two steps:

First, the discriminator is trained in order to maximize its ability to distinguish between real and fake samples. Training is carried out by a supervised deep-learning algorithm using the known real samples (label 1) and the realizations generated by G θ (label θ ). The loss (cost) function to compute the misclassification error is given by:

By combining an SRCNN, representing the generator, with an image classification network as the discriminator, an SRGAN is formed. Although SRGANs produce features that appear realistic, due to pixel-wise mismatch, the accuracy of the generated SR images is lower than that of SRCNN. Since micro-CT images usually contain significant amounts of image noise and texture that manifest themselves as high-frequency features, they are also inadvertently recovered. According to [100], SRCNNs usually suffice because they contain some form of intrinsic noise suppression while maximizing edge recovery. [85] used the same dataset, DeepRock-SR, with SRGAN and compared its performance with that of SRCNN. Figure 7 shows the comparison of the results with those produced by SRCNN and BC-generated images.

The second approach is the CCSIM method [39–42], an ML approach without using any NN that represents the digital image DI using a computational grid G, partitioned into overlapping blocks of sizes T x × T y , where G and DI have the same sizes. The neighboring blocks share overlapping regions OL with sizes ℓ x × ℓ y (see Figure 8). One then begins from a corner block of G (or any other block) and visits each grid block along a 1D raster path. For each grid block, a pattern of heterogeneity from DI is selected at random and inserted in the visiting block. The inserted pattern is referred to as the data event D T , with the word “event” implying that the inserted pattern of heterogeneity in the block may change again. Then, the next pattern is selected based on the similarity between the neighborhood points and DI, meaning that, instead of considering all the previously constructed blocks (by filling them with patterns of heterogeneity from DI), only those in the neighborhood of the current blocks are used for the calculations. Next, the similarity between, or closeness to, the neighboring blocks and DI is quantified based on the a cross-correlation function that represents a convolution between DI and D T ( x , y ) : ψ i , j ; x , y = ∑ x = 0 ℓ x − 1 ∑ y = 0 ℓ y − 1 D I x + i , y + j D T x , y , (7) which represents a convolution of D I ( x , y ) with D T ( x , y ) , where i ∈ 0 , T x + ℓ x − 1 and j ∈ 0 , T y + ℓ y − 1 . Thus, an overlap region of size ℓ x × ℓ y between two neighboring blocks and a data event D T are used to match the patterns in the DI. The overlap region contains a set of pixels or voxels selected from the previously constructed blocks and uses them in Equation 7 to identify the next pattern of heterogeneity. For Euclidean distance (difference) between the constructed block and the data to be minimum, ψ ( i , j ; x , y ) must be maximum or, in practice, exceed a preset threshold. After calculating ψ ( i , j ; x , y ) and selecting those patterns for which ψ ( i , j ; x , y ) exceeds the preset threshold, one of the acceptable ones is selected at random and inserted in the block currently being visited in G. The process is repeated until all the blocks of the grid G have been reconstructed. As a rule of thumb, the neighboring regions might have an overlap of size of approximately 1/5–1/6 of the size of the blocks. Large grid blocks increase the computations as computing ψ ( i , j ; x , y ) requires considering many points, whereas small regions may cause discontinuity in the patterns. The method has been shown to successfully reconstruct models of various complex 2D and 3D porous media based on their images.

[92] proposed a method for enhancing images of porous materials and computing its effective properties (such as their permeability) when only a limited number of HR images—as few as one—are available. The method is based on data augmentation and generating a large and diverse dataset for the training of a CNN. [92] used the CCSIM algorithm to generate a large and diverse dataset to be used for training the CNN, and thus, they dubbed their approach a hybrid stochastic-CNN (HS-CNN) method. They developed their approach for shales, one of the most complex porous media, but the method is applicable to any type of porous material. They used a limited number of images—30 2D images with a size of 500 × 500 —with the CCSIM algorithm to generate 2,000 plausible realizations of the same, which were then used as the training images with the CNN. The size of 500 of the output realizations was 1000 × 1000 , with the remaining having the same size as the original images. The success of any learning algorithm depends heavily on the number and diversity of the images that are used for its training. Thus, to diversify, the 2,000 training images were transformed using various filters, such as the Gaussian noise, scale, crop, rotation, and flip to increase the diversity between the training datasets. The CNN uses the high-quality training data and extracts their differences with the upscaled images of the same set, using a residual learning strategy. The training images all had high resolution, whereas the image at hand that required enhancement was a LR one, i.e., one with a smaller size. Therefore, the target image was enlarged, i.e., upscaled, to the size of the training images by using an interpolation method with bicubic functions. Then, the CNN learned iteratively how to estimate the residuals. After the training was complete, i.e., after the CNN learned how to estimate the residuals, the high-quality image was reconstructed by adding the original enlarged image to the estimated residuals. Therefore, the upscaled images were used as the input, whereas the estimated residuals represented the output. The CNN that [92] used had 22 individual layers that learned mapping of HR training images onto the LR input image. The utilized images were similar in their content but differed in their details. To reduce the computational time, 64 random patches of size 41 × 41 from the training data were selected from each training image and used in the NN, rather than utilizing the full-size images. While such patches can jeopardize the capture of large-scale structures in the images, the use of the aforementioned filters—particularly, the scaling operators, ensured that the large-scale structures were still accounted for. The constructed small patches were then fed to the CNN over several iterations.

The LR image represented the first layer. The next layer was the convolution layer that contained 64 filters of size 3 × 3 , thereby assigning one filter to each patch. The performance of the CNNs improves with increasing the number of filters, but the training time also increases. Choosing a smaller filter size is usually preferred in the sense that it reduces the computational time, but it may also result in missing the large-scale structures of the image. On the other hand, larger filters complicate the training process. Thus, the optimal filter size should be decided a priori or selected by trial and error. Zero padding was also used in order to generate feature maps with the same input layer. All the convolution layers, except the last one, contained the ReLU activation function. The initial weights were assigned randomly and optimized during the learning. The first layer applied the Softplus, or SmoothReLU, function as the rectifier to the input values without changing their depth information, given by Equation 8: A x = log 1 + exp x . (8)

The last layer of the CNN had a single filter of size 3 × 3 × 50 , followed by a regression layer that calculated the loss function defined be Equation 9, given by: L = ∑ i , j , k y Resid i , j , k − y HS - CNN i , j , k 2 ∑ x , y , z y i , j , k , (9) where y ( i , j , k ) Resid represents the value of each pixel (voxel) at ( i , j , k ) for the residual image D I Resid and similarly for image predicted by the HS-CNN algorithm, D I HS - CNN .

The CNN was trained with stochastic gradient descent momentum optimization. Since one deals with complex images and very large datasets, the gradient descent method does not by itself solve the problem of accurately estimating the optimal values of the weights w . Hence, the algorithm was used with momentum at each iteration, with the momentum defined as the moving average of the gradients; this approach retains the update Δ w , the difference between the weights in two consecutive iterations, which is then used in the subsequent update. The momentum was used to prevent the optimization computation from converging to a local minimum or a saddle point. A high momentum parameter accelerates the speed of convergence to the true minimum of the loss function, but setting the momentum parameter at too high a value may give rise to overshooting the true global minimum, thereby making computations unstable. On the other hand, if the momentum coefficient is too low, it cannot reliably help the minimization in order to avoid local minima, thereby slowing down the training. A moving average was also used because the data were just images.

Figure 9 presents a visual comparison of the results, indicating excellent accuracy of the HS-CNN algorithm. The initial LR image represents a very smooth and opaque view of the pores in a shale sample, whereas the enhanced image generated by the HS-CNN reveals features as observed in the reference image. Similarly, the image generated by the BC interpolation method is a smooth reconstructed image. As expected, the CNN alone, without the large dataset produced by CCSIM algorithms, did not perform well. Note, however, that in the modeling of porous materials, one usually has only a few HR images, which are often insufficient. [92] also made a quantitative comparison using various statistical measures. One measure, for example, was peak signal-to-noise ratio (PSNR) R , which was used with the images generated by both the bicubic interpolation and the HS-CNN algorithm and compared with that of the reference high-resolution image. R is defined as follows: R = 10 ⁡ log 10 M I 2 σ 2 , (10) where M I denotes the maximum possible pixel value in the image DI and σ 2 is the mean-squared error. The computed statistical measures, such that given by Equation 10, indicated the high accuracy of the enhanced images.

Every reconstruction method developed for RLPM can also be applied to biological materials, with the results used for training ML algorithms. As an example, Figure 10 presents a 2D MRI image of a baby’s brain and three realizations reconstructed from it using the CCSIM algorithm. [101] used a GAN to generate synthetic chest X-rays for infections caused by COVID-19 to train the model. The training data consisted of actual and synthetic chest X-ray images, which were then fed into a customized CNN for deep learning-based chest radiograph classification in order to distinguish the COVID-19 cases from patients with regular pneumonia, along with normal cases free of the disease. The accuracy of the model for detecting COVID-19 cases were 93.94 and 54.55 percent, respectively, with and without data augmentation, i.e., without generating synthetic data by GAN reconstruction, thereby indicating the accuracy of the GAN.

We describe the problem using a concrete example and then briefly discuss related works. [102] used images of sandstone to link the effective permeability of the porous media to their morphology. Their proposed network was composed of two ML structures, namely, a CNN for feature extraction and a regular fully connected NN for estimating the permeability. We describe in detail the CNN that extracted the features. The filters in the convolution layers slide along the input data, with each producing a specific feature map. The input data, 3D images labeled with their permeabilities, were then convolved using 3D filters, each of size k × k × k . For simplicity of describing the procedure, the following equation was used, which represents a linear mapping g x , y , z = ∑ i = − m m ∑ j = − m m ∑ k = − m m f x − i , y − j , z − k w i , j , k + b , (11) where f and g represent, respectively, the input and output; ( x , y , z ) are the coordinates of the voxels in the input; and w and b are the weights and biases, respectively. Each set of the 3D filters produces a 3D output image, which is the input for the next convolutional layer. Note that in Equation 11, only the bias does not depend on the location; otherwise, every pixel value f ( x , y , z ) is multiplied by a weight, and therefore, the resulting feature map depends on the location. If, however, two pixels have the same value in the input data and the weights are also the same, then the output will also be the same. This problem is addressed by the pooling layer that CNNs usually contain. The convolutional layer convolves the input images using the filters in order to extract their key features. To filter part or all of the input data, they are multiplied by the corresponding matrices, with the results summed up and placed in a single output pixel, as shown in Figure 11, where a 2 × 2 filter was used. The process is repeated by sliding the filters along the input data using the stride operation in order to generate new output features. Since the filter moves one unit along the data, the stride is one. The entire process of multiplying and summing up represents the convolution operation as it is similar to the classical convolution. In principle, one may require several filters because image recognition requires various feature maps, and multiple filters capture their essential properties. [102] used fifty 5 × 5 × 5 filters. As Figure 11 indicates, the pixels at the center have more influence in the convolution than those on the border of the input matrix (the 3D images), which can be problematic. To remedy the issue and take the effect of the marginal pixels into account, the edges were padded with extra zero-value pixels. The activation function A ( x ) used was ReLU, which was applied to the extracted features at each pixel x to produce their maps A ( x ) . Therefore, several maps were produced that hopefully captured all the important features that connected the input data to the output.

Despite their effectiveness in extracting various features from the input images, the resulting maps may still be limited in terms of recording features’ locations in the input data. This means that if the same pattern appears in various locations in the input data due to, for example, cropping, rotation, resolution, and size changes, then its feature map will be different after the convolution layer, thereby giving rise to a major problem with the convolved maps, namely, their sensitivity to the features’ location in the input. The reason is that the filters in the convolutional layers are applied with very small strides, usually stride 1, and thus, the maps generated by the convolutional layer will have very similar locations for each feature in the input. Hence, when a convolutional filter is applied and slid over the image, the output feature map may still depend on the data in each region. To resolve the problem, one may downsample, or coarsen, the feature maps, making them more robust to changes in the features’ position using the next layer of the CNN after the activation, the pooling layer, that downsamples the feature maps by summarizing the presence of the features in the map’s patches, while preserving the essential features of the data. This not only reduces the size of the data entering the fully connected layer but also reduces the computation. Thus, the pooling layer is very useful when one must preserve a specific feature without considering its exact location in the image because the layer combines all the semantically similar features. Reducing the dimensionality of the feature map also reduces the sensitivity to the location. Note that the pooling operation is specified, rather than learned. The pooling layer acts on each feature map separately in order to generate a new set of the same number of pooled feature maps. The size of the pooling operation or filter is smaller than that of the feature map. Two common functions used in the pooling operation are average pooling, which calculates the average value for each patch on the feature map, and maximum pooling (max-pooling), which computes the maximum value for each patch of the feature map. [98] used the former, which is defined as follows: g x , y , z = ∑ k = 0 s − 1 ∑ j = 0 s − 1 ∑ i = 0 s − 1 1 s 3 f s x + i , s y + j , s z + k , where s is the coarsening length scale.

The output of the pooling layers was then flattened to a 1D vector to produce the input to the fully connected layer, the final layer in the proposed network, before the output layer. The fully connected layer keeps a collection of the most important outputs from all the convolutional layers. Since the purpose of the network was regression, the number of neurons in the output layer depended on the number of targets, i.e., the permeabilities to be predicted. Each neuron in the output layer had a continuous value that represented the prediction for each target. All the feature maps in the last convolutional layer were connected to a unit in a fully connected layer, with the output layer producing the prediction y i —the permeability for image i —calculated by a logistic regression layer, y i = A R ( w i v i + b ) , where A R ( x ) is the activation function of the regression layer, taken to be the logistic function, [ 1 + exp ( − x ) ] − 1 .

This basic scheme has been used by many to link various properties of RLPM to their morphology, including diffusivity, elastic moduli, and the dispersion coefficient in solute transport through porous media [103].

As mentioned above, extracting feature representations from images at the pixel level is a difficult problem, particularly for MIs. Since CNNs have the capability to learn meaningful features at multiple levels and classify them, their use for feature extraction from medical images has naturally attracted attention. Extracting features from MIs falls within a general class of techniques called content-based image retrieval (CBIR)—also referred to as query by image content (QBIC) and content-based visual information retrieval (CBVIR)—which use computer vision techniques to address the problem of image retrieval, or the search for digital images in large databases. “Content” in this context is a reference to colors, shapes, textures, and similar information that can be extracted from the image, and therefore, “content-based” implies analyzing the contents of the image, rather than such metadata as descriptions associated with the image. A comprehensive review of the application of ML algorithms to the CBIR problem is provided by [104].

[105] combined feature descriptors of CNNs to extract 1 0 3 features for overlapping patches in 3D MRIs for the prostate and to construct a large feature matrix over all the data volumes. Hashing forests are an approach for mapping a query onto a target in a database, enabling similar nearest neighbors to be searched for and retrieved efficiently and accurately. This is typically done by learning a hashing function [106] that maps the database entries onto compact binary codes in the Hamming space. A hashing forest is a supervised variant of random forests developed for the task of nearest neighbor retrieval. A Hamming space is the set of all 2 N binary strings or vectors of length N in which two binary strings are considered to be adjacent if they differ only in one position. Then, the distance between any two binary strings, called the Hamming distance, is the number of positions at which the corresponding bits are different. [107] used a CNN with five layers to extract features from the fully connected layers of the CNN. They utilized the last layer and a pre-trained CNN and fed the extracted features to a support vector machine classifier to obtain the distance metric. The conclusion was that incorporating gender information produces better performance than just CNN features.

The three most common types of brain lesions are gliomas, meningiomas, and pituitary tumors. Thus, we first describe each type of lesion and then discuss the progress that has been made in using ML methods for analyzing and interpreting MIs associated with brain tumors.

According to Mayo Clinic [111], “Glioma is a growth of cells that starts in the brain or spinal cord. The cells in a glioma resemble healthy brain cells called glial cells. Glial cells surround nerve cells and help them function. As a glioma grows, it forms a mass of cells called a tumor. The tumor can grow to press on the brain or spinal cord tissue and cause symptoms.” Gliomas are primary tumors that grow within the central nervous system; a subset of them, glioblastoma (GBM), is the most common primary adult brain tumor. Diagnosis of GBM involves a combination of neurological examination, advanced neuroimaging, such as MRI, and histopathological analysis of composite subtypes after surgical resection or biopsy [112]. The current “gold standard” of treatment for GBM is radiation therapy (RT) with concurrent temozolomide (TMZ) or equivalent alkylating chemotherapy. The former, when added to surgery, improves survival by approximately 6 months [113]. More recent studies have demonstrated that the addition of temozolomide, both concurrently with and following radiation, increases survival by an additional two months [114]. Temozolomide, an FDA-approved chemotherapy drug sold under the brand name Temodar, is used to treat certain types of brain tumors, including GBM and anaplastic astrocytoma. It is an alkylating agent that, by damaging the DNA of cancer cells, slows or stops their growth. Recurrence is, however, inevitable and will almost always occur within the previously radiated area. GBM is highly heterogeneous with multiple cellular subtypes identified within a given tumor on transcriptomic sequencing. Studies have demonstrated that molecular subtypes have varying responsiveness to RT. Moreover, the impact of subtype analysis on radiation delivery is limited as it is usually obtained too late to impact RT planning due to the need to start radiation immediately after surgery. In addition, the high rate of recurrence of GBM suggests that a subset of radio-resistant cells is persistent and that there are likely infiltrative tumor regions not readily apparent on conventional MRI, which, therefore, receive sub-therapeutic doses of radiation.

Given the aforementioned problems–the varying radio-sensitivity, the longitudinal tumor dynamics, and difficulties in identifying recurrent cell reservoirs–there exists a critical gap in the clinical capabilities for providing adaptable, longitudinal care that adequately addresses each person’s unique tumor burden. Current qualitative neuro-imaging approaches provide some distinguishing information that can predict prognostic molecular markers based on such features as texture and density [115], but conventional systems lack the ability to determine the extent of disease infiltration and the underlying molecular heterogeneities to adequately target them based on each patient’s unique tumor burden. Due to such difficulties and the knowledge gap, it was hypothesized that one can bridge the gap by leveraging new MRI technologies to extract relevant imaging features, in addition to patient outcome data, and construct a predictive NN based on the data. Addressing these limitations, investigators have recently reported on leveraging conventional MRI-guided GBM tissue biopsies to contrast-enhancing and non-enhancing MRI regions in illustrating the potential for ML-based development of predictive radiogenomic modeling of underlying intratumor molecular heterogeneity [116].

A meningioma, according to Mayo Clinic [117], is “a tumor that grows from the membranes that surround the brain and spinal cord, called the meninges. A meningioma is not a brain tumor, but it may press on the nearby brain, nerves, and vessels. Meningioma is the most common type of tumor that forms in the head. Meningiomas occur more often in women. Most meningiomas grow very slowly over many years. They can grow without causing symptoms. However, sometimes, their effects on nearby brain tissue, nerves, or vessels may cause serious disability.” Most meningiomas are not cancerous, but some can be cancerous. More than 39,000 Americans are diagnosed with meningioma every year.

In humans, pituitary tumors, also called pituitary adenomas, are unusual growths that develop in the pituitary gland, or hypophysis, which is an endocrine gland (a network of glands and organs throughout the body) in vertebrates. It is behind the nose at the base of the brain. Some of these tumors cause the pituitary gland to produce excessive amounts of certain hormones that regulate important body functions, while others may cause the gland to produce insufficient amounts of the same hormones. Most pituitary tumors are benign, not cancerous, and typically do not spread to other parts of the body.

Deep NNs have been used extensively for analyzing brain images with multiple applications, including classification of disorders (such as schizophrenia and Alzheimer’s disease); segmentation of tissues, lesions, and tumors; detection and classification of tumors and lesions; predicting survival rate against certain diseases; and image reconstruction. Most ML-based methods learn mappings from local patches to representations to labels. However, if the local patches do not contain the contextual information required for anatomical tasks, then a more refined method may be needed. For example, as mentioned above, [75] used patches obtained through non-uniformly sampling, which was carried out by gradually lowering the sampling rate in patch sides in order to span a larger context. Convolutional NNs have been used extensively in many analyses of brain images, typically obtained by brain MRI, for which there are many public MRI datasets on brain tumor classification, such as Kaggle, OpenNEURO, and fastMRI.

[118] used a three-layer CNN to classify the three types of brain tumors described above. The ReLU activation function was used in the first layer, followed by a structure in which the activation function could be adjusted to normalize the input layer. Dropout operations (i.e., randomly setting the outputs of hidden neurons to zero, usually carried out during the training of an NN) were used in both the second and third layers to mitigate the risk of overfitting. One CNN was used to distinguish between meningioma, glioma, and pituitary tumors with 3,064 enhanced images for 233 patients. The World Health Organization (WHO) classifies the risk levels associated with gliomas into four grades, with grade I being the least dangerous and grade IV being the most dangerous. [118] used a second CNN that differentiated between the three glioma grades of higher risk—grades II, III, and IV—using 516 images for 73 patients. The accuracies of the two distinct cases were, respectively, 96.13 and 98.7 percent. [119] undertook a similar study to classify glioma and meningioma tumors of the brain. They used a CNN for the classification problem and a second one, faster region-CNN (faster R-CNN), for segmentation. Faster R-CNN, proposed by [120], shares full-image convolutional features with the detection NN, which makes it possible to obtain region proposals by essentially cost-free computations. It consists of two modules, with one being a deep CNN that proposes regions and the second one being the faster R-CNN detector [53, 55], which uses the proposed regions. The entire system is a single, unified NN for object detection. [120] used the faster R-CNN with 218 images as the training set, employing a CNN with two convolutional layers. Their model achieved an accuracy of 100 percent for meningioma classification and 87.5 percent for glioma classification, with an average confidence level of 94.6 percent in the segmentation of meningioma tumors.

CNNs were used by [121] (see also [122]) to classify tumor grades (see above), which were segmented using a pre-trained CascadeCNN, after which the deep features were extracted and the tumors were classified. CascadedCNNs were introduced by [123] for the segmentation of brain tumors. Their idea was that since the predictions should, in principle, be influenced by the model’s beliefs about the value of nearby labels, one should feed the output probabilities of a first CNN as additional inputs to the layers of a second CNN by concatenating the convolutional layers. Thus, [123] concatenated the output layer of the first CNN with any of the layers in the second CNN, which, effectively, represented a cascade of two CNNs with a two-way processing of an image. [121] carried out the classification using a fine-tuned VGG-19 network (see above), a CNN with 16 convolutional layers and 3 fully connected layers. Two datasets, Radiopaedia and brain tumor [124], were employed, which were extended using eight different augmentation techniques. The datasets were divided into 50, 25, and 25 percent sets for, respectively, training, cross-validation, and testing. The accuracy for predicting the Radiopaedia dataset for grade I was, respectively, 90.03 and 95.5 percent without and with data augmentation, with similar accuracies for grades II, III, and IV. The corresponding numbers for the brain tumor dataset were 88.41 and 96.12 percent.

There have also been a few published studies for the specific case of glioblastoma. [125], for example, studied the problem of resection after surgery, considered to be one of the main prognostic factors for patients who have been diagnosed with glioblastoma. To achieve this, one must carry out segmentation and classification of residual tumors from postoperative MR images, and [125] used DCNNs to study the problem. The dataset used consisted of pre- and early post-operative MRI scans from 956 patients (in 12 hospitals in the United States and Europe) who had undergone surgical resection of glioblastoma. With very few exceptions, all the EPMR MRIs (early postoperative magnetic MRIs) were obtained within 72 h after surgery. To have anatomical consistency among the various input sequences, an initial image-to-image registration procedure was implemented. The brain masks were automatically generated using a pre-trained slab-wise AGU-Net model with input of 256 × 192 × 32 voxels. AGU-Net (annotation-guided U-net) was introduced by [126] for rapid one-shot video object segmentation. It consists of two parts, a fully convolutional Siamese NN and a U-Net. In the first part, Siamese NNs are used to label the foreground and background in a new frame that corresponds to a reference object image, while the second part fuses the annotation information obtained by Siamese networks into the skip connection of a U-net to form an annotation-guided U-net for segmenting a specific target. A Siamese NN uses the same weights while working concurrently on two different input vectors to compute comparable output vectors. Since the AGU-Net analysis required a lower resolution, all inputs were resampled to an isotropic spacing and resized to 128 × 128 × 144 voxels. Based on the analysis of the dataset by the nnU-Net framework (see above), the 3D full-resolution U-Net was selected for further analysis, using 192 × 128 × 80 voxels as the size of the input patch, with the NN using five levels, downsampling using strided convolutional layers (a stride of one was used), and upsampling utilizing transposed convolutional layers. The size of the kernel was 1 × 3 × 3 voxels in the first level and 3 × 3 × 3 for the remaining four, while those of the filter were { 32,64,128,256,320 } for the corresponding levels. The loss function was a combination of the Dice score and cross-entropy (see above). All models were initialized using pre-trained weights from the best pre-operative glioblastoma segmentation model. The input layer was the only layer that was adapted to account for the various input combinations. The Adam optimizer was used with an initial learning rate, l r = 1 0 − 3 . Residual tumor tissue was predicted during inference by each trained model, generating a probability map of the same resolution as the EPMR CE scan. The accuracy of the trained NNs was evaluated based on their ability to segment the residual tumor and classify patients into those with gross total resection and those with residual tumor. Only two classes were considered for segmentation, whereby a positive voxel exhibited tumor tissue, whereas a negative voxel indicated either background or normal tissue. A rest tumor volume threshold was selected for classification that served as the cut-off value. A Dice score of 61 percent was the best achieved segmentation performance, while the best classification performance reached 80 percent. More importantly, the segmentation performance of the most accurate models was comparable to that of human expert raters.

[127] developed an enhanced YOLO (so above) architecture for detecting brain tumors, which improved significantly the accuracy of detection using convolutional block attention modules (CBAMs), squeeze-and-excitation blocks (SE), and residual blocks. The CBAM increases the ability to focus on critical spatial and channel-wise features in MIs, while the SE blocks further improve feature representation by emphasizing important channels. RepVGG blocks [128] can efficiently extract features, while residual blocks help mitigate the vanishing gradient problem. The model was evaluated based on three brain tumor datasets and was demonstrated to provide significant improvements over previous models.

Other medical problems associated with the human brain have also been studied recently using ML approaches. [129] studied the problem of automatic identification of epilepsy in electroencephalography (EEG) signals. They pre-processed the data using adaptive wavelet denoising models in order to remove noise while preserving the actual information-carrying part of the signal. Then, multiple variable permutation entropy and multiple variable multi-scale fuzzy entropy (mvMFE) were extracted from the data that contained complexity and frequency-specific variations. To enhance the discriminative power of the features, uniform manifold approximation and projection were also utilized for non-linear dimensionality reduction. Next, the ResNet neural network (see above), integrated with bi-directional LSTM, was used to capture temporal dynamics and spatial features. When implemented in Python, the model achieved an accuracy of 94 percent, an F1-score of 96 percent, a recall value of 93 percent, a specificity of 87.70 percent, and a precision of 82.21 percent. [130] used deep CNNs, together with various transfer learning models (see above), to analyze resting state functional MRI (fMRI) images in order to predict the autism disorder features. Their work indicated that VGG16 NN achieves a classification accuracy of 95.8 percent. Their work supports our contention that ML models can be generalized across various neuroimaging-based diagnoses.