Researchers have long been interested in analyzing the in-situ physical, chemical, and biological properties and processes that take place in plants and soils. To accomplish this, researchers have widely adopted the use of X-ray micro-computed tomography (X-ray μCT) for 3D analysis of flower buds, seeds, leaves, stems, roots, and soils (Petrovic et al., 1982; Crestana et al., 1985, 1986; Anderson et al., 1990; Brodersen et al., 2010; Hapca et al., 2011; Mooney et al., 2012; Helliwell et al., 2013; Cuneo et al., 2020; Théroux-Rancourt et al., 2020; Xiao et al., 2021; Duncan et al., 2022). In plants, researchers have used X-ray μCT to visualize the internal structures of leaves, allowing for the quantification of CO₂ diffusion through the leaf based on path length tortuosity from the stomata to the mesophyll (Mathers et al., 2018; Théroux-Rancourt et al., 2021). Other applications of X-ray μCT in plants include the visualization of embolism formation and repair in plant xylem tissue, allowing for the development of new models to better understand drought stress recovery, along with non-destructive quantification of carbohydrates in plant stems (Brodersen et al., 2010; Torres-Ruiz et al., 2015; Earles et al., 2018). In soils, X-ray μCT was used to visualize soil porosity, soil aggregate distribution, and plant root growth (Tracy et al., 2010; Mooney et al., 2012; Helliwell et al., 2013; Mairhofer et al., 2013; Ahmed et al., 2016; Yudina and Kuzyakov, 2019; Gerth et al., 2021; Keyes et al., 2022). Despite the wide use of advanced imaging techniques like X-ray μCT and imaging more generally in agricultural research, major limitations in both hardware and software hinder the speed and accuracy of image processing and data analysis.

Historically X-ray μCT data collection was extremely time consuming, and resource intensive as individual scans can exceed 50 Gb in size. Data acquisition rates were limited by the ability of X-ray detectors to transfer data to computers, limited hard drive storage capacity once the data was transferred, and intensive hardware requirements that limited the size of files that could be analyzed at any given time. Many of these constraints have been removed as detector hardware has improved, hard drive storage transfer speed and space has increased, and computing hardware has advanced. Now a major limiting step to the widespread use of X-ray μCT in the agricultural sciences is data analysis; while data can be acquired in hours to seconds, the laborious task of hand segmenting images can lead to analysis times of weeks to years for large data sets (Théroux-Rancourt et al., 2020).

Recent advances in machine learning, specifically the application of convolutional neural networks to image analysis, have enabled rapid and accurate segmentation of image data (Long et al., 2015; Ronneberger et al., 2015; Chen et al., 2018; Smith et al., 2020; Raja et al., 2021; von Chamier et al., 2021). Such applications have met with great success in medical imaging analysis, outperforming radiologists for early cancer diagnosis in X-ray μCT images (Lotter et al., 2021). However, challenges remain to applying convolutional neural networks to the analysis of agriculturally relevant X-ray μCT images. Specifically, training accurate models for image segmentation requires the production of hand annotated training datasets, which is time consuming and requires specialized expertise to properly annotate training image data (Kamilaris and Prenafeta-Boldú, 2018). Further, the computing resources required for training and applying deep learning models are unique, more common to computer gaming systems or graphics design work, rather than traditional computational systems (Gao et al., 2020; Ofori et al., 2022).

To navigate these challenges, we developed a modular workflow for image annotation and segmentation using open-source tools to empower scientists that use X-ray μCT in their work. Specifically, image annotation is done in ImageJ; while this does not prevent the need for experts to annotate images, it does allow experts to annotate their images without using proprietary software. The semantic segmentation of X-ray μCT image data is accomplished using Google’s Colab to run PyTorch implementations of a Fully Convolutional Network (FCN) with a ResNet-101 backbone (He et al., 2015; Long et al., 2015; Ronneberger et al., 2015; Chen et al., 2017; Paszke et al., 2019). The FCN architecture, while older, allows for model development on variable size images due to the exclusion of fully connected layers in the FCN architecture (Long et al., 2015; Ronneberger et al., 2015). In addition to X-ray μCT datasets, the workflow is flexible enough to work on virtually any image dataset, as long as corresponding annotated images are available for model training. Additionally, by developing and deploying the code in Google’s Colaboratory, users have access to free or low-cost GPU resources that might otherwise be cost prohibitive to access (Rippner et al., 2022b). If users have access to better hardware than is available through Colaboratory, the notebooks can be run locally to utilize advanced hardware. Additional code is also available to use this workflow on high performance computing systems using batch scheduling (Rippner et al., 2022b). This method for analyzing X-ray μCT data allows users to rapidly extract information on important biological, chemical, and physical processes that occur in plants and soils from complex datasets without the need to learn to code extensively or invest in expensive computational hardware.

In the following section we will describe the parameters under which our CT data was collected, how the CT image data was annotated for model training, and the parameters used to train the various models. The actual workflow and corresponding training video can be found on Github (Rippner et al., 2022b). For reproducibility purposes, the data sets used for training the models featured in this paper can be found a repository hosted by the United States Department of Agriculture, National Agricultural Library (Rippner et al., 2022a).

Accuracy, precision, recall, and F1 scores for a general model to identify and segment specific walnut leaf tissues on a pixel-wise basis plateaued after training and validation on at least 3 annotated leaves. For epidermis and mesophyll tissues, prediction F1 scores were > 80% while prediction F1 scores were generally > 75% for bundle sheath extensions and > 70% for airspaces. The lowest prediction F1 scores were achieved for veins tissues (∼60%) and the highest for the background class (>95%) (Figures 2, 3). Precision scores were generally higher than recall scores except in the case of vein tissue identification, where the models tended to over-predict the occurrence of the vein tissue class. F1 score variability across all prediction classes decreased as the number of leaves used during training and validation increased. Model F1 score variability was highest for the bundle sheath extension and vein tissue classes, likely due to colocation of the two tissue classes. When evaluating generalized walnut leaf model performance with increasing training epochs, model F1 scores plateaued after 50 epochs. Model F1 score variability was consistent after 50 epochs, with no improvement after additional training time (Figures 4, 5). This was particularly true for the vein tissue class which took the most training for consistent identification.

Model performance for the segmentation of an almond flower bud and a soil aggregate was hindered by the downscaling necessary to fit the training and validation data on video cards with 16 Gb of VRAM. When downscaled to 0.5 size in the x and y dimensions (25% size), the best model F1 scores were 99.8, 99.1, 92.4, and 71.5% for the background, bud scale, flower, and air space classes, respectively. At the same scaling for the soil aggregate, the best model F1 scores were 99.4, 83.7, 55.6, and 71.3% for the background, solid, pore, and organic matter classes, respectively. With access to the A100 GPU with 40 Gb of VRAM, training images were only downscaled to 0.85 size in the x and y dimensions (72% size). This yielded best model F1 scores of 99.9, 99.4, 94.5, and 74.4% for the background, bud scale, flower, and air space classes, respectively. At the same scaling for the soil aggregate the best model F1 scores were 99.4, 91.3, 76.7, and 74.9% for the background, solid, pore, and organic matter classes, respectively. While the improvements in model performance with increased scaling were modest for the flower bud, they were large for the soil aggregate, likely due to the intricacy of the aggregate that was lost with downscaling (Figure 6).

Model outputs are binary 2-dimensional data from which information like material area, perimeter or other morphological traits can be extracted using downstream image analysis functions. These data are saved as sequences of arrays which allows for the 3d visualization of the segmented materials (Figure 7). Additionally, 3d data can be extracted from the array sequences using Python libraries like NumPy or using other programming languages like R or MATLAB.

Generalized models for image segmentation are typically generated after training on hundreds to millions of images (Belthangady and Royer, 2019; O’Mahony et al., 2020; Shahinfar et al., 2020; Khened et al., 2021). Due to the limited availability of leaf, bud, or soil x-ray CT scans, such a training image library simply doesn’t exist (Moen et al., 2019). However, we found that we can generate accurate models using 5 annotated slices from at least 3 unique leaf scans. This discrepancy is likely the result of the consistent image collection settings (resolution) on the same imaging platform (Beamline 8.3.2) which simplifies the learning process (Fei et al., 2021; Silwal et al., 2021). Our results are comparable to those previously achieved by Théroux-Rancourt et al. (2020) on X-ray CT images of plant leaves. That method, based on random forest classification, requires hand annotating 6 images from every single scan and is designed for extracting data from leaf X-ray CT images exclusively. Specifically, the precision and recall scores for the background (>95%), mesophyll tissue (>80%), epidermis tissue (>80%), and bundle sheath extension (>75%) classes were equal to those achieved by Théroux-Rancourt et al. (2020). The current approach had lower precision and recall scores (>75%) for air space identification compared to Théroux-Rancourt et al. (2020) (>90%), but higher precision and recall scores for vein tissue identification (>60% vs. < 55%, respectively). Despite similarities in the quality of results, the current method decreases segmentation time from hours to minutes compared to Théroux-Rancourt et al. (2020) and can be applied to any X-ray CT image data set.

Our training batch size was limited to 1 by a combination of factors including variably sized training images for the leaf scans and hardware limitations for the almond bud and soil aggregate scans. Typically, batch size selection is constrained by a combination of hardware and the number of images used for training; the smaller the batch size, the longer training takes (Smith et al., 2017). This presents a significant barrier when training using millions of images but is not an issue when only tens of training images are available.

Epoch selection is an important component of maximizing model accuracy, precision, recall, and F1 scores. Training models for too few epochs leads to substandard model performance while over training with too many epochs wastes time and can lead to overfitting (Li et al., 2019; Baldeon Calisto and Lai-Yuen, 2020; Pan et al., 2020). Typically models trained with small batch sizes require more training epochs for satisfactory performance compared to models trained on large batch sizes (Smith, 2018). This can greatly increase training times if many images are required for training to achieve satisfactory model performance (von Chamier et al., 2021).

Beyond batch size and epoch number, image size plays a significant role in model performance. Large images take up more GPU memory than smaller images (Sabottke and Spieler, 2020). GPU memory can be conserved by decreasing batch size, but once batch size has been reduced to 1, the only option is to downscale images for training. However, this results in a significant loss of information in the images, hindering model performance as fine details are lost (Sabottke and Spieler, 2020). This was particularly problematic with the soil aggregate scans which contained fine pore spaces which were lost when the images were downscaled to 0.5 in the x and y dimensions to fit on GPUs with 16 Gb of VRAM. Downscaling to this degree results in a loss of 75% of the image information. Only when GPUs with more VRAM were used could images be downscaled less, resulting in improved performance for models trained on these higher resolution images.

By stacking the sequences of data arrays produced by the model, novel information can be gained about processes that occur over 3 dimensions. Taking this approach, Théroux-Rancourt et al. (2017) previously showed that mesophyll surface area exposed to intercellular air space is underestimated when using 2D rather than 3D approaches. Similarly, Trueba et al. (2021) found that the 3D organization of leaf tissues had a direct impact on plant water use and carbon uptake. In soils, it well understood that pore tortuosity plays a key role in understanding processes like water infiltration and O₂/CO₂ diffusion. As Baveye et al. (2018) highlighted in their review, the rapid segmentation of soil X-ray μCT data has long been a major hurdle to understanding these processes. Our workflow simplifies and accelerates this process, enabling researchers to rapidly extract information from their X-ray μCT data. Our approach is similar to those developed by Smith et al. (2020) and von Chamier et al. (2021), but is more flexible as it works with variably sized images and allows for multi-label semantic segmentation.

With the current work, we present a workflow for using open-source software generate models to segment X-ray μCT images. These models can be specific to an image set (segmenting a single soil aggregate or almond bud) or be generalized for a specific use case such as segmenting leaf scans. We demonstrated that a limited number of annotated images can achieve satisfactory results without excessively long training time. The workflow can be run locally, in Google’s Colaboratory Notebook, or adapted for use on high performance computing platforms. By using GPU resources, the rate of segmentation can be dramatically increased, taking less than 0.02 s per image. This allows users to segment scans in minutes, a significant speed gain compared to other methods with similar precision and recall (often > 90%) across a variety of sample scans (Arganda-Carreras et al., 2017; Théroux-Rancourt et al., 2020). This will allow researchers to gain novel insights into the role that 3d architecture of soil and plant samples plays in a variety of important processes.

The datasets presented in this study can be found on the National Agricultural Library Ag Data Commons website https://doi.org/10.15482/USDA.ADC/1524793.

AM, DR, JE, PR, EF, and DP contributed to the conception and design of the study. MM, FD, and KS annotated images. PR, DR, JE, JN, and AB wrote code for image segmentation and data extraction. DR wrote the first draft of the manuscript. MM helped write the “Materials and Methods” section of the manuscript. All authors contributed to the manuscript revision, read, and approved the submitted version.