Radiomics, the high-throughput mining of image features from routine medical images, provides a quantitative and robust method to assess tumor heterogeneity. It can serve as a powerful tool for precision medicine in cancer treatment. However, the current work primarily focuses on diagnostic metrics which neglects the treatment effect. The involvement of in-treatment image should be investigated for a direct estimation of treatment outcome.

Daily cone-beam computed tomography (CBCT) images are originally developed for patient setup and have accordingly been acquired with low imaging dose. The images have substantially more scatter than diagnostic CT due to the flat-panel detector design. However, in many cases, these images are acquired at every fraction of treatment during the whole course of radiation therapy and may function as a timely biomarker for treatment-induced changes (1–4). Benjamin et al. (1) reported that serial changes from CBCT images during head and neck (H&N) radiotherapy can improve chronic xerostomia prediction. Hebert et al. (5) revealed that intersite heterogeneity captured from CBCT could predict outcomes in patients with high-grade serous ovarian cancer. Despite the promising potential of radiomics features from longitudinal CBCT images, the stability of these features should be addressed.

In addition, given the increasing number of radiomics-based studies, investigators have built many in-house software packages, and several radiomics platforms are available for public use. The inherent variations from algorithm implementation, image preprocessing, and mathematical definitions could cause large differences in radiomics feature computation. Moreover, the differences in disease-specific image settings such as mAs, kVp, and image resolution could also contribute to computational variations. The lack of understanding of stability has slowed the clinical implementation of many promising radiomics-based diagnosis or prognosis schemes. One can be guided on the use of radiomics features derived from CBCT images in clinical studies only after the sources of variations are understood.

As such, in this work, we evaluated the reproducibility and repeatability of radiomics features derived from longitudinal CBCT images for two distinct clinical sites. The agreement of these features across two commercially available platforms was also analyzed.

Figure 2 gives an example of a H&N case and a pelvic case with different-day CBCT images. The values of first-order feature such as histogram_skewness and of second-order feature such as GLCM_energy from LIFEx package are shown. Histogram_skewness which focused on total intensity distribution was very similar across days, but GLCM_energy which focused on internal heterogeneity showed variations.

In many cases of radiotherapy, CBCT is acquired throughout the patient radiation treatment and, thus, is a feasible image modality to detect sequential changes during treatment (1–5). However, before extending its role as an effective biomarker, the stability as well as the reproducibility and repeatability of CBCT-based radiomics features should be assessed. In this study, the reproducibility of CBCT-based radiomics features was assessed using two different softwares on two clinical sites with different acquisition settings, while repeatability was evaluated by comparing scans acquired at different days.

Recently, the stability of radiomics features has gained great attention from the society, and there have been over dozens of studies on related topics (17–22). However, previous works primarily focused on pretreatment diagnostic-level CT or PET images. Fave et al. (22) were one of the very few who tried to investigate the stability of CBCT-related radiomics features. They focused specifically on a test–retest dataset with 10 lung patients scanned twice for 15 min apart. They found that feature repeatability using CBCT was adversely affected by motion. However, there are many other questions that have not been answered, e.g., whether the finding in lung CBCT can be generalized to other disease sites such as H&N and pelvis, where motion is not commonly seen; and whether findings on other modalities, such as CT, can be extended to CBCT. To our best knowledge, limited work has been done to understand the roles of CBCT-based radiomics features. In this study, we assumed the tumor had minimal changes especially during the first 2 days of treatment; in this way, we can evaluate the stability of CBCT-based radiomics features. Our results revealed that most first-order histogram-based parameters on CBCT were reproducible for both disease sites compared with second-order features, with similar results confirmed by previous studies but on other modalities (12, 22–25). Notably, the robust features identified in H&N imaging protocol were not always extendable to pelvic imaging protocol. This might be due to the position of the tumor which is sometimes affected by the filling status of the bladder or rectum. The inherent differences of the image acquisition settings such as larger scatters, higher noise, and larger image resolution also contributed to larger variations as seen in pelvic protocols. Thus, the stability of radiomics features could be disease-/imaging protocol-specific and should be evaluated respectively and carefully.

The agreement of CBCT radiomics features across the two widely used software packages was demonstrated in this study, and variations were observed between the packages for both clinical sites. These sources of variations among packages included differences in image preprocessing, algorithm implementation, and feature-specific parameters. To ensure direct comparison, only those features shared with the same mathematical definitions were selected. Additionally, in LIFEx, features were calculated on the largest cluster of continuous voxels, while in IBEX, whole ROI was used regardless of whether voxels were connected or not. In the current study, only the connected ROI was used for calculation. Moreover, IBEX did not allow non-negative HUs of the CT scans, despite the fact that the lowest HU for a CT scan is −1,014. Thus, HU transformation was applied for feature calculation. Several studies have previously demonstrated that features can vary when calculated in different software platforms (21, 26, 27). The Image Biomarker Standardization Initiative (IBSI) is an international collaboration developed to help standardize radiomics feature calculation in terms of feature definition and nomenclature (28, 29). However, IBSI did not give guidelines for feature calculation settings. To eliminate variation from image preprocessing, no additional and only the default preprocessing was applied. What we found is that many first-order features showed good agreement across packages, with nearly all features differed. All second-order features showed poor–moderate agreement and had large variations when using package-specific default parameters. Therefore, when these radiomics features are used for predictive modeling, computer-aided diagnosis, or image segmentation, for example, the results could greatly differ depending on the software being used. It is unlikely for a single institutional research-oriented work to use different radiomics analysis software packages. However, for multicenter clinical trials and future accreditation work, the reproducibility of radiomics features using different analysis algorithms/packages should be documented and carefully evaluated.

Previously, the repeatability of radiomics features was tested in a “coffee-break” dataset of patients scanned with 15-min intervals (30–32). Due to ethical reasons by introducing an extra image dose, limited patient data can be collected and sometimes phantom measurement has to be used as an alternative (33). However, it is unclear whether stable features measured from the physical phantom or limited data of the patients with “coffee-break” intervals can truly represent the clinical scenario in which the time between scans is in the order of days. On the other hand, daily CBCT is commonly used in the radiation department for most of the patients, thus providing an informative test–retest dataset for radiomics feature repeatability assessment. Especially during the first- and second-day CBCT, patient anatomy changes are relatively minimal, which may allow for radiomics feature stability evaluation. It is also noticed that features from 5-day apart CBCTs were less consistent than the 2-day apart images. It cannot be excluded that in this time period, the tumor changes subclinically and that this change is detected by radiomics. When prognostic information is derived from image features in a radiomics study, one should be aware of changes in a tumor. It is advisable to avoid using features that are not robust in a test–retest study. However, if the dataset with a large time interval is used for test–retest analysis, it would mean that we discard features that are actually informative. Although beyond the scope of the current study, our future work will expand the current finding to broader-scope clinical studies to identify the most reliable and informative radiomics predictors for clinical outcome. Moreover, feature repeatability and its correlation with volume were further assessed in the current study. It is in consensus that from a multi-institutional trial that volume was one of the robust features in the clinical test–retest analysis (34). It was shown that two GLRLM features were highly correlated with volume, and this could partly explain the good repeatability of these features. However, the size dependence could introduce a certain level of uncertainty to extend the work to other studies. As such, we emphasize the importance of a proper test–retest study with a close control on the imaging acquisition protocol, interval of scans, target volume range, etc.

There were some limitations in this study. Several factors could have reduced the robustness of radiomics features. The variability of feature values, however, was compounded by differences in segmentation methods and institution-specific factors, whereas the dependence of the variability in features due to image-specific parameters (e.g., tissue type, imaging modality, and image acquisition) was not discussed. Although in this study we had patients for a test–retest study in a more clinical-oriented scenario compared with previous studies, the dataset was very small to be able to analyze subsets to test these effects. Future studies of images, where predictive performance for the outcome of interest is investigated, taken at different time points during treatment with a multi-institutional trial are necessary to address these considerations. Future studies should be designed to tightly control all aforementioned factors in a radiomics study. Nevertheless, to minimize the risk of using unstable and unreproducible features in a radiomics analysis, it is advisable to perform treatment site-specific and time-, scanner-, and imaging protocol-controlled analyses.

Despite CT/CBCT being the most common image modalities in the radiation oncology world and stringently maintained for patient setup/dose calculation purpose, there is a lack of consensus in terms of calibration process of the image system for additional radiomics (texture) analysis. Previous works primarily focused on understanding the inherent characteristics of CT-based radiomics features using water/solid water phantoms, the Gammex phantom (Sun Nuclear, Melbourne, FL, USA) or other commercially available phantoms designed for CT performance check (33, 35–37). However, those phantoms were originally designed with uniform materials offering HU close to human tissues with minimal internal patterns. These led to recent studies designing radiomics phantoms that consist of different textural compositions. Among them, the most used one was the credence cartridge radiomics (CCR) phantom which consisted of 10 different cartridges with rubber, polyurethane, cork, etc. (38). However, the capability of those materials in recapitulating human tissue in terms of a wide range of radiomics features was not evaluated. Additionally, radiomics feature itself as well as its stabilities can be disease-specific and material-dependent. Moreover, studies to understand CBCT-based radiomics features are even limited, and whether the findings in CT can be translated into CBCT is unknown. We are in the process of designing radiomics phantoms with various compositions to replicate radiomics features of different disease sites. These phantoms can be further used to evaluate feature reproducibility in terms of different material designs with varied acquisition settings and acquisition techniques including CBCT. Further effort can be expanded on understanding not just the variations on acquisition settings but also on which range and what methods can combat these variations.

In summary, the reproducibility and repeatability of CBCT-based radiomics features were assessed and compared for the first time on both H&N and pelvic sites. There were overlaps of stable features in both disease sites, yet the overall stability of radiomics features may be disease-/protocol-specific and a function of time between scans. More investigations are needed to further evaluate the stability of CBCT-based radiomics features before establishing its role as clinical biomarkers.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

The studies involving human participants were reviewed and approved by the Ethics Committee of Shanghai Chest Hospital (committee reference number: KS1863). The patients/participants provided their written informed consent to participate in this study.

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

This work was partially funded by Shanghai Hospital Development Center (Grant No.16CR3056A), the Interdisciplinary Program of Shanghai Jiao Tong University (Grant No. YG2019ZDB2019), and Nurture Projects for Basic Research of Shanghai Chest Hospital (Grant No. 2019YNJCM05).

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.