The studies considered for this review met the following criteria:

· The study does not fulfill the inclusion criteria.

· The article is a systematic review or meta-analysis.

The present review was conducted according to the PRISMA guideline for systematic reviews ( Figure 2 ) (13). The literature research on Pubmed (MEDLINE) was carried out until November 2022 using MeSH keyword search. The search terms were the following: (machine learning) AND (tumor board); (machine learning) AND (multidisciplinary team meetings); (machine learning) AND (multidisciplinary cancer teams); (artificial intelligence) AND (multidisciplinary cancer teams); (artificial intelligence) AND (multidisciplinary team meetings); (artificial intelligence) AND (tumor board); IBM Watson for Oncology; (machine learning) AND (multidisciplinary team); Watson for Oncology; (artificial intelligence) AND (multidisciplinary team); (clinical decision support system) AND (multidisciplinary team meetings); (clinical decision support system) AND (multidisciplinary team); (clinical decision support system) AND (multidisciplinary cancer teams); (clinical decision support system) AND (tumor board).

For the search terms “Watson for Oncology” and “IBM Watson for Oncology”, the search was limited to literature from 2015 onwards, because commercial use of Watson for Oncology began in 2015 (14). For all other search terms, no time limit was set. In total, 4078 records were identified through database searching. Preliminary screening of titles, abstracts and duplicates yielded 139 articles. The aim of the paper was to include studies that focused on CDSS and then review the concordance. However, the very general selection of search terms resulted in in a large list of papers that deal with AI in oncology but did not cover any CDSS. Indeed, this was recognizable in most cases by title and abstract. Note, a decent amount of duplicates have been removed as well (n = 823). After the initial selection process, the articles were read in full and care was taken to review both treatment recommendations and concordance between CDSS and MDM. Excluded were articles that compared CDSS to a guideline, investigated how CDSS influences the actions of MDMs, articles that investigated the acceptance of CDSS by physicians or patients, or articles in which CDSS provided a prognosis or could decide on possible inclusion in a trial. Meta-analyses or reviews on the topic were also excluded. Finally, after independent assessment of full text articles by two different researchers (RO, SN), 31 articles were included. No separate checks on study quality like patient selection or study population were done. Studies were included when they had performed an analyzation of concordance rate between MDM and CDSS. If there was disagreement on this, an additional independent arbitrator (FK) was consulted for further resolution.

Review Manager (RevMan) 5.4.1 (The Cochrane Collaboration, 2020) software was utilized to conduct a comprehensive analysis of the extracted data. To enhance the clarity and ease of interpretation of the results, forest plots were generated. The primary objective was to assess the level of agreement between treatment decisions made by WFO and MDT for various cancer types. The data was analyzed dichotomously, and odds ratios (ORs) with corresponding 95% confidence intervals were calculated for each variable (stage, histology type, etc.) Heterogeneity among the studies was evaluated using the I² test. I²> 50% indicated considerable heterogeneity, whereas no heterogeneity was present in the absence of these conditions. P < 0.05 was considered significant. If the data provided could not be meta-analyzed, only descriptive analysis was done. Because not all studies could be included in the meta-analysis due to the unavailability data, an additional descriptive analysis was performed.

Most of the studies which matched the review criteria used Watson for Oncology (WFO). Twenty-three of the 31 studies were on WFO and concordance ( Table 1 ). Other Clinical Decision Support Systems (CDSSs) included were OncoDoc (15), Lung Cancer Assistant (LCA) (17) and the Multidisciplinary meeting Assistant or Treatment sElector (MATE) (16). Two studies using a decision tree model based on Dutch guidelines were included (41). In addition to the CDSSs mentioned above, there were two prototype decision tree models created by the working group Andrew et al. and Lin et al. that conducted a concordance study (18, 42).

Across all studies, a total of 16,472 participants were included. The number of included subjects varied greatly within the included studies. Five studies had a very small number of cases (< 100) (20, 24, 25, 27, 40) while the other studies had a relatively large number of included cases(> 1000) (16–18, 36, 43). Three studies examined multiple tumor entities, and included only a small number of participants in the subgroups (21, 38, 44).

Thirteen studies of breast cancer were conducted, involving a total of 7786 subjects (15, 16, 18–21, 27, 34, 36, 38, 43–45). The number of participants per study varied widely, ranging from 55 (27) to 1,977 (36). The most common treatment decisions reviewed in this study were for breast cancer, with the MDM and CDSS evaluated in multiple studies. Colorectal cancer was the subject of eight studies (21, 23, 25, 37–39, 41, 44), followed by lung cancer with seven studies (17, 21, 22, 30, 31, 35, 38). Gastric cancer was reviewed in three studies (21, 24, 29), while cervical (21, 32) and prostate cancers (33, 44) were each the focus of two studies. Thyroid cancer was examined in two studies (26, 40), and there was one study each for ovarian cancer (21), basal cell carcinoma (42), and hepatocellular carcinoma (HCC) (28).

Of the analyses evaluating the concordance rate between therapy decisions and CDSS, the majority were retrospective. Only three analyses were prospective (15, 16, 44).

As seen in Table 1 , there are several AI-based CDSSs used regularly in clinical oncology. The most common is Watson for Oncology (WFO); its use is widespread in the US and in Asia. Other systems like OncoDoc, LCA or other decision tree-based CDSSs are often prototypes which are only used at a single hospital, region or country. The CDSSs that were reviewed for concordance with treatment decisions are appointed below.

First, we conducted an overall meta-analysis of patients with different cancer stages (see Figure 3 ). In studies concerning WFO, treatment was deemed concordant if it was categorized as ‘Recommended’ or ‘For consideration’. A total of 18 studies were included in the analysis. The results showed a concordance rate of 72.7% (1992/2739) for stages I-II and 73.4% (2289/3117) for stages III-IV across various carcinomas, although this difference was not statistically significant (P=0.18). However, the meta-analysis revealed significant statistical heterogeneity (I² = 88%) across different cancer stages. As a result, we conducted a subgroup meta-analysis to examine specific cancer types and stages. In the case of breast cancer, five studies were included in the analysis (see Figure 4 ), revealing a concordance rate of 72.8% (1209/1661) for stages I-II and 84.1% (557/662) for stages III-IV, P≤ 0.00001.

The concordance rates for different cancer types and stages were as follows: for colorectal carcinoma ( Figure 5 ), 63% (245/392) for stages I-II and 67% (669/993) for stages III-IV; for gastric carcinoma ( Figure 6 ), 55% (33/60) for stages I-II and 45% (127/282) for stages III-IV; for cervical cancer ( Figure 7 ) for stages I-II 73% (105/144) and 68% (88/130) for stage III-IV; and for lung carcinoma ( Figure 8 ), 85% (137/162) for stages I-II and 83% (494/593) for stages III-IV. However, none of these differences were statistically significant (P>0.05). In addition, we analyzed different types of lung cancer, including SCLC and NSCLC, in three studies ( Figure 9 ). The results showed a concordance of 94.3% (134/142) for SCLC and 82.7% (416/503) for NSCLC, with a statistically significant difference (P=0.004). Analysis of histopathology subtypes in lung cancer revealed a concordance rate of 90% (450/495) for adenocarcinoma and 86% (230/266) for squamous cell carcinoma ( Figure 10 ), with a statistically significant difference (P=0.02). For ECOG 0-1, the concordance rate was 66.6% (330/495), while for ECOG 2-5 ( Figure 11 ), it was 58% (69/120), although this difference was not statistically significant (P=0.23).

Breast cancer has been analyzed by various CDSSs, showing generally high concordance. In the study of Somashekhar et al., the overall concordance rate between WFO and MDM is near 93% being at the ´Recommend´ level 62% and the `For consideration´ level 31% (19). Across the different stages, the concordance is above 80% (19), which is the same in the study of Zhou N et al. (21) As for the other CDSSs, there is also a high concordance rate of 93,4%, 93,2% and 85,3% using OncoDoc2, MATE and decision clinical tree system based on Oncoguide respectively (15, 16, 44). McNamara et al. conducted a study to analyze the concordance of WFO with decisions made by oncology experts and its impact on decisions made by newcomers to oncology. In breast cancer, the overall concordance rate among experts was found to be 87.9%. Novice oncologists had a concordance rate of 75.5% without the use of WFO, which improved to 95.3% with WFO (27).

In a study by Zhao et al., concordance rates between MDM and WFO were found to be only 77% for the adjuvant treatment group and 27.5% for the metastatic group (34). Xu et al. conducted an interesting study on the influence of WFO on treatment decisions, which showed that treatment decisions changed in only 5% of cases after reviewing WFO recommended treatment options for patients (36). However, there were also studies on breast cancer with low concordance rates, such as a study by Suwanvecho et al., which found a concordance rate of only 59.3% (38). In a study by Pan et al., the overall concordance rate was only 69.4%. Interestingly, the concordance rate was worse in the adjuvant chemotherapy group, whereas in the neoadjuvant chemotherapy group, the overall concordance rate was 96.7% (43).

Studies evaluating the use of WFO in patients with colorectal carcinoma have shown highly variable results. Some studies, including Zhou et al., Lee et al., and Mao et al., reported low overall concordance rates of 64%, 48.9%, and 66.9%, respectively, for colorectal cancer (21, 23, 37). However, other studies, such as Kim et al. and Aikemu et al., reported good agreement rates with overall concordance of 87% and 91%, respectively, for colorectal cancer (25, 39). Additionally, two studies that did not use WFO as a clinical decision support system also reported good overall concordance rates above 80% (41, 44).

Several studies have been conducted on lung cancer and WFO. Kim et al. achieved a high concordance rate of 92.4% (35). Zhou et al. showed an overall concordance rate of 83%, 92% for SCLS and 80% for NSCLC (21). In contrast, Liu et al. reported only an overall concordance rate of 65.8%, but also achieved 83% for SCLS but only 61.1% for NSCLC (22). Two of the studies discussed in this paper were conducted just for NSCLC, You et al. recorded a high overall concordance rate of 85.16% compared to the other studies, and Yao et al. achieved 73.3%, which was higher than the work of Liu et al. (30, 31) Sesen et al. used the LCA system, in which the rule-based decision support of the LCA guideline achieved an exact concordance rate of 0.57 with the recorded treatments. For the probabilistic LCA decision aid, the result was worse, with 0.27 and 0.76 for the exact and partial concordance rates, respectively. In this study, MDM was not performed, but patient treatment from the English National Lung Cancer Audit Database was compared with the LCA decision (17).

The overall concordance rate for gastric cancer was low at 54.5% by Tian et al. (32) In a study by Choi et al, concordance at the recommended level was also low at 41.5%, but higher at the recommendation level at 87.5%. For various stages and low ECOG scores, consensus was also low (24).

Two cervical cancer studies were found for this review. In both studies, overall agreement was below 75% with 64% and 72.8%, respectively (21, 32).

Yu et al. showed an overall concordance of 73,6% for prostate cancer. Looking at the different stages there was a higher concordance for lower stages (33). Ebben et al. showed in there study a similar overall concordance (78,8%) but using a different CDSS (44).

For thyroid cancer the results are diverse. The study of Yun et al. showed only an overall recommendation of 48% (40) in contrast to 77% overall concordance shown in the study by Kim et al. (26).

For ovarian cancer Zhou et al. showed a concordance rate above 90% overall and for stages as well (21).

Andrew et al. did a study on a Machine-learning algorithm to predict multidisciplinary team treatment recommendations in the management of basal cell carcinoma (42). They stated that the choice of conventional treatment (surgical excision or radiotherapy) by the MDT could be reliably predicted based on the patient’s age, tumor phenotype and lesion size. The algorithm reliably predicted the MDT decision outcome of 45.1% of nasal Basal cell cancer (42).

Zhang et al. conducted a study on hepatocellular carcinoma (HCC), where only surgically treated patients were included. The study aimed to compare the concordance between the decision made by WFO and the decision made by surgeons regarding the need for surgery, without comparing with MDM. The overall concordance rate was found to be 72%. In subgroup analyses, concordance varied from 66% for major hepatectomy to 88% for BCLC stage 0-A, indicating a higher level of agreement in less complex cases (28) ( Table 2 ).

While conducting a review on concordance, it was found that many studies from Asia were focused on the use of the WFO system. WFO was originally based on vast cancer treatment experience in North America and the National Comprehensive Cancer Network guidelines (14). Therefore, it is not surprising that there have been numerous studies on its concordance in other countries and this could affect the results and match rates. For example treatment recommendations for different types of cancer can differ significantly between countries, for example gastric cancer treatment in the US and Chinese population (48). Another example, while WFO recommends three immunotherapies, namely pembrolizumab, nivolumab, and atezolizumab, for metastatic NSCLC, these are not yet approved by the China Food and Drug Administration (CFDA) (21). Although WFO does not require all information, studies have shown that entering more data into the system could increase the concordance rate (20). It therefore also seems important to collect as much data from the population the system is used in.

When considering other CDSSs used, the studies available for analysis are limited, making it more challenging to draw conclusions about the consistency of treatment decisions compared to MDM.

Another significant issue is the variability in the definition of concordance. WFO overall concordance rate is often listed as ´Recommended´ and ´For consideration´ with these two categories sometimes being reported separately. It is crucial to carefully examine how the overall match is evaluated, as a high overall agreement may not always translate to a high “recommendation” but may only be viewed “for consideration”.

When treating cancer patients, an MDM is an integral part of treatment planning and approach (3, 49). Studies have already shown that oncology patients benefit from a multidisciplinary approach to health care (50–52). Therefore, discussion in an MDM should be considered fundamental in treatment decision-making. Consequently, the decisions of the CDSS should only be compared with the decisions of the MDM. In some studies, however, only a comparison between decisions regarding the actual treatment of patients and the CDSS was made (26–28, 30, 36, 38, 43). In part, even in some studies the CDSS decision was only compared to national guidance (17, 40). Moreover, the decisions of an MDM or actual treatment are not always consistent with the guidelines (53).

This review highlights a range of different tumor types with particular focus on breast, lung and colorectal cancer. These cancers are among the most frequently diagnosed worldwide (2), so it is understandable that more studies have been conducted on these types. The number of studies conducted for each cancer type allows for reasonable conclusions to be drawn about the agreement rate between the CDSS and MDM. However, for other tumor types, such as HCC, thyroid, prostate, cervical, ovarian and basal cell cancer only a few studies have been conducted, making it difficult to draw definitive conclusions. Among the various tumor types, breast cancer studies are the most consistent, with high agreement rates observed across different CDSS. These studies also tend to involve larger sample sizes, with most studies including more than 1000 patients compared to studies on other tumor types (16–18, 36, 43).

The review demonstrates a wide range of concordance rates across different studies, with some studies showing rates above 90% (19, 35, 39), while others are below 60% (29, 40). Therefore, it is crucial to only use a CDSS in clinical practice when there is a high concordance rate to ensure high confidence in decision-making. Breast cancer studies have shown the highest overall concordance rate, exceeding 90% in some studies (15, 16, 19) but still showing a wide range with even reported concordance rates below 60% (38). The concordance rates for gastric, thyroid and basal cell cancer are consistently the lowest. Regarding the agreement rates for individual stages, there is no general statement as it varies between studies and tumor types. The meta-analysis for different carcinomas showed no significant difference between stage I-II and stage III-IV ( Figure 3 ). For breast cancer, however, there was a significant difference, so the concordance rate was higher at advanced stages. For colorectal carcinoma, the studies that performed a staging analysis also showed low concordant rates. Thus, it is important to note that some studies showed a high overall concordance rate when no differential stage analysis was performed.

However, a lower ECOG score seems to be associated with a higher concordance in the results. Furthermore, in studies comparing the treatment recommendation for NSCLC and SCLS, SCLC shows a higher concordance rate than NSCLC. In NSCLC, adenocarcinoma has a higher concordance rate than Squamous cell carcinoma.

Jie et al. (14) published a meta-analysis on the application of WFO in 2021. However, only studies on WFO were included here (n = 9). Since then, multiple studies on AI in MDM have been published. The main purpose of the review by Jie et al. was to analyze the concordance rate between MDM and CDSS which was similar to our review. In comparison to our study only WFO was analyzed and less studies were included. One important difference was the concordance rate between the stages. The study by Jie et al. showed a higher agreement for lower stages, but without statistical significance, and a slightly higher overall agreement in comparison to our study. In our study, there was no significant difference in this regard, except for breast cancer. However, the subdivision was different. Thus, in contrast to us, Jie et al. subdivided into stages I-III and IV. Gastric cancer also showed the lowest agreement rates in Jie et al. A low ECOC also seemed to have been associated with a higher agreement rate in Jie et al. They also showed a higher consistency of SCLC compared to NSCLC, which was similar to our study.

In the near future, CDSS could be used in daily clinical routine. However, it is necessary to train the various systems based on large patient data sets. Moreover, verification of the accuracy of these data must take place on large patient collectives. The highest medical evidence is desirable and can be reached by conducting multicenter studies. This is certainly a major obstacle, since many hospitals use their own hospital information systems, making it more difficult to develop systems that can be used between different hospitals.

Should these systems prove to be highly accurate, then the use of CDSS in MDM can bring both a time saving and a qualitative gain. However, complete decision-making power by a CDSS should not be granted yet due to the importance and complexity of the decisions made during MDMs. However, it is conceivable that decision proposals are made by the CDSS and that the medical staff only has to approve them. Furthermore, the system should also recognize and indicate complex or individual cases and serve the latest scientific studies for the cases. Lastly, the automatic preparation of MDM cases is also a conceivable support for the medical staff.