We systematically searched eight databases, including Cochrane Library, PubMed, Embase, Web of Science (WOS), China National Knowledge Infrastructure (CNKI), Wanfang Data, Chongqing VIP (CQVIP), and Chinese BioMedical Literature Database (CBM), with the search terms “PD-1”, “PD-L1”, “gastric”, “colorectal”, “oesophageal” and “randomized controlled trial” to identify RCTs published from inception to 1 July 2022. The search strategy was preformulated by the research team and finally implemented by a team member (SL Ou). Furthermore, the reference lists of relevant systematic reviews were reviewed, and ClinicalTrials.gov was also checked to avoid omissions. Duplicate studies were removed by Endnote X9. The search strategy is detailed in Supplementary Table S1.

Studies were included that met the following criteria: 1) population: patients with EC, GC, gastroesophageal junction cancer (GEJC) and CRC; 2) intervention: PD-1/PD-L1 inhibitor monotherapy or in combination with chemotherapy (CT); 3) control: placebo or CT; 4) outcomes: hazard ratio (HR) of overall survival (OS), progression-free survival (PFS) or disease-free survival (DFS), grade 1–2 adverse events (AEs) and grade 3–4 AEs, quality of life (QoL); 5) study: Phase 2/3 RCT. Studies were excluded following exclusion criteria: 1) studies did not report survival curves or the rates of grade 1–2 AEs and grade 3–4 AEs; 2) non-Chinese or English literature.

The advanced disease and adjuvant or neoadjuvant therapy settings forms of ASCO-VF version 2 and ESMO-MCBS version 1.1 were used to assess the value scores (Schnipper et al., 2016; Cherny et al., 2017). ASCO-VF is designed for only in phase II or III RCT, including clinical benefit, toxicity and bonus points. The net health benefit (NHB) score is obtained by the final sum of the three module scores. The clinical benefit score is subtracted HR value the survival outcome indicator from 1, multiply by 100 points and then multiply by the weight (OS weighted 1, PFS weighted 0.8, ORR weighted 0.7). The toxicity score is the percentage difference between the total toxicity points of the intervention regimen and the control regimen multiply by 20 points. If the intervention regimen is more toxic than the control regimen, the toxicity score is subtracted from the clinical benefit score. If the toxicity of the intervention regimen was lower than the control regimen, the toxicity score is added to the clinical benefit score. Bonus points include 20 points for long-term survival (OS weighted 1, PFS weighted 0.8), 10 points for improvement in cancer-related symptoms, 10 points for quality of life, and percentage improvement in treatment-free interval multiply 20 points.

The ESMO-MCBS framework is designed for use only in positive trials, including clinical benefit, toxicity/quality of life. The clinical benefit grade is based on the lower limit of the 95% confidence interval (CI) of HR of survival outcome associated with a particular grade in a prespecified manner (e.g., grade 4 for control regimen with median OS < 12 months, HR ≤ 0.65 and OS gain ≥3 months). Upgraded 1 level if improved quality of life or/and less specific 3–4 AEs are shown.

Finally, the net health benefit (NHB) scores of ASCO-VF are continuous data; ESMO-MCBS grades are distributed as 5, 4, 3, 2 or 1 for advanced disease setting and as A, B, or C for adjuvant or neoadjuvant therapy setting. ASCO-VF does not clearly define what score is considered the “meaningful value threshold”, whereas ESMO-MCBS defines “meaningful clinical benefit” as a grade of 5, 4, A or B.

Two authors (SL Ou and JL) independently screened the titles and abstracts and full texts of eligible studies and used a standardized extraction form to extract the data. The extracted contents included the study name, phase, sample size, type of cancer, PD-1/PD-L1 inhibitors used, dosage regimen, follow-up time and outcomes. ASCO-VF scores and ESMO-MCBS grades were also independently evaluated by two authors (SL Ou and XL Qin). Any discrepancies were adjudicated by a third author (HW) to establish the final score or grade.

To assess the monthly cost of all anti-tumor drugs in the intervention and control groups of the included RCTs, we used the price of the branded name and generic drugs (often generic) from the Hospital Information System (HIS), which derived from the lowest wholesale pricing of the centralized procurement and drug price supervision platform of Sichuan Province and represented the actual purchase price of drugs in public medical institutions of the inter-provincial alliance. The monthly cost was calculated according to the dosage schedule in the included RCTs for a patient weighing 60 kg with a body surface area of 1.70 m². We reported the incremental monthly cost as the difference between the intervention and control groups. If the control group was placebo or best supportive care, the cost was set at zero. The most expensive one was recorded when the control group had multiple therapeutic regimens. The monthly cost of the therapeutic regimen was calculated over an average period of 30 days. Therapeutic regimens not available in China were not counted.

All data were collected using a standardized extraction form in an Excel file. Statistical analysis was performed with IBM SPSS (version 25.0). Continuous data were plotted to assess the normality of the underlying distribution. Comparisons between study groups were made using Student’s t-test or the Wilcoxon signed-ranked test, as appropriate. We generated a receiver operating characteristic (ROC) curve to assess the predictive value of the ASCO-VF score in relation to the threshold of the ESMO-MCBS grade and evaluate the consistency of the two value frameworks. We used scatterplots and Pearson’s or Spearman’s correlation to show the association between incremental monthly cost and ASCO-VF scores or ESMO-MCBS grades. All analyses were deemed significant if p < 0.05.

We identified 2086 records through initial retrieval. Ultimately, 33 studies reporting 23 RCTs published in English were considered eligible for this study (Kang et al., 2017; Bang et al., 2018; Shitara et al., 2018; Eng et al., 2019; Kato et al., 2019; Chen E. X. et al., 2020; André et al., 2020; Chen L. T. et al., 2020; Huang et al., 2020; Kojima et al., 2020; Shitara et al., 2020; Andre et al., 2021; Van Cutsem et al., 2021a; Boku et al., 2021; Van Cutsem et al., 2021b; Janjigian et al., 2021; Kelly et al., 2021; Luo et al., 2021; Moehler et al., 2021; Sun et al., 2021; Adenis et al., 2022; Antoniotti et al., 2022; Diaz et al., 2022; Doki et al., 2022; Fuchs et al., 2022; Kang et al., 2022; Lu et al., 2022; Mettu et al., 2022; Okada et al., 2022; Park et al., 2022; Shitara et al., 2022; Wang et al., 2022; Xu et al., 2022) (Figure 1). Of these, two (8.7%) RCTs were conducted in the setting of adjuvant therapy, while the others (91.3%) were conducted in the setting of advanced disease. Ten (43.48%) RCTs involved treatments for EC, five (21.74%) involved treatments for CRC, and eight (34.78%) involved treatments for GC/GEJC. Four (17.4%) RCTs had three arms, and the others (82.6%) had two arms. The median sample size was 493 (IQR 307–724), and all included studies were supported by pharmaceutical companies. More characteristics are presented in Table 1.

For the adjuvant therapy setting, durvalumab showed a negative value even compared with placebo, with an ASCO-VF score of −18.7. The application of ESMO-MCBS for nivolumab versus placebo resulted in a grade of A, which met the meaningful value threshold. For advanced diseases, all 25 therapeutic regimens met the evaluation criteria of ASCO-VF. The scores were normally distributed, ranging from −12.5 to 69. Since ASCO-VF has no clearly defined threshold for the meaningful value threshold, we used the mean score of 26.5 (95% CI 18.4–34.6) for subsequent analyses. Therefore, 12 (48%) regimens fell above the threshold, and 13 (52%) regimens fell below the threshold. The mean score of positive therapeutic regimens was 37.2 (95% CI 27.6–49.2), and the mean score of negative therapeutic regimens was 12.8 (95% CI 3.4–22.2). The value score of positive therapeutic regimens was significantly higher than that of negative therapeutic regimens (p < 0.001, Student’s t-test). Fourteen positive therapeutic regimens met the evaluation criteria of ESMO-MCBS. Six (42.9%) of the regimens met the ESMO-MCBS benefit threshold grade, and eight (57.1%) of the regimens did not meet the ESMO-MCBS benefit threshold grade (Table 2).

The ROC curve was used to forecast the meaningful value threshold of ASCO-VF to meet the ESMO-MCBS in advanced disease. The threshold score was 38.2, which was close to that in our previous study (Jiang et al., 2020). Excitingly, the area under the curve was 1.0 (p = 0.002), suggesting exactly the same predictive value. Based on this result, ASCO-VF scores and ESMO-MCBS grades showed that pembrolizumab met the meaningful value threshold in the first-line treatment of EC and microsatellite instability–high CRC. Toripalimab and camrelizumab met meaningful value threshold in the first-line treatment of squamous cell EC, and nivolumab and camrelizumab met meaningful value threshold in second-line treatment. PD-1/PD-L1 inhibitors did not meet valuable threshold in GC/GEJC.

The incremental monthly cost data of RCTs assessed by ASCO-VF were not normally distributed, thus, we analysed the correlation between value scores/grades and incremental monthly cost with Spearman’s correlation. The incremental monthly cost and ASCO-VF scores were negatively correlated (Spearman’s ρ = −0.465, p = 0.034, Figure 2). For ESMO-MCBS grades, the incremental monthly cost and value grades also showed a negative correlation (Spearman’s ρ = −0.211, p = 0.489, Figure 3).

The rising price of new anticancer drugs has led to public criticism of the pricing policies of manufacturers (Kantarjian et al., 2013). Coupled with the high burden of cancer, value assessment of new anti-tumor drugs has become an urgent need (Bach and Pearson, 2015). In this study, we assessed the value of PD-1/PD-L1 inhibitors in EC, GC and CRC using ASCO-VF and ESMO-MCBS. We found that only a few treatment regimens showed clinical value in EC and CRC. The association between ASCO-VF and ESMO-MCBS in this study was very well, and the value score/grade was negatively correlated with the incremental monthly cost.

Adjuvant chemotherapy after surgery is generally required for resectable locally advanced EC or GEJC. However, no treatment regimen has been shown to be effective, and the standard of care is best supportive care (Stahl et al., 2013; Ajani et al., 2019). In our study, nivolumab met valuable threshold in resectable locally advanced EC/GEJC (Kelly et al., 2021), which provides a new reference for clinical treatment and a new direction for clinical trials.

In regard to advanced diseases, 14 positive therapeutic regimens of 13 trials were assessed with both ASCO-VF and ESMO-MCBS, and 11 negative therapeutic regimens of 9 trials were assessed with only ASCO-VF. The NHB scores of positive trials were significantly higher than those of negative trials, and all negative trial scores were below the threshold predicted by the ROC curve. Considering that none of the 11 negative therapeutic regimens showed an improvement in QoL, we may conclude that a treatment is of no value when survival outcomes are not significantly increased while QoL is not improved, which is consistent with the use of ESMO-MCBS for non-inferiority (equivalence) studies (Cherny et al., 2015; Cherny et al., 2017). In GC/GEJC, none of the therapeutic regimens achieved the threshold value score or grade even when the PD-L1 combined positive score (CPS) was ≥1. PD-L1 inhibitor monotherapy or in combination with CT did not reach the threshold in CRC, but the PD-1 inhibitor pembrolizumab showed clinical value with an improvement in efficacy, toxicity and QoL as first-line therapy for microsatellite instability–high CRC (André et al., 2020; Andre et al., 2021; Diaz et al., 2022). In EC, pembrolizumab, toripalimab or camrelizumab in combination with CT showed clinical value in first-line treatment (Luo et al., 2021; Sun et al., 2021; Wang et al., 2022), and nivolumab and camrelizumab monotherapy showed value in second-line treatment (Kato et al., 2019; Huang et al., 2020; Okada et al., 2022). Although significant differences in survival outcomes have been at the forefront of drug approval and clinical decisions for many years, various stakeholders are increasingly focusing on the value (Vivot et al., 2017). In our study, we found that 8 of 14 positive therapeutic regimens did not meet the threshold value (Kang et al., 2017; Chen E. X. et al., 2020; Janjigian et al., 2021; Antoniotti et al., 2022; Doki et al., 2022; Lu et al., 2022; Xu et al., 2022), which suggests that the majority of positive interventions improved overall survival while compromising QoL or increasing the risk of toxicity. Therapeutic decisions should not be made solely on the p < 0.05 of survival indicators, and the clinical value of therapeutic regimens should be considered comprehensively.

Traditionally, we assume that the high price of new drugs is due to the need to support research; however, an analysis of transformative drugs shows that the main source of drug innovation is government-funded academic research (Kesselheim et al., 2015). As the payer of medical activities, the price paid by patients for drugs should have a positive relationship with the value created. In recent years, a series of studies have shown that there is no statistically significant association between the value and prices of anticancer drugs (Vivot et al., 2017; Jiang et al., 2020; Vokinger et al., 2020). Interestingly, in this study, we found a negative correlation between the prices of PD-1/PD-L1 inhibitors and their value. This negative correlation between prices and the ASCO-VF value score was even statistically significant (Spearman’s ρ = −0.465, p = 0.034), resulting in an urgent demand for value-based pricing. Camrelizumab and toripalimab showed clinical value in EC and have relatively low prices in the Chinese market, so their value may be more worthy of payment, which was also consistent with the results of China’s national price negotiations (Zhang et al., 2022).

This study has some implications. Firstly, this study shows no clinical value for PD-1/PD-L1 inhibitors in GC/GEJC, which suggests that subsequent clinical trials on the treatment of GC/GEJC with PD-1/PD-L1 inhibitors should fully follow the current evidence. Secondly, the prices of PD-1/PD-L1 inhibitors are not aligned with their value. Price negotiation for higher-priced PD-1/PD-L1 inhibitors should be prioritized to improve patient access to beneficial drugs, thereby contributing to patient-centred cancer treatment goals. Thirdly, all therapeutic regimens with improved QoL showed clinical value (Kato et al., 2019; Huang et al., 2020; Luo et al., 2021; Sun et al., 2021), suggesting that clinical trials and clinical treatment strategies should pay more attention to QoL.

We comprehensively assessed the value of PD-1/PD-L1 inhibitors in oesophageal and gastrointestinal cancer with ASCO-VF and ESMO-MCBS, and we acknowledged some limitations. Firstly, the number of RCTs included in this study was small, and there were only 14 therapeutic regimens that met both the ASCO-VF and ESMO-MCBS criteria. Although the consistency of the two value frameworks in this study was perfect, the conclusion may exist the risk of bias due to the influence of the small sample size. Secondly, as ASCO-VF did not define toxicity scores for subgroup analyses, they could not be evaluated in the subgroup analyses. Therefore, the subgroup results of PD-L1 expression and microsatellite stability level were partially incomplete. Thirdly, we used the pricing system of public hospitals and centralized procurement and drug price supervision platforms of Sichuan province in China, so the results of the correlation between the value scores/grades and cost do not necessarily apply to countries outside of China. Finally, we only considered drug costs when calculating monthly increments, without taking into account the patients and their spouses or other important people due to absence, emergency treatment, hospitalization and medical expenses. In fact, because these costs are not easy to obtain directly, value frameworks consider only the cost of drugs as a rough estimate of the cost of treatment.