Overall survival (OS) is the gold-standard endpoint used to demonstrate the clinical efficacy of new cancer drugs in randomized clinical trials (RCTs). The primary effect measure of interest is the ratio of the hazards of death, namely the OS hazard ratio (HR_OS), assessed over the entire follow-up period (FUP) and estimated using the proportional hazards (PH) Cox model.

However, a reliable estimation of HR_OS requires large RCTs with long FUP, resulting in increase in costs and time required before a new cancer drug is available to patients. To expedite drug approvals, the evaluation of new treatments in RCTs often relies on the assessment of their effects on surrogate endpoints, under the assumption that these effects accurately predict those on OS at the final analysis (1).

Progression-free survival (PFS) has been long used as a surrogate endpoint for OS in RCTs testing chemotherapy and targeted therapy in patients with advanced solid tumors. Also, the HR estimated from a Cox PH model for PFS (HR_PFS) is routinely used as a measure to empirically compare experimental versus control arms.

The weak correlation (R²<0.40) between HR_PFS and HR_OS resulting in RCTs testing immune checkpoint inhibitors (ICIs) (2, 3) may challenge the belief that HR_PFS is a potential surrogate endpoint for OS. However, the poor correlation between HR_PFS and HR_OS may be attributed to ICIs’ novel mechanisms for activating or rehabilitating self-immunity against tumors, which could result in delayed clinical effects and long-term responders, as well as in disease progression followed by tumor shrinkage (pseudo-progression). In these instances, the PFS curves may take some time to separate, and the immunotherapy agent curve may have a long tail, leading to the violation of the PH assumption on which the calculation of HR_PFS is based (4).

The restricted mean survival time (RMST), namely the mean survival time to some prespecified time point t^* , was proposed as an alternative treatment effect measure to address both delayed response and long-term responders issues, accounting for deviation from PH assumption (5). The treatment effect on PFS can be measured as the ratio of RMST (rRMST), which is the ratio of the area under the Kaplan-Meier (KM) PFS curve for the control group vs experimental group, from time 0 to a chosen time t^* .

Early overlapping PFS curves may also depend on pseudo-progression events, a documented type of response to ICIs that occur when an initial apparent RECIST-defined progression is observed prior to eventual disease improvement. To consider pseudo-progressions, Wang et al. (6) recently proposed a novel endpoint, the modified PFS (mPFS), which omits the events of disease progression (but not deaths) within n months (e.g., 3 months) after randomization, showing that HR_mPFS outperformed HR_PFS as surrogate for HR_OS in ICI trials.

Here, we performed a systematic review and meta-analysis of RCTs testing ICIs in patients with advanced solid tumors to compare the surrogacy value for HR_OS of both PFS and mPFS as endpoints and HR and rRMST as treatment effect measures, in strata of type of treatment administered in the experimental arm [i.e., ICI alone, ICI plus chemotherapy, ICI plus ICI or other treatment(s)].

The value of PFS and mPFS as surrogate endpoint for OS in RCTs testing ICIs was assessed using a meta-analytical approach based on pseudo individual patient-level data (IPD) (see details below). The treatment effect was measured by the HR for OS, and by the HR and the rRMST for the two surrogate endpoints.

Overall, 61 RCTs comprising a total of 36,034 patients were included in the analysis ( Supplementary Figure S2 , Supplementary Table S1 ).

Table 1 shows the main features of included trials (12–78). Six phase I/II or II or II/III trials (10%) and 55 phase III trials (90%) were included. The publication years spanned from 2011 to 2021, with 29 (48%) published in 2021. Forty-one studies (67%) were in the first-line setting and 27 (44%) enrolled patients with non-small cell lung cancer. OS alone was the primary endpoint in 25 (41%) trials, PFS alone in 12 trials (20%), OS and PFS or OS and overall response rate (ORR) as co-primary in 22 (36%) and 1 trial (2%), respectively. The sole phase I/II included trial used ORR alone as primary endpoint.

The median FUP of trials was 18.9 months, ranging from 5.1 to 69.5 months.

Six trials had three treatment arms, for a total of 67 comparisons analyzed ( Table 2 ). Thirty comparisons (45%) tested ICI alone, 22 comparisons (33%) evaluated the combination of ICI with chemotherapy, and 15 comparisons (22%) tested other ICI(s)-containing combinations, including an anti-PD-(L)1 combined with anti-angiogenesis drugs in 5 trials, the combination of an anti-PD-(L)1 with an anti-CTLA-4 drug in 7 trials, combo-immunotherapy (i.e., anti-PD-(L)1 plus anti-CTLA-4) plus chemotherapy in 2 trials, and an anti-PD-L1 combined with targeted therapy in one trial. The treatment administered in the control arm was chemotherapy alone in 56 comparisons (84%), anti-angiogenesis agent alone or in combination with chemotherapy in 6 comparisons (9%), targeted therapy in 2 comparisons (3%) and placebo in 3 comparisons (5%). Chemotherapy was administered as control arm in all the comparisons testing ICI plus chemotherapy, in 27 (90%) comparisons testing ICI alone, and in 7 (47%) comparisons testing ICI plus ICI or other treatment(s).

The proportional hazards assumption was not rejected (i.e. Grambsch-Therneau test p-value greater than 0.05) for both OS and PFS in 31 out of 67 comparisons (46%), for OS only in 15 comparisons (22%), for PFS only in 2 comparisons (3%), and neither for OS nor PFS in 19 comparisons (28%).

The PH assumption held for both OS and PFS in 82% of comparisons testing ICI plus chemotherapy, in 53% of comparisons testing ICI plus ICI or other treatment(s), and in only 5 (17%) comparisons testing ICI alone.

Supplementary Table S2 reports the quality assessment of trials according to the Cochrane Risk of bias tool. Overall, the quality of trials was high, as the risks of selection, attrition, reporting and other forms of bias for all the RCTs included in the analysis were low. The only potential biases affecting trials were performance and detection bias, since only 21 out of 61 RCTs had a double blinding design.

Supplementary Table S3 reports the treatment effect estimates and their ratios derived from pseudo IPD extracted from the published KM curves for each included comparison. The HRs for OS ranged between 0.43 and 1.03 in comparisons testing ICI alone, between 0.56 and 1.26 in comparisons testing ICI plus CT, and between 0.59 and 0.86 in comparisons testing ICI plus ICI or other treatment(s). Notably, pooled HR_OS were very similar across strata: HR_OS=0.77 (95% CI: 0.73, 0.82) for ICI alone, HR_OS=0.78 (95% CI: 0.73, 0.84) for ICI plus chemotherapy, and HR_OS=0.78 (95% CI: 0.73, 0.82) for ICI plus ICI or other treatment(s) ( Figure 1 ).

Pooled HRs for PFS and mPFS varied according to the type of treatment administered in the experimental arm. Pooled HR_PFS=0.86 (95% CI: 0.77, 0.97) in ICI alone, HR_PFS=0.68 (95% CI: 0.62, 0.74) in ICI plus chemotherapy, and HR_PFS=0.74 (95% CI: 0.65, 0.85) in ICI plus ICI or other treatment(s). Pooled HR_mPFS was 0.69 (95% CI: 0.64, 0.74), 0.72 (95% CI: 0.66, 0.78) and 0.68 (95% CI: 0.62, 0.73) respectively ( Figure 1 ).

Across the 67 comparisons, the time horizons t^* defining RMST ranged from 8.7 months to 5.1 years (median 20.9 months [IQR: 15.0, 26.0]) and no important differences emerged between strata.

Pooled rRMST_PFS and rRMST_mPFS were similar within and across strata of treatment administered in the experimental arm. In comparisons testing ICI alone pooled rRMST_PFS=0.85 (95% CI: 0.78, 0.92) and rRMST_mPFS=0.77 (95% CI: 0.74, 0.81), in comparisons testing ICI plus chemotherapy pooled rRMST_PFS=0.78 (95% CI: 0.73, 0.83) and rRMST_mPFS=0.82 (95% CI: 0.78, 0.87), while in comparisons testing ICI plus ICI or other treatment(s) pooled rRMST_PFS=0.80 (95% CI: 0.75, 0.86) and rRMST_mPFS=0.79 (95% CI: 0.74, 0.83; Figure 1 ).

When all the RCTs were pooled together, none of the endpoints (i.e., PFS and mPFS) or metrics (i.e., HR and RMST) investigated had a strong association with HR_OS: the R² of the association with HR_OS was respectively 0.38 (95% CI: 0.23, 0.52) for HR_PFS, 0.60 (95% CI: 0.39-0.73) for HRm_PFS, 0.39 (95% CI: 0.23, 0.52) for rRMST_PFS, and 0.31 (95% CI: 0.11, 0.50) for rRMST_mPFS.

Figure 1 shows the pooled ratio between each surrogate treatment effect measure and HR_OS by type of treatment administered in the experimental arm.

Notably, The HR_PFS showed a larger treatment effect for the experimental treatment compared to that observed for HR_OS (ratio of the two treatment effect measures <1) in 7 (23%) comparisons testing ICI alone, in 20 (91%) comparisons testing ICI plus chemotherapy, and in 8 (53%) comparisons testing ICI plus ICI or other treatment(s) ( Supplementary Table S3 ).

Overall, the HR_PFS significantly underestimated the protective treatment effect size observed on HR_OS in comparisons testing ICI alone (pooled ratio HR_PFS/HR_OS=1.14, 95% CI: 1.06, 1.24) and, on the contrary, significantly overestimated it in comparisons testing the combination of ICI plus chemotherapy (pooled ratio HR_PFS/HR_OS=0.88, 95% CI: 0.83, 0.93).

Conversely, the HR_mPFS significantly overestimated the protective treatment effect size observed on HR_OS regardless of the type of treatment administered in the experimental arm (pooled ratio HR_mPFS/HR_OS was 0.90 (95% CI: 0.86, 0.95) for ICI alone, 0.93 (95% CI: 0.89, 0.97) for ICI plus chemotherapy, and 0.89 (95% CI: 0.83-0.95) for ICI plus ICI or other treatment(s); Figure 1 ).

Supplementary Table S4 shows the pairwise agreement between the statistical significance of HR_OS and each surrogate measure. In comparisons testing ICI alone, there was no agreement between statistical significance of HR_OS and HR_PFS (Cohen’s Kappa coefficient was 0.02 [95% CI: -0.32, 0.36] and McNemar’s test p-value 0.197), while a stronger agreement was observed between HR_OS and the other surrogate measures (Cohen’s Kappa coefficient ranged from 0.30 for HR_OS vs rRMST_PFS [McNemar’s test p-value = 0.035] to 0.51 for HR_OS vs HR_mPFS [McNemar’s test p-value = 0.014]).

In comparisons testing ICI plus chemotherapy, a fair agreement was observed between HR_OS and all the surrogate measures (Cohen’s Kappa coefficient was 0.37 [95% CI: 0.04, 0.71] and McNemar’s test p-value 0.014 for HR_OS vs HR_PFS, rRMST_PFS and rRMST_mPFS). In ICI plus ICI or other treatment(s) strata, the agreement was very poor with all the surrogate measures (Cohen’s Kappa coefficient ranged from -0.06 for HR_OS vs HR_mPFS [McNemar’s test p-value = 0.257] to 0.12 for HR_OS vs HR_PFS and rRMST_PFS [McNemar’s test p-value = 0.414]).

Figure 2 shows the correlations between effects of ICIs on OS (y-axis) and the potential surrogate endpoints (x-axes; i.e., panel A: HR_PFS, panel B: HR_mPFS, panel C: rRMST_PFS, and panel D: rRMST_mPFS) according to the type of treatment administered in the experimental arm.

Among comparisons testing ICI alone, a moderate association with HR_OS was observed for HR_PFS (R² = 0.58, 95% CI: 0.24, 0.78), HR_mPFS (R² = 0.55, 95% CI: 0.25, 0.71) and rRMST_PFS (R² = 0.54, 95% CI: 0.30, 0.66), while for rRMST_mPFS the correlation was weak (R² = 0.27, 95% CI: 0.07, 0.51).

Among comparisons with ICI plus chemotherapy as experimental arm, a strong association with HR_OS was observed for HR_PFS (R² = 0.74, 95% CI: 0.35, 0.90) and HR_mPFS (R² = 0.81, 95% CI: 0.49, 0.93), while the R² coefficient was 0.55 (95% CI: 0.07, 0.79) for rRMST_PFS and 0.63 (95% CI: 0.17, 0.82) for rRMST_mPFS.

In the ICI plus ICI or other treatment(s) strata, the associations were very weak for all the surrogate endpoints and treatment effect measures with the R² ranging from 0.01 to 0.22.

The surrogacy equations between the log-transformed treatment effects and the ln-HR_OS estimated from the WLS regression, along with the R² coefficient, the prediction bands, and STE were displayed in Figure 3 , Supplementary Figures S3 and S4 for ICI alone, ICI plus chemotherapy, and ICI plus ICI or other treatment(s), respectively.

The slope of the surrogacy WLS regression lines significantly differed from zero for all the analysis on ICI alone and ICI plus chemotherapy comparisons.

There is controversy on the value of PFS as surrogate endpoint for OS in RCTs testing anticancer immunotherapy in patients with advanced solid tumors (2, 3, 79, 80). Another issue of intense debate is the most adequate metric to capture the treatment effect on PFS, since the widely adopted HR relies on the PH assumption, which is frequently violated in immunotherapy trials (4).

To our knowledge, we reported here the most comprehensive and updated analysis exploring these two related issues: i) the value of the PFS as surrogate endpoint for OS in trials testing ICIs, as compared with the alternative endpoint represented by mPFS; ii) the suitability of the HR to measure the treatment effect on PFS, as compared with the alternative metric represented by RMST.

Our findings tried to increase the understanding of the afore-mentioned conflicting results regarding the surrogacy value of PFS (2, 3, 79, 80). When all the RCTs were pooled together, none of the endpoints (i.e., PFS and mPFS) or metrics (i.e., HR and RMST) investigated had a strong association with HR_OS.

For example, we found that the R² of the association between HR_PFS and HR_OS was 0.38 (95% CI: 0.23, 0.52). This means that only 38% of the variability among treatment effects on OS is explained by the effects observed on PFS, far from the R² cut-off value of 0.70, which is considered optimal for a candidate surrogate endpoint by international guidelines (8).

However, when comparisons testing different types of treatments were analyzed separately, we observed a moderate (i.e., 0.5≤R² ≤ 0.69) or a strong (i.e., R²≥0.7) association between HR_PFS and HR_OS in the two groups of comparisons testing ICI alone or combined with chemotherapy, respectively.

This discrepancy between results of overall and stratified analyses might be explained by the fact that the HR_PFS and HR_OS estimates were well aligned within the two groups, but along two different regression lines. The regression line fitting the HR_PFS over HR_OS in comparisons testing ICI alone had lower values of both the intercept (-0.22) and the slope (0.43), compared to those estimated in the ICI plus chemotherapy group (intercept=0.05, slope=0.77).

The slope indicates the steepness of the regression line and captures the average rate of HR_OS change when HR_PFS increases by 1 unit. Slope values near zero indicate that negligible HR_OS changes occur as the HR_PFS changes, while slope values that are progressively further from zero indicate rates of HR_OS change increasingly larger. Our results suggests that in both groups of trials an improvement of PFS favoring the experimental arm translated into an OS improvement, but the amount of OS gain with the same treatment effect on PFS was different. As a matter of fact, even if the pooled HR_OS was very similar in the two groups (0.77 [95% CI: 0.73, 0.82] and 0.78 [95% CI: 0.73, 0.84] for ICI alone or combined with chemotherapy, respectively), the pooled HR_PFS resulted significantly heterogeneous (0.86 [95% CI: 0.77, 0.97] and 0.68 [95% CI: 0.62, 0.74], respectively). It might be explained by the fact that the biological effects and the impact on the natural history of disease of each type of treatment can differ substantially, and thus the amount of PFS improvement that eventually translates into an effect on OS is strictly treatment-dependent.

Homogeneity of treatment types and their biological mechanisms of action should be guaranteed in surrogacy analyses (8). If this principle is overlooked, even a strong association between the candidate surrogate and the true endpoint can be hidden, leading to the erroneous conclusion of absence of surrogacy.

Consistently, the absence of PFS surrogacy observed in the third group of comparisons [i.e., ICI plus ICI or other treatment(s)] can be attributed to the heterogeneous spectrum of therapies considered, including the combination of ICI with anti-angiogenesis agents, targeted therapies, or different types of ICIs. The limited number of RCTs available for each of these treatment types precluded the possibility to perform more specific analyses. Therefore, no conclusions can be drawn on the surrogacy value of PFS for immunotherapy strategies other than ICI alone or combined with chemotherapy.

Finally, our results showed that mPFS did not confer an advantage over PFS.

Concerning the issue of the best metric for measuring the treatment effect on PFS, it has been reported that the RMST can provide theoretical advantages over the HR. RMST is not affected by deviations from the PH assumption, which is common in immunotherapy trials due to the intrinsic mechanisms of action of ICIs that lead to delayed responses and long-term responders (4, 81).

Although in 54% of included comparisons (36 out of 67) the PH assumption was violated (for PFS or OS only, or both), we unexpectedly found that the HR outperformed the rRMST, yielding higher R² values for PFS over OS in all of the explored contexts. Notably, despite the fact that the largest difference favoring HR versus rRMST was observed in the group of trials testing ICI plus chemotherapy, where the PH assumption held in the majority of cases for both OS and PFS, no advantage for rRMST was observed in the other two groups of comparisons, where the PH assumption was largely violated.

Our analysis revealed that HR_PFS is limited by the tendency to significantly underestimate the treatment effect size observed on HR_OS in comparisons testing ICI alone (pooled ratio HR_PFS/HR_OS=1.14, 95% CI: 1.06, 1.24) and, on the contrary, to significantly overestimate it in comparisons testing the combination of ICI plus chemotherapy (pooled ratio HR_PFS/HR_OS=0.88, 95% CI: 0.83, 0.93).

This had in turn relevant impact on the STE of HR_PFS, which is defined as the minimum value of the HR_PFS necessary to be observed in a future RCT to confidently predict an OS benefit. Our results showed that a HR_PFS lower than 1 could be enough to predict a significant OS benefit in RCTs testing ICI alone, while a HR_PFS lower than 0.70 is required in trials testing ICI plus chemotherapy.

The overestimation of the final effect on HR_OS by HR_PFS is expected, since all the events considered in the OS endpoint are also included in the PFS. Similar findings were previously reported in surrogacy analyses on RCTs testing chemotherapy or targeted therapy (80, 82). In contrast, the underestimation of the final effect on HR_OS by HR_PFS observed in trials testing ICI alone is quite unusual and it could be due to the pseudo-progression events. These events are specifically observed in patients treated with ICI alone and they can lead to an erroneous and systematic underestimation of the ICI effect on PFS without affecting patients’ OS. In accordance with this hypothesis, the underestimation of the treatment effect on OS in comparisons testing ICI alone was not observed when considering the mPFS that accounts for the pseudo-progression events. However, the weak value of the R² precluded the possibility to use such endpoint in this context.

It’s worth noting that the incidence of pseudo-progression events has been reported to be higher among patients treated with anti-CTLA4 monotherapy as compared to anti-PD(L)1 monotherapy. Consequently, the surrogacy value of mPFS might be substantially higher in trials exclusively testing anti-CTLA4 treatments. However, the limited numbers of RCTs assessing anti-CTLA4 monotherapy precluded conducting a more detailed analysis.

The immune-related RECIST (iRECIST) are new response criteria specifically designed to assess the activity of immunotherapy, and to correctly categorize real versus pseudo-progression events (83). The adoption of iRECIST to categorize the events included in the PFS would probably improve its tendency to underestimate treatment effects on OS in trials testing ICI alone, but additional study and validation of iRECIST are required.

Our study has several strengths. It is the most comprehensive and updated analysis on such relevant topic, and it includes a large number of RCTs and patients. The efforts to reconstruct the IPD of more than 36,000 patients using a validated algorithm allowed to reliably assess and compare different endpoints and metrics. Also, the wide range of treatment effect estimates reported in the included trials for both PFS and OS contributed to ensuring adequate generalizability of the analysis.

The main limitation of our analysis is that it is based on reconstructed rather than original IPD. Original IPD allow for checking the plausibility of randomization sequences, verifying data integrity and consistency, fitting bivariate and copula-based models, which are among the preferred methods of assessing trial-level associations, adjusting the analyses for baseline prognostic covariates, and accounting for the fact that each within-trial surrogate outcome is estimated with error (84). Nevertheless, the specific goal of our analysis was to assess surrogacy at trial-level, and we used only data from high-quality RCTs. Therefore, an analysis based on original IPD is unlikely to substantially change our conclusions.

Furthermore, results from the analysis stratified by type of treatment administrated in the experimental arm deserve further validation to be considered conclusive. Finally, it could be possible that the shape of the curves for PFS and thus its surrogacy value could meaningfully be affected by specific inclusion criteria used to select patients populations enrolled in RCTs, especially the enrichment for molecular and clinical biomarkers predictive of response to immunotherapy, such as expression levels of PDL1, tumor mutational burden, smoking habits and gender. The lack of original IPD precluded conducting such types of granular analyses.

A relevant consideration should be highlighted. Regulatory agencies consider the HR_OS as the gold-standard measure for assessing treatment effects in RCTs and approving new drugs (85). As a result, we used it as the reference measure in our surrogacy analysis. In conclusion, our results showed that HR_PFS had a strong surrogacy value for HR_OS in comparisons testing ICI in combination with chemotherapy and moderate in comparisons testing ICI alone. Therefore, it should remain the reference surrogate endpoint in such contexts. Even in the presence of significant deviation from the PH assumption, the available evidence does not support the use of alternative endpoints, such as the mPFS, or metrics, such as the RMST.

Finally, two caveats should be highlighted. First, the available evidence does not allow for an adequate investigation of the value of HR_PFS as surrogate for other types of immunotherapy treatment strategies. Second, when using treatment effects on HR_PFS to predict those on HR_OS, the tendency to either underestimate or overestimate the final OS should be taken into account, depending on the type of treatment under investigation.

The lead authors (VB and FC) affirm that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.

The original contributions presented in the study are included in the article/ Supplementary Material . Further inquiries can be directed to the corresponding author.

IS: Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing. EP: Data curation, Formal Analysis, Investigation, Methodology, Project administration, Writing – review & editing. LP: Conceptualization, Investigation, Supervision, Visualization, Writing – review & editing. CO: Data curation, Formal Analysis, Investigation, Methodology, Validation, Visualization, Writing – review & editing. MM: Data curation, Formal Analysis, Validation, Visualization, Writing – review & editing. CS: Conceptualization, Data curation, Formal Analysis, Investigation, Methodology, Supervision, Validation, Visualization, Writing – review & editing. TP: Supervision, Validation, Visualization, Writing – review & editing. JC: Conceptualization, Investigation, Methodology, Supervision, Validation, Visualization, Writing – review & editing. GG: Conceptualization, Investigation, Methodology, Supervision, Validation, Visualization, Writing – review & editing. MP: Investigation, Methodology, Supervision, Validation, Visualization, Writing – review & editing. RG: Conceptualization, Formal Analysis, Investigation, Methodology, Project administration, Supervision, Validation, Visualization, Writing – review & editing. VB: Conceptualization, Data curation, Formal Analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing. FC: Conceptualization, Data curation, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing.