Assuming that the number of cases during the surveillance period for each group follows a Poisson distribution, with parameter λ₁ for the investigational vaccine group and λ₀ for the placebo group, then c₁ is binomially distributed B(c, π) conditional on the total number of cases c where c = c₀+c₁, and with π=T1λ1T1λ1+T0λ0. The relationship between π and VE is expressed as

Where, the constant r = T₁/T₀ is the ratio of the total person-time at risk in the vaccine and placebo groups.

The interval estimation for VE is then converted to the problem of constructing CI for a single Binomial proportion of π. If (L_π, U_π) is the 100(1−α)% CI for π, then the 100(1−α)% CI for VE can be derived as:

Assume that π follows a prior beta distribution beta(a, b) with parameters a and b. The mean of the prior distribution is u = a/(a+b) and the variance is u(1−u)/(a+b+1). The posterior distribution of π given c₁ infected cases in the vaccine group conditional on the total number of cases c is again a member of the beta family, denoted beta(a + c₁, b + c − c₁). The resulting limits of the Bayesian interval are the α/2^th and (1−α/2)^th quantiles of the posterior beta distribution, respectively. The 100(1- α)% equal tailed Bayesian interval is given by

Notably, when a = b = 0.5, the above interval corresponds to the 100(1- α)% equal-tailed Jeffreys interval.

According to Pfizer's protocol, a minimally informative beta prior, beta (0.700102, 1), which is centered at θ = 0.4118 (VE=30%), was applied to construct the posterior probability for π (16). The 100(1- α)% equal tailed interval for π is given by

The interval for π can then be converted into the CI for VE by using Equation (2).

This method is based on normal approximation for a logarithmic transformation of the incidence rate ratio, the variance of log(IRR) can be written as the sum of variances of the log incidence rates in vaccine group and placebo group.

By a Taylor's series approximation, the variance of log(c_x/T_x) is var(cx)/cx2, and since c_x follows a Poisson distribution, an estimate of var(c_x) is given by c_x. This yields the following large sample two-sided 100(1−α)% CI for the incidence rate ratio (14):

The above CI can also be obtained from the Poisson regression using the maximum likelihood estimate (MLE) of the parameters with treatment as a fixed effect and time at risk for each subject as offset in the model.

The 100(1−α)% CI for VE can then be expressed as:

When c₁ = 0 or c₀ = 0, the approximate Poisson interval doesn't exist.

At a fixed vaccine efficacy ve with given c, r, and a, the coverage probability is defined as the proportion of the two-sided (1−α) CI containing the true ve. An ideal interval estimation method has the coverage probability equal to (1−a) under all parameter settings, e.g., a 95% CI contains the true ve with 95% probability. However, this is not always the case for data that follow discrete distributions.

Given the total number of cases c, the possible combinations of the number of cases in the vaccine group c₁ = 0, 1, 2, …, n and the number of cases in the placebo group c₀ = c−c₁ are exhaustive. The binomial probability for each combination of (c₁, c₀) is given by (cc1)πc1(1-π)c-c1 where π=1-11+(1-VE)r according to Equation (1). The coverage probability can then be estimated by summing the probabilities of all combinations for which the resulting interval contains the true ve. For each configuration of (ve, c, r, a), the coverage probability can be calculated using the following formula.

where I(k, ve) is an indicator variable that equals to 1 if the CI includes ve when c₁ = k, otherwise I(k, ve) = 0.

VE is always demonstrated by comparing the lower limit of the CI with a given margin. When the non-coverage probability at the lower tail is greater than the nominal level a/2, it may lead to inflation of the type I error rate. Owing to the asymmetry of the Poisson and Binomial distributions, the non-coverage probability at the two tails is always unequal. For a given method under each configuration of (ve, c, r, a), the non-coverage probability at the lower tail can be calculated as the sum of the binomial probabilities of all combinations for which the lower limit of the resulting CI is greater than the true vaccine efficacy ve.

Where, indicator J(k, ve) = 1 if the lower limit of the CI is greater than the true ve when c₁ = k, otherwise J(k, ve) = 0.

The expected interval width can be calculated using a formula similar to that of the coverage probability and non-coverage probability. For each configuration of (ve, c, r, a), it can be calculated as follows.

where Len(k, ve) is the interval width of the resulting CI when c₁ = k. Because ve ranges from −∞ to 100%, the interval width was set to 2 when the lower limit is lower than −100%; otherwise, it was calculated as (upper limit—lower limit).

Reported VE always implicitly assumes that the diagnostic test has a sensitivity and specificity of 100%. However, this assumption is invalid according to the reported sensitivity and specificity of COVID-19 diagnosis testing (17, 18). The estimated VE can be biased by an unknown amount when the sensitivity or specificity of the diagnostic test is <100% (19, 20). Diagnostic tests are rarely totally accurate. We assume that the diagnosis of COVID-19 with a specificity of 100%. However, the sensitivity for identifying a COVID-19 infection varies based on the type and quality of the specimen obtained and duration of illness at the time of testing (21). Ridgway et al. (18) reported that sensitivity of a single NAAT test ranged from 82 to 97% among symptomatic patients utilizing 34348 SARS CoV-2 NAAT results from two health systems (18).

The sensitivity of a diagnostic test is the conditional probability that the test will be positive (Test+) if the disease is present (Disease+).

Specificity is the conditional probability that the test will be negative (Test–) if the disease is not present (Disease–).

When the sensitivity and specificity of the test are known, the adjusted number of cases, also called the actual number of cases C_actual, is calculated according to the following formula.

Where, C_observed, Se, and Sp denote the observed number of cases, sensitivity, and specificity, respectively.

When the specificity is 100%, the above formula can be written as C_actual = C_observed/Se.

Let s₁ and s₀ denote the sensitivities in the Vaccine and placebo groups, respectively. The number of the observed cases in the vaccine and placebo groups can then be derived as c_1o = c₁s₁ and c_0o = c₀s₀, respectively. The impact of under-sensitivity of diagnosis test on the non-coverage probability at the lower tail and expected lower limit are investigated. The expected lower limit is calculated in a similar way with expected interval width.

The reported vaccine efficacy data from the following three COVID-19 studies will be used to illustrate our results. The examples are sorted by the vaccine name in an alphabetical order.

Example 1 (BNT162b2 vaccine): This is an ongoing multinational, placebo-controlled, observer-blinded, pivotal efficacy among persons 16 years of age or older. Participants were randomized in a 1:1 ratio to receive two doses, 21 days apart, of either placebo (N = 17511) or the BNT162b2 vaccine (N = 17411). The primary endpoint is VE against confirmed Covid-19 with onset at least 7 days after the second dose. There were 170 cases of symptomatic COVID-19 included in the primary efficacy analysis (3).

Example 2 (ChAdOx1 vaccine): This is an ongoing single-blind phase 3 trial in Brazil among adults 18 years and older. Participants were randomized to control group (N = 2025) or ChAdOx1 group (N = 2063) in a 1:1 ratio. All participants were offered 2 doses with administration 4 weeks apart. The primary endpoint is VE against symptomatic COVID-19 more than 14 days after the second dose of vaccine. There were 131 cases of symptomatic COVID-19 included in the primary efficacy analysis (4).

Example 3 (HBO2 vaccine): This is an ongoing randomized, double-blind, phase 3 trial in the United Arab Emirates and Bahrain among adults 18 years and older. Participants were randomized to receive 1 of 2 inactivated vaccines developed from WIV04 (N = 13 459) and HB02 (N = 13 465) strains or Placebo (N = 13 458); they received 2 intramuscular injections 21 days apart. The primary endpoint is VE against laboratory-confirmed symptomatic COVID-19 that occurred at least 14 days after a second vaccine dose. There were 142 symptomatic COVID-19 cases (95 cases in Placebo group; 26 cases in WIV04 group; 21 cases in HB02 group) and we randomly selected HB02 group for the analysis (6).

The two-sided coverage probability over the ve overlapped by c are presented in Figure 1 using line plots with the horizontal and vertical axes indicating the ve and coverage probability, respectively. Similarly, the non-coverage probability at the lower tail is plotted in Figure 2. In addition, a descriptive summary of the coverage probability and non-coverage probability at the lower tail is provided in Table 1.

As shown in Figure 1, the coverage probability is not fixed at the nominal level 1−α = 95% for all methods. As c increases, the coverage probability approaches the nominal level 95%, and oscillation decreases but still exists even for large values of c. Under all parameter settings, the exact conditional method has a two-sided coverage probability on or above the nominal level 95%. It is clear from the plot that the oscillatory is more significant for Bayesian intervals using either the non-informative Jeffreys prior or minimal informative Pfizer prior, when compared to the other three methods. As indicated in Table 1, the minimum coverage is 86.8% for the Bayesian method with the Jeffreys prior and 86.2% for the Pfizer prior. The Pfizer method performs slightly better than the Jeffreys prior and its low-coverage mainly comes from the region where the true ve is close to 100% when c reaches 60. The approximate Poisson method has minimum coverage below 90% when the true ve is close to 100% and above 92.5% when the true ve is not close to 100%. The mid-p interval has minimum coverage probability above 92.5% under all scenarios.

As shown in Figure 2, the non-coverage probability of the exact conditional method is no greater than the nominal level α/2 = 2.5% under all scenarios. As shown in Figure 2, the Bayesian intervals using either non-informative Jeffreys prior or minimal informative Pfizer prior has a higher non-coverage probability at the lower tail, when compared to the other three methods. The non-coverage probability can be three times higher than the nominal level 2.5% for the Jeffreys prior, and two times higher than the nominal level 2.5% for the Pfizer prior. For the mid-p method, the non-coverage probability at the lower tail is always below 5%. The non-coverage probability of the approximate Poisson interval at the lower tail is desirable, with a maximum value of 2.8% as shown in Table 1. However, the coverage probability of Poisson method is low when the ve is close to 1, as shown in Figure 1, which indicates that the non-coverage of Poisson method mainly comes from the upper tail.

The expected interval width of two-sided 95% CI constructed by the five methods for c = 10, 20, and 60, and VE from 0.5 to 1 is presented in Figure 3 using box plot. A descriptive summary of the expected interval width is provided in Table 1. The exact conditional and approximate Poisson methods provide wider interval widths under all scenarios, and the approximate Poisson method has slightly narrower interval compared to the exact conditional method. Bayesian methods are consistently narrower than the other methods. The pattern of the interval widths for all five methods become increasingly more similar as c increases. When c reaches 60, the interval width of all five methods is very close.

The exact conditional method is often considered too conservative, with an unnecessarily wider interval width. In this section, we demonstrate that this is not always the case. Figure 4 plots the cumulative percentage of the non-coverage probability at the lower tail of the observed VE interval using the exact conditional method with a variable ratio of sensitivity. Figure 5 shows the expected lower limit of the observed VE interval with a variable ratio of sensitivity. As shown in Figure 4, when the ratio of sensitivity is below one, the non-coverage probability may exceed the nominal level 2.5%, and the lower the ratio of sensitivity, the higher the probability that the non-coverage probability exceeds the nominal level 2.5%. Moreover, Figure 5 shows that the lower the ratio of sensitivity, the higher the expected lower limit of the observed VE interval especially when the actual VE is close to 0.5. This indicates that the reported CI constructed using the number of observed cases tends to be overly optimistic when the ratio of sensitivity in the vaccine group to that in the placebo group is below one.

In this section, we illustrate our results using the reported efficacy data for COVID-19 vaccines. Three studies were selected (see Example COVID-19 Vaccine Efficacy Trials), and their primary efficacy endpoint were analyzed using the five methods presented in this paper, with the results presented in Table 2.

As indicated in Table 2, the VE interval constructed by the five methods are very similar when total number of infected cases is large (example 1). Among the five methods, the exact conditional method is always wider than the other methods, followed by the approximate Poisson method. The mid-p interval is slightly wider than the Bayesian intervals when the total number of cases is small (example 2).

Assume that the sensitivity of the diagnosis test is 90% in the vaccine group and 100% in the Placebo group, then the estimated VE% using actual number of cases in examples 1–3 are 94.5, 60.2, and 75.6, respectively. They are lower than the estimated VE% presented in Table 2, which indicates that the reported VE% and corresponding CI will be overly optimistic when the sensitivity of the diagnosis test in the vaccine group is lower than that in the Placebo group. The exact conditional method will be affected less than other methods due to wider interval width.

This method was converted from the Clopper-Pearson interval for a single proportion. This is the only method that guarantees two-sided coverage probability and non-coverage probability at the lower tail under all scenarios. The Clopper-Pearson interval is often considered to be too conservative for Binomial proportions (22–25), and Ewell (14) showed that the exact conditional interval for VE is overly conservative with a wide interval width (14). In our simulations, the exact conditional method produced an interval that was consistently wider than that of the other methods, especially when the total number of cases was small. When the total number of cases reaches 60, the interval width of all five methods are very similar.

This method was converted from the mid-p interval of a single proportion. Although the minimum coverage is below the nominal level, it exhibits slight oscillation among the methods except for the exact conditional method. The minimum coverage is closer to the nominal level, compared to approximate Poisson method and Bayesian methods. The mid-p interval is slightly wider than the Bayesian intervals and narrower than the approximate Poisson and exact conditional intervals.

The approximate Poisson method performs well when the VE is not close to one. When the VE is close to one, this method may lead to very low coverage, but non-coverage mainly comes from the upper tail. As a result, the lower limit of this method is conservative relative to the mid-p and Bayesian methods. This method provides a slightly narrower interval than the Clopper-Pearson method but wider than that of the other methods. Joshi, Geroldinger, and Jiricka et al. (26) showed that a Poisson interval does not exist when the number of infected cases is zero in the vaccine group (26).

The coverage probability of Bayesian methods with either the Pfizer prior or Jeffreys prior behaves more erratically than the other methods, and may lead to inflation of the type I error rate under some parameter settings because of low coverage.

In conclusion, we showed that the coverage probability for VE intervals is not fixed at the nominal significance level for all methods, due to the discreteness of the count data.

For the exact conditional method, it equals the nominal level or more while for the rest methods it may below the nominal level, even for a large number of total infected cases. In addition, our investigation shows that the exact conditional interval is too wide when the total number of infected cases is small, and hence, may not provide an informative CI. The narrower interval obtained using the Bayesian, mid-p, and approximate Poisson methods is at the cost of not preserving the nominal significance level. As a result, we suggest a mid-p method should be used when the total number of cases is below 60 to obtain a narrower interval width with a slight loss of coverage. When the total number of cases reaches 60, the exact conditional method has a similar interval width to that of the other methods without a loss of coverage.

Furthermore, our investigation of the effect of under-sensitivity of diagnosis testing shows that coverage probability decreases when the sensitivity in the vaccine group is lower than that in the placebo group. In such cases, the exact conditional method is preferred to guarantee coverage.