The manuscript was prepared by nine volunteer members of the BeCOME BIR working group, who are current or former PV and pharmacoepidemiology experts of various research affiliations (pharmaceutical companies, contract research organizations and independent consultancy) based in Canada, Europe, UK and US with specific experience in COVID-19 vaccine safety surveillance and/or safety studies. While this manuscript describes MAH experiences and lessons learned, objectivity is maintained through the use of published evidence, systematic consensus development, and integration of diverse stakeholder perspectives (see targeted literature review and consensus approach below for methodological details).

Rather than conducting a systematic review, a targeted literature review approach was selected and conducted by working group members to identify and synthesize key initiatives that generated BIRs to support the assessment of COVID-19 vaccine safety by the MAHs during the WHO’s Public Health Emergency of International Concern (ending on 5 May 2023) (World Health Organization, 2025). The search covered publications and reports available through 31 December 2023. This approach enabled a focused analysis and review of the challenges encountered by the MAHs during post-marketing monitoring, with particular emphasis on issues that emerged during large-scale vaccination campaigns. To this end, the working group used multiple strategies including (i) a targeted search and review of the scientific literature that were known, available and actively used by the MAHs during this specific period (ii) a targeted search and review of prespecified Public Health and Health Authority websites such as EMA, FDA, and CDC and (iii) a snowball search of the reference lists from the reviewed documents in the first two strategies. Additionally, BIR resources and knowledge were exchanged with other subject matter experts or task forces within trade associations including International Federation of Pharmaceutical Manufacturers and Associations (IFPMA) and Vaccines Europe – a specialized group of European Federation of Pharmaceutical Industries and Associations (EFPIA) (Bauchau et al., 2023). As the targeted literature component focused on identifying key BIR publications and datasets that were actively utilized by the MAHs, they were determined through the approaches outlined above. A formal systematic literature review was beyond the scope of this work; this is acknowledged in the limitations section.

Data were extracted on characteristics of the key initiatives from their reports, publications, dashboard or web apps, where available. These initiatives were selected based on their collaborative efforts to produce general population-based incidence rates across diverse geographic regions or countries, while establishing the infrastructure to integrate multiple data sources including health insurance, general practice, hospital, and registry data. For each initiative and where available, study characteristics (study countries, observation period, type of data sources, study population, number of AESI, etc.), information describing challenges, limitations, and recommendations for future practice were summarized.

This working group followed the nominal group technique (Harvey and Holmes, 2012) using structured group discussions over a period of 10 months (September 2022 - June 2023) to aid contribution and agreement on factors considered critical to best practices in generation and utilization of BIRs. The expert-based judgments were collated into preliminary sets of challenges and priorities, which were then presented for critical review at the inaugural BeCOME meeting held in June 2023 in Annecy, France (Bauchau et al., 2025). This meeting convened a diverse group of international experts representing a broad spectrum of stakeholders, including representatives from industry, academia, public health and research institutes, supra-national organizations, non-government organizations, and regulatory agencies. Following discussions and documented recommendations from this multi-stakeholder workshop, the final findings were organized into the following four domains:

Summary of key initiatives that generated BIRs during the COVID-19 pandemic

The BIR working group identified five key initiatives: ACCESS (Willame et al., 2023; Willame et al., 2021b), GVDN (Phillips et al., 2023), OHDSI (Li et al., 2021; Voss et al., 2023), BEST (Moll et al., 2023; Tworkoski et al., 2021), and Ontario studies (Nasreen et al., 2021; Nasreen et al., 2022). These initiatives generated BIRs for varying numbers of AESIs ranging from 8 (first Ontario study) (Nasreen et al., 2021) to 41 (ACCESS) (Williame et al., 2023; Williame et al., 2021b), with significant overlap among them. These BIRs were widely used by vaccine manufacturers to support COVID-19 vaccine safety monitoring during the pandemic period and continue to be used up to the present for other vaccines as well. While additional BIR publications from countries such as Korea, Scotland, and Australia were identified during this period (Huh et al., 2021; Cullen et al., 2023; Pillsbury et al., 2023), they were not considered as key initiatives due to their limited applicability for the MAHs. Table 1 summarizes these initiatives. The detailed description of these initiatives, including their respective reports, dashboards or publications on BIRs for selected AESIs, is provided in Supplementary Table S1.

ACCESS, GVDN, and OHDSI generated BIRs using different types of databases from multiple countries (primarily from Europe or North America), while BEST utilized exclusively US claims databases and Ontario studies generated BIRs from a single Canadian provincial linked health administrative database. These initiatives primarily targeted general pediatric and adult populations, with ACCESS and BEST additionally generating BIRs for specific subpopulations. For example, ACCESS covered subpopulations with underlying conditions who were at risk for developing severe COVID-19 and pregnant women (Willame et al., 2023; Willame et al., 2021b), and BEST included subpopulations such as nursing home residents and influenza vaccinated individuals (Moll et al., 2023; Tworkoski et al., 2021). Although all these initiatives generated BIRs stratified by age and sex separately, only the GVDN dashboard (Global Vaccine Data Network, 2023), OHDSI studies (Li et al., 2021; Voss et al., 2023), and Ontario studies (Nasreen et al., 2021; Nasreen et al., 2022) provided BIRs with dual stratification by both age and sex. Depending on data source and study design, these initiatives also generated BIRs stratified by other factors such as time periods (pre-COVID and peri-COVID), data source types (electronic health records [EHR], claims, registries), and/or healthcare settings (inpatients, emergency department visits, outpatients). Due to the urgent need to monitor COVID-19 vaccine safety, ACCESS and BEST released their reports in 2021 (Willame et al., 2021b; Tworkoski et al., 2021) and subsequently published their findings as manuscripts in 2022 (Willame et al., 2023; Moll et al., 2023). OHDSI and Ontario studies published their initial manuscripts in 2021 (Li et al., 2021; Nasreen et al., 2021), while GVDN published their BIRs in 2023 (Phillips et al., 2023) and simultaneously released them on their public dashboard (personal communication, May 2025).

The initiatives acknowledged several common challenges or limitations. These included the inability to validate case definitions since diagnostic codes do not always represent true disease occurrence, variable data quality and completeness across sources, and considerable variability (heterogeneity) in BIRs across populations, regions, and data sources. Some initiatives faced specific challenges related to their data sources. For example, governance issues (such as administrative constraints or lengthy governance approval process) affected French data access in the ACCESS initiative (Willame et al., 2021b), resulting in delay which was especially problematic during a pandemic response. In addition, the COVID-19 pandemic introduced specific challenges affecting BIRs, such as altered healthcare utilization patterns due to resourcing constraints, limits on elective procedures/encounters during the early pandemic months and increasing use of virtual healthcare. These factors could potentially lead to underestimation of less severe events. Overall, these challenges or limitations underscore the complexity of establishing reliable BIRs for vaccine safety monitoring during pandemic situations and emphasize the need to consider the context for generation of BIRs and careful interpretation when using these data for safety signal detection.

Additionally, the ACCESS, GVDN, OHDSI, and BEST initiatives offered high-level recommendations for generation and/or utilization of an appropriate BIR: (i) use the same database for comparing background and post-vaccination rates; (ii) select data sources with the most complete event identification; (iii) consider population characteristics when selecting appropriate BIRs; and (iv) account for temporal trends and seasonal variations when selecting BIRs.

The unprecedented scale and urgency of the COVID-19 vaccination campaign presented new levels of challenges for vaccine safety surveillance worldwide, although some similar challenges had been experienced during the 2009 H1N1 pandemic. Furthermore, the pandemic environment created multiple obstacles to effectively generate and utilize these rates for the extensive and evolving lists of AESIs. Built on the review of key initiatives that generated the BIRs during the COVID-19 pandemic, five major challenges were further identified through a combination of the BIR working group members’ professional experience and structured discussions and collaborative discussions held during the inaugural BeCOME meeting. These challenges are described below.

During the COVID-19 pandemic, regulatory requirements mandated COVID-19 vaccine manufacturers to conduct O/E analyses on numerous prespecified AESIs as part of their signal detection activities (European Medicine Agency, 2022). This deviated from the traditional application of O/E analyses, which typically focuses on signal refinement for safety concerns arising from various sources (Mahaux et al., 2016). The results and interpretations of these O/E analyses were required to be included in the monthly SSR to facilitate a rapid review of the safety profile and enable swift signal identification for COVID-19 vaccines.

As a critical parameter of O/E analysis, the BIRs should closely reflect the true incidence in the target population with minimal bias. Any inaccuracies in the BIRs can substantially distort the O/E ratio and consequently affect the interpretation of safety signals. The absence of standardized methods for selecting a suitable BIR led different organizations to apply varying criteria when choosing a BIR estimate, making it difficult to compare O/E results across different COVID-19 vaccines. However, stakeholders may use BIRs differently in O/E analysis, depending on their objectives. For example, vaccine manufacturers may conduct safety signal assessments during clinical development or post-marketing surveillance, while regulatory agencies use BIRs to monitor the safety profile of vaccines under development or newly approved vaccine products for regulatory decisions.

The selection of an appropriate BIR was further complicated by the heterogeneity in the estimates, as described above. EMA strongly recommended using ACCESS-derived BIRs in Europe (European Medicine Agency, 2022), but significant inter-source variability was observed, as illustrated in Figure 2. This variability introduced uncertainties in calculating the number of expected AESI cases, which consequently affected the O/E ratio. For instance, assuming 5 million COVID-19 vaccine doses administered and a fixed 42-day risk window for transverse myelitis, the minimum number of observed cases needed for a statistically significant O/E ratio varies substantially based on the selected ACCESS BIR: 4-6 reported cases when using lower BIRs (0.14-0.33 per 100,000 person-years) versus 8-11 cases with higher BIRs (0.57-0.92 per 100,000 person-years). This variability could lead to biased O/E estimates and incorrect safety signal assessments. Therefore, multiple sensitivity analyses across a range of BIR estimates were needed to address these uncertainties.

The availability of granular-level BIRs (e.g., age- and sex-specific BIRs) was also essential. For example, with myocarditis, one of the prespecified AESIs under investigation in the O/E analysis requested by regulators, age- and sex-stratified analyses proved particularly important. While overall O/E analyses indicated no elevated ratios, age-and sex-specific analyses revealed statistically significant O/E ratios within the younger age group, predominantly in men following the second dose of mRNA COVID-19 vaccines (European Medicines Agency, 2021). This finding underscores the importance of stratified BIR and granular vaccine exposure data for effectively generating evidence while fine tuning potential safety signals.

Beyond BIR selection challenges, it should be noted that O/E analyses also face other challenges and methodological considerations when estimating the “observed” component from passive reporting systems. These challenges and proposed improvements to current analytical methods are detailed in a separate publication by the BeCOME dedicated working group on O/E analyses (Serradell et al., 2025).

Background incidence rates serve as a cornerstone of vaccine safety surveillance, providing essential context for interpreting adverse events following vaccination. By establishing a critical baseline for O/E analyses, BIRs help distinguish true safety signals from coincidental events that would have occurred regardless of vaccination. This baseline provides the context necessary for effective safety signal assessment and supports evidence-based regulatory decision-making. Therefore, generating unbiased and precise BIRs in target populations is essential.

For a BIR to be unbiased and precise, the observed BIR value would have minimal uncertainty from the two fundamental sources of error: (i) random error (reduces with increasing sample size and is reflected in the width of the confidence interval) and (ii) systematic error (not affected by sample size and results in underestimation or overestimation of the true incidence) (Figure 4). Random error occurs by chance and in an unpredictable direction. On the other hand, systematic error causes BIRs to deviate from the true incidence in directions that may be predicted based on the type of systematic error.

To inform the best practices in estimating BIRs, two factors should be considered: (i) is a candidate bias certain to be present? and (ii) if present, is the candidate bias likely to introduce a small or large deviation from the true incidence? Generators and users of BIRs should therefore consider which major sources of error are likely to profoundly impact the results and which may have more inconsequential effects. For example, random error may well be the highest priority if estimating a BIR from a small database (e.g., for rare diseases), whereas if estimating a BIR using large databases, minimizing sources of systematic error may be most important. Factors such as the condition being studied, the context of care, the level of missingness, and the specific research question all play a role in determining which data source to prioritize when BIRs are heterogeneous (Clothier et al., 2024).

While random error can be reduced by increasing the sample size, when possible, systematic error requires careful consideration of many epidemiologic study design principles. Table 3 presents the BeCOME BIR working group expert opinion on a decision tool to reduce systematic error through study design that applies both in producing new BIRs from electronic healthcare databases and in assessing the appropriateness of published BIRs for a given use case. The suggested decision tool includes five methodological categories: geographic area, time period, sampling scheme, phenotype validation, and coverage of healthcare settings.

Researchers are encouraged to assess the relative importance of each category when generating or using BIRs. For example, if an AESI has recently undergone a major change in diagnostic practices, time period should be considered a key priority, since it would largely bias the estimated BIR to use data from prior to the diagnostic change. To aid in assessing BIR appropriateness, we suggest assigning a quality score of 1 (low weight) through 3 (high weight) to each methodological category based on the specific scenario and recommendation. Scores can thus range from 5 to 15 (5: all scores = 1; 15: all scores = 3). Higher total scores can be generally interpreted as having less bias compared to the optimal BIR in the given application. However, critical consideration of all sources of bias, the specific objective, and intended use beyond this simplified tool is necessary in downstream uses of BIRs. Table 3 includes an illustrative example assessing a published BIR for autoimmune hepatitis (AIH) (Gronbaek et al., 2020) to be used in a hypothetical O/E analysis in the US. In this example, the BIR of AIH received a total score of 11 which can be interpreted as appropriately robust to proceed with the O/E analysis, noting potential bias introduced through extrapolation of the BIR from the UK to the US and potential measurement error due to lack of validation of the coding algorithm used to define cases.

It is imperative that all factors in Table 3 be considered together and in the context of the use case. Specifically, it is possible that a published BIR may have a high total score suggestive of low risk of biases, yet there is a key consideration that deems the particular BIR not fit-for-purpose. For example, a high-quality study using recent population-based data may have produced a BIR that is substantially higher than other published rates. For applications to PV where the goal is to be conservative in order to capture safety signals, use of a high outlier BIR may not be recommended despite receiving a high-quality score.

Background incidence rates generated from systematic reviews with meta-analyses were not considered in scope for the application of Table 3. It may be preferable to use such systematic reviews as a source to identify a single primary study of high quality and use that in downstream analyses for the relevant population rather than directly applying the meta-analytic BIR, which may be heterogenous. Considering the multifactorial study design elements critically will aid in the goal of selecting and/or generating BIRs that are unbiased and precise and therefore serve as good proxies for the source population BIRs.

It should be noted that this tool may encounter feasibility challenges, particularly in LMICs where BIRs are frequently unavailable. This highlights broader ethical concerns about inequality in vaccine safety monitoring capabilities between regions. The BeCOME LMIC working group has identified challenges in resource-constrained settings, including inadequate infrastructure, insufficient trained personnel, and poor stakeholder coordination (Bauchau et al., 2025). To address these challenges, strategic priorities have been identified: securing sustainable human and financial resources for PV systems; developing context-appropriate training programs for local staff; and fostering coordination among multiple stakeholders (e.g., vaccine manufacturers, donors, non-governmental organizations, regulators, and health ministries) (Bauchau et al., 2025). While international collaborations and WHO support are crucial, they must focus on building local capacity rather than creating dependency. Implementing these strategies will enable local BIR generation and strengthen indigenous expertise, advancing equitable vaccine safety monitoring across all populations regardless of economic circumstances.

In addition to the recommendations in Table 3, BIR generation from electronic healthcare databases should adhere to established RWE study guidelines including, but not limited to, recommendations issued by the EMA, the FDA, and the CIOMS Working Group XIII (EMA, 2024; FDA, 2023; Hennessy et al., 2025). Digitalization of healthcare data has expanded over the past few decades, allowing proactive development of more granular subpopulation data during non-emergency periods. These efforts would enhance preparedness for future public health emergencies by enabling more nuanced and equitable safety signal assessment across diverse populations. Although valuable, even BIRs derived from robust, recent, population-based studies with validated case definitions have inherent limitations that must be considered in BIR utilization and interpretation, particularly when utilized for O/E analyses.

Several limitations warrant explicit acknowledgment. First, the methodological transparency of our targeted review process may be limited, as we did not employ a systematic review with predefined search strategies and inclusion criteria. However, this approach allowed us to conduct a focused analysis of challenges that emerged during large-scale vaccination campaigns. Second, our analysis predominantly relies on key initiatives where datasets originate primarily from high-income countries, which may not adequately represent the situations in resource-constrained settings such as LMICs. Third, some initiatives discussed depend on case definitions that have not been fully validated across diverse populations and healthcare contexts, potentially affecting the accuracy and comparability of BIRs. Finally, this paper presents experiences and perspectives from industry professionals. As stated in methods, we have endeavored to provide a balanced and objective assessment throughout this work. We encourage readers to interpret our findings and recommendations within this context, recognizing the unique insights that industry experience brings to understanding the practical challenges and solutions in vaccine safety monitoring. These limitations underscore the need for continued collaboration between industry, academia, regulatory agencies, and public health organizations to ensure comprehensive and unbiased approaches to vaccine safety surveillance.