Peer-reviewed scientific papers have become near-synonymous with credible scientific communication, both for primary research and for evidence syntheses (Goldbeck-Wood, 1999). While these papers benefit from peer review, they are also largely static outputs, and are not necessarily the best format for continuous updates, dynamic data exploration, or secondary data use.

COVID-19 has highlighted these challenges for evidence synthesis, as we struggle to keep up with an ever-evolving evidence base (Convenes, 2022). However, thousands of COVID-19 systematic reviews were registered–in some cases, more systematic reviews than primary studies (Pérez-Gaxiola et al., 2021). Of these, many reviews rapidly became outdated, and a small fraction had to be curated as the highest-quality efforts (McMaster Health Forum, 2021a).

At the same time, the pandemic has incubated a new generation of evidence synthesis products, including and beyond the research paper (McMaster Health Forum, 2021b; Convenes, 2022). We have seen more regularly-updated systematic reviews and rapid reviews designed to be responsive to decision-maker needs (Macdonald et al., 2020). Exceptional science journalism has helped synthesize scientific advances for the lay public (Yong, 2021). Finally, several dashboards have synthesized complex evidence across a variety of sources, such as the Oxford Covid-19 Government Response Tracker (Hale et al., 2021), the COVID-NMA systematic review dashboard of clinical trials (Boutron et al., 2020), and our project, SeroTracker (Arora et al., 2021). These evidence synthesis products are designed to meet user and decision-maker needs: curating, analyzing, and disseminating heterogeneous data in near-real-time.

In this piece, we will discuss our experiences building SeroTracker, a systematic review, dashboard, and data platform for SARS-CoV-2 seroprevalence studies. We propose a user-centered model for evidence synthesis, describe the challenges we faced in implementing this model, and reflect on questions for the scientific community to consider as these efforts become more commonplace.

Early in the pandemic, many scientists started to publish seroprevalence studies–population-based antibody testing investigations that measure the prevalence of SARS-CoV-2 antibodies and help map the extent of SARS-CoV-2 infection and immunity (Bobrovitz et al., 2020). However, the results from these studies were published across research papers, preprints, and the news, with no unified data source.

Recognizing this gap, we built SeroTracker, a dashboard and data platform for SARS-CoV-2 serosurveys (Arora et al., 2021). SeroTracker rigorously curates and critically appraises SARS-CoV-2 seroprevalence data through an ongoing systematic review. To provide up-to-date evidence, we search for new published papers, preprints, and government reports weekly and update our dashboard continuously. We provide interactive analyses and open data on SeroTracker.com, aiming to enable rapid access and integration into other modeling and research efforts. The SeroTracker platform today includes data from over 3400 seroprevalence studies in 137 countries and territories, and has users across sectors including research groups, global public health organizations, the media, and the private sector (SeroTracker, 2022).

Reflecting on our work, we believe that SeroTracker's value has resulted from adopting a user-centered approach. We started by understanding users' needs through interviews and meetings, which led us to combine ideas from systematic reviews and data visualization to build a novel research product. We continuously engaged end-users throughout our design process, through both formal and informal quality improvement cycles. The user-centered design thinking approach has long been successfully applied in engineering (Razzouk and Shute, 2012), and is reflected by other widely-used health data projects such as Our World In Data and OpenPrescribing (Curtis and Goldacre, 2018; Mathieu et al., 2021).

We faced numerous challenges while building SeroTracker, which raised key considerations for future user-centered evidence synthesis products. Here, we reflect on these considerations, our approaches to addressing them, and their implications for subsequent efforts.

We argue that design thinking should be used as a core tool for evidence synthesis. By engaging data users and decision-makers as “end users” of research products, we can thoughtfully apply the evidence synthesis process to create useful, targeted outputs, whether it be dashboards, living papers, or other scientific products. Ultimately, this approach will make evidence synthesis more effective in its end goal–to help governments, researchers, and institutions act quickly on new information.

It is essential to consider how to preserve the strengths of the existing scientific communication paradigm in a user-centered evidence synthesis approach. These well-established strengths include the rigor of pre-registration and reporting guidelines (e.g., PROSPERO, PRISMA), the validation and legitimacy provided by peer review, and clear dissemination mechanisms by which knowledge users can learn of new relevant work (Carpenter et al., 2014; Page et al., 2021).

Novel evidence synthesis products demand new standards for rigor and impact evaluation (Convenes, 2022). We posit several questions that we hope others can help explore. What does it mean to peer review a dashboard? What is the best way to measure the impact of these digital tools? Counting the number of page views is (perhaps too) simple, but it is much more challenging to measure impact through informing policy. How can we appropriately fund this sort of research infrastructure, which often does not directly fit the mandate of traditional academic funding sources (Hershberg, 2022)? And how do we make new products easily discoverable, to help researchers publicize their work and knowledge users find useful tools?

Through SeroTracker and other initiatives, we have seen the benefits of groups collaborating on widely accepted, open-access core datasets. Many different users can apply these datasets to their own research questions, reducing duplication of effort and enabling scientists to focus on new analyses. Closed datasets privilege those with access to said data. In contrast, the open nature of this data creates opportunities for any researcher to participate–particularly those without traditional forms of power.

We expect user-centered evidence synthesis to become even more relevant as the volume of data grows, researchers become more technologically adept, and health emergencies demand us to close the time gap between data and decisions (Convenes, 2022). In the wake of COVID-19, we have the opportunity to move toward user-centered evidence syntheses purpose-built for maximum impact.

TY and RA jointly conceptualized, wrote, and revised this piece. Both authors contributed to the article and approved the submitted version.

Separate from this work, SeroTracker receives funding from the Public Health Agency of Canada through Canada's COVID-19 Immunity Task Force, the World Health Organization Health Emergencies Programme, the Robert Koch Institute, and the Canadian Medical Association Joule Innovation Fund. These funding sources had no role in this study, including conceptualization, writing, or the decision to submit for publication. This manuscript does not necessarily reflect the views of the World Health Organization or any other funder.

RA was previously a Technical Consultant for the Bill and Melinda Gates Foundation Strategic Investment Fund, is a minority shareholder of Alethea Medical, and was a former Senior Policy Advisor at Health Canada. TY was a former Senior Policy Advisor at Health Canada. Each of these relationships is unrelated to the present work.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.