We searched PubMed for systematic reviews (with or without meta-analyses) published between 2010 and 2019 in five general cardiology/cardiovascular medicine journals with the highest impact factors according to the 2019 Clarivate Analytics Journal Impact Factor (JIF) list (Circulation [23.6], European Heart Journal [22.7], Journal of the American College of Cardiology [20.6], Circulation Research [14.5], and JAMA Cardiology [12.8]). Although the scope of “Circulation Research” relates mainly to basic cardiovascular research, it publishes systematic reviews of clinical studies on emerging biological and molecular interventions in different cardiovascular diseases. Therefore, it was deemed relevant for this analysis. The detailed search strategies, used in the present analysis, are illustrated in Supplementary Table 1.

A systematic review was defined as per the MeSH database as “A review of primary literature in health and health policy that attempts to identify, appraise, and synthesize all the empirical evidence that meets specified eligibility criteria to answer a given research question,” while a meta-analysis was defined as “Works consisting of studies using a quantitative method of combining the results of independent studies.” Screening for eligible studies was performed by two independent authors (AO and AIA) in two subsequent steps: title and abstract screening followed by full-text screening.

The following data were extracted from eligible systematic reviews: Type (direct, individual-patient data [IPD], network meta-analysis [NMA], others), country of origin (classified into single and multinational collaborations), number and type of included studies (randomized trial, observational and diagnostic test accuracy studies), number of searched databases and filters used during the search, whether the reviewers searched protocol registration sites, conference abstracts, and reference lists of retrieved reports, and used risk of bias assessment tools. We further extracted data on the primary outcome, its statistical significance, and the presence of statistical heterogeneity. Further, we checked the sources of funding and classified them if present into governmental/academic and private sources.

In addition, we assessed whether the eligible reviews performed publication bias assessment and the methods used (Egger's test, Begg's test, Funnel plots, correction by trim and fill method, others). For reviews that used any of these tests, we extracted the results of publication bias assessment. For studies that did not perform such assessment, we searched the article for the reason of not performing this analysis.

We used the “Assessing the Methodological Quality of Systematic Reviews (AMSTAR)” checklist online tool to assess the quality of included systematic reviews (https://amstar.ca/Amstar_Checklist.php). The AMSTAR tool was developed by the Universities of Ottawa, Bristol, Bond, and Toronto to assess the methodological quality of systematic reviews. It consists of 16 questions (majority: yes/no answers) that assess the performance of different steps of the systematic review process as determination of inclusion criteria, screening and data extraction, risk of bias and publication bias assessments, as well as analysis methods. It generates a score for each assessed review as either of high, moderate, low, or critically low quality (5).

We used R software (version 3.6.3 for Windows) to conduct the statistical analysis. Data were expressed as count (proportion, 95% confidence interval [CI]) or median (interquartile range) for categorical and numerical data, respectively. We used Chi-Square, ordinal logistic, and Poisson regression tests to evaluate the association between methodological characteristics, publication bias assessment, AMSTAR quality scores, and citations (after adjusting for the journal and publication year). We used the Wilson method without continuity correction to calculate 95% confidence intervals for proportions (24). A p-value < 0.05 was considered statistically significant.

It was not appropriate or possible to involve patients or the public in the design, or conduct, or reporting, or dissemination plans of our research.

Among 659 retrieved search results, we excluded 278 after title/abstract screening. Eventually, we identified 352 eligible systematic reviews that were published in the target cardiology journals between 2010 and 2019. In systematic reviews that included a meta-analysis, the most frequent type was direct head-to-head comparison, followed by IPD meta-analyses, while only few studies used NMA. Only eight studies performed a qualitative systematic review without meta-analysis. The eligibility criteria in the assessed meta-analyses focused on RCTs (N = 164; 46.6%), observational studies (N = 104; 17.6%), or both RCTs and observational studies (N = 65; 16.8%); Table 1. The majority of eligible reviews (N = 193) were the result of a multinational collaboration; other reviews were published most frequently from the United States (N = 80), followed by the Netherlands (N = 13), Italy (N = 12), and Canada (N = 12).

Sixty-four reviews (18.2%, 95% CI: 14.5–22.6) registered/published their protocol. The median number of searched databases was 3 (IQR: 2, 3). The frequently searched databases were Medline (N = 310; 88.1%), EMBASE (N = 196; 55.7%), Cochrane Central Register of Clinical Trials (N = 168; 47.7%), and Web of Science (N = 49; 13.9%). A full list of the used databases and search engines in the assessed reviews is illustrated in Supplementary Table 2. Among the assessed systematic reviews, 130 (36.9%) used search filters, most frequently limiting search results by language, date of publication, human species, and study type. Manual screening was employed in 219 reviews (62.2%), while 48 (13.6%) reviews added conference abstracts to their eligibility criteria. The median number of included studies was 13 (IQR: 7, 30); Table 1.

Among the retrieved systematic reviews, 208 (58.4%, 95% CI: 53.9–64.1) reported their risk of bias assessment in adequate details using validated tools and 177 (52.3%, 95% CI: 45.1–55.5) examined the risk of publication bias. Among the latter, publication bias was present in 35 reviews. The commonly used methods to assess for publication bias were Funnel plots, Egger's and Begg's tests. Among reviews that did not assess for publication bias, only 21 reviews cited reasons, that were mostly related to the small number of included studies.

The primary outcomes in the assessed reviews were commonly related to mortality and stroke. In 274 (79.6%, 95% CI: 75.1–83.6) reviews, the comparison for the primary outcome was statistically significant and statistical heterogeneity was significant in 167 (48.5%, 95% CI: 43.3–53.8) meta-analyses.

Of the assessed studies, 87 (24.7%, 95% CI: 20.5–29.5) did not report on whether they received funding or not, 33 (9.4%, 95% CI: 6.8–12.9) reported receiving no funding, 70 (19.9%, 95% CI: 16.1–24.4) reviews reported receiving governmental/academic funding, 120 (34.1%, 95% CI: 29.3–39.2) reported private funding from pharmaceutical companies, while 42 (11.9%, 95% CI: 9–15.7) reviews received funding from both sources. The distribution of the number of citations for the assessed reviews was clearly skewed (median 92, IQR: 38, 196); the majority was the publications with smaller number of citations; Table 2.

The number of reviews with critically low, low, moderate, and high quality scores were 78 (22.2%, 95% CI: 18–27), 171 (48.6%, 95% CI: 43.2–53.9), 85 (24.1%, 95% CI: 19.6–28.8), and 18 (5.1%, 95% CI: 3.1–8), respectively. This means that the majority of assessed meta-analyses (71%, 95% CI: 65.7–75.4) had low or critically low quality. Only 221 (62.8%, 95% CI: 57.6–67.7) reviews stated clearly that they followed the EQUATOR reporting guidelines (PRISMA, MOOSE, etc.).

We found a significant association between the existence of publication bias and statistical heterogeneity of the primary outcome (OR, 3.93; 95% CI, 1.43–10.85), but not significance of the primary outcome (OR, 1.79; 95% CI, 0.58–5.51). Interestingly, regression analysis showed that meta-analyses with complex design (e.g., IPD), those that included RCTs, included a larger number of studies, had a significant primary outcome, adhered to EQUATOR reporting checklist, or had a higher AMSTAR quality score were associated with a significantly higher citation count (p < 0.05) after adjusting for journal and publication year. On the other hand, we found no significant association between AMSTAR quality score and publication journal, registering the review protocol (OR, 1.33; 95% CI, 0.69–2.56), multinational collaboration (OR, 1.09; 95% CI, 0.73–1.61) or receiving private funding (OR, 1.02; 95% CI, 0.03–29.43). Interestingly, the publication year positively predicted AMSTAR quality score (OR, 1.03, 95% CI: 1.02–1.04).

Although our study, to our knowledge, provides the first comprehensive analysis on the methodological characteristics and gaps of systematic reviews in the cardiology literature, some limitations should be addressed. First, journal selection in the present study was based on the impact factor. While we recognize the limitations of such metric, the selected journals are indeed the most reputable in our field. Second, our analysis focused more on methodological rather than statistical properties of systematic reviews, such as analysis models or effect estimates. Third, the number of reviews with significant publication bias was relatively small to fully analyse the possible contributing factors that can be improved in the systematic review methodology.