A first key factor in assessing the trustworthiness of meta-analytic systematic reviews in CCAP is their transparency. Concerns about transparent reporting and ultimately reproducibility (i.e., the same analysis based on the same data yields the same results; Epskamp, 2019) and replicability (i.e., results at a later time support these conclusions; Epskamp, 2019) have accompanied the field of research synthesis from its inception (e.g., Mulrow, 1987). In part, these concerns sparked the development of the currently ubiquitously used (Caulley et al., 2020) and recommended (Kolaski et al., 2023) PRISMA guidelines (Page et al., 2021a) and their extensions that cover state-of-the-art reporting of protocols (Moher et al., 2015) or search strategies (Rethlefsen et al., 2021) among others. In their current version (March 2021; Page et al., 2021a) the PRISMA guidelines provide detailed guidance on transparent reporting of (meta-analytic) systematic reviews, including on the structure and content of the background information and study aims; the eligibility criteria; systematic search, screening and coding procedures; bias and certainty assessments; meta-analytic procedures; the structure and content of the discussion; conflict of interest statements; protocol preregistration; public availability of materials such as data and code; and using automation tools during the systematic review process (Page et al., 2021a, 2021b). Importantly, PRISMA does not advise on methodological quality (Kolaski et al., 2023). For example, PRISMA recommends reporting the number of authors involved in study screening or coding (Page et al., 2021a), but it does not recommend double coding or interrater reliability thresholds, which indicate sufficient methodological quality (or conduct) within dedicated tools (Shea et al., 2017).

In Psychology and related biopsychosocial disciplines, the ongoing replication crisis (Open Science Collaboration, 2015; Wiggins and Christopherson, 2019) has also increased attention to the transparency and reproducibility of meta-analytic systematic reviews. This has culminated in several meta-scientific investigations within Psychology (Lakens et al., 2017; Hohn et al., 2020; Maassen et al., 2020; Polanin et al., 2020; Steil et al., 2022), Clinical Psychology (López-Nicolás et al., 2022), forensic science (Chin et al., 2022), cognitive training (Sandoval-Lentisco et al., 2024), or biopsychosocial interventions (Page et al., 2018, 2021c; Nguyen et al., 2022) among others. These studies, typically focusing on transparency-related aspects in line with PRISMA, show a relatively consistent pattern: Broad aspects of transparency and those related to narrative sections were most commonly reported, including the background information (Steil et al., 2022), eligibility criteria (Hohn et al., 2020; Polanin et al., 2020; López-Nicolás et al., 2022; Nguyen et al., 2022; Steil et al., 2022; Sandoval-Lentisco et al., 2024), databases searched (Chin et al., 2022; López-Nicolás et al., 2022; Steil et al., 2022; Sandoval-Lentisco et al., 2024) and number of hits retrieved (Nguyen et al., 2022), or the meta-analytic model used (e.g., fixed-effect or random-effects model; Polanin et al., 2020; López-Nicolás et al., 2022; Nguyen et al., 2022; Sandoval-Lentisco et al., 2024). However, detailed information required for reproducibility were often missing, including full and replicable search strategies for all databases searched (López-Nicolás et al., 2022; Nguyen et al., 2022; Steil et al., 2022; Rethlefsen et al., 2024), specific screening and data extraction procedures (Polanin et al., 2020) (but see Sandoval-Lentisco et al., 2024), meta-analytic weighting procedures and estimators of variance components (López-Nicolás et al., 2022; Nguyen et al., 2022; Sandoval-Lentisco et al., 2024), or formulae, information, and raw data to calculate effect sizes (Lakens et al., 2017; Maassen et al., 2020; Polanin et al., 2020; López-Nicolás et al., 2022). Regarding open science practices, the availability of preregistrations or study protocols was rare (Tsujimoto et al., 2017; Polanin et al., 2020; López-Nicolás et al., 2022; Nguyen et al., 2022; Sandoval-Lentisco et al., 2024) and, if existent, deviations from them were often not declared (Tricco et al., 2016; Koensgen et al., 2019; Hu et al., 2021; Sandoval-Lentisco et al., 2025). In a related vein, the sharing of data in interoperable format as well as statistical syntax to reproduce analyses was virtually absent (Polanin et al., 2020; Chin et al., 2022; López-Nicolás et al., 2022; Nguyen et al., 2022; Sandoval-Lentisco et al., 2024).

Currently, it is unclear to what extent these findings also apply to meta-analytic systematic reviews within the field of CCAP. The closest investigation is a meta-review on transparency and reproducibility within meta-analytic systematic reviews of clinical psychological interventions as cited above (López-Nicolás et al., 2022), which concluded that the adherence to these practices was somewhat higher than in other fields (López-Nicolás et al., 2022). Supporting this, three meta-meta-analyses in CCAP, focusing on externalizing behavior problems (Mingebach et al., 2018; Weber et al., 2019) and ADHD (Türk et al., 2023) used a now outdated version of PRISMA guidelines (Moher et al., 2009) as a proxy for study quality (not transparency). They concluded that the adherence to these guidelines was generally adequate (Mingebach et al., 2018; Weber et al., 2019; Türk et al., 2023). However, their scoring approach prevented identifying areas of high or low transparency, and current PRISMA guidelines would allow for a closer examination of finer-grained transparency practices as mentioned above.

Additional transparency considerations may be needed for meta-analytic systematic reviews in CCAP. This notion has been put forward for research syntheses involving pediatric populations, leading to the planned (Kapadia et al., 2016) but, as of November 2024, not yet published PRISMA guideline for such syntheses (PRISMA-C). This guideline is primarily geared toward systematic reviews for pediatric medical intervention research. However, preparatory works related to the development of this guideline (Kapadia et al., 2016; Farid-Kapadia et al., 2017a, 2017b) suggest several reporting domains relevant to CCAP: These include explicitly mentioning the pediatric population in the title, referencing (and justifying) this population within the study objectives, providing replicable age-related eligibility criteria, the reporting of age-specific search terms or tested pediatric search filters, and addressing the implications for children, adolescents, and caregivers specifically within the discussion section (Kapadia et al., 2016; Farid-Kapadia et al., 2017a, 2017b). Given these CCAP-specific considerations, it is crucial to assess their current uptake within meta-analytic systematic reviews in leading journals in the field.

A second key factor in assessing the trustworthiness of meta-analytic systematic reviews in CCAP, besides transparency, is their methodological quality (Kolaski et al., 2023). Its assessment (but not the quality itself) necessarily hinges upon transparent and reproducible reporting, but they are not equivalent, as described above. AMSTAR 2 (Shea et al., 2017), the recommended tool within psychology and health-related research (Kolaski et al., 2023), assesses methodological quality within several domains. These include providing a preregistered protocol; the adequacy of the eligibility criteria, literature search, data extraction, and meta-analytic procedures; a duplicate study selection and data extraction procedure; justifying the exclusion of individual studies; conducting risk and publication bias assessments; using appropriate meta-analytic methods, and reporting on potential conflicts of interest within primary studies and the review itself (Shea et al., 2017).

Contrasting the lack of evidence on transparency, the methodological quality of (meta-analytic) systematic reviews in CCAP has been assessed more frequently, likely due to the increased popularity of overviews of reviews in psychology and health-related research that require a quality assessment (Gates et al., 2022). These quality-related assessments as part of overviews of reviews have examined diverse areas of CCAP, such as treatments for anxiety (Byrne et al., 2023; Galán-Luque et al., 2023; Zbukvic et al., 2024), depression (Espada et al., 2023; Zbukvic et al., 2024), eating disorders (García-Fernández et al., 2023), conduct problems (Romero et al., 2023), or consequences of sexual abuse (Sánchez De Ribera et al., 2020); mental health correlates of cyberbullying (Kwan et al., 2020); mental health burdens among sexual and gender minority youth (O’Shea et al., 2024); or the impact of paternal mental health on child development (Scarlett et al., 2024).

Except for one study (Zbukvic et al., 2024), these investigations documented poor methodological quality within most (Kwan et al., 2020; Sánchez De Ribera et al., 2020; Byrne et al., 2023; Romero et al., 2023) or all (Espada et al., 2023; Galán-Luque et al., 2023; García-Fernández et al., 2023; Scarlett et al., 2024; O’Shea et al., 2024) included systematic reviews (notably not all of them meta-analytic). Domains of consistently low quality included the lack of preregistration (Kwan et al., 2020; Sánchez De Ribera et al., 2020; Byrne et al., 2023; Espada et al., 2023; Galán-Luque et al., 2023; García-Fernández et al., 2023; Romero et al., 2023; Scarlett et al., 2024), failure to justify excluding individual studies (Kwan et al., 2020; Sánchez De Ribera et al., 2020; Byrne et al., 2023; Espada et al., 2023; Galán-Luque et al., 2023; García-Fernández et al., 2023; Romero et al., 2023; Scarlett et al., 2024), and no assessment of risk of bias (Kwan et al., 2020; Sánchez De Ribera et al., 2020; Byrne et al., 2023; Galán-Luque et al., 2023; Romero et al., 2023), publication bias (Sánchez De Ribera et al., 2020; Byrne et al., 2023), or primary studies’ funding sources (Kwan et al., 2020; Espada et al., 2023; Romero et al., 2023; Scarlett et al., 2024). These investigations paint a rather bleak picture of the methodological quality of systematic reviews within certain domains and areas of CCAP. They also highlight the need for larger-scaled investigations into this matter (most overviews included fewer than 15 systematic reviews) and clearer guidance on improving methodological quality.

A third key factor in assessing the trustworthiness of meta-analytic systematic reviews is the consistency of statistical reporting within null hypothesis significance testing (NHST; Sterling, 1959). Typically, this involves checking whether reported p-values are consistent with their accompanying test statistics and degrees of freedom (Bakker and Wicherts, 2011; Nuijten et al., 2016). Such inconsistencies can indicate various underlying issues, ranging from simple mistakes over misunderstandings of statistics to engaging in questionable research practices, such as incorrectly and deliberately rounding p-values below 0.05 (John et al., 2012; Nuijten et al., 2016). Regardless of their cause (and of discussions about whether to abandon NHST), such reporting inconsistencies can lead to false conclusions about the degree to which the data support the presence or absence of an effect (e.g., an effect is erroneously deemed to be significant), hinder reproducibility, and preclude calculating correct effect sizes based on test statistics (Nuijten et al., 2016; Polanin and Nuijten, 2018; Nuijten and Polanin, 2020).

Research has mainly focused on statistical reporting inconsistencies within primary studies (e.g., Bakker and Wicherts, 2011; Nuijten et al., 2016; Olsen et al., 2019; Nuijten and Wicherts, 2024), indicating a disconcertingly high prevalence of statistical reporting errors. For example, a re-analysis of over 250,000 p-values from eight major psychology journals (1985–2013) using the R package {statcheck} (Nuijten and Epskamp, 2024) revealed at least one inconsistency between test statistics, degrees of freedom, and p-values in about 50% of publications as well as at least one gross inconsistency (i.e., altering conclusions based on statistical significance) in about 13% of publications (Nuijten et al., 2016). Recently, statcheck has been adapted to include test statistics used in meta-analyses (e.g., the Q statistic), enabling investigations of meta-analytic systematic reviews (Nuijten and Polanin, 2020).

Some might argue that the primary purpose of a meta-analysis is to provide an estimate of a meta-analytic summary effect (or an exploration of the heterogeneity around this effect), and hence, NHST is of little interest within meta-analytic systematic reviews, including those within CCAP. However, NHSTs in meta-analytic studies are common (previous works documented averages between 29 and 68 NHSTs per article; Polanin and Pigott, 2015; Polanin and Nuijten, 2018) and, for example, routinely used to assess whether the data support the presence or absence of moderator effects, statistical heterogeneity, or, in small samples, even meta-analytic summary effects (Polanin and Pigott, 2015; Nuijten and Polanin, 2020). Furthermore, arguments about inconsistent statistical reporting as litmus tests for deeper underlying issues within a study (Nuijten et al., 2016), as outlined above, similarly apply to meta-analytic investigations (Nuijten and Polanin, 2020). Meta-scientific studies have revealed that 39% of meta-analyses within the social sciences included at least one inconsistent result, whereby 8% contained a gross inconsistency (Polanin and Nuijten, 2018; Nuijten and Polanin, 2020). However, no systematic investigation has yet focused on meta-analytic systematic reviews in CCAP.

Transparency, quality, and statistical consistency of reporting are crucial for the trust that researchers, clinicians, policymakers, and other stakeholders within the field of CCAP place in meta-analytic systematic reviews. Further, in seeking guidance for future studies, applied meta-analysts within the field of CCAP may want to know in which areas the reporting transparency or quality could be improved to guide their own efforts.

Within this meta-review we aim to answer three research questions (RQs) that all refer to meta-analytic systematic reviews from the field of CCAP within leading journals of this discipline published between 2022 and 2024.

As a secondary aim, we plan to gain further insights into transparency-promoting factors by exploring associations between reporting transparency as well as methodological quality, statistical consistency, and other transparency-related variables within our review sample, as meaningful associations between these or related factors have been documented within the literature (Page et al., 2016, 2018; Chin et al., 2022; López-Nicolás et al., 2022; Nguyen et al., 2022). We plan to assess associations between:

Reporting transparency and methodological quality

As a tertiary aim, we plan to provide practical recommendations for researchers within CCAP by highlighting “what works” (i.e., areas where reporting transparency and methodological quality are high) and “what could work better” (i.e., areas where more transparent reporting or higher quality is advisable for the field), which we will incorporate into our discussion section.

All study and Supplementary materials (S1–S5) for this study protocol can be found in Supplementary material as well as via https://osf.io/qhrau/ (including preprint publication). We will make all data (in machine-readable format) and all materials (including R code to reproduce all analyses, figures, and tables) for the final publication available via this project on OSF, alongside a preprint publication on PsyArXiv. Regarding guideline adherence, there are currently no specific guidelines available for conducting meta-reviews (Dal Santo et al., 2023). Thus, we adhered to the PRIOR statement for overviews of reviews (Gates et al., 2022) together with the PRISMA (Page et al., 2021a), PRISMA-P (Moher et al., 2015) (checklist provided within Supplementary material S1) and PRISMA-S guidelines where applicable (Rethlefsen et al., 2021). The final publication will similarly adhere to these guidelines where applicable. Note that, in contrast to PRIOR, we did not include “overview of reviews” in our title (Gates et al., 2022), as this would imply a content focus rather than a methodological focus inherent to meta-reviews by our definition (Biondi-Zoccai, 2016; López-Nicolás et al., 2022).

During preparation of this manuscript, ChatGPT (versions 3.5 and 4.0; OpenAI, 2024) was used on author-generated text [MS] within narrative sections (Introduction, Discussion) to edit for typography, grammar, clarity, and brevity only, akin to a human co-author (Nakagawa and Lagisz, 2024). Prompts included varieties of the statement “Please edit the following paragraph for brevity and clarity and point out what was corrected: [author-generated text copied in full].” MS incorporated these suggestions as seen fit and, together with the other authors, takes full responsibility for the content of this manuscript. No new text segments, ideas, or references were generated with the help of ChatGPT, any other large language model, or any other generative artificial intelligence assistance.

This meta-review focuses on meta-analytic systematic reviews from the field of CCAP. To translate a definition of this field into precise eligibility criteria, we will first present our definition of CCAP for the purpose of this meta-review and highlight conceptual differences from related disciplines.

In line with the American Psychological Association (Division 53), we define CCAP as a subdiscipline of psychology that examines mental (emotional, behavioral, and developmental) disorders in children and adolescents, including research on the development, maintenance, diagnosis, and clinical-psychological as well as psychotherapeutic interventions/treatment of disorders with a focus on infants, children, and adolescents across all age groups (Brown et al., 2023). We consider developmental psychopathology to be a subarea of CCAP, as it deals with developmental processes that contribute to the etiology, onset, and chronification of mental disorders in children and adolescents, mostly focusing on examining risk and protective factors and trajectories of internalizing and externalizing problems throughout childhood and adolescence in the general population, i.e., including subclinical phenotypes (Barry et al., 2023).

The field of CCAP shares considerable conceptual overlap with Child and Adolescent Psychiatry as well as (Pediatric) Clinical Neuroscience, which are not within the scope of this meta-review. Regarding Child and Adolescent Psychiatry, we base our definition on American Academy of Child and Adolescent Psychiatry (2021) and Schlesinger et al. (2023), which defines it as a medical specialty area that primarily deals with medical aspects of mental disorders and drug/psychopharmacological therapies among children and adolescents. We define the field of (Pediatric) Clinical Neuroscience as an interdisciplinary field combining aspects of neuroscience and clinical medicine: It encompasses the study and treatment of diseases of the nervous system, focusing on the application of neurobiological findings to clinical problems (Kandel et al., 2021) with a typical focus on biological markers, such as genetic (e.g., DNA analyses), neuroimaging (e.g., fMRI), electrophysiological (e.g., EEG), biochemical (e.g., salivary cortisol), and cardiovascular (e.g., heart rate variability) markers.

A concise summary of our eligibility criteria is reported in Table 1. We give additional clarifying information below. We [MS, MZ] conceptualized a first draft of our eligibility criteria based on theoretical and methodological considerations. As recommended (Foo et al., 2021), we then refined these criteria through piloting the screening of a random sample of 55 records (10%) retrieved by our search strategy (see below), as well as through full-text assessments of published meta-analytic systematic reviews from the field of CCAP (Alozkan-Sever et al., 2023; Davis et al., 2023; Manuele et al., 2023; Harris et al., 2024; Lin et al., 2024; Tung et al., 2024; note that these did not need to fulfill final eligibility criteria).

We focus on meta-analytic systematic reviews from leading journals (based on content and citation metrics criteria) within the field of CCAP to examine reporting practices of the (potentially) most impactful publications within this field. This may represent an upper threshold of the uptake of good reporting practices, but substantial reporting non-transparency has also been identified within top-ranking journals in Psychological Science (e.g., Polanin et al., 2020).

This is the first (notably unfunded) investigation into transparency, methodological quality, and statistical consistency of reporting within meta-analytic systematic reviews within CCAP. Thus, we deem this approach reasonable to gain a first impression of the field and deduce practical recommendations for researchers, as well as to lay the foundation for future works within this field (e.g., meta-reviews assessing time trends or prevalence estimates on reporting practices within the field). Since there is not one single, non-invitational flagship journal dedicated to publishing systematic reviews within CCAP that we could sample from (a commonly used approach within other psychological disciplines, e.g., Polanin et al., 2020; Siegel et al., 2022), and we deemed a keyword search as too broad (and therefore impractical, if not impossible) to identify eligible systematic reviews, we identified eligible journals through a three-step procedure based on expert opinion and journal metrics, as detailed in Supplementary material S2.

Ultimately, this yielded a list of seven journals we considered to be representative of the top-ranking journals publishing meta-analytic systematic reviews within the field of CCAP (in alphabetical order): Child Development, Clinical Psychology Review, Development and Psychopathology, European Child & Adolescent Psychiatry, Journal of Child Psychology and Psychiatry, Journal of the American Academy of Child and Adolescent Psychiatry, and Psychological Bulletin. Together, these journals cover a broad spectrum of clinical topics and areas of application relevant to CCAP and encompass all age groups specified in our eligibility criteria, i.e., from infancy to adolescence/young adults. They publish widely disseminated research syntheses (and primary studies) from around the world using a diverse range of methods and are highly relevant to an international audience (with a broad ethnic background), both in terms of readership and authorship. All journals are indexed in Web of Science (Core Collection, as subscribed by the University of Vienna). Thus, we rely on this database as our single information source.

Our full search string for Web of Science (Core Collection, as subscribed by the University of Vienna) is reported in Table 2. We combined (1) all seven unique journal ISSNs (see also Chin et al., 2022, for a similar approach and Table 3 for journal ISSNs) with (2) keywords relating to systematic reviews or meta-analyses (appearing in titles, abstracts, or keywords). These records were then (3) limited to publication years (early access) 2022, 2023, or 2024. Within sets (1) and (2), search terms were combined using the OR-operator, whereas sets (1), (2), and (3) were combined using the AND-operator. No other filters or limits were used. The search string was constructed by the first author [MS] and peer-reviewed for accuracy by the second author [SF]. We did not validate the search string against a pre-specified set of studies, because the keywords used within our search were limited and eligible articles largely defined through their publishing outlets. On September 25, 2024, this search strategy yielded 550 records. After peer review for this protocol has been concluded, we will re-run this search. If peer review has been completed prior to 2025, we will update our search after January 1, 2025, for records from 2024.

To derive search terms for set (2), we screened previous meta-reviews (Koffel and Rethlefsen, 2016; Cortese et al., 2019; Wayant et al., 2019; Hohn et al., 2020; Chow et al., 2021; Chin et al., 2022; López-Nicolás et al., 2022; Nelson et al., 2022; Nguyen et al., 2022; Steil et al., 2022; Wong and Bouchard, 2023; Pellegrini et al., 2024; Rethlefsen et al., 2024; Sandoval-Lentisco et al., 2024) for keywords and deemed variants of “systematic review” and “meta-analysis” useful for the purpose of this study. We supplemented these with variants of the term “meta-regression” to also include meta-analytic systematic reviews that primarily focused on exploring heterogeneity within effect sizes. We piloted including the term “quantitative review” (as used, for example, in López-Nicolás et al., 2022) as well as “research synthes*” (own considerations), which yielded no additional records and were dropped.

We deliberately chose to search within titles, abstracts, and keywords (vs. a title-only search) and to include variants of the keyword “meta-analysis” (vs. “systematic review” only) because we wanted to avoid limiting our search to those meta-analytic systematic reviews that fulfill the first PRISMA—and thus transparency—criterion, namely to include the term “systematic review” within the title (Page et al., 2021a). Our journal- and study type-based search strategy will also retrieve systematic reviews unrelated to children and adolescents (e.g., from general clinical psychology journals such as Clinical Psychology Review) and/or clinical psychology (e.g., from developmental journals such as Child Development or multidisciplinary journals such as the Psychological Bulletin). We will screen out these reviews manually (see below), a strategy that is feasible given the moderate number of records.

We opt for a classic two-stage screening procedure (Page et al., 2021a), including screening of titles and abstracts, followed by full-text screening. We piloted this approach based on the same sample of 55 records that was also used to pilot our eligibility criteria. Title and abstract screening will be undertaken by three authors [MS, SF, JR], who will assess titles and abstracts of every record independently. Records will be advanced to the full-text stage via liberal acceleration (Page et al., 2018), i.e., every record deemed as eligible for full-text assessment by one (or more) author(s) will be assessed as a full-text. For descriptive purposes, we will report our interrater reliability (percentage agreement) between all three authors and between pairs of authors.

Full-text assessment will also be undertaken by [MS, SF, JR], in such that we will assess full-texts’ final eligibility prior to data extraction. Final eligibility will be based on consensus among all three authors. The senior author [MZ] will arbitrate in cases where no consensus can be reached or additional expertise is needed. For descriptive purposes (our inclusion decision is consensus-based in all cases, Shea et al., 2017), we will report initial interrater reliability (percentage agreement between all three authors and pairs of authors). References of records that could not be retrieved in full or that were excluded at this stage will be published together with the reason for their exclusion. For inclusion decisions that we made on a case-by-case basis, we will provide a separate table (possibly within Supplementary material) including a detailed reasoning for each case. The number of excluded records by category will also be made available within the PRISMA flow chart.

For both steps, we will not use any automation tools or rely on crowdsourcing or previously “known” assessments (Page et al., 2021a). We do not expect needing to translate abstracts or articles because of our journal-based search strategy, including English-language articles only. We also do not expect having to contact authors for information necessary to determine eligibility. We will publish a PRISMA flow chart.

We estimate that our final sample of eligible meta-analytic systematic reviews will include about 60 reviews. We base this estimate on our piloting procedure, where we [MS, SF] piloted our eligibility criteria as well as our systematic search procedure, as recommended (Foo et al., 2021). Based on a random sample of 55 out of 550 retrieved records (10%; September 25, 2024), we deemed six as eligible for inclusion, thus totaling 6 * 10 = 60 studies based on extrapolation.

Data collection for research questions (RQ 1; transparency) and (RQ 2; quality) will be undertaken by [MS, SF, JR]. We will randomly split eligible records into two groups. The first author [MS] will code all records once and the second authors [SF, JR] will each code records from one of the two groups. Thus, every eligible record will be coded twice and independently. We will resolve discrepancies through discussion and/or arbitration by another author (SF and JR for records they did not code; MZ for all records). We piloted our data collection process as well as our data collection sheets (see Section ‘Data Items’) on two meta-analytic systematic reviews that were deemed eligible within piloting of the screening procedure (England-Mason et al., 2023; Farkas et al., 2024). In addition, we will code three meta-analytic reviews jointly to reach a common understanding of the codebook and coding materials, in particular regarding quality- and transparency-related items. Through regular meetings between the raters regarding overarching coding questions, we will further strive to ensure a common understanding of the codebook and minimize systematic rater bias or discrepancies.

We will assess and report interrater reliability for every variable (percentage agreement as recommended within AMSTAR-2; Shea et al., 2017) and report on the mean (M), standard deviation (SD), and range across all variables. Because our assessment of transparency-related variables is based on a self-developed, PRISMA-based checklist (see section 2.8.1.2), we will also report the prevalence-adjusted, bias-adjusted Kappa (PABAK; as calculated within Stevenson and Sergeant, 2025) for all categorical transparency-related variables and pairs of raters and the intraclass correlation (ICC) for continuous variables (as calculated within Gamer et al., 2019). We will not contact authors for additional information, because we are interested in the transparency and quality of reporting as is. We will not use any automation tools for data collection regarding these two research questions.

Data collection for research question 3 (RQ 3; consistency of statistical reporting) will be carried out through statcheck (Nuijten and Polanin, 2020; Nuijten and Epskamp, 2024). Specifically, we will rely on the CRAN version at first date of analysis for the R package {statcheck} (Nuijten and Epskamp, 2024). Statcheck assesses whether reported p-values are consistent with their accompanying test statistics by recalculating them (Nuijten and Polanin, 2020). As basis for further analysis via statcheck, we will download .html (preferable) or .pdf-versions of all eligible studies within our meta-review, which will then be read into R and analyzed through functions provided within the package (see section ‘Analysis’). Statcheck can only reliably detect reporting inconsistencies within (correctly) APA-formatted texts (Nuijten and Epskamp, 2024). At time of writing, four out of seven journals included within our meta-review require APA-formatting (Table 3), although this requirement often refers to formatting of references specifically (and not test statistics). We will still include all articles in our analysis (i.e., we will attempt to analyze all downloaded articles through statcheck functions), because it is possible that test statistics from other journals might be formatted in a way that is consistent with APA style and thus analyzable. However, it could be possible that we can only report on results from four journals.

We will code three groups of items and generate a fourth group of items via statcheck: (1) items related to general publication and study information, (2) transparency-related items (necessary for RQ 1), and (3) quality-related items (necessary for RQ 2) in the form of the AMSTAR 2 tool for assessing methodological quality (Shea et al., 2017). The full codebook is available as a Word document within Supplementary material S3, and as the raw coding sheet (Excel; excepting AMSTAR 2 items which will be coded verbatim within a separate sheet not included for copyright reasons) within Supplementary material S4. We will also analyze items from group (4), which are not manually coded but generated through statcheck.

Our aim is to assess meta-analytic systematic reviews from seven leading journals within the field but not to generalize to a wider population (which would, ideally, include a random sample). Thus, we will generally present our results descriptively and will not, for example, provide confidence intervals (e.g., for proportions). Notable exceptions include our exploratory analyses, where we will provide p-values and/or confidence intervals for the sake of completeness, but emphasize the descriptive nature of our analyses. We assume a two-tailed p-value of < 0.05 as our threshold for significance for these purposes.

See Supplementary material S5.

Upon submission to the journal and concurrent preregistration as a preprint (November, 2024), the status of our project was the following: Preliminary searches as well as piloting of the eligibility criteria and the screening procedure in a randomly drawn sample of 55 records had been conducted. The coding procedure had been piloted on two meta-analytic systematic reviews. We had not started the main search, screening, and coding procedures.

Upon reception of reviews (June, 2025), the status of the project is as follows: We have completed the systematic search and title-and-abstract screening and established a final sample of eligible meta-analytic systematic reviews. Coding for all three main pillars of this meta-review (transparency, quality, and statistical consistency) is currently underway. Reviewers’ comments did not address any steps completed in the meantime that would need to be repeated.

We plan to conduct further investigations based on the review sample obtained from this study. First, we plan to analyze adherence to open science practices of eligible meta-analytic systematic reviews in more detail within a separate report, including practical recommendations to improve adherence to open science principles within meta-analytic systematic reviews. Second, we plan to assess the computational reproducibility of meta-analyses within our sample, similar to López-Nicolás et al. (2024). Third, we plan to compare review protocols and preregistrations with published reviews, similar to Sandoval-Lentisco et al. (2025). All of these investigations will be preregistered.