Description: A well-crafted review question is essential for any evidence synthesis because it establishes the core focus and system boundaries for the review process (Pullin et al., 2009). The key elements must be clearly defined.

Example: Zahra et al. (2021) do not state a review question or hypothesis but rather vaguely state a problem and an objective: “[…] this study was designed to determine the nexus between biochar and compost […].” The lack of clarity in the question means that the review’s system boundaries are unclear.

Solution: Authors aiming for a systematic review should separate and specify their question’s key elements using a framework such as the PI/ECO framework for comparative intervention/exposure questions (Population, Intervention/Exposure, Comparator, Outcome) (see, e.g., Higgins et al., 2019; CEE, 2022).

Description: Defining an a priori protocol that outlines the methods is critical when conducting evidence syntheses to improve transparency, comprehensiveness, reproducibility and to avoid mission creep and selective study choices (bias). It also helps to ensure focus and efficiency throughout the review process, precluding major deviations from the review objectives, search strategy, inclusion, and appraisal criteria. An a priori protocol reduces bias by explicitly specifying procedures that will be followed at each step, reducing the opportunity for subjectivity and transparently outlining the planned methods in detail (CEE, 2022).

Example: Viana et al. (2022) and Palomo-Campesino et al. (2018) lack a protocol and details in their methods description, for example, specifics about searched databases within the Web of Science platform and the handling of consistency and subjectivity among reviewers. As in many cases, the authors argue that their review is systematic because of the use of PRISMA. However, this is a common misconception and demonstrates that the authors are confusing reporting guidance for conduct guidance. The review by Delitte et al. (2021) is particularly problematic because it lacks both method description and protocol. The lack of internationally accepted best practices and minimum standards for conduct and reporting raises the risk of bias, and insufficient details impede the assessment of the review’s validity.

Solution: Authors aiming for evidence synthesis should design an a priori protocol outlining planned methods for searching, screening, data extraction, critical appraisal, and synthesis. Deviations from the protocol during study conduct should be described and justified. Organizations such as Cochrane, Campbell Collaboration and Collaboration for Environmental Evidence (CEE) provide standards and guidance for conducting high-quality evidence synthesis. In addition, authors can apply internationally recognized review reporting standards, such as ROSES (Reporting standards for Systematic Evidence Syntheses) (Haddaway et al., 2018b) or PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) (Page et al., 2021). They assist authors in ensuring that they accurately document all their actions in a clear language, whereas review protocols guide review conduct. Ideally, authors can avoid shortcomings by having their protocols peer-reviewed and published beforehand in suitable journals (e.g., Environmental Evidence), obtaining feedback from methodological experts during the peer-review process. However, authors facing monetary and/or time constraints may consider registering protocols using PROCEED’s free service (JKI and CEE, 2023). Protocols are not peer-reviewed but undergo an editorial process with guidance and possible revisions before acceptance (Pullin, 2023).

Description: Searching in evidence syntheses requires a detailed approach involving tried-and-tested search strings that carefully balance sensitivity and specificity. Validation against benchmark papers is essential to ensure the comprehensiveness of the searches. Reviewers should ensure that they search across multiple, relevant, and diverse sources for evidence, including bibliographic databases (e.g., Scopus, LILACS), theses and grey literature repositories and other relevant organizational websites. Neglecting grey literature may introduce publication bias, especially for negative or non-significant results that are typically underrepresented in peer-reviewed journals (Franco et al., 2014). Commonly used search facilities, such as Google Scholar, may be deemed inappropriate because they may introduce publication bias, among other limitations (Møller and Jennions, 2001; Aguillo, 2012; Bramer et al., 2016; Kugley et al., 2017; Gusenbauer and Haddaway, 2020). The exclusion of non-English languages can lead to bias (Konno et al., 2020a). Reviews that examine regionally specific topics must consider local language evidence - both in terms of search design and study inclusion.

Example: Dagunga et al. (2023) may have overlooked valuable sources by omitting grey literature, particularly in “agroecology” research, where it can serve as an important repository, as highlighted by D’Annolfo et al. (2014) and Hussain et al. (2014). Malhi et al. (2021) exclusively relied on Google Scholar for their search, an unrepeatable resource from which results cannot be exported in full.

Solution: Including or consulting information specialists or librarians in the review process is an important means of avoiding common mistakes when constructing a search strategy (Rethlefsen et al., 2015; Meert et al., 2017). To enhance comprehensiveness and representativeness, a suite of different search methods should be employed, including: multiple databases, forward and backward citation chasing, web searches, manual checking of non-indexed journals (Gusenbauer, 2024), and calls for submission of relevant studies. Having a peer-reviewed a priori protocol enhances the validity of the search strategy by facilitating constructive feedback.

Description: Systematic eligibility screening or study selection requires a priori and well-defined criteria that are consistently applied to all studies identified in the search process. This is to minimize errors or selection bias, which occurs when the included articles fail to be representative of the evidence base (McDonagh et al., 2013). Consistency of screening decisions and resolving disagreements among the review team are essential to reduce subjectivity in study inclusion. The methods used for consistent screening decisions must be described and reported transparently.

Example: Dagunga et al. (2023) do neither report on consistency checking nor define what a “smallholder farmer” is. It is not clear which studies are included, those farmers who have a certain smallholding size, or those that use the label “smallholder.” The main terms “agroecology” and “resilience,” as well as their measurements, are only defined in the results section as part of the analysis. As central definitions and inclusion criteria for studies (e.g., type of measurement and empirical vs. non-empirical studies) should be defined a priori, this indicates that the review might not be entirely systematic. Ricciardi et al. (2021) aimed at comparing farms with similar management systems but lack details on eligibility criteria for such management systems, how many screeners were involved and how they dealt with consistency. Authors may have conducted a noble study, but unreported numbers and excluded articles during screening (Muneret et al., 2018; Jat et al., 2020) or the absence of a list of included studies (Bai et al., 2022), limit transparency and trust in the results and conclusions.

Solutions: The review question should be used as the basis for defining eligibility criteria. Evidence reviews applying PICO/PECO-type research questions may use these as key elements [Population(s), Intervention(s), Comparator(s), Outcome(s)] for defining the eligibility criteria. Additional entries, such as study design, language, geography, and study design should also be clarified (Bernes et al., 2018; Macura et al., 2019). Involving multiple reviewers and piloting the screening process for disagreements further enhances consistency (Waffenschmidt et al., 2019). Before full-text screnning begins, disagreements must be resolved, screening decisions kept consistent, and criteria clearly defined for operational clarity. This process is crucial at multiple stages of the review. Transparency and replicability are ensured by providing details on the eligibility criteria application, the complete list of screened articles, included studies, and reasons for article exclusion at the full-text stage.

Description: Critical appraisal aims to assess the internal validity (quality) and external validity (generalizability) of individual studies included in a review. This process is important because research studies vary considerably in terms of their reliability, for example, study design (CEE, 2022), and overlap with review aims. Evidence syntheses should be based on reliable primary research, but moderate-quality research can be included; however, all studies should be weighted accordingly and tested for robustness (CEE, 2022). The validity assessment process could be influenced by implicit bias (i.e., the reviewer’s perspective) (Konno et al., 2024) and should hence not depend on a single reviewer. It should be based on tried-and-tested frameworks for critical appraisal, for example, tools for assessing risk of bias in randomized control trials, such as RoB2.0. It is essential to provide detailed information regarding the decision-making process during critical appraisal.

Example: Abdalla et al. (2018) and Liu et al. (2018) omitted critical appraisal or failed to report a validity assessment; hence, their review’s reliability cannot be assessed by the reader. Only three systematic reviews in our CEEDER assessment are rated “high-quality” (Gold) for their critical appraisal (Bernes et al., 2018; Eales et al., 2018; Meurer et al., 2018). All three studies clearly presented a critical appraisal process and validity assessment.

Solution: Critical appraisal should be planned and tested a priori to ensure consistency among the review team. The critical appraisal criteria should reflect what the review team deemed to be critical variables influencing the reliability of the study, as for example in Eales et al. (2018). The study validity rating should avoid complex scoring systems, but instead use a simple rating, such as low, high, and unclear validity, to avoid introducing errors (Higgins et al., 2011). Various established and verified critical appraisal tools are available (e.g., Frampton et al., 2017).

Description: Data extraction and coding in evidence synthesis involves systematically retrieving relevant information from included articles in a standardized manner, including qualitative and/or quantitative data. This process requires a description of the data extraction method and coding of the studies. This includes variables that are extracted as data and others that are coded (metadata). Additionally, it should be outlined how the process was tested for repeatability and consistency between reviewers (Frampton et al., 2017). The documentation of each extraction step and the approach to acquiring missing or unclear data are crucial for transparency (CEE, 2022).

Example: Adil et al. (2022) partly report how they extracted data but do not report specific steps, criteria, or procedures followed during the data extraction process, nor how the data extraction was validated to minimize errors and bias in the results. Additionally, the extracted data are not presented anywhere, despite a data availability statement stating the opposite. In comparison, Blouin et al. (2019) provide a more detailed account of their data extraction process. However, more explicit information on the validation process for the extracted data and the approach taken for handling missing or insufficient data are lacking.

Solution: Good data extraction practices for enhanced objectivity, reproducibility, and transparency involve: (i) fully documenting the type of data to be extracted and extraction method; (ii) employing a data extraction form pretested by two reviewers; (iii) including appendices with full extracted data/meta-data from each primary study; (iv) specifying pre-analysis calculations and data transformations, e.g., calculation of effect sizes; (v) ensuring consistency through multiple reviewers’ testing for human error, and in the best case, two independent reviewers extracting data from each study (CEE, 2022).

Description: Data synthesis combines and analyses data from studies included in the evidence synthesis to answer the review question, draw conclusions, or identify patterns. This involves the synthesis of study characteristics through narrative, quantitative (e.g., meta-analysis), qualitative (e.g., thematic synthesis) or mixed methods synthesis (Sandelowski et al., 2010, 2012; Pearson et al., 2015; Higgins et al., 2019; CEE, 2022). Vote counting as a form of quantitative synthesis should not be used, as it is misleading and neglects each study’s validity, statistical power, precision and magnitude (Friedman, 2001).

Example: Dittmer et al. (2023) and Cozim-Melges et al. (2024) used vote counting to show that agroecology and diversified farming practices positively affect climate change adaptation and enhance biodiversity. The identified interventions’ effect size, magnitude, and statistical power are missing. Dhaliwal et al. (2019) opted for a narrative synthesis. Nonetheless, the effect of agricultural addendums in a comparative way is missing, given the lack of a quantitative synthesis of the included studies. On the other hand, Tuttle and Donahue (2022) clearly state that their evidence synthesis is a narrative synthesis. This is accompanied by tables and visualizations that summarize the information retrieved from the included articles.

Solution: The synthesis of data should follow tried-and-tested methods. In quantitative synthesis, vote counting should never be used instead of meta-analysis. In cases where meta-analysis is not feasible (e.g., due to data heterogeneity), a narrative synthesis should be employed to describe the evidence base. All included studies’ characteristics, outcomes, and validity should be listed and tabulated. The chosen methods/models for synthesis should be justified, assumptions clarified (e.g., for missing data), and decisions made explicit (e.g., choice of effect measure). Following the call for ‘Open synthesis’ (Haddaway, 2018a), input data for quantitative synthesis and analytical code should be provided where not legally restricted or data summaries where restrictions exist. Authors should transparently outline potential limitations and uncertainties of the analysis approach, detailing their potential impact on the overall conclusions.

We recognize the challenging nature of systematic reviews, which demand substantial resources, expertise, and time. Moreover, the pressure in academic research to frequently produce and publish (Van Dalen, 2021; Becker and Lukka, 2023) can drive researchers to cut corners to meet deadlines, thus reducing methodological rigor and validity. In resource-constrained scenarios, researchers may employ a ‘rapid review’ or ‘scoping review’ methodology, incorporating systematic review elements. However, the risk of bias can increase (CEE, 2022; Sabiston et al., 2022). Pursuing methodological rigor in evidence synthesis must be a shared responsibility among authors, peer reviewers, and editors to enhance the quality, reliability, and credibility of evidence synthesis in agri-food science. Despite challenges such as lack of awareness, limited training opportunities, and time constraints, it is critical to foster commitment within the research community to adhere to best practices. This requires willingness to learn, appreciation of the efforts involved in rigorous reviews, and collaboration among stakeholders. To avoid unreliable syntheses, all actors in the research community should be cognizant of common mistakes.

Many researchers, as well as postgraduate and PhD students, have not been trained in best-practice evidence synthesis. Providing comprehensive and accessible training to researchers can address the current gaps in awareness, skills, and best practices. Several organizations, such as Collaboration for Environmental Evidence and Cochrane, offer training opportunities (CEE, 2024c; Cochrane, 2024). However, fostering a proactive approach requires the active involvement of universities and other research institutions in incorporating such training programs into their curricula and continuing education.

Peer review and publication of review protocols are regarded as best practices and can enhance the quality of evidence synthesis thanks to the critical evaluation of the review’s design, methods, and planned analyses. Thus, redundancies and wasted research efforts can be avoided before conducting the study. Overall, this practice promotes transparency, reduces bias, and enhances credibility.

The lenient acceptance of papers claiming to be “systematic” is turning “systematic” into a meaningless buzzword devoid of its original rigor and purpose. To enhance gatekeeping at publication, editors and publishers must establish robust peer-review processes that ensure that evidence syntheses meet established methodological and reporting standards. Journals can provide authors with clear guidelines that require transparency and reporting in sufficient detail, as well as adherence to recognized protocols (e.g., PRISMA, ROSES). Additionally, editors can enlist expert reviewers with knowledge of evidence synthesis methodology to thoroughly assess submitted papers.