To identify existing holistic assessments, we searched two online databases: Web of Science (WoS) and CAB Abstracts. These two databases were chosen because they are large, high-quality, relevant databases and accessible to the research team. We explored these databases using a search string with three main components. The first component focused on terms relating to different sustainable agrifood systems approaches (building on approaches identified in HLPE, 2019), the second concerned their application to agrifood systems, and the third focused on terms relating to some form of assessment: TI = (Holistic OR Socio-ecological OR Social-ecological OR Sustainab* OR Agroecolog* OR Agro-ecolog* OR Regenerat* OR Organic OR Resilien* OR Climate-smart OR “Climate smart” OR Diversified OR Ecological OR Agrobiodivers* OR Agro-biodivers* OR Inclusi*) AND (Agro-ecosystem* OR Agricultur* OR farm* OR “Food systems” OR “Food system”) AND (Framework* OR Tool* OR Indicator* OR Metric* OR Assessment* OR Evaluat* OR Measur* OR Monitor* OR Index* OR Indices).

To limit the number of articles retrieved and improve article relevance, this search string was applied to the article title only. Article language was set to English and document type was set to article, review article, book or chapter. No restriction of year of publication was set. All searches were conducted on the 28th of September 2022. The database searches yielded 3,525 articles: 1,601 articles from WoS and 1,924 articles from CAB Abstracts. These retrieved articles were supplemented with holistic assessment methods already known to the research team, resulting in an additional 13 assessments being added to the review. Retrieved article data were imported into Mendeley reference management software (Mendeley, 2022) and duplicates removed before being exported to Rayyan (Ouzzani et al., 2016) – an online platform for systematic literature reviews—for further screening. Figure 1 provides an overview of the selection process using a PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) flow diagram (Page et al., 2021). A total of 1,261 duplicate articles were removed and following the screening of abstracts and full texts, a further 2,071 articles were excluded at various stages, resulting in a final set of 206 assessments included in the review.

Given our focus on holistic assessment, we excluded assessments that did not assess all three dimensions – economic, environment and social – thus failing to meet our first characteristic of holistic assessment. In this review, we define ‘assessment’ or ‘assessment method’ to include “the diversity of approaches used in the literature (referred to as “approach,” “method,” “tool” or “framework”)” (Chopin et al., 2021:4) to assess the multifunctional performance of food and agricultural systems. This includes one-off and routine monitoring and performance data.

We discarded articles that were of poor quality or lacked sufficient information on their methodology, were reviews of assessment approaches, were irrelevant to agrifood systems, or were not deemed to be performance assessments (e.g., papers focused on the adoption rates of practices or suitability mapping). We also excluded ex ante evaluations using simulation models. Further, given that our unit of analysis was the assessment method, several articles referring to the same assessment approach (e.g., three articles referred to the use of the same approach) were treated as one entry in the final review. We included assessments irrespective of the geographies (e.g., low- to high-income countries, temperate, tropics), scale (plot, farm, landscape, national etc.) and farming systems (small-scale, commercial, pastoral etc.) for which they were developed.

The search and evaluation process used for the review was effective in that it generated a large number of diverse assessments. Importantly, the diversity of ways concepts such as sustainability and social components have been included in assessments are represented in our results and were not pre-determined by the search terms used. Such a review can never be complete and we acknowledge that the results are dependent on the decisions we made. For example, assessments for which there are no references in English will not have been captured. Likewise, assessments that aim to be holistic and systems oriented but that do not use any of our search terms to describe themselves will have been missed. As in most research projects, there is a lag between collecting data and publishing results. Our results should be complete up to time of data collection, but new assessments or new versions of existing assessments published since September 2022 have not been included.

Several frameworks have been developed for characterising and evaluating agrifood systems assessments, each with their own focus. While many focus on evaluating the characteristics of tools (e.g., Gasparatos and Scolobig, 2012; Schader et al., 2014), others have focused on the process of tool development (e.g., de Olde et al., 2016) or the implementation of assessment tools in terms of their ability to inform strategic decision-making (e.g., Coteur et al., 2020). In our review we extend and apply a framework developed by Binder et al. (2010) and adapted by Chopin et al. (2021) (see Supplementary Table S1 for details of the variables used in this review).

The framework originally outlined by Binder et al. (2010) has been widely adapted and used to review a range of characteristics and assessment tools (Bonisoli et al., 2018; e.g., de Olde et al., 2016; Linder et al., 2017). It was developed for evaluating sustainability assessments based on three interlinked dimensions: normative, systemic, and procedural (Binder et al., 2010). The normative dimension refers to “how to assess whether the studied system is sustainable” and covers aspects such as the goal of an assessment and underlying conceptual framing and theory (Binder et al., 2010:73). The systemic dimension refers to “whether a system is properly described by means of the set of indicators used” (Binder et al., 2010:73). This includes aspects such as the dimensions, themes and metrics used, which also fall under the normative dimension, but also whether an assessment captures relationships between metrics and potential trade-offs and synergies. Lastly, the procedural dimension covers “how the assessment was carried out” (Binder et al., 2010:73). This includes the methods used to collect data, the types of data collected, resource requirements and the user-friendliness of the tool. A more detailed description of these three dimensions and the rationale behind their associated variables can be found in Binder et al. (2010) and Chopin et al. (2021).

Given our focus on holistic systems assessment, we drew on the framework outlined by Binder et al. (2010) and its adaptation by Chopin et al. (2021) and extended it to include the four characteristics of holistic assessment proposed by Lamanna et al. (2024). In addition, we collected general information regarding each assessment, such as year of first publication, geographic focus, and whether they have been widely promoted.

Although there is nothing wrong with an assessment focusing on a particular issue of interest and selecting themes and metrics based on what matters to them, certain dimensions may receive more attention than others. This risks important aspects of agrifood systems performance receiving less attention and contributes to the ‘level playfield’ problem described earlier. Selective measurement of a system opens the possibility of the designer of the assessment influencing the results.

It is widely claimed that social dimensions of holistic assessments remain relatively underdeveloped compared to the economic and environmental dimensions (de Olde et al., 2016; Röös et al., 2019; Schader et al., 2014; Slätmo et al., 2017; Springer et al., 2015). Springer et al. (2015) and Nadaraja et al. (2021) found that assessments contain higher numbers of metrics related to the environmental dimension compared to the social and economic dimensions. Of the 47 indicators reviewed by Nadaraja et al. (2021), 29% pertained to social, 20% to economic, and 5% to governance aspects. In contrast, 47% of indicators were related to the environmental dimension.

We found no substantial differences in the number of metrics for each dimension but observed greater diversity in the themes associated with the social dimension. This finding aligns with other studies and suggests lack of consensus over how and what to measure within this dimension (Chopin et al., 2021; Janker and Mann, 2020; Springer et al., 2015; Wohlenberg et al., 2020). This reflects the diverse and subjective nature of many social dimensions of sustainability or a lack of consensus on how to define ‘social sustainability’ and difficulties in developing quantifiable metrics (Janker and Mann, 2020; Latruffe et al., 2016; Springer et al., 2015). What social sustainability means and entails differs around the world and across agricultural systems, geographies, and scales (Janker and Mann, 2020). An illustrative example is Röös et al. (2019) who reported that two popular farm-level assessment tools (SAFA and IDEA) struggled to capture the social conditions of Swedish farmers and had limited relevance to the social context of farming in Sweden.

Using social components of SDGs as a guide, our analysis uncovered several social themes that have received less attention by existing assessments. We found that far fewer assessments focused on the more social and institutional-related goals such as “Reduced Inequalities” (SDG 10), “Gender Equality” (SDG 5), “Peace, Justice, and Strong Institutions” (SDG 16), and “Partnerships for the Goals” (SDG 17), compared to those directly related to environmental and economic outcomes, such as “No Poverty” (SDG 1), “Life on Land” (SDG 15), and “Climate Action” (SDG 13). Additionally, a common reason for the exclusion of assessments from our review was their failure to incorporate aspects of the social dimension (see Figure 1).

It is possible that designers of holistic assessments of agrifood systems do not see such social issues as characteristics of the system they are able to influence and change. Aspects such as peace, justice and equity are determined by socio-political systems larger than those which agricultural researchers tend to focus on and are able to influence and thus such aspects may be deemed beyond the scope of their assessments.

These findings echo those of Zou et al. (2022), who identified ‘governance’ as the most neglected dimension at the food systems level. Similarly, at the farm level, Chopin et al. (2021) noted that few sustainability assessments used a ‘governance-oriented’ framing. Nevertheless, we speculate that the consideration of social and governance related aspects such as equity, peace, food sovereignty, and the intrinsic and relational values of nature will become more prevalent in the assessment of agrifood systems as more systemic framings which include a greater emphasis on social dimensions, such as agroecology, continue to gain traction.

It is widely claimed that the development of sustainability assessments should involve those who will use or be affected by the results and that they should be involved early and throughout, from the assessment design to the interpretation of results (Alrøe et al., 2016; Chopin et al., 2021; Schindler et al., 2015). Yet, our findings indicate that the stated intended users of assessments—farmers, researchers and policy makers—are seldom involved from the beginning. Instead, their input is often solicited after the goals and metrics of an assessment have been established, raising questions about the extent to which an assessment can inform and impact their decisions. In line with Arulnathan et al. (2020), we also found a severe lack of information on how users were engaged in the development of the reviewed assessments and whether their views were taken on-board. Most descriptions of how assessments were developed simply listed the types of actors consulted.

For holistic assessment to support transitions to sustainable agrifood systems, they need to be used and inform people’s decisions and actions. Yet, at least in the case of farm-level sustainability assessment, the uptake and use of tools by end users has been limited (Binder et al., 2010; de Olde et al., 2016; Triste et al., 2014). Triste et al. (2014) emphasise the importance of how a tool is developed rather than the content of the tool itself. They claim that the limited uptake of the MOTIFS tool by farmers is due to failings in the design process and insufficient engagement with end users (Triste et al., 2014). Farmers interviewed by de Olde et al. (2016), despite seeing relevance in the farm-level assessment tools, were sceptical as to whether the results were useful for informing their decisions. The farmers questioned whether the results provided new knowledge (i.e., the assessment did not tell them anything they did not already know) and reported that they “felt restricted in their opportunities to improve their sustainability due to the complexity of the system they are a part of” (de Olde et al., 2016:396).

Co-designing assessments with the intended users of the resulting data is likely to increase the relevance and utility of an assessment and its outcomes (Alrøe et al., 2016). Such assessments are more likely to meet the needs of their intended users, increasing the likelihood of wider uptake and use. Anyone organising and promoting an assessment needs to be realistic about whose needs are really being served by it, and the fact that farmers, while at the heart of an agricultural system, may not see the need for additional data. Participation can also lead to greater sense of ownership and allow for negotiating multiple user needs and interests (Namirembe et al., 2022; Schindler et al., 2015).

Care is needed when deciding who, how and when to engage different interest groups in the development of a holistic assessment. Such decisions determine the quality of participation and its outcomes (Reed, 2008). For example, evidence from co-production research indicates that stakeholders do not necessarily want to be included in every single step of the research process, rather they prefer to be consulted in strategic ways (Bieluch et al., 2017). Others warn of “consultation fatigue” among stakeholders who are “increasingly asked to take part in participatory processes that are not always well run, and as they perceive that their involvement gains them little reward or capacity to influence decisions that affect them” (Reed, 2008). Whether to engage those who are affected by the outcome of an assessment is another key decision (Reed, 2008). On the one hand, involving such actors is likely to increase their trust in assessment results and subsequent decisions. On the other, it may be impractical to involve stakeholders at this level. For example, involving everyone who may be impacted by the outcome of tracking progress on the UN SDGs in assessment design is neither feasible – since it includes everyone in the world – nor necessarily desirable as not everyone who is part of a system wants or needs data about its performance.

Assessing from multiple perspectives does not only imply involving stakeholders in assessment design. Data on many of the indicators used in holistic assessments are collected as subjective responses from individuals and depend on those individual experiences and values. The levels of these indicators will depend on who is answering. Taking an example from TAPE, rating of the amount of stress animals experience will be done differently by farmers and an animal welfare expert, and access to new knowledge rated differently by a farmer and extension officer. A holistic assessment that takes account of multiple perspectives would measure these different perspectives. Although many of the assessments collected self-reported data from farmers and used local indicators (indicators based on what local actors use to evaluate their system), we did not explicitly score for whether they captured multiple perspectives on the same aspect of system performance. Nevertheless, none of the reviewed assessments stood out as consistently generating data from multiple perspectives or stated this as an explicit aim.

A system is distinguished from a collection of parts by the interactions and interdependencies between those parts (Betley et al., 2021). If we are to understand and manage a system as a whole, it is inadequate to examine multiple dimensions in isolation. Instead, we must understand their interactions, trade-offs and synergies, as well as their associated themes and metrics (Binder et al., 2010; Capello and Nijkamp, 2002; Van Passel et al., 2007). Understanding synergies and tradeoffs is in part a function of how the data are analysed and presented. However, there are also implications for the design and implementation of data collection. For example, it will usually be necessary to ensure that all variables used in a tradeoff analysis have been collected on the same set of units (e.g., farms) at the same time. If understanding is to go beyond correlation, then evidence on the functional connection between system components will also need to be collected. Like Soulé et al. (2021), Binder et al. (2010), and Bonisoli et al. (2018), we found that, despite acknowledging the importance of interactions, few assessments analysed interactions. It is common practice to use numerical and visual integration in the form of composite indices and graphs such as radar charts. Both approaches allow for easy communication of assessment results. Yet, they can also hide the complexities and interactions between indicators. For example, the metrics displayed using a radar chart are not linked together and trade-offs and synergies are not shown despite being displayed next to each other.

Similarly, composite indices involve the integration of multiple sub-indicators into one single numerical indicator and are thought to be attractive to policy makers because they can be easily and quickly communicated and are interpretable by a wide audience (i.e., a higher score is ‘better’ than a lower score) (Siva Muthuprakash and Damani, 2017; Van Passel and Meul, 2012). Yet, composite indices are unable to provide an understanding of system dynamics as they do not reveal anything about the interactions between metrics (Roy et al., 2019). They can hide important details regarding the complexity of an issue and lack transparency on what sub-indicators may be driving an overall score (Latruffe et al., 2016; Van Passel et al., 2007). Further, they can obscure the differences in scores between systems. For example, two systems may have the same score, yet it is unclear if the systems are fundamentally similar or simply happen to have the same values of composite indicator (Peano et al., 2015).

The methods used to compute such composite indices are often subjective or arbitrary (Siva Muthuprakash and Damani, 2017). As highlighted by de Olde et al. (2016), even experts in each field can disagree markedly on the importance of different indicators. The use of composite indices thus requires transparency in how they are developed and if composite indices are used, they should not be presented in isolation from their more detailed sub-indicators (Magrini and Giambona, 2022; Van Passel and Meul, 2012).

Of the assessments we reviewed that did consider interactions, most used some form of correlation analysis between different indicators and dimensions (e.g., Rodrigues et al., 2010). To ensure such analysis is possible, data must be collected in ways that allow for interactions to be considered. In addition to the co-measurement requirement described above, there needs to be some basis for assuming that a correlation describes interaction and is not spurious. At the same time, given that assessment results need to be understandable and relatable to those making the decisions if they are to influence decisions, we need to find simple ways of communicating interactions and their complexity (Shields et al., 2002; Van Passel and Meul, 2012).

What happens after the assessment is perhaps more important than the assessment design itself in terms of changing practice and moving towards sustainability. The need for support in interpreting results to inform action has been clearly articulated in the case of farm-level tools and farmers decision making (Marchand et al., 2014). Coteur et al. (2020) reported that despite an awareness among tool developers of the importance of supporting farmers in interpreting assessment results and providing advice, only eight of the 18 tools that they reviewed provided support in interpreting results and recommendations based on results. This gap is also noted by de Olde et al. (2018), with many developers focusing on the methodology of a tool rather than how assessment results will be used by users and influence decision making. Ensuring that the presentation of results is interpretable by the assessment audience is key for action and influencing decision making. Given that different audiences will often have different preferences on how data is presented (Bourne et al., 2021) it is crucial for assessments to consider options in data representation (Lamanna et al., 2024). The use of online platforms for automating the analysis and visualisation of data collected using promoted tools could help increase the usability and interpretation of assessment results by those who lack data analysis skills. Yet we found that online platforms are uncommon. Most promoted tools we reviewed were open access and provide guidance on their use for generating data, but less so on interpretation and how to use the results of the assessment.

The design of a holistic assessment is challenged by the number of system components and processes that could be included in the measurement scheme. However, a holistic view includes understanding the system as a whole—its emergent properties—not only each of its components. Emergent properties of agrifood systems, such as resilience, equity, and empowerment, result from complex interactions among system components across scales. Prosperi et al. (2016) argue that concepts like resilience and vulnerability offer a more effective framework for assessing food systems, and hence determining what should be measured and monitored, than system components alone. Only 26% of the assessments that we reviewed addressed themes related to such emergent system properties, perhaps because it is difficult to do so. For instance, despite increasing interest in resilience and development of several indicators for its measurement (Douxchamps et al., 2017), measuring resilience remains challenging given its varied definitions, multidimensional nature and need for the inclusion of multi-spatial and temporal scales (Prosperi et al., 2016). Resilience first requires clear definition in the context of the system being assessed, the measurement at multiple levels and scales, yet most assessments of resilience focus on one scale (Douxchamps et al., 2017).

Evaluating changes in system performance over time, rather than through a single snapshot, can provide important insight into a system’s trajectory and response to shocks, stresses, and management changes (Chopin et al., 2021). Measuring resilience necessitates the measurement of a system over time (Douxchamps et al., 2017). Nevertheless, we found that most assessments are used to provide one-off snapshots of performance or to compare the performance of two or more cases at a single point in time, even if the authors hope their measurement scheme will be taken up and used repeatedly. In their review of farm-level assessment tools, Marchand et al. (2014) present a potential trade-off between how comprehensive or complex an assessment is and how easy or quick it is to use. On one end, there are full assessments aimed at providing high accuracy and often complex evaluations, and on the other end, there are rapid assessments aimed at providing a quick picture and overview with lower complexity and accuracy. Neither end point is strictly defined since it is never possible to assess everything and unclear when a simplified assessment stops being holistic. However, there are real choices to make. This trade-off closely relates to other trade-offs identified in the literature, including scope versus precision (Schader et al., 2014) and breadth versus depth (Namirembe et al., 2022). Complexity of an assessment tool has implications for the cost, time, and skill requirements for carrying out an assessment. This, in turn, has implications for using a method repeatedly or routinely to monitor change and data collection by non-specialists such as activists or communities themselves. For instance, the more complex a tool is the less likely it is to be widely taken up (Marchand et al., 2014).

In our review, we classed most assessments as having ‘medium complexity’ and having a ‘medium’ time requirement, in that they are “generally applicable for professionals working in the environmental or agricultural sector with advanced experience” and where “some additional training is necessary to become familiar with these methods” (Kaufmann et al., 2023:22) and take 2 to 7 h to complete. This prevalence of medium complexity and time requirement could reflect a recognition of this trade-off between complexity and useability among tool developers and efforts to find a middle ground.

Nevertheless, this compromise can come at the expense of assessment reliability and validity. Of the assessments we reviewed, 34% were classed as being non-reproducible, in that you might not get the same result if they were to be repeated by more than one person, and over half of the assessments we reviewed relied on questionnaires and recall data rather than field measurements. Subjective data is liable to biases and limits the utility of a metric for evaluating agrifood systems approaches on a level playing field. There are potential sources of bias in objective measurements too, but they are usually easier to manage by design.

Assessments that were classified as non-reproducible were often those where metrics used were dependent on value judgements and likely to vary depending on the perceptions of those carrying out the assessment. For example, TAPE relies on several subjective metrics such as whether a farm has many trees or not. As noted by Marchand et al. (2014), more objective measures also allow for greater comparability across cases and the potential development of benchmarks. Yet tools that are quick and rapid to apply often rely on respondent recall and perception which may reduce the accuracy and correctness of an assessment (Coteur et al., 2020; Marchand et al., 2014).

Coteur et al. (2020) and Payraudeau and van der Werf (2005) note that there is often a trade-off between the use of performance-based indicators and feasibility. System indicators can often be described as either (1) practice-based indicators that describe what is done in the system, such as practices used by a farmer, and (2) performance-based indicators that measure what the system is doing in social, economic or environmental dimensions. The latter provides direct evidence of the system function. The former are often interpreted as indicators of performance based on an assumption that use of a practice will lead to specific performance (Coteur et al., 2020; FAO, 2014). FAO (2014) proposes that performance-based indicators, given their closeness to reality and the impact of interest, should be prioritised over practice-based indicators.

We note that one person’s practice-based indicator may be another’s performance-based indicator. For example, consider the level and diversity of tree cover. It is a description of a system and not its performance. However, for someone interested in the biodiversity value of an agricultural landscape and aiming to increase tree cover, the level of tree cover could well be a performance indicator. The tensions between assessment scope and accuracy, breadth and depth, and complexity and use ability, mean trade-offs will need to be made. These should be done with the assessment objectives and intended users clearly in mind and involved in the development of the assessment.

The 206 holistic agrifood system assessment frameworks, methods, and examples that we identified and reviewed display a wide diversity of approaches and practices. The reason is simple: there is no one assessment tool that will work for everyone and new concepts and purposes of assessment are continually emerging. Previous, less extensive reviews have reached similar conclusions (Alaoui et al., 2022; Arulnathan et al., 2020; Bonisoli et al., 2018; Marchand et al., 2014; Nadaraja et al., 2021; Schader et al., 2014; Van Passel and Meul, 2012). The diversity of themes and metrics employed by assessments is to be expected since agrifood systems encompass so much. For instance, while one assessment may focus on measuring soil health as the key environmental dimension, another may focus on measuring biodiversity. Neither of these two assessments is right or wrong in their selection of themes and metrics. They are different because they were developed for different users, each with different objectives and perspectives on what is important to measure and how to measure it (Alaoui et al., 2022; Chopin et al., 2021; Coteur et al., 2020; Gasparatos and Scolobig, 2012; Janker and Mann, 2020; Marchand et al., 2014). As Alaoui et al. (2022) note, different groups have different reasons for using farm-level assessment tools. Farmers may use them for evaluating the performance of their own farm; agricultural advisors for advising farmers on how to improve their sustainability; and researchers for comparing farm performance across different cases, regions, or countries (Alaoui et al., 2022). The same will be true for any other scale of assessment.

The development of conceptual bases for assessments is one driver of development of new methods. Among the assessments we reviewed, holism first appeared as a frame in 1997, agroecology in 2004, resilience in 2008, ecosystem services in 2011, climate adaptation and sustainable intensification in 2015, and vulnerability in 2019 (see Figure 4). This trend of conceptual development will inevitably continue, resulting in further innovation in assessment methods.

The objectives of an assessment also influence the level at which it is conducted. For instance, farmers and agricultural advisors may have a greater interest in assessments conducted at the field or farm level, while national governments and international bodies may have greater interest in assessments conducted at the regional, national, or sectoral level. We found that most assessments focused on the farm scale and production stage of agrifood systems. This focus on farm-level assessment could reflect a potential bias in our search terms. It could also reflect the fact that many decisions over agricultural production are made by farmers at the farm level and thus the farm level is often identified as an entry point for influencing system change (Marchand et al., 2014). Further, measuring performance at the field and farm level is often easier than measurement at higher levels (e.g., landscape, regional, national) making it a likely focus for research projects trying new ideas with limited resources. In addition, many metrics collected at farm level can be aggregated to provide a view at higher levels but the converse is not true. Nevertheless, as Gharsallah et al. (2021) note, there are important aspects of agrifood systems performance that cannot be scaled in this way such as ecosystem services that emerge at landscape scale, water regulation, or biodiversity protection. IDEA4 (Zahm et al., 2024) is an example of an assessment that is based explicitly on a multiple scales, with a framework that merges farmers’ self-centred goals with wider interests of the community, country and rest of the world.

The appeal of universal system indicators is obvious. We have extensive databases of conventional production and economic data about food and agriculture systems that are global in extent and aim to be globally comparable (e.g., FAOStat). Recognising that these are inadequate for giving a holistic picture of systems leads to demands for assessments and indicators that will fill the gap (Béné et al., 2019; Schneider et al., 2023). However, the utility of a globally relevant assessment tool is limited given the diversity of assessment objectives and varied nature of agrifood systems (Binder et al., 2010; Dong et al., 2016; Hayati et al., 2010). We identified numerous promoted or ‘readymade’ tools developed with the intention of being globally applicable and meeting the needs of multiple audiences (e.g., farmers, researchers, and policy makers). Despite many of these tools being co-designed, they are unlikely to meet all users’ aims and need context specificity. For instance, TAPE (FAO, 2019) was developed with the intention of being a global tool, yet Namirembe et al. (2022) in their evaluation of using TAPE across multiple projects and contexts conclude, that “it is not a readymade approach or set of tools to use in every situation” and that “the same will be true for any other ‘readymade’ assessment tool that involves processes, indicators, and tools that have been defined without reference to the specific context and objectives.” In recognition that no one tool fits all, several authors have argued for combining tools or using them sequentially to address different scales and user needs, thus catering to the various “layers and players” involved in an assessment (Alrøe et al., 2016; Van Passel and Meul, 2012).

Wide scale adoption of new standards for system measurement depends on the power of people demanding it. UN agencies, for example, have power and can influence what data is collected—for example, most countries report national progress on sustainable development goals and international assistance is available to help in that effort. In our review we noted most of the ‘promoted’ assessment approaches and tools (those behind which there is a concerted effort to persuade others to use them) are developed by international bodies and NGOs. The most common tool developer was the United Nations Food and Agricultural Organisation (FAO) with eight named tools having been developed by the organisation, followed by Biovision Foundation with three tools. Whether a framework is taken up depends not only on its inherent strengths but also on who is promoting and supporting it and the influence of those groups. This applies at national as well as international scales, with states having the authority to impose adoption of an approach and provide resources to implement it.

The ideas of technology lock-in (Foxon, 2013) also play a role in whether a new assessment tool will be used. There are always reasons to keep doing what we are doing rather than something new, even when the new approach is demonstrably superior. Foxon (2013) identifies four sources of technology lock-in: economies of scale, learning and skills, perceptions of success and network effects. All these will apply to assessment approaches and hence we see a method such as TAPE being widely used even if it is not necessarily the most appropriate in each case.

Many past reviews aimed at identifying deficiencies and gaps to be filled and call for the development of new assessment approaches and metrics. This thinking—that we or anyone else will be able to fill such gaps—is flawed. There are and always will be gaps to holistic agrifood systems assessment. While the appeal of universal methods and indicators is obvious, creating a new standard that everyone will use is unlikely given the diversity of assessment objectives, and varied and unlimited nature of agrifood systems. There will never be one tool or approach that will work for everyone, can measure everything and be used everywhere. There is large diversity and continuous innovation and adaptation in the assessment of agrifood systems. There will always be new frameworks, concepts and perspectives on agrifood systems assessment emerging. Improving the holistic assessment of agrifood systems is therefore not a question of improving or combining existing tools to meet the vast needs and interests of different users. Instead, the gap to be addressed is the lack of guidance for setting up and designing effective holistic systems assessments.

There are two broad options for someone wanting to conduct a holistic systems assessment: (1) to select or adapt an existing tool or combination of tools to best fit their needs; or (2) to develop their own tool. Several past reviews have aimed to provide guidance on how to select the right assessment tool based on needs (e.g., Bonisoli et al., 2018). Yet, there is no guarantee that such a tool already exists given the diversity of global agrifood systems and user objectives.

Lamanna et al. (2024) describe the second approach. While they include the design principle of “do not reinvent the wheel” in their guidance (i.e., if there is already a tool that meets the assessment objectives, then use it), they also outline a flexible stepwise approach to developing holistic systems assessments that meet specific user needs. This stepwise approach has several advantages, one of which is the emphasis on clearly articulating the purpose of the assessment and identifying an appropriate conceptual framing. Objectives and conceptual framing drive the effective design of assessments (Namirembe et al., 2022). Yet, like Siva Muthuprakash and Damani, 2017 and Janker and Mann (2020), we found that many assessments lack a clearly articulated conceptual framework on which their selection of themes and metrics are based, with the risk that indicators are selected because they ‘might be interesting’ rather than because they contribute to a consistent view. Following a stepwise approach to developing a holistic systems assessment, with a strong emphasis on theoretical framing and clearly articulated goals, can help ensure that what matters is measured (i.e., the choice of what to measure is determined by what users care about) and that context-specific aspects are not overlooked.

An argument against designing individual context-specific assessment frameworks is that resulting datasets will not be comparable across time and contexts. However, if that is a requirement or aim, then it becomes part of the design criteria (e.g., monitoring progress towards SDGs where coherence is needed) (Rosenstock et al., 2017). There may also be a tension and balance to be made between measuring what a user cares about (i.e., “what matters”), which is often value-based, and selecting a holistic conceptual frame that is logical, coherent and complete. Nevertheless, those interested in holistic assessment need clear guidance to navigate the abundance of existing frameworks and develop assessments that meet their needs while avoiding the extremes of narrowly focused metrics or attempting to measure everything simultaneously. Frameworks for selecting and developing assessments, such as that outlined by Lamanna et al. (2024), could offer such guidance.