It is a common goal of society to keep antimicrobials effective for the coming generations. One way of supporting this goal is to have surveillance in place for antimicrobial use (AMU) and resistance in different domains and sectors. This should preferably be done in an integrated manner, because genes coding for antimicrobial resistance (AMR) are spread within and among different human, animal and environmental domains. To ensure surveillance effectiveness and efficiency, there is a need to evaluate existing surveillance systems or components at regular intervals (1). This will help to reach the objective of surveillance which, among others, is to determine why and where action is needed to modify AMU and hereby reduce AMR.

Several tools have been developed to assist in such evaluations. Evaluation may be done by different types of professionals, who may be acting internally or externally to the surveillance system under evaluation. The users will have varying levels of experience in surveillance evaluation, access to detailed data and time to dedicate to the evaluation. Moreover, evaluation may be pursued for different purposes. This makes it necessary to choose the right tool for a given evaluation context, team and question.

During 2019–2020, an international network of scientists called “Convergence in evaluation frameworks for integrated surveillance of AMU and AMR” (CoEvalAMR) developed guidance for the evaluation of integrated surveillance for AMU and AMR (2). In this network, we defined integrated surveillance of AMU and AMR in the context of One Health as surveillance that is based on a systemic, cross-sectoral, multi-stakeholder perspective to inform mitigation decisions with the aim to keep antimicrobials effective for future generations (https://guidance.fp7-risksur.eu/welcome/evaluation-of-surveillance/). In Phase 1 of the CoEvalAMR project, a methodology was developed to gather user feedback on evaluation tools for integrated surveillance for AMU and AMR in an easy and standardized way. The focus was on compiling user subjective experience on the application of the tools; the approach chosen was partly inspired by websites using user feedback and scoring to inform decision-making of other users. The methodology consisted of four different approaches that complemented each other. The first consisted of a brief description of the case study, whereas the second covered the assessment of 11 pre-defined functional aspects of the tool including workability regarding the need for data, time and people (Table 1). The third approach covered an assessment of seven predefined content themes related to the tools' scope (Table 2). The functional aspects and content themes were scored semi-quantitatively using a scale from 1 to 4, and a comment was requested explaining the score. The fourth approach consisted of the subjective perception of the tool assessors based on an assessment of the strengths, weaknesses, opportunities, and threats (SWOT) (Table 3).

During Phase 1, six tools were assessed using the described methodology, by applying them to eight national surveillance systems as country cases. The tools were: ATLASS (The Assessment Tool for Laboratories and AMR Surveillance Systems developed by the Food and Agricultural Organization (FAO) of the United Nations), ECoSur (Evaluation of Collaboration for Surveillance tool), ISSEP (Integrated Surveillance System Evaluation Project—now called ISSE), NEOH (Developed by the EU COST Action “Network for Evaluation of One Health”), PMP-AMR (The Progressive Management Pathway tool on AMR developed by the FAO), and SURVTOOLS (Developed in the EU FP7 project RISKSUR). An overall description of this work can be found in (2) whereas (3), described the Danish case study in detail. Moreover, a description of users' experience for each country case study can be found on the website of CoEvalAMR (https://guidance.fp7-risksur.eu/welcome/case-studies/). Some of these case studies consisted of full evaluations based on the tools used. In others, the focus was mostly on the tool, and therefore, the case study only included a superficial evaluation of the surveillance system.

We learned that some tools can be directly used to evaluate a given question, a surveillance component or a system. Such tools have a pre-defined set of steps that need to be conducted. Other tools are better described as frameworks, which provide a theoretical background and explanation as to how the evaluation should be designed. These frameworks guide users toward the most appropriate evaluation method based on the evaluation question and context. According to Calba et al. (4), a framework acts as a skeletal support for something being constructed. Hence, it is an organization of concepts that provides a focus for inquiry. In contrast, Calba et al. (4) define a tool as a process with a specific purpose. Therefore, a tool is used as a means of performing an operation or achieving an end. The ISSE is an example of a framework (5), whereas PMP-AMR is an example of a tool (6).

Among the tools and frameworks investigated, only the ISSE framework is dedicated specifically to the evaluation of integrated surveillance of AMU and AMR, outlining a logic model that can be used to conceptualize surveillance evaluations. Other tools, such as ATLASS and PMP-AMR, are designed specifically for AMU and AMR surveillance and management, NEOH for One Health initiatives in general, SURVTOOLS for surveillance in general, and ECoSur for integrated collaboration (2).

It was concluded that all tools investigated were suitable to evaluate relevant—but not necessarily all—aspects of integrated surveillance for AMU and AMR. Moreover, each tool has its specific purpose and consequently distinct advantages and drawbacks. This makes it important to define a clear evaluation question and objective to choose the right tool. We also learned that the complexity of the tool application appeared to be proportional to the comprehensiveness of the evaluation results. Moreover, governance and impacts of integrated surveillance for AMU and AMR were not fully covered by the assessment of the tools in Phase 1.

Hence, ample experience was collected regarding assessment of the tools and the developed methodology. It was concluded that the methodology worked, but the wording and definitions could be clearer, the evaluation coverage could be broadened, and the scoring system could be more standardized. It was also of interest to understand better the expectations of tool users. Moreover, we wanted to compare the CoEvalAMR methodology with the assessment process used in the newly published Surveillance and Information Sharing Operational Tool (SISOT) (7), developed by the Tripartite (FAO/WHO/WOAH) of the United Nations (UN). These aspects have been dealt with in Phase 2 of the CoEvalAMR project, which runs from 2021 to 2023. The objective of this paper is to present the updated methodology, including an example showing the changes, as well as the considerations behind the update.

In spring 2021, monthly virtual meetings began in the network, allowing members to convene and discuss how to update the methodology. A common document was set up enabling all members to provide comments and suggestions, which were subsequently discussed with the aim of obtaining consensus. This process continued until autumn 2022. Three elements were discussed: (1) lessons learned from using the initially developed methodology, (2) an analysis of expectations of tool users, and (3) the assessment process used in SISOT. Regarding lessons learned, the approach was a brainstorm in the groups' monthly meetings.

Regarding expectations of the tool users, we considered the results of a survey by Rüegg et al. (8). The survey was conducted in Phase 1 of CoEvalAMR to gather information on evaluation of existing or planned AMU and AMR surveillance systems and people's use of available evaluation tools, as well as their expectations on tools. An analysis of the 23 answers received was undertaken. We studied and discussed how we could best make use of these results to update the CoEvalAMR methodology.

Further, we looked at the list of functional aspects and content themes in SISOT and assessed if any of these would be of value for the update of the methodology. We also studied the definitions, use of scales, and visual appearance. Based on discussions in the CoEvalAMR network group, we aimed at identifying additional functional aspects, which would make the description of the individual tools more complete.

Finally, the updated methodology was tested using a case study undertaken as part of our network. Here, ECoSur was applied to the French surveillance system for AMU, AMR and antimicrobial residues in humans, animals and the environment (9). The overall objective was to evaluate the degree and quality of multisectoral collaboration within the surveillance system. In accordance with the aim of ECoSur, the focus was on evaluating the organization, functioning and functionalities of collaboration taking place in the French multi-sectoral surveillance system. The tool is available online (https://survtools.org/wiki/surveillance-evaluation/doku.php?id=quality_of_the_collaboration), for more information about ECoSur, please see Bordier et al. (10).

In Phase 1 of the CoEvalAMR network project, it was found that the users scored the individual functional aspects and content themes in a slightly subjective way. As the project progressed, a higher degree of consensus arose regarding interpretation of the methodology, including the way of scoring (2). Moreover, we discovered that the third question in the SWOT analysis was misunderstood by some of the users. We expect that with the update of the methodology, subjectivity will be reduced. Similarly, the likelihood of misunderstanding the questions will be lower.

The importance of considering gender and equity to tackle AMR has been underlined by the WHO (12, 13) but is currently rarely integrated into surveillance system evaluation. As explained by WHO, unless we think about how AMU and AMR affect men and women and different groups in society in their day-to-day lives at home, work and in their communities, we may inadvertently design programs that fail to address what matters. Hereby, effectiveness may be reduced, and impacts lost, and we may even contribute to gaps and inequities (12). As a first step toward enhancing the inclusion of this aspect, we have added consideration of gender to the list providing a general description of the tools (Supplementary Table S2). Still, we foresee a discussion on how to assess and evaluate gender aspects and other equity issues of importance for AMU and AMR. These issues may become part of a future Phase 3 of our network. Here, chapter 4 in the Handbook for Evaluation of One Health may provide inspiration for the next steps to take (14).

The respondents of the questionnaire survey undertaken as part of Phase 1 of CoEvalAMR pointed to the need for standardization of tools (8). In response to that, we have focused on standardizing our methodology by introducing clearer definitions and scales. The question arises as to which extent further standardization of our methodology is needed. It may be argued that standardization is an essential requirement in academia, but a less important issue for persons involved with the human health and veterinary authorities, where the process initiated by the tool would be more important than the tool itself. Moreover, the intention is not to compare tools, but to describe the tools to such an extent that the future users will be guided in choosing the right tool for their purpose.

According to the survey, the users prefer tools that are easy to use, without much need for preparation or training (8). The question is how this can be operationalized. Grants are usually targeting the development of tools, whereas limited resources are available for supporting their uptake and long-term maintenance. Moreover, the results of simple evaluations may not be sufficiently valuable. Still, it is relevant to discuss the balance between required training, allocated resources, details and overview. To address this, the intended outcome of the evaluation becomes crucial. This reiterates the need for careful description of the evaluation purpose before choosing the evaluation tool.

In our updating of the methodology, we have been inspired by the SISOT matrix developed by the Tripartite. The SISOT matrix is very detailed and can be used for evaluating different kinds of tools and resources for any zoonotic risk-reducing activities. The questions and possible ways of answering show how well-developed the SISOT matrix is. Our revised CoEvalAMR tool is targeting integrated surveillance for AMU and AMR. Based upon our own experience as well as the French case study (11), the CoEvalAMR methodology appears simpler and quicker to use than the SISOT matrix, while it still contains most of the elements that form part of the SISOT matrix. In conclusion, each approach was developed for its own objectives and has its value.

The case studies reported by Sandberg et al. (2) and Nielsen et al. (3) and the French case study (9, 11) covered both multi-component and single component surveillance systems. Multisectoral means that more than one sector is working together in a joint program or response to an event. Similarly, multidisciplinary means collaboration across several disciplines. Taking a One Health approach means that all relevant sectors and disciplines are involved (15). However, it does not imply that all sectors must work together and at all stages of surveillance. The key regarding the degree of integration is relevance. For example, the Competent Authority may need AMU and AMR data in animals and humans to evaluate the effect of a ban on use of a specific kind of antimicrobial in agriculture. However, data on AMR from the environment may not be needed. In contrast, if we are trying to understand the spread of AMR in the environment, data about AMU and AMR are needed from all three sectors. The methodology we have developed is useful to provide an overview of the advantages and disadvantages of the tool investigated, irrespective of whether the tool was used for evaluation of an integrated or non-integrated surveillance system.

Evaluation of One Health surveillance is an active field, and there is a growing number of evaluation tools becoming available. The Canadian One Health Evaluation of Antimicrobial Use and Resistance Surveillance (OHE-AMURS) tool is an example of such a new tool. It has been created to evaluate progress toward integrated, One Health surveillance of AMU and AMR while focusing among others on policy and programme sustainment (16). In Sandberg et al. (2), six tools were retained for evaluation. The ambition in Phase 2 of CoEvalAMR is to apply the updated evaluation methodology to other tools, in accordance with the needs or interests of the network members. The French case study is an example of this. It showed that there is a diversity of individual surveillance programs in France (9). This makes it difficult to get an overview of the surveillance system and its level of integration (11). The ECoSur evaluation provided this overview and helped to identify recommendations, which were shared with policy makers to improve One Health collaborations within the French system for surveillance of AMR, AMU, and AM residues (11).

An ongoing common activity in WG4 of CoEvalAMR is an evaluation of the OH-EpiCap tool, which is under development by the MATRIX consortium, funded by the One Health European Joint Program (17). In a common paper about OH-EpiCap, it will be investigated how we can combine the scores of the different assessors and case studies in a way which ensures that the variation is reported.

Other persons involved in surveillance evaluation are welcome to make use of our methodology. Moreover, the tool developers can get valuable information from our case studies for further improvements of their tools.

The CoEvalAMR evaluation methodology is developed with a focus on the users’ experience. It is free to use, simple and easy to work with. It has been updated to improve clarity, broaden the evaluation coverage, increase the standardization, and improve the visual appearance. The update was based upon experience from the CoEvalAMR network group from applying the methodology using country case studies, a questionnaire focused on the users' needs as well as a comparison with SISOT Evaluation Matrix developed by the Tripartite.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

LA took the initiative to update the methodology and headed the revision process through a series of meetings in WG4 of the CoEvalAMR network. Based upon the inputs from the group, LA drafted the first version of the paper, which was commented by all authors. The excel version of the CoEvalAMR tool was developed by PM. LC is responsible for the French case study. All authors read and approved the final version of the manuscript.