We focus on the ‘Market-based approaches to improving the safety of pork in Vietnam’ (SafePORK) program to explore the use of a combined ToC and OM. SafePORK is a 5-year program funded by the Australian Center for International Agricultural Research and implemented by the International Livestock Research Institute, Vietnam National University of Agriculture, Hanoi University of Public Health, and national (National Institute of Animal Sciences) and international partners (University of Sydney). The development of SafePORK was motivated by a growing concern for food safety, one of the most pressing issues among people in Vietnam (22). In particular, the safety of pork is a major concern as pork is the most widely consumed animal source food in Vietnam (23, 24). Pork safety is a shared responsibility among many actors along the pork value chain, making risk management for pork safety a complex challenge. SafePORK operates in several areas of Vietnam (Hanoi, Hoa Binh province, Hung Yen province, and Nghe An province). Applying a One Health approach, SafePORK aims to reduce the burden of foodborne disease in the informal, emerging, and niche markets of Vietnam.

SafePORK can be considered a complex program, characterized by a plurality of stakeholder perspectives and multiple interacting components (25). The research team is comprised of veterinarians, medical doctors, public health experts, farming systems experts, and agricultural economists. They work closely with actors along the pork value chain (e.g., farmers, slaughterhouse workers, wet market retailers, and consumers) and other decision-making partners (e.g., local authorities). Research and development activities of SafePORK often overlap and include generating evidence on feasible approaches; identifying, developing, and piloting light-touch interventions; and, building capacity to manage food safety risks among government partners, private sector actors, journalists, and pork value chain actors. One of the core objectives (number three) of SafePORK is to develop a roadmap showing how, why, and in what context SafePORK leads to safer food (Box 1).

Given the complexity of the food safety challenge in Vietnam, the engagement of multiple perspectives characteristic of One Health approaches, and the need for learning support throughout the SafePORK program, we were interested in a framework that was responsive to dynamic, real-world environments. We wanted to systematically capture and learn from our outcomes to inform adaptations to the program. As everyone has different ideas, hypotheses, and assumptions (“theories”) about how change happens, going through a ToC process can help make these theories explicit. We used ToC to establish a shared roadmap toward change and identify potential areas for monitoring and evaluation. A ToC, however, does not tell us how to assess change; we combined ToC with tools offered by OM to support SafePORK in not only learning about its change process but also in measuring it. OM is often considered well-suited to assess programs implemented under complexity in which multiple influences make it difficult to predict what will happen as a program proceeds (26). We used OM to help the team be specific about the actors SafePORK intends to work with, the behavioral changes it hopes to see, and the strategies needed to achieve such changes. Furthermore, we used OM as a framework to monitor outcomes.

We developed a ToC following advice from several guidance documents (27–29) along with consultations with the team. Often absent from ToC guidance documents is the focus on systems change, yet, capturing systems change is particularly important for food safety programs that influence (and are influenced by) food systems (30). Our adapted 5-step ToC process is iterative, cyclical, and reflective, involving: (1) contextual analysis; (2) identifying the goal; (3) working backward to identify what changes must occur to reach the goal; (4) working forward to identify how the program will contribute to changes; and, (5) stating assumptions underlying change processes (Figure 1). By starting with an overview of potential long-term outcomes at the end of the program (2022), a focus is placed on the bigger picture of systems change. Proposed SafePORK contributions (from 2017 onwards) are added to the change pathway only after systems change is envisioned.

To operationalize the methodological framework for developing a ToC, a full-day workshop was conducted with 12 SafePORK researchers (seven women and five men). The facilitator (SL) described the ToC approach using examples and then asked participants to undertake an exercise following the 5-step process. As SafePORK works extensively with several actors along the pork value chain, participants agreed to create separate actor-based ToCs while acknowledging that ToCs might be combined into one comprehensive ToC later. Participants were split randomly into two teams; one worked on slaughterhouse workers and retailers while another worked on consumers and policymakers. Toward the end of the workshop, participants were asked to reflect on the challenges and opportunities of developing a ToC.

OM is a three-stage process of intentional design, outcome and performance monitoring, and evaluation planning (14). In the first stage, stakeholders create a vision of desired behavioral outcomes and outline strategies to be used in achieving such outcomes. The second stage provides a framework for monitoring progress toward changes identified in stage one. The third stage provides a framework for identifying evaluation priorities and conducting an evaluation. To design SafePORK's monitoring and evaluation, we adapted OM; we focused on intentional design to build on the ToC (Figure 2).

To operationalize OM, we convened a half-day workshop with SafePORK researchers (5 women and 4 men). The facilitator (SL) explained the theory of OM and provided examples of OM in practice. As most team members were already familiar with OM through the previous phase of SafePORK (PigRisk program; ACIAR LPS/2010/047; 2012-2017), we worked together quickly through the initial OM steps (i.e., drafting the vision and mission statement). More time was spent focusing on boundary partners, outcome challenges, and progress markers often considered the “essence of OM” (31). Specifically, participants were split randomly into two teams to explore boundary partners, outcome challenges, progress markers, and strategies, and how these relate to SafePORK's vision and mission. Toward the end of the workshop, teams planned for the outcome monitoring.

The resulting ToC in Figure 3 visually describes the presumed mechanisms of change occurring within the food system in Vietnam. Here, we expand on the ToC by narratively describing the pathway as well as assumptions and context underlying change. The overall goal of SafePORK is to reduce the burden of food-borne disease in traditional, emerging, and niche markets of Vietnam. To achieve this goal, SafePORK proposes that wide-scale adoption of safe food practices among all pork value chain actors is needed. Two pre-conditions are required to achieve this wide-scale adoption: (1) small-scale adoption of safe food practice and (2) updated policies.

According to SafePORK researchers, the identification of cost-effective practices is important for supporting small-scale adoption of safe food practices among women and men. SafePORK intends to contribute to this outcome by piloting light-touch incentive-based interventions, training, and communication along the pork value chain. Several assumptions underlie the causal link between SafePORK activities and improved safe food practices, such as retailers', slaughterhouses', and consumers' demand safer pork. Furthermore, safe food practices need to be supported by broader environmental factors including slaughterhouse and market infrastructure, food safety procedures and enforcement, and certification and branding, which may be indirectly influenced by SafePORK.

Secondly, researchers emphasized that policy-makers should strengthen policies, support the scaling-up of SafePORK pilots, develop a model for small-scale slaughterhouses, improve surveillance and inspection, and increase the budget for food safety interventions. SafePORK intends to influence these actions by presenting evidence from pilot interventions to policymakers through policy brief workshops and study tours. At the provincial level, for example, SafePORK is engaging the sub-Department of Animal Health in Hung Yen in dialogue surrounding the slaughterhouse intervention model. There are several assumptions behind this causal link, such as policy-makers must be interested in improving food safety. While this assumption may seem obvious, the experiences of SafePORK (and PigRISK) demonstrate that buy-in from policy-makers is essential and must be fostered for interventions to succeed.

Researchers also considered the social and physical environments to be important factors underlying the success of SafePORK. For example, Hung Yen is an appropriate province to implement interventions given its high pig production, proximity to the capital city (Hanoi), and room for improvement of hygienic practices. In Hanoi, greater awareness of food-borne diseases, higher income, and generally stronger infrastructure make Hanoi a conducive environment to conduct food safety interventions. Interventions also need to consider who participates in and benefits from efforts aiming to improve food safety and the different roles and responsibilities of women and men. For example, slaughtering is mostly done by men while retailing and purchasing are mostly done by women, providing opportunities for targeted risk management.

The articulated program theory provided us with a starting point for planning evaluation activities through outcome mapping's intentional design stage. According to SafePORK researchers, the vision of SafePORK is to improve public health by reducing the burden of food-borne disease in traditional, emerging, and niche markets of Vietnam. Its mission is to develop and test market-based, light-touch, and incentive-based interventions. While SafePORK interacts with many boundary partners with a critical role in ensuring food safety, program monitoring and evaluation will focus primarily on slaughterhouse workers and retailers. We consider slaughterhouse workers and retailers to be within SafePORK's direct ‘sphere of influence’ whereas other value chain actors are within SafePORK's indirect ‘sphere of interest’ (14). The main outcome challenge for direct partners is to maintain more hygienic pork handling practices taught in SafePORK training. Progression toward this outcome will be measured by indicators ranging from agreeing to take part in identifying promising interventions to maintaining practice change (Figure 4).

Researchers agreed that the progress of boundary partners toward the achievement of the outcome challenge will be measured on an ongoing, real-time basis. Monitoring journals provided by OM will be used to guide this process. The outcome journal will track the behavioral changes of partners using progress markers whereas the strategy journal will document the activities conducted to achieve outcomes. Several focal points from the SafePORK team will contribute to one shared journal integrating outcomes and strategies. Specifically, the focal points will document (1) activities/strategies implemented, including with whom, where, and when; (2) reflections on what changes occurred, what worked well, and what could be done better; and, (3) and share pictures of before and after. The collected information will be used to inform adaptations to interventions and provide evidence for the final evaluation.

From our ongoing monitoring efforts, we are starting to see behavioral changes during implementation (in some areas and not others), informing adaptations to the intervention. For example, at a slaughterhouse in Hung Yen, we are seeing the provided grid and tables being used during carcass handling. Importantly, some tables were co-invested by the slaughterhouse owner, highlighting the slaughterhouse owner's interest in the program. We are also seeing better separations between clean and dirty areas. However, sometimes knives are not properly cleaned after use and in some cases are put on the floor. The team makes regular visits to the slaughterhouse to encourage hygienic practices. At the traditional wet markets, we are seeing retailers now using separate cutting boards for raw meat and cooked meat. However, many retailers prefer wooden boards because they are better for chopping bones. To address this challenge, the program co-invested in wooden cutting boards with retailers.

Combining ToC and OM can address some criticisms associated with each approach. For example, ToCs can sometimes be seen as vague, generic, and simplistic (34). This case study demonstrated that developing progress markers for certain outcome pathways can provide further detail into outcome indicators that are typically missing in ToCs. Conversely, OM is critiqued for neglecting systems thinking by focusing solely on outcomes as behavioral change (35). Yet, OM is generally considered to align with select concepts of systems thinking. For example, interrelationships are acknowledged when ‘outcomes’ are defined as patterns of behavior and interactions among stakeholders; perspectives of specific actors are accounted for when setting ‘outcome challenges’ for specific actors; and, boundaries are considered when selecting ‘boundary partners’, including some actors and excluding others. We consider the interrelationships between stakeholders to be particularly important to monitor; in previous (e.g., PigRISK) and current phases of the program, we considered our partnership as a separate unit of analysis, collecting and sharing reflections about the partnership to ensure various actors operate smoothly as a functioning team (36). Furthermore, because of OM's orientation toward understanding complex and non-linear relationships between different actors that can shape a program, OM is often understood as a complexity-sensitive method (17, 26). To address the critique that OM lacks systems thinking, mapping systems change using a ToC can help to illuminate how boundary partners are influenced by (and are situated within) a social-political system. Furthermore, the progress markers of “expect to see,” “like to see,” and “love to see” reflect the direct response to program inputs and not necessarily a temporal sequence (37); mapping outcomes and their inter-relations in a ToC enables a stronger understanding of when outcomes might occur.

When comparing our process in combining ToC and OM to other programs operating in low-resource settings, we find a variety of processes. We used ToC as a starting point for OM; we kept the findings from the two tools separated to allow for cross-comparisons and to maximize the potential of both tools. In Balls and Nurova, a ToC was created at the program design stage to guide the monitoring and evaluation of sanitation and hygiene research projects in Zambia, Kenya, Malawi, and Tanzania (38). Their ToC illustrated how outcomes will be monitored through OM progress markers, suggesting an effort to combine ToC and OM findings. In other studies, OM was conducted first. For example, OM and ToC were used to encourage 12 non-governmental organizations working on sustainable forest management in Papua New Guinea to align their efforts (39); the ToC was created after the development of OM progress markers to help visualize the relationships between drafted outcome statements. Similarly, in an evaluation of a disaster risk reduction network in the Asia-Pacific region, OM was used to visualize the relationship among stakeholders, the desired behavioral changes, and progress markers; then, ToC was used to identify and test assumptions behind such changes (21). We see the variation in ToC/OM combinations as a strength and a response to the different needs and priorities of programs. We encourage evaluation practitioners to be explicit about their approach in combining the two and to reflect on implications.

The identification of assumptions underlying the change processes was not found to be particularly difficult, as typically reported in ToC case studies (34); however, one participant emphasized some assumptions were large and require dedicated interventions to address them. A big assumption, for example, is that SafePORK can contribute to the development of food safety certification. Yet, achieving certification is challenging due to short project timelines and the lack of consumer trust surrounding certifications (40). While evaluators have provided clarity on what assumptions are and how to identify them (41, 42), specific guidance is required on whether certain assumptions are better considered as a step along the change pathway or as an assumption underlying the change. In terms of outcome monitoring, we are starting to accumulate a lot of journal entries but struggle in presenting this data in a meaningful way. Some practical examples from the literature visualizing outcome monitoring data would be helpful.

We note several limitations to our study. Similar to the experiences reported in Taye et al. (20), we find the progress markers developed may not have been appropriate or realistic. For instance, progress markers for slaughterhouse workers and pork retailers such as “agreement to take part in intervention” might be too simplistic, while “consistent practice change” might be beyond the scope of the project. And like the experiences reported in Balls and Nurova (38), we found some preliminary data from monitoring journals to be messy and inconsistent. Continued reflections by evaluators, researchers, and participants on the development and use of outcome mapping tools would provide important insights to improve evaluation practice. Furthermore, because SafePORK is ongoing, our ToC is a ‘work in progress’; we will keep in mind design considerations, such as a better description of connections (45), to ensure our ToC is testable. Finally, due to resource constraints, our ToC and OM were based on researcher perspectives only. Developing ToC and OM with multiple stakeholder groups along the pork value chain might have led to a more nuanced ToC and OM and a better understanding of priorities to be included. However, through active participation in research and intervention design, stakeholders indirectly contributed to these evaluation activities.