A number of frameworks and methodologies have emerged to underpin co-production (35) and a recent systematic review of co-production in healthcare identified the same shared principles of various co-production approaches required in the democratic mobilisation of knowledge to improve health care and delivery including: bringing people together as active and equal partners, valuing all knowledge, using a creative approach, and iterative prototyping techniques (36). In operationalising these elements, all share versions of the same five core steps, namely: Set-up, this is where a range of participants are recruited reflective of all stakeholders, including user groups and broader community. It involves clear and collaborative agreement of the intended aims and outcomes of the process, including levels of involvement, the decision-making process, and the use of training to support the process and upskilling of participants; Discovery, where you gain an understanding of the context and issues at hand including exploring various perspectives, concerns and assumptions, and preferences and priorities; Definition, where the insights gained are prioritised, themes, patterns, and key problems are identified, and the feasibility of various solutions understood; Development, this is the opportunity to generate ideas and creative solutions. Early protypes might be developed to support learning through doing and help identify risks and previously unforeseen consequences. As ideas coalesce buy-in from external groups might be sought; and Delivery, the final step involves producing and launching the final solution, and map next steps and future sustainability. It is also an opportune moment to consider the performance and continuance of the co-production process (12, 37) Figure 2.

Below recent evidence of best practice in stakeholder engagement and digital health co-production is presented within each of the five key steps and constructs. This evidence is summarised in Table 1 and further explored below Table 2.

This first step consists of the recruitment of diverse stakeholders and the delineation of the scope and approach particular to the co-production initiative. In the context of digital health it has been recommended that hybrid co-production teams are created consisting of at least three diverse, stakeholder groups: patients and/or citizens, a range of senior decision-makers and providers from health and social care organisations, and digital health technology developers and suppliers from companies of various expertise, size, and maturity (38). These three groups are hereon referred to as “patients”, “staff” and “developers”.

There is consensus that the recruitment of patients or citizens to any healthcare related activity can be improved by addressing issues of awareness, opportunity, and resource (39, 40). To ensure engagement of all patient participants including underserved populations it is likely that some basic training in the core elements of digital health technologies is warranted (39). This might also include a lengthier period of formative design and development as the process continues to allow individuals to become adjusted to the concepts involved (40). Participation can be supported by flexible approaches in the timing, mode, and location of co-production activities, such as utilising community-situated venues and timing co-production activities to accommodate responsibilities of work and family (41). The payment of travel and expenses and the provision of vouchers is commonplace in patient engagement in health improvement activities, often guided by national bodies (84) but in the context of digital co-production this might usefully involve the provision of hardware or data packages to support familiarisation with digital technology. Finally, to ensure the intended diversity it is important to monitor the demographics of those enrolled (37).

The importance of enlisting a broad range of staff, representative of a range of digital experience, roles, and responsibilities, is understood: not only for the value of their individual experience, but also their subsequent influence on the acceptance of the digital solution amongst colleagues (42). There is evidence that it tends to be the digitally literate staff that enlist or support digital co-production with those less digitally inclined often overlooked (42). Similar to patients, to encourage the involvement of staff with limited digital experience some education may be needed as to the key concepts and capabilities of digital health. There are also more broadly recognised barriers to recruiting and retaining staff for healthcare initiatives relating to time and resource, with the understanding that meaningful participation can be encouraged by providing them with dedicated time and cover, acknowledging their participation as continuous professional development (43).

Arguably the least understood or explored group of stakeholders in digital co-production are those developing digital health technologies. Although many companies in the sector will be aware and have experience of user-centred design, the closely related process used in computer sciences (45); it is distinct from co-production in that understanding the needs of users occurs apart from the design process (whereas users are integrally involved in the design process in co-production) (44). The range of companies involved can range from non-profit social enterprise companies to international “Big Tech” such as Google and Palantir, with global interests in the use of data banks, aggregation platforms, and artificial intelligence (46). This can lead to significant variation in the understanding of patients and the healthcare environment, with similar differences in the resource and opportunity to engage in iterative co-production (47).

As with newly combined co-production teams in other domains of healthcare, this step should include the provision of training in what the process entails, the principles of equality and equitability, and the ask of their time and resource (48). This is also a timely opportunity for transparent and consensual agreement of the scope of the work to align the ambitions of those participating (49, 73).

The discovery step contains the discussion of challenges, assumptions and the setting of priorities for the co-production team. In a growing number of patients, particularly those from underserved populations this might involve addressing or overcoming issues of mistrust in centralised care organisations (41), as well as concerns around the security and confidentiality of digital health tools and data (50, 51, 85, 86). There is evidence that moderators might help to build trust by accommodating participants cultural backgrounds, primary languages, and cultural and faith practices (52–55).

Hesitancy towards digital health has also been reported amongst care providers, with concerns around the safety and reliability of digital health tools (56), with the recognition that moderators might start with an explicit discussion of the potential issues of using digital health tools (87). These include perceptions that digital solutions will reduce the in-person patient contact that many value and the role of hybrid/analogue or digitally-enhanced systems (38). There are also previously reported suspicions amongst senior-decision that co-production risks their commitment to potentially expensive or resource intensive solutions in order to meet emerging needs and preferences (58).

It is important that the potential funding model for the solution is discussed particularly for private sector technology developers, and commissioners where the additional cost can render digital solutions unsustainable (59). Although there are recommendations for the need for adaptable and value-based financing mechanisms, evidence of successful funding models is lacking (61, 88). Associated with this is the need to explore the expected profit margins (89), and a strategy agreed for where intellectual property lies (60), particularly in publicly funded and hybrid public/privately funded healthcare systems (61). These discussions require balancing what is desirable vs. what is affordable, not only to the health service but also patients, particularly underserved populations (62).

To support these initial conversations and potential solutions co-production initiatives typically use methods that involve abstraction and verbal communication such as open brainstorming (19). However, such approaches are more familiar to those exposed to formal education, and it has been suggested that carefully constraining the issue under discussion and using iterative sessions better suits a broader range of participants and provides more workable solutions (63, 64). This includes individual sessions structured around who is the target user, why they would use the tool/solution, the context in which they will use it, and early thoughts as to how the project team will gauge success of the solution (65).

This step involves discussing the feasibility of the proposed solutions in terms of their fit, predominantly with patients and existing health services, though with implications for those developing and producing the technology (90). For underserved populations there are a number of well-rehearsed barriers to the take-up of digital health offers including access to hardware such as smart phones or PCs, the cost of data packages, and reliable internet connectivity and broader issues of digital and health literacy (66).

Health service representatives have a different set of constraints to consider. These relate to the characteristics on individual providers, their work practices, as well as the compatibility with existing care pathways and workflows (67). It is at this step where the training requirements of existing staff might be considered with acknowledged difficulties of self or experiential learning in delivering novel digital offers (70), and whether digital champions might be used to support patients and staff (71). This is also the point at which the impact on existing work processes must be understood as well as broader considerations of infrastructure including the necessary data assets, compatibility with existing IT systems, and the return on investment (68). These decision can be supported by paper prototyping, including sketches, diagrams, and storyboards are fast and inexpensive to create, and recommended for these earliest stages where the design direction is vague (69).

There are potentially difficult conversations to be had during this step where conflicting agendas are being aligned and difficult decisions being made, and there are recommendations that this is an appropriate time to understand the level of continued engagement across stakeholders. There are tools developed for this purpose that enable an understanding of levels of ownership, responsibility, and interactivity in the co-production team (37, 72).

This step involves deciding which idea or ideas should be taken forward and refined, alongside developing the plan for implementation. In making these choices it is important to recognise that each group of stakeholders will have varying priorities, and proposed solutions might be usefully scored on the different attributes valued by each (37), and categorised within the three domains of good design in healthcare; efficiency, safety, and usability (74). There are tools available designed to support equitable decision making amongst diverse groups which may be appropriate in this instance (91). These were borne of multi-criteria decision making that is more commonly understood within the field of operational research where alternatives are analysed with respect to a set of multiple (and often conflicting) criteria (75).

Refinement of the solution or tool requires digital rapid prototyping techniques are used to test more realistic and solid ideas; they should be realistic enough to accurately test most interface elements (69). They can be built using purposely developed prototyping tools and software (e.g., Marvel or Proto) or simple versions can also be made using presentation software like PowerPoint or Keynote (69). They are a cost-effective means of creating a prototype which would then be explored and further improved through a multi-phase (and pre-clinical) testing process (28, 76, 92, 93). Finally a high-fidelity prototype might be produced, and while valuable are also expensive and time-consuming and best used to refine near final versions or where complex interventions require accurate simulation (69).

A pre-determined time period of implementation should be agreed, with clearly defined roles and responsibilities of the stakeholders involved (78). A number of structured implementation frameworks for digital health have been developed (77, 79, 80). The implementation of novel digital health technologies is complicated by the intended functionality and identity of the intended user e.g., patients vs. providers) and it can evolve as they interact with surrounding system and processes (94). Though there is a lack of standardisation of which frameworks apply in which context, its recognised that they should involve a mixed method analysis of the impact, uptake, user experience and working mechanisms of the digital solution (77, 80) including a prior data-collection plan, agreed criteria for success or failure (78),.

The final step considers the long-term sustainability of the implementation, including assessing aspects of interoperability and integration with existing workflows, how well they support patient engagement and empowerment, and the degree to which the data collected can be used to evidence success and inform further innovation (68). In the UK the National Institute for Clinical and Health Excellence has created a useful three-tiered “Evidence Standards Framework” for digital health, where the level of evidence required to demonstrate effectiveness varies based on the technology's function and potential risk with one Tier 1 being the lowest (82). Specifically, these tiers are (1) Information and support tools (e.g., health tracking apps, symptom diaries); (2) technology used for health behaviour management, or preventative care; and the highest is Tier (3) Technologies that provide treatment or diagnosis (e.g., medical devices, digital therapeutics) (82).

It is also important to grasp the opportunity to learn more of which elements of co-production have proven most effective and how successfully a diverse range of stakeholders have been engaged (83). For example metrics might include the number of participants that remained throughout the process, or self-reported changes in their self-efficacy (38). This includes what moderators have learnt from the process, including their understanding of individual stakeholder groups and how they might be combined (83). Part of this process includes how the co-production dialogue can become continual, where previous co-production initiatives have been criticised for being short-term in their approach instead of continuing to provide feedback and providing tangible demonstration of the value placed on their time and input (95).