Three interactive sessions by the Co-production Network for Wales¹ for the Superbugs core members (Tyrrell, Hatch, and Eberl) and two professional web designers (Fricker, Hullis) emphasised the underlying concepts and philosophies of co-production and how to design, implement, and evaluate the process. Recruitment of co-production partners was primarily done by the existing communication network of all schools across Wales, which was compiled by the Cardiff University School of Medicine Engagement Team. While being limited to virtual meetings due to COVID-19 restrictions in place at the time (Frobel et al., 2022) and not being able to take advantage from in-person teambuilding as foundation for constructive co-production, we were able to involve teachers from the whole of Wales, thus covering a geographical area of 20,000 km². These virtual meetings benefited from our extensive experience with the all-Wales ‘Life Science Challenge’ inter-school competition, which features online quiz rounds (Cardiff University, 2023), and the development and delivery of online and hybrid teaching materials for both school children and university students. We recruited 25 teachers to attend an initial introductory session, delivered over Zoom, where they were introduced to the background and aims behind the ‘Superbugs’ initiative, the idea of developing an educational tool for remote learning, and their potential role in this project. Teachers represented primary and secondary schools across Wales and covered both English- and Welsh-medium education. A total of 15 teachers expressed an interest to continue to be involved in the project and became our official co-production partners.

Once partners had been recruited, co-production unfolded through three online workshops, delivered over Microsoft Teams, and facilitated using interactive tools, such as Mentimeter,² Slido,³ and Padlet.⁴ All co-production workshops were attended by university scientists and engagement professionals, primary and secondary school teachers, and web designers to ensure that all scientific, educational, and technical aspects of the project were represented. With the official launch of the Superbugs website⁵ in October 2021, the project entered the post-co-production phase.

As with our Superbugs pop-up shop project (Tyrrell et al., 2022), a multifaceted evaluator approach was taken, which was, in part, directed by the use of the ‘Measuring What Matters’ tool, provided by Co-production Network for Wales,⁶ a practical task which allowed us to identify the important questions to ask in order to rigorously achieve our evaluation aims. As a result, it was agreed that our evaluation would consist of four distinct elements.

The horizontal evaluation comprised of a self-assessment by project participants combined with peer review, primarily via a self-audit at the beginning, during, and at the end of the project.

For the participatory approach evaluation, we collected direct feedback from primary stakeholder participants through bespoke questionnaires for teachers and school children during the ‘Piloting Draft’ element of the project.

The impact evaluation consisted of two parts—an evaluation of the impact upon our co-production partners with regard to their own personal knowledge of microbiology and AMR and their self-reported confidence and competence in teaching these topics.

Finally, for the product evaluation, we assessed the success of the Superbugs website itself, gauged through monitoring website traffic, feedback collected through questionnaires and interactive elements within the website, and anecdotal feedback.

The overall study design, including a series of co-production workshops, is visualised in Figure 1. When conceiving the project, an initial logic model was developed during the pre-production phase to define our aspirations and the processes by which we would achieve our aims and objectives. Upon recruitment of our co-production partners, the logic model evolved further to more accurately reflect the changing focus and priorities of the wider team. This revision formed a repeated exercise through the course of three co-production workshops, the final output of which is shown in Figure 2.

Workshop 1 was an in-depth introduction to the project and what we were hoping to achieve. This workshop provided a useful understanding of teachers’ individual experience in designing, using, and evaluating digital teaching during the COVID-19 pandemic when schools were forced to resort to remote and hybrid learning over extended periods—a setting which was entirely novel for both teachers and pupils at the time (Table 1). In particular, insights were gained into a range of educational online platforms already employed and how best to communicate for the remainder of the project (Figure 3A). These insights formed the basis of our delivery strategy for the pilot development of Superbugs.online. Teachers also undertook a benchmarking exercise in the form of a simple quiz, delivered via Slido, to gauge their level of knowledge of basic microbiology.

Workshop 2 was centred around developing Superbugs.online in close alignment with the needs of our educational partners and was split into two parts. First, we delivered an interactive lecture introducing teachers to key concepts of microbiology, infection, and AMR. This lecture ensured that all participants were on a consistent level of knowledge for the remainder of the project and were able to engage constructively with the scientific aspects of the discussions. The second half of the workshop included discussions around the new curriculum for Wales (Welsh Government Hwb Team, 2023) to determine how our ambitions might fit into this new scheme and whether and how its confinements might affect the design and content of Superbugs.online. A key point raised unanimously by all teachers was the preference for a non-prescript, flexible resource that would allow educators to apply and adapt Superbugs.online for their own purposes. To quote our partners directly, “Prescription takes away from being able to adapt material to variety of ELOs [expected learning outcomes] across the curriculum,” and “having golden nuggets [of microbiology] that people can incorporate into their own teaching… would be more useful.”

In Workshop 3, we began to develop the initial pages of the website to be reviewed and discussed, and co-produced feedback questionnaires that would sit across the Superbugs platform to allow for continuous monitoring of usage and feedback. Pupil feedback questions were co-produced with the help of a year 5 class at Llanedeyrn Primary School, Cardiff, and year 11 pupils at Ysgol Clywedog, Wrexham, with guidance from their teachers. Additionally, we collected valuable feedback from teachers on the pilot version of Superbugs.online and on how best to promote this website (Table 2).

As a direct result of the co-production workshops, we created a website aiming to be informative, self-explanatory, and non-prescriptive and, at the same time, easy to navigate, interactive, and fun to visit (see Supplementary material). Careful consideration of software solutions and design options ensured that the final product was straight-forward to maintain and develop further by the Superbugs team, without the need for extended support by professional web designers and programmers after the initial set-up and training. Superbugs.online was launched bilingually in English and Welsh in October 2021 (Supplementary Figure S1, Supplementary Table S1). The ease with which the content could be hosted in different languages quickly made us realise the full potential of the project, and with the help of Stòrlann Nàiseanta na Gàidhlig, we were able to launch a Scottish Gaelic version in 2022. Grant funding from An Chomhairle um Oideachas Gaeltachta agus Gaelscolaíochta (COGG) enabled us to have an Irish version in place ready for the new school year 2023/2024. Thus, Superbugs.online now covers all four languages officially used for education at public and private schools across the British and Irish Isles. With a paucity of high-quality modern teaching resources in Welsh, Gaelic, and Irish (compared with English), especially in STEM subjects, this puts the Superbugs project in a unique position to fill an urgent educational need and meet the demands and ambitions from teachers and pupils (Thomas and Parry, 2021; Royal Society for Chemistry, 2023). Although the website is aligned closely to the new Curriculum for Wales, its cross-disciplinary nature and its interactive elements are relevant to teachers and pupils across the United Kingdom, Ireland, and beyond, interweaving science and health education with mathematics, humanities, languages and expressive arts, and training learners to develop numeracy, literacy, and problem-solving skills and stimulate critical thinking, creativity, planning and organising, and digital competence (Table 3).

At the end of the co-production process, the partners were asked to undergo a self-evaluation and provide feedback on individual aspects of the project. All teachers (100%) among the co-production team agreed or strongly agreed with the notion that the project structure had provided enough opportunities for them to contribute, that they had been made to feel comfortable and confident to engage, that they had felt their views and inputs were respected and valued, and that co-production was a positive approach for developing innovative and impactful educational tools (Figure 3B). Importantly, all teachers (100%) also considered that the level of their involvement had been just right for them, with nobody feeling that the level had been too little or too much (data not shown). This positive evaluation was complemented by constructive and supportive verbal feedback (Table 4), demonstrating that the co-production element had been designed optimally to benefit from involving key stakeholders in the conception, design, and implementation of the project.

The Superbugs core team together with the web designers completed the Co-production Network for Wales’s self-audit tool before, during, and at the end of the project. At the beginning of the projects, all five categories (Assets, Networks, Outcomes, Relationships, and Catalysts) only ranked either as “Made a start” or “Making progress” (Figure 4). This initial assessment had improved considerably by the end of the project, with all five categories either ranking as “Doing well” or “Doing as well as you can,” as a testimony of the achievement and learning during the co-production phase.

It was important to the outcomes of the project that we imparted a positive impact on all co-production partners. As such, during Workshop 1, a simple benchmarking exercise was carried out to ascertain the baseline knowledge of our partners on topics around microbiology, AMR, and antibiotic stewardship. All teachers (100%) showed a basic working understanding of antibiotics, their selective toxicity against microorganisms, and some basic good stewardship practice. These competences were encouraging, as being informed to this level would enhance the intellectual level of input that could be gained throughout the co-production phase. However, many teachers suggested to have reservations in their ability to communicate/educate these topics further, with only 46% of teachers feeling confident enough to deliver teaching on AMR and microbiology. Additionally, 31% of teachers indicated at the beginning of the project that they did not feel informed enough to teach on these topics compared with 69% who answered “A little bit” or “Very much.” Importantly, this original hesitation quickly disappeared so that, by the end of the project, all teachers (100%) felt informed and confident in their abilities to design and deliver classes framed around the content of Superbugs.online.

During the development of Superbugs.online, early versions of the website were shared with the pupils of several co-production partners, and feedback from 247 pupils was received. Despite only accessing a prototype of our resources, it was reassuring that 78.1% of pupils rated the preliminary website ‘excellent’ or ‘good’ and that 38.0% would have been happy to share it with friends (compared with 11.4% who would not be happy to do so) (Figure 5). Verbal feedback emphasised the breadth of content that the students found interesting (Supplementary Table S3).

From the time of the official launch, student feedback was collected via feedback forms embedded in the Superbugs website. To avoid biases due to responses from individual visitors with a markedly positive or negative view of our resources, we here focused on feedback given by whole groups of students accessing the website jointly. In one particularly informative exercise at the time of the launch, all students of the same year 6 class attending an English-medium primary school in Cardiff were encouraged by their teacher to complete the online feedback form. Out of 22 submitted questionnaires, 59.1% of pupils rated Superbugs.online ‘excellent’ or ‘good’, 72.7% of those found it excellent or good to navigate, and 68.2% of those found the graphics and visuals excellent or good (Figure 6A).

Of the pupils who visited the corresponding sections, 66.7% of them rated the ‘Adventure Trail’, 88.9% of them rated the ‘Story Time’ section, and 62.5% of them rated the ‘Being a Scientist’ section as excellent or good. Encouragingly, only very few pupils considered Superbugs.online overall or aspects of it ‘poor’ or ‘bad’ (Figure 6B), and at least one of these negative responses may have been due to a mistake as it was accompanied by positive comments in the free text parts of the questionnaire (not shown). After visiting Superbugs.online, 38.1% of pupils felt more confident about speaking regarding science with friends and family, 68.2% of pupils were more interested in science in general, and 27.3% of pupils stated that they were more likely to choose one or more science subjects at A levels (the school leaving certificate in England and Wales). Moreover, only very few pupils gave negative responses (Figure 6C). These overall positive impressions from the questionnaires were reinforced by anecdotal comments provided by pupils about the most interesting thing they had learned and their favourite part of Superbugs.online (Supplementary Tables S3, S4).

The co-production process led to the development of lasting relationships. Several partners remained with the project in a productive and cross-disciplinary team, even after the end of the official project, and are still being consulted on many aspects of Superbugs.online and the wider Superbugs initiative. We very much see these partners now as a part of the core Superbugs team and, as such, no longer officially evaluate their input. Although perhaps idealistic, this observation is certainly an organic route of travel and testifies to the long-term success of our co-production. However, a major learning from this process is also that true co-production is difficult to achieve. The overall project needed to be driven by a core team, and maintaining a co-productive team with partners from external organisations turned out to require significant time and effort on behalf of key individuals to ensure that all project outputs, outcomes, benefits, and impacts were the result of true co-production.

With its provision of multilingual materials, Superbugs.online meets a key educational need in the United Kingdom’s devolved nations (Wales, Scotland, and Northern Ireland) and the Republic of Ireland. However, accurate provision of educational content in four different languages has turned out to be more demanding than anticipated as it needs to be cross-checked by translators and education experts to make sure that the texts are both scientifically accurate and at an appropriate language level for pupils (Department of Education and Skills, 2015; Education Scotland, 2015; Thomas and Parry, 2021; Royal Society for Chemistry, 2023). It is important to note that some Superbugs content is, at present, provided in English only—e.g., certain videos, animations, and online games, where simultaneous translation is technically not possible. Ongoing work aims at replacing those English materials with multilingual alternatives and/or appropriate subtitles wherever possible. The inclusion of a glossary that lists scientific terms and explains them in simpler words for each language is planned but has not been started.

Creating an educational resource is only the beginning of raising awareness and increasing understanding as it needs to be promoted heavily to make it visible and stand out against potential competitors (Hayes et al., 2020; Tarín-Pelló et al., 2022). As such, social media campaigns, direct contact with teachers, a printed brochure, and word of mouth recommendations were all important in promoting Superbugs.online and ensuring its relevance for science education (Supplementary Figures S2, S3). The co-production process helped considerably in this regard, allowing us to gauge the wishes and interests of teachers and pupils alike while simultaneously developing materials that directly catered for our stakeholders’ needs. As of October 2023, Superbugs.online was already listed as a recommended educational website by Hwb (Digital Learning for Wales, the Welsh Government’s provision of educational tools), Gaelic Education (maintained by Stòrlann Nàiseanta na Gàidhlig), and the Federation of European Microbiological Societies (FEMS), with evidence of direct web traffic referrals from each site.

Collecting meaningful feedback from anonymous visitors remains a challenge. Although Superbugs.online features three bespoke (and co-produced) questionnaires for pupils, teachers, and other members of the public alike, uptake of this option has been very poor to date outside a supervised setting in the classroom. This is likely to be due to general survey fatigue (de Koning et al., 2021; Shiyab et al., 2023) and needs to be addressed going forward. The long-term aim is to embed more seamless evaluation strategies as part of the engaging activities. In the meantime, we will continue and intensify our collaborations with teachers and schools to learn how to improve our resources further and keep monitoring web traffic to understand which parts of Superbugs.online are particularly successful. To date, the web and Twitter statistics, together with direct feedback from teachers and pupils, demonstrate the effectiveness of our engagement with key stakeholders and the interest in our resources, thus directly addressing the original objectives set out in the co-production workshops.

Developing and debugging an educational resource such as Superbugs.online and keeping it up to date comes with considerable expenses, even if the content itself is largely contributed free of charge. Costs for long-term website maintenance currently amount to approximately £750 per year for domain registration and Squarespace and Weglot subscriptions in addition to translation and proof-reading services. To date, our team has covered these expenses from grants related to the development and translation of Superbugs.online; however, sustaining a stable provision of our educational website in the long run will necessitate ongoing support.

The possibility to provide online materials to complement our in-person events is an attractive option for extended engagement with the public and was successfully explored at small Superbugs workshops held during the 2023 summer holidays at Swansea Central Library (Swansea, United Kingdom) and the Broadlands Fun Day (Bridgend, United Kingdom). In addition to our ever popular activities such as a microscope station and a ‘Grow your own Microbe’ body swab station, these events also featured guided tours through Superbugs.online and the possibility for visitors to explore the online content at their own pace. The ‘Behind the Scenes’ blog section at present regularly contains summaries and photographs from live events, thereby increasing the audience reach and promoting in-person activities and website alike.