To achieve the goals of the AIMS study, two site champions—one physician and one nurse—were identified at each study site. The site champions function as the primary site contacts for the study and are responsible for overseeing the implementation of the assigned bundle at their site. In addition to identifying champions, we assessed for readiness with a formative evaluation (19), and planned to re-examine implementation with: (a) this process and (b) a future summative evaluation. Beyond these four strategies, six other Expert Recommendations for Implementing Change (ERIC) implementation strategies were selected to target fidelity to the assigned sepsis bundles in the emergency departments of 18 hospital sites (20). First and second, we established ongoing learning collaboratives through which we provide ongoing training. These are monthly virtual meetings offered to all providers and staff within sites randomized to the same bundle arm. Each learning collaborative meeting was focused on a key topic, informed by the developmental formative evaluation (19), and typically included didactic and/or interactive components. In-person meetings with site champions and the Steering Committee took place as an extension of the learning collaboratives and happened at the beginning and middle of the study. Access to a message board hosted on the Society of Critical Care Medicine (SCCM) Connect website was also considered part of the learning collaboratives, as participants could review learning collaborative materials (recordings, slides) and ask or respond to questions via this platform. Third, we developed and distributed educational materials focused on the sepsis bundle components. These materials were designed to address frequent issues that arose with implementation and were provided at the beginning and throughout the study. Fourth, sites received monthly audit and feedback of their adherence to bundles via reports sent by email to the site champions and data abstractors. Meetings with the implementation science team were available to sites upon request to aid in site-specific tailoring on bundle implementation based on audit and feedback reports. Fifth, we developed and implemented tools for quality monitoring through ongoing training of data abstractors at each site. Finally, we supported changes to record systems (i.e., Epic, the electronic health record system) through coaching to sites about ways to more easily abstract data from Epic for the study and to modify Epic order sets and alerts.

The process evaluation occurred at the mid-point of the 30-month implementation effort (in July–August 2024; see Figure 1) and a multi-method data collection process was used. Semi-structured qualitative interviews assessed how implementation efforts have supported bundle implementation, ways to improve the implementation process, and barriers not being addressed. Two sources of quantitative data were collected: (1) learning collaborative satisfaction and (2) usefulness of activities offered at the mid-study in-person meeting. Following analysis of qualitative interviews and learning collaborative surveys, the implementation team and AIMS Study Steering Committee identified and enacted next steps to modify implementation strategies and address ongoing barriers. The overall goal was to understand and address barriers and facilitators of fidelity to the assigned bundle, the primary implementation outcome. Of note, the original plan for the process evaluation was to review recorded learning collaboratives for themes (12); however, given that learning collaboratives often included significant didactic components and were narrow in their scope (i.e., would primarily provide information on the learning collaborative experience), the alternative process specified above was agreed upon by the implementation team as a way to obtain broader feedback on implementation challenges and successes to date. Effectiveness outcomes were not included in this implementation-focused process evaluation as interim analyses could compromise study integrity (21). For similar reasons, fidelity was not aggregated and reviewed; however, monthly creation and review of audit and feedback reports indicated room for improvement in bundle adherence across sites. A manuscript presenting the primary outcomes will be published at the conclusion of the trial.

A semi-structured guide was created by the implementation team for the interviews. The guide asked about: (1) perceptions of the learning collaboratives; (2) ways in which the study had supported bundle implementation; (3) ways in which support could be improved; (4) ongoing barriers not currently being addressed; and (5) desired outcomes from the mid-study in-person meeting. The nurse and physician champions from each site were invited to participate in a virtual interview held on a video conference platform. A total of 24 qualitative interviews were conducted with nurse and physician champions from 17 out of the 18 sites (one site did not respond to our requests to schedule an interview). Although saturation, or the lack of novel data being collected, was observed after 12 interviews, we continued to conduct interviews until we had at least one representative from the 17 sites who responded to our outreach. Interviews lasted approximately 20 min. Rapid qualitative analysis procedures (22, 23) were used given the tight turnaround to inform the in-person meeting agenda and changes to implementation strategies. Coders (HEF, AL, LEG, JFH) met and developed a structured summary template that was used to code individual interviews. Two interviews were coded by the analysis team together to ensure coding consistency and to revise the summary template as needed. The remainder of the interviews were split up and summarized by individual coders. All coding questions were reviewed in weekly coder team meetings. Once complete, the interview summaries were input into a matrix with interview questions as columns and respondents as rows. All coders collectively and collaboratively reviewed the matrix and identified themes by column. In addition to the interview guide questions that asked about learning collaboratives and the upcoming mid-point in-person meeting, two other implementation strategies were repeatedly mentioned in the interviews in response to probes (i.e., audit and feedback and educational materials). Thus, themes were re-organized by implementation strategy after initial analysis of responses in the matrix.

Quantitative surveys regarding perceptions of the learning collaboratives were collected at the end of the August 2024 learning collaborative from attendees (N = 19 respondents). Surveys included 6-items participants rated on a 7-point Likert scale from 1 (strongly disagree) to 7 (strongly agree) regarding satisfaction and appropriateness of the learning collaboratives. Surveys regarding usefulness of the activities offered at the in-person meeting held in December 2024 were collected at the meeting conclusion (N = 25 respondents). Surveys asked about how well each in-person meeting activity (e.g., storyboard rounding) supported (1) networking with and (2) learning from other sites. Responses were provided on a 10-point Likert scale with 10 representing the highest perceived usefulness of each activity. Means of each item were averaged to understand how each activity supported networking with and learning from other AIMS sites.

Following review of the results of the quantitative and qualitative data, the implementation team reviewed barriers and identified areas for improvement. One of the coders (AL) created an exhaustive list of all recommendations that arose from qualitative interviews that might be used to address common barriers. Each of these recommendations was reviewed and rated for both feasibility and impact by the implementation team. Recommendations that were considered highly feasible and highly impactful were brought for consideration by the Steering Committee. The Steering Committee approved the recommended changes and worked collectively to operationalize and enact these changes. Changes made to the implementation strategies were tracked with the FRAME-IS by the implementation team (18). The FRAME-IS catalogs modifications made to implementation strategies by: (1) what is modified; (2) the nature of the modification; (3) the primary goal and rationale for the modification; (4) the timing of the modification; (5) participants in the modification decision-making process; and (6) the scope of the modification.

Audit and Feedback. Champions indicated that audit and feedback reports were helpful to identify performance trends, optimize practices, and assess the progress of their site in comparison to other hospital sites. Some respondents also reported successfully using these reports to engage hospital leadership, presenting data as evidence to support requests for resources or process changes (e.g., to hire a sepsis coordinator or make changes in Epic). A common concern noted with audit and feedback reports were that monthly reports were not frequent enough to be actionable, particularly if they had a more comprehensive internal process for audit and feedback. Others stated that reports were too complex and often led to confusion, and that uploading data for the reports was slow, burdensome, and time-consuming, reducing staff motivation to participate. Champions suggested focusing reports only on key metrics, such as compliance and mortality, clarifying what patients are included in data, and providing more tailored feedback at the site and/or individual staff level. Respondents were generally unaware that meetings with the implementation team were available to review audit and feedback reports, and no respondent indicated utilizing this offering.

Develop and Distribute Educational Materials. Champions reported that the visual educational materials, such as a diagram of the sepsis pathway, were the most utilized resources as they helped organize thinking and could be easily referenced. There were no barriers to use indicated, but participants suggested providing more concise educational content, such as short videos on sepsis concepts.

Learning Collaboratives. Participants appreciated opportunities in the learning collaboratives to engage with and learn from other sites, particularly through interactive activities. They also described enjoying the diversity of topics and included multiple provider groups (e.g., physicians and nurses). The learning collaboratives were viewed to be most helpful when focused on practical solutions and overcoming barriers; topics related to Epic sepsis workflows, causal pathway diagrams, and fluid requirements were particularly favored. For barriers, time constraints limited participants’ abilities to engage with learning collaboratives. While the meetings were recorded and available for viewing, recordings were considered to be less engaging than attending live. Champions also indicated that some learning collaborative topics (e.g., need for fluid resuscitation) were viewed as too basic or redundant, limiting their appeal to more seasoned clinicians. There was also mixed feedback on some components, with some respondents indicating that learning collaboratives focused on presentations from participating sites, staff turnover, and specific case examples were helpful and relevant, while others found them to be irrelevant to their position or context. For example, hearing about experiences of a highly resourced site (e.g., with multiple funded roles to support sepsis bundle implementation) was often perceived as unhelpful for less-resourced sites. There was also some disagreement around frequency of meetings, with some appreciating the consistent monthly offering and others indicating that learning collaboratives took up too much time. Suggested modifications from respondents were to provide discussion topics and the opportunity to submit questions ahead of the learning collaborative. They also identified a desire for practical and actionable takeaways from each meeting, as well as resources and materials for sites to use locally. Quantitative data from the learning collaborative survey (n = 19) indicated high satisfaction with the learning collaboratives overall, with means of at least 6 out of a possible 7 on all queries (Table 1).

SCCM Connect. Respondents noted that the SCCM Connect library was initially helpful for answering questions and accessing resources related to the learning collaboratives. While many noted that their SCCM connect usage decreased over time as familiarity with the study increased, it remained a valuable reference for some. Technical challenges, including logging in, slow or limited responses from peers, difficulties using the platform on a mobile device, and time constraints were noted as barriers to using the site. Participants requested user-friendly technological improvements, a resource that provided an overview of SCCM functions, and structured discussions on the message board.

In-person Meeting . Champions reported that they appreciated the chance to connect, collaborate, and network with other sites at the initial in-person meeting and found interactive and small group activities helped to create a sense of community. However, some respondents noted that they initially had felt concerns that they would not be performing on par with other AIMS sites. The respondents had several suggestions for the upcoming mid-study in-person meeting, including hearing about and sharing successes among sites without promoting competitiveness and expanding discussions and interactive activities where sites can share and learn from other sites’ experiences. Requests were also made for a debrief of the meeting to be shared for individuals who would be unable to attend.

Cross-Cutting Barriers to Sepsis Bundle Compliance. A primary barrier to bundle adherence noted by respondents was that identifying sepsis in complex patients or patients who present with atypical symptoms leads to difficulties in diagnosis and slows the implementation of bundles. Additionally, participants described heavy workloads and competing priorities that interfered with full engagement with sepsis protocols and educational and training initiatives. Epic (EHR) issues were also noted as ongoing barriers, including data extraction from outdated data systems, alert fatigue and desensitization to sepsis-related alerts, and software updates that negatively impact upgraded Epic sepsis processes. There were also several technological barriers related to engaging with some of the implementation strategies, including firewalls and software restrictions that made it difficult to access online resources such as SCCM Connect. Leadership and bureaucratic challenges were reported, with some sites reporting difficulty gaining leadership buy-in, which inhibited interest in and approval of sepsis care improvements to support bundle implementation. This seemed to be particularly challenging within larger hospital networks in which decisions proceed through multiple levels of administration. Finally, participants indicated that high turnover rates made it challenging and burdensome to provide sufficient sepsis bundle training and led to inconsistent application of sepsis protocols.

Cross-Cutting Facilitators to Sepsis Bundle Compliance. While some sites noted difficulties with leadership, most sites reported increased leadership buy-in largely due to AIMS study participation. With study participation, champions indicated the ability to increase sepsis awareness among leadership and to successfully seek and receive additional resources for bundle implementation, such as a dedicated sepsis coordinator. Overall, they noted a more supportive environment for adopting new sepsis protocols. Participants also reported improvement in inter-organizational environment and networks. They cited valuable learning in their interactions with other sites and a sense of community and collaboration that was a product of participation in AIMS study implementation activities.

Implementation team synthesis of the qualitative and quantitative data led to the identification of several areas for improvement. As described above, modifications that were rated by the implementation team as highly feasible and likely to have high impact were brought forth to the Steering Committee. At this stage, suggested adaptations to implementation strategies were operationalized and plans for their execution commenced. Table 2 summarizes the final outcomes of this process, which is also described briefly.

For learning collaboratives, in order to increase engagement with and relevance of topics and to provide resources for those unable to attend live, the discussion topics were publicized 3 months in advance to allow for preparation and question submission. In addition, a template for the structure of the learning collaboratives was developed to guide presenters in their planning and presentation, and summaries including key take away of the meetings began to be distributed. Meetings with the implementation team were advertised at the conclusion of learning collaboratives and at the in-person meeting to increase awareness and promote uptake.

To improve access to and engagement with SCCM Connect, a video-based user guide was created and distributed. Opportunities to connect with other sites about shared topics of interest were also developed by creating breakout rooms with different topics at one of the learning collaborative meetings. Most of the suggestions for the in-person meeting were rated to be feasible and impactful and were ultimately implemented, including increased opportunities for structured and unstructured networking and teambuilding, small group activities that highlighted strengths of each site to reduce competitiveness, and a summary of activities and takeaways for those unable to attend. Attendees at the meeting who completed the post-meeting survey (n = 24) rated the opportunity for unstructured discussion (i.e., breaks and lunch) most supportive of networking and rated storyboard rounding most supportive of learning from other sites, though generally all activities were considered to be supportive of networking and learning (Table 3).

To supplement existing educational materials and ensure access to concise educational materials, short videos on key sepsis concepts and problem areas were developed and distributed among sites. Thus, we expanded upon the plan to distribute educational materials and developed additional materials in response to feedback. Additional detail was added to audit and feedback reports to clarify limitations of data and discrepancies between in-house site reports and AIMS study reports. However, no feasible solutions were identified to address site challenges in generating data and resultant delays in receiving feedback. Notably, these activities were difficult for sites but related more to study procedures than to bundle implementation.

Final modifications were cataloged using the FRAME-IS (Table 4). A total of 15 adaptations were documented, with 7 content modifications, 4 context modifications, 2 evaluation modifications, 1 training modification, and 1 modification that was categorized as both a content and a context modification. The primary goals of the modifications were to improve fidelity to bundle implementation (n = 2), to increase adoption of bundle implementation (n = 1), and to increase the acceptability, appropriateness, or feasibility of the implementation effort (n = 12). All modifications occurred in the implementation phase and were planned and enacted by the research team.