The St. Jude Children’s Hospital (St. Jude) Institutional Review Board approved this study as an exempt, minimal-risk study. Additional local approvals were obtained from centers participating in the CritCom pilot when required.

The preliminary version of CritCom was developed using a 7-step method (1. Literature Review, 2. Measure Development, 3. Cognitive Interviews (English), 4. Translation, 5. Cognitive Interviews (Spanish); 6. Language Synthesis, and 7. Final Review), which has been previously described (28) and is briefly summarized below. This methodology, specifically the rigorous translation process, was used to ensure that the measure was usable in multiple contexts and languages. Throughout the process, we aimed to design a pragmatic measure, which has been defined as a measure that is “important to stakeholders in addition to researchers, low burden, broadly applicable, sensitive to change, and actionable” (29).

First, a literature review was conducted to identify existing tools developed or utilized in healthcare settings to evaluate inter-professional communication. Literature on teamwork was also included at this stage, as these tools often contain domains of communication. Studies with measures addressing communication elements in healthcare were reviewed for common themes, and all relevant survey items were collated. A database comprising 421 questions and 45 domains of communication was obtained from this literature review. The initial domain selection included the constructs with the most significant evidence, frequency of occurrence, and relevance to clinical care. This database of items was then iteratively reviewed by a 21-member panel of global experts in pediatric oncology, interdisciplinary communication, and measure development from 21countries ( Supplemental Table 1 ) to establish content validity and improve cultural sensitivity, producing a draft measure with 52 items across seven domains. This measure focused on childhood cancer care due to the high risk of clinical deterioration in hospitalized children. During these events, interdisciplinary care is necessary for efficient care and improved clinical outcomes.

We conducted cognitive interviews with 36 clinicians from 15 countries. Interviews in English were conducted with nurses and physicians working in the intensive care unit (ICU) or medical wards to identify problematic survey items and to establish face validity. Interviews were conducted by JR, KP, and SM using a standardized interview guide (28) in phases of 3-5 interviews, with changes to the survey based on feedback. Interviews were stopped after eight rounds of weekly meetings when no further changes were needed for the English survey version. To address regionalism, CritCom was translated into Spanish using a forward-back translation process with iterative review by a group of five native Spanish speakers from different countries. Cognitive interviews were then conducted in Spanish using the same techniques as in English (JR and MPT). During this round of cognitive interviews, changes were made to the Spanish and English instruments based on feedback. As edits and clarifications were made, the bilingual research team worked to ensure that the intent of the original items was preserved. (See Supplemental Table 2 for participant demographics of cognitive interviews).

The bilingual expert panel completed a final review to confirm that the measures reflected all relevant communication components identified in the initial review. Additionally, bilingual members of the expert panel reviewed the two versions to ensure that the meaning was maintained between the two languages. This process resulted in a preliminary CritCom tool with 52 items across seven domains (see Supplemental Figure 1 for a summary of the initial CritCom development process).

We piloted a preliminary 52-item CritCom measure globally among hospital staff (ICU and ward nurses and physicians) providing childhood cancer care. Participants were recruited from the St. Jude Global Critical Care Program (30) network of collaborators and pediatric critical care research networks such as Proyecto EVAT (31), POKER (PICU Oncology Kids Europe Research Group) (32), and PALISI (Pediatric Acute Lung Injury and Sepsis Investigators) (33). Recruitment asked clinicians to fill out an application indicating interest in participating individually or as a hospital; those selecting hospital participation were instructed to provide a list of emails for eligible participants at their center. Eligible participants included any clinical staff involved in the clinical care of hospitalized children with cancer who may have experienced deterioration. Those who do not take care of children with cancer or do not care for these children during deterioration were excluded from this study.

After identifying the eligible participants, CritCom was administered electronically via an anonymous Qualtrics survey in English or Spanish (based on the participant’s country). The participants were given six weeks to respond and receive weekly reminders. Participants provided demographic information about themselves and their organizations. Finally, they were asked to complete a set of questions regarding CritCom usability (see Supplemental Figure 2 for the demographic and usability questions of the pilot measure).

The data for the Spanish and English versions of the tool were managed and analyzed using R, a programming language for statistical computing (34). Data were explored and described before performing psychometric analyses, which focused on the measure’s reliability. Within R, the packages used for psychometric analysis were Classical Test Theory (CTT) and lavaan, which were used for latent variable analysis. Our team has expertise in quantitative measurement development, and these analytical methods were informed by our prior work (35).

After initial data cleaning and descriptive analyses, psychometric data analysis was performed, and these results provided further measurement refinement. Confirmatory factor analysis (CFA) was initially used to confirm the hypothesized domain structure that emerged from earlier development stages (36). Confirmatory factor analysis consists of developing a statistical model to test the pre-identified factor (domain) structure compared to a structure where all items exist within one domain. These analyses helped identify poorly performing domains and items and understand if our proposed subscale structure was correct. We anticipated that some CritCom domains would have intercorrelations because of their conceptual overlap. Additionally, we used a robust full-information maximum likelihood to handle non-normality in the data appropriately. These psychometric analyses were then used to exclude the items and restructure the domains. Items were dropped if they had poor loadings on the construct or required more variability. One domain was dropped from the instrument due to poor performance in the CFA, and the other was re-conceptualized after dropping poorly performing items.

After the final tool was developed, we re-conducted CFA (37). These analyses were used to assess the final conceptual structure of the domains. We assessed three measures of fit: comparative fit index (CFI), root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR) (38). The CFI ranges from 0-1, where larger values indicate a better model fit. RMSEA assesses the covariance between the models, and the ideal output is less than 0.05. Finally, the SRMR is an analysis of the residuals in the model, with a desired output less than 0.05.

The usability of the Critcom tool was assessed through descriptive statistics of the usability questions. Additionally, we used the pragmatic scale of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS) to assess the quality of the developed measure (39). This scale consists of five categories and provides a Likert scale assessment ranging from -1 (poor) to 4 (excellent).

A total of 433 participants from 42 Spanish- and English-speaking hospitals in 22 countries completed the pilot CritCom ( Table 1 ), representing a response rate of 62.8%. Participants included nurses (57.9%), physicians at all levels of training (36.1%), and other clinical staff, including respiratory therapists. The participants performed clinical work across a range of hospital units/ward types, including the ICU (34.9%), oncology ward (26.3%), and general medical ward (18.7%). The participants were primarily from upper-income countries (50%; Table 1 ; Supplemental Figure 3 ).

After the initial development, the 52-item preliminary CritCom measure was assessed for its structure using CFA and individual item analyses. The results of the CFA during this process are shown in Table 2 . The initial baseline model included 52 original items in one domain, and the original pilot included all original items in the seven-domain structure. After assessing these models, 14 items were dropped because of poor performance, such as items with low item-total correlations or those loaded poorly onto the domain structure ( Table 2 ). One of the domains (systems) was split into two as the items did not fit within a single construct, resulting in eight domains. Two domains (mechanisms, modes, and systems) were dropped due to conceptual ambiguity and poor psychometric performance, resulting in a final instrument that included 30 items within six domains.

The final CritCom tool measured the quality of clinical communication using the following domains: (1) actionable, (2) clarity, (3) tone, (4) empowerment, (5) collaboration and teamwork, and (6) leadership ( Table 3 ; Supplemental Figure 4 ). CFA results demonstrated an improvement in the overall structure throughout the refinement of the measure. This culminated in the results for the final structure, which had a good fit with the model. This is illustrated through the CFI = 0.94 (desired statistic greater than.90), RMSEA = 0.045 (desired statistic less than 0.05), and SRMR = 0.049 (desired statistic less than 0.05). These indices indicate a good fit of the measurement model (i.e., the six domains of CritCom) to the observed data (28) ( Table 2 ). The CFA approach we used here follows established analytical and reporting best practice guidelines (40).

Table 4 presents the number of items and Cronbach’s alpha, which measures the internal consistency (reliability) for each domain in the original measure and after-measure refinement. These scores highlighted the internal consistency of each domain. The final measure had excellent internal consistency, with Cronbach’s alpha ranging from 0.81 – 0.86, suggesting good subscale reliability. This indicates that the items fit well within one domain and target the same underlying component (i.e., the construct) of communication quality.

CritCom results were calculated by computing the average of each item within a domain and then calculating the overall average for the total score. Table 4 presents the pilot’s final measure scores, with overall scores ranging from one (representing poor-quality communication) to five (high-quality communication) in each domain. Overall, tone had the lowest and actionable the highest domain scores, respectively. Figure 1 illustrates the distribution of the overall CritCom scores, showing good variability in the sample, although most total scores ranged from 3 to 5.

Additionally, we assessed domain scores by language (English or Spanish) to understand how CritCom performed in each language ( Figure 2 ). The profile plot shows that the pattern of domain scores did not vary appreciably between assessment languages, indicating similar measure performances in English and Spanish.

After completing the CritCom measure, the participants were asked to assess the instrument’s usability ( Figure 3 ). The vast majority of the participants agreed or strongly agreed that the survey was easy to use (94.0%), described the questions as clear (94.7%), felt it correctly described communication in their setting (89.8%), and agreed to cover concepts that are important within their clinical setting (96.1%). Overall, these findings demonstrate that participants found the tool usable and that it resonated with the concept they believed to be important.

In the PAPERS categories, CritCom scored good (3) on brevity (30 items), readability (between 8^th and 12th-grade reading levels), and burden (manual calculation, although it provides recommendations for handling missing data). It scored excellent (4) for cost (free) and training (no training required). Overall, this resulted in a PAPERS score of 17 out of 20, indicating that this tool is usable and practical for clinicians and researchers (41). ( Supplemental Figure 5 ).