Front. Public Health Frontiers in Public Health Front. Public Health 2296-2565 Frontiers Media S.A. 10.3389/fpubh.2021.740238 Public Health Study Protocol Application of the RE-AIM Framework for the Pediatric Mild Traumatic Brain Injury Evaluation and Management Intervention: A Study Protocol for Program Evaluation Tucker Paula W. 1 * Bull Rachel 2 Hall Alex 2 Moran Tim P. 2 Jain Shabnam 3 Sathian Usha 4 Simon Harold K. 3 Gioia Gerard A. 5 Ratcliff Jonathan J. 2 Wright David W. 2 1Nell Hodgson Woodruff School of Nursing, Emory University, Atlanta, GA, United States 2Department of Emergency Medicine, Emory University School of Medicine, Atlanta, GA, United States 3Departments of Pediatrics and Emergency Medicine, Emory University School of Medicine and Children's Healthcare of Atlanta, Atlanta, GA, United States 4Urgent Care and Community Care Services, Children's Healthcare of Atlanta: Department of Pediatrics, Emory University School of Medicine, Atlanta, GA, United States 5Division of Pediatric Neuropsychology, Children's National Hospital, Rockville, MD, United States

Edited by: Takeo Mukai, The University of Tokyo, Japan

Reviewed by: Shunichiro Tsuji, Shiga University of Medical Science, Japan; Alia Lauren Yasen, University of Oregon, United States

 *Correspondence: Paula W. Tucker paula.tucker@emory.edu

This article was submitted to Children and Health, a section of the journal Frontiers in Public Health

 17 02 2022 2021 9 740238 12 07 2021 29 12 2021 Copyright © 2022 Tucker, Bull, Hall, Moran, Jain, Sathian, Simon, Gioia, Ratcliff and Wright. 2022 Tucker, Bull, Hall, Moran, Jain, Sathian, Simon, Gioia, Ratcliff and Wright

This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

 Background

Children who experience a mild Traumatic Brain Injury (mTBI) may encounter cognitive and behavioral changes that often negatively impact school performance. Communication linkages between the various healthcare systems and school systems are rarely well-coordinated, placing children with an mTBI at risk for prolonged recovery, adverse impact on learning, and mTBI re-exposure. The objective of this study is to rigorously appraise the pediatric Mild Traumatic Brain Injury Evaluation and Management (TEaM) Intervention that was designed to enhance diagnosis and management of pediatric mTBI through enhanced patient discharge instructions and communication linkages between school and primary care providers.

 Methods

This is a combined randomized and 2 × 2 quasi-experimental study design with educational and technology interventions occurring at the clinician level with patient and school outcomes as key endpoints. The RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework will be utilized as a mix methods approach to appraise a multi-disciplinary, multi-setting intervention with the intent of improving outcomes for children who have experienced mTBI.

 Discussion

Utilization of the RE-AIM framework complemented with qualitative inquiry is suitable for evaluating effectiveness of the TEaM Intervention with the aim of emphasizing priorities regarding pediatric mTBI. This program evaluation has the potential to support the knowledge needed to critically appraise the impact of mTBI recovery interventions across multiple settings, enabling uptake of the best-available evidence within clinical practice.

 RE-AIM mix-methods program evaluation mild traumatic brain injury concussion pediatric intervention Centers for Disease Control and Prevention10.13039/100000030 Introduction

Mild Traumatic Brain Injury (mTBI) is a considerable public health problem that is responsible for more than 70% of traumatic brain injuries in the United States (1, 2). Between 2004 and 2013, head injury among children up to 17 years of age accounted for 1.6 million outpatient (68%) and Emergency Department (ED) (32%) visits (3). In 2014 alone, TBI-associated ED visits, hospitalizations, and deaths comprised more than 837,000 children (4).

Children who sustain an mTBI may be seen in multiple healthcare settings across their recovery, such as primary care, urgent care, specialty clinics, ED, and inpatient settings (5, 6), while most of the recovery phase occurs in the home and school settings (5). Unfortunately, inconsistency in management approaches and treatment recommendations exist across the various healthcare settings (5, 7). Adding to this complexity is the reality that communication linkages between the healthcare systems and the schools are frail. As a result, this lack of communication and care coordination increases the risk of re-exposure to mTBI, less effective management in healthcare settings, and delayed recovery (5).

Current needs and opportunities for improvement exist in the management of mTBI in children, explicitly traversing across the care continuum from emergency and urgent care to primary care settings and then to the schools where the child spends a significant amount of their day. Therefore, we view the ecology of mTBI services as akin to a neighborhood, with various elements that must work together to be a successful community.

A growing number of studies offer insight into the consequences of mTBI (8–13). Children who sustain a TBI, including mTBI, may encounter cognitive and behavioral changes that can negatively influence school performance (13–16). These changes may even influence the quality of life in childhood and adult years (8, 17, 18). For most children who experience symptoms related to mTBI, recovery typically occurs between 1 and 6 weeks (19–21); however, symptoms can persist up to 3 months or longer following injury (10, 17, 21, 22).

In 2018, the Centers for Disease Control and Prevention (CDC) and the Pediatric Mild Traumatic Brain Injury Workgroup published the first evidence-based guidelines encompassing 19 best practice recommendations regarding identification, prognosis, and management of mTBI among children including recommendations for return to school (23). These recommendations were based on the findings of a recent systematic review (24). Specific intervention strategies and recommendations have also been proposed to promote improved academic and healthcare outcomes for students with mTBI (5, 7, 25–28). Despite a better understanding of pediatric mTBI, there remains heterogeneity in diagnosing and managing the condition. This variation may be attributed to a lack of standardization and widespread adoption of evidence-based clinical guidelines, including guidance on return to school (29–34). Therefore, a systematic approach and the integration of clinical decision support tools to facilitate knowledge translation for advancing diagnosis and management of pediatric mTBI are promising strategies that may improve clinician training and decision making in the care delivery process for this vulnerable population (31–33). To ensure maximum effectiveness, the diagnosis and individualized management strategies from the primary healthcare provider must be communicated to the child, parent, and school personnel through a standardized, consistent pathway (25, 28).

In response to the pediatric mTBI public health concern, the CDC released a notice of funding opportunity to improve pediatric mTBI outcomes through healthcare provider training, clinical decision support, and discharge instructions. The study funded through this notice, the mild Traumatic Brain Injury Evaluation and Management (TEaM) Intervention, is a multi-faceted intervention that includes sophisticated decision support in pediatric primary care (PC), pediatric urgent care (UC), and pediatric EDs. The study's focus is directed toward improved diagnosis and management of pediatric mTBI, including patient discharge education and enhanced communication among school and healthcare providers. The purpose of this manuscript is to apply the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework to critically evaluate the planned multi-disciplinary, multi-setting intervention designed to improve outcomes for children who have experienced mTBI (35–37). In addition to overall program evaluation, the RE-AIM Qualitative Evaluation for Systematic Translation (RE-AIM QuEST) developed by Forman et al. will be used to inform qualitative data collection for the planning and formative evaluation of the TEaM Intervention (38). The focus of this paper is to define the feasibility, sustainability, and capability of scaling up implementation of the planned TEaM Intervention, including implementation of the mTBI training program across clinical settings.

The RE-AIM model was first developed in 1999 by Glasgow et al. (35) as an evaluative framework to assess the public health influence of health promotion initiatives with an emphasis on gathering information regarding their Reach, Effectiveness, Adoption, Implementation, and Maintenance (35). It assists researchers in the planning, evaluating, and reporting of healthcare intervention effectiveness (35–37). The RE-AIM model considers the long-term impact of an intervention within real-world settings, using a multi-level (individual, organizational) approach (35), and is adaptable to effectiveness studies such as randomized control trials (39, 40), time-series, as well as quasi-experimental designs (41, 42). In this model, Reach is the number, percentage, and representativeness of individuals agreeing to participate in an intervention, including the rationale for participation and non-participation. Effectiveness is the influence of the intervention on significant outcomes such as quality of life, economic, and potential negative or positive effects. Adoption is the number, percentage, and representation of settings and participants agreeing to deliver the initiative, including the rationale for adoption and non-adoption. Implementation is the consistency to which various intervention components are conducted as proposed, including intervention modifications and implementation strategies. Lastly, Maintenance is the degree to which the intervention becomes part of routine organizational processes, and the intervention's long-term impacts on individual-level outcomes (36, 37). Emphasis on external validity originates from the Reach and Adoption dimensions, and internal validity originates from Effectiveness and Implementation dimensions of the model (43).

As a mix-methods (quantitative, qualitative) approach to evaluate the planned TEaM Intervention, this manuscript will attempt to answer the following questions directed by both the RE-AIM (36, 44) and RE-AIM QuEST (38) frameworks: Is the TEaM Intervention reaching healthcare providers who diagnose and manage children with mTBI? How effective is the TEaM Intervention on mTBI-related outcomes, including adverse effects on academic performance and health-related quality of life among school-aged children with concussions? To what extent did healthcare settings that were proposed to deliver the TEaM Intervention actually participate? To what degree was the TEaM Intervention consistently deployed across study sites? To what degree and time point did the TEaM Intervention become standard of practice and sustain effectiveness? See Table 1 for TEaM Evaluation Data Collection Plan.

TEaM evaluation data collection plan.

RE-AIM dimensions Quantitative Qualitative
REACH
Numerator # of HCPs who participate in the study (intervention/control) Denominator Total # of HCPs eligible to consent (intervention/control/non-participation) across settings (ED, UC, PC) Data sources ED personnel data eMR surveys/questionnaires semi-structured interviews focus groups provider training records Inclusion criteria # of HCPs who consent to the study Exclusion criteria # of HCPs unwilling or ineligible to consent Representativeness among HCPs Basic demographics Similarities/differences: participation/non-participation, control/intervention groups Barriers/facilitators to HCP recruitment, reasons for participation and non-participation Formative evaluation (study phase 2-provider recruitment) Assess barriers to recruitment and how barriers were addressed Assess provider knowledge and skills in mTBI diagnosis and management Obtain provider feedback regarding mTBI training modules
EFFECTIVENESS
Validated study instruments CDC ACE Tools (45, 46) CLASS-3 (parent/student report) (14, 47) PCSI-2/PCEI (48, 49) PEDS QL4 (50, 51) Data sources eMR surveys/questionnaires telephone interviews Definition Comparative change/improvements in mTBI clinical outcome measures over time (1, 2, 4, and 12 weeks) mTBI Clinical Outcome Measures Primary outcomes academic problems/decline after return to school CLASS-3 (parent/student report) (14, 47) (weeks 1, 2, 4) Secondary outcomes mTBI symptoms post injury PCSI-2 (48) (weeks 1, 2, 4, and 12) anxiety/mood changes post injury AND after return to school PCSI-2/PCEI (48, 49) (week 1) PCSI-2 emotion scale (48) PCEI emotional control scale (49) CLASS-3 school stresses (14, 47) after return to school (weeks 2, 4, and 12) PEDS QL4 quality of life after return to school (50, 51) (weeks 4 and 12) Time to Recovery (45, 46) Circumstances and processes influencing TEaM Intervention effectiveness Differences in patient-level outcomes between intervention vs. control groups across study sites (ED, UC, PCP)
ADOPTION
(ED, UC, PC settings) Numerator # of settings that adopted TEaM Intervention components Denominator Total # of settings approached Data sources eMR training records semi-structured interviews site visits observations Definition #/% of settings willing to adopt TEaM Intervention components Representativeness of settings Basic demographics Similarities/differences among settings: ED, UC, PC (baseline initiatives vs. intervention components), adoption vs. non-adoption Understand contextual factors influencing adoption of TEaM Intervention components Identify barriers and facilitators to adoption and non-adoption across study sites Formative evaluation (study phase 1-building a foundation) Assess stakeholder perceptions and feedback regarding educational and CDS tools Identify anticipated barriers/facilitators to adoption of TEaM Intervention components across delivery systems (ED, UC, PC)
IMPLEMENTATION
Numerator # of individual TEaM intervention components implemented Denominator Total # of TEaM intervention components Data sources eMR informal interviews focus groups among providers site visits observations Definition Degree of TEaM intervention components consistently implemented across sites Setting/TEaM intervention components ED and UC settings1. Provider Training Modules2. eMR Triage screening/provider alert3. eMR Concussion Smart Form Template (documentation, decision support, management pearls)4. Triggered by eMR Concussion Smart Form Template   → CDC evidence-based discharge instruction   →Communication Linkage Letters- Return to School with suggested accommodations- Return to Primary Care with ED findings Primary care settings1. Concussion Academy Skill Training (CAST) and Training Modules2. eMR Concussion Smart Form Template (documentation, decision support, management pearls)3. CDC ACE tool (45, 46)4. CDC ACE care plan/education (45, 46)5. Linkage letters- Return to School with STAMP suggested accommodations and supports Understand variations of implementing the TEaM Intervention components Approach to adapting each component of the TEaM Intervention across delivery systems Implementation barriers and facilitators of the TEaM Intervention including contextual factors and operational practices and how they were addressed
MAINTENANCE
data sources eMR informal interviews site visits Definition Utilization of TEaM intervention components among HCPs 6 months following the study period Identification and engagement of champions across delivery systems Understand barriers to sustainability of the TEaM Intervention across study sites
Formative evaluation (completion of study phase 3-subject enrollment and pre-decision support)/study phase 6 (study wrap up) Evaluate provider experiences, perceptions, barriers and facilitators regarding TEaM Intervention components Asses contextual factors underlying facilitators and barriers to implementation of TEaM Intervention components
Methods Study Design Overview

The TEaM study is a combined randomized and 2 × 2 quasi-experimental study design with training and technology interventions occurring at the provider level involving ED, UC, PC healthcare providers (HCPs) across the Children's Healthcare of Atlanta system, with patient and school outcomes as key endpoints. At the HCP level, this study is randomized, and at the patient level, this study is a quasi-experimental with a pre and post design to facilitate analysis of the impact of our Electronic Medical Record (eMR) support tool on children's health outcomes.

 Participants/Setting

Participants in this study include HCPs in ED, UC, and PC settings affiliated with Children's Healthcare of Atlanta. HCPs will include Physicians and Advanced Practice Providers (nurse practitioners and physician assistants). HCP level randomization will be stratified by practice site to ensure that the distribution of control and intervention providers is approximately equal within sites. Providers will be sent an electronic informed consent form (eICF) using Research Electronic Data Capture (REDCap™) tools (52) for consent to participate.

Children ages 5–18 years old who screen positive and/or are diagnosed with mTBI/concussion will be recruited following discharge from their index injury visit, which may have occurred in either ED, UC or PC settings. Regular reports of children who screen positive through the decision support tool will be sent to the research team throughout the 32-month study period. After confirmation of an evaluation by a participating provider and either a positive screen or diagnosis of mTBI/concussion, an initial phone call will be made to the parent.

Upon the first contact, the parent will be given a very brief explanation of the study at which point, an option to opt-out will be offered. If they choose to opt-in, a simple validated symptom questionnaire, the CDC Acute Concussion Evaluation (ACE) tool (45) will be used to ensure that the child meets eligibility requirements. At this point, if the child is eligible, parental verbal consent will be obtained in order to proceed with further explanation of the study. A total of 532 consecutive children (see section Data Analyses) with mTBI meeting inclusion/exclusion criteria will be recruited. The goal is to maintain a lost to follow up (LTF) rate of <20% for a total sample of 425. Enrolled children will be treated for up to 6 months post-injury to assess the impact of the TEaM Intervention on pediatric mTBI-related outcomes.

 <italic>TEaM</italic> Intervention

The TEaM Intervention was developed by the investigators through rigorous evaluation of the literature, incorporation of the CDC guidelines (23, 45, 46), and assimilation of best practices currently adopted by national experts (31, 45). The TEaM Intervention will be adapted for this study in collaboration with stakeholders from pediatric neuropsychology, emergency medicine, neurosurgery, sports medicine, primary care, ambulatory care, clinical informatics, nursing, school linkage experts, educators, school health personnel, as well as members of the CDC Pediatric Mild Traumatic Brain Injury Guideline Workgroup (23). The intervention consists of provider training in mTBI evaluation, care coordination and management protocols, eMR decision support system, and linkage materials. The eMR will be leveraged to refine the provider training portion of the intervention to improve screening, management, and linkage throughout the clinical care continuum with the goal of producing an effective and scalable intervention aimed at reducing mTBI-related school problems/performance, improving patient symptom recovery, and reducing post-injury complications. The TEaM Intervention is designed to enhance adoption and standardize best practices; the study tools are designed to fully function without interfering with a system that already has a high-functioning mTBI program. Most importantly, this intervention enables potential generalizability across various clinical settings; thereby, improving the translation and adoption of evidence-based management for patients, caregivers, providers, and school personnel to promote recovery following pediatric mTBI.

 Data Collection Reach (Provider Level): Is the <italic>TEaM</italic> Intervention Reaching HCPs Diagnosing and Managing Children With mTBI?

We will determine how many and what proportion of HCPs receive TEaM Intervention training within each proposed site (ED, UC, PC) (36). The numerator is defined as the number of HCPs who agree to participate and receive the TEaM Intervention training. The denominator is defined as the total number of potential HCPs who are eligible for the study across each study site (53). We will consider representativeness by identifying similarities and/or differences between HCPs (intervention vs. control group and participation vs. non-participation). Comparisons among the control and intervention groups regarding compliance to the protocol and basic demographic characteristics will also be assessed (36). Qualitative inquiry will complement our data collection on Reach. Our objective is to understand how HCPs perceive the TEaM Intervention training, including barriers and facilitators to recruitment, participation, and non-participation across sites (38). Data sources will include eMR, provider questionnaires, semi-structured interviews, focus groups, and attendance records from provider training.

 Effectiveness/Efficacy (Patient Level): How Effective Is the <italic>TEaM</italic> Intervention at Improving mTBI-Related Outcomes, Including Academic Performance and Health-Related Quality of Life Among School-Aged Children With Concussions?

The primary outcome of the TEaM Intervention is to determine if implementation of an evidence-based/best practice mTBI intervention with multi-setting communication linkages will decrease mTBI-related complications among school-aged children, a universal endpoint. Effectiveness of the intervention for the intervention group vs. controls (38) will be examined on important patient level outcomes utilizing validated instruments. The primary outcome is degree of academic problems as reported by the parent and student on the Concussion Learning Assessment and School Survey, 3rd Ed (CLASS-3) (14, 47). Secondary outcomes will be the emotional responses of the students as measured by the Emotional Symptom scale of the PostConcussion Symptom Inventory-2 (PCSI-2) (48), the Emotional Control scale of the PostConcussion Executive Inventory (PCEI) (49), and the School Stresses scale of the CLASS-3. General well-being will be assessed by the Pediatric Quality of Life Inventory (PEDS QL4) (50, 51). Each of these measures will be administered at 1, 2, 4, and 12 weeks. Finally, time to recovery (45, 46) will be assessed for each study group. Attrition rates of patients will also be determined. Further data collection will include qualitative inquiry on effectiveness. Our objective is to determine the circumstances and processes that resulted in TEaM Intervention effectiveness and the differences in patient-level outcome measures across study sites (38). Data sources will include eMR, surveys, telephone interviews with patients and parents.

 Adoption (Organizational Level): To What Extent Did Healthcare Settings That Were Proposed to Deliver the <italic>TEaM</italic> Intervention Actually Participate?

We will assess adoption of the TEaM Intervention at the organizational level. We will determine the number and proportion of study sites (ED, UC, PC) that adopt the components of the TEaM Intervention (increased utilization of best practices in diagnosis, follow-up by patients and their parents or guardians, and number of Return to School/Primary Care Letters provided), and how many did not adopt the intervention (36). The numerator is characterize as the number of settings that adopted individual TEaM Intervention components. The denominator is defined as the total number of settings approached (53). We will also consider the representativeness of each study site (36) by assessing similarities or differences between current practices (baseline programs and initiatives) against TEaM Intervention components (use of validated symptom inventories, appropriate exam documentation, provision of CDC or institutional discharge instructions, return to school/primary care letters, appropriate use of diagnostic tools (e.g., computed tomography when indicated), identification and referral for children with persistent post-concussion symptoms). Our objective is to understand contextual factors that influence adoption, including the barriers and facilitators to adoption and non-adoption across study sites (38, 44). Data sources will include eMR, semi-structured interviews, site visits, and observations.

 Implementation (Provider Level): To What Degree Was the <italic>TEaM</italic> Intervention Consistently Deployed Across Study Sites?

Individual provider-level evaluation will determine the percentage of provider utilization of the TEaM Intervention components across study sites and the use of evidence-based practices for diagnosing and managing mTBI. The provider utilization rate of TEaM Intervention components will be determined by a numerator (# of individual TEaM Intervention components consistently implemented) and a denominator (Total # of TEaM Intervention components). We will also evaluate the extent of communication linkages between providers (ED, UC, and PC), patients, parents, and school personnel (25). To evaluate implementation components of the TEaM Intervention (provider training, eMR-based screening, notification system and decision-support, patient discharge instructions, post-mTBI management), we will determine the consistency of implementation across each study site through provider utilization rate of individual intervention components, the length of time, cost, and resource utilization associated with the implementation of the intervention (36). Strategies to improve implementation of the TEaM Intervention will include provider training, technical assistance, and ongoing feedback surrounding the utilization of the TEaM Intervention components (36). Our goal is to understand the variations (similarities and differences) of implementing the TEaM Intervention and the approach to adapting each component of the TEaM Intervention across delivery systems (38, 44). Implementation barriers and facilitators of the TEaM Intervention will be determined, including contextual factors and operational practices triggering the barriers to implementation and how they were addressed (38). Data sources will include eMR, informal interviews, focus groups among providers, site visits and observations.

 Maintenance (Provider Level): To What Degree and Time Point Did the <italic>TEaM</italic> Intervention Become Standard of Practice and Sustain Effectiveness?

Individual provider-level maintenance will be determined by utilization of the TEaM Intervention components and evidence-based practices in diagnosing and managing mTBI among HCPs 6 months following the end of the study period (36). To maintain the sustainability of the TEaM Intervention across sites and facilitate communication between HCPs and school personnel, engagement of stakeholders across delivery systems will be reinforced on mTBI best practices to promote patient recovery (25). Qualitative inquiry will complement data collection on individual level maintenance by understanding the barriers to sustainability of the TEaM Intervention across study sites (38). We will explore strategies to facilitate appropriate work-flow and practice changes and seek new sources of funding opportunities to help promote sustainability and scalability of the TEaM Intervention. Data sources will include site visits and interviews with site specific provider champions following the study period.

 Formative Evaluation

Formative evaluation of the TEaM Intervention will be guided by the RE-AIM QuEST Framework (38) as an expansion of RE-AIM (36) to inform qualitative data collection throughout each phase of the study period (Figure 1). To facilitate Adoption and Implementation, during study phase 1 (Building a Foundation), we will identify and assess stakeholder (providers, hospital administrators, IT system personnel) perceptions and feedback regarding the training and CDS tools, and anticipated implementation barriers and facilitators surrounding utilization of the TEaM Intervention components across delivery systems (ED, UC, PC). To facilitate Reach, during study phase 2 (Provider Recruitment), we will assess any barriers to recruitment and how these barriers were addressed (38). We will also assess provider knowledge and skills relating to mTBI diagnosis and management during recruitment and obtain provider feedback regarding the mTBI educational training modules. To facilitate Maintenance, at the completion of study Phase 3 (Subject Enrollment and Pre-Decision Support) and during study Phase 6 (Study Wrap Up) of the TEaM Intervention, we will evaluate provider experiences, perceptions, barriers, and facilitators regarding TEaM Intervention components across delivery systems. Reasons for adaptations to the TEaM Intervention components, and assessment of contextual factors underlying facilitators and barriers to implementation of the TEaM Intervention across delivery systems will be completed (38). Finally, we will evaluate provider, patient, family and school personnel experiences, perceptions, barriers, and facilitators regarding communication linkages across delivery systems. Qualitative data collection will include focus groups, formal meetings, semi-structured interviews, and surveys.

TEaM study timeline.

 Data Analyses Sample Size

Sample size calculations were conducted with the aim of achieving 80% power for the primary outcome. Computations were based on previous research using the Concussion Learning Assessment and School Survey, 3rd Ed. (CLASS-3) which reported on the mean, variability, and intercorrelations for metrics derived from the CLASS (47). The sample size was calculated for a 1:1 treatment allocation scheme with 80% power to detect an overall 20% difference in the mean of the CLASS-3 Academic Problems Scale at the two-sided α level of 0.05 (corresponding to a Cohen's ≈ 0.15). With these parameters, we will randomize 532 patients after assuming a loss-to-follow-up rate of 20%. Assuming minimal loss-to-follow-up, power is expected to be ~88%.

 Descriptive Statistics

Frequencies and percentages will be used to describe categorical variables. Means, standard deviations, medians, and interquartile ranges will be used to describe continuous variables. The reliability of variables obtained via chart review will be evaluated using intraclass correlations, Cohen's κ, or Fleiss' κ depending on the characteristics of the variable.

 Primary Outcome—CLASS-3 Academic Problems Scale

Analyses will be conducted under the intention-to-treat principle (ITT). Scores on the CLASS-3 Academic Problems scale (42 pt. scale) will be compared between intervention and control patients using a mixed-effects/multilevel linear regression. Experimental group and Timepoint (weeks 1, 2, and 4) will be included as independent variables. The mixed-effects model will be used in order to account for clustering within the data (e.g., clustering within provider, patient, etc.). When possible, the intercept and intervention slope will be included as random effects to evaluate the variability between providers and sites. Alternatives will be considered if model assumptions are not met or if the model fails to estimate properly.

 Secondary Outcomes—(PCSI Emotion Scale, PCEI Emotional Control Scale, CLASS-3 School Stresses Scale, PEDS QL4, Time to Recovery)

For continuous variables, the analyses will be the same as above: either a mixed-effects/multilevel linear regression. For other variables, the distribution family and link function will be determined by the characteristics of the data.

 Study Covariates

Demographics (age, sex, race), premorbid conditions (migraines and other headaches, depression, ADD/ADHD, other learning disorders, and other psychiatric illnesses), prior mTBI history and duration of symptoms, baseline academic performance, provider type (e.g., APP/Physician) and specialty, initial number of mTBI symptoms, and mechanism of injury.

 Subgroup Analyses

The following planned subgroup analyses will be conducted in order to assess for heterogeneity of treatment effects: (1) Patients seen by resident physicians will be compared with those not seen by a resident, (2) patients treated by providers in compliance with the protocol will be compared to those who were not, (3) patients who received school accommodations will be compared with those who did not, (4) the effectiveness of the interventions will be compared across provider type and specialty.

 Missing Data

In the event of substantial missing data, data will be imputed using a multiple imputation procedure. The type of procedure (e.g., fully conditional specification, hot-decking, etc.) and the number of imputed data sets will be determined by the amount of missing data and the pattern of missingness. Efforts will be made to minimize loss-to-follow up; however, in the event that a patient's primary outcome (CLASS-3 score at the 1, 2, and 4-week follow ups) is unobtainable, we will (1) assume the worst and assign an “unfavorable” outcome for that patient in order to conduct the ITT analysis and (2) assume the best and assign a “favorable” outcome for that patient as a sensitivity analysis for the ITT analysis. These analyses will describe the boundary conditions for the models described above.

 Multiple Testing

To the extent possible, we will track the number of analyses to report the false discovery rate, that is, the expected proportion of false positives. We will test the primary outcome variable at an α level of 0.05; for secondary outcomes, the false discovery rate will be formally controlled.

 Discussion Significance

The CDC has informative, evidence-based resources and existing instruments to support healthcare providers in the evaluation, management, and treatment of pediatric mTBI (https://www.cdc.gov/headsup/providers/index.html) (54). Although these resources are made available to all stakeholders and clinicians, the dissemination and implementation of intervention programs has not been rigorously evaluated in real-world settings, e.g., utilization and adoption across healthcare delivery systems (ED, UC, and PC). This study targets the public health concern and translational research gap in pediatric mTBI. We will use RE-AIM as a pragmatic model (44) to assist us in evaluating the implementation and effectiveness of a multi-component clinical intervention using clinician training, electronic clinical decision support, customized discharge instructions, and communication linkage tools to ultimately improve personalized diagnosis, management, and treatment of pediatric mTBI established from high-quality evidence.

 Public Health Impact

The TEaM Intervention's public health impact will be determined by healthcare provider reach and efficacy on the individual level and the implementation and adoption on the organizational level (35, 55, 56). We seek to carefully evaluate any translational and dissemination issues related to the application of the TEaM Intervention within real-world settings (43, 56), and believe this evaluation will provide further knowledge and inquiry for future studies, updates to evidence-based guideline recommendations and dissemination materials. Furthermore, evaluation of this intervention utilizing the RE-AIM framework (35, 55) is an appropriate method in determining the effectiveness and generalizability of the TEaM Intervention, providing the knowledge needed to inform our stakeholders (patients, parents, school personnel, providers and grant funders), and facilitate the translation of research into the clinical setting, thereby improving practice change, program sustainability and outcomes for children with mTBI.

 Contributions to the Field Statement

This study evaluates the impact of an evidence-based, multi-component pediatric mTBI intervention across multiple pediatric healthcare settings leveraging the eMR to facilitate clinical decision support and improve communication linkages since the release of the new CDC pediatric mTBI guidelines. Utilization of the RE-AIM framework (36) complemented with qualitative inquiry (RE-AIM QuEST) (38) to evaluate the impact of TEaM Intervention across multiple healthcare settings is a pragmatic approach in facilitating the translation of research into clinical practice.

The emphasis on HCP education, training, and communication linkage tools addresses the research priority in mTBI. Furthermore, this model evaluates the sustainability of the TEaM Intervention following the funding period through ongoing engagement and collaboration of stakeholders within the healthcare and school systems, community partners, and funding agencies. This knowledge may assist researchers, funders, and policymakers in identifying specific outcome measures, contextual factors and, clinician practices surrounding pediatric mTBI. The TEaM Program Logic Model (Figure 2) was inspired by and adapted from the work of Manca et al. (57) to assist in program planning. The logic model describes the inputs, outputs/activities defining each applicable dimension of the RE-AIM framework, projected short, intermediate, and, long-term outcomes of the TEaM Intervention. The TEaM Intervention has the potential to provide a nationally recognized evidence-based intervention developed and guided by the best available evidence, ultimately promoting usability, feasibility, and scalability across organizational settings. The program evaluation component of the TEaM Intervention may provide the knowledge needed to better understand the application of a multi-component, multi-setting intervention into the real-world and increase the uptake and dissemination of pediatric mTBI guidelines.

TEaM Logic Model inspired by and adapted from “Logic model for the BETTER 2 program” by Donna Patricia Manca et al. licensed under CC BY 4.0.

 Project Status

Study interventions requiring provider enrollment are in the development and refinement stages. Due to COVID-19, a delay within the study timeline has occurred as the investigative study staff as well as HCPs who will be randomized participating providers are Emergency Medicine providers or involved in programs that directly support that setting. This delay has impacted the expected timeline of provider recruitment. Similarly, patient presentations to the study sites initially decreased when distancing measures were first implemented in the United States (Figure 3). However, patient presentations have begun to return to pre-COVID-19 levels. Other project efforts such as publication and eMR/CDS refinement continue.

The number of patients screened per day at participating study sites. The red vertical line identifies March 11th, 2020 when the WHO declared the COVID-19 outbreak to be a pandemic. The black fit line was computed using smoothing splines. The λ parameter was set using generalized cross validation.

 Conclusion

Application of the RE-AIM framework (36) employing a mix-methods approach (RE-AIM QuEST) (38) is a model for evaluating the effectiveness of the TEaM Intervention to emphasize priorities of this public health concern surrounding pediatric mTBI. This evaluation method has the potential to provide the knowledge needed to critically appraise the impact of pediatric mTBI post- injury recovery interventions across multiple settings, enabling the uptake of best-available evidence within clinical practice.

 Ethics Statement

The study protocol was approved by the Institutional Review Board of Emory University (IRB Registration No. IRB00108674).

 Author Contributions

PT drafted the manuscript. TM wrote the data analysis. All authors contributed to the writing of the study protocol for grant funding, revised sections of the manuscript, and approved the final version of the manuscript.

 Funding

This study was funded by the Centers for Disease Control and Prevention (CDC), Grant Number 5U01CE002939-03.

 Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

 Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

We would like to acknowledge and thank the TEaM Key Partners Ann Glang, Andrew Reisner, Daniel Wu, Srikant B. Iyer, Brad Weselman, and Laura Jones for their contribution.

 References National Center for Injury Prevention Control. Report to Congress on Mild Traumatic Brain Injury in the United States: Steps to Prevent a Serious Public Health Problem. Atlanta, GA: Centers for Disease Control and Prevention (2003). Available online at: https://www.cdc.gov/traumaticbraininjury/pdf/mtbireport-a.pdf (accessed September 14, 2020). Cassidy JD Carroll L Peloso P Borg J Von Holst H Holm L . Incidence, risk factors and prevention of mild traumatic brain injury: results of the WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury. J Rehabil Med. (2004) 36:28–60. 10.1080/1650196041002373215083870 Zogg CK Haring RS Xu L Canner JK AlSulaim HA Hashmi ZG . The epidemiology of pediatric head injury treated outside of hospital emergency departments. Epidemiol Camb Mass. (2018) 29:269–79. 10.1097/EDE.000000000000079129240568 Centers for Disease Control Prevention. Surveillance Report of Traumatic Brain Injury-related Emergency Department Visits, Hospitalizations, and Deaths—United States, 2014. Centers for Disease Control and Prevention, U.S. Department of Health and Human Services (2019). Available online at: https://www.cdc.gov/traumaticbraininjury/pdf/TBI-Surveillance-Report-508.pdf (accessed September 14, 2020). Haarbauer-Krupa J Ciccia A Dodd J Ettel D Kurowski B Lumba-Brown A . Service delivery in the healthcare and educational systems for children following traumatic brain injury: gaps in care. J Head Trauma Rehabil. (2017) 32:367–77. 10.1097/HTR.000000000000028728060211 Arbogast KB Curry AE Pfeiffer MR Zonfrillo MR Haarbauer-Krupa J Breiding MJ . Point of health care entry for youth with concussion within a large pediatric care network. JAMA Pediatr. (2016) 170:e160294–e160294. 10.1001/jamapediatrics.2016.029427244368 Centers for Disease Control and Prevention. Report to Congress: The Management of Traumatic Brain Injury in Children. Atlanta, GA: National Center for Injury Prevention and Control; Division of Unintentional Injury Prevention. (2018). Available online at: https://www.cdc.gov/traumaticbraininjury/pdf/reportstocongress/managementoftbiinchildren/TBI-ReporttoCongress-508.pdf (accessed September 14, 2020). Yeates KO Kaizar E Rusin J Bangert B Dietrich A Nuss K . Reliable change in postconcussive symptoms and its functional consequences among children with mild traumatic brain injury. Arch Pediatr Adolesc Med. (2012) 166:615–22. 10.1001/archpediatrics.2011.108222393171 Babcock L Byczkowski T Wade SL Ho M Mookerjee S Bazarian JJ. Predicting postconcussion syndrome after mild traumatic brain injury in children and adolescents who present to the emergency department. JAMA Pediatr. (2013) 167:156–61. 10.1001/jamapediatrics.2013.43423247384 Barlow KM Crawford S Stevenson A Sandhu SS Belanger F Dewey D. Epidemiology of postconcussion syndrome in pediatric mild traumatic brain injury. Pediatrics. (2010) 126:e374–81. 10.1542/peds.2009-092520660554 Hessen E Anderson V Nestvold K. MMPI-2 profiles 23 years after paediatric mild traumatic brain injury. Brain Inj. (2008) 22:39–50. 10.1080/0269905070184617918183508 Moser RS Schatz P Jordan BD. Prolonged effects of concussion in high school athletes. Neurosurgery. (2005) 57:300–6. 10.1227/01.NEU.0000166663.98616.E416094159 Lowry R Haarbauer-Krupa JK Breiding MJ Thigpen S Rasberry CN. Lee SM. Concussion and academic impairment among US high school students. Am J Prev Med. (2019) 57:733–40. 10.1016/j.amepre.2019.08.01631753255 Ransom DM Vaughan CG Pratson L Sady MD McGill CA Gioia GA. Academic effects of concussion in children and adolescents. Pediatrics. (2015) 135:1043–50. 10.1542/peds.2014-343425963014 Ewing-Cobbs L Prasad MR Kramer L Cox CS Baumgartner J Fletcher S . Late intellectual and academic outcomes following traumatic brain injury sustained during early childhood. J Neurosurg Pediatr. (2006) 105:287–96. 10.3171/ped.2006.105.4.28717328279 Taylor HG Orchinik LJ Minich N Dietrich A Nuss K Wright M . Symptoms of persistent behavior problems in children with mild traumatic brain injury. J Head Trauma Rehabil. (2015) 30:302–10. 10.1097/HTR.000000000000010625629259 Novak Z Aglipay M Barrowman N Yeates KO Beauchamp MH Gravel J . Association of persistent postconcussion symptoms with pediatric quality of life. JAMA Pediatr. (2016) 170:e162900–e162900. 10.1001/jamapediatrics.2016.290027775762 Anderson V Brown S Newitt H Hoile H. Long-term outcome from childhood traumatic brain injury: intellectual ability, personality, and quality of life. Neuropsychology. (2011) 25:176–84. 10.1037/a002121721219074 Purcell L Harvey J Seabrook JA. Patterns of recovery following sport-related concussion in children and adolescents. Clin Pediatr. (2016) 55:452–8. 10.1177/000992281558991526063756 McCrea M Guskiewicz K Randolph C Barr WB Hammeke TA Marshall SW . Incidence, clinical course, and predictors of prolonged recovery time following sport-related concussion in high school and college athletes. J Int Neuropsychol Soc JINS. (2013) 19:22. 10.1017/S135561771200087223058235 Zemek R Barrowman N Freedman SB Gravel J Gagnon I McGahern C . Clinical risk score for persistent postconcussion symptoms among children with acute concussion in the ED. JAMA. (2016) 315:1014–25. 10.1001/jama.2016.120326954410 Barlow KM Crawford S Brooks BL Turley B Mikrogianakis A. The incidence of postconcussion syndrome remains stable following mild traumatic brain injury in children. Pediatr Neurol. (2015) 53:491–7. 10.1016/j.pediatrneurol.2015.04.01126421987 Lumba-Brown A Yeates KO Sarmiento K Breiding MJ Haegerich TM Gioia GA . Centers for disease control and prevention guideline on the diagnosis and management of mild traumatic brain injury among children. JAMA Pediatr. (2018) 172:e182853. 10.1001/jamapediatrics.2018.285331848109 Lumba-Brown A Yeates KO Sarmiento K Breiding MJ Haegerich TM Gioia GA . Diagnosis and management of mild traumatic brain injury in children: a systematic review. JAMA Pediatr. (2018) 172:e182847–e182847. 10.1001/jamapediatrics.2018.284730193325 Gioia GA Glang AE Hooper SR Brown BE. Building statewide infrastructure for the academic support of students with mild traumatic brain injury. J Head Trauma Rehabil. (2016) 31:397–406. 10.1097/HTR.000000000000020526709582 Glang AE Koester MC Chesnutt JC Gioia GA McAvoy K Marshall S . The effectiveness of a web-based resource in improving postconcussion management in high schools. J Adolesc Health. (2015) 56:91–7. 10.1016/j.jadohealth.2014.08.01125438964 Reisner A Popoli DM Burns TG Marshall DL Jain S Hall LB . The central role of community-practicing pediatricians in contemporary concussion care: a case study of Children's Healthcare of Atlanta's Concussion Program. Clin Pediatr. (2015) 54:1031–7. 10.1177/000992281557346825715824 Popoli DM Burns TG Meehan WP Reisner A CHOA concussion consensus: establishing a uniform policy for academic accommodations. Clin Pediatr. (2014) 53:217–24. 10.1177/000992281349907023960266 Lumba-Brown A Wright DW Sarmiento K Houry D. Emergency department implementation of the centers for disease control and prevention pediatric mild traumatic brain injury guideline recommendations. Ann Emerg Med. (2018) 72:581–5. 10.1016/j.annemergmed.2018.03.04529753521 Suskauer SJ Yeates KO Sarmiento K Benzel EC Breiding MJ Broomand C . Strengthening the evidence base: recommendations for future research identified through the development of CDC's Pediatric Mild TBI Guideline. J Head Trauma Rehabil. (2019) 34:215–23. 10.1097/HTR.000000000000045530608306 Arbogast KB Curry AE Metzger KB Kessler RS Bell JM Haarbauer-Krupa J . Improving primary care provider practices in youth concussion management. Clin Pediatr. (2017) 56:854–65. 10.1177/000992281770955528521519 Zonfrillo MR Master CL Grady MF Winston FK Callahan JM Arbogast KB. Pediatric providers' self-reported knowledge, practices, and attitudes about concussion. Pediatrics. (2012) 130:1120–5. 10.1542/peds.2012-143123147981 Donnell Z Hoffman R Myers G Sarmiento K. Seeking to improve care for young patients: development of tools to support the implementation of the CDC Pediatric mTBI Guideline. J Safety Res. (2018) 67:203–9. 10.1016/j.jsr.2018.09.00630553425 Purcell LK Davis GA Gioia GA. What factors must be considered in ‘return to school’ following concussion and what strategies or accommodations should be followed? A systematic review. Br J Sports Med. (2019) 53:250–250. 10.1136/bjsports-2017-09785329500251 Glasgow RE Vogt TM Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. (1999) 89:1322–7. 10.2105/AJPH.89.9.132210474547 RE-AIM – Reach Effectiveness Adoption Implementation Maintenance. (2021). Available online at: https://www.re-aim.org/ (accessed September 16, 2020). Glasgow RE Harden SM Gaglio B Rabin B Smith ML Porter GC . RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. (2019) 7:64. 10.3389/fpubh.2019.0006432719761 Forman J Heisler M Damschroder LJ Kaselitz E Kerr EA. Development and application of the RE-AIM QuEST mixed methods framework for program evaluation. Prev Med Rep. (2017) 6:322–8. 10.1016/j.pmedr.2017.04.00228451518 White SM McAuley E Estabrooks PA Courneya KS. Translating physical activity interventions for breast cancer survivors into practice: an evaluation of randomized controlled trials. Ann Behav Med. (2009) 37:10–9. 10.1007/s12160-009-9084-919255819 Glasgow RE Strycker LA King DK Toobert DJ Rahm AK Jex M . Robustness of a computer-assisted diabetes self-management intervention across patient characteristics, healthcare settings, and intervention staff. Am J Manag CARE. (2006) 12:137–45.16524346 Holland WC Nath B Li F Maciejewski K Paek H Dziura J . Interrupted time series of user-centered clinical decision support implementation for emergency department–initiated buprenorphine for opioid use disorder. Acad Emerg Med. (2020) 27:753–63. 10.1111/acem.1400232352206 Larsen AL Liao Y Alberts J Huh J Robertson T Dunton GF . analysis of a school-based nutrition education intervention in kindergarteners. J Sch Health. (2017) 87:36–46. 10.1111/josh.1246627917488 Glasgow RE McKay HG Piette JD Reynolds KD. The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns. (2001) 44:119–27. 10.1016/S0738-3991(00)00186-511479052 Glasgow RE Estabrooks PE. Pragmatic applications of RE-AIM for health care initiatives in community and clinical settings. Prev Chronic Dis. (2018) 15:170271. 10.5888/pcd15.17027129300695 Gioia GA Collins M Isquith PK. Improving identification and diagnosis of mild traumatic brain injury with evidence: psychometric support for the acute concussion evaluation. J Head Trauma Rehabil. (2008) 23:230–42. 10.1097/01.HTR.0000327255.38881.ca18650767 Zuckerbraun NS Atabaki S Collins MW Thomas D Gioia GA. Use of modified acute concussion evaluation tools in the emergency department. Pediatrics. (2014) 133:635–42. 10.1542/peds.2013-260024616361 Gioia GA Babikian T Barney BJ Chrisman SPD Cook LJ Didehbani N . Identifying school challenges following concussion: psychometric evidence for the Concussion Learning Assessment & School Survey, 3rd Ed. (CLASS-3). J Pediatr Neuropsychol. (2020) 6:203–17. 10.1007/s40817-020-00092-5 Gioia GA Vaughan CG Sady MD. PostConcussion Symptom Inventory- 2: Technical Manual. Lutz, FL: Psychological Assessment Resources, Inc. (2019). Gioia GA Isquith PK. PostConcussion Executive Inventory (PCEI). Lutz, FL: Psychological Assessment Resources, Inc. (2018). Desai AD Zhou C Stanford S Haaland W Varni JW. Mangione-Smith RM. Validity and responsiveness of the Pediatric Quality of Life Inventory (PedsQL) 40 Generic Core Scales in the pediatric inpatient setting. JAMA Pediatr. (2014) 168:1114–21. 10.1001/jamapediatrics.2014.160025347549 Varni JW Burwinkle TM Seid M Skarr D. The PedsQLTM* 4.0 as a pediatric population health measure: feasibility, reliability, and validity. Ambul Pediatr. (2003) 3:329–41. 10.1367/1539-4409(2003)003<0329:TPAAPP>2.0.CO;214616041 Harris PA Taylor R Thielke R Payne J Gonzalez N Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. (2009) 42:377–81. 10.1016/j.jbi.2008.08.01018929686 Gaglio B Shoup JA Glasgow RE. The RE-AIM framework: a systematic review of use over time. Am J Public Health. (2013) 103:e38–46. 10.2105/AJPH.2013.30129923597377 Centers for Disease Control Prevention. HEADS UP to Health Care Providers. (2020). Available online at: https://www.cdc.gov/headsup/providers/index.html (accessed September 16, 2020). Glasgow RE Nelson CC Strycker LA King DK. Using RE-AIM metrics to evaluate diabetes self-management support interventions. Am J Prev Med. (2006) 30:67–73. 10.1016/j.amepre.2005.08.03716414426 Estabrooks PA Gyurcsik NC. Evaluating the impact of behavioral interventions that target physical activity: issues of generalizability and public health. Psychol Sport Exerc. (2003) 4:41–55. 10.1016/S1469-0292(02)00016-X Manca DP Aubrey-Bassler K Kandola K Aguilar C Campbell-Scherer D Sopcak N . Implementing and evaluating a program to facilitate chronic disease prevention and screening in primary care: a mixed methods program evaluation. Implement Sci. (2014) 9:135. 10.1186/s13012-014-0135-725293785