The first step in C4 development was to identify published scales that had been used in mTBI. We identified two validated scales from the research literature, the NSI (7) and Everyday Memory Questionnaire (14); a third that was published in a textbook (26) and recommended by colleagues; and a fourth that was a component of a published treatment protocol (15). Each had items relevant for mTBI treatment planning, but no single scale focused on cognitive activities and was validated in mTBI, and there was considerable overlap among scales. Thus, the group began by identifying scale items that were relevant to clinical practice and eliminating duplicates, then mapped scale items to cognitive domains to identify areas of overlap and gaps. This version was sent to five mTBI speech-language pathology (SLP) or occupational therapy (OT) experts in the Department of Defense (DoD), Veterans Administration (VA), and academic settings across the U.S.

SLP and OT experts provided item-level feedback and recommendations for additional items and/or wording changes (e.g., phrasing all items as “I” statements). Feedback was aggregated and a revised version was sent back to experts for review in two subsequent rounds. After the final round, all five experts agreed for the inclusion of the selected items. In addition to item-level feedback, experts provided suggestions regarding questionnaire instructions and follow-up interview questions to facilitate using the questionnaire specifically as a treatment planning tool. We incorporated this feedback, for example, specifying in the instructions that identified challenges should be “new since [examinees’] injury and affect [their] everyday functioning” because experts emphasized the clinical utility of identifying changes from baseline. In addition, we included interview questions to be asked post-questionnaire, which experts identified would facilitate the treatment planning process. These included clarifying questions for items rated as frequently bothersome (e.g., “Can you give me an example of this happening in your everyday life?”), as well as a question asking examinees to identify their top three greatest challenges, as a means of identifying treatment targets and addressing possible negative bias (i.e., identifying their greatest challenges even if participants identify several items as frequently bothersome). The initial list of 49 items with sources is shown in the Supplementary Appendix with initial expert feedback (i.e., initial qualitative ratings of “possibly include” and “possibly exclude”). The final 23-item version with instructions and follow-up interview questions is shown in Appendix A. The final version of the C4 included 22 items evaluated everyday complaints commonly reported in the chronic mTBI population, and one item was a bias check (item 16). The bias-check item was selected to be a non-declarative memory task that would be unaffected by mTBI.

A separate group of five mTBI experts, including physicians and neuropsychologists, sorted the items into cognitive domains to assess content validity. These experts were not given domain categories in advance; instead, they each generated their own domain categories. Items were assessed for consensus across experts to determine if cognitive domains were adequately covered.

The C4 was administered to service members with a history of mTBI (n = 21), veterans with a history of mTBI (n = 32), and veterans with orthopedic injuries but no history of mTBI (n = 25). Table 1 presents demographic and clinical characteristics for the full sample (N = 78). Participants were recruited from a military treatment facility and a VA Polytrauma/TBI System of Care program.

Participants with mTBI were recruited as part of STAR-C. Mild TBI was determined based on clinical judgment of a medical provider embedded within a clinic specialized for TBI care and defined using the VA/DoD criteria (16): 30 min or less of loss of consciousness, less than 24 h of alteration of consciousness, or 1 day or less of posttraumatic amnesia; and no abnormalities on neuroimaging, if available. The mTBI must have occurred at least 6 months prior to the participant starting the study. Clinical determination was based on both self-report and medical record review. Participants must have endorsed at least one cognitive complaint as “moderate” or more severe on the NSI (7) and demonstrated at least a 6th grade reading level on the Wide Range Achievement Test, 4th Edition, Word Reading subtest (WRAT-4 Reading; (17)) to qualify for STAR-C. Exclusion criteria included a history of moderate, severe, or penetrating TBI; a lifetime diagnosis of schizophrenia or other psychotic disorder; a history of neurological disease (e.g., multiple sclerosis, cerebrovascular accident); active suicidal ideation; current participation in intensive behavioral health treatment, defined as more than 5 appointments/encounters per week; inability to attend treatment three times per week for 3 weeks; use of daily narcotic medication; or failure on measures of performance validity, per the Test of Memory Malingering (18) manual cutoff score or symptom validity, per the NSI Validity-10 cutoff score (19).

Participants in the orthopedic injury group were recruited from the Polytrauma Musculoskeletal and Orthopedic Injury clinic at the same VA hospital. Reasons for orthopedic treatment included traumatic orthopedic injuries, wear-and-tear injuries, and chronic pain. Individuals who indicated a history of mTBI during prescreening were not invited into the study. Medical records of enrolled participants were reviewed to confirm a lack of documented mTBI. Time since injury was calculated from the date of the presenting orthopedic/musculoskeletal injury to date of enrollment.

Table 1 presents demographic and clinical characteristics of our sample, separated by groups of service members with a history of mTBI, veterans with mTBI, and veterans with orthopedic injuries. We statistically compared the groups using contingency tables, t-tests, or ANOVAs depending on the variables’ level of measurement. These tests were conducted to ensure the samples were representative of what is typical for their respective population (e.g., to ensure the orthopedic group presented with fewer mental health conditions than the mTBI groups). The three samples differed in age, with service members being the youngest and the orthopedic sample the oldest. Participants with orthopedic injuries had fewer years of military service and fewer Operation Enduring Freedom/Operation Iraqi Freedom combat deployments than those with mTBI. Fewer veterans with a history of mTBI also had a history of prior mental health conditions than the other two samples. Veterans with a history of mTBI had the highest current posttraumatic stress disorder (PTSD) rate overall, which was higher than the current PTSD rate in the orthopedic sample. Participants with orthopedic injuries also had fewer other anxiety disorders than the two samples with a history of mTBI, and the fewest mood disorder diagnoses, with the most being in the veteran sample with a history of mTBI.

The C4 questionnaire was presented to participants in the format shown in Appendix A, which includes instructions to participants. A 5-point frequency of occurrence scale (“Not at all” to “All the time”) was used to rate an item’s persistence over a 2-week period. For items rated as “Often” or “All the time,” clinicians asked the additional questions, “In what setting is this most disruptive (e.g., home, school, community, work)? Can you give me an example of this happening in your everyday life? What do you do when it happens? What physical or psychological factors influence this happening and how you deal with it?” These follow-up questions were intended to assist clinicians and participants in identifying the contexts in which difficulties tend to occur, concrete examples of cognitive difficulties in daily life, existing coping strategies, as well as perceived causes of these difficulties. Participants were then asked to identify the top three items from the questionnaire that represented the most bothersome functional limitations. For the veteran and active-duty mTBI groups, these three items were the treatment targets which served as the starting point for STAR-C therapy.

In addition to the C4, participants completed the Minnesota Multiphasic Personality Inventory – 3 (20). The MMPI-3 is a 335-item, true-false, self-report measure of psychopathology. It yields ten validity scales/indicators, three higher-order scales, eight substantive restructured clinical scales, 26 specific problems scales, and five personality psychopathology scales. The specific problems scales of Cognitive Complaints and Inefficacy (e.g., decision-making difficulties) were chosen to evaluate the C4’s convergent validity, given that these scales also capture subjective cognitive complaints.

After providing written informed consent to participate, all participants completed a demographic data questionnaire, the C4, and other questionnaires relevant to the larger STAR-C feasibility trial. Participants with a history of mTBI also completed other procedures as part of STAR-C (further details are provided in Turkstra et al. (25); only relevant details are summarized here). If a participant was not able to identify at least three C4 items that occurred “Often” or “All the time,” they were to be excluded from the study; however, all eligible participants could identify three problem items on the C4 at intake, so none were excluded. Participants who advanced to treatment collaborated with clinicians to reframe C4 problem statements as treatment targets (e.g., “I get overwhelmed by things I have to do” might be reframed as “Each morning, I will choose three tasks to complete that day”). Participants worked to attain those targets in 6–10 individual strategy-focused cognitive rehabilitation sessions within 4 weeks. Treatment was discontinued when the participant had attained three treatment targets or completed 10 sessions, whichever came first. Target attainment was measured using goal attainment scaling (21), and was measured immediately after therapy and at 1 and 3 months post-therapy. In addition to patient data, we informally solicited feedback from the clinicians who administered the C4 regarding the C4’s ease of use and clinical utility. Moreover, the C4 was re-administered to participants with orthopedic injuries approximately 4 weeks after the first administration, and we used these data to evaluate test–retest stability of the instrument because scores were not expected to change over time for this group given that they received no treatment. No participants within the orthopedic injury group were lost to follow-up. All procedures involving human subjects were approved by the Institutional Review Board for human participants’ protection at both enrolling sites.

Given that the C4 was developed as a treatment planning tool and not an assessment tool, our planned analyses focused on demonstrating the clinical use of the C4 and not on demonstrating a breadth of psychometric properties. For treatment planning, the essential psychometric properties are face validity (e.g., items capture functions that patients and clinicians judge as relevant to therapy) and content validity (e.g., items are based on a conceptual framework and capture appropriate content domains). The face and content validity of the C4 are largely reflected in our scale development procedures, such that the items for the C4 were developed and vetted by outside clinicians not involved in the STAR-C study. We also calculated percentages of item endorsement frequences among our three groups (service members with TBI, veterans with TBI, veterans with orthopedic injury) to determine if C4 items covered content that was relevant and specific to mTBI functional outcomes as opposed to generalized functional outcomes that would be expected in any group with a history of injury (e.g., orthopedic injury). We reported relative differences in endorsements as opposed to statistical differences due to Type I error inflation and power concerns within the context of the STAR-C feasibility trial’s small sample. To demonstrate the clinical utility of the C4, we included information regarding STAR-C outcomes; specifically, we reported the percentages of participants who successfully attained treatment goals at post-treatment and follow-up data collection time points. Additionally, though a formal qualitative analysis was not conducted, we included excerpts of the qualitative information provided by STAR-C clinicians regarding their perceptions of the C4 as a treatment planning tool.

Importantly, though it was not the focus of the current study to demonstrate the psychometric properties of the C4 as if it were an assessment tool, we conducted basic psychometric analyses to provide preliminary evidence of its psychometric soundness. Specifically, the factor structure of the C4 was evaluated using exploratory factor analysis (EFA) in the mTBI samples only (n = 53). The bias-check item was excluded from EFA procedures because this item was intentionally designed to represent a cognitive task that would be unaffected by mTBI (i.e., a non-declarative memory task) and therefore would not be expected to load with other items. Item scores were submitted to principal-axis factor analysis with squared multiple correlations as the initial communality estimates. Items were judged to load on a factor if the highest factor loading was ≥0.400 (22), and cross-loadings were resolved by assigning an item to the factor with the stronger loading. In addition, we assessed reliability using internal consistency and stability measures. Internal consistency was evaluated within factors by calculating Cronbach’s alpha. Stability was evaluated by calculating test–retest reliability correlation coefficients on the data obtained from the orthopedic injury sample (n = 25) at two timepoints: baseline and 4 weeks later. Validity was examined by using data from the mTBI sample (n = 53) by correlating factor scores with selected MMPI-3 problems scales. It was predicted that convergent validity would be demonstrated by significant positive correlations with Cognitive Complaints and Inefficacy.

Content domains are reported in Table 2. Out of the five experts tasked with sorting the 23 C4 items into cognitive domains, at least 80% agreed that 4 items captured the attention domain, 3 items captured the memory domain, and 3 items captured the processing speed domain. At the level of 60% or more agreement, 4 items were agreed to capture the attention domain, 7 items captured the memory domain, 5 items captured the processing speed domain, and 1 item captured the executive functioning domain. Therefore, in total, five experts agreed on the content sorting for 17 out of 23 items at the level of 60% or more agreement, and they agreed upon the content categories for 10 items at the level of 80% or more agreement. Notably, these four content categories were generated by the experts themselves.

See Appendix A for a list of all items as they were presented to participants. Specifically, Figure 1 depicts the percentage of each item endorsed as “Often” or “All the time” by each group, while all item endorsement frequencies are presented in more detail in Supplementary Table 1. All items were endorsed as being problematic “Often” or “All the time” relatively more frequently by the mTBI samples than by those with orthopedic injuries. Seven problems were endorsed by more than 50% of both mTBI samples as occurring “Often” or “All the time.” In order from most to least frequent, these were: (1) “I have trouble remembering to do things I said I would, like passing on a message or making an appointment.” (2) “I have trouble remembering what I’ve done yesterday, or conversations I’ve had the previous day.” (3) “I cannot do things as quickly as I used to, or I make mistakes.” (4) “I have trouble finding a word that is on the tip of my tongue.” (5) “I get very fatigued during or after activities where I have to pay attention or sustain mental effort.” (6) “I have trouble focusing on a task in a distracting environment, like background noise or other people talking.” (7) “I have trouble doing more than one thing at a time.” Notably, our bias check (item 16) was endorsed as being problematic “Often” or “All the time” by only one participant out of our entire sample of 78 participants, reflecting its utility as an infrequently endorsed item.

STAR-C outcomes are described in detail in Turkstra et al. (25). In brief, of the 53 mTBI participants who completed STAR-C, 51 (96.23%) successfully attained at least one of their three identified targets at the end of treatment, with 81% successfully attaining all three. Of the 51 who successfully attained targets, ratings were available for 40 participants at 4 weeks and 12 weeks post-treatment. Of those 40, 80% continued to successfully attain at least one of the three C4-identified target areas at both follow-ups.

STAR-C clinicians reported that the C4 helped participants talk about their cognitive complaints and how those complaints translated into specific functional challenges. Clinicians noted that it was helpful to have specific daily living task examples for participants to reflect on as they described the impact of their cognitive challenges in everyday life. As one clinician noted, patients often reported concerns with ‘memory’ and ‘attention’ but were unable to expand on those concerns. The C4 questions provided practical terminology to help participants express how their cognitive difficulties affected everyday functioning, which then transferred to identifying meaningful targets. As one clinician said, the C4 allowed a “seamless transition” from assessment to treatment. Clinicians noted that patients seemed to “resonate with” the questions, and said that the questions put their thoughts into words (e.g., a participant reportedly said that “it’s like you were reading my mind”), which was an “immediate in” for the therapeutic relationship.

According to EFA, eigenvalues for the first three factors in the unrotated solution were: 7.187, 2.246, and 1.692; and accounted for 30.34, 7.62, and 5.29 percent of the variance, respectively, for a total variance accounted for of approximately 43 percent. The final varimax-rotated solution is presented in Table 3, and descriptive statistics for each factor are presented in Table 4. We chose to label factors using terms that describe functional impact to align with our goal for the C4 to identify activity-based treatment targets. We labeled our factors Keeping Attention Focused (9 items; e.g., “I have trouble doing more than one thing at a time”), Seeing Things Through (7 items; e.g., “Once I get started, I have trouble finishing things”), and Being Present (4 items; e.g., “I have trouble remembering what I just said, repeating myself in conversations”). To confirm the appropriateness of the three-factor solution, we correlated unit-weighted scores for the three factors (see Table 4). Correlations ranged from 0.327 to 0.656 (ps < 0.05), suggesting that these three factors are related yet separate constructs (i.e., positively correlated but not very strongly correlated), thereby providing additional support for the three-factor solution.

The Keeping Attention Focused and Seeing Things Through factors showed homogenous construct measurement, r_αs = 0.859 and 0.851, respectively. The Being Present factor had an internal reliability coefficient of 0.636, reflecting the small number of items comprising this scale. Moreover, though alpha values >0.70 are preferred, values >0.60 may be considered acceptable for the construction of new measures (23). All three factor scales showed high stability across a 4-week period among the orthopedic group (ps < 0.001): Keeping Attention Focused r = 0.785, Seeing Things Through r = 0.783, and Being Present r = 0.871.

As presented in Table 4, the C4 factors overall showed positive correlations with Cognitive Complaints and Inefficacy scales, consistent with hypotheses related to convergent validity. Specifically Keeping Attention Focused and Seeing Things Through were moderately positively correlated with Cognitive Complaints and Inefficacy (rs = 0.419–0.592, ps ≤ 0.002). Results were not as consistent with our hypotheses for Being Present, such that relationships with Cognitive Complaints and Inefficacy were positive, yet weaker and nonsignificant (rs = 0.175–0.184, ps ≥ 0.187). Notably, the third factor retained, Being Present, was the weakest in terms of eigenvalue, number of items, internal reliability, and convergent validity. Yet, several observations reinforced our decision to retain it. This factor was weakly positively correlated with Cognitive Complaints and Inefficiency, a pattern providing some support for the validity of this factor as an area of cognitive complaint in individuals with a history of mTBI. Moreover, despite having only four items that loaded onto this factor, one of its items was among the most frequently endorsed within our sample, “I cannot do things as quickly as I used to, or I make mistakes.” This item was endorsed as a problem “Often” or “All the time” by over half of both the military and veteran mTBI samples (see Figure 1). Therefore, this level of endorsement supported Being Present as an area of concern that warrants clinical attention in cognitive rehabilitation for individuals with a history of mTBI. Thusly, we retained Being Present as a factor.

Relatedly, two items, “I have trouble finding a word that is on the tip of my tongue” and “I have trouble remembering where my everyday items are, like my phone or keys,” did not load on any factors. We retained these items in the C4 given the C4’s original purpose as a treatment planning tool. These items are still useful for treatment planning purposes, such that 61.9 and 47.6% of active duty and 53.1 and 31.3% of veteran military service members with a history of mTBI endorsed, respectively, difficulties with word-finding and misplacing items at the levels of “Often” or “All the time” (See Figure 1).

Despite support regarding the C4’s utility as a treatment planning tool, the current study is not without limitations. Several of the current study’s limitations were due to data being collected as part of a mTBI treatment demonstration project (STAR-C; 25) where the C4 was the clinical instrument used for identifying intervention targets among military service members beginning cognitive rehabilitation. Ultimately, the C4 was not originally intended to be assessed as a potential outcome measure. Therefore, one major limitation of this study was the small sample sizes as well as our restriction to military service member samples, whose responses might not generalize to the general population. Future research may wish to use a larger and more diverse sample to test if sociodemographic factors influence C4 responses. A larger sample would also allow for examining more complex relationships between variables, such as with the use of methods like structural equation modeling.

Moreover, though we present preliminary psychometric analyses, including a tentative factor structure for this instrument, the results should be replicated with a much larger sample and using additional analytic methods, such as confirmatory factor analysis. Furthermore, though we present preliminary qualitative feedback from clinicians, the current results could be replicated with a larger sample of clinicians and/or with a structured interview to elicit more qualitative data from clinicians to facilitate the use of formal content analyses to increase the objectivity of qualitative data analysis. Future research may also wish to include an additional uninjured control group to test if the difficulties captured by the C4 reflect general difficulties unrelated to injury. Lastly, for the purposes of STAR-C, potential participants who demonstrated performance validity or symptom validity failure were intentionally excluded and therefore did not complete the C4. Future research could compare responses to the C4’s bias-check item with established symptom validity measures to establish a cutoff score that corresponds with symptom validity pass versus failure. This, as well as other future research addressing the aforementioned limitations, would bolster the C4’s utility as a brief, activity-focused outcome measure tailored to capturing common cognitive complaints among adults with mTBI.

As a means of bolstering the C4’s utility as a treatment planning tool, future research may consider adaptations. For example, two separate ratings, one for pre-injury and one for post-injury, could help patients to reflect on their baseline cognitive abilities in a manner that is not idealized, and thereby suggest that the end goal of cognitive rehabilitation is not to function perfectly. Moreover, C4 items could be further refined using strategies such as the inclusion of additional response options (e.g., “Not sure” or “Not applicable”) which would allow for the identification and subsequent rephrasing or removal of confusing or unclear items. Additional items could be developed with the inclusion of a patient-generated item (e.g., “Is there something you are struggling with that is not included in this survey?”) that would allow for the identification of additional often-reported functional difficulties. Lastly, while evidence suggests that the C4 is a useful clinical tool for the purposes of treatment plan development to learn compensatory strategies through cognitive rehabilitation, it is also important to note that a holistic approach to mTBI treatment might be necessary to address comorbid conditions (e.g., chronic pain, sleep disturbances, mental health difficulties) that can contribute to cognitive inefficiencies in daily life as opposed to solely focusing on compensatory strategies. Notably, participant’s responses to the C4’s follow-up question, “What physical or psychological factors influence this [cognitive difficulty] happening and how you deal with it?” could provide insight into what additional interventions might be needed to address a patient’s symptoms holistically (e.g., psychotherapy if cognitive difficulties tend to increase when under emotional stress). A holistic approach may be especially indicated if some patients are found to not respond as well to cognitive rehabilitation protocols, like STAR-C, as compared to other patients.