Researchers are increasingly interested in the emergence of implicit social cognition in childhood. As such, the race-attitude IAT has been modified for use with children (see Olson and Dunham, 2010; McKeague et al., 2015, for reviews) and has been used to assess children’s implicit racial biases in North America (e.g., Baron and Banaji, 2006; Newheiser and Olson, 2012) and beyond (e.g., Rutland et al., 2005; Dunham et al., 2006; Newheiser et al., 2014). Younger White children typically show racial preferences favoring their own racial ingroup on both implicit and explicit measures. However, by at least 8 years of age, implicit and explicit racial attitudes have been found to diverge. Despite a marked decrease in explicit racial bias that typically occurs by the age of 8-years (see Raabe and Beelmann, 2011, for a review), implicit racial bias as measured by the IAT remains stable across childhood and into adulthood (Rutland et al., 2005; Baron and Banaji, 2006). With consistency, children demonstrate relative implicit preference for ingroups and dominant outgroups relative to less dominant outgroups (see Olson and Dunham, 2010, for a review), often at a magnitude that is comparable to the implicit racial preferences of adults.

In addition, as with adults, the few studies that have examined children’s behavior suggest that responses on Child-IATs may be meaningfully related to intergroup behavior. White 8- to 11-year olds who displayed more positive racial outgroup attitudes on the IAT tended to report more cross-race friendships (Turner et al., 2007) and primarily White 4- to 6-year-olds who demonstrated stronger implicit bias favoring their minimal ingroup relative to a minimal outgroup allocated more resources (coins) to minimal ingroup members (Dunham et al., 2011). Although more research is needed, these findings suggest that racial biases as measured by the Child-IAT may meaningfully predict children’s behavior in their interactions with racially diverse peers.

Although results with adults, and to some extent children, support the predictive validity of the IAT, little is known about the psychometric properties of the Child-IAT when administered to children or adults. It is important to examine the reliability of the race-attitude Child-IAT if we wish to further predict children’s intergroup behavior and/or assess the effectiveness of prejudice-reduction interventions. For example, if an intervention alters implicit racial attitudes, then we have a greater chance of identifying this as an effective approach when using psychometrically sound, reliable measures as assessment tools. Although the adult version of the IAT has strong psychometric properties, “it cannot be assumed that there would be similar rates of reliability and validity in these measures with children” (Olson and Dunham, 2010, p. 251). Given the increasing popularity of Child-IATs, as an initial investigation into its psychometric properties, we first review the existing information on the reliability of Child-IATs. Second, through two studies we examine the reliability of race-attitude Child-IATs when completed by both children and adults, with a specific focus on (1) internal consistency and (2) test–retest reliability.

Reliability provides an indication of the dependability of a measure by identifying the proportion of total observed variance that reflects consistent (or true) as compared to random error variance (Cronbach, 1951). Internal consistency, which refers to the degree to which responses on a measure are consistent or related to one another, is one method used to assess reliability. Measures with low internal consistency, or a high degree of measurement error, may not provide useful information as scores will be unpredictable and not extend to meaningful interpretations of test performance (Cronbach, 1951). Further, measures with low internal consistency will have attenuated effects, such as limited correlations with other measures (Nunnally, 1978; Cunningham et al., 2001) and reduced effect sizes (Baugh, 2002), which can limit researchers’ ability to appropriately interpret their results. By using psychometrically sound measurement tools, researchers will be in a better position to comment on the potential disassociation between implicit and explicit attitudes, and when this might occur in development (e.g., Rutland et al., 2005).

Researchers reporting the internal consistency of IATs typically present split-half correlations (Spearman–Brown corrected) between D-scores calculated separately for “practice” and “test” trials from the critical blocks presenting the combined attribute and target discrimination task (Greenwald et al., 2003). However, this approach may be limited as different methods of splitting the data (e.g., practice vs. test trials, even vs. odd trials, etc.) can produce different internal consistency coefficients (Cronbach, 1951; Nunnally, 1978). By contrast, coefficient alpha is, by mathematical definition, the average of all the possible split-half coefficients (Cronbach, 1951). In line with recommendations that coefficient alpha be used to assess the internal consistency of new measures (Nunnally, 1978), researchers who have specifically investigated the psychometric properties of the IAT with adults have often used coefficient alpha to estimate internal consistency (e.g., Bosson et al., 2000; Cunningham et al., 2001; Gawronski, 2002; Gschwendner et al., 2008). Because including more items in the analyses can result in higher coefficient alpha, these researchers have used trial-based difference scores or scores based on test quarters as items in their analyses. For these reasons, we also use coefficient alpha based on test quarters to estimate the internal consistency of children’s responses.

Research examining the internal consistency of race-attitude IATs in adult samples have reported coefficient alphas that range from 0.55 to 0.88 (Cunningham et al., 2001; Gawronski, 2002; Nosek and Smyth, 2007; Bar-Anan and Nosek, 2014) and split-half reliabilities from 0.43 to 0.67 (Greenwald et al., 2003; Nosek et al., 2005). Although not as high as what is typically found for self-report measures, the internal consistency of the IAT falls within the acceptable range for reaction time measures which, by their very nature, show greater fluctuation across trials (Cunningham et al., 2001; Nosek et al., 2011). The reason for increased response variability on implicit measures could stem from a variety of sources, including having a larger response range (i.e., ranging 0 to 10,000 ms on an implicit measure as compared to 1 to 7 on an explicit measure), momentary inattention to the task (i.e., blinking, sneezing), and/or participants adopting different response strategies within the task (Buchner and Wippich, 2000; De Houwer, 2003; Fiedler and Bluemke, 2005; Lane et al., 2007). Such sources of error variance do not similarly influence responses on explicit measures. Although the IAT is not without its limitations (a point returned to in the General Discussion), it is worth noting that this measure is generally found to have stronger internal consistency as compared to other measures of implicit social cognition (e.g., Bosson et al., 2000; Cunningham et al., 2001; Degner and Wentura, 2010; Bar-Anan and Nosek, 2014), a fact that has contributed to its continued use and popularity.

To date, only a small proportion of published studies report any internal consistency information when race-attitude Child-IATs are completed by children. In two of the papers that reported internal consistency, coefficient alphas ranged from 0.73 to 0.81 (Turner et al., 2007; Zezelj et al., 2015). In other studies, split-half correlations ranged from 0.55 to 0.73 (Degner and Wentura, 2010; Haye et al., 2010; Dunham et al., 2014; see Supplementary Materials for a comprehensive review). These preliminary findings suggest that when administered to children, the internal consistency of race-attitude Child-IATs is comparable to what is found when adults complete a traditional version of this task. However, given the increasing use of this measure with children, a more systematic investigation into the internal consistency and stability of race-attitude Child-IATs is needed (Olson and Dunham, 2010; McKeague et al., 2015).

The goal of the present research was to systematically examine the internal consistency and test–retest reliability of race-attitude Child-IATs completed by 5- to 11-year-olds and by adults. To replicate the conditions under which these tasks are frequently administered, we examined the reliability of the race-attitude Child-IAT when (a) the measure was reduced in length (Study 1) and (b) two Child-IATs were completed in succession (Study 2).

A second main goal of this research was to examine the test–retest reliability of race-attitude Child-IATs. To the extent that the construct of interest is stable, a measure should yield similar scores across time and these scores should be reliably correlated (e.g., test–retest correlations; Nunnally, 1978; Bosson et al., 2000). Although reported less frequently, researchers examining the test–retest reliability of race-attitude IATs completed by adults have found Pearson rs ranging from 0.17 to 0.50 (Cunningham et al., 2001; see also Lane et al., 2007; Gschwendner et al., 2008; Bar-Anan and Nosek, 2014). On other versions of the Child-IAT administered to children, test stability has been comparable to that of the traditional race-attitude IAT completed by adults; test–retest rs ranged from 0.20 to 0.70 for smoking attitudes (Andrews et al., 2010) and from 0.14 to 0.39 for aggressiveness (Gollwitzer et al., 2007; Lemmer et al., 2015). In the present research we provide the first examination of the test–retest reliability of the race-attitude Child-IAT.

The goal of this paper was to examine the psychometric properties of modified race-attitude Child-IATs. To examine whether the length of the measure would impact internal consistency, in Study 1 we administered a reduced-length White–Black race-attitude Child-IAT (Rutland et al., 2005) to children. In Study 2, we examined the internal consistency of a traditional-length (Greenwald et al., 2003) White–Black Child-IAT completed by both children and adults. In addition, in Study 2 we examined (a) whether the internal consistency of a second White–Black Child-IAT completed in the same testing session was comparable to the first, and (b) the test–retest reliability of this measure. In Study 2, we included an adult sample to determine whether the psychometric properties of children’s responses would be comparable to those made by adults. This research is an initial step in addressing the void in the literature regarding the psychometric properties of this increasingly popular race-attitude Child-IAT.

Using a sample that was more racially diverse than what is typically seen in research on children’s implicit racial attitudes in North America, coefficient alpha for the reduced-length race-attitude Child-IAT was 0.72. This is comparable to the internal consistency coefficients for adults completing traditional-length versions of the IAT (αs ranged 0.55 to 0.88; Cunningham et al., 2001; Gawronski, 2002; Nosek and Smyth, 2007; Bar-Anan and Nosek, 2014) and children completing reduced versions of similar measures (split-half rs ranged 0.55 to 0.73; Degner and Wentura, 2010; Dunham et al., 2014), suggesting that a reduced-length race-attitude Child-IAT demonstrates adequate internal consistency when completed by children. Replicating previous research, one-sample t-test comparing the overall D score to 0 confirmed that children demonstrated implicit racial bias favoring Whites relative to Blacks (D = 0.08, SD = 0.39), t(191) = 3.15, p = 0.002, d = 0.23. Taken together, these results suggest that a reduced-length race-attitude Child-IAT shows comparable internal consistency as when the traditional-length IAT is administered to adults.

A first goal of Study 2 was to examine whether the internal consistency of a traditional-length Child-IAT would be comparable to that of the reduced-length Child-IAT administered in Study 1. Coefficient alpha for the first completed Child-IAT was 0.76 for younger and 0.85 for older children (Table 1). Although comparable, these slightly exceeded the internal consistency of the reduced-length Child-IAT (0.72; Study 1). Further, the internal consistency of children’s responding was comparable to, but slightly higher than, adults completing the same measure, α = 0.75. We suspect that this may be due to differences in administration procedures; children completed the Child-IAT in the presence of an experimenter who re-directed them as necessary, whereas the adults completed the measure independently. Despite these differences, both children and adults demonstrated internal consistencies that were comparable to published studies with adults (αs ranged from 0.55 to 0.88; Cunningham et al., 2001; Gawronski, 2002; Nosek and Smyth, 2007; Bar-Anan and Nosek, 2014). These findings suggest that reducing the length of the Child-IAT does not improve the internal consistency of this child-friendly measure. Instead, internal consistency may be slightly improved by a longer version of the measure (Cronbach, 1951; Nunnally, 1978).

The second goal of Study 2 was to examine the impact of repeated administrations within a single testing session on the internal consistency of the Child-IAT. For the second Child-IAT, α = 0.67 for younger children, α = 0.77 for older children (Study 2a), and α = 0.74 for adults (Study 2b; Table 1). For children, coefficient alpha was lower for the second completed Child-IAT. For adults, coefficient alpha was comparable across both administrations. Considering that two traditional-length Child-IATs were completed in succession, children may have fatigued during the second measure, increasing the error variance and thus decreasing internal consistency. Nevertheless, for each age group, the internal consistency of the first and second Child-IAT was still comparable to published studies with adults (0.55 to 0.88; Cunningham et al., 2001; Gawronski, 2002; Nosek and Smyth, 2007; Bar-Anan and Nosek, 2014).

Finally, we examined the test–retest reliability of the two race-attitude Child-IATs completed within a single testing session. One-sample t-tests comparing D-scores to 0 confirmed that, similar to other research using race-attitude Child-IATs with children, on both Child-IATs, younger children, older children, and adults demonstrated implicit racial bias favoring Whites relative to Blacks, ts ≥ 2.53, ps ≤ 0.01, ds ≥ 0.18, see Figure 1. Importantly, paired-samples t-tests revealed that the magnitude of bias did not differ between the first and second completed Child-IAT for younger children, t(63) = 1.31, p = 0.20, d = 0.16, older children, t(70) = 1.47, p = 0.15, d = 0.17, or adults, t(189) = 1.62, p = 0.11, d = 0.12. This provides some initial evidence of reliability, as the measure yielded similar scores at both times.

In addition, for both children, r = 0.24, n = 135, p = 0.005 [ICC(2,1) = 0.39], and adults, r = 0.33, n = 198, p < 0.001 [ICC(2,1) = 0.50], significant test–retest correlations and moderate Intraclass Correlation Coefficients emerged, although for children this relationship was slightly lower than for adults. These correlations fall within the lower range of test–retest reliabilities typically seen for adults completing race-attitude IATs (rs from 0.17 to 0.50; Cunningham et al., 2001; see Lane et al., 2007, for a review), and are comparable to published studies examining children’s smoking attitudes (0.20 to 0.70; Andrews et al., 2010) and aggressiveness (0.14 to 0.39; Gollwitzer et al., 2007; Lemmer et al., 2015).

Researchers often decrease the number of critical trials to create a child-appropriate task (e.g., Rutland et al., 2005; Dunham et al., 2014). Somewhat counter-intuitively given that children have shorter attention spans and are susceptible to fatigue, but consistent with classical test theory (Cronbach, 1951; Nunnally, 1978), the reduced-length Child-IAT (Study 1) did not demonstrate higher internal consistency than the first completed traditional-length measure (Study 2). However, both versions demonstrated comparable internal consistencies to what has been found with adults, suggesting that reducing the length of a Child-IAT should not compromise the potential of this measure to provide useful data.

Children may be asked to complete multiple Child-IATs within a single testing session for both theoretical and practical reasons (e.g., Baron and Banaji, 2006; Andrews et al., 2010; Cvencek et al., 2011, 2014). Therefore, we examined the internal consistency of the Child-IAT when completed in succession. Although the first completed Child-IAT demonstrated higher internal consistency than the second measure, all estimates fell within the expected range.

Evidence that responses on the first Child-IAT may be relatively more consistent than responses on subsequently completed tasks can have methodological implications. For example, the results from Study 2 highlight the importance of counterbalancing the order of different Child-IATs to control for erroneous variability that may confound meaningful interpretation of scores. The lower coefficients for the second Child-IAT suggests the need to carefully interpret scores from measures completed in succession. The higher proportion of error variance may attenuate meaningful relationships (Nunnally, 1978; Cunningham et al., 2001) and effect sizes (Baugh, 2002) when multiple Child-IATs are completed in a single testing session. It is also worth noting that comparable reliabilities across both administrations could be due to the fact that children completed the Child-IATs in the presence of a trained experimenter. In cases of response uncertainty, the experimenter would provide a prompt, which could have reduced error variance. Future research will be needed to examine whether Child-IATs are comparably reliable when self-administered by older children or adolescents.

In Study 2 we examined the test–retest reliability of two child-IATs completed within the same testing session using Pearson’s r and ICCs. Although less satisfactory than the internal consistency estimates, these coefficients fell within the low end of the range observed in published studies with children and adults. Reconciling acceptable internal consistency with lower test stability is an issue faced by researchers using the IAT. Teige-Mocigemba et al. (2010) have identified two potential explanations: That the IAT (a) captures trait rather than (or in addition to) state associations and/or (b) is sensitive to construct-unrelated response variability (i.e., response strategies, learning effects). The weak test–retest reliability demonstrated by Pearson’s r in combination with slightly higher ICCs highlights the possibility that on the Child-IAT children’s and adults’ responses may reflect variability unrelated to their implicit attitudes or be sensitive systematic error (e.g., learning effects, situational cues; Bedard et al., 2000; Weir, 2005) – particularly when completed within the same testing session. Factors contributing to occasion-specific variance, and in particular children’s sensitivity to these factors, is an avenue for future research.

One issue to emerge from this research is the degree to which we should be concerned by the lower internal consistency of implicit as compared to explicit measures. Lower reliability coefficients also emerge with adult responses, indicating that this problem is not restricted to child participants completing Child-IATs. Low reliability of implicit measures is a potential concern as this may reduce the probability of finding group differences (Buchner and Wippich, 2000), which has implications for the effectiveness of the Child-IAT as a tool to assess interventions and relationships between constructs. However, despite lower reliability estimates than what might be expected for explicit measures, race-attitude Child-IATs may be sensitive in capturing differences amongst groups. For example, Asian and White 9- to 12-year-olds exposed to a positive Black exemplar, as compared to positive White exemplar, demonstrated less implicit bias on a Child-IAT (Gonzalez et al., 2016).

Although currently scarce in the literature, we anticipate that evaluating interventions aimed at reducing children’s implicit racial prejudice will be a primary focus of future research. Given that internal consistency may be reduced for subsequently completed child-IATs, we recommend that researchers consider using between-participant designs, as compared to within-participants designs (cf. Schnabel and Asendorpf, 2015), to examine the effectiveness of their interventions. That IAT D-scores have been found to meaningfully differentiate between existing and experimental groups for adults (see Teige-Mocigemba et al., 2010, for a review) and children (e.g., Dunham et al., 2007; Gonzalez et al., 2016), perhaps reliability coefficients falling below the generally accepted value of 0.8 should not be interpreted as reflecting a critical deficiency in the Child-IAT (Nunnally, 1978; Pedhazur and Schmelkin, 1991). Implicit measures are subjected to greater sources of error than explicit measures (e.g., momentary inattention, etc.; Buchner and Wippich, 2000; De Houwer, 2003; Fiedler and Bluemke, 2005; Lane et al., 2007), therefore it is perhaps not surprising that responses on the Child-IAT demonstrate greater error variance than what is expected from explicit questionnaires. Indeed, it has been suggested that “low interitem consistency may be a characteristic of response-latency measures more generally” (Cunningham et al., 2001, p. 163), and should perhaps be both expected and accepted. Other methods for calculating reliability and conducting analyses involving the IAT that control for this expected increased measurement error, such as structural equation modeling, may provide more stringent evidence for hypothesis testing (e.g., Cunningham et al., 2001) and should be considered in future research.

It is worth noting that lower internal consistency and test stability are acceptable only to the extent that the Child-IAT is a valid measure of the construct of interest. With adults, the race-attitude IAT demonstrates adequate validity (see Lane et al., 2007; Teige-Mocigemba et al., 2010, for reviews), and importantly responses on this measure can be better predictors of behavior during intergroup interactions as compared to self-reported racial bias (Greenwald et al., 2009). Preliminary evidence suggests that responses on Child-IATs may similarly correspond to children’s behavior. Dunham et al. (2011) found that children’s implicit intergroup attitudes predicted the allocation of resources to minimal ingroup members. Although this is a great first step, additional research is needed to address the validity of the race-attitude Child-IAT, with a particular focus on whether this measure can predict children’s spontaneous intergroup behavior (e.g., non-verbal behavior) above and beyond explicitly reported attitudes.

Although the IAT is a widely popular measure that has been used to meaningfully assess children’s social cognition across a range of important topics (see Supplementary Table S1 in Supplementary Materials for a summary), it should be noted that the IAT is not without limitations. For example, the processes underlying the IAT are unclear (see Fiedler et al., 2006; Teige-Mocigemba et al., 2010, for comprehensive reviews) and responses are collapsed into a single relative score of categorical bias, which may mask important differences in the developmental trajectory of implicit attitudes (e.g., Degner and Wentura, 2010; Williams and Steele, unpublished). Therefore, the importance of validating other implicit measures for use with children cannot be understated (Degner and Wentura, 2010; Williams et al., 2016; Williams and Steele, unpublished). We believe that priming tasks and variants of the IAT [i.e., single-category IAT (Karpinski and Steinman, 2006), Go/No Go Association Task (Nosek and Banaji, 2001)] show particular promise in addressing the criticisms of the IAT, and may reveal exciting nuances in the developmental trajectory of children’s implicit social cognition. With this in mind, in the future researchers should aim to select the best measurement tool to assess the impact of prejudice-reduction interventions for improving intergroup relations in childhood.