We discuss how we generated or revised items for the measure and how the items relate to the literature on early math development as evidence of the measure’s content validity (Joint Committee on the Standards for Educational and Psychological Testing, 2014). Our goal was to have reliable subscales of early math domains, given previous research indicating that preschoolers can and should be learning about early math skills including patterning and spatial skills (Verdine et al., 2014; Fyfe et al., 2019). Thus, we created items to measure a wider variety of early math skills given that more than half of the items in the original measure focused on numeracy. For example, we created a new item, “Fill in the missing part of a pattern made of repeating objects (for example, circle, square, square, circle, square, _____, circle, square, square)” based on Rittle-Johnson et al. (2020) in order to include more items on patterning skills. In general, we referenced prior research on early math development to identify math tasks most children in the U.S. are able to accomplish independently before kindergarten or their fifth birthday (e.g., National Research Council, 2009; Claessens and Engel, 2013; Clements and Sarama, 2014; Rittle-Johnson et al., 2017, 2019; Litkowski et al., 2020a; Kaufman et al., 2021). See Supplementary Table S1 for details about the final measure’s items including the origin of each item.

Overall, sixteen items (8 patterning, 5 numeracy, and 3 spatial) were inductively added to measure a wider variety of math skills and to make the measure similar across subscales. Nine items were dropped primarily because they were ambiguous or were very similar to other items (e.g., “Use a computer with age-appropriate software to learn math concepts” did not focus on a specific numeracy, patterning, or spatial skill). Ten items from the original measure were also revised for clarity. The instruction was also expanded with the addition of a sentence that reads “Please select ‘yes’ for each skill that you think most children in the United States correctly master by age five. Please select ‘no’ for each skill that you do not think most children in the United States correctly master by age five.”

After the first round of edits, the measure included 30 items (10 numeracy, 10 patterning, and 10 spatial). Within each subscale, there were seven items on skills children typically develop by age five (within the developmental range) and three on skills children typically do not develop by age five (beyond the developmental range). Parents’ correct responses to each item were scored as a 1 and incorrect answers were scored as a 0. Parents’ scores were averaged to create a measure of their knowledge across numeracy, patterning, and spatial. Parents’ scores were also averaged to create separate composite measures for numeracy, patterning, and spatial for their knowledge (1) across all items, (2) about skills within the typical developmental range for most preschoolers, and (3) about skills beyond the typical developmental range for most preschoolers.

We used Cronbach’s alpha as an indicator of internal consistency and interpreted alpha levels based on previous research (Cooper and Schindler, 2003; Cohen et al., 2007). The measure had moderate/acceptable internal consistency across all items (α = 0.60) but had unacceptable reliability when considering subscales composed of all numeracy (α = 0.36), patterning (α = 0.32), and spatial (α = −0.02) items. When considering only the items that are within the typical development range for preschoolers, the numeracy subscale had moderate/acceptable reliability (α = 0.69), the patterning subscale had low reliability (α = 0.53) and the spatial subscale had unacceptable (but much better) reliability (α = 0.45). When considering the items that are beyond the typical development range for preschoolers, both the numeracy subscale (α = 0.69) and patterning subscales (α = 0.67) had moderate/acceptable reliability while the spatial subscale had unacceptable reliability (α = 0.46). Given the poor reliability of the spatial subscales, we focused further analyses and revisions on the numeracy and patterning subscales.

We used Confirmatory Factor Analyses (CFA) to examine whether each subscale’s items measured a shared construct as indicated by each item having a significant factor loading. All numeracy items loaded significantly onto a model with two correlated factors, with one factor including items that measure skills that are beyond the typical developmental range for preschoolers and the other factor including items that measure skills that are within the developmental range (see standardized factor loadings in Table 3). All patterning items except for item 5 (“Sort a set of objects into 3 groups based on color such as red, blue, and green”) loaded significantly onto a similar 2-factor model (see standardized factor loadings in Table 4). Thus, we concluded that we have evidence of construct validity: all numeracy items do measure the same construct and most patterning items do measure the same construct.

We also compared the 2-factor models to 1-factor models which did not consider items measuring skills that are within the typical developmental range separately from items that measure skills that are beyond the typical developmental range. We found that the 2-factor model fit the data significantly better than a 1-factor model for both numeracy, χ²(3) = 98.28, p < 0.001, and patterning, χ²(3) = 31.7, p < 0.001. This finding suggests that the subscales (within the typical developmental range for preschoolers and beyond the typical developmental range for preschoolers) were unique. The subscales (within the typical developmental range for preschoolers and beyond the typical developmental range for preschoolers) were negatively correlated providing additional evidence that they were unique and needed to be treated as separate measures, r_numeracy(159) = −0.25, p < 0.001 and r_pattern(159) = −0.33, p < 0.001. Further studies focused on revising and validating the subscales consisting of items that are within the developmental range for most typically developing preschool-aged children given that the intended audience of the measure was parents of preschoolers. Additionally, the subsequent studies focused on validating the subscales among parents of 3-year-olds and/or 4-year-olds since the measures ask about skills that are relevant to preschool-aged children who are younger than 5 years old.

In contrast to study 1, and perhaps due to the much smaller sample size, the measure’s internal consistency was low for the items measuring numeracy skills within the typical developmental range for preschoolers (α = 0.53) and moderate for the items measuring patterning skills within the typical developmental range (α = 0.64) at Time 1. Notably, both subscales had high reliability at Time 2 after parents received some information designed to change their knowledge about early numeracy or patterning development (α_numeracy = 0.72 and α_patterning = 0.87). Additionally, these subscales had high test–retest reliability, with strong correlations between the numeracy scores, r(19) = 0.65, p = 0.001, and patterning scores, r(19) = 0.80, p < 0.001, across two weeks.

Two authors made notes about parents’ comments and questions during the think-aloud sessions and identified themes that emerged among parents after discussing their notes. First, several parents posed clarifying questions to the experimenter as they participated in the think-aloud. Overall, parents asked nine questions as they read and responded to the knowledge measure. This included questions about the difference between “numerals” and “numbers” which suggested that some items’ phrasing needed to be clarified as well as uncertainty about what was meant by “most children” in the knowledge measure question. We revised the measure based on this feedback. Second, many parents expressed being unsure about their answers to knowledge measure items. Parents frequently said “I do not know” (25 times) or “probably” (10 times) when thinking through their answers to individual items. This finding suggests that parents have little confidence in their knowledge about early math skills.

Items focused on numeracy skills that are within the developmental range had low (but not unacceptable) reliability (α = 0.59). The Kuder–Richardson Formula 20 (KR20), often considered a better measure of reliability for tests with dichotomous variables, yielded a similar estimate (0.60). One item (“Count a row of 15 objects”) had a low item-total correlation. As such, it was excluded from further analyses. The patterning items that are within the developmental range had moderate/acceptable reliability (α = 0.65 and KR20 = 0.66). As with patterning, one item (“Sort a set of objects into 3 groups based on color such as red, blue, and green”) had a low item-total correlation and was excluded from further analyses.

We used CFA to examine whether each subscale’s items measured a shared construct as indicated by each item having a significant factor loading (see standardized factor loadings in Tables 3, 4). All numeracy items measuring skills within the typical developmental range for most preschoolers loaded significantly onto a factor. Similar to study 2, all patterning items measuring skills within the typical developmental range except for item 5 loaded significantly onto a factor. Importantly, the models fit the data well according to several indices such as a nonsignificant chi-square, Comparative Fit Index >0.9, Root Mean Square Error of Approximation <0.08, and Adjusted Goodness of Fit >0.9 (see Supplementary Table S2). Thus, we concluded that after dropping item 5, we have evidence of construct validity: numeracy items do measure the same construct and patterning items do measure the same construct.

Next, we measured the convergent validity of the numeracy and patterning subscales. The Joint Committee on the Standards for Educational and Psychological Testing (2014, p. 16–17) reports that “relationships between test scores and other [external] measures intended to assess the same or similar constructs provide convergent evidence.” We view parents’ knowledge and beliefs as similar constructs- two types of parental cognitions that potentially influence their support. We examined whether parents’ knowledge about numeracy skills that most children develop by age 5 was related to their beliefs about their child’s numeracy ability and did the same for patterning. We found evidence of convergent validity for both. Specifically, parents’ knowledge about early patterning development was significantly correlated with their perception of their child’s patterning abilities, r(342) = 0.24, p < 0.001. Parents’ knowledge about early numeracy development was also significantly correlated with their perception of their child’s numeracy abilities, r(342) = 0.11, p = 0.036.

In line with research indicating that the home numeracy environment and the home literacy environment are separate constructs (even though they have some shared variance; Napoli and Purpura, 2018), we view parental belief about literacy as a distinct construct from their knowledge about math. As such, we examined the relationship between these two constructs as an indicator of discriminant validity. We found evidence of discriminant validity for both types of parental knowledge. Specifically, parents’ knowledge about early numeracy development was not correlated with their belief about their child’s literacy abilities, r(343) = 0.05, p = 0.342. Additionally, parents’ knowledge about early patterning development was not correlated with their perception of their child’s literacy abilities, r(343) = 0.09, p = 0.112.

To explore potential sources of parents’ knowledge about early math development, we examined their reports of how they decided on their answers (see Supplementary Table S2 for additional details). Forty-three percent of parents’ responses did not fit the coding scheme (e.g., some parents responded to the question with a single word). Of the 196 parents whose responses were coded, most mentioned that they thought about their experience with other children (54%) and/or their perception of their participating child’s current abilities and their expectations for their participating child’s math development (45%). Very few parents mentioned their knowledge of benchmarks or other information about children’s early math development (16%).