There are currently three sets of tests aimed at young learners of English that are also available in China: the Cambridge Young Learners English Test (CYLE), developed by Cambridge ESOL Examinations, the Pearson Test of English Young Learners (PTE Young Learners), developed by Pearson, and the TOEFL Primary test, developed by ETS.

The CYLE test series tests how well 7–12-year–olds are performing in the skills of listening, speaking, reading, and writing. It has so far been administered in 68 countries and has more than 360,000 test takers annually (Chik and Besser, 2011). The PTE Young Learners, called Starters is aimed at learners between 6 and 13 years, also assessing the four language skills. TOEFL Primary is for learners 8 years and up and assesses reading and writing and speaking. All three tests are administered at designated test centers, with TOEFL Primary being the only one that also offers institutional testing at schools.

Although these three tests are aimed at children above kindergarten age (3–6 years in China), they are nonetheless often used in Chinese kindergartens. This is highly problematic for several reasons. A major feature of these tests is that their format requires that children can already read and write, which is not the case for most kindergarten children in China who are just beginning to learn English. When used with preliterate children, these tests will therefore produce inaccurate results—a child may have a certain level of oral proficiency, but this will not be captured by a test that requires literacy. Furthermore, these tests are designed for children above elementary school and lack the play-based interface that is necessary to engage young children. And finally, the tests are designed according to the cultural norms of Western countries (specifically the United States, the United Kingdom, Canada, and Australia), which makes them less accessible for Chinese children who lack the cultural background knowledge, as we will discuss in more detail in the next section.

Another issue with these tests is that they are designed to cover all areas of language ability, with a focus on phonological awareness and grammar. However, the consensus in early language teaching is that for very young children, teaching should be focused on developing lexical competence and no explicit grammar instruction, as children do not develop the necessary metalinguistic skills until much later (Curtain and Dahlberg, 2010; Shin and Crandall, 2014). In line with this, any instrument for assessing young learners’ proficiency should focus on vocabulary.

When it comes to measuring the vocabulary of young Chinese EFL learners, researchers and teachers typically rely on assessments such as the PPVT (Dunn and Dunn, 2007) or the British Picture Vocabulary Scale, BPVS (Dunn et al., 2009). These tests have been developed over many years, and typically have very good psychometric properties. However, they are aimed at children learning English as their first language (L1). Using these tests for EFL learners is problematic for two different but related reasons.

The first reason is that the learning environment for children who learn English in an educational setting differs from that of children learning it in their home environment. Children who initially acquire words in their home environment learn these from conversations with and among adults as well as through activities such as pretend to play and book reading (Ninio, 1983; Snow et al., 1991). Their early vocabulary is characterized by words that reflect what is relevant for the child, such as family members (e.g., mommy, daddy), toys (e.g., ball, teddy), body parts (e.g., toe, hand), or food items (e.g., sandwich, cookie). In line with this, the structure of vocabulary tests like the PPVT is based on the idea that in the child’s experience, some words are more frequent than others, and that children will acquire more frequent words before less frequent words (Dunn and Dunn, 2007).

However, the words used in English lessons in Chinese schools are different from what a child may encounter when it learns English from parents and others at home. English lessons often introduce school-related vocabulary (e.g., classroom, pencil, ruler), places (e.g., library, hospital, post office), shapes and colors (e.g., rectangle, red), and animals (e.g., elephant, snake). Thus, testing Chinese young EFL learners with words that children growing up in an English-speaking environment typically learn at home will not provide an accurate picture of their vocabulary knowledge.

The second, related reason why these tests are problematic is that the items are based on the dominant culture of the country for which the test was developed (e.g., United States, United Kingdom). This applies both to the types of items used as well as the way items are depicted. An example for the first issue items (‘items’ being used here to mean both targets and distractors) like muffin and pretzel (both used in the PPVT), items that are unlikely to be known to young Chinese children. Examples of culturally specific depictions are a traditional English teacup with a handle on a saucer (used in the BPVS), which are not common in China, or a castle in European medieval style (used in the PPVT), which is very different from the way castles look in China.

Recent research found the PPVT to be less reliable for L2 learners with limited English experience and proficiency. Wood et al. (2015) tested both Spanish-speaking kindergarteners’ and monolingual kindergarteners’ vocabulary using the PPVT-4. They observed that the relationship between the difficulty level of the items in the PPVT (which is indicated by the order of items in the test) was positively related to children’s error scores in both groups (i.e., children made more errors with more difficult items). However, this relation was much stronger in English monolingual children than in Spanish L2 learners of English. In other words, the findings suggest that the difficulty assumptions that the test is based on do not hold to the same extent for L2 learners as they do for L1 learners. Goriot et al. (2018) administered the PPVT-4 to pupils in the Netherlands in six different age groups (4–15 years old). They found that the test had low reliability scores (as measured by Cronbach’s alpha) for learners with limited proficiency; these were predominantly children in the youngest age of 4–5 years. They also found an effect of the children’s L1. English words that had cognates in Dutch (e.g., penguin and pinguin) tended to be easier for participants than words that did not, most likely because participants were able to guess the meaning. Like Wood and Pena’s study, this study showed that the test’s reliability is lower with L2 speakers. In addition, the participants in Goriot et al. (2018) had the advantage of speaking a typologically close L1 and of being culturally closer to America. Potential issues with tests like the PPVT are arguably more pronounced in young Chinese learners of English, whose L1 does not have any linguistic similarities with English and who have less experience with Western culture.

The learning environment of young Chinese English learners is different also from that of dual language learners in an English-speaking country (e.g., Hispanic English language learners in the United States, who speak Spanish at home, but who learn and speak English at school, or EAL learners in the United Kingdom who speak Pashtu at home but speak English at school). Children whose home language is not English have less exposure to English than children whose home language is English (e.g., Cote and Bornstein, 2014), and they score consistently lower on English vocabulary tests than their monolingual counterparts (REF). But unlike foreign language learners in China, these children are immersed in English continuously at school (not only during English lessons), and are likely to interact with others (peers, teachers) in English on a regular basis. Researchers who work with this population have pointed out that test validity is threatened when available norms are based on monolingual children, when the child’s cultural experiences do not match test expectations, or when the items are not presented in a way that allows the child to demonstrate competence (Peña and Halle, 2011). This problem is exacerbated for children who do not even have minimal experience with the cultural norms reflected in test items, such as young Chinese learners of English living in China.

Against this background, we developed a new vocabulary test, designed specifically for young Chinese learners of English. But before we move on to describing this new instrument, we briefly want to discuss what it means for a young learner to “know a word” (in comprehension).

One-hundred-and-eighty-one children between 3 and 7 years of age (M = 5;04¹) were recruited from two preschools in two major cities in China (see Table 1): one in eastern China (n = 72) and one in southern China (n = 109). Socio-economic background is often correlated with educational outcomes. We therefore wanted to include some information on the participants’ SES. As it was not possible to collect information about parents’ income or their educational background, we used the tuition costs of the preschools and publicly available economic information about the catchment areas of the preschools as indicators (here: housing prices). The first preschool served predominantly middle SES families: Its tuition was 28% higher than the average tuition in the city, while the average housing price was ¥ 45,280/m², which is slightly lower than the average housing price of both the cities (which is ¥ 50,000/m² in both cities). The second preschool served predominantly high SES families, as evidenced by the fact that its tuition is ten times the average tuition in the southern China city and the housing price in its catchment area is more than two times the average (¥ 108,920/m²). Based on the total number of children enrolled in each preschool, 40% of all children were randomly selected from each grade. Of the 181 children in our sample, 75 were female and 106 were male.

Nine-hundred-and-eleven children participated in the second field test. The children were randomly selected from each grade in each preschool. Of these, 405 were female and 506 were male (mean age = 5;01).

We employed the same selection criteria as in the first field test and recruited six kindergartens and one elementary school for the second field test. Four kindergartens were located in a metropolis in southern China (N = 694), one from a low SES neighborhood with tuition 36% lower than the average school tuition in the city, and with an average housing price of ¥ 26,500/m²; two from middle SES neighborhoods with tuition 28% higher than the average tuition and with an average housing price of ¥ 54,580/m², and one from a middle-to-high SES neighborhood with tuition 28% higher than the average tuition and with an average housing price of ¥ 93,410/m². The elementary school (with tuition 28% higher than the average tuition in the city) was from a middle-to-high SES neighborhood. Two kindergartens were in a metropolis in eastern China (N = 217; one from a middle SES neighborhood with tuition six times the average tuition and with an average housing price of ¥ 77,930/m² and one from a high SES neighborhood with tuition ten times the average tuition and with an average housing price of ¥ 108,920/m²).

Some children did not attend preschool on the second day of testing, or they did not want to complete one or several of the tests, often because they were shy. Of the children in the first cohort (N = 558), 22 children did not complete either of the two forms of the ACCE-V, and 21 of these did also not complete the PPVT-4. One child completed only Form A, but none of the other tests. Of the children in the second cohort (N = 353), 12 did not complete either of the two forms of the ACCE-V, and of these twelve, four did not complete these tests on the re-test date, either. One child did not complete Form B on the first day but did complete Form A and both forms on the re-test date.

The main purpose of the ACCE-V is to assess the level of receptive vocabulary knowledge relevant in the context of Chinese primary English education. Our search for target items therefore began by surveying the most widely used English textbooks and working books developed for first and second graders in China (see Table 2). From these books, we extracted the English words used in exercises, texts, and instructions (excluding pronouns and conjunctions). Altogether, 595 words (nouns, pronouns, adjectives, adverbs, verbs, and prepositions) were extracted. The procedure of the ACCE-V development was illustrated in Figure 1.

Each word was assigned to one of 18 content categories (actions, animals, body parts, attributes², people, buildings and spaces, vehicles, household objects, clothing and accessories, shapes and colors, nature and landscapes, food, plants and fruit, books and money, toys and recreation, times, numbers, prepositions, pronouns, abstract concepts). We assessed the usability of the words based on frequency measures, concreteness, and imageability.

A word’s frequency is measured by counting how often it occurs in a corpus (e.g., in a collection of books or newspapers). We included frequency measures as a criterion because it seems fair to assume that more frequent words will more likely be found in different textbooks (and consequently be used in more English language classes and known to more children), whereas less frequent words reflect more idiosyncratic choices by a textbook publisher.

In the absence of a corpus of words used in Chinese English-language classes, we used two widely used databases: the CELEX database (Baayen et al., 1996), which is based on the COBUILD corpus with around 17.9 million tokens (from both written and spoken sources), and the SubtlexUS database (Brysbaert and New, 2009), which contains 50 million tokens and is based on American movies and TV series subtitles. We used the Cob frequency (from CELEX) and the SUBTLwf (from SubtlexUS) to identify and exclude low-frequency words (e.g., narrator, magnificent), and to decide between semantically similar words (e.g., lollipop and candy), assuming that more frequent words were more likely to be taught (in this case candy has a higher frequency than lollipop).

Concreteness and imageability were used to restrict the pool of possible target words to those that would be more likely to be known to children and that could be depicted clearly using drawings. Imageability is defined as the ease with which a word gives rise to a sensory mental image (Paivio et al., 1968), while concreteness refers to the ability to see, hear, and touch something. Empirically, words like difference or against tend to get lower concreteness ratings than words like banana or running.³ Not surprisingly, words with lower concreteness ratings are also typically more difficult to illustrate. We gauged a words’ imageability and concreteness using our own and the illustrator’s (see below) introspection and our experience with creating visual stimuli for young children.

To compare ACCE and other vocabulary tests, we chose the Chinese version of PPVT (Lu and Liu, 1998, henceforth: PPVT-C) and the English version of PPVT (Dunn and Dunn, 2007, the PPVT-4).

The PPVT-C contained 115 items with a possible score ranging from 0 to 115. It was translated and validated in Taiwan. Like the ACCE, the PPVT-C is a forced-choice picture selection format, in which the child is presented with a word and then asked to select the target picture matching that word from an array of four pictures. If a child answers five out of seven consecutive items wrong, the examiner would stop the test and record the score. The internal consistency of the PPVT-C was 0.83, and that of the PPVT-4 was 0.89.

In addition to the revised ACCE-V vocabulary test (form A and form B) and PPVT-C (Lu and Liu, 1998), we also included the English PPVT-4 (Dunn and Dunn, 2007) and the Chinese Expressive Vocabulary Test (EVT) to test criterion validity (concurrent validity). The EVT in Chinese was adapted by the Child Language Research Center (CLRC) at East China Normal University (see Chen et al., 2018 for details). Possible scores range from 0 to 124. The internal consistencies (Cronbach’s alpha) of the PPVT-C, the English PPVT-4, and the Chinese EVT in our sample were 0.82, 0.96, and 0.73, respectively.

To assess the psychometric properties of the ACCE-V, we used both Item Response Theory (IRT) and classical test theory to determine the test’s reliability and validity. After the first field test, we modified or removed items that did not have satisfactory properties, and then tested the modified version again in a second field test. We used a two-parameter (2PL) IRT model to fit the data.

Once we had established that the ACCE had good psychometric properties, we used multiple regression models to compare the children’s age, gender, and English vocabulary scores. In the regression models, the outcome (English vocabulary score) was modeled as a linear combination of the predictor variable (age), controlling for gender, foreign teacher, housing price, and tuition.

Analyses were conducted using STATA 15.0 (StataCorp, 2017) and R 3.6.2 (R Core Team, 2019).

The uni-dimensionality assumption of the data was evaluated by using a principal components analysis (PCA). For Form A, the PCA showed that the first component (26.89%) accounted for substantially more variation than the second component (7.00%) and subsequent composites, indicating that Form A was measuring a unidimensional ability. Results were similar for Form B, where the first component accounted for 28.43% of the total variance whereas the second component accounted for only 7.72%. The uni-dimensionality assumption was thus met. We used a 2PL model to fit the data, allowing us to estimate difficulty and discrimination parameters for each item. Items with lower discrimination parameter estimates and items that were too easy were removed. Examples were illustrated in Figure 3.

To shorten the measurement while preserving the reliability and information, the original 30-item test was shortened to a 20-item test by using the Test Information Function (TIF) and Conditional Standard Error of Measurement (CSEM). Both measures allowed us to compare the total information provided by a different shortened version of a test (see Figure 4). For each form in the field test, we selected 20 items out of 30 were that minimized the CSEM and maximized the TIF. In addition, difficulty parameter estimates were used to arrange items from easy to difficult in each form.

Internal consistency measures were calculated for each 6-month age band. Table 4 shows the internal consistency measures (split-half reliability, Cronbach’s α, and standard error of measurement) for each of the age bands. All measures indicate high to very high internal consistency of the ACCE-V, indicating that it is suitable for all age ranges.

Both forms of the ACCE-V were completed twice by 340 children, with a test interval of 1 week. The correlation for Form A was r = 0.86 and r = 0.85 for Form B, indicating good test–retest reliability.

We determined alternate form reliability by correlating scores from Form A and Form B. Overall, the alternate form reliability was 0.85. Table 4 shows the correlations for each individual age band. There were moderate to strong positive correlations between both forms, indicating good alternate form reliability. At r = 0.65 the correlation is lowest in the youngest age band (3.0–3.5 years), and the only band with a correlation below 0.8. We believe that the youngest age group may have had more difficulty keeping their concentration throughout the multiple assessments, despite the pauses in between. In addition, this age band had the smallest sample size (N = 36).

The ACCE-V scores were correlated with the PPVT-4, the PPVT-C, and the Chinese EVT. Table 5 shows the correlation of the test scores with each other.

ACCE-V scores were most strongly correlated with PPVT-4 scores. Since the PPVT and the ACCE-V both assess children’s English receptive vocabulary, this is to be expected and desirable. Correlations with the PPVT-C are moderate, showing that children’s Mandarin skills and their L2 English skills are related. The strength of the correlation is in line with what is generally found for receptive L1 and L2 vocabulary in young learners (Atwill et al., 2007; Karlsen et al., 2017; Grøver et al., 2018). The correlations with the Chinese Expressive Vocabulary Test are lowest at around 0.3. This is not surprising, given that both tests are measuring proficiency in different languages (English vs. Mandarin) and in different modalities (receptive vs. productive).

We again used a 2PL model to measure the items’ difficulty and discrimination. All items had very good discrimination indices and provided a good range of difficulty indices (see Table 6; Figure 5). The most difficult items tended to be verbs (e.g., putting) and prepositions (e.g., on).

We also compared the ACCE-V’s difficulty parameters with those of the PPVT-4. As can be seen in Figure 6, some items from the PPVT that are presented early in the test and thus assumed to be relatively easy were in fact quite difficult for the children in our sample. Examples are item 3 (spoon) and item 8 (cup). At the same time, there were also items that are assumed to be relatively difficult that turned out to be less difficult than some of the earlier presented items. Examples are item 38 (penguin) and item 58 (panda). Note that these items were assumed to be more difficult by the developers of the PPVT, as indicated by their position in the tests (easy items occur early, harder items later). Overall, the items in the ACCE-V increase monotonously in difficulty, whereas the difficulty of the items in the PPVT does not.

Note that the current version of the ACCE-V does not have a stopping rule, since the test is brief. We re-analyzed the data using a (hypothetical) stopping rule of three incorrect responses in a row and four incorrect responses in a row, respectively. Using the stopping rule did not affect the instrument’s reliability.

Finally, we used the Mantel–Haenszel method (Mantel and Haenszel, 1959) for Differential Item Functioning (DIF) to test for each item whether it was more difficult for either boys or girls. Three items in Form A were potential DIF items. Two showed slight advantages for girls and one an advantage for boys: grandmother (65% correct among girls, 56.6% correct among boys), butterfly (68.1% correct responses among girls, 56.8% correct responses among boys), and triangle (41.1% correct responses among girls, 49.2% correct responses among boys).