In 2009, (Spencer et al. 2009) argued for the importance of taking a systems approach to studying development with a particular emphasis on considering “a broader interpretation of experience and an appreciation for the non-obvious nature of development.” In the decade and a half since their publication, more work in the field of development has explored the non-obvious factors that contribute to developmental outcomes, expanding mechanisms to include neurological, cognitive, social, and behavioral processes. Although these studies have given us greater insight into the processes behind developmental outcomes, considerably less focus has been given to the methodological choices researchers make along the way (at times, leaving some of those decisions at the level of a footnote).

Take, for instance, the body of research on children's early word learning. A long-held belief was that exposure to any speech was sufficient to help children build a vocabulary (e.g., Hart and Risley, 1995; Huttenlocher, 1998). However, though the link between overall language exposure and expressive vocabulary remains strong, more recent research has shown that the impact of the input is more nuanced with multiple features and properties being important. This includes findings that being exposed to child-directed speech facilitates children's processing of linguistic input and leads to long-term vocabulary gains (Weisleder and Fernald, 2013) and lexically diverse speech has been shown to aid early word acquisition (Hills et al., 2010).

Critically, this body of research has also moved beyond the input itself to explore the contextual factors that help young children capitalize on word learning opportunities. For example, (Yu and Smith 2012) used head-mounted eye-trackers to assess where 18-month-old children looked during a free-play session with novel objects. They found that the physical constraints of infants' short arms led to moments where one object was dominant in their visual field; if caregivers provided labels in those hot spot moments, infants were more likely to retain the novel word-referent mapping at a later test. More recently, (Schroer and Yu 2022) demonstrated that these pivotal word learning opportunities are also driven by children holding the object; simply looking at an object while hearing its label was not sufficient to support the learning at test.

Collectively, the previous research has shown, then, that word learning is not influenced by a single, obvious factor. Rather, components such as the properties and features of the input as well as the context in which it is presented matter. Knowing that there are many non-obvious factors that influence the word learning process then leads to the question—what else should researchers consider when making claims about what influences language learning? One possibility that has been given considerably less attention is the role of the stimuli, especially their salience, typicality, and presentation context.

A key (yet often forgotten) part of methodological discussions is the stimuli (images, sounds, contexts, etc.) used. To assess the extent to which developmentalists are currently transparent about their stimuli choices, we conducted a preliminary review of word learning studies published since 2020 in five developmental journals (Child Development, Developmental Science, Developmental Psychology, Infancy, Frontiers in Psychology). Specifically, naive undergraduate coders answered three questions: 1) are the stimuli described in depth?; 2) are the stimuli shared via OSF or similar framework?; 3) were the stimuli normed or from a normed set? While 76.5% of articles had “yes” for at least one question, very few (7%) answered “yes” for all three questions. Less than half of the articles were able to be coded as “yes” for questions individually − 1) 42.9% described stimuli in depth, 2) 32.7% shared the stimuli, and 3) 48.0% used normed stimuli. Taken together, this suggests that though most researchers share information about their stimuli in some form, a notable 23.5% did not share information at all, and most papers may have not provided enough information for other researchers to truly consider the effect that the stimuli may have had.

In many of these studies, the effect of stimuli are an afterthought—while stimuli may be reported, differences in children's behavior because of those decisions are not. There are of course exceptions; some studies include supplements, appendixes, or footnotes about stimuli decisions—see for instance, as noted in Figure 1, (Perry and Saffran 2017) clarify that the type of “duck” being used was a critical factor in order to accurately assess children's knowledge of features in the main study. However, few of the reviewed studies discuss how such seemingly minor decisions about which “duck” to use may have changed the results (in this example, for the better!). Here, we focus on the area of word learning as an example for how the objects themselves and the contexts in which they are presented have important implications for children's behavior and thus, the conclusions we draw.

Central to all research is the reliability and validity of the methods used. As apparent by multiple recent publications (e.g., Byers-Heinlein et al., 2022; Kominsky et al., 2022; Kucker and Chmielewski, 2022; Zettersten et al., 2022), this has become a particularly salient discussion within the developmental area, especially as the field continues to shed light on the non-obvious factors that might influence our results. Outside of word learning, classic theories in motor development traditionally posited that children navigated visual cliffs due to a fear of heights. However, work by Karen (Adolph 1995) showed that adjusting the slope of a visual cliff creates different affordances and tactile experiences that interact with the child's motor ability which ultimately influences whether or not they descend a slope. Likewise in visual search paradigms, children's attention and processing of stimuli varies based on if the stimuli presented are faces or salient objects, with object saliency diminishing as children age (Kwon et al., 2016). Thus, paying attention to the stimuli and environments we choose for a study is likely relevant for multiple developmental areas.

So if we want to improve our conclusions, regardless of the topic of study, what should developmentalists consider when designing our methods and specifically our stimuli? One idea is that researchers can continue to increase their transparency—not only can we share the datasets associated with our individual studies, but we can also provide detailed descriptions and samples of the stimuli for use by other labs. Part of this is utilizing public databases of stimuli, such as the NOUN database (Horst and Hout, 2016), OSF submissions (https://osf.io/), and Databrary (https://databrary.org/) and if there isn't a validated database or stimuli that works for your purposes, then norm the stimuli prior to running the full study (much like the first steps used when creating standardized measures and assessment tools). By taking such steps we can assure that our stimuli are not just arbitrary choices that can lead to artifacts, but are intentionally selected and quantified in ways that help us reduce noise and enhance focus on the core variables of interest.

Transparency and validation of our stimuli is important not just at the lab level, but across the field. Cross-lab collaborations, such as that of the ManyBabies consortium (https://manybabies.org/), have begun doing just this. In their seminal study, The ManyBabies Consortium et al. (2020) were able to conclude that babies preferred infant-directed speech; importantly, though, they also found that the effect was stronger “in those children for whom the stimuli matched their native language and dialect.” The effect was also specific to the head-turn preference procedure, demonstrating that children's individual experiences interact with the stimuli and context. Looking at these phenomena beyond a single lab and more cross-culturally, allows us to be more mindful of possible effects of methodological choices before making claims about universal developmental milestones, whether in language studies, those that focus on theory of mind (e.g., Schuwerk et al., 2021), rule learning (e.g., Visser et al., 2021), and more.

Moreover, another option for tackling these challenges is to take advantage of the growth in statistical modeling available to us. For instance, in addition to systematically testing for order effects, researchers can use mixed-effects models to account for item (and even participant) variability (Baayen et al., 2008; Muradoglu et al., 2023). Accounting for that variability in our analyses should enable us to more clearly see the throughline, or answers to, the research questions at the heart of developmental science even when the stimuli and methods may vary.

Many of these are things we have been aware of for years and may seem obvious, but may be ignored or dismissed as not relevant or applicable to our specific study. But, the evidence suggests it might indeed be critical to consider these details about our stimuli and methods. In short, we need to go back to our Methods 101 lessons—remembering that the way we ask our questions, the sequence of those questions, and what we present, all alter behavior and present possible confounds that can cloud our understanding of the true variables of interest. So as we look forward to the next quarter century for our dynamic field, let's move beyond the footnote to collaborate and share methods and stimuli that will allow us to draw stronger conclusions about cognitive development.