Formal education occupies only a small portion of a person’s life, making out-of-school experiences crucial in shaping scientific understanding over time (Kim and Dopico, 2016). Therefore, improving STEM education requires identifying barriers and developing collaborations across informal, after-school, and formal platforms (Labov and Olson, 2014). These kinds of experiences are often categorized as Informal Science Learning (ISL), experiences that extend educational opportunities beyond the traditional school day (National Research Council, 2009a). A key implication for ISL is learners’ prior knowledge and understanding of the world are just as important as the new knowledge conveyed through specific experiences (Schweingruber and Fenichel, 2010). Pairing ISL hands-on activities with prompts that encourage self-reflection can support the learning process and foster deeper understanding (Schweingruber and Fenichel, 2010; Schweingruber et al., 2014).

Targeted interventions through informal science learning play an important role in reinforcing STEM identities during key transitions, like the shift from middle to high school, when important academic decisions are made (Archer et al., 2010). Middle school is a critical period when students start to lose interest in science and mathematics (Christensen and Knezek, 2017; Thomas and Larwin, 2023) and start to solidify their academic interests and career aspirations (Christensen and Knezek, 2017; Wang and Degol, 2013).

The STEM activities described in this paper were designed to incorporate science, technology, and math, but all were structured around the engineering design process (EDP) framework (Cunningham, 2018), which consists of five steps (Ask, Imagine, Plan, Create, and Improve). Using the EDP has been shown to enrich scientific literacy (Bethke Wendell and Rogers, 2013; Yoon et al., 2014), attitudes (Cunningham and Lachapelle, 2012), and interests (Aguirre-Muñoz et al., 2021; Guzey et al., 2016). Early exposure to the EDP can also influence students’ academic (Kelley et al., 2015) and STEM identities (Talafian et al., 2019).

EDP exposure yields positive outcomes, especially when integrated with culturally responsive teaching (Feder et al., 2009; Gay and Howard, 2000; Ladson-Billings, 2022). Bell et al. (2009) emphasized effective learning experiences consider the intersection of individual, social, and historical contexts. Additionally, Krapp and Prenzel (2011) argue that interest-driven interactions with the environment create optimal learning experiences by blending cognitive engagement with positive emotions.

In response, educators from a research university in the Central Valley collaborated with two underserved middle school populations in the region to introduce engineering concepts through embodied learning in ISL settings. These beyond-the-classroom environments offer opportunities for students to explore STEM concepts more freely than formal classrooms, potentially having long-lasting impacts on their STEM engagement (Staus et al., 2021). This research explores whether repeated exposures to hands-on activities using the EDP framework influences STEM identity, self-efficacy, and perceptions of both general and lesson-specific engineering knowledge.

We hypothesize a single exposure will significantly improve students’ STEM identity and self-efficacy compared to pre-test levels, with subsequent exposures yielding smaller gains due to a ceiling effect. In this context, a ceiling effect implies that as students’ STEM identity and self-efficacy reach higher levels, there is less room for noticeable improvement, even with additional interventions (Judson, 2012; National Research Council, 2009b). This aligns with Bandura’s (1997) research indicating initial successes boost confidence, while repeated experiences result in diminishing returns as familiarity grows. Similarly, we expect a single exposure to significantly enhance perceptions of basic engineering knowledge, supported by evidence that structured out-of-school programs foster science interests and achievement (Bell et al., 2009). We also hypothesize prior exposures will not significantly affect perceptions of knowledge related to the specific engineering type taught (e.g., electrical v. environmental); these perceptions are shaped mainly by the current lesson’s content.

The following research questions guide this study:

We conducted two studies: Study 1 was during an immersive Summer STEM Academy, and Study 2 took place over a series of Saturday STEM Academies held throughout the school year. These differing contexts allowed us to explore the effects of both concentrated and repeated experiences on student outcomes.

This study included approximately 50 middle school students from a large school district in California’s Central Valley, selected from a year-long African American Student Leadership Program. Among those who responded to the surveys, 32 identified as female (68.1%), 12 as male (25.5%), and 3 as other or preferred not to disclose their gender (6.4%).

This intervention was based on “Water, water everywhere: Designing water filters” (The EiE Team, 2020c). The concepts were enhanced with water testing strips and upgraded filter materials (activated charcoal and filter fiber), along with recommended items like sand and screens. The story provided in the EiE was replaced with information about local watersheds and pollutants in the Central Valley of California, highlighting the region’s serious water pollution issues caused by agricultural practices. This context was chosen to resonate with students’ lived experiences, fostering motivation to design water filter prototypes as solutions to challenges directly affecting their communities. The lesson was designed to actively engage students in the learning process, incorporating both theoretical and practical components to ensure a comprehensive understanding of the subject matter. Please refer to Table 1 for details on the engineering focus and the specific activities related to the engineering field.

The instructional activities spanned approximately five hours, divided across two days. The lesson began with an activity called “Technology in a bag,” where students were given a bag with an object inside and determined whether it was technology. This interactive activity allowed students to physically move to a designated area in the room, labeled ‘technology’ or ‘not technology,’ to express their answers. This activity facilitated a discussion on common misconceptions about technology, which were addressed, and students collaborated to refine their understanding of technology.

Members of the research team then presented scientific concepts relevant to the upcoming activities, focused on the water cycle, watersheds, and groundwater contaminants, laying the foundation for understanding the environmental context in which the engineering challenge would occur. The EDP was introduced through a brief video presentation, and its importance in engineering was emphasized, establishing it as a critical framework for approaching the hands-on problem-solving tasks with which they would engage.

Students explored the issue of water pollution, with a particular focus on the Central Valley, where fertilizers and industrial waste exacerbate the problem. The lesson concluded with hands-on activities, during which students conducted contaminant tests on water samples from local watersheds and evaluated the effectiveness of various materials in filtering pollutants. Based on the data they collected, students designed and constructed water filter prototypes, which were tested for their contaminant removal efficiency. Throughout the process, students were encouraged to document their findings and use the data to refine their designs.

When asked to describe the steps an engineer would use to solve a problem, thematic analysis of their responses revealed two major themes: ‘Problem-solving’ and ‘Describing steps of the EDP’ (please see Table 2). Students emphasized identifying and resolving problems as central to engineering. For example, students mentioned the need to ‘find the problem’ before devising potential solutions. One student stated, “They would try to find what’s the problem, then see what they can build or do to fix it, and then create it.” This reflects a basic, yet crucial, understanding of problem-solving as a fundamental part of the engineering process.

Additionally, 58% of students who answered the prompt demonstrated a solid grasp of what is required for effective problem-solving as an engineer. They described EDP steps such as brainstorming, planning, testing, and refining solutions. One noted, “Identify needs, Research the problem, Brainstorm possible solutions, Choose the most promising solution, Carry out the solution, Evaluate, Reflect and Improve.” These themes indicate students grasped the EDP’s systematic and iterative nature, reflecting the intervention’s success in teaching a foundational understanding of engineering problem-solving.

A Wilcoxon signed rank sum test with continuity correction found no significant change in PEK scores from pre-test (M = 2.45, SD = 0.93) to post-test (M = 2.27, SD = 0.95), Z = 0.67, p = 0.507, 1-β = 0.73. The mean difference in perceptions of engineering knowledge (PEK) scores was −0.17, (95% CI [−2.75, 2.00]). Please see Table 3 for descriptive and inferential results of all constructs.

Participants were recruited from a high school and middle school in a mid-sized school district in California’s Central Valley, where the student population is 90% Hispanic or Latino, and 83% of students are eligible for free lunch (National Center for Education Statistics, 2024). Students were incentivized to attend Saturday academies to clear unexcused absences, and due to the informal science nature, some students left before completing the post-test survey, as noted in each lesson’s section. Participants for each lesson consisted of a different population, with possible overlap across lessons, meaning some students may have participated in more than one academy.

As an icebreaker activity, students were asked their reasons for attending the STEM academies. Many shared that they attended because it helped clear their absences or gave them priority for a week-long university Summer STEM camp. Several students expressed excitement about the opportunity to visit the university campus and experience something different, describing it as a chance to step out of their hometown.

A total of 44 students attended the Geotechnical Engineering lesson, with a balanced gender distribution (47.7% male, 52.3% female). The Optical Engineering lesson had 20 middle school students (55% male, 45% female), mostly 8th graders (85%). The Electrical Engineering lesson had 14 students, primarily female (71.4%), with attendees from 7th to 9th grade. Finally, 20 students participated in the Green Engineering lesson, evenly split by gender, with most in 7th and 8th grade.

Instructional materials were adapted from the following sources: ‘A Stick in the Mud: Evaluating a Landscape’ (The EiE Team, 2020a); ‘Lighten Up: Designing Lighting Systems’ (The EiE Team, 2020b); ‘An Alarming Idea: Designing Alarm Circuits’ (The EiE Team, 2005); and ‘Now You’re Cooking: Designing Solar Ovens’ (The EiE Team, 2020d). Each lesson was structured around the EDP framework and tailored to be culturally relevant and age-appropriate for the study population. Four lessons were conducted—two in the fall and two in the spring—each lasting approximately four hours. Every session began with introducing technology and the EDP, ensuring a consistent foundation. An overview of the lessons and deviations in context and materials from the original EiE plans is provided below. For detailed descriptions of specific activities and their corresponding NGSS codes, please refer to Table 4.

Students demonstrated a good understanding of the content, as evidenced by the high proportion of correct answers for the “True/False Formative Assessment” included in the Optical Engineering lesson (please see Figure 1). Qualitative analysis of engineering journals revealed several key themes in how students communicated their understanding of the specific engineering concepts and the EDP. Students demonstrated a greater understanding of technology compared to their initial ideas of technology; they successfully recalled engineering concepts, utilized discipline-specific terminology, and effectively integrated science with engineering in their reflections.

Additionally, program participants communicated a deeper understanding of the EDP by showing an awareness of how to achieve success through specific aspects of the process. The hands-on activities served to support understanding in meaningful ways, as students often referred to these experiences to articulate their understanding of the engineering concepts. For example, in their geotechnical engineering journal, one group wrote: “We think the bedrock will keep the skyscraper stable. The skyscraper fell on the first layer. In the second layer the skyscraper started learning instead. On the third layer the skyscraper stayed straight.” This response was directly informed by their experience with the “Engage with Models” activity. The themes identified from the engineering journals, along with examples, are detailed in Table 5.

The themes reflect how students understand and apply engineering concepts in a structured educational setting across different engineering fields. One of the primary themes is the students’ understanding of technology. This theme captures how students define and describe technology as a human-made process or object designed to solve problems. For instance, in discussions related to optical engineering, students explain that turning on a light is considered a technology because it involves processes, parts, and wires designed by humans. Similarly, in green engineering, students perceive technology as “anything made by a person to solve a problem.” This reflects a growing awareness among students that technology is not just about gadgets or machines but rather about the underlying processes and problem-solving nature of human inventions.

Another important theme is the successful recall of engineering concepts, which emphasizes the students’ ability to remember and apply science and engineering concepts. For example, in geotechnical engineering, students demonstrated their understanding of foundational stability by explaining that a skyscraper should be anchored in the bedrock layer, which is the most stable during an earthquake. Similarly, in electrical engineering, students recalled that a power source, wire, light bulb, and battery are necessary to light up an LED. This ability to recall and apply specific engineering knowledge signifies that students are internalizing core concepts and can actively engage with them.

The theme of discipline-specific terminology illustrates students’ use of specialized vocabulary from both science and engineering. This theme captures how students apply the correct terminology when describing scientific phenomena and engineering processes. For instance, in geotechnical engineering, students correctly associate faster-moving water with increased erosion, showing their understanding of the relationship between speed and geological changes. In electrical engineering, they correctly predicted that when one LED in a series circuit burns out, it might act as an open circuit. This proper use of vocabulary reflects their growing mastery of the language of engineering and science, which is crucial for deeper learning and communication within these fields.

In addition to understanding concepts and terminology, students also demonstrated an integration of science and engineering. This theme is reflected in students’ ability to use scientific knowledge to make engineering decisions. For example, in geotechnical engineering, students applied their understanding of erosion to determine the best site for building, considering how water movement affects soil stability. In electrical engineering, students integrated their knowledge of circuits to explain how a series circuit differs from an open circuit in terms of electron flow. This ability to bridge science and engineering shows that students are not only learning isolated facts but are also connecting different domains of knowledge to solve real-world problems.

The theme of gained awareness of how to achieve success captures students’ understanding of the steps required to complete a task successfully. Here, students articulated the processes they would follow to achieve success, demonstrating clarity of thought and the ability to envision outcomes. In geotechnical engineering, students mentioned testing and collaboration as part of the design process, expressing confidence that understanding the task would lead to a successful outcome. Similarly, in electrical engineering, students discussed how improving the clarity of a design could lead to better results. This theme highlights students’ developing metacognitive skills as they reflect on the actions necessary for problem-solving and success.

Finally, the theme of hands-on activities serving a representational role emphasizes how students used prior practical experiences to inform their responses. For example, in geotechnical engineering, students referred to earlier experiments where they tested the stability of skyscrapers on different layers, using this knowledge to predict the outcome of anchoring a building in bedrock. In electrical engineering, they recalled which materials were strong or weak conductors based on prior activities involving metal, paper clips, and other materials. This theme underscores the importance of experiential learning, where hands-on activities enhance students’ understanding and retention of abstract concepts by giving them tangible experiences to draw upon.

This pattern of themes features the multifaceted ways in which students interact with hands-on engineering activities. These insights suggest that students are not only learning the content but also applying critical thinking skills and displaying an appreciation for the problem-solving nature of engineering.

Paired t-tests showed significant increases in self-efficacy scores from pre-test (M = 3.28, SD = 0.85) to post-test (M = 3.75, SD = 0.73), t (19) = −2.39, p = 0.027, 1–β = 0.62, with a medium effect size, d = −0.53; perceptions of engineering knowledge (Basic) scores showed a significant increase from pre-test (M = 3.53, SD = 0.89) to post-test (M = 3.80, SD = 0.73), t (19) = −2.52, p = 0.021, 1–β = 0.67, with a medium effect size, d = −0.56. Additionally, a significant increase was observed in perceptions of engineering knowledge (Green) scores from pre-test (M = 3.12, SD = 1.30) to post-test (M = 3.63, SD = 0.99), t (19) = −3.12, p = 0.006, 1–β = 0.84, with a large effect size, d = −0.70. There were increases in STEM identity scores from pre-test (M = 2.80, SD = 0.94) to post-test (M = 3.19, SD = 0.85), though this difference was not statistically significant, t (19) = −1.85, p = 0.079, 1–β =0.42, with a small effect size, d = −0.41. Detailed statistics, confidence intervals, and effect sizes are provided in Table 9.

The findings from both studies provide valuable insights into how students communicated their understanding of engineering concepts and the EDP, addressing our first research question: How do students communicate their understanding of the engineering concepts from the lessons and the EDP process?

Qualitative analysis revealed students engaged with the EDP both conceptually and practically. The lessons fostered a foundational understanding of engineering, enabling students to approach challenges systematically. Embedding the EDP in culturally relevant, hands-on activities reinforced students’ learning, aligning with prior research highlighting the benefits of experiential, culturally responsive STEM education (Feder et al., 2009; Gay and Howard, 2000; Kaldi et al., 2011; Ladson-Billings, 2022; Weisberg and Newcombe, 2017).

Students also showcased their growing mastery of discipline-specific terminology, a critical indicator of deeper learning and communication within STEM fields. Their ability to articulate concepts like erosion, circuit design, and structural stability reflects the effectiveness of integrating scientific vocabulary into engineering tasks. This mastery highlights the importance of teaching students not just the “how” of engineering but also the language necessary to communicate their understanding effectively.

Moreover, the theme of integrating science and engineering emerged prominently in student responses. Their reflections revealed a capacity to use scientific knowledge to make informed engineering decisions, bridging disciplinary divides. For example, students applied their understanding of erosion to geotechnical engineering tasks and used their knowledge of electrical conductivity to design functional circuits. This integration reflects the intervention’s success in fostering interdisciplinary thinking, a key skill for real-world problem-solving.

An additional insight was students gained awareness of the steps required to achieve success, including collaboration, testing, and iterative improvement. These metacognitive skills, articulated through their reflections, illustrate their ability to plan and execute tasks methodically, reinforcing their understanding of the EDP as a systematic framework.

Finally, students’ expanded understanding of technology, moving beyond gadgets to viewing it as a problem-solving process, signifies a critical shift in their conceptualization of engineering. This broadened perspective aligns with the intervention’s goals of fostering a comprehensive understanding of engineering’s problem-solving nature.

Together, these themes highlight the multifaceted ways students interacted with the EDP, demonstrating that the intervention enhanced their conceptual knowledge and critical thinking and problem-solving skills. These insights further support the efficacy of embedding culturally relevant, hands-on learning into STEM education, particularly in fostering transferable skills and deeper engagement with engineering concepts.

The quantitative analysis explored how repeated exposure to the EDP within these lessons impacted students’ STEM identity, self-efficacy, and perceptions of engineering knowledge. While the singular concentrated exposure in the Summer STEM Academy reflects a statistically significant decrease in STEM identity, the effect size (r = 0.22) indicates the practical importance may not be substantial, and the 95% confidence interval [−1.80, 1.38] includes 0 suggesting some uncertainty about the precision of the observed difference.

In contrast, students who attended Saturday STEM Academies to clear absences exhibited slower but measurable increases in STEM identity, suggesting repeated exposure to the EDP gradually enhanced their identification with STEM over time, supporting the role of consistent engagement in building long-term interest (Bell et al., 2009). However, STEM identity plateaued after multiple exposures. This plateau aligns with Staus et al. (2021) who observed rapid engagement with STEM subjects slowed as students become more familiar with the concepts. Without new and progressively challenging content, repeated exposure may not further strengthen STEM identity (Bethke Wendell and Rogers, 2013).

Unlike STEM identity, self-efficacy gains were slower and required multiple exposures to produce significant improvements. This finding aligns with Bandura’s (1997) theory of self-efficacy, which emphasizes mastery experiences—repeated, successful engagements with tasks—are essential for building confidence. In the Summer STEM Academy, students did not have significant self-efficacy growth, while students in the Saturday STEM Academies exhibited gradual improvements. This difference may suggest students in the Summer STEM Academy who voluntarily participated had insufficient time to build their self-efficacy or may have already entered with higher levels of self-efficacy. For the Saturday Academy students, continuous engagement was crucial for building confidence, supporting the idea that sustained exposure is particularly beneficial for reluctant learners (Schweingruber and Fenichel, 2010). This suggests the need for providing students with repeated, hands-on experiences that allow them to internalize and apply engineering principles over time (Kosmas et al., 2019).

For Study 2, basic and specific engineering knowledge showed significant gains after repeated lessons, even in topics unrelated to students’ previous exposures, suggesting students developed a generalized understanding of engineering principles. The cumulative learning effect supports the idea that repeated engagement with core concepts allows students to transfer knowledge across domains (Bell et al., 2009). Contrary to our hypothesis, prior exposure to the EDP framework did influence students’ perceptions of their knowledge of specific engineering types. Students with more prior exposure reported higher knowledge even when the lesson content was new. This result suggests cumulative engagement with the EDP helped students develop a transferable understanding of engineering concepts, emphasizing the importance of repeated exposure in ISL settings.

The contrasting results between the Summer and Saturday STEM Academies offer insight into how different learner profiles respond to ISL interventions. Summer STEM Academy students, part of an African American leadership program, did not show significant gains after a single exposure to culturally relevant, immersive lessons. However, the Saturday STEM Academy students who had to be encouraged to attend exhibited slower but positive growth across all constructs measured. These findings emphasize the need to adapt ISL programs so that they address individual student motivation levels and are available to them more than one time (Schweingruber and Fenichel, 2010). However, caution is necessary in attributing long-term changes to these interventions alone, as noted by Staus et al. (2021). Future research is needed to assess the persistence of these impacts.

Reinforcing STEM identity is crucial to strengthening students’ academic trajectories, especially during transitions like middle to high school, particularly for underrepresented groups (Archer et al., 2010; Wang and Degol, 2013). Several implications for practice emerge to enhance future STEM lessons. Embedding the EDP in culturally relevant, hands-on activities suggests incorporating real-world problems that engage students and help them understand and apply engineering concepts. Additionally, directly involving students in identifying topics they find relatable and relevant at the end of their study period could help guide the development of future lessons. This approach would ensure that topics resonate with students’ interests and experiences, fostering greater engagement. Programs targeting students with varying motivation levels should adapt their approaches to provide learners with consistent, long-term STEM exposure to foster gains in STEM identity and self-efficacy.

While short-term experiences may spark initial interest, sustained engagement with progressively challenging content is essential for continued growth in STEM identity and self-efficacy. Programs should focus on scaffolding lessons to build on prior knowledge and integrate various topics over time, promoting cumulative learning. This allows students to deepen their understanding and transfer engineering principles across different fields. Sustained, hands-on experiences are especially beneficial for reluctant learners, enabling them to internalize and apply concepts effectively over time (Kosmas et al., 2019).

Balancing experiential learning with reflective practices is necessary. While hands-on activities engage students, dedicating time for structured reflection deepens understanding (Cheryan et al., 2017; Schweingruber et al., 2014). STEM lessons could benefit from guided journaling or group discussions to encourage students to articulate their learning in more detail.

Ultimately, incorporating culturally relevant, hands-on experiences, sustained engagement, and reflection in STEM education equips students with both the skills and confidence to pursue engineering and related fields. Such practices ensure students—especially those with varied motivations—develop a robust STEM identity and are prepared for future academic and professional challenges.

Several limitations must be acknowledged. First, the outdoor data collection during the Optical Engineering lessons may have limited students’ written reflections, as they focused more on gathering temperature data than on deeper conceptual reflection. Similarly, the informal STEM Academy settings, while beneficial for engagement and participation, may have led to concise journal responses, suggesting a need for more structured reflection time in future interventions.

Additionally, there was an uneven emphasis on different engineering disciplines, particularly geotechnical engineering, which received more attention in the qualitative analysis due to the lack of quantitative data. This imbalance may have affected the depth of insights across Saturday STEM Academies.

The voluntary nature of student participation also contributed to the small sample size and inconsistent data collection, limiting our ability to generalize the results for larger groups and conduct a true longitudinal analysis. However, this variability in attendance provided an opportunity to assess whether different levels of exposure to the EDP had varying effects on students’ STEM identity, self-efficacy, and perceptions of engineering knowledge. Future studies could address these limitations by incorporating a more varied and larger sample size, as well as measuring student attitudes 6 months after exposure to the EDP to assess whether their attitudes toward STEM and Engineering are retained long-term. These adaptations would enhance the generalizability of the results and offer a more comprehensive understanding of the impact of the EDP across different contexts.

Further, differences in student engagement between the two cohorts likely influenced the magnitude of construct gains. Students in the Summer STEM Academy (who voluntarily participated) may have had higher initial motivation levels compared to those in the Saturday STEM Academy (who were incentivized to attend to clear absences). This variation in engagement could have also contributed to differences in the results. Additionally, the reliance on self-reported measures may introduce bias, as students’ perceptions may not fully reflect their actual abilities or knowledge.

Finally, while we observed short-term gains in STEM identity, self-efficacy, and perceptions of engineering knowledge, this study does not include long-term data to assess whether these gains are sustained over time. As Staus et al. (2021) argued, understanding the long-term effects of informal science education requires longitudinal approaches. Without follow-up data, we cannot determine whether the improvements in identity, self-efficacy, and knowledge persist after the program ends.