Scientific models are tools for explanation and prediction. A complete scientific explanation should “explain observed relationships between variables and describe the mechanisms that support cause and effect inferences about them” (National Research, 2012). Thus, to support student explanations, a scientific model of a phenomenon should include important components (“variables”), their interactions (“relationships”), and define the mechanisms involved. This approach is similar to the Structure–Behavior–Function model (SBF; Hmelo-Silver et al., 2007). However, the focus on interactions in our approach allows for more abstract relationships that do not rely on one particular behavior or function.

When modeling an ecosystem, these correspond to the organisms in the ecosystem (animals, plants, insects, fungi, etc.), how these organisms interact with each other and the environment (predator, prey, producer, decomposer, etc.), and the involved processes (abiotic, biotic, etc.). Professional biologists use this information to measure the biodiversity of an ecosystem in terms of species richness, evenness, and divergence (Holling, 1973; Gunderson, 2000; Mason et al., 2005).

In this work, we characterize variation in students’ models by examining the number of organisms present, the various types of organisms present, the number of interactions between organisms that students have identified, and the diversity of these interaction types. We also look at how these features are distributed within a model. These measures are used to understand the complexity of a student model. This approach is similar to prior research understanding student models, particularly work from Jordan et al. (2017). Our approach to analyzing student models primarily differs from the focus on components and interactions rather than using the SBF framework. This change allows us to automatically characterize the different pieces of models, which can be used to support real-time analysis and feedback in the future.

Interestingly, understanding the complexity of an ecosystem has been shown to support students to develop empathy and other affective stances toward nature (Kellert and Wilson, 1995). Student understanding the flow of matter and energy through ecosystems has also been shown to vary strongly across cultural boundaries (Bang et al., 2007), providing further motivation for supporting equitable opportunities to learn scientific modeling.

Constructing scientific models is part of the “inquiry” tradition in science education, where students learn scientific concepts through hands-on “doing” (Kolodner et al., 2003). Understanding what students are doing at a fine-grained level can provide teachers with useful insights into learning processes, as well as provide teachers with feedback as to where and when students need additional assistance. Toward this end, several scholars have developed frameworks characterizing effective student modeling practices (Schwarz et al., 2009; Bryce et al., 2016). Schwarz et al. (2009) identify a series of the following seven practices: (1) identifying the anchoring phenomena to be modeled, (2) constructing a model, (3) testing the model, (4) evaluating a model, (5) comparing the model against other ideas, (6) revising the model, and (7) using the model to predict or explain phenomena. Bryce et al. (2016) identify a similar set of practices as important to support student learning during modeling, namely, (1) observation (paralleling the anchoring phenomena), (2) model construction, (3) model use, (4) model evaluation, and (5) model revision. Their research suggests that supporting students to engage in these practices can lead to positive learning outcomes (Schwarz et al., 2009).

Here, we focus on a subset of these practices—constructing, evaluating, revising, and using models—incorporating them into our analysis framework (Cukurova et al., 2016). We focus on these four practices as they are directly supported through the EcoSurvey interface and can be readily observed and tracked in the usage log. In addition to these four practices, we examine the degree to which students engaged in iterative design of their models. Iteration occurs when students cycle between the other four modeling practices, where the four practices correspond directly to individual actions in the EcoSurvey interface, such as adding an organism or relationship (construction), editing an organism or relationship (revision), or generating a graph of the entire ecosystem to support explanations (using). Iteration is an important modeling practice that is used to both expand the scope of a model and to improve its accuracy (Homer, 1996; Bryce et al., 2016). Learning analytic techniques are used to identify the degree to which students used these practices and to examine variations in student modeling practices. While these usage log analysis methods are an excellent passive way to collect data on student practices (Pelánek et al., 2016), it is important to note that these methods do not capture information about how students are reasoning with their models. Exploring student reasoning with models and how they generate explanation using models is beyond the scope of this study and would require deep exploration of students’ cognitive processes using think-alouds, cognitive interviews or other learning and cognitive sciences research methods (e.g., Schauble et al., 1995).

Understanding student activity is an active area of Learning Analytics research. There is a large body of work focused on detecting students’ skill acquisition using digital tools. One example of this is a study focused on young students’ ability to make numeric and fractional estimates in a number line game. In this study, they found significant gains in student accuracy over time.

One important area of research around understanding student activity focuses on detecting variance between students as a predictor of future activity. One example is Park et al. (2017), understanding student use of online classroom resources using features drawn from student clickstreams. These features focus on generalizing beyond the course content, focusing on simple frequency measures (e.g., number of clicks per day) and abstractions of how the content accessed relates to the course schedule, determining whether the content being accessed is being “previewed” or “reviewed.” The team used these metrics to categorize whether students’ engagement with online resources increased, decreased, or stayed the same, creating three different categories of students. Overall, they found that students who specifically increased their engagement with the tool had a better success rate in the class.

While these aggregate features are useful in understanding activity, another approach to understanding this activity and variance is the use of sequence modeling (Xing et al., 2010). This approach focuses on a fine-grained distinction of different activities and analyzes patterns in how activities lead into one another. This approach parallels that used by d’Aquin and Jay (2013), where they used sequential pattern mining to study student course enrollment patterns.

These innovations inspire my approach to analyzing student engagement with modeling practices. By adapting the approaches of aggregate activity and sequence analysis to scientific modeling in the classroom, I develop new insights into how students participate in this crucial activity.

Scaffolding, or the use of external supports to help a person accomplish a task, is a common approach in the learning sciences. Vygotsky (1980) discussed such supports when describing the zone of proximal development for apprentices in trade fields. Scaffolds are natural extensions of this idea, tools that support a learner in this process, providing guidance or removing levels of complexity from the problem. These scaffolds can then be removed from the application, setting, or activity when the learner is ready to move on.

This scaffolding approach has been adopted in software-supported learning scenarios (e.g., Quintana et al., 2004; Gadiraju and Dietze, 2017). One particular line of work has focused on digital applications that support scientific inquiry. Quintana et al. (2004) describe the following three phases of scaffolding scientific inquiry: (1)

Characterizing the cognitive tasks, social interactions, tools, and artifacts that constitute the scientific practices in which learners are engaged.

Our work naturally builds on these guidelines. By focusing the design of EcoSurvey around the practices outlined in learning science literature on modeling in the classroom, we naturally connect to the real practices of the task. Our analytics are designed to automatically measure how successful students are at engaging with these practices and the points at which students run into difficulty. We have used this feedback in design, and our results show that students are using the revised modeling tool to create more complete models of their local ecosystem.

Ms. Smith instructs students to map the ecosystem within a selected site on their school grounds or in the local area, taking pictures and making field notes on the organisms and interactions between organisms that they observe. Maria’s group makes observations along the creek that runs next to the school. They find a lady beetle, a honey locust tree, some mushrooms, a gray squirrel, and a few other organisms. Using their smartphones, they take pictures of these organisms and upload them to EcoSurvey, creating a “card” for each organism while out in the field. Each card automatically captures information about the date, time, and location of the observation being recorded. Cards also include a “relations” field to capture interactions between organisms and information about the organism’s role in the ecosystem. Students begin entering this information as they observe it in the field, and then continue to augment this information back in the classroom through additional research. In Figure 1, we see Maria’s lady beetle card under construction. While in the field, she created the card, uploaded a picture, and added details about interactions they saw. At the same time, her team members are also creating cards for other organisms they are observing.

As students create cards, their organisms are added to a shared class “survey.” The survey view shows all of the organism cards and their detailed information, ordered by how recently they were edited. Maria can see that her classmates have created many cards, including a Blue Jay card (Figure 2).

Ms. Smith organizes the student groups into pairs and asks each group to review the other’s cards for correctness and completeness. Maria’s group is paired with Group 2, who completed several cards. Andre, a member of Group 2, asks Maria to first review the blue jay card he created. Maria uses the search feature of the survey view to quickly find the blue jay among the cards. She notices that this card is missing many details, including interactions with other organisms.

Maria recommends that Group 2 do further research into how the blue jay contributes to the local ecosystem. She also takes the chance to update her group’s honey locust card. She discovered that blue jays nest in honey locust trees during her earlier research. She did not realize that their school ecosystem included blue jays until she reviewed the work of her classmates, as her group did not see one. Once Maria has completed editing her group’s cards, she continues her review of Group 2’s cards. She uses the group select function to view only the cards created by members of Group 2.

Group 2 notices that two people in Maria’s group created duplicate lady beetle cards. Maria decides to add her lady beetle information to the other card, since it is more detailed, and uses the delete function to remove her lady beetle card from the model.

In reviewing Group 2 cards, Maria sees a card for geese, but notices that the group did not add a predatory relationship to grass, even though she observed geese eat the grass on the soccer field. She uses the search functionality and discovers that no one in class created a card to document grass as an observed organism. Maria adds a new card for grass and includes a predatory-prey relationship with geese. By cycling back through earlier modeling practices (creating new cards), Maria is iteratively improving the class model to be more complete and accurate.

Once the class has created a robust model of their local ecosystem, students use this model to construct arguments for choosing a particular tree to plant. Maria presses the “create relation graph” button, which generates the graph representation of the model and exports it to a digital graphing tool (Figure 3). Maria and her team study the resulting diagram that enables them to visualize the relationships (links) between all the organisms (nodes) they have cataloged. It is clear from looking at her graph that the English Oak trees are an important keystone species in their site, involved in a large number of relationships with a wide variety of organisms. The geospatial locations in the observational data indicate that there are only two English Oak trees located in their site; Maria and her group recommend planting an additional tree of this type.

Maria’s scenario illustrates how EcoSurvey supports students to engage in the practices of creating, evaluating, revising, iterating, and using models. To use a learning analytics approach to study modeling practices, we must map specific actions, or sequences of actions, taken in the EcoSurvey interface to specific modeling practices. Table 1 describes the mapping between modeling practices and specific EcoSurvey interface actions that we use in our analyses. As students interact with EcoSurvey, the system captures and logs each of the actions shown in Table 1. Each log entry includes the time, user, survey, and action type.

Consistent with a design-based research approach, we are iteratively improving the design of the EcoSurvey tool and the supporting curriculum after each field deployment. Classroom observations, feedback from users, and analysis of the usage patterns from the first version drove several important changes.

The foremost change is the redesign of the survey view, incorporating the graph representation of the model into the students’ main workflow as seen in Figure 4. This view presents the model as a collection of components (organisms) and interactions (relationships). This development grew from results from our first deployment that students failed to engage with relationships for many organisms in their models, and many classrooms showed limited engagement with exporting their models to the graph view. Therefore, we designed the graph layout to emphasize the relationships between organisms, naturally promoting the task of adding relationships to disconnected cards.

The second change we made was to the types of relationships that could be added as seen in Figure 5. In the first version, the relationship field was open and would accept any response. This led to a wide variety of responses, many of which did not accurately reflect possible relationships (e.g., “brown trout” is not a relationship type). While we normalized the relationships for analysis (as discussed below), this still left a large number of “unknown” relationships. By using a closed list of relationship options based on language from the science standards, version 2 of EcoSurvey scaffolds student model development and scientific understanding by driving them to consider how their intuitive representation of the relationship maps to the terms used by scholars in the field.

The work presented here builds on two deployments, one of each version of EcoSurvey. Both deployments took place in the same district and with the same professional development routines. However, the data we have analyzed for each deployment used a separate cadre of teachers, which allows us to avoid effects due to previous experience with using EcoSurvey in the classroom.

For our first deployment, EcoSurvey usage log data were collected across 10 high school classrooms during Fall 2015. A total of 9 models were created, which included 586 organism cards and 545 interactions. Our second deployment in Fall 2016 featured final models from students in 35 classes across 11 teachers. These models included 4,136 organism cards and 4,701 interactions.

The deployment of the first version of EcoSurvey also incorporated activity logging. From the sample, we recorded actions for 204 students generating 3,160 action logs, whereas 58 students did not record any activity. All classrooms in both samples followed a 3:1 device deployment where three students used one laptop together; thus it is not surprising that there are students with no recorded activity.

Our first research question examines variation within student models, focusing specifically on the richness of students’ models in terms of the number of organisms and their relationships. We analyze the relative number of organisms and interactions within each class survey. We also look at the balance of interactions per organism by evaluating both the average number of interactions per organism and variance in the distribution of interactions. Examining variance allows us to distinguish different patterns in the assignment of interactions to organisms. Some classes may create models where most organisms have a similar number of interactions, whereas other classes may create models where only a few organisms have been assigned many interactions.

We also analyze the distribution of relationship types using evenness. This measure considers how each type of relationship is represented within the survey. We calculated evenness using the Shannon index, the same formula for species evenness in the study of ecosystems (Shannon and Weaver, 1998). The Shannon index gives an evenness score from zero to one. A survey with an equal number of relationships of each type would have a perfect evenness score of one. Conversely, a survey with many predator–prey relationships and few others would have a low program type evenness score. The Shannon index is calculated using the following formula: (1a)J′=H′H′MAX, where (1b)H′=−∑i=1NPiln(Pi), where p_i = proportion of relationships of type i in the surveys and (1c)H′MAX=−∑i=1Nln(Si), where S_i = total number of relationships in the surveys.

Our second research question seeks to understand how design changes in digital modeling tools can have an impact on students’ models. These measures are important for the iterative process of our design-based research approach, providing evidence of what impact the design changes have on students’ models.

To evaluate the impacts of design, we aim to compare directly across deployments of different versions of EcoSurvey. We run the same statistical comparisons for each version and compare across conditions. In cases where direct comparison of means and variance is possible, we use a standard Student’s t-test (Student, 1908) to determine significance.

Our last research question examines variation in student modeling practices, focusing on action variety, frequency, and iteration. Action variety refers to the range of actions a student performed. For example, some students may have only created and edited cards, whereas others may have used the full range of EcoSurvey actions. Frequency refers to the total number of actions completed by an individual student and the number of usage sessions they engaged in. Sessions are defined by a series of actions from a single user without a large break in activity (greater than 2 h). Defining a session using a 2 h gap allows for any student activity within a long class period to occur within one session; several of our classrooms employ 1.5 h block periods.

To characterize iteration practices, we look for evidence of design cycles within the log information. Design cycles can be recognized when students engage in multiple sequences of construct–revise–use practices. This focus on a sequence of practices is consistent with Schwarz et al. (2009), which characterized modeling practices as a series of steps. By extension, a design cycle consists of returning to a previous modeling step after moving on in the sequence (e.g., creating a new card after editing a different card). We counted the number of cycles as a measure of iteration.

Combined, these three metrics—action variety, frequency, and iteration—yield an eight feature vector for each student consisting of total number of EcoSurvey actions, total number of create actions, total number of evaluate actions, total number of revise actions, total number of use actions, total number of EcoSurvey action types taken, number of sessions, and number of iterations. We combined the feature vectors for students with the same teacher, and performed a Kruskal–Wallis H test (Kruskal and Wallis, 1952) for each feature to determine differences between teachers. A Kruskal–Wallis H test is a non-parametric adaptation of an ANOVA to compare samples of different sizes, as we have in our groups. We further explored these differences using Tukey’s HSD test (Tukey, 1949) to test the significance of pairwise differences between teachers.

This understanding of modeling practice allows us to characterize the variation in student activity in EcoSurvey. To expand on this characterization, we examine the degree to which we can use sequences of student modeling actions to predict that student’s teacher. We plan to use this prediction in a support system for students and teachers embedded within the tool.

For this prediction task, we use the previously described features of variety, frequency, and iteration as well as automatically extracted sequence patterns. In our work, a sequence pattern consists of a series of EcoSurvey actions (e.g., “New Card,” “Edit,” “Generate Graph”) embedded within a student’s complete action log. To extract sequence patterns, we used the Colibri Core (van Gompel and van den Bosch, 2016) software package. This software package, originally designed for natural language processing tasks, treats every action as a token and determines the frequency of consecutive token sequences (n-grams) from student usage logs. These token sequences can include wildcard actions (skip-grams). For instance, the software will extract the sequence “New Card,” “Edit,” “Generate Graph” as either an n-gram or as the skip-gram “New Card,” {*}, “Generate Graph.” This skip-gram will capture similar sequence patterns, where one action occurs between New Card and Generate Graph actions. This yielded 2,893 unique sequence patterns, which occurred at least three times, across all student usage logs. Once we extracted these sequence patterns, we used them as a new series of features to augment each student’s existing feature vector.

To understand which features that characterize a student’s modeling actions are most predictive of his or her teacher, we input subsets of each student’s feature vector into four Naive Bayes classifiers using Weka (Hall et al., 2009). The first classifier used the eight features related to variety, frequency, and iteration of actions. The second classifier used the full set of sequence pattern extracted by Colibri Core for each student. The third classifier implemented a best-first search (Hall and Holmes, 2003), which automatically reduced the full set of sequence patterns to the eighteen most predictive features. The last classifier combines the eight varieties, frequency, and iteration features with the eighteen most predictive sequence patterns. Each test was run using 10-fold cross validation.