Winne and Hadwin’s (1998) conditions, operations, products, evaluations, and standards (COPES) model of SRL explains the dynamic process students engage in while they regulate their learning during a specific task. The model describes a cyclical sequence of phases, starting with task definition and contextual analysis, where students evaluate and orientate themselves in the learning environment and identify task-relevant influencing factors and information (Winne, 2017; Winne and Hadwin, 1998). Relevant factors may include internal influences, such as task-related prior knowledge, and external conditions, such as the technical environment or the specified timeframe for the task. The subsequent phase involves goal setting and strategic planning. In this phase, learners formulate standards for the products of their task performance, according to which they plan their approach (Winne, 2017; Winne and Baker, 2013). The third phase of the COPES model, implementation, describes learners’ actual use of the chosen strategies. In this phase, students consider feedback from diverse sources, such as the learning environment, but also through self-reflection, such as satisfaction with their own performance (Winne, 2017). Following the completion of the task, students evaluate their approach and the effectiveness of their strategy use in a reflective phase (Winne and Hadwin, 1998). The COPES model highlights the iterative nature of self-regulation, as students continuously adapt and refine their approaches in response to feedback and emerging task demands. This iterative process of SRL leads to improved learning outcomes and problem-solving skills over time and is included in most SRL models (Panadero, 2017).

Within this complex SRL process, various metacognitive, cognitive, and motivational strategies come into play (Boekaerts, 1996). One of the most important components of SRL is metacognition (Gascoine et al., 2017). Nelson and Narens (1994) have described metacognition as the interplay of monitoring and control. Monitoring behavior involves the active awareness and assessment of one’s cognitive processes and ongoing tasks. Learners continuously evaluate their understanding, progress, and the strategies they apply during learning or problem-solving activities. Control refers to the strategies and actions taken based on the outcome of the monitoring process. Once individuals become aware of their cognitive states by monitoring, they can implement various control strategies. Control strategies can include adjusting the learning approach or changing study techniques. In essence, control mechanisms are the conscious, goal-directed efforts learners employ to enhance their performance. Learners, for instance, monitor discrepancies between their aims and their actions and use metacognitive control to adapt their behavior to their standards if necessary (Winne and Nesbit, 2009). These theoretical models (Winne and Hadwin, 1998; Winne and Nesbit, 2009) emphasize the dynamic interplay between monitoring and control.

Accordingly, SRL is not necessarily a linear process with a clear order, but rather a repetitive interplay between monitoring and control. Therefore, SRL can be seen as a sequence of events that occur during task performance, which offers a new perspective on metacognition and the whole SRL process (Azevedo, 2014). Based on this assumption, SRL can be interpreted as a successive sequence of states, with a clearly defined start and end point of each state (Winne and Baker, 2013). However, SRL and the occurrence of such events is a partially invisible process. Not all of the SRL activities of learners, in particular cognitive and metacognitive events, are directly visible. The covert nature of SRL can be described as a learner’s dynamic SRL approach consisting of a sequence of observable behavioral events that indicate not directly observable states (van Laer and Elen, 2018). Accordingly, this conceptualization of SRL is based on the assumption that meaningful insights into learners’ latent behavioral states can be gained through a careful examination of ordered observable behavioral events or sequences (Molenaar and Järvelä, 2014). For example, when a learner is rereading the task instruction or the own written product, this could be an observable action or event, indicating metacognitive monitoring, a state difficult to identify without such indicators. However, rereading the own written product can also be interpreted as an indicator for evaluating, which demonstrates that observable actions cannot always be objectively or uniquely assigned to one covert SRL state and highlights the need to take this into account when assessing SRL.

Based on the common agreement to define SRL as a process with different phases (Panadero, 2017), the exclusive use of offline measurements has often been criticized in recent SRL research (Veenman and van Cleef, 2019). The retrospectively reported use of SRL strategies in self-report questionnaires is usually inaccurate and prone to socially desirable responses and to over- or underestimation (Panadero et al., 2016; Roth et al., 2016). Thus, the general validity of self-reports, especially for children, has frequently been questioned (Veenman and van Cleef, 2019; Whitebread et al., 2009). Instead of assessing what students recall or believe to do during learning, online measures attempt to assess what learners actually do while working on a task. Therefore, these methods are considered to be more objective, precise, and valid than offline measures (Greene and Azevedo, 2009; van Halem et al., 2020). However, as event-based measurements are a relatively recent development in SRL research, the psychometric properties of online measures are not yet as well documented as for offline measurements, such as self-reports (McCoy, 2019). Nevertheless, studies show high correlations between online methods and between SRL online assessment and performance, which indicates the validity of these methods (Veenman and van Cleef, 2019).

Despite the major concerns regarding validity, offline methods have been the most common assessment method in SRL research for a long time, based on their time-efficient administration. Data from SRL self-report questionnaires and adult ratings can be collected and analyzed easily in large samples (Chen et al., 2015; Gascoine et al., 2017; McCoy, 2019; Veenman, 2011). In contrast, online measurements, such as observations, interviews, or think-aloud protocols, lead to an extensive database of verbal and non-verbal behaviors. The data collection, transcription, and analysis of this data are time consuming (Vandevelde et al., 2015; Veenman and van Cleef, 2019) and need a lot of resources. In addition, some of these methods are intended for laboratory research and are impractical for applied school-based research (Marulis et al., 2016; McClelland and Cameron, 2012). Furthermore, previous research has discussed the potential reactivity of the different assessment methods of metacognition, drawing attention to the relevance of task and person characteristics, and cues (Double and Birney, 2019). A meta-analysis suggests that general think-aloud assessment is non-reactive, but that asking participants to verbalize or explain further details in addition, such as reasons for their behavior or thoughts, is positively reactive (Fox et al., 2011). Nonetheless, both methods are shown to increase task performance time.

When comparing and evaluating the validity of different instruments and considering the efficient analysis of SRL data, the granularity of the SRL assessment seems relevant (Bannert et al., 2014; Fan et al., 2022; Molenaar, 2014; Winne, 2010). Offline measures, such as self-reports, assess SRL as a global measure of how an individual usually uses SRL strategies in learning by aggregating learning approaches across learning contexts, episodes, and tasks (Winne and Perry, 2000). In contrast, within online measurements, granularity can vary substantially (Azevedo, 2014; Zhou et al., 2010). Greene and Azevedo (2009) describe the different types of granularity with the help of levels. For example, in think-aloud protocols and log data, students’ specific behaviors and mental processes are recorded, resulting in microlevel data that can also be encoded into macrolevel categories of SRL (Greene and Azevedo, 2009; Siadaty et al., 2016). Coding of log data, for example, can range from milliseconds and smallest mouse movements, which are linked to specific SRL strategies, to the general appearance of SRL behavior, by, for example, clicking on a URL link as a broad indicator for SRL (Winne, 2017).

Moreover, assessing SRL in young learners poses a major challenge (Whitebread et al., 2009). Overall, many instruments have been developed for older students. Most offline methods, such as strategy questionnaires, are highly dependent on student’s literacy skills (Perry and VandeKamp, 2000; Whitebread et al., 2009). Being less dependent on language, some online methods, such as observations, are already suitable for preschool age (Veenman and van Cleef, 2019; Whitebread et al., 2009; Winne and Perry, 2000). In addition, think-aloud protocols have also proven to be a practical method for a more objective assessment of SRL strategy use, with young learners (Veenman, 2011). Despite the recent developments in SRL assessment methods, there is still a lack of research on young learners.

Based on the paradigm shift, defining SRL as a series of events resulting in a complex and dynamic process, which develops over time (Molenaar, 2014), online measures seem to be the more valid option to capture SRL (Veenman and van Cleef, 2019). Nevertheless, self-reports are still frequently used because they are the most efficient option. Even if online SRL assessment methods are repeatedly discussed as being more valid, more sensitive to the process nature of SRL, and allow a finer-grained analysis, there is still a lack of instruments that can be administered and analyzed efficiently and in flexible contexts especially with young students. With the increasing technical possibilities, computerized online assessment methods have received growing interest in SRL research (van Laer and Elen, 2018; Winne and Nesbit, 2009). In recent research, various forms of trace data are used to assess SRL, and consequently offer the potential to provide learners directly with feedback on their learning approach. Thus, learning analytics can serve to evaluate and to promote the SRL process.

The actual application areas of log data in SRL research are very diverse. For example, log data are used to assess SRL in reading and writing tasks (e.g., Fan et al., 2022; Rakovic et al., 2022), in the context of hypermedia use (e.g., Kinnebrew et al., 2013) and in broad digital learning environments (e.g., Maldonado-Mahauad et al., 2018; Saint et al., 2020b). Previous research used log data, for instance, to investigate different learning strategies and cluster different SRL approaches of students (e.g., Maldonado-Mahauad et al., 2018; Malmberg et al., 2014; Saint et al., 2020b). Other studies investigated the relation of SRL strategy use assessed with log data and the actual learning outcome (e.g., Maldonado-Mahauad et al., 2018; Rakovic et al., 2022). Using log data, several studies have already investigated the impact of interventions, such as prompting (e.g., Bannert and Reimann, 2012; Siadaty et al., 2016).

Nevertheless, most of the aforementioned studies investigated SRL among secondary school students (e.g., Kinnebrew et al., 2013; Munshi et al., 2018), university students (e.g., Bannert and Reimann, 2012; Fan et al., 2022; Saint et al., 2020a; Saint et al., 2020b), or adults (e.g., Siadaty et al., 2016), whereas only a few studies have examined the SRL of young learners with the help of log data so far (e.g., Malmberg et al., 2014; Molenaar, 2014). A recent systematic review by Saint et al. (2022) indicated that merely 21% of the reviewed studies using online assessment focused on K–12 students, with only some of those investigating SRL in primary school students or younger children. So far, multimedia learning environments, such as gStudy (Winne et al., 2006) and Betty’s Brain (Biswas et al., 2016) have been used to investigate upper primary school students’ learning processes with different content focus.

Various methods, such as process mining (e.g., Bannert et al., 2014; Saint et al., 2020b), transition graphs (e.g., Malmberg et al., 2014; Saint et al., 2020a; Siadaty et al., 2016), cluster analysis (e.g., Maldonado-Mahauad et al., 2018), sequence analysis (e.g., Kinnebrew et al., 2013), and network analysis (e.g., Paquette et al., 2021; Saint et al., 2020a), are used to analyze log data to detect SRL. To investigate the methodological potential of log data, only a few studies triangulated log data with different online measurements, such as eye tracking, to analyze the particular value of the various methods (e.g., Fan et al., 2022). Nevertheless, some studies have highlighted the potential of combining log data with other online or offline SRL instruments, such as think-aloud or self-reports (e.g., Fan et al., 2023; Jansen et al., 2020).

Despite progress in log data research, the structure and preprocessing of raw log data for such analyses are challenging, often not described in detail, and justification for methodological decisions is often neglected (van Laer and Elen, 2018). In the first step, indicators in the form of log events have to be defined as meaningful actions (Zhou et al., 2010). In this context, distinctions from other related constructs, as well as the theoretical explanations, must be taken into account (Winne and Baker, 2013). Generating a sequence of meaningful actions from the raw log data so that each action or sequence of actions corresponds to a strategy used by a learner can be challenging. Non-strategic or irrelevant actions act as noise in the action stream and can lead to misinterpretations. For example, non-strategic actions might be deemed irrelevant, but they can also indicate latent relevant processes in the context of the complex SRL process. In addition, not all actions are unambiguous, which can lead to misinterpretations or subjective dependency of the extracted SRL behavior. Researchers often rely on statistical methods for the temporal and sequential analysis of the log data, but fail to argue theoretically why the sequences identified with the help of sequential analysis are meaningful and explanatory (van Laer and Elen, 2018; Zhou et al., 2010). This leads to a lack of transparency in previous SRL research with log data (van Laer and Elen, 2018) and substantial differences in the quality and type of data, the granularity of SRL is measured and the actual operationalization of SRL (Azevedo, 2014). Overall, the numerous unique learning environments, the lack of transparency in preprocessing log data, and insufficient theoretical foundation make it difficult to replicate the studies and transfer methods and operationalizations.

As a reaction, some researchers have started to develop frameworks to guide log data analysis (Saint et al., 2020b; Siadaty et al., 2016; van Laer and Elen, 2018). Three main steps can be distinguished in the work with log data: 1. preprocessing, 2. characterization and operationalization, and 3. analysis. Within these general steps, the recently developed frameworks and protocols (Saint et al., 2020b; Siadaty et al., 2016; van Laer and Elen, 2018) highlight the relevance of theoretical foundation, transparent description, and the preprocessing of the gathered data, followed by defining possible log events in libraries and transforming the data based on defined SRL actions. It is possible to carry out analyses only after having performed these important yet often insufficiently described steps. In addition to the vast potential of the variety of analyses, special attention should, therefore, be paid to data quality and the theory-driven processing of the mass of log data that is easily accessible.

In the following, an analytical approach for the reconstruction of deterministic actions and states is briefly described. The strategy aims to make the definition of so-called low-level features explicit and transparent. Low-level features are understood as the elementary pieces of evidence extracted from the log events that either indicate an interpreted action (i.e., traceable behavior that occurs at a particular point in time) or a state (i.e., a temporary section of traceable behavior that starts and ends at identifiable time points in the log event data). Focusing on single events separately is not necessarily sufficient to extract these low-level features and unlock the full potential of information contained in the log event data. To effectively address the context in which events occur (referred to as contextual dependency, see Kroehne and Goldhammer, in press), algorithms, such as FSMs, can be utilized. FSMs are a formal way to define algorithms, which can be used to analyze log data by processing them event by event. These FSM algorithms defined a finite set of theoretically described and observed log events, resulting in states and triggers for transitions (Kroehne and Goldhammer, 2018).

To use FSMs, the log data are structured as individual events, and each event combines several information. First of all, a timestamp indicates when the event occurred. Moreover, the log data include a person identifier, indicating the individual participant, and a reference to the element of the computer-based instrument; for example, the task name. In addition, different event types are recorded and defined based on the software used to computerize the assessment content. Beyond that, the log data may include additional event-specific data. Which event-specific data are available depends on the event type. For example, events of type “button click” directly provide the button’s name as an event-specific attribute. Events, such as “button click,” may be interpreted directly, depending, for example, on the name of the clicked button. For instance, a click on a button to reset a task to the initial condition would have the direct interpretation of an action as “Reset.” An action only has a time stamp when it happened but no time duration, although it would be possible to differentiate mouse or touch down and up events. In addition to actions, observed states can be extracted from log data as low-level features. Clicking on button “A” could, for example, open a pop-up window, which is visible until it is closed again via a click on button “B.” The two events of the type “button click” would identify a particular state, “State 1,” with a duration: the time between the two events. In contrast to the previous example of the “Reset” button, not only the click on the button “A” does have a certain interpretation but also the time between these two events (“button click” on “A” and “B”) do have certain interpretations, which represents a certain state as low-level feature.

As the arrows show in Figure 1, differentiating observed states is possible by identifying the transitions between these states with the available log events. For that purpose, events are distinguished by event type (e.g., “button click”), the provided element (e.g., the name of a task), and event-specific attributes (e.g., the name of a clicked button, provided by all events of type “button click”). Since log events typically have time stamps, the decomposition shown in Figure 1 automatically provides time measures for each state visit. The duration time for each occurrence of “State 1” can be calculated by determining the time interval between the timestamps of the “button click” events for button “A” and button “B.” Since potential off-task events that are not logged or not recognized as being relevant may have occurred between these timestamps, it can also be described as the maximum duration time of this state. Figure 1 illustrates that actions become labels for specific transitions and can be used to separate observed states using an FSM. However, the FSM visualized in Figure 1 is only one possible decomposition for a process into observed states based on log events using FSMs.

Using FSMs allows deriving timed low-level features, that is, states with start and end times for computing the duration, and actions with a single timestamp. Based on these low-level features, process indicators can be extracted as aggregated values of the low-level features. These are variables at the person-by-task level, with one value for each test-taker for each administered task. Depending on the desired process indicator, different low-level features can be extracted using additional FSMs. Not all log events have to be defined as meaningful actions. Log events that are not used to trigger transitions are absorbed by the self-transitions (marked as “*Ignored events” in Figure 1). In addition, the documentation of log data that describes when an event of a particular event type occurs and that provides details about the meaning of all event-specific attributes, the FSM operationalization of low-level features offers the opportunity to achieve transparency concerning log data analyses and the extraction of observed states. It can also help to achieve replicability by providing additional descriptions of the low-level features.

The study aims to introduce the digital TTT as an innovative instrument to assess young students’ SRL. To enable the use of the newly developed task for future learning analytics, the following research questions guide the study:

As in the original task, the digital TTT requires children to reconstruct a rail circle with a set of train tracks according to a plan. In our study, each child had to solve two different TTTs. The digitized task starts by automatically opening a window that shows students a plan with a shape they have to rebuild. The train tracks are arranged in an unsorted way for each task. With the pointing device (mouse, touchpad, or touch-screen), the tracks can be moved within the playing space. When two tracks are moved close to each other with matching connection spots and in the correct orientation, they automatically connect in the digital TTT. With specific buttons children can rotate the individual train tracks in two directions (clockwise and counterclockwise), can zoom in and out to enlarge the playing space, or take a closer look at a track or their current construction. Moreover, a help button allows the participants to (re)watch the explanatory video for the different features within the digital environment. With the “plan” button, students can reopen the plan showing the shape they have to rebuild at any time while working on the task. The plan and help buttons are presented as so-called modal dialogs (see Kroehne, 2023b), and no interaction with the digitalized TTT was possible while either the “plan” or “help” window was visible on the screen. Finally, participants could jump to the next task or complete the task by clicking the ‘next’ button (see Figure 2).

After closing the plan window to start the task, the experimenter connected via telephone and made the children aware of the help button, which included a video that introduced the different buttons in the digital environment and their functions. As a first task, all students had to rebuild a simple circle shape. Participants received a more complex shape depending on their age in the following task. For this purpose, the shapes already evaluated in other studies (Bryce and Whitebread, 2012; Spektor-Levy et al., 2017) were utilized for the different age groups. Previous research with the TTT as an observational instrument showed that age is related to the development of SRL abilities (Spektor-Levy et al., 2017) and demonstrated individual differences in monitoring and control (Tzohar-Rozen et al., 2021).

The digital TTT was implemented in JavaScript/HTML5 as an extension to be used with the CBA ItemBuilder (Kroehne, 2023b). In addition to the log events provided by the CBA ItemBuilder, the extension was implemented to provide additional replay-complete log data (see Kroehne and Goldhammer, 2018 for a definition) for all interactions within the digital TTT, allowing the creation of animated video replays based on the log event data. Test deployment for online assessment was implemented using the IRTlib Software (Kroehne, 2023a), embedded into a learning management system (moodle) using LTI, with IRTlib functioning as an LTI tool provider serving the CBA ItemBuilder tasks, including the digital TTT according to the study design (i.e., the age-related tasks as shown in Table 1).

Participating children engaged in a think-aloud procedure, where they were asked to verbalize their thoughts, feelings, and actions while performing the TTT. Participants were told once again that the experimenter, who gave the introduction on the phone, could not see what the children were doing. Before working on the task, participating children were trained to think aloud. When students remained silent, experimenters encouraged intensive think aloud with short standardized prompts, such as “please keep on thinking aloud.” Due to COVID-19 social distancing practices children’s think aloud was recorded via telephone and subsequently transcribed. Transferring think-aloud to a remote procedure is an innovative approach, nevertheless, some researchers have already suggested the benefits of this novel approach from the participants’ perspective (Alhejaili et al., 2022).

To identify indicators of monitoring and control, Bryce and Whitebread (2012) developed a coding scheme for analyzing observational data. Codes have extended the existing coding scheme from think-aloud research (Greene and Azevedo, 2009; Vandevelde et al., 2015) as well as by inductive codes to capture metacognition as comprehensively and fine-grained as possible. Three coders were trained in an extensive training followed by an iterative coding process. In this process, sequences of audio data were discussed to achieve a shared understanding of the different codes and to develop potential inductive codes if necessary. Metacognitive statements were coded regularly. This implies that if a child made multiple statements about, for example, discovering errors during task execution, each statement was coded individually unless they were consecutive and referred to the same mistake within the same sentence. This led to a substantial overall agreement (Fleiss Kappa κ = 0.70) for all operationalized metacognitive behaviors coded in the think aloud. Table 4 describes all codes, definitions, and examples of think-aloud protocols.

The raw log data were provided by the IRTlib software as JSON files in a proprietary format containing a data structure for each raw log event. Log events from three different sources (the deployment software/LTI tool provider IRTlib, the CBA ItemBuilder tasks, and the digital TTT embedded as an extension) were combined in postprocessing into a flat and sparse log data table (see Kroehne et al., submitted for publication) using LogFSM. After processing, all log events contain the core attributes (in which timestamps, event types, element name/task IDs), and additional event-specific attributes for events of a particular type. Event-specific attributes contain precise information on the moved track, coordinates of the moved tracks (start and end coordinates), directions of the rotations, connections between tracks, and a list of the track types in connected track groups were stored per individual and task. The data are sorted by anonymized ID and timestamps. In total, we obtained 19,867 log events for all 85 participants and all tasks. Table 3 shows the average performance time between the start event and the click on the “NEXT” button to end the task and the number of log events per task. Three children did not perform the simple TTT due to technical problems. No information was deleted from the database, as all log events appeared relevant. Cases with very short or very long processing times, as outliers, were also not removed, as it was assumed that these children either had finished the task very quickly or had taken a long time to complete it.

In the first step, all possible log events within the digital environment were explored, and meaningful actions were identified. The initial raw log data were made usable with the help of a video replay. The video replay visualizes the log data of an individual, showing the start and end positions of train track movements, rotations, and zooming, and lists the log events of the buttons. To explore the log data, video replays of 10 participants were recorded. Participants with different task processing times were selected, including some participants with particularly long task performance times and high metacognitive scores in think-aloud coding, to investigate complex behavior and possibly less stringent monitoring and control behavior. ELAN computer software (version 6.7) (Max Planck Institute for Psycholinguistics and The Language Archive, 2023; Lausberg and Sloetjes, 2009) was used to explore and analyze the selected cases. In this first step, a team of at least two trained research assistants separately coded one video and the underlying log data in an inductive manner and commented in ELAN which actions and metacognitive states could be derived from observable events or sequences of events. These preliminary indicators were discussed, and initial operationalization for the metacognitive codes of the think-aloud coding scheme was developed. This procedure was iteratively repeated for 10 participants for 20 tasks. Meaningful actions were extracted, and definitions for metacognitive states and transitions were reviewed, adapted, and optimized until a set of operationalized indicators was obtained. In addition, identified indicators were compared interindividually to review if the present sequence could be interpreted differently in the log data of other individuals. Using video replay and extensive inductive exploration and development of indicators of metacognitive behavior, consistent with previous research and theory, an attempt was made to avoid misinterpretation of the raw log data. In addition, indicators were chosen that required minimal interpretation to remain as close as possible to participants’ actual behavior and reduce subjective judgments. In total, 14 distinguished meaningful actions were defined based on the log events action library. These relevant actions were based on clicks on the different buttons, rotations, movements, and the associated changes in the existing track connections.

During the data exploration with the help of the video replay, the frequency of same events occurring directly one after the other was also determined as a meaningful sequence of actions and indicator for transitions to the different states. Therefore, count indicators were generated in addition to the 14 actions listed above, which are based on the initial events and the resulting connections. Count indicators were created for (a) the actions listed above, (b) the use of the individual tracks, and (c) directions of rotation. If the same event type occurred several times in succession (e.g., move_minus), the count always increased by one. If a different action was executed within a sequence, the count started again at one. For (b) the individual tracks, a count indicator was created according to the same pattern, which counts how often the same track was used in succession. And (c) a count indicator was created for the direction of the rotations, counting how often the participant rotated the track in the same direction in direct succession.

The administration of a particular task always starts with the plan window open; hence, the action “PLAN_CLOSE” is the first action expected for each test-taker. As described above, the actions represent indicators for possible transitions into different metacognitive states. Based on the think-aloud data coding scheme, the state library shown in Table 2 was developed. The state library includes eight metacognitive states on the microlevel, which are assigned to monitoring and control on the macrolevel. In addition, three neutral, technical states are defined.

To extract metacognitive features from the log data using FSM, we utilized the LogFSM R package (Kroehne and Goldhammer, 2018). The automated processing of log events implemented in the R package LogFSM processes the event data for each test-taker event by event, changing the observed state if a transition is defined for a particular event type (and possible event-specific additional attributes), and remains in the identical (current) observed state if no transition is defined. To extract the low-level features from the log data, an algorithm was defined based on the action and state libraries (Tables 2 , 4). Actions were defined as triggers for transition in and out of defined states representing the microlevel metacognition codes. In total, 11 states were defined and extracted with the help of LogFSM, whereby eight states are SRL related. Two states are technical and describe the task processing (startstate and endstate). In contrast, the construction state describes the continuous task performance in which monitoring and control behavior is not identified based on the defined indicators.

To examine monitoring and control behavior variations between simple and complex digital TTT, we performed a descriptive statistical analysis, investigating differences in the frequency, duration, and associated proportions of each metacognitive state. With the help of the LogFSM, the total frequency and duration of each state for each participant were extracted. The frequency of each state was determined by counting its total occurrences. State duration represents the cumulative minutes students spent in the state, calculated based on the timestamp information. Since the frequency and duration data were not normally distributed, we report the mean, median, and 25th and 75th percentile values. To compare the differences between the two tasks, Wilcoxon signed-rank tests (with Bonferroni correction) were conducted for pairwise comparison.

The Wilcoxon signed-rank test revealed significant differences in the frequencies in the metacognitive states between the two tasks (see Table 5). The negative Z-value indicates a consistent and significant increase in the occurrence of most of the states during the performance of the complex digital TTT. However, the SEEKING_HELP and the TECHNICAL_ORIENTATION, operationalized by clicking on the help button in different phases of the SRL process, occurred significantly less when working on the complex task. This seems reasonable, as the complex task was carried out directly after the simple task, and the children were, therefore, already familiar with the technical functions. The Wilcoxon signed-rank test on differences about the relative frequencies also showed significant differences across all states. However, analyzes revealed a change in the direction for SEARCH_SORT_MAKESPACE and the neutral CONSTRUCTION state. Although, overall, these states occurred more frequently while the students were working on the complex task, they were more frequent in the simple task in relation to the total number of states. This indicates the sensitivity of the digital TTT to measure differences in monitoring and controlling the behavior of young students during simple and more complex tasks.

In addition, Wilcoxon signed-rank tests also demonstrated a significant increase in duration in the states CHECKING_PLAN, MONITORING_UNDERSTANDING_MISTAKE, CHECK_CONSTRUCTION, SEARCH_SORT_MAKESPACE, CORRECT_MISTAKE, CHANGE_STRATEGY and CONSTRUCTION in the complex task compared to the simple task. Moreover, the duration comparison showed that the children also spent more time in the states TECHNICAL_ORIENTATION and SEEKING_HELP during the simple task than during the complex task. The comparison of the proportional time spent in the different states between the two task types also revealed significant differences across nearly all of the metacognitive states, with the same pattern as in the comparison of the total duration. However, this analysis reveals a different pattern regarding the state SEARCH_SORT_MAKESPACE. In relation to the total duration, the participating children spent almost the same proportion of time searching, sorting, or making space in the two task types.

As Table 6 illustrates, even the comparison of simple metrics, such as the proportional durations, gives valuable insights into the SRL process and shows that children spend a higher percentage of time looking at the plan, monitoring, and correcting mistakes in the complex task. These findings underscore task-specific variations in children’s metacognitive behavior and the potential of the introduced digitized TTT to measure such differences.

Transition graphs were utilized to investigate differences in the sequential characteristics of the metacognitive processes during the simple and the complex TTT. They provide a helpful visualization to explore the data and identify meaningful patterns. DigrammeR package in R was used to create the generic visualization of the LogFSM results. The graph nodes were defined as the states, while transitions, displayed by arrows, showed the empirically observed transitions. The indicated weight of the path relates to the proportion (the transition frequency divided by the total number of all transitions from this state). Accordingly, the sum of all outgoing paths from one state node is 1.0. However, rounding the independent probabilities for the transition paths may result in an inaccuracy of 1% for the sum of all transitions (0.99 or 1.01 instead of 1).

As the frequency and the duration analyses are already suggested, the transition graph of the simple TTT shows the relevance of the states TECHNICAL_ORIENTATION in the orientation phase and SEEKING_HELP during task performance (see Figure 3). In more than half of the cases, participating children opened the help video at the beginning of the task performance, while in 34% of the total transitions, children changed directly to the CONSTRUCTION state after starting the task performance. The simplicity of the shape is demonstrated by the fact that only 25% of the participants reopened the plan after TECHNICAL_ORIENTATION at the beginning of the task; only 7% did so directly after starting, and merely 8% of the transition from CONSTRUCTION led to CHECKING_PLAN. Moreover, the transition graph shows that, in most cases, CORRECT_MISTAKE directly followed the MONITORING_UNDERSTANDING_MISTAKE state (83%). Interestingly, a third of the transitions from CHECK_CONSTRUCTION led to the end state. This could be an indicator for evaluating the own task performance, by, for example, comparing the own construction to the shape plan before finishing the task.

Compared to the simple TTT, most children directly transferred to the CONSTRUCTION state after the start of the complex TTTs (see Figure 4). However, 40% of all transitions from the start state in the complex TTTs lead to CHECKING_PLAN, which suggests that the given shape was indeed more difficult in this task (see Figure 4). Compared to the simple task (42%), participating children changed more often from CONSTRUCTION state to MONITORING_UNDERSTANDING_MISTAKE in the complex tasks (50%). This also applies to the transition from CHECKING_PLAN to MONITORING_UNDERSTANDING_MISTAKE. While in the simple task, only 9% of the transition from the CHECKING_PLAN state leads to the MONITORING_UNDERSTANDING_MISTAKE state, this appears in 19% of the cases in the complex tasks. The transition graph of the complex TTTs visualizes the higher proportion of monitoring and control loops in these complex tasks compared to the simple digitized TTT.