According to Hattie (2023) meta-analysis, the use of digital tools has, on average, a moderate effect on learning gains (d = 0.34). Beyond cognitive outcomes, digital learning environments have also been shown to influence affective dimensions of learning. For instance, prior research suggests that such environments may help reduce students’ test anxiety (Akram and Abdelrady, 2023). In addition, interactive digital tools can enhance students’ self-efficacy, perceived instructional clarity, and learning expectations (Akram and Abdelrady, 2025), indicating that their benefits extend to motivational and experiential aspects of learning. Taken together, these findings suggest that digital media can support students’ learning in multiple ways. However, the effectiveness of digital tools depends strongly on how they are implemented. Simply introducing technology into the classroom does not automatically lead to improved performance (Hattie, 2023). To ensure that digital media contribute meaningfully to learning, their use should be aligned with four well-established educational principles: multimedia learning, self-paced learning, guided activity, and feedback (Hillmayr et al., 2020). Multimedia learning is conceptualized in the cognitive theory of multimedia learning (CTML; Mayer, 2014). This theory asserts three underlying assumptions. First, humans process information via two separate channels: the visual pictorial and the auditive verbal channel. Second, both channels only have limited, but separate bandwidths, meaning the speed at which information can be processed is limited. Third, currently presented content must be engaged with and processed actively. Active engagement with content is defined via active learning “which entails carrying out a coordinated set of cognitive processes during learning (i.e., active processing assumption)” (Mayer, 2014, p. 43). The first two assumptions explain the benefit of using multimedia in education, since more information can be processed if both channels are used without causing cognitive overload. The third assumption suggests that many digital tools can facilitate learning because they encourage students to actively engage with the content through interactive learning environments.

Furthermore, self-paced learning is considered to be an important part of effective learning because it enables students to progress through topics at their own pace (Hillmayr et al., 2020; Moreno, 2007). Students perceive tasks as less difficult if they can control the speed at which they work on them (Moreno, 2007). It has been shown that students perform better when self-pacing than following prespecified pacing, even when controlling for the total study-time (Tullis and Benjamin, 2011).

Additionally, guided activity holds significant value in interactive digital educational settings. Students find tasks enriched with guided activities less demanding than those without (Moreno and Mayer, 2007). Belland et al. (2017) concluded in their meta-analysis that scaffolding, a form of guided activity, has a significant positive effect (g¹ = 0.46) on learning outcomes in STEM education. Encouraging students to actively engage in selecting, organizing, and integrating new information fosters essential and generative processing, thereby enhancing learning outcomes (Moreno and Mayer, 2007).

Finally, feedback is an important part of effective learning processes and has been proven to enhance learning gains (Hattie and Timperley, 2007). Feedback can be categorized in different ways. Two common categorizations include differentiation by timing (immediate vs. delayed feedback), and comprehensiveness (correct response vs. elaborated feedback). Immediate feedback has been shown to be more beneficial than delayed feedback in applied settings such as classrooms (d = 0.28; Kulik and Kulik, 1988). Van Van Der Kleij et al. (2015) showed that elaborated feedback results in larger effect sizes (g = 0.49) than feedback only consisting of correct responses (g = 0.32) or feedback only consisting of information whether the answer given was correct (g = 0.05).

As interactive learning environments enable students to engage in multimedia, individualized, self-paced learning processes with direct feedback, they are expected to facilitate effective learning and enhance learning outcomes.

An example of a digital tool designed to incorporate all these features —multimedia, self-pacing, guided activity, elaborated feedback—are ITS. The following section provides an overview of ITS and the current state of research regarding their effectiveness.

Computer tutoring systems in schools have been around for almost 60 years (Atkinson, 1968). They are generally divided into two generations. The first generation is known as computer assisted instruction (CAI) (Kulik and Fletcher, 2016; VanLehn, 2011), while systems from the second generation are usually called ITS (VanLehn, 2011). The main difference between CAI and ITS lies in the level of adaptivity and the types of scaffolding provided. While there are various definitions of ITS (e.g., Ma et al., 2014; Shute and Zapata-Rivera, 2007), most emphasize that, unlike CAI, ITS not only offer corrective feedback and hints but also allow students to choose their approach. Additionally, ITS provide adaptive feedback and feedback for intermediate steps, known as sub-step feedback (VanLehn, 2006, 2011). VanLehn calls this behavior the inner loop. The outer loop, in contrast, governs the type and difficulty of tasks a student is presented with within the system. In practice, this means ITS can, for example, assign more challenging tasks to well-performing students while directing underperforming students to review fundamental topics.

Ma et al. (2014) have summarized the three features an ITS needs in order to conduct these loops properly:

“An ITS is a computer system that for each student:

We opted to analyze the effect in a real, ecologically valid setting, where individual teachers had the freedom to determine the extent of ITS usage in their respective classes. This design was chosen because it allowed to investigate the effect sizes of ITS usage in real-world conditions. As mentioned above, the effectiveness of ITS depends on many different factors—some of which may differ substantially between experimental and field settings—including factors beyond our control. Therefore, we chose to observe the implementation of an ITS in an authentic educational context, and, more specifically, to analyze how the frequency of ITS use is related to learning outcomes.

If the ITS is effective, we would expect students who use it more frequently to show greater learning gains in mathematics performance, even when controlling for covariates relevant to learning. Since the system is designed to support students in mastering curricular content, these learning gains will be assessed based on students’ performance on standard curricular tasks. Hence, our research question is:

As mentioned above, on the one hand, there are reasons to expect a positive effect of the ITS due to the effectiveness of multimedia learning, adaptive and immediate feedback, adaptive tasks, and self-paced learning. On the other hand, reduced direct teacher-student interactions and the resulting reduced social support lead to an expectation of a negative effect, as does the constraint to solve tasks in a specific way dictated by the ITS through its scaffolding. Therefore, it is not clear whether the use of ITS in learning mathematics leads to a higher or lower performance gain.

The study was conducted as a one-year longitudinal study with pre- and posttest. In order to investigate authentic ITS use, the teachers were not given any regulations regarding ITS use. They were free to use the system in their classes as much or as little as they deemed appropriate. Consequently, substantial variance in ITS usage was anticipated. In our design, this natural variability in ITS usage was used to estimate the expected effect of ITS usage on learning gains under common conditions.

At the beginning and end of the school year, a computer-based questionnaire and test was administered. The pretest was conducted between September and November 2021, while the posttest took place in June or July 2022. The exact timing of test administration was determined by the respective teachers, resulting in varying time intervals between pre- and posttests across classes. The average time interval between pre- and posttest was M = 250 days (SD = 21). Mathematics performance was measured in both tests. Given the correlational nature of the study design, a comprehensive set of student-related variables was recorded in order to control for individual differences and thus enable the estimation of the effect of ITS usage (see Section 3.4 Instruments).

Furthermore, throughout the school year, the activity of individual students in the ITS was automatically recorded in the form of aggregated data. Students completed so-called worksheets within the ITS. Each worksheet comprised a set of similar tasks on a common topic. Each time a worksheet was completed by a student, one datapoint was collected, containing three pieces of information: The number of tasks on the worksheet, the number of tasks solved correctly, and the start time of the work on the worksheet. For technical reasons, more detailed data could not be collected.

The study was conducted accompanying the rollout of a test license for the used ITS in three counties in the German federal state Schleswig-Holstein. All schools—and by extension, all students—in those counties had free access to the ITS, regardless of their participation in the study. The study was conducted with 7th and 8th grade students from 82 classes across 13 schools. The pretest was completed by 1,673 students, while the posttest was completed by 1,309 students. In total, 1,062 students were successfully matched with both a pretest and a posttest. Of these students, 120 could not be linked to an account in the ITS and were therefore excluded from the longitudinal analysis. Additionally, two more students had to be removed because they were the only participants in their respective classes with parental consent, which would have caused complications during the analysis. Thus, data from 940 students across 57 classes were available for longitudinal analysis. However, data from students with only one available measurement point were still used when feasible (e.g., scaling the Rasch model).

The average age of the sample used for longitudinal analysis at the pretest was 12.8 years, with a standard deviation of 0.7 years. Of these, 489 were female, 445 were male, and 8 did not specify their gender.

The ITS Bettermarks is a digital learning platform designed to align with the state-level mathematics curricula of German secondary education and can be used both in the classroom and as a supplementary tool (Bettermarks, 2025). The ITS is intended to support the assessment and development of students’ competencies within the curricular frameworks governing secondary mathematics education in Germany (Bettermarks, 2025). Therefore, it offers a comprehensive range of tasks covering the prescribed content of secondary education in Germany. The material provided by the ITS for Grades 7 and 8 shows substantial correspondence with the TIMSS framework (Mullis et al., 2021), aligning largely with the content domains Number, Algebra, and Geometry and Measurement, as well as with the cognitive domains Knowing and Reasoning.

Access to the platform is available to both students and teachers via a browser or an app. The content within the ITS is organized in a folder structure (Figure 1). Each topic has its own book, comprising an introduction, several chapters, a test to assess understanding of the covered content, and a review section for students to revisit the topic and assess their proficiency. Each chapter consists of various worksheets, each containing multiple tasks.

Teachers can assign digital worksheets, known as to do’s, to their students, allowing them to give digital homework, for example. Teachers can view the completion rates of individual students as well as their inputs. Students can also independently select tasks from all books and work on them autonomously.

Most tasks in the worksheets focus on promoting procedural knowledge, but there are also tasks that promote conceptual knowledge (e.g., Hiebert and Lefevre, 1986). Tasks often involve performing common standardized procedures, such as adding fractions, determining a function equation, or drawing and calculating the area of a triangle. The ITS contains few tasks that encourage constructive reasoning. As a result, input formats often consist of numbers or fractions. Sometimes more interactive inputs are required, such as digitally drawing a function graph using a toolbox or iconically representing a fraction by dividing and coloring an area.

More complex tasks are broken down into substeps. For example, the breakdown of adding two unlike fractions is as follows:

These substeps are worked on sequentially. The ITS provides direct feedback for each input. Some of the feedback is corrective (“Sorry, that’s not right.”), while other feedback is adaptive (“You have found a common denominator, but the lowest common denominator is smaller.”). If an incorrect input is made twice, this substep is marked as incorrect, and the solution to this substep is shown as a worked example. These features correspond to the inner loop according to VanLehn (2006, 2011). Additionally, the ITS recognizes when students consistently make specific types of errors. For instance, if a student demonstrates proficiency in adding like fractions but has difficulty with unlike fractions, the system detects this specific knowledge gap. It then assigns practice tasks that directly target the identified area of difficulty. These tasks are presented individually within the domain labeled knowledge gaps, enabling focused and tailored practice. This mechanism corresponds to the outer loop according to VanLehn. The structure of this system also qualifies Bettermarks as an ITS, in line with the definition of Ma et al. (2014) mentioned in section 2.2.

There were no guidelines for the teachers on how the ITS should be used. Teachers and students could decide freely on the extent of ITS usage and when it occurred (whether during or outside school hours). There were also no guidelines regarding the topics or usage format (group or individual work) although the design of the ITS lends itself more to individual use.

To investigate the impact of the frequency of ITS usage on learning progress, we first needed to operationalize what “usage” means in a way that fits the structure of the available data. The log data contained one entry for each time a student opened a worksheet, including the start date and time, but they do not contain any information on when students stopped working. Because of this limitation, we were not able to calculate actual time-on-task and instead focused on how often students initiated work on worksheets. To capture different facets of students’ engagement with the ITS, we developed five plausible operationalizations of usage frequency.

The five operationalizations are defined as follows:

This operationalization counts every instance of a student opening a worksheet during the measurement period, including repeated interactions with the same worksheet. It is conceptually close to common indicators of engagement, such as time spent using the system, as it reflects the overall volume of interactions. However, it may overrepresent short, dense bursts of activity (e.g., repeated re-openings), which can inflate usage counts without necessarily reflecting meaningful learning activity.

Here, each worksheet is counted only once, regardless of how often it is revisited. This measure emphasizes the breadth of content students engaged with and reduces inflation due to repeated openings of the same worksheet. At the same time, it ignores repetition and revisiting, which may be pedagogically relevant, and is less closely related to time-based measures of system use.

In this approach, each worksheet is counted at most once per hour. This reduces the influence of very dense activity within short time spans while still allowing repeated engagement with the same worksheet to be reflected over time. A limitation is that the choice of an hourly window is somewhat arbitrary, and the measure may overestimate the usefulness of engagement when the same worksheets are worked on repeatedly.

This operationalization counts each worksheet at most once per day. It further reduces the impact of short-term bursts of activity and aligns well with daily study routines and typical homework structures. However, it masks variation in within-day intensity and duration of work and treats brief and extensive engagement on the same day equivalently, resulting in a weaker correspondence with time-based engagement measures.

This measure captures the number of days on which at least one worksheet was opened. It emphasizes the regularity of engagement over time and is closely related to concepts of spaced practice. At the same time, it provides no information about the amount or depth of work completed on a given day and does not distinguish between minimal and extensive daily usage.

Because the period between pre- and posttest differed across classes, all operationalizations were adjusted by dividing the absolute count n by the number of weeks t between the two measurements. This yields the standardized usage score.

We consider all five operationalizations valid, as each captures a different facet of how students used the ITS. Since none is theoretically superior, we selected the operationalization with the strongest raw effect size for the main analyses. Raw effect size refers to the association between usage and learning gains without adjusting for covariates. Section 5.1 reports the comparison of these raw effects and identifies which operationalization was chosen for further analysis.

Overall, the students completed 61,051 worksheets, indicating that each student on average completed about 2 worksheets per school week (SD = 2.4). The average solution rate was 34%, meaning roughly one in three given answers was correct. Knowledge gaps (see section 3.5) comprised 1.5% of the opened worksheets. If students worked on worksheets multiple times, each instance was counted separately. Approximately one-third of ITS usage occurred during school hours (Monday to Friday, 8 a.m. to 2 p.m.), while the remaining two-thirds occurred at home.

The operationalizations for ITS usage discussed in 4.5 were compared in terms of their raw effect size on both the class level (class mean) and the individual level (centered at the respective class means). The standardized effect sizes are shown in Table 1.

As can be seen in Table 1, operationalization 2 (counting each worksheet only once per student, no matter how often it was completed during the school year) has the strongest significant raw effect size on the posttest of all operationalizations. Therefore, this operationalization was used for subsequent analysis. Under this operationalization, the ITS usage is visualized in the histogram in Figure 3.

The average ITS usage is about 1.3 unique worksheets per school week. A total of 469 students completed less than one worksheet per school week, 301 completed between one and two, 170 completed 2 or more. It can be seen that the ITS usage varied between students and the ITS usage itself is substantial. Therefore, a potential effect of ITS usage on learning gains should show up in the final model.

The items assessing students’ perspective on working with the ITS—specifically in terms of demand, affective and cognitive engagement, and usefulness—were averaged into their respective Likert scales.

As shown in Figure 4, most students found working with the ITS to be relatively low in demand (M = 1.9, SD = 0.6). They reported moderate levels of engagement (M = 2.5, SD = 0.8), and half of the students somewhat agreed that the ITS was helpful in learning mathematics (M = 2.8, SD = 0.7).

On the German grading scale (ranging from 1, very good, to 6, insufficient), students rated the ITS perceived usefulness with an average score of 2.7 (SD = 1.0). Overall, most students expressed neutral or mildly positive feelings about using the ITS on both the subjective ratings and the grade.

As described in the Methods section, a series of multilevel models (Table 2) were used to analyze the data. To build a model with a good fit to the dataset, the initial model was progressively made more complex by successively adding predictors. Predictors were added generally following the approach outlined by Field et al. (2012), if (a) there is empirical evidence which links the predictor to student performance and (b) adding the predictor increased the predictive power of the model. Models were compared based on their Akaike information criterion (AIC) and Bayesian information criterion (BIC) values. Both information criteria are measures of goodness of fit. While the values cannot be interpreted in an absolute way, a lower value indicates a better fit. In model M0, only the hierarchical structure of data with no predictors was considered. The intra class correlation (ICC) was 0.53, indicating there was about as much variance on the individual level as on the class level. In models M1 and M2, assessments of prior knowledge (pretest score and marks in mathematics) were added. In model M3, the noncognitive measures regarding mathematics and students’ perspective were added. The five different noncognitive measurements regarding mathematics were expectedly moderately correlated (on average 0.43), so it made sense to only add one of them as a covariate. Out of five different scales, attitude towards mathematics had the strongest effect of the posttest score and was therefore included in model M3. In model M4, structural variables at the student and school level—grade and school type—were added. School type showed a substantial association with posttest performance, even after controlling for prior achievement and other covariates. Grade level, in contrast, was included primarily as a theoretically relevant control variable to account for differences in curricular progression, but did not exhibit an additional effect once prior performance was taken into account. As shown in Table 2, the stepwise addition of predictors from models M1 to M4 was associated with successive improvements in model fit, as indicated by decreasing AIC and BIC values, and with increases in explained variance at both the individual and class level. Including any other predictors, such as gender or self-concept in other subjects, resulted in an increase in both information criteria, AIC and BIC.

To examine the influence of ITS usage frequency, this factor was incorporated into model M5 at two levels. At the class level, the group-centered mean of ITS usage frequency was introduced for each class. At the individual level, the usage frequency of individual students, centered on the class mean, was included as part of the model. At the class level, we investigated whether classes that used the ITS more frequently (and thus probably traditional materials less frequently) showed greater learning gains compared to classes with lower usage frequency (between-group effect). At the individual level, we examined whether students who completed more tasks than their classmates exhibited higher learning gains (within-group effect). A between-group effect could therefore be interpreted as an improvement in learning gains compared to traditional learning materials, while a within-group effect could suggest that increased learning time enhances learning gains.

Table 2 presents the hierarchical inclusion of predictors across models M1 to M5. In M1, performance in the pretest is a strong predictor of the performance in the posttest, accounting for approximately 56% of the variance at the individual level and approximately 83% at the class level. The inclusion of marks as an additional predictor in M2 increases the proportion of explained variance and improves model fit. Model M3 shows that a more positive attitude towards mathematics is significantly associated with higher posttest scores. In M4, the influence of school type becomes evident. Controlling for prior performance, mathematics attitude, marks in mathematics and grade, the difference in posttest performance between students attending academic track schools (Gymnasium) and those from comprehensive schools (Gemeinschaftsschule) is 0.57 logits, indicating substantially greater learning gains among students in the academic track. The inclusion of ITS usage as a predictor in M5 does not improve model fit, as indicated by higher AIC and BIC values compared to Model 4, and the explained variance remains largely unchanged. Furthermore, ITS usage does not exhibit a significant effect on student performance at either the individual or class level.

Based on a longitudinal pre–posttest design, our study examined whether the frequency of ITS use influenced students’ learning gains. The multilevel analysis revealed a significant positive effect of ITS usage on mathematics performance (β = 0.22) on the individual level when not controlling for other variables (left part of Table 1). However, when controlling pretest scores, the observed effect of ITS usage on mathematics performance was notably smaller (β = 0.09, right part of Table 1), a pattern consistent with findings from a meta-analysis by Ma et al. (2014), which showed that studies accounting for baseline differences tend to report smaller effect sizes. This smaller, yet still significant effect aligns with results reported by Spitzer (2022) for the same ITS; however, that study did not account for additional covariates beyond prior performance. In contrast, in our study, when further covariates—such as students’ attitudes toward mathematics, grade level, and school type—were included in the model, the effect of ITS use was no longer statistically significant, neither on the individual nor the class level (Model 5 in Table 2). Therefore, the answer to the research question is that no significant relationship was found between the frequency of ITS use and student learning gains in curricular mathematics tasks, once prior performance and other relevant covariates were controlled for. This non-significant effect shows the importance of controlling relevant covariates in a correlational design, even though doing so substantially increases the analytical workload.

The analysis was repeated using alternative operationalizations of ITS usage (as defined in section 4.5), including counting every opened worksheet, limiting each worksheet to one count per hour or per day, and counting only days in which the ITS was used at least once, yielding no significant deviations from the aforementioned results. The effect of ITS use on posttest performance was smaller once pretest performance was controlled for. This indicates that high-achieving students use the ITS more and learn more over the course of a school year. However, our findings do not support the assumption that increased ITS usage directly leads to greater learning gains. Simply using the ITS more frequently—potentially at the expense of other instructional approaches—does not appear to enhance student learning outcomes. As Higgins et al. (2012) aptly put it, “…it is not whether technology is used (or not) which makes the difference, but how well the technology is used to support teaching and learning” (p. 3). In addition, students perceived the ITS as having a moderately positive impact on their mathematics learning. This aligns with the findings from the multilevel model, which suggest that the ITS supports mathematics learning to a similar extent as traditional media.

As the study was conducted in the context of the introduction of the ITS in a federal state in Germany, it was not possible to conduct an experiment with random assignment to a control group not using the ITS. Therefore, the lack of a control group restricts our ability to firmly establish causal effects of the ITS’s impact on learning outcomes. Nevertheless, we found substantial variance in the “natural” usage of ITS among different classes and we could show a large discrepancy between raw and controlled effect sizes of ITS usage, both of which will be important for further investigation.

Due to the limited available log data, it was only possible to analyze the usage frequency of the students. This reduces the explanatory power of our analysis, as other factors—such as the social form employed during ITS usage as well as the way teachers implemented the ITS during and beyond school hours—could also influence the ITS’s effectiveness.

As an additional consequence resulting from the limited log data, ITS usage was operationalized as the frequency with which students opened unique worksheets. As a result, it was not possible to distinguish between different forms of engagement, such as careful, effortful work and rapid task completion, nor to identify non-learning behaviors. This limitation may have weakened the observed relationship between ITS usage and learning gains. Nevertheless, the moderate raw association between ITS usage and performance suggests that this measure still captured a meaningful, though coarse, indicator of students’ interaction with the ITS.

The results of our study raise the question of why the use of the ITS was not effective. In the following, several plausible reasons are discussed. Because the extent to which students actually used the system determines the potential impact of any instructional technology, the discussion first considers the intensity of ITS use observed in our data. After outlining the extent of actual system use, the discussion begins with specific features of the ITS itself including the types of tasks available and the nature of the feedback, and the extent to which the system’s adaptive functionality was utilized. Then moves on to challenges related to its use in the classroom, including self-pacing, homework integration, collaborative learning, and the role of teachers in shaping instructional use. Furthermore, a broader perspective is taken by examining the ITS’s alignment with principles from CTML and guided activity, before reflecting on aspects of the study design—such as the long measurement period—that may also have played a role. The section concludes with a discussion of the alignment between the ITS and the performance test.

Although students completed a total of 61,051 worksheets, the average individual usage corresponded to only about 1.3 unique worksheets per school week, with nearly half of the students completing fewer than one unique worksheet per week. Taken together with the null effects in our multilevel models, this suggests that many students may not have reached an intensity of ITS use that is sufficient to produce detectable gains on a broad curriculum-based test.

One contributing factor may be the limited types of tasks that students could engage with in the ITS. Although the system offers a large number of items, many of them focus on highly procedural exercise formats and comparatively few engage students in forms of mathematical activity linked to deeper learning, such as constructive reasoning, modelling real-world situations, communicating mathematical ideas, sketching representations, or working through open or non-routine problems. These forms of activity are widely recognized as essential components of mathematical proficiency because they require students to make sense of mathematical structures, connect representations, and articulate their reasoning (e.g., Santos-Trigo, 2024). As outlined in the introduction, the ITS primarily offers tasks with clearly structured scaffolding and step-by-step guidance, but includes relatively few opportunities for tasks that actively promote such constructive or representational reasoning. In this regard, the system may be well suited for supporting procedural fluency but less effective for fostering flexible and conceptual mathematical understanding.

The type of feedback provided by the ITS could also account for the results. Although the system offers immediate feedback at each substep, the depth and quality of this feedback vary considerably: some items provide elaborated explanations, whereas many others offer only corrective “right/wrong” responses. Research has shown that elaborative feedback is generally more beneficial for learning than purely corrective feedback because it supports the development of conceptual understanding and helps students diagnose their misconceptions (Van Der Kleij et al., 2015). In our study, this pattern aligns with students’ own perceptions: they rated the ITS as only moderately engaging and only somewhat helpful for learning, suggesting that the predominantly corrective feedback may have facilitated procedural success without consistently fostering deeper understanding. For more complex or conceptually demanding tasks, students may therefore have required additional forms of scaffolding beyond immediate correctness indicators.

Another factor that may help explain the absence of significant effects concerns the limited use of the ITS’s adaptive knowledge gap functionality. Although the system is designed to identify and address individual misconceptions through targeted follow-up tasks, knowledge gaps accounted for only 1.5% of all opened worksheets, despite an average solution rate of approximately one third (section 5.1). This indicates that, while students frequently made errors during regular worksheet work, these errors translated into only minimal engagement with targeted follow-up tasks addressing identified difficulties. One possible explanation is that knowledge gaps were assigned relatively conservatively by the system; another is that students often did not engage with recommended follow-up tasks once they were available. The present data do not allow a clear distinction between these explanations. Previous research suggests that learning effects of digital tools are particularly pronounced when systems are adaptive (Hillmayr et al., 2020). Against this background, the largely underutilized adaptive functionality observed in the present study provides a plausible explanation for why the ITS did not yield stronger effects.

In this context, how teachers handled the use of the ITS also likely influenced the effectiveness. For the portion of ITS usage that occurred during school hours, it is not entirely clear to what extent students were able to work with the ITS at their own pace, as this likely depended on how much autonomy teachers allowed during ITS use. As found by Moreno (2007), allowing students to self-pace lowers perceived difficulty.

In addition to usage during school hours, a substantial part (approximately two thirds) of the ITS usage was conducted outside school hours. This indicates that a significant fraction of homework was completed and handed in through the ITS. As found by Hattie (2023), the evaluation and discussion of homework is important for students. If the digital homework was not properly discussed in class because the ITS provided worked examples, this could hinder students to ask questions and get clarification about certain tasks and therefore negatively impact their learning gains.

Another important factor is collaborative learning, which has been shown to enhance learning outcomes. As outlined in the theoretical framework, students who engage in collaborative learning tend to learn more effectively, especially when technology facilitates this process (Chen et al., 2018; Sung et al., 2017). However, the ITS analyzed here is not designed to support collaborative learning in pairs or small groups. Given that the majority of ITS use in our sample occurred outside school hours, much of the work with the system likely took place individually rather than in pairs or groups. This usage pattern, together with the lack of collaborative features in the system, may help explain why potential benefits of collaborative learning with technology did not materialize here. Consequently, greater use of the ITS may limit opportunities for students to collaborate and engage in meaningful discussions about mathematical concepts. This lack of interaction could result in smaller learning gains overall.

Teachers’ decisions regarding the adoption of the ITS constitute another important factor in interpreting the findings. Although the system was available to all participating schools, its use was entirely voluntary, and teachers retained full discretion over whether they used the ITS at all and how intensively it was employed in their classes. Even though the ITS was adopted in many classrooms, overall usage intensity remained low and varied substantially across classes, indicating that mere availability did not translate into sustained or systematic use. This pattern is consistent with research showing that teachers’ adoption of digital tools depends strongly on their perceived usefulness, attitudes toward technology, and institutional conditions (Teo, 2011). Moreover, when digital tools are not embedded in shared instructional routines at the school level, uptake tends to remain uneven, which is typically associated with reduced or absent effects on student learning (Ertmer and Ottenbreit-Leftwich, 2010).

Beyond adoption, the instructional use of the ITS is therefore likely to have influenced its effectiveness. Although teachers who chose to use the system were free to embed it into their teaching as they saw fit, no mandatory training or instructional guidance accompanied its implementation. Meta-analytic evidence indicates that digital interventions yield substantially larger learning effects when teacher training is provided than when it is not (Hillmayr et al., 2020). Teacher training typically focuses not only on technical operation but also on how digital tools can be integrated into instruction, how students’ work can be monitored, and how digital tasks can be connected to classroom discussion and follow-up activities. As a result, the pedagogical use of the ITS likely varied widely, and in some cases may not have been closely aligned with instructional goals or curricular demands. Earlier experimental work further suggests that insufficient preparation can, in some cases, even be associated with negative learning effects (Koedinger and Anderson, 1993). Taken together, these findings highlight the teacher’s role in shaping the effectiveness of digital tools and provide a plausible explanation for the null effects observed under conditions of voluntary and unsupported ITS use.

In addition to the lack of collaborative learning, the results can in part be explained with CTML (Mayer, 2014). Since most tasks found in the ITS operate at a symbolic or algebraic level, according to CTML, they do not make full use of the pictorial channel, leading to less effective learning.

In contrast to the previously discussed theoretical concepts, the ITS adheres to the principles of guided activity very well. The ITS incorporates well-structured substeps with clear instructions, aligning with the principles of guided activity. This design facilitates task completion by providing explicit guidance to students and, hence, assumingly fosters effective learning.

The study design itself also plays a role in interpreting the results. As Kulik and Fletcher (2016) found, longer measurement periods correlate with smaller effect sizes. One possible explanation for this is that the novelty factor of the ITS diminishes over time.

The ITS and the performance test were closely aligned with respect to both curricular content and the types of competencies addressed. As described in sections 4.3 and 4.4, both were developed with reference to the same state-level curricula, resulting in substantial overlap in the mathematical content covered. In addition, the performance test primarily assessed procedural knowledge, which corresponds to the predominant focus of the tasks provided by the ITS. The observed pre–post learning gains were of a magnitude that is typical for students at this grade level over a school year, suggesting that the performance test was sensitive to the mathematical learning that occurred during the measurement period. Against this background, it is unlikely that the absence of a significant effect of ITS use can be attributed to a mismatch between the learning environment and the performance test.

In summary, while the ITS aligns well with the principle of guided activity, other aspects may help explain its limited effectiveness. The types of tasks and the feedback provided may not have been sufficient to support deeper learning. Opportunities for self-paced work in class were likely restricted, and digital homework may have lacked in-class follow-up. Key principles from CTML also appear only partially addressed. Lastly, the long measurement period may reduce the measured effect sizes in digital learning due to diminishing novelty effects.

Considering limitations of the ITS and likely challenges in its integration into classroom settings, the absence of significant effects are reasonable. They align with Hattie (2023) findings that the mere presence of technology in the classroom does not guarantee improved learning outcomes. Instead, it is probably the way technology is integrated into the teaching process that truly matters.

Given the ecologically valid nature of the study, the results can be interpreted as a realistic evaluation of the implementation of an ITS. In Germany, if schools are provided with technology, such as an ITS, the frequency and manner in which this technology is used is almost entirely at the discretion of the individual teacher. The hope associated with the provision of an ITS is that this type of technology use will optimize learning processes and lead to sustainably higher performances in international (PISA) and national standardized evaluations in Germany.

A more precise delineation of the capabilities and limitations of digital media in education is necessary. In particular, it is important to address how effective digital tools are in supporting students when revisiting familiar concepts as opposed to engaging with new ones. Also, exploring the importance of students’ knowledge about the ITS and their metacognitive awareness of learning with ITS merit attention.