Learning describes an act of information processing through various levels and units of human memory (see Atkinson and Shiffrin, 1968; Baddeley, 1992). These considerations emphasize learners' active role in cognitive, metacognitive, affective, and motivational processes during learning (see Boekaerts, 1999; Winne, 2001). Given that framework, multi-dimensional constructs seem plausible but also inevitable in regard to describing the learning process, always in a descriptive tradeoff between process components and possible (sub-) outcomes. As LE has predominantly been defined as the observable consequence of learning motivation and participation in learning activities (e.g., Hu and Hui, 2012; Lan and Hew, 2020), the highly complex construct comprises abundant dimensions. Following that understanding of LE's versatility, the construct is frequently divided into behavioral, cognitive, or emotional dimensions which will be captured in this study as effort, attention, and content interest (e.g., Jamet et al., 2020; Lan and Hew, 2018; Deng et al., 2020).

Given those facts, LE is considered a major factor influencing learning outcomes or academic success to the same extent as learning persistence and performance (Kuh et al., 2008). While previous positive effects of LE (e.g., on knowledge retention or processing depth of learning material; e.g., Sugden et al., 2021) retain their relevance beyond traditional learning settings, the role and potential of LE become even more important in learning contexts with time- and location-independent learning activities that rely on high levels of self-control and provide educators with fewer possibilities to moderate learning processes immediately.

In addition, considering that learning is not only an individual but also a social process (e.g., Young and Collin, 2004), all sub-dimensions of LE are relevant in the context of both institutionalized AOSs and non-institutionalized AOSs (e.g., Massive Open Online Courses; MOOCs). Although MOOCs often differ in terms of their learners, for example, because they are primarily independent of university curricula and participants do not identify as students (Watted and Barak, 2018), the role of LE is widely discussed in this domain (e.g., Zhang et al., 2021). With MOOCs and other open educational resources being integrated into institutionalized curricula (e.g., in blended learning; Feitosa de Moura et al., 2021), the lines between MOOCs and institutionalized AOSs are blurred. Not capable of educators' traditional in-class impressions, gained in classical learning settings, a higher value must be placed on learner activities and information that can be derived from their behavior. Accordingly, recent studies explored both sides of the coin, taking into account that learners' active engagement with the learning material in AOSs is a relevant determinant of student learning (cf. Bosch et al., 2021; Koszalka et al., 2021; Lan and Hew, 2020) while underlining the predictability of LE by learner characteristics (Daumiller et al., 2020; Doo and Bonk, 2020; Gillen-O'Neel, 2021). A popular application that uses LE inferences is, for example, in the prediction or determination of drop-out rates (Landis and Reschly, 2013).

Following the above-mentioned explanations, the study attempts to answer two research questions.

The first research question examines the extent to which trace data (independent variables) can be assigned to respective behavioral, cognitive, or emotional sub-dimensions derived from LE research and its combined potential to explain self-reported LE (dependent variables). Therefore, it is formulated as follows:

The second research question deals with the stability of possible model-based predictions. In the sense of measurement repetition, it applies linear models generated using one data subset [lesson 2 (L2); t1] to a comparable dataset [lesson 5 (L5); t2]. The chosen procedure not only makes it possible to check the stability of a model generated at t1 concerning the explained variance over both measurement times but also offers the opportunity to show the extent to which the explained variance of a model optimized at t2 (benchmark) differs from the model generated at t1, providing a framework in which the significance of the context in which learning takes place can be discussed. Therefore, the following formulation adequately summarizes the line of thought.

Participants were 764 teacher students from two German universities. For answering both RQs, subsets of two comparable lessons were used further on (L2: n = 608, L5: n = 408, Table 2). Both samples derive from the same population, and two different measurement times of the same online course are the subject of the study; therefore, 372 people are included in both datasets, which generate dependencies.

Since participants actively registered for the AOS, only driven by the pre-released content description and not by an explicit interest in participating in the study, data must be interpreted as results of an ad-hoc sample.

The sample consisted of teacher students, who attend different study programs, combining at least two majors (from Natural Sciences, Linguistics, Arts, Sports, Humanities, and modules in Educational Sciences). Given that, the sample is able to incorporate tech-close/-distant study routines (e.g., programming or text-based research-intensive courses vs. laboratory work or gym lessons) that might map LMS-usage variance. Limited by its homogeneity in regard of the student population, the sample yet allowed a non-biased material-wise insight, due to the fact that all participants were interested in the topic of how to teach (the respective content) with digital media. On a voluntary basis, students indicated the following course attendance (multiple selection possible; across all teaching degree programs): 4% Sport, 6% Art, 19% Humanities, 34% Natural Sciences, and 36% Linguistics.

The study was approved by the ethics committee as participation in the AOS was possible regardless of the provision of data. Declarations of consent were obtained from all participants for the data used in the study.

The results for both lessons L2 and L5 showed weak bivariate correlations of individual indicators with the LE dimensions of effort, attention, and content interest (Tables 5, 6). While 26 correlations of 16 unique indicators were significant (p < 0.05), the correlations of only three unique indicators were relevant in both lessons, respectively (r ≥ 0.1 (rounded); 10012, 1071, 2022).

Taking into account the used indicator pool and its mostly weak correlation toward the respective LE dimensions, it has to be stated that there are existing weak-to-moderate correlations, mostly highly significant, which could justify the existence of suppressor variables affecting the designed models (Table 7).

The regression models for each LE dimension identified from best-subset regressions for the L2 data are shown in Table 8. The models were composed of nine indicators from a pool of 16 unique indicators.

To examine the stability of the identified models, we applied them on L5 data. Table 9 shows similar R² values for the L2.model on both, L2 as well as L5, datasets. Surprisingly, the R² values on the L5 data are slightly higher. Optimizing the model fit on L5 data by repeating all previous analysis procedures (computing a L5.model on L5 data), displayed in an L5.model R² benchmark, puts the potential of the L2.models in perspective to explain 53–60% of the variance of the optimized model but also highlights further amplification effects evoked by a contextual indicator fit.

Bootstrapping results (5,000 repetitions) confirm the robustness of the models (effort R²: L2 model: 95%CI [0.02,0.08], M:0.07, SD:0.02; L2 model applied on L5 data: 95%CI [0.02,0.10], M:0.09, SD:0.03; attention R²: 95%CI [0.02,0.07], M:0.07, SD:0.02; L2 model applied on L5 data: 95%CI [0.01,0.06], M:0.06, SD:0.02; content interest R²: 95%CI [0.01,0.05], M:0.05, SD:0.02; L2 model applied on L5 data: 95%CI [0.02,0.10], M:0.08, SD:0.02). Regarding the given data structure, it should be critically noted that linear dependencies exist within the models, which make it necessary to take into account the earlier presented bivariate correlations and the nature of some indicators. Calculation of the variance inflation factors for all models, however, shows that this circumstance has hardly any influence on the presented process (VIFs: effort: min 1.02, max 1.88; attention: min 1.02, max 2.19; content Interest: min 1.02, max 1.46).

The question of generalizability, the given course structure, and dependencies within the study design mark the major hurdles of the study.

In addition to all thoughts invested into the presentation of why the given sample is capable to come with an acceptable amount of behavioral variance (age, semester, and study routines derived from various study majors), the study's external validity needs to be tested. A similar problem needs to be faced in the context of the used questionnaires. Although the factor structure within self-reported LE was examined and found to be conclusive, it remains unclear what specific proportion of self-report is predicted. Future work should address the relationship between LE and performance, as well as continue to model LE as a (robust) latent construct (Wong and Liem, 2022).

Despite the AOS comprising five lessons, only two lessons, nearly identical in their instructional design and structure, were comparable, due to the context-specific characteristics of indicators. While normalization and standardization of time-on-task indicators on video or text might still be possible (e.g., Crossley et al., 2023), other ordinal data such as action counts or time differences are heavily influenced by learning design elements (e.g., tasks that come along with vast behavioral changes such as group work/forums, concept maps, or deadlines within lessons; Ahmad et al., 2022). Next, the role of (inter-)personal characteristics, offline activities in general and especially regarding the intensity of peer interactions during the course, was not investigated in the study. Even though the original study design was capable to address such considerations, we opted for a narrower, yet in the sense of the assessment—better communicable, and more comprehensible way but underline the lack of a by far holistic perspective that needs to be met by further research. Therefore, further studies are needed to investigate more holistic models of further generalizable indicators that can be applied to various contexts.

Finally, the study setup itself is susceptible to dependencies, starting with the fact that the dataset is the product of a university course and therefore 92% of L2 students are part of the L5 dataset, which contributes significantly to answering the question of stability. While this situation could be resolved through cohort comparisons, another issue arises through students' familiarization with the LMS and the question of how to define sort of baseline behaviors. Regarding indicator selection, not only context fit but also habituation effects (e.g., focused vs. exploratory navigation/more deliberate or strategic behavior per se, but also customization status toward the LMS in general; power/expert vs. novice users) need to be discussed with the perspective of its impact on changing baseline attributes in more detail in future research.

Beyond conceptual considerations, this argumentation line must be supplemented at the level of the existing data structure as follows. The existence of moderating, mediating, or suppressor variables as well as collinearity is nurtured throughout the research project due to three factors: (a) by indicator inherent correlations, (b) by the design of the used behavioral indicators per se, and (c) by the applied best-subset regression method. All factors are intertwined, which makes a holistic representation of influencing factors almost impossible. To report the mechanisms of action as transparently as possible anyway, examples are given below.

Inherent correlations can be shown, for example, by the situation that the number of breaks taken helps to explain the LE dimension effort, in which the probability of the number of breaks increases with the time invested in processing learning resources. However, the estimated fundamental force at work here lies in the time-on-task logic. The significance of the design of behavioral indicators, on the other hand, can be outlined by focusing on such two indicators as “opening the additional content page” and “visited additional content”, as one indicator is a prerequisite for initiating the other one. Finally, the best-subset regression approach used addresses precisely the mechanisms described here by declaring its aim to include collinear independent variables in model proposals instead of ignoring them due to penalties. However, this increases the probability of the effect of unnoticed suppressor variables, which in doubt reduces effects of not only one but also several independent variables of the proposed models. To put this in perspective, we are aware about the capabilities of machine learning techniques that would probably be able to be more precise concerning modeling but want to point out that the final selection would no longer happen in a sense of deductive, prior assumption-made manner but more on performance. This in turn would contradict the previously outlined balance of interpretability and performance. Knowing about the possibilities of more flexible non-linear modeling approaches, we made a very conscious decision in favor of the chosen path, deploying variables that reflect all three dimensions of LE, not interested in eventually downsizing dimensions by ignoring conceptual interpretability (Luan and Tsai, 2021; Hellas et al., 2018; Khosravi et al., 2022).

Overall, we call for a future research orientation that contributes to assessment considerations, indicator and learning design, firmly anchored in theory.