Black and Wiliam (1998) argue that to best support learning, an assessment must produce evidence of a student’s level of knowledge, skills, and abilities (KSAs) and elicit performance associated with demonstrating the state of KSAs expected at that level. They suggest a combination of cognitive theory of learning, assessment design, measurement models, and curriculum provides the most beneficial information to promote student learning. Expressing a similar viewpoint for assessment more generally, the National Research Council (2001, 2014) proposed an Assessment Triangle that emphasizes the theoretical and empirical connections among cognitive/substantive theory, task design, and analytic methods in order to create valid assessment and support reliable inferences for the purpose at hand, whether formative, summative, certification, etc. Therefore, a central challenge in assessment design is developing suitable frameworks that link theory, tasks that provide observable evidence about a student’s capability relative to those substantive theories, and analytic models that interpret student performance accordingly.

Formative assessment in particular is of increasing interest, where the objective is assessing students’ learning progress during instruction in order to guide further learning, rather than focusing on achievements at their end of a program of study (Black and Wiliam, 1998; Elwood, 2006; Bennett, 2011; Briggs et al., 2012). Huff and Goodman (2007) found that a large percentage of teachers wished they had more individualized diagnostic information from these assessments. The National Research Council (2001, 2014) reported that formative and timely feedback is important to students in their learning development (also see Gordon, 2020). One of the major purposes of formative assessment is to check student’s progress on learning tasks as well as to monitor the effectiveness of the teachers’ instruction, which can result in identifying a gap between a student’s actual level and desired level of performance, thus providing information to bolster a student’s understanding of a topic (Havnes et al., 2012; Kusairi et al., 2019).

Further, the COVID-19 pandemic has challenged education institutions around the world in teaching by traditional means. This crisis prompts administrators to adopt alternative strategies to address learning assessment (Khan and Jawaid, 2020). Online formative assessments are increasingly of interest as an alternative solution, which can be implemented by mailing/e-mailing, messaging platforms, discussion boards, and online educational platform tools (Nagandla et al., 2018; Choi and McClenen, 2020; Khan and Jawaid, 2020). For examples, CAN provides online educational platform in high schools, colleges, and community organizations around the world, drawing on curricula, instruction, and interactive assessment.

An evidence-centered design framework (ECD; Mislevy et al., 2003b) is a principled assessment design framework that can provides guidance for generating tasks that evoke evidence about students’ KSAs, and for coherently connecting theory embodied in an application with task design, and for choosing analytic models that best characterize the relationship between them. Furthermore, Arieli-Attali et al. (2019) proposed an expanded ECD including a learning layer as assessment design framework. The expanded ECD supports a creation of a system for blended assessment and learning at the design stage. The learning layer consists of (1) e-Proficiency model for identifying learning processes, (2) e-Task model for specifying features of learning support in the task design, and (3) e-Evidence model for addressing statistical methods of inferring latent learning processes. In a similar vein, Deonovic et al. (2020) proposed a master model that incorporates learning and assessment. The master model provides latent-variable statistical model that supports detailed diagnostic feedback related to the learning model.

Learning progressions provide a grounded theory for creating tasks and making inferences about a student’s progress. Interest in LPs has been increasing in many educational areas because they provide substantive evidence in the development of formative assessment (Arieli-Attali et al., 2019). LPs are defined by measurable pathways that a student may follow in the process of developing their knowledge and gaining expertise over time (National Research Council, 2001). Wilson’s (2012) research on measurement for structured assessments focused on learning progressions defined by means of construct maps.

Generally, an LP consists of several ordered levels or units, each of which represents a given state of KSAs required for a student to achieve mastery at that level. Major objectives in the study of an LP are to provide (1) information regarding the state of a student with respect to the level of understanding of a given concept and (2) diagnostic information regarding the strength and weakness of a student’s understanding along a curriculum (Gotwals et al., 2009; Schwarz et al., 2009; Briggs and Alonzo, 2012; West et al., 2012). To provide such information about the learning states of a student (i.e., the current, past, and prospective future levels of a student on LPs), the first step is to develop tasks for gathering student responses that provide evidence about students’ KSAs in relation to their levels on LPs. More specifically, if key task features that can evoke evidence about student states have been identified by drawing on research, then the information can be used for constructing tasks that can elicit student responses containing evidence about student KSAs. Through the assessment design framework, tasks are generated to reflect the targeted aspects of KSAs by incorporating the identified task features that evoke evidence about the KSAs or the targeted strategies. As an exemplar of theory-based task design, Embretson’s (1998) “cognitive design system” integrates the principles of cognitive psychology into task design.

In a study of LPs, Gotwals et al. (2009) presented a set of designed tasks linked to the LP of inquiry reasoning to gather evidence of how students use their content knowledge to formulate scientific explanations associated with a range of ecology, classification, and biodiversity domains. As an example, Table 1 shows a scenario for assessing the concept of biodiversity. Given the scenario, two tasks are generated relative to different levels of the LP of biodiversity. Both tasks ask the student to provide an answer and a rationale for their answer. Question A, relating to a lower level of the LP, asks students to identify which zone has the highest animal richness. The answer and its evidence are straightforward in the task because Zone B clearly has the highest animal richness. In contrast, Question B, related to a higher level, asks students to identify which zone has the highest biodiversity, given the same scenario. While the answer is the same as the previous task, providing appropriate and sufficient evidence supporting the answer is not as straightforward because students need to understand the difference between the concepts of the richness and the abundance of animals. Therefore, by using the key task features associated with the LP, a teacher obtains evidence about the level a student may have attained and what a student knows with respect to the LP domain.

Once tasks have been developed, another major issue is modeling the relation that links student performance on assessment tasks to their levels on the LP. Historically, measurement of proficiency change in accordance with development theory, cognitive psychology, and learning science has been a significant issue in educational and psychological research, such as Piaget’s (1950) stages of cognitive development, Siegler and Campbell’s (1989) multiple strategies in proportional reasoning ability of children, and Rock and Pollack-Ohls’s (1987) math learning as a dynamic latent variable consisting of a series of discrete stages. Various approaches in psychometric models have been proposed for addressing the measurement of proficiency change.

Historically, measurement of proficiency change in accordance with development theory, cognitive psychology, and learning science has been a significant issue in educational and psychological research, such as Piaget’s (1950) stages of cognitive development, Siegler and Campbell’s (1989) multiple strategies in proportional reasoning ability of children, and Rock and Pollack-Ohls’s (1987) math learning as a dynamic latent variable consisting of a series of discrete stages. Various approaches in psychometric models have been proposed for addressing the measurement of proficiency change. A suitable psychometric model requires certain properties for addressing learning progressions: (1) Observations involve student performances in task situations; most often, observables are categorical variables such as from selected response items or human or automated evaluation of more open performances; (2) a learning progression is operationalized as a latent variable with several ordered latent classes representing qualitatively different levels in the learning progression; and (3) there is a structural, generally probabilistic, latent-variable relation in an LP over time when the variable of interest is unobservable. (4) Task design and theory provide a theoretical framework for creating and modeling observable evidence as well as information about the nature and structure of expected change.

Some psychometric models matched to LP research have been proposed in Latent Class Analysis (McCutcheon, 1987), Rule Space model (Tatsuoka, 1990), Cognitive Diagnosis models (CDM, Leighton and Gierl, 2007), Mixture IRT (Sen and Cohen, 2019), Bayesian networks (Madigan et al., 1995), and hidden Markov models (Wiggins, 1955; Collins and Wugalter, 1992). Most research in CDMs and rule space modeling has focused on the classification at a given time point; the movements from one attribute at one point in time to others at the next point in time, as might be expressed in terms of transition proportions of skills, levels, and strategies between consecutive measurement time points, are not addressed.

Markov chain models can describe transition proportions of latent classes between consecutive time points. These models have been applied in situations such as attitude change, learning, cognitive development, and epidemiology (Langeheine and Van de Pol, 2002). Variations of Markov chain models (i.e., hidden Markov models, mixed hidden Markov models, and mixed hidden Markov models with multiple groups) have been proposed (Langeheine and Van de Pol, 2002). The models concern modeling change over time in observed categorical variables by using transition probabilities for unobserved variables. The hidden Markov model thus combines features of a latent class model and those of a simple Markov chain model. The model is also referred to as a latent Markov model, as proposed by Wiggins (1955) or a Latent Transition Analysis (Collins and Wugalter, 1992). This model has been applied to identify unobservable latent state changes such as strategies, levels, and skills based on observable student responses at each point in time.

The conditional probabilities and transition probabilities of a hidden Markov model correspond to the expression of a dynamic Bayesian network, or DBN (see Dean and Kanazawa, 1989, for an introduction to DBNs and Reichenberg, 2018, for a recent review of their use in educational assessment).

Dynamic Bayesian networks are an extended model of Bayesian Networks, which are a probability-based flexible statistical modeling framework rather than a specific statistical model. This framework supports reasoning and decision-making with uncertain and inconsistent evidence. BNs use graphical representation and linked probability theory (Kjaerulff and Madsen, 2007). The graph consists of nodes representing unobservable and observable variables and directed edges representing stochastic or logical relations among variables. The joint probability distribution of a set of (typically finite-valued) variables is represented recursively as the product of conditional distributions,

where the “parents” of a node A are the nodes with edges from them to A. If A_i has no parents, the conditional probability is regarded as a marginal probability. An ordering of the variables for the recursive expression is selected to take advantage of conditional independence relation. In particular, latent variables are typically modeled as parents of observable variables. The formal expression of the joint distribution is

A conditional probability table (CPT) for A_i has columns for each of its possible values, and rows for each possible combination of values of its parents pa(A_i) that are conditional probability distributions for A_i conditional on the given values for pa(A_i).

In assessment applications, evidence is received from external sources in the form of observable variables such as task responses or raters’ scores. A likelihood distribution over the states of its parent variables, typically but not necessarily latent variables representing KSAs, is induced by a value of an observable variable. Once all of interrelationships are expressed in terms of the recursive representation of the joint distribution of variables, it is possible to calculate the updated states of any variables effected by new information about another set of variables via Bayes’ rule. As the size of the collection of variables and the complexity of their interrelations increases, computational algorithms more efficient than definitional application of Bayes rule can be applied (e.g., Lauritzen and Spiegelhalter, 1988).

The preceding sections describe how LPs and ECD can provide a coherent, theory-based assessment design framework in formative assessment, and how Bayesian networks can be used to manage issues of evidence and inference. This section describes how DBNs can be used to model learning progressions over multiple time points in such a system. Specifically, it addresses the questions of how the current, past, and future levels of a student’s LPs are related and can be inferred from student responses. We note that DBNs are an extension of the hidden Markov model to represent multivariate latent and discrete spaces. Taken together, the discussions explain how the DBNs can model LPs over time by connecting defined LPs, assessment design, and the interpretation of student performances.

In this study, DBNs can be used to model learning progressions over multiple time points under a formative assessment system. DBNs have been applied to intelligent tutoring systems for modeling changes in students’ knowledge. As examples, Reye (1996, 1998) used dynamic belief networks in the intelligent tutoring system. Chang et al. (2006) developed a Bayes net toolkit for student modeling in an intelligent tutoring system. Almond (2007, 2010) extended ECD to incorporate DBNs, in terms of a partially observable Markov decision process (POMDP). As an example, Almond used the POMDP to model student growth in a study of simulated music tutoring program. DBNs thus offer a psychometric model to link the theory embodied in a learning progression and tasks that provide evidence about a student’s level on that progression through ECD assessment design framework. That is, DBNs can support inference about students’ levels and expected change in an LP over time when the variable of interest is unobservable, when task design and theory provide a theoretical framework for creating and modeling observable evidence as well as information about nature and structure of expected change.

Dynamic Bayesian networks are an extension of hidden Markov models (HMM) that is used to represent multivariate latent and discrete spaces. The elements and structural relationships in DBNs correspond to a standard algebraic expression of HMMs through the concept of Markov property and conditional independence. An HMM is comprised of a Markov chain and observables (Cappé et al., 2005). A Markov chain is a sequence of discrete random variables with the Markov property. The term “hidden” refers to the fact that variables of the Markov chain are latent; that is, their values are never directly observed, although evidence about them arrives in the form of observable variables that depend on them stochastically. Three kinds of distribution parameters are then to be estimated: (1) the initial multinomial state distribution, P(X_t=1) in a DBN; (2) the transition model, set of conditional multinomial distributions that represent the transition probabilities P(X_t | X_t–1); and (3) the observation model, which corresponds to the conditional probability distributions of observables Y_t, or P(Y_t | X_t) in the DBN.¹ The formal probabilistic notation of the hidden Markov chain in the HMM is denoted as follows:

with P(X₁|X₀) interpreted as P(X₁). Under the assumptions of conditional independence and the first-order Markov property, the observations {Y_t} are independent given the states of a hidden Markov chain {X_n} at all time points but X_t:

Dynamic Bayesian Networks thus extend static Bayesian Networks (BNs) to model probability distributions over multiple time points (Murphy, 2002). A common approach to representing DBNs is to combine multiple static BNs for a desired number of time slices (Kjaerulff and Madsen, 2007). Therefore, the DBNs can be used for inference about previous states, current states, and possible future states of a system over time (Murphy, 2002). A DBN contains a prior for the initial hidden state, P(X₁), a transition function of the hidden states over multiple time points, P(X_t | X_1:t–1), and observable variable(s) given each hidden state, P(Y_t | X_t).

This discussion assumes three properties. First, the links of time slices are defined by the conditional probability of the variables at a current Time t given the variables at previous Time t-1, the first order Markov property. Second, observations are structured under assumptions of conditional independence, in that P(Y_t) is conditionally independent of P(Y_t‘), given X_t for t ≠ t‘. Third, the sense of “dynamic” refers to state change over time, not network or structure change over time. Under the three assumptions, the formal notation of DBN at Time t can be expressed with respect to a graphical model (Murphy, 2002) as

where A_t = (X_t, Y_t), incorporating the latent variable (X) and the observation (Y) at time t and Ati is the i^th element of this concatenation. The joint probability distribution is then as follows:

Again it is possible to calculate the updated states of any variables as affected by new information about another set of variables through Bayes theorem, directly or by more efficient algorithms.

In Bayesian networks with discrete variables, Bernoulli and categorical distributions are used for the conditional probability distributions. Their parameters are incorporated into the full model and can also be learned from the observations, using Beta and Dirichlet distributions as conjugate priors for them (see Almond et al., 2015, Chap. 8, for more parsimonious prior distributions that can be employed when substantive theory is available). Bayesian inference is based on posterior distributions of variables of interest, obtained by combining the prior distribution on the variable(s) of interest with the likelihood induced by observations through the appropriate conditional distributions. A common point estimate is the maximum a posteriori (MAP) estimate; that is, the value that yields the maximum posterior probability for an unobserved variable given the realized data. In LP research, the values representing the levels of LPs are not directly observed. The MAPs for an LP variable are the levels that are most probable for each student given their responses.

Expectation and Maximization (EM) algorithms, gradient ascent, and Markov chain Monte Carlo Estimation (MCMC) are commonly used in BN software programs to estimate the values of the parameters of the distributions of the latent variables. This study uses the EM algorithm (Dempster et al., 1977) as implemented in Netica (Norsys Software Corp, 2010) to estimate parameters of DBNs.

As an illustration of DBNs, we suppose that the same students are repeatedly measured at more than one point during a period of instruction (e.g., a course in a semester or an intervention). The tasks used are designed by incorporating task features with which students can be differentiated in terms of levels of understanding or achievement that are theoretically grounded in a substantively based learning progression theory. In such situations, the investigation of the patterns of change in student levels across measurement occasions can offer diagnostic information customized to reflect individual learning and provide an informative evaluation of the effectiveness of instruction. For a simple example of modeling LPs through the DBNs, suppose that four measurements are designed. At each time point, there is a latent variable representing an LP and the observables that depend on them in probability. It is assumed that three levels are identified in the LP. Each measurement consists of six tasks across time points. Figure 1 shows an example of modeling LPs with a DBN.

In this example, the probabilities of the initial marginal probability table of the first measurement, CPTs, and transition probability tables are hypothetically set in order to illustrate the structure of DBNs. They could be estimated from observations or determined through theory or domain expert opinions. In this example, the hypothesized transition probabilities are restricted in such a way that all the probabilities of reverse changes are zero. This constraint reflects an LP considering only forward movements over time. Other types of transition probability patterns can be considered depending on different substantive theory such as a forward movement, an adjacent movement, and all possible movements. The different patterns can be modeled by (1) constraining sets of transition probabilities to be equal to zero, (2) restricting them to be a particular value, or (3) fixing them to be equal to each other such as a forward movement, an adjacent movement, and all possible movements.

To understand how the transition function works for the purpose of investigating state change over time, one could consider a situation where student status at the first measurement occasion is known. This information is propagated through the network by Bayes theorem. The posterior distribution of the next three variables given the student’s states at the first measurement occasion can be updated by using the transition function (see Figure 2, which shows an initial probability table at the first measurement occasion and transition probability tables). Figure 3 shows the posterior distributions of three variables (i.e., LP_Time2, LP_Time3, and LP_Time 4) given a student latent Level 1 at the first measurement occasion. The grayed-out coloring of the first node indicates that the value of that variable is known with certainty at the point in time at issue. It can be inferred that the student is most likely at Level 2 at the second measurement occasion with 0.40, at Level 3 with 0.39 at the third measurement occasion, and at Level 4 with 0.63 at the fourth measurement occasion, respectively.

The next step considers the DBNs with tasks. The conditional probability distribution for Task 1 given the first measurement occasion is shown in Table 2 as an example; note that this is a task that is most useful for differentiating students at Level 1 from Levels 2–4. Once a student’s responses have been observed at any given time point, this information is propagated through the network. The posterior distributions of the latent variable at that time point as well as the latent variables at previous and future time points are obtained. Figure 4 shows a situation in which observations are [1,1,0,0,0,0], indicating that Tasks 1 and 2 are correct while Tasks 3, 4, 5, and 6 are incorrect at the first measurement occasion.

Based on these observations, the posterior distributions of the four latent variables are updated. Note that the first six Task variables are shaded, indicating that their values are known, while the node for LP_Time1 is not shaded because knowledge about its value is improved by the information in the responses but is still not known with certainty. This shows that the student is more likely to be at Level 2 at the first occasion with 0.74, at Level 3 at the second with 0.46, at Level 4 at the third with 0.54, and at Level 4 at the fourth with 0.83 when the student has the particular pattern of responses to the six tasks at the first measurement occasion.

An application study was conducted with data from a blended online-classroom course provided by the CNA. Each course contains several chapter exams and a final exam. The same students were measured several times over the course of the curriculum. The target populations of the courses are high school students, 2- and 3-year community college and technical school students, and 4-year college and university students. The study addressed a learning progression concerning IP (Internet protocol) addressing. The 26 tasks are modeled as conditionally independent observable variables on a single discrete latent variable with values that indicate four levels of the LP. The student sample size was 1,450. A DBN was constructed and observations collected on two occasions approximately one month apart.

The learning progression for IP Addressing skills was identified as a subset of larger pool of assessment items, based on a combination of confirmatory discrete factor analyses and subject matter experts’ opinion (as further described in West et al., 2010, 2012). The analyses were conducted using Mplus (Muthén and Muthén, 2006) and the fit of competing models was carried out with the BIC and the bootstrapped likelihood ratio test (McLachlan and Peel, 2000; Nylund et al., 2007). Table 3 shows the four levels in the resulting LP. Each level contains the descriptions of KSAs, where Level 1 can be defined as describing novice students that possibly have pre-course KSAs.

The Conceptual Assessment Framework in the ECD framework provides technical specifications for the evidentiary arguments that the operational assessment embodies. Modeling the LPs for IP Addressing is organized around five guiding questions (see Mislevy et al., 2003a,b, for further details in the general case, and Zalles et al., 2010, for the case of learning progressions):

(1) Construct Model: What complex of knowledge, skills, or other attributes should be assessed?

The construct model variables can be specified by the aspects of KSAs associated with levels, progress, levels of learning progressions, and diagnostic information. A construct model variable can represent either an LP of a particular domain when the levels are ordered, or as a level of a particular LP when the ordering may be partial. The construct model shows that the levels are defined for assessing the LP in the domain and information regarding the KSAs that are required for students at that level. In addition to this information, the structure of the construct model variables is also specified. For this LP, based on expert opinion and the preliminary analyses, a hierarchical relationship among the construct model variables is posited to be an adequate structure. We used two instances of the same four-level IP Addressing LP variable for the two occasions. The LP variable at the first measurement time point influences the second LP variable at the second time point. This construction becomes the representation of latent variables in a DBN (Figure 5). The construct model is connected with the task model through the evidence model, which explains how each observable depends on the construct model variables.

(2) Evidence Model: What behaviors or performances should reveal those constructs? How are they connected?

For the study of LPs, the evidence model provides (1) information about how student performances are modeled and interpreted relative to the level of an LP, (2) information about the criterion for comparing observed and expected LPs, and (3) information about feedback within and across task level. Consequently, the evidence model provides inferential reasoning from observables of tasks and expectations in the student model. This study illustrates these relationships with a DBN, one of the suitable psychometric models for modeling LPs. The DBN of this LP model after estimating the CPTs from the data set of 26 items and 1450 students is shown in Figure 5 in the results section. Task model: What tasks or situations should elicit these behaviors?

For assessing LPs, the task model provides information for developing tasks to elicit student performances relative to the levels of a learning progression. Specifically, it contains the following information: (a) the key features of tasks that are important to elicit student’s understanding with respect to the targeted KSAs at a particular level of an LP, (b) the key features of tasks, which are more likely to classify student performances into different levels of an LP, (c) the key features that make a task more or less difficult, (d) other characteristics/contexts of a task that affect its difficulty, and (e) the aspects and features that inform the quality of tasks for assessing LPs. By designing assessment tasks that target different levels associated with different aspects of targeted KSAs, it becomes possible (a) to infer the level of the LP a student may have attained, (b) to draw conclusions about the value, sequence, and structure of a student’s learning, and (c) to gather empirical evidence to guide the development and refinement of the hypothesized LPs associated with assessment and curriculum. West et al. (2010, 2012) identified the features of tasks relative to different levels of the LP in IP Addressing Skills. As an example, two tasks in Figure 6 require different levels of IP Addressing skills to obtain the correct answer. The two tasks look similar on the surface, but the stem of Task A is/24, while that of Task B is/28. This change requires students to use a more advanced IP addressing skill (specifically, reasoning across octal boundaries; West et al., 2010, 2012).

Appropriate and sufficient evidence with respect to levels in LPs can be obtained by incorporating task features associated with the key aspects of knowledge and skill required of students to complete the tasks. This study used tasks identified by their features relative to levels of the IP Addressing Skills progression. LP1_Task1 to LP1_Task7 and LP2_Task1 to LP2_Task3 are the tasks identified by the features relative to Level 2 of the IP Addressing Skills progression. LP1_Task 8 to LP1 Task 14 and LP2 Task 3 to LP2_Task 5 are the tasks identified by the features relative to Level 3 of the progression. Lastly, LP1_Task15 to LP1_Task17 and LP2_Task6 to LP2_Task9 are the tasks identified by the features for assessing Level 4.

(3) Assembly model: How much do we need to measure?

For assessing LPs, the assembly model describes how the three models above are combined for inferring a student’s level on a learning progression in a given assessment situation. For instance, the number of tasks (i.e., test length) with respect to the different levels on an LP and the balances of task type and focal KSAs are determined to construct an optimal assessment.

(4) Presentation model: How does the assessment look?

The presentation model describes how a task is presented to students. There are many different means for delivering an assessment, such as paper and pencil format, computer and web-based format, and simulation- and game-based format. The requirements for presenting assessments differ depending on the format. Formative assessment can also be delivered using mailing/e-mailing, messaging platforms such as Messenger and WhatsApp, and online educational platform tools, such as Questbase and Woot Math. The presentation model also contains specifications for the interactions between the student and the assessment, and the work products that will be captured for subsequent evidence identification.

Formative assessments are increasingly of interest in the field of education. A formative assessment system provides information for teachers and students about whether specific learning goals are being reached, and what is needed to master a given concept (Bennett, 2011; McCallum and Milner, 2020). For this purpose, an assessment must produce evidence for revealing student levels and their change over time (Black and Wiliam, 1998; Bennett, 2011). While substantive, psychological, instructional, and task developmental aspects of formative assessment have been considered, few assessment design frameworks have been proposed to link the theory embodied in a learning progression, tasks that provide evidence about a student’s level on that progression, and psychometric models that can link them. More specifically, few statistical analytic models have been studied to characterize the relationship between student performance and levels on learning progressions under a formative assessment system. In this study, we review a coherent assessment design framework for modeling learning progression under a formative assessment system using an ECD framework. Then we describe how DBNs can be used to addresses the question of how a learner’s current, past, and future levels in learning progressions are inferred under a formative assessment. Finally, we conduct an application study of DBNs using real data from the domain of beginning computer network engineering drawn from an online formative assessment in the CNA. Consequently, this study describes a design framework and learning analytics method for measuring students’ advancement along learning progression in a formative assessment system.

Dynamic Bayesian Networks are a useful statistical modeling method that can support inferences about level change over time when task design and theory provide not only a theoretical framework for creating and modeling observable evidence, but also information about the nature and structure of expected change. DBNs provide real-time updating of estimates for student levels during instruction, so that they offer beneficial information to students, instructors, and curriculum developers for enhancing student learning (Almond, 2007; Arieli-Attali et al., 2019; Attali and Arieli-Attali, 2019).

In addition, DBNs can serve as a psychometric model in research related to learning progressions. LPs inform the state of a student with respect to their level of understanding of a given concept and diagnostic information regarding the strengths and weaknesses of a student’s understanding along a curricular strand. However, challenges have emerged in LP research, including (1) designing a coherent assessment system, (2) inferring student learning progression levels based on the responses to assessment tasks, and (3) interpreting the difference between expected and observed students’ progress mapped to the conceptually defined learning progression. An ECD Framework can be a useful tool for modeling learning progressions by linking among the theory embodied in a progression, tasks that provide evidence about a student’s level on that progression, and psychometric models that can characterize the relationship between student performance and levels on learning progressions. Bayesian Networks can work as a psychometric model that can provide real-time updating of estimates for a student learning progression.

Furthermore, BNs can help lead to a valid task design. BNs confirm the levels and progressions by comparing the results from data analysis, allowing task designers to specify the levels of KSAs at which they are aiming assessment tasks. This helps make task design more principled, better connected with targeted inferences, and ultimately more valid. Lastly, BNs also help connect curriculum to assessment. For example, curriculum designers can take information from a BN structure and make decisions about which content areas are more important to emphasize so that students will have a greater probability of mastering future KSAs (West et al., 2010, 2012).

Although BNs provide a promising means for modeling LPs under an assessment system with formative and summative components, there are some issues that need to be considered in the future study. First, since BNs are a flexible statistical framework, there are many decisions that must be made when designing a BN. The structure of the relation among variables must be determined. The structure can be determined by communicating with content experts as well as by using statistical methods such as structural equation modeling. In addition, one may need information about the probability distributions of variables in the BN as prior information. While the values of the variables of the probability distributions can be determined from data, the structure can be posited before data are collected (although it may be revised in light of subsequent data; see Levy, 2006). Therefore, it is important to keep in mind the purpose of the assessment in order to determine which relations are important to model. This process is not always straightforward and may require some iterative work before the model can be said to be good enough (Almond et al., 2015). Second, this study considered a first order Markov property in the transition probability. The designing of DBNs with more than two measurements with a higher order Markov property could reflect a more realistic educational setting.

Third, this study was initially designed with one LP variable corresponding to a domain. However, more than one LP reflecting more than one domain may be of interest to be modeled in future studies. For example, tasks can be reflected by more than one LP and students might have different learning patterns in terms of multi-LPs representing different domains. A student may require a higher level on one LP while additionally requiring a lower level on the other LP to complete a given task in the domain. In the case of multiple LPs, different learning paths along the multi-LPs can be modeled over time (Wilson, 2012).

Fourth, this study considered only the forward transition movement in the simulation studies because the forward transition movement is the most appropriate structure for representing student movements along an LP. This is a strong constraint but a typical hypothesis and appropriate for a preliminary study of DBNs for modeling LPs. However, there can be different types of transition structures, such as selected backward transition movements and all-transition movement. It is left to future studies to investigate the selection of the best fit transition movement structure for a particular content area, student population, and instructional program.

Lastly, the data example used to illustrate the DBM used only discrete-coded responses of conditionally independent tasks. The same LPs can be used to describe targeted KSAs, task demands, and student performance for more complex tasks such as the simulation-based troubleshooting and design tasks as are employed in CNA in the Packet Tracer environment (Williamson et al., 2004).