This study adopts a situational understanding of diagnostic competence, meaning that difficulties and errors must be perceived and addressed in the moment. Such a perspective aligns with the concept of situation-based diagnostic competence which is understood as a continuum according to Blömeke et al. (2015a) with the specific content of diagnosis (Hoth et al., 2016). This includes “cognitive abilities and affect-motivational skills […] as well as the skills to perceive relevant diagnostic information during class, to interpret these aspects and to decide how to respond” (Hoth et al., 2016, p. 43), which leads to an observable action by the teacher.

For analyzing diagnostic behavior in error situations, the model of diagnostic competence in error situations according to Heinrichs (2015) serves as central framework. It follows a three-step process: (1) perception of an error – recognizing deviations from mathematical norms; (2) analysis – identifying possible causes to develop a deeper understanding; and (3) decision-making – choosing an appropriate response (Heinrichs, 2015; Heinrichs and Kaiser, 2018).

In mathematical modelling, diagnostic competence represents a core teaching competence, as identifying and interpreting students’ difficulties and errors serves as a key requirement to provide appropriate support (Blum, 2011; Borromeo Ferri, 2018). Wess et al. (2021) further conceptualized these abilities as modelling-specific pedagogical content knowledge, complemented by beliefs and self-efficacy.

According to Leiss (2007), diagnosing the type and cause of a difficulty in the modelling process forms the basis for selecting an appropriate intervention, which is comparable to Heinrichs’ (2015) model of diagnostic competence in error situations. The identification of the type of a difficulty can be compared to perceiving an error, while there is an analysis of the causes and a decision for an action in both approaches. Building on Leiss (2007) definition of adaptive interventions, Klock and Siller (2020) derived four criteria for evaluating teacher interventions in modelling situations. First, a (potential) difficulty needs to be diagnosed. Since modelling tasks can often be solved using multiple approaches, adaptive interventions require that the current solution path or difficulty of the learners be specifically addressed in the intervention. Additionally, since learners should work as independently as possible when modelling, an adaptive intervention should only affect the solution process of the learner to a minimal extent and should preserve the learners’ independence. To this end, Zech (1998) distinguishes between different levels of support for learners, ranging from motivational to strategic to content-related support, representing increasing degrees of teacher guidance.

Türling et al. (2012) investigated situational error perception among pre-service teachers (PSTs) and in-service teachers (ISTs) in accounting using video vignettes and interviews. They found that PSTs showed a relatively low ability to perceive and correct errors, whereas ISTs performed significantly better. In contrast, Cooper (2009) found that PSTs in mathematics could identify students’ written errors and error patterns accurately. For modelling, Alwast and Vorhölter (2022) observed large individual differences in PSTs’ perception of student difficulties in videos with regard to the number of perceived difficulties – a finding also confirmed for outdoor modelling (Buchholtz, 2024).

Although cause diagnosis is crucial for dealing with errors successfully (Borasi, 1996; Loewenberg Ball et al., 2008), different studies show that both PSTs and ISTs often lack this skill. This is evident in the relatively low depth of error analysis or interpretation of students’ difficulties (Alwast and Vorhölter, 2022; Larrain and Kaiser, 2022; Seifried and Wuttke, 2010). Heinrichs (2015) defines a high level of cause diagnosis as the ability to specify multiple potential causes for a specific error. In modelling contexts Alwast and Vorhölter (2022) also emphasized the aspect of using observed indicators for an analysis of difficulties. Qualitatively weaker cause diagnoses merely describe the error or refer to a very general cause without reference to the specific error (e.g., lack of attention in the teacher’s explanation) (Larrain and Kaiser, 2022). A diagnosis of the cause of an error or difficulty can involve questioning the students about their conceptual understanding or their solution strategies (Shaughnessy et al., 2021; Klock and Siller, 2020). However, empirical studies have shown that both PSTs and ISTs often overlook this step and move directly to error correction (Seifried and Wuttke, 2010; Shaughnessy et al., 2021).

Several studies have also identified different strategies of (pre-service) mathematics teachers for handling students’ errors (Benecke and Kaiser, 2023; Heinrichs, 2015; Heinze, 2005; Matteucci et al., 2015; Oser et al., 1999; Santagata, 2005; Tulis, 2013). Approaches such as teachers ignoring or immediately correcting errors are considered non-productive (Brodie, 2014; Gardee and Brodie, 2022; Santagata, 2005) and have been termed outcome-oriented, as they focus on quick correction without deeper analysis (Rach et al., 2012). This was also evident in one third of PSTs responses to errors in a study on modelling tasks (Cai et al., 2022). In contrast, a process-oriented approach focuses on error analysis and students’ understanding, which is why this approach is considered to have learning potential (Rach et al., 2012). In modelling, process-oriented interventions correspond to addressing the learners’ specific difficulties. Still, studies show that teachers often unconsciously impose their own preferred solution methods on students (Blum, 2011; Blum and Borromeo Ferri, 2009). With regard to the type of intervention different results were found. While Stender’s (2016) research highlighted the prevalence of strategic interventions, Leiss (2007) found predominantly content-related interventions. For outdoor modelling, Buchholtz (2024) found a preference among PSTs for instructional interventions in error situations.

Open-ended responses were analyzed using qualitative content analysis (Kuckartz, 2018). Approximately 25% of the data were double-coded, yielding a Cohen’s kappa of 0.88 (Brennan and Prediger, 1981).

In the first step, responses were coded for the types of errors (e.g., calculation error), error causes (e.g., lack of knowledge), and error handling strategies (e.g., explanation) described by the PSTs. If different errors, causes or strategies were mentioned, multiple codes were assigned. In a second step, the responses to the error causes and error handling were characterized by additional categories. The described error causes were characterized by the depth of error analysis. For each error perceived it was looked at whether there was no cause, an impossible cause, only a further description of the error (Larrain and Kaiser, 2022), a general cause (Larrain and Kaiser, 2022), an error-specific cause, several causes for one error (Heinrichs, 2015) or a justified cause (e.g., with indicators of the video) (Alwast and Vorhölter, 2022) described in the error analysis. The error handling was further characterized by whether the suspected cause of the error was verified by diagnostic actions (Seifried and Wuttke, 2010; Shaughnessy et al., 2021) and whether the error handling interventions were process-oriented or outcome-oriented (Rach et al., 2012).

To provide an overview of the beliefs held by the sample group, a confirmatory factor analysis (CFA) using Mplus (version 8.8) was conducted to verify the two-dimensional structure of beliefs about teaching and learning for this specific sample. Model fit was assessed via the χ²/df ratio and global fit indices (CFI, RMSEA, and SRMR). An acceptable model fit is defined by χ²/df < 3 (Schermelleh-Engel et al., 2003), CFI ≥ 0.90 (preferably ≥ 0.95), RMSEA < 0.06, and SRMR < 0.08 (Hu and Bentler, 1999).

The qualitative and quantitative data were integrated following a multi-step procedure. First, qualitative data were analyzed for patterns conceptually aligned with specific belief items. Four items were identified as relevant to these patterns (see Section 4.1). In a second step, all participants’ quantitative responses to these items were analyzed for particularly high or low agreement. Values were considered notable if they deviated by at least ± 1 SD from the item mean. For each participant, both patterns in the qualitative and notable agreement in the quantitative data were documented. In a third step, it was examined for each participant whether qualitative patterns found and notable agreement to items match in content. The extent of correspondence between qualitative and quantitative findings was then determined. A low, medium, or high relation was assigned depending on the ratio of matching features across both data sources to the total number of identified features.

This integration procedure is explained by the following example. Step 1: In the qualitative answers of PST 5, one pattern was found, as the lack of knowledge of formulas was mentioned as a cause of errors in two vignettes, indicating that formulas play a particular salient role for this PST. Step 2: In the quantitative data this participant showed a strong agreement to the transmission-oriented items 1 [formulas], 6 [correct answer] and 9 [non-standard procedures] (see Section 4.2.2), resulting in a total of four notable features in the qualitative and quantitative responses. Step 3: In both data sources a notable feature regarding formulas was found and are therefore matching in content. For the other two items no corresponding qualitative pattern was identified. Consequently, two of the four identified notable features are matching across both data sources, which was classified as a medium relation between this PSTs’ diagnostic competence and their beliefs.

Because the errors differed by vignette, the categories for the first diagnostic process are vignette-specific. In vignette 1 (circumference), two target errors were present: an error in the formula (treating the circumference as 2 · r) and an error in the assumption of the “meter step.” Both were frequently reported in the PSTs’ responses (56 and 71 mentions). Many PSTs, however, concentrated on other aspects in the application of the step strategy (45 mentions), such as “obstacles in the way” (PST 14), “increasing the radius by walking outside the fountain” (PST 44), “taking steps of varying lengths” (PST 102) or “counting errors” (PST 174). Additionally, 27 participants problematized the step strategy itself as an error (e.g., PST 13 below). Overall, the distribution of mentioned errors in the two error situations showed that the PSTs mainly focused on errors regarding the step strategy (total of 143 mentions) rather than the formula.

PST 13 sees using the formula as the “correct approach,” while using the steps is a “mistake,” as no “exact result” can be achieved with this alternative strategy. Such responses were not considered as appropriate error perception in this outdoor modelling context, since the learners’ strategy indeed may lead to an acceptable result with more accurate assumptions or actual measurement of a step and illustrates a typical strategy of working on the object with physical body measurements.

Looking at the focus patterns of the participants, the results showed that 41 PSTs (24.8%) mentioned both the formula error and an appropriate aspect of the step strategy as being flawed (e.g., PST 113 below). In 8 responses (4.8%) the focus lay on the formula error and on the step strategy itself.

By contrast, 101 PSTs described an error in only one area, as 75 participants (45.5%) mentioned only issues in the step strategy, 19 PSTs (11.5%) considered the strategy itself as the sole error, while in 7 responses (4.2%) exclusively the formula error was described. 15 participants (9.1%) perceived other aspects to be an error. For example, PST 75 states that “the students thought they needed the diameter, but this is not the case. The radius is enough to calculate the circumference of a circle” and concludes that “this shows that they are not aware of how the circumference of a circle is calculated”. In this case it is evident that the PST’s incomplete mathematical knowledge guided the perception of errors, emphasizing the relevance of teachers’ knowledge to interpret students’ solution processes adequately.

Based on how many and what aspects were described as errors by each PST, we interpreted the quality of their responses. Answers which contained multiple relevant aspects (e.g., PST 113) show a higher quality, while responses like the ones from PST 13 and 75 imply a lower quality as they do not focus on productive aspects.

In the second vignette (area), PSTs predominantly described calculation errors and errors regarding the value of the radius (94 and 106 mentions). The error in reading the formula was mentioned less often (40 mentions). 25 responses (16.7%) captured all three errors; 55 PSTs mentioned two (36.7%), 55 one of the errors (36.7%), and 15 (10%) perceived none.

In the third vignette (volume), the most frequent category was the use of the elimination or estimation method (96 mentions), which we did not treat as an appropriate error perception, as this strategy could lead to the right result, if the height of the fountain would be considered. The relevant errors, which related to confusing key concepts (25 mentions) or incorrect scaling of the result (33 mentions) were identified much less frequently. Only 6 PSTs (4%) described both relevant errors (three also flagged the use of the elimination method), 46 (31.1%) noted one relevant error (18 additionally mentioned the use of the elimination method), 75 (50.7%) flagged only the elimination or estimation strategy, and 21 (14.2%) focused on other aspects.

For all three vignettes, we found that PSTs focused on different aspects and that the number of perceived relevant errors varied among PSTs, which can indicate different levels of quality in PSTs’ error perception. Especially in vignette 3, the participants’ attention was primarily on the alternative strategy of estimating the result. Notably, some PSTs treated non-standard strategies (step counting; elimination/estimation) as “errors,” which is undesirable in modelling contexts. Therefore, potential links to beliefs were examined via item 2 [exact procedures] and item 9 [non-standard procedures] in order to answer RQ2 (see Section 4.2).

For cause diagnosis, categories applied across vignettes, but different focal points emerged between the vignettes. For example, strategic (63 of 127 mentions) or metacognitive aspects (34 of 56 mentions) such as “not using the tools provided” (PST 33) or “not taking enough time to reflect on and develop their idea” (PST 110) are primarily used as explanations for errors in the vignette about the circumference. PSTs also cited incorrect assumptions (15 of 27 mentions) like the “assumption that one step equals one meter” (PST 177) or external influences (24 of 30 mentions) such as “the water being an obstacle that makes it difficult to measure the diameter/radius” (PST 146), “the size of the fountain” (PST 19 and 136) and “several people standing in the way” (PST 89) as obstacles. Here, it is apparent that characteristic aspects of outdoor modeling such as assumptions or metacognition, but also the specific circumstances on site, were highlighted by the PSTs.

In comparison, in vignettes 2 and 3 (area/volume) the emphases shifted to content-related or affective error causes. Lack of knowledge (76 and 75 of 190 mentions) was the most frequently mentioned cause here. Moreover, lack of understanding (68 of 126 mentions) and confusing mathematical concepts (19 of 32 mentions) like “confusing squaring with doubling” (PST 104) were increasingly mentioned as causes in the area calculation vignette. Motivational aspects (51 of 107 mentions) and teacher-related factors (26 of 44 mentions) such as “providing various possible solutions” (PST 152) were mostly cited as reasons for errors in the volume calculation. These patterns plausibly track the modelling steps that are needed in the respective task (e.g., only working mathematically in the second video) and the type of errors made.

The given causes also varied with regard to their quality. For example, PST 141 attributed the error of using an “inaccurate and varying step size” (PST 141, perception) to “missing knowledge of the term radius or the formula” (PST 141, analysis). While the error is located in the execution of the step strategy, the proposed cause does not relate to that and was coded as an impossible explanation (37 codings) for the perceived error and indicates a low analytical quality. Low quality was also evident, when no cause (53 codings) or merely a further description of an error was provided (68 codings). Altogether, 158 low-quality causes were identified across participants and vignettes. Responses offering a general cause (238 codings) like, “lack of formula knowledge” (PST 5), “lack of motivation” (PST 103) or “insufficient understanding of mathematics” (PST 14), were considered to be of slightly higher quality. Such explanations provide possible causes, e.g., for the formula error or the calculation error, but lack specificity, as these could apply to multiple types of errors. Among general causes, the most frequent references were to knowledge (123 codings), motivation (52) and understanding (32). Higher-quality analyses were found when PSTs formulated specific and plausible causes, such as:

This explanation regarding students’ understanding of mathematical concepts could be specifically attributed to the error of the incorrect radius and therefore was coded as error-specific cause (190 codings). The most common specific causes referred to understanding (53 codings), strategic aspects (39), knowledge (34) and metacognition (23). In some cases, PSTs proposed multiple causes for a single error (26 codings) or justified their cause hypothesis using evidence from the video (28 codings, e.g., PST 91), indicating a deeper diagnostic reasoning.

PST 91 attributes the cause of the error in the step strategy to the unreflected imitation of another student, demonstrating a justified cause supported by video indicators.

In summary, the quality of cause diagnosis varied substantially. Most responses were of lower quality, offering general or even non-existent causes (396 of 640 causes), while only 54 instances reflected more advanced analysis through multiple or justified causes. Moreover, the proportion of modelling-specific versus content-related causes differed between vignettes, depending on the modelling steps involved.

Across all vignettes, lack of formula knowledge emerged as one of the most frequent causes of errors (over 120 mentions). 35 PSTs (21.2%) mentioned this cause in at least two of the vignettes, indicating that formulas play a particular salient role in how PSTs interpret errors. Consequently, the relation between this pattern and PSTs’ beliefs was further examined in RQ2, specifically using item 1 [formulas] (see Section 4.2).

Across vignettes, PSTs’ approaches to error handling could be characterized into general-strategic hints, content-related-strategic hints, and content-related hints (142, 187, and 69 mentions). For example:

While strategic hints appear to promote student-focused and independence-preserving interventions, many PSTs also provided teacher-centered and controlling interventions, such as directly naming or correcting errors (49 codings), supervising the whole solution process (21 codings) or frequently offering teacher explanations, mainly in vignette 2 (74 of 140 codings, e.g., PST 8). For instance:

PST 8’s approach reduces learners’ independence, while PST 18’s content-related hint supports guided exploration. Similarly, PST 118’s intervention is highly controlling, guiding students toward a formula-based solution by providing the formula and asking about the procedure rather than supporting their initial approach:

Such strategies like guiding the students to an exact procedure when they chose another approach (step strategy or elimination) were reported by 49 PSTs (29.7%) and are considered undesirable in modelling contexts, as they suppress exploration. Such responses offer another indicator for the influence of the belief items that deal with exact or non-standard procedures (items 2 and 9), as these PSTs prefer to work with an exact procedure instead of the learners’ alternative approach.

Moreover, responses like PST 118’s were characterized as outcome-oriented (235 codings), focusing on producing correct answers rather than understanding the underlying reasoning. In contrast, PST 160 is trying to get insights into the students’ thought process and attempts to make the error tangible through physical activity. So, in this case a diagnostic action and a process-oriented approach (194 codings) with efforts to clarify the error were evident, which is adaptive in terms of diagnosing first and addressing the current problem of the learners.

Diagnostic actions like asking about the solution process or “asking the students to show the radius, diameter, and circumference at the fountain itself” (PST 79) to check their conceptual understanding were only used in 56 responses of 39 participants (23.6%). The use of the object for explanations or physical actions to clarify errors occurred in 68 instances, particularly in the circumference vignette, highlighting a context-specific form of dealing with specific errors unique to outdoor modelling.

Overall, PSTs’ error handling reflects both strategic, autonomy-supportive interventions, which preserve the learners’ independence, and teacher-centered, corrective interventions, such as guiding the students to an exact procedure. It was also noticeable that certain characteristics, such as explanations or physical actions, were mentioned particularly in connection with specific errors or vignettes. Thus, the nature of error situation appears to influence the chosen intervention. However, many PSTs focused more on correcting outcomes and getting the right result than on diagnosing students’ thinking, resolving the errors and promoting understanding.

Indeed, 77 PSTs (46.7%) exhibited an outcome-oriented approach in at least two vignettes, suggesting that achieving the correct result outweighed fostering conceptual understanding. Consequently, item 6 [correct answer] was examined further in RQ2 in order to investigate possible relations with beliefs.

The error-handling response of PST 174 consisted mainly of task-completion aids rather than diagnostic interventions, reflecting an outcome-oriented approach. In the quantitative data, high agreement with item 1 [formulas] was found for this PST, as the individual agreement of 4 is higher than the item-specific threshold of 3 (Table 4).

However, no overlap was found for these notable features, as no high agreement in the content-related item 6 [correct answer] was found for the pattern of outcome-orientation, while the corresponding qualitative pattern for the high agreement in the formula item could not be identified either. Thus, no relation between diagnostic competence and beliefs was found for this participant.

PST 5 described lack of knowledge of formulas as a cause of errors in two vignettes (Table 5), indicating strong alignment with item 1 [formulas]. This was confirmed as the score of 5 shows a high level of agreement to item 1. The same participant also showed high agreement with items 2 and 6 [exact procedures and correct answer] (Table 5), though corresponding qualitative indicators were absent.

PST 78, in contrast, consistently favored exact procedures or avoided alternative procedures. This PST viewed the step strategy in determining the circumference as an error and guided students toward formula use, emphasizing correctness over exploration. A similar procedure can also be observed in the error handling in the water volume-vignette, where the error is corrected by the teacher in order to discuss the solution process of the task. These descriptions indicate that this PSTs error handling is outcome-oriented. In the belief items, high agreement of 5 was found for item 2 [exact procedures], but not for the item on outcome-orientation. In addition, no qualitative pattern for the increased agreement with the formula item could be identified in the video questionnaire (Table 5).

For both participants, two of the four identified notable features aligned across data sources, resulting in a medium degree of relation.

PST 4 perceived the error in the volume vignette in not making an exact measurement or calculation. In addition, for the circumference vignette it is noted, that the students’ chosen strategy is not suitable for an exact result (Table 6), which indicates the subjective relevance of exact procedures for this PST. This interpretation was confirmed by the high agreement on item 2 [exact procedures] (Table 6), resulting in a high relation between diagnostic competence and beliefs, since all notable features are matching the other data source.

Among the 139 PSTs with identifiable notable features, 95 (68.3%) displayed low or missing relations between error-diagnostic competence and beliefs. Medium or even high relations were found much less frequently (Table 7). Overall, the data indicate rather weak associations between PSTs’ diagnostic competence in error situations and their beliefs about teaching and learning mathematics.