National standards describe the mathematical competence that students should achieve at a given time in their school careers (KMK, 2004; Common Core State Standards Initiative, 2010).² Content and process-related standards specify the mathematical content students should know, and how they can learn and apply it. The German Educational Standards for primary school list expectations along the five guiding content domains of Numbers & Operations; Space & Shape; Measurement; Pattern & Structures; Data, Frequency, Probability and include Problem Solving, Mathematical Communicating, Modeling, Reasoning, and Representing as process-related proficiencies (KMK, 2004). Thus, mathematical competence encompasses not only knowledge, but also skills that are demonstrated in actively applying the content. In addition to content domains and process-related proficiencies, the German standards describe requirement levels (Anforderungsbereiche) that should be covered in teaching and in the selection of tasks. The spectrum of requirements ranges from rather simple reproduction of results to the more complex tasks of making connections to generalizing and reflecting.

To reach the described goals, a high quality of mathematics instruction is necessary. We know that instructional support, among other factors, is related to student achievement in general (OECD, 2020). In particular, cognitive activation, clarity of instruction, and a supportive climate play important roles in this context (Blömeke et al., 2016). Schoenfeld (2018) differentiates these three aspects with his five dimensions for robust understanding in mathematics: the Content has to be presented in a way that allows for focused and coherent discussions and provides opportunities to make connections. Mathematics teaching should guarantee Cognitive Demand, which means to challenge students with tasks ranging from moderate to demanding (as described in the level of requirements) and provide situations of “productive struggle.” Students should have Equitable Access to Content: all students need to engage with the content and be involved in meaningful ways. Mathematics education needs to provide opportunities to discuss disciplinary ideas and to build on each other’s ideas (Agency, Ownership, and Identity), which will train especially the process-related proficiencies reasoning and communicating. The mathematics teacher should pay attention to Formative Assessment and give students the opportunity to deepen their understanding with appropriate guidance for further work, tailored to their level of learning.

To assess whether the mathematical proficiencies required by the German Educational Standards are achieved by the students, proficiency level models based on the standards were developed by the Institute for Educational Quality Improvement (IQB) together with subject experts. In comprehensive standard-setting processes experts divided the continuous scale in every content domain into multiple sections referred to as proficiency levels. For each level, there is a description of the cognitive requirements that students can meet once they reach that proficiency level. In this way, it is possible to assign qualitative descriptions of the proficiencies acquired by students and to determine what percentage of students are most likely to be able to meet specific requirements. To examine the effects of school closures on mathematics performance, the broad spectrum of mathematical competence with all the proficiencies that students should acquire needs to be considered.

Whether a high quality of mathematics instruction as described above was realized in distance learning settings is presented below. Since the numerous surveys on distance learning often do not differentiate between school forms and surveys of primary school children are rather scarce, results of all school forms are reported below. If the school forms can be differentiated, this is indicated.

The instruction given to students (aged 10–16 years) during distance learning in mathematics consisted mainly of solving mathematical tasks (Heller and Zügel, 2020). 80% of parents reported that their children had to work on worksheets provided by their teachers, whereas only 29% reported that classroom teaching via video took place several times a week or daily (Wößmann et al., 2020). About 40% of the parents and also of the teachers stated that there was no possibility for a communicative exchange with classmates (Huber et al., 2020; Wößmann et al., 2020). These results were confirmed by a qualitative analysis of teaching descriptions of 700 teachers from four European countries (including Germany). The descriptions showed that more than two-thirds of the teachers tended to use a transmissive teaching style in distance learning. Some teachers stated that they changed their teaching style in distance learning compared to classroom teaching, because the particular conditions did not allow for collaborative teaching, or because transmissive learning was easier to organize (Aldon et al., 2021). A survey of 82 Australian elementary teachers found that the teachers’ attitudes toward productive struggle or cognitive demand differed depending on whether the instruction was distance or face-to-face (Russo et al., 2021). The main reasons given by the teachers were the absence of a teacher-facilitated, synchronous learning environment, as well as parents’ negative attitudes toward struggle when learning mathematics. These are clear indications that less cognitive activation took place in distance learning and that students had fewer opportunities to engage in discussions about disciplinary ideas and their ways of thinking (for an overview, see also Helm et al., 2021).

There is also evidence that instruction may have changed in terms of content and performance expectations. Secondary school teachers in structurally disadvantaged schools in particular argued for lowering the standard level during the period of distance learning (Bremm, 2021), and there are indications that the procedural aspect of mathematical concepts (including calculating, applying algorithms, or exercising skills and techniques) was more present in distance learning than conceptual understanding (Aldon et al., 2021).

In terms of a supportive climate or formative assessment, there is some indication that teachers gave comparatively less positive reinforcement to their students at home (Mantasiah et al., 2021). While parents reported that almost all students received worksheets to work on weekly, just under two-thirds of the students received feedback at least once a week (Wößmann et al., 2020). In particular, students of parents without an academic background and lower-performing students very rarely had individual contact with their teachers (Wößmann et al., 2020). There is some evidence that teachers supported students who struggled with given tasks primarily by generally providing more materials and additional explanations rather than offering individualized learning tasks (Aldon et al., 2021). On the other hand, only a few students and parents indicated that teachers were not available to answer students’ questions (Helm et al., 2021). Regarding parental support for home-learning in mathematics, some parents stated that they found it difficult to provide the best possible support for their children (Darragh and Franke, 2021).

To summarize, there are indications that distance learning was less cognitively activating and more focused on procedural aspects such as calculating or exercising skills than on conceptual understanding. Moreover, it seems that students received less individual support and teachers considered lowering the standards.

As reported in the introduction section, home schooling significantly reduced the children’s weekly learning time. Students report between 3.6 h (Wößmann et al., 2020) and 5 h of learning time per day (Schober et al., 2020; Helm et al., 2021). The reported learning time appears comparatively low if one assumes that a school day has at least 4.5 h of pure lesson time (six times 45 min) plus time for homework. A survey in seven European countries (including Germany) with 6,285 parents and their children (n = 5,767, 10–18 years) found that girls spent 18 h of per week studying in addition to any online sessions, and boys spent 17 h (Kantar Market Research Institute, 2021). The fact that boys spent less time for learning than girls was also confirmed in other surveys (Wößmann et al., 2020; Lampert and Thiel, 2021; Nusser et al., 2021).

It seems that girls adapted to distance learning better than boys (Forsa, 2020). Boys more often did not know what to do for school compared to girls (Sturzbecher et al., 2021), and parents of boys were more likely than parents of girls to report that their child was more difficult to motivate to learn at home (Lockl et al., 2021). They saw a main challenge in their boys’ ability to concentrate in online learning sessions, while girls’ parents saw the lack of social connection as their child’s main challenge during school closure (Kantar Market Research Institute, 2021).

All the findings on distance learning (“Quality of distance learning”) show that the conditions for teaching to achieve the desired objectives (“Goals of mathematics education in primary school”) are poorly fulfilled. There are already some results on whether this also reflected in lower performance in mathematics. We report performance comparisons (time frame before and after the school closures) first from Germany, followed by international comparisons.³

Schult et al. (2022) used a mandatory, teacher-administrated test in the German federal state of Baden-Wuerttemberg in grade 5 to compare students’ performance in the three pre-pandemic-years 2017–2019 with performance in September 2020 (Schult et al., 2022). After 2 months of school closure, students’ performance was slightly below the average results of the 3 years prior to the pandemic. The difference was more pronounced in the subtest Operation (d = −0.09) than in the subtest Number (d = −0.03) and more in low-achieving than in high-achieving students.

The study by Depping et al. (2021) showed hardly any differences in performance before and after the school closures in the German federal state of Hamburg. Depping et al. compared the results of teacher-administered tests at the end of grade 3 before the pandemic (2019) in the content domains Space & Shape as well as Data, Frequency, Probability with teacher-administered tests at the beginning of grade 4 (2020). Additionally, they compared mathematical performance in all content domains at the beginning of grade 5 (2019 vs. 2020). Although both group comparisons showed slightly lower performance in 2020 than in 2019 (the loss in performance was most pronounced for the disadvantaged school group in grade 5), these differences proved to be statistically nonsignificant (d < 0.2).

Some international comparison studies show a drop in performance. Tomasik et al. (2021) used data generated from a computer-based formative assessment tool in Switzerland and compared students’ learning gain in Mathematics and German 8 weeks before the school closure with the learning gain during 8 weeks of school closure separately for primary and secondary school students. While no differences were evident at the secondary level, learning gains at the primary level were twice as high in the 8 weeks prior to school closures than during the period of school closure. In addition, heterogeneity in achievement increased significantly (Tomasik et al., 2021).

In the Netherlands, schools were closed for 8 weeks. Engzell et al. (2021) report a loss of performance in mathematics, spelling and reading of about one fifth of a school year (3 percentiles or 0.08 standard deviations) during school closures. The authors analyzed a sample of students from 15% of all Dutch schools (aged 8–11). They compared scores from standardized tests immediately before and after the school closures with scores from the same periods in the 3 years prior to the pandemic. Learning loss was particularly pronounced in mathematics and among students from disadvantaged families, with no overall gender differences found (Engzell et al., 2021).

Similar results were reported by studies with standardized tests in California (2.5-month learning lag in mathematics in grade 4–8; Pier et al., 2021), in Belgium (school mean scores decreased by 0.19 standard deviation; Maldonado and de Witte, 2022) and in South Africa (schools were closed for 155 days, loss from 59 to 50%—interpreted as a loss of almost a full school year; van der Berg et al., 2022). The South African study tested five content domains in grade 3, but was not balanced across content domains: more than 60% of the items tested the Number, Operation, and Relationships domain. The other for domains were tested with four to six items (7–10% of all test items). The greatest loss of learning was reported in Data Handling (13%), followed by Numbers, Operations and Relationships (9%), Measurement (8%), Patterns, Functions and Algebra (7%), and Geometry (6%; van der Berg et al., 2022).

However, there are further studies that show no significant drop in performance. A study of more than 2,000 students (average age 9.7) in Australia did not find an impact of school closures on performance in general (d = 0.06) but significant negative effects on performance among students from disadvantaged schools (d = −0.16; Gore et al., 2021). A drop in performance was also not evident in the study by Meeter (2021), who analyzed successively collected performance data from a digital learning environment in the Netherlands. During the school closure, younger children in particular even showed higher performance than children in previous years. It seems that they were able to use the lockdown period to practice key skills. However, this performance advantage disappeared when schools reopened (Meeter, 2021). Since the situation after the initial school closures differed in many countries, the findings of other countries cannot be easily transferred to the German context. Some of the German studies refer to a long period of time (see Introduction, Stanat et al., 2022), to only one federal state (Depping et al., 2021), not to the primary level (Schult et al., 2022) or only to some content domains (Depping et al., 2021). For this reason, specific results from the primary level that do not only focus on mathematical competence in general, but also on the different content domains in an adequate time frame around the school closures, is thus far missing.

This study is based on data from two Germany-wide pilot studies in elementary schools (grade 3) with an overlapping set of mathematics items. The data were collected in May 2019 and May 2021 as preparation of the nationwide written comparison tests VERA (Vergleichsarbeiten), which students take in the third grade in Germany as part of the national educational monitoring strategy. The samples were comparable in that they were drawn as stratified cluster-random samples that were intended to cover the main characteristics of German third graders with respect to the outcome. A total of 1,905 students from 100 classes in eight German federal states completed the mathematics assessment in the pilot study in 2019 and 3,203 students from 172 classes in mostly different eight German federal states in 2021. There was always one class per school sampled. The students’ mean age was 9.48 years (SD = 0.65) in 2019 and 9.52 years (SD = 0.62) in 2021. In 2019 and 2021, 49.0 and 50.3% of the children were female, respectively.

In both years, the test consisted of two parts, with working times of 40 min (part 1) and 20 min (part 2) and a break of 10 min in between.

For our analysis, we used data from 537 test items. Among these, 85 were identical, 145 items were uniquely administered in 2019 and 307 items were uniquely administered in 2021.

The test items were developed by teachers in close collaboration with experts in mathematics education and psychometricians and were based on the proficiency level models that were developed according to the German Educational Standards as briefly described above. They cover all five content domains defined in the German Educational Standards. The items have multiple-choice format (e.g., “Put a cross next to the drawing showing this building from above.”—Content domain: Space and Shape) and constructed-response formats (e.g., “Anna, Bert and Cesar are sitting on a bench. Write down all the ways the children can sit.”—Content domain: Data, Frequency, and Probability) and were grouped into disjoint blocks that were combined to booklets according to a multi-matrix design (e.g., Frey et al., 2009). Because our performance measure is based on proficiency level models, we will use the terms performance and proficiency in the following interchangeably.

In order to examine measurement invariance assumptions as prerequisites for linking, we calculated differential item functioning (DIF) statistics according to the Educational Testing Service (ETS) classification (e.g., Penfield and Algina, 2006). We found 11 of the 85 items to exhibit DIF. We inspected these 11 items and the students’ answers to these items in order to examine whether there could be any construct-irrelevant source of variance that could have caused the differential functioning. During this procedure, we found two items that had to be excluded for the year 2019 because their coding guidelines were misunderstood. The remaining nine DIF items were omitted from the link by relaxing measurement invariance for these items opting for partial invariance but were retained in the analysis with unique difficulty parameters for each year. Consequently, there were 74 common items remaining (13 Numbers & Operations, 19 Space & Shape, 12 Pattern & Structures, 21 Measurement, and 9 Data, Frequency, Probability).

Generalized Linear Mixed Models (GLMMs) allow us to model item and person properties and assess their effects on the individuals’ responses. In order to investigate the effects of the factors addressed in our research questions on students’ correct responses (their score) to the 546 mathematics items in 2019 and 2021, we specified a linear logistic test model with additional error term in the GLMMs framework with a multi-level structure (students were nested in classes). Further, GLMMs allow for the specification of fixed and random parts. Our fixed-effects part consisted of the following predictors: content domain, proficiency level and requirement level as item features on the item side and gender and year of measurement (two distinct samples in 2019 and 2021) on the person side. Interactions with year of measurement were allowed for all factors in order to investigate differential changes for different types of items as well as for boys and girls. The models’ random part consisted of random item and random person effects that allow for individual deviations for each item and each person from the predicted value of the models’ fixed part.

The factor content domain consists of five levels (Numbers & Operations, Space & Shape, Pattern & Structures, Measurement, and Data, Frequency, Probability) that were effect coded. With effect coding, the intercept is equal to the grand mean of all five content domains. Thus, we were assuming that all content domains contribute with equal weight to the proficiency level models. The same was done for the factor gender. The factor requirement level consists of three levels (I–III) as described in the German Educational Standards. As there were only a few items in the third category, we decided to collapse categories II and III into one category (II-III). The binary factor requirement level was weighted effect coded. In weighted effect coding, all levels of the factor are weighted by their sample proportions. This was done because we assumed that the three requirement levels are not equally distributed in the potential item pool. The factor year of measurement was dummy coded with 2019 as reference group. Because the proficiency levels are based on a continuous scale, this covariate was added as continuous and centered at the medium stage (3 out of 5). Thus, the model intercept can be interpreted as the average across all content domains and all boys and girls at a moderate proficiency level and over not necessarily equally frequent levels of requirement level in the year 2019. The models were estimated using the lme4 R package (Bates et al., 2022). In lme4, the model specifies as score ~ year*content_domain + year*sex + year*requirement_level + year*proficiency_level + (1|class/student) + (1|item).

As mentioned above, with the exception of score and proficiency level, all variables were factors with 5,108 levels for student, 272 levels for class, 546 levels for item, five levels for content domain, two levels for gender, and two levels for requirement level. In addition to the main effects and interaction effects, we investigated also simple effects in order to explore the exact nature of our interactions using the emmeans R package (Lenth et al., 2022). In particular, we compared the mean of each content domain in the year 2021 to the corresponding mean in 2019. The same was done for gender and requirement level.

The results for answering RQ1 showed a negative main effect of the year, with a lower probability to respond correctly to an item in 2021 compared to 2019, therefore indicating an overall increase in item difficulty or a drop in performance (p < 0.001, see Table 1). This result can be interpreted as averaged over content domains, and for items at a moderate proficiency level (3 out of 5) for a population with similar characteristics regarding gender, controlled for differences in class means due to other variables, and for items with similar requirement levels as in our sample.

This general finding differed across the different content domains (RQ2): the interaction effects in Table 1 indicate an average decline for Measurement (M) and Pattern & Structures (P&S). Contrastingly, the drop compared to the average decline was significantly larger for Data, Frequency, Probability (DFP; p = 0.006) and Space & Shape (S&S; p = 0.003) on the one hand and smaller for Numbers & Operations (N&O; p = 0.017) on the other hand.

Figure 1 depicts these domain-specific effects in terms of model-derived estimates for the probability of a correct answer in the different domains and years, as well as the significance of the simple effect of the year for each domain. The probabilities to answer an item correctly were mostly homogenous for all domains in 2021, ranging from 38 to 40%. In contrast, in 2019 we could identify some heterogeneity; that is, a group of domains had correct answer probabilities of around 42% (N&O, M, P&S), and a group had probabilities around 48% (S&S and DFP), indicating a lower difficulty for the latter in 2019 and a larger decline between 2019 and 2021. Nevertheless, all simple effects except for Numbers & Operations and Pattern & Structures were significant, indicating a relevant change in the probability to answer an item correctly. Interestingly, seven (2 DFP, 5 S&S) out of the nine aforementioned items with severe DIF according to the ETS classification that were excluded from the link were omitted more often and answered correctly less often in 2021 than in 2019. In contrast, the last two DIF items (both N&O) showed no such differences in omission, but even had a higher rate of correct answers in 2021 than in 2019. That is, the qualitative analysis of the DIF items is in line with the quantitative effects reported above.

Regarding the first two research questions, we can confirm that students’ proficiencies in mathematics decreased in general between 2019 and 2021 (RQ1), with differences between domains (RQ2). Space & Shape and Data, Frequency, Probability decreased more than the other domains, whereas Numbers & Operations decreased less. Pattern & Structures and Measurement did not decrease differentially from the average of all domains.

The main effect of gender in Table 1 indicates that boys were more likely to answer an item correctly than girls in 2019 (p < 0.001). For both genders the estimated probability of a correct response declined in 2021. This decline appeared to be slightly larger for girls. However, this interaction effect is not significantly different from zero (p = 0.057).

As can be seen in Figure 2, the probability of a correct answer for girls declines from 42 to 34% between 2019 and 2021, and for boys from 47 to 42%, also showing a slightly larger decline for girls. Both drops can be regarded as small effects in terms of Cohen’s d (−0.161 and −0.092, respectively), when converting the corresponding odds ratios to effect size d according to Sánchez-Meca et al. (2003).

Thus, to answer the third research question, we can attest that the performances of boys and girls do not decline significantly differently, but clearly do decline and more for girls than for boys.

To assess the influence of the complexity of the task we compared test items with a requirement level of I to those with a level of II or III. In 2019, the probability of a correct answer was higher for requirement level I than for level II–III (p = 0.004). This was also the case in 2021 (i.e., there is no significant interaction effect; see Table 1). Figure 3 shows that in both years items at requirement level I have a higher probability of being answered correctly (in 2019, about 47%; in 2021, about 41%) than items from level II-III (in 2019, about 43%; in 2021, about 37%). The average probability for items at both levels declined in 2021 by 6% compared to 2019. Both declines are significant.

With regard to the fourth research question, we were therefore able to conclude that the decrease was not more severe for more complex tasks.