To identify relevant studies that investigated the effect of COVID-19-related school closures on student achievement, we searched the Web of Science database for articles published between March 1, 2020 and April 30, 2021. We used the following keywords and search string: [Covid OR Corona OR “SARS-CoV-2” AND school AND learn^* OR “test score” OR performance OR competenc^* OR achievement OR grades]. The results were refined by using the following categories: education, educational research, economics, education scientific disciplines, psychology educational, psychology multidisciplinary, social sciences interdisciplinary, and education special. The indexes searched were SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH, BKCI-S, BKCI-SSH, ESCI, CCR-EXPANDED, and IC. Because the COVID-19 pandemic was still ongoing at the time this review was written, and the field of research on the effects of COVID-19-related school closures on student achievement is rapidly evolving, we additionally searched the preprint servers PsyArXiv, EdArXiv, and SocArXiv using the aforementioned keywords. With this initial literature search, we obtained 601 potentially relevant studies. After selecting relevant articles out of these studies, we used the backward reference searching method (i.e., examining the works cited in the selected articles) to identify additional potentially relevant studies. See Figure 1 for a PRISMA flowchart of the literature search process.

The abstracts of the studies selected were carefully read by the authors, and further inclusion was decided based on the following initial criteria. The studies (1) had to have a clear focus on COVID-19-related school closures, they (2) had to focus on primary and secondary education, and they (3) had to have student achievement (or test scores) as the dependent variable. This initial selection left 109 studies for potential inclusion in the review. These studies were thoroughly read by the authors and two research assistants. We carefully assessed the quality of included studies and based the decision to include studies in the review on the following primary set of inclusion criteria: Studies were required (1) to have collected actual data prior to and during/after COVID-19-related school closures, and (2) to have applied statistical analyses and to report an effect size. This set of inclusion criteria was chosen in order to select studies that provided the aforementioned evidence-based insights. Thus, reviews or discussions on how COVID-19 affects educational processes were excluded. Likewise, exploratory analyses or simple surveys (where only percentages were reported) were also excluded. For example, Chadwick and McLoughlin (2021) investigated the impact of COVID-19 related school closures on student's science learning. However, they only questioned teachers on the impact of COVID-19 related school closures on teaching, learning, and assessment. Because the study did not meet the inclusion criteria of having collected actual data on student achievement, including a comparison of data prior to and during/after COVID-19-related school closures, and applying statistical analyses rather than solely reporting percentages, the study was excluded from the systematic review. Similarly, studies by Haeck and Lefebvre (2020), Kaffenberger (2021), and Kuhfeld et al. (2020a) were excluded from the systematic review because they reported predicted effects of COVID-19 related school closures on student achievement but did not collect actual data prior to and during/after COVID-19-related school closures.

To determine the degree of rater agreement on the selection of the studies, a randomly selected subset of 20 studies was evaluated by both the authors and the research assistants. Any remaining divergent evaluations were highlighted in the evaluation forms and subsequently discussed. The second selection procedure yielded nine studies that were suitable for inclusion in the review. Subsequently, a backward search of references within the nine selected studies yielded two additional studies, which were then also included in the review.

The Cochrane Risk Assessment of the included studies was conducted independently by the first and second author using the “Risk Of Bias in Non-Randomized Studies of Interventions” tool (ROBINS-I; Sterne et al., 2016). The result of the risk assessment is summarized in Figure 2. Taken together, the highest risk of bias is due to the lack of inclusion of potential confounding variables (Domain 1). Most studies, however, included at least a few relevant controls. Bias due to selection of participants was unlikely as the groups were formed naturally. Similarly, where applicable, interventions were classified correctly. Except for Clark et al. (2020), no information could be obtained about deviations from intended interventions. This is because the COVID-19 related school closures were not intended interventions. Thus, although there is no information, the risk due to deviations from intended interventions was deemed low. Lastly, bias due to missing data, measurement of outcomes, or selection of reported results is unlikely, as most studies exhibited very small proportions of missing data (except Depping et al., 2021), and were highly transparent in their reporting of the results (except, partially, van der Velde et al., 2021, who only report significant mean differences between groups in sufficient detail). Depping et al. (2021), however, thoroughly discuss and provide convincing reasons for assuming that missing data does not substantially influence their results.

We synthesized the eleven studies by extracting the following information that was relevant for our research questions: (1) country, (2) duration of school closure, (3) sample description (type of school and sample size), (4) subjects for which student achievement was investigated, (5) statistical method, (6) general effects of the COVID-19-related school closures on student achievement, and (7) differential effects as reported by subgroup analyses (see Table 1 for a detailed list of the studies included). The focal piece of information was the reported general and differential effects. Where possible, general effects reported in different metrics (e.g., percentile scores), were converted to changes in SD. We then calculated the median of the reported effects, for the overall general effect, as well as for the general effect on reading and mathematics. In light of the relatively small number of studies, random- or even mixed-effects meta-analytic models were not feasible.

The studies on the effect of COVID-19-related school closures on student achievement selected for our review reported mixed findings, with effects ranging from−0.37 SD to +0.25 SD (Mdn = −0.08 SD). Most studies found negative effects of COVID-19 related school closures on student achievement. Seven studies reported a negative effect on mathematics (Clark et al., 2020; Kuhfeld et al., 2020b; Maldonado and De Witte, 2020; Tomasik et al., 2020; Depping et al., 2021; Engzell et al., 2021; Schult et al., 2021), five studies on reading (Clark et al., 2020; Maldonado and De Witte, 2020; Tomasik et al., 2020; Engzell et al., 2021; Schult et al., 2021), and two studies on other subjects, such as science (Maldonado and De Witte, 2020; Engzell et al., 2021). This is in line with expected learning losses due to COVID-19 related school closures and the assumption that, in spring 2020, the ad hoc implementation of online teaching gave students, teachers, schools, and parents little time to prepare for or adapt to measures of remote learning.

Three studies reported positive effects of COVID-19 related school closures on student achievement. Meeter (2021) and Spitzer and Musslick (2021) showed students to improve their mathematics achievement when learning with an online-learning software during the COVID-related school closures. Similarly, van der Velde et al. (2021) reported an increase in correct solutions on open questions within a French learning program. Interestingly, these three studies focused on online-learning software. Thus, the positive effects may be explained by the students under investigation being familiar working with the corresponding online-learning software prior to school closures. Hence, they did not have to adapt to a new learning environment when in-person teaching was interrupted due to COVID-19. Moreover, students increased the time using the online-learning software at home, were less distracted or experienced less time pressure in a home-schooling rather than classroom setting, or were presented with individualized assignments within the online program (see also Meeter, 2021; Spitzer and Musslick, 2021; van der Velde et al., 2021).

Additionally, two studies found positive effects on student achievement in mathematics and reading (Gore et al., 2021), or in reading only (Depping et al., 2021). This result might be accounted for by the achievement measurement being timed some months after school closures in both studies and the possibility of effective compensatory measures being implemented by teachers, schools, and local policy makers during this time to counteract learning losses, such as offering learning groups during summer vacation in parts of Germany (Depping et al., 2021).

Even though the median for the effect on mathematics and reading is comparable when averaging above all studies (d = −0.10 SD and−0.09 SD for mathematics and reading, respectively), some included studies found different effects for different subjects. On the one hand, reasons for finding larger learning losses in reading than in mathematics might be that “mathematics is easier to teach in distance learning, as it is simple to provide exercises and tests digitally or as worksheets” (Maldonado and De Witte, 2020, p. 13). As another explanation, many students might not speak the language in which they are tested in at home, hence, not benefitting much in their language skills during school closures (e.g., Maldonado and De Witte, 2020). On the other hand, reasons for finding larger learning losses in mathematics than in reading might be that students spent more time on reading during school closures and that supporting children in their reading skills might have been easier to realize for parents than supporting children in improving their competencies in mathematics (e.g., Depping et al., 2021; Schult et al., 2021).

The studies selected for our review reported three main differential effects of COVID-19-related school closures on student achievement in different groups of students. First, the main finding was that younger children were more negatively affected in their learning than older children were (-0.37 SD vs.−0.10 SD; Tomasik et al., 2020). Second, children from families with a low socioeconomic status (SES) were more affected than children from families with a high SES were (Maldonado and De Witte, 2020; Engzell et al., 2021). In this context, one study reported an interaction between grade and SES, that is, for younger children from schools with low school-level SES, learning losses of 0.16 SD were found, while younger children from schools with medium school-level SES experienced learning gains of 0.15 SD (Gore et al., 2021). Third, low-performing students were more affected by COVID-19-related school closures in mathematics, while high-performing students were more affected by COVID-19-related school closures in reading (Schult et al., 2021). Finally, low-performing students benefited more from systematic online-learning methods (Clark et al., 2020; Spitzer and Musslick, 2021).

As the original studies were not designed to identify the reasons for these effects, additional studies are required to explain the three main differential effects exhaustively. In the following, we provide potential explanations as stated in the original studies. Regarding the first main differential effect (younger students are more affected compared to older students), Tomasik et al. (2020) state that the slower pace of students in primary school may be due to younger children relying more on cognitive scaffolding during instruction, because their capability for self-regulated learning might not be sufficiently developed. From a socio-emotional perspective, younger children might have been more sensitive to stressors related to the COVID-19 pandemic (Tomasik et al., 2020).

The reasons for students from low SES families being more affected relate to access to remote learning, their learning behavior, and the support provided from families and schools. Children from families with a low SES are less likely to have access to remote learning (UNESCO, 2021), are less often provided with active learning assistance from their schools (Tomasik et al., 2020), and spend less time on learning (Meeter, 2021) than children from families with a high SES. Moreover, parents with a high SES are more likely to provide greater psychological support for their children (OECD, 2019), which seems to be specifically relevant in a situation such as the COVID-19 pandemic.

The differential effect on low-performing and high-performing students may be due to high-performing students being capable of improving their performance regardless of the learning environment, while low-performing students specifically benefit from systematic online learning (Clark et al., 2020). Additionally, low-performing students might be less distracted in comparison to learning in a classroom setting (Spitzer and Musslick, 2021). Finally, with the possibility to adapt the assignments in online programs individually to the students, low-performing children might have been addressed more thoroughly according to their needs (Spitzer and Musslick, 2021).