Automated feedback and writing: a multi-level meta-analysis of effects on students' performance - Fleckenstein, Johanna; Liebenow, Lucas W.; Meyer, Jennifer (Frontiers in Artificial Intelligence, 2023)
Adaptive learning opportunities and individualized, timely feedback are considered to be effective support measures for students' writing in educational contexts. However, the extensive time and expertise required to analyze numerous drafts of student writing pose a barrier to teaching. Automated writing evaluation (AWE) tools can be used for individual feedback based on advances in Artificial Intelligence (AI) technology. A number of primary (quasi-)experimental studies have investigated the effect of AWE feedback on students' writing performance. This paper provides a meta-analysis of the effectiveness of AWE feedback tools. The literature search yielded 4462 entries, of which 20 studies (k = 84; N = 2,828) met the pre-specified inclusion criteria. Overall, results based on a three-level model with random effects show a medium effect (g = 0.55) of automated feedback on students' writing performance. However, the significant heterogeneity in the data indicates that the use of automated feedback tools cannot be understood as a single consistent form of intervention. A moderator analysis investigated the impact of the characteristics of the learner, the intervention, and the outcome measures. We discuss these findings in light of automated feedback use in educational practice and give recommendations for future research.