Nino Buliskeria
Ali Elminejad
Tomas Havranek
Zuzana Irsova
Stepan Jurajda
Marek Kapicka
Martina Luskova

Replication; Robustness; Meta-analysis; COVID-19; Education; Learning deficit

I21, I24, I28, C68

Buliskeria N., Elminejad A., Havranek T., Irsova, Jurajda S., Kapicka M., Luskova M. (2024): " A comment on “A Systematic Review and Meta-analysis of the Evidence on Learning During the COVID-19 Pandemic”" IES Working Papers 41/2024. IES FSV. Charles University.

Betthäuser et al. (2023) examine the effects of the COVID-19 pandemic on the learning progress of school-aged children. They collect 291 estimates from 42 studies. Their meta-analysis-corrected estimate implies a substantial decline in students’ learning (Cohen’s d = −0.14, 95% confidence interval −0.17 to −0.10). First, we successfully reproduce the main results and the majority of supporting figures. Second, we provide additional analysis addressing publication bias by implementing correction techniques: PET-PEESE (funnelbased), 3PSM (selection model), and RoBMA (model averaging). Additionally, we implement novel approaches that account for the strength of biased selection favoring affirmative results in the sample of analyzed studies. Third, we use techniques that assume the presence of p-hacking (MAIVE, RTMA). Using these methods, the corrected effect ranges from −0.25 to −0.11 with high statistical significance. While our analysis does reveal some evidence of publication bias and p-hacking, these phenomena do not appear to systematically distort the overall findings of the original study.