Schools closed all over the world in the context of the COVID-19 pandemic, leaving over 1.6 billion children without in-person classes. For many countries, that was the case for a very long time; in Brazil, schools were closed for over 700 days. Dramatically, by end of March 2022, 23 countries were still to fully reopen schools -- affecting over 400 million children worldwide.
By now, it is clear that learning losses during the pandemic were tremendous, and that such losses were larger in countries that kept schools closed for longer.
But how do we know whether such losses really stemmed from differences in learning between remote and in-person classes or, rather, from other effects of the pandemic on teachers, students and their families? And how do we know that the correlation between the length of school closures and learning losses does not reflect, instead, the fact that countries where schools were closed for longer might also have worse techonology infrastructure or quality of instruction?
In a new paper, we provide evidence that remote learning indeed caused worse learning outcomes during the pandemic, above and beyond other direct effects of COVID-19, and that longer school closures indeed causally magnified these effects.
Why this research matters
How much did children learn with remote classes during the pandemic? This question is particularly important for low- and middle-income countries, where average vaccination rates are still dragging -- creating pressure for classes to remain at least partially remote even as we enter the third year of the global pandemic.
On the one hand, low- and middle-income countries' efforts to transition from in-person classes onto remote learning were unprecedented. In Brazil, during the H1N1 epidemic in 2009, the reaction was to prolong school recess rather than to offer remote classes. This time around, education secretariats tried to respond quickly, through a variety of approaches: from offering printouts for students to pick up at the school gate to televised classes to zero-rating platforms where students could study online, asynchronally or with teachers' support. In effect, over 80% of Brazilian students claimed to have had classes every week during the pandemic.
On the other hand, limited internet access, lack of dedicate spaces to study at home, and little support from parents who often have not attended school for as long as their children have likely made remote learning particularly ineffective in these settings. Not surprisingly, despite having classes, students also reported very limited study time during the pandemic -- less than 2.5 hours a day. That is above and beyond additional detrimental factors in the context of the pandemic, from demand for child labor to violence against children in a context of psychological distress.
Several studies have now shown that learnings losses were dramatic during the pandemic, across several low- and middle-income countries and even in even in high-income countries, such as the Netherlands. Having said that, these studies have two critical limitations: 1) they do not parse out the effects of remote instruction from other direct effects of the pandemic, and 2) most of them focus solely on learning losses, ignoring the issue of student dropouts.
The first issue matters because, if children learned little during this period because their family experienced job or income losses, or because themselves or other family members got sick, then these studies might overestimate the negative effects of remote learning -- which could actually work in the aftermath of the most acute phase of the pandemic.
In turn, the second issue matters not only because a surge in student dropouts could reverse decades of progress in developing countries to make primary and secondary education nearly universal, but also because if the profile of the students who took exams before and after the pandemic changed, then these studies might actually underestimate the negative effects of remote learning -- since students who drop out are not typically faring well at school.
The study design
In this study, we overcome these limitations by combining administrative data on the universe of middle- and high-school students in São Paulo, Brazil, with natural experiments for the timing of school closures and reopenings in the State, to study the causal effects of remote learning on learning losses and student dropouts during the pandemic.
São Paulo State, with over 3 million students before the pandemic hit, offers a great empirical setting to study these questions. The State featured in-person classes during the entire first school quarter of 2020, transitioning to remote instruction afterwards -- which allows us to compare how each student fared in remote learning relative to in-person classes under the same teachers and within the same grade. This is thanks to the fact that the State conducted quarterly standardized assessments before and during the pandemic, and because it offers socioeconomic data and scorecard attendance and grades for all its students over the same period, allowing us to carefully account for compositional changes and to study effects on student dropouts. Because learning typically evolves unevenly over the course of the school year, we use differences between the first and the remainder school quarters in 2019 and 2018 as conterfactuals.
Causal evidence on the effects of remote learning
Our findings provide a pretty striking account of learning losses during the pandemic. On average, students learned only 27.5% of what they would learned with in-person instruction. Losses were not concentrated in any particular grade, but were higher for black and brown students, those in poorer neighborhoods and in schools without experience with online academic activities before 2020. Learning losses were higher for math classes, where students learned only 20% of the in-person equivalent; that figure was 40% for Portuguese.
Detailed data on COVID-19 case-fatality rates for each of the 645 municipalities of São Paulo State allow us to parse out direct effects of the pandemic on educational achievement, by letting learning losses vary non-parametrically with the local severity of the pandemic (measuref by log per capita cases or deaths). The fact that losses were not higher in municipalities affected by the pandemic to a greater extent suggests that the learning losses that we document are indeed the results of remote learning during that period.
Estimating effects on student dropouts throughout that period is challenging. After all, just like every Brazilian State, São Paulo automatically re-enrolled students in 2021 to prevent that anyone would miss the opportunity to remain in school just because of lack of connectivity or because their caregivers did not feel comfortable showing up in person to sign them up. In effect, the State recorded 0% dropouts among secondary students in 2021, whereas dropouts hovered around 10% in a typical year. In reality, however, many students were not taking part in any academic activity, as we could see from their attendance records or their scorecard grades. In tandem, national polls suggested 40% of students were not motivated to come back by the time in-person classes returned.
To track student dropouts in face of these challenges, we define a proxy of high dropout risk as students missing scorecard grades for both Portuguese and math classes in that school quarter. In the paper, we show that this proxy reliably predicted actual dropouts before the pandemic, and lack of participation in 2021 school activities. We find catastrophic effects of remote learning on dropout risk, which suggest that dropouts among secondary students in the State might have increased from 10% to 35% in the absence of in-person classes.
Causal evidence on the impacts of the length of school closures
Can we conclude that secondary schools should refrain from remote instruction even after 2 years after it was first introduced? Or were learning losses and the surge in dropout risk the result of a costly emergency transition to remote learning? If that were the case, our results would overstate the costs of remote learning relative to in-person classes outside of this transition.
To answer that question, we take advantage of the fact that some municipalities in in São Paulo State allowed high-school classes to resume in the last school quarter of 2020. By then, (after over two quarters of remote instruction), teachers and students had had time to adapt to the new instruction mode.
We find that resuming classes in 2020 partially mitigated learning losses from remote instruction. Our estimates for the weekly rate at which test scores improved under in-person classes are similar to our estimates of that at which they deteriorated under remote classes, confirming that the effects of the length of school closures on learning losses are causal. Reassuringly, within municipalities that authorized schools to reopen, test scores improved relative to the control group only for high-school students -- as in-person middle-school classes did not resume in 2020.
From research to policy
Without rigorous evidence to quantify the contribution of remote learning to those educational impacts, decisions about what to do about in-person classes in the pandemic were largely influenced by the potential health costs of reopening schools, without weighing those against the educational costs of keeping them closed. In effect, other economic activities for which it is easier to quantify losses from closures – such as shopping centers, bars and restaurants – have reopened systematically earlier than schools (and for longer) in middle-income countries. Our findings show, however, that remote classes had enourmous societal costs during the pandemic.
As new variants threaten to bring about new mobility restrictions, and potentially close schools once again, the public debate should move from whether schools should be open or not to how to reopen them safely: whether to prioritize school staff in the vaccination schedule, what is the appropriate but feasible personal protection equipment in place, what safe capacity limits should be, what changes in school infrastructure are required (e.g., to ensure appropriate ventilation), and how to adapt public transportation to mitigate contagion risks on the way to school –- the very same discussions that have been part of the public debate about shopping centers, bars and restaurants since the onset of the pandemic.
