Fool Me Twice? New York City Rejects Research Critical of Renewal Schools
New York City is currently testing out a policy to help struggling "Renewal schools" by providing them with extra resources and community services in order to improve their performance, rather than shutting them down. However, in some cases, more aggressive measures have had to be taken. For example, it was recently announced that after three years of attempting to turn around two Renewal high schools, all teachers and staff will have to reapply for their positions.
Unfortunately, recent evaluations of the Renewal Schools program have shown disappointing results. One report from the Manhattan Institute found that while there were statistically significant improvements in average test scores for Renewal schools after the policy was implemented, the improvements were too small to justify the high cost of the program. Another analysis by Teachers College professor Aaron Pallas found that test score gains in Renewal schools were no different from those in other schools with similar demographics. Although these findings may seem contradictory, they are the result of slightly different methodologies and assumptions used in the evaluations.
It’s important to note that all findings from empirical research rely on assumptions, and different research methods have different levels of reliance on assumptions. While the administration of Mayor Bill de Blasio has challenged the assumptions behind both studies, their main concern with Pallas’s study is that it doesn’t take into account the fact that Renewal schools were specifically selected because they were struggling. As for my study, it’s unclear why the administration questions its validity.
However, it is reasonable to challenge assumptions in research. Academics are constantly questioning each other’s assumptions in order to improve the quality of their work. That being said, if the Renewal schools policy were truly effective, there should be some evidence of its impact in studies like these. The assumption that both studies are missing a significant policy effect is simply implausible.
The administration’s view that no research design can inform us about the effects of Renewal schools is concerning. This stance has been taken in the past when empirical research results didn’t align with their beliefs. However, this position is unacceptable, especially considering the significant cost of the program. It could have been avoided with better implementation.
Studies like mine and Pallas’s use quasi-experimental designs, which aim to replicate the results that would have been obtained through a randomized experiment. While these methods are often convincing, real experiments are always preferred. In the case of the Renewal schools program, the administration could have conducted a randomized field trial, randomly assigning Renewal school status to a selection of schools that were expected to benefit from the treatment. This would have allowed for more reliable results and fewer assumptions.
Overall, it is crucial to critically evaluate the assumptions and methodologies used in research studies. The administration’s dismissive approach towards research undermines the potential for evidence-based decision-making and raises concerns about the effectiveness and accountability of the Renewal schools program.
There are concerns that Randomized Field Trials (RFTs) unjustly withhold a beneficial intervention from those in need. One issue with this concern is that the effectiveness of an intervention cannot be determined until thorough research has been conducted. However, a more practical matter is that limited budgets often restrict the availability of treatments. (I believe that if the de Blasio administration had the necessary budget, they would have expanded the Renewal schools program further.) In such cases, randomization not only allows for rigorous study of the policy’s effects but also ensures the fairest distribution of scarce resources.
Having convincing evidence of the effects of a program is crucial for effective policymaking. In the case of Renewal schools, the evidence may not be perfect, but it strongly suggests that the policy has resulted in underwhelming outcomes. If we desire more conclusive evidence about the effectiveness of future policy interventions, policymakers must implement them with research design in mind.
Marcus A. Winters is currently an associate professor at the Boston University School of Education and a senior fellow at the Manhattan Institute.
