The difficulty of generalizing RCT results (the RCT result)

The recent wave of randomized trials in development economics has provoked criticisms regarding external validity and the neglect of political economy. We investigate these concerns in a randomized trial designed to assess the prospects for scaling-up a contract teacher intervention in Kenya, previously shown to raise test scores for primary students in Western Kenya and various locations in India. The intervention was implemented in parallel in all eight Kenyan provinces by a non-governmental organization (NGO) and the Kenyan government. Institutional di fferences had large e ffects on contract teacher performance. We nd a signi cant, positive e ffect of 0.19 standard deviations on math and English scores in schools randomly assigned to NGO implementation, and zero e ffect in schools receiving contract teachers from the Ministry of Education. We discuss political economy factors underlying this disparity, and suggest the need for future work on scaling up proven interventions to work within public sector institutions.

A new paper by Bold, Kimenyi, Mwabu, Ng’ang’a and Sandefur.

30 Responses

  2. Pingback: Links of the day | Rachel Strohm

  4. This is a great empirical result but also completely unsurprising result as there never was any plausible positive theory of education policy formulation and implementation in Kenya in which rigorous knowledge about what works was the key constraint to better policy.
    (http://www.hks.harvard.edu/fs/lpritch/Education%20-%20docs/ED%20-%20Gov%20action/NAPwrong_PolicyIrrelevance_Brookings_final.pdf)
    (or, for that matter, in any other place or any other policy domain).

  5. So if now the only available “rigorous” evidence acceptable to the randomista crowd “proves” (with equal rigor to anything being proved by any RCT) that rigorous evidence doesn’t impact policy will the advocates of rigorous evidence be consistent with their own ideology and stop or will they continue their faith based advocacy for more resources for RCTs? I’ll give everyone exactly one guess.