Following my post on misleading methodological wars in political science this morning, I saw for the first time David McKenzie’s blog post on whether randomized control trials (RCTs) have taken over development economics:
Another claim is that the “best and brightest talent of a generation of development economists been devoted to producing rigorous impact evaluations” about topics which are easy to randomize and that they take a “randomize or bust” attitude whereby they turn down many interesting research questions if they can’t randomize
To explore this, I examined the publication records of the 65 BREAD affiliates (this is the group of more junior members), restricting attention to the 53 researchers who had graduated in 2011 or earlier (to give them time to have published). The median researcher had published 9 papers, and the median share of their papers which were RCTs was 13 percent. Focusing on the subset of those who have published at least one RCT, the mean (median) percent of their published papers that are RCTs is 35 percent (30 percent), and the 10-90 range is 11 to 60 percent. So young researchers who publish RCTs also do write and publish papers that are not RCTs. Indeed this is also true of Esther and her co-authors on this paper (Abhijit Banerjee and Michael Kremer) – although known as the leaders of the “randomista” movement, the top-cited papers of all three researchers are not RCTs.
And as for journals:
RCTs are a much higher proportion of the development papers published in general interest journals than in development journals. However, even in these journals they are the minority of development papers – there are more non-RCT development papers than RCTs even in these general journals. Moreover, since most of the development papers are published in field journals, RCTs are a small percentage of all development research: out of the 454 development papers published in these 14 journals in 2015, only 44 are RCTs (and this included a couple of lab-in-the-field experiments). As a result, policymakers looking for non-RCT evidence have no shortage of research to choose from.
Read the full post. Here is a graph.
David was responding to Esther Duflo’s talk on the subject.
See my comments on why I think you can explain these trends with normal responses to technological change. Does this mean we have reached peak RCT? I think so.
Hat tip to the excellent FAI newsletter, to which I subscribe..