Go short on randomized control trials?

What would be the effects of disbursing $1-1.5 billion of foreign aid to Pakistan? RCTs [randomized control trials] do not, and cannot, have anything to say on the matter—not only because of their narrow focus and applicability, and hence non-generalizability, but also because they cannot speak to macroeconomic effects.

…We are still left to rely on other evidence—economic and historic—about the effects of aid in stunting institutional development, in creating aid-dependence, in entrenching the hold of the bad guys, and in making the export sector uncompetitive in a way that is detrimental to long run development.

That is Arvind Subramanian responding to Nick Kristof’s ringing endorsement of RCTs in development.

Clearly I think RCTs have a huge role to play in aid and development research, but should young academics be answering this clarion call?

If I had to pick development research like stocks, I would be shorting program evaluations and buying the unfashionable stuff–including the macroeconomic, historical and institutional research.

I have substantive reasons for saying so, but as a general rule, any research trumpeted in the opinion column of the Times is probably on its way off the frontier.

5 Responses

  1. Jason, I’d urge you to take a look at “Contesting Development” by Barron, Diprose, and Woolcock.
    http://yalepress.yale.edu/book.asp?isbn=9780300126310

    It presents a complicated institutional reform initiative (the KDP project in Indonesia) which established mechanisms for participatory local governance, and discusses in detail the evaluation the team did to evaluated it (NOT an RCT), and presents the findings.

  2. The questions is: will RCT become standard fare in development practice (meaning the governments will move to a post Paris agenda, away from the Big Plans and into more focused action; or will RCT go the way of the dodo, ignored by the politicians and institutions who hold the keys.

    In the first case, young academia could start ignoring it, but graduates can get a good professional life out of doing RCT for the next generation.

    In the second case, everybody can ignore it.

  3. In my view, expecting RCTs, for that matter a single method, to answer every question is like trying to put in your trouser through your neck. It defies commonest of all common senses.

    However, to respond to Subramanian, Alberto Abadie has been thinking about this problem for some time. He has suggested what is called a Synthetic Control Technique that is very suited to handle macro effects. Although it is short of exact RCT, in my view, it is by far better than cross-country regressions that try to compare countries of heterogeneous character.

    An interested person needs to see his AER 2003 joint paper to get see how synthetically generated control can help us take into account macro effects, although much improved and refined paper which came out in Journal of the American Statistical Association 2011 is the best one.

  4. My experience of developing country government suggests that the binding constraint is not knowing what policies to pursue, but knowing how to implement them. I don’t think RCTs can answer this, and it is the reason why so far they have been largely irrelevant for policy decisions.