What’s the alternative to randomized control trials in development research?

Daron Acemoglu calls for more structural theory, more general equilibrium, and more political economy in development economics.

Here’s an example.

As an example of composition e¤ects, consider the problem of estimating the importance of credit market imperfections. Banerjee and Du‡flo (2005) survey a large body of evidence that small and medium-sized businesses in less-developed economies are credit constrained and an extension of credit to these businesses will make them increase production.

Now consider the effect of a large-scale policy of credit expansion to small- and medium-sized businesses. This policy could lead to a different type of composition effect than the one operating in partial equilibrium.

For example, it may be the case that in partial equilibrium estimation focusing on …firm-level variation we found that …firms with better access to credit expanded, but this was at the expense of other …firms that did not have access to credit (that is, partly by “stealing business” from others).

And yet, the same response cannot take place in general equilibrium. As a consequence, when additional credit becomes available to a large fraction of… firms, total output may not increase by as much or at all.

One could thus imagine a situation in which partial equilibrium estimates of relaxing credit constraints are large, while the general equilibrium e¤ects would be small.

Well, ‘complement’ is probably a better word than ‘alternative’.

The entire profession has gravitated towards more field experiments. One of the leading lights has just gotten the MacArthur genius award and the Clark Medal. If I were a first year graduate student, I’d start running in the opposite direction of RCTs. Acemoglu’s giving you a new target.

13 thoughts on “What’s the alternative to randomized control trials in development research?

  1. As a grad student about to start his first year, thanks for the link! :)

  2. I couldn’t disagree more.

    If I understand you correctly, you’re implying there is about to be some sort of over supply of folks doing RCT or other field research, which if so, is quite questionable.

    Moreover, what is the right equilibrium between theory and field work?

    Perhaps the movement toward field work is just the natural response toward its unnatural suppression for so many years — like that seen by women or minorities who are now entering professions from which they were previously discouraged or prohibited from entering…. Still waiting for that to happen in econ though….

  3. I guess I’m conflicted. On one hand, I’d like to see more RCTs, especially where they replicate prior work in new contexts in an effort to establish more external validity. On the other hand, such an approach doesn’t always work. For example, I’m a little concerned about the conflation of experiments that target solutions to poverty with “aid effectiveness,” as the latter is a mechanism that (occasionally) actually targets particular interventions. These experiments aren’t well connected to the fact that many of these interventions are funded by foreign organizations and often channeled through international, then national, then regional, then local bureaucracies. If you could randomly assign the bureaucratic nonsense that happens at each level, then you might start making some headway on “aid effectiveness.”

  4. I’m not saying there is no role for RCTs in development. Many will continue, as they should. But as a grad student, your job is to write a dissertation at the frontier of your profession. I’m simply suggesting that you don’t look for the frontier in what was popular yesterday, and begun a decade ago.

  5. Any advice for undergrads who will need to be on the frontier of research in 5 years?

  6. RCTs are so expensive it’s hard to imagine them ever dominating the field.

    Drug trials provide a private benefit to specific companies with deep pockets. whereas RCT researchers will always be competing for a small limited pool of funds.

  7. the problem of resource allocation is more than a hundred years old, should I abandon that?

    Just focus on what’s in your interest people.

  8. Why position yourself on the frontier of a field? Why not pick an interesting topic and do it as well as you can, using the best methods available? It means that the scholarship might continue to be cited even once the material is less trendy.

    As for RCTs, sure they’re trendy today and wont look as shiny and hip in 10 years, but that doesn’t mean they’ll necessarily be marginalized in that period. This could be like the explosion of econometrics that followed the rise of desktop computation in the 1980s.

  9. i’d love to see undergrad’s question answered as well. what’s your idea in which direction the profession is heading?

  10. Thanks for the link and advice Chris. This just gave me a John Bates Clark medal idea – I’d like to start a JPAL for general equilibrium studies called PAGES, the P A General Equilibrium Studies for development.

  11. I’m a tad skeptical. Didn’t the shift toward RCTs represent a shift in the scope of questions the field had legitimate space to provide answers? A shift of emphasis toward intervention-effect validity and away from generalizability? No doubt GE and PE considerations need to be made in the course of making policy interventions, but I do wonder how fruitful (in the sense of capably informing practice to truly improve outcomes) the empirical work will turn out.

    I’d second Paul Johnson’s comment above, and argue, if anything, that more resources might be put into developing (pun?) more sustainable revenue models for RCT studies.

  12. What’s THE alternative to RCTs? It reminds me of that great line in the Blues Brothers “We like BOTH kinds of music – country AND western.”

    It’s important for reseachers and policy makers to keep in mind four approaches to causal inference in impact evaluation – experimental designs (RCTs), quasi-experimental designs (eg constructed comparison groups), non-experimental designs. (eg process tracing, comparative case studies), and statistical modeling approaches such as general equilibrium.

    There are many cases where non-experimental approaches are needed for impact evaluation, and we do a disservice to the field to leave them out of our toolkits – for example:
    When interventions are not focused at individual change but at system level change (eg national anti-corruption initiatives)
    When interventions cannot be rolled out gradually but do have universal application immediately (eg national policy)
    When access to the intervention cannot be controlled or randomized (eg information campaigns with a lot of leakage and/or secondary transmission, or selective programs such as scholarships or advanced training)
    When population numbers are too small for adequate sample size
    When the intervention is responsive and adaptable so the intervention is not constant
    When an experimental design has not been established at the beginning and a credible comparison group cannot be constructed
    When there are not resources to undertake an experimental impact evaluation