So you want to be an impact evaluator? A cautionary tale

Time for a little general advice (if only to avoid writing the same inadequate e-mails over and over again).

Several aspiring graduate students have written me about becoming an impact evaluator. ‘What degree to get and where?’ is the most common question.

I think the best advice is: don’t get a PhD to do evaluations. The randomized evaluation is just one tool in the knowledge toolbox. It’s currently the rage, but that means it will probably be old news by the time you finish your PhD.

Yes, the randomized evaluation remains the “gold standard” for important (albeit narrow) questions. Social science, however, has a much bigger toolbox for a much broader (and often more interesting) realm of inquiry. If you want to know the effects of small binary treatments, you are in business. If you find any other question in the world interesting, you have some more work to do. Dani Rodrik has made a similar point here.

Don’t get me wrong: a large number of my projects are randomized control trials. They are eminently worth pursuing. But to be honest, uncovering the causes of effects excites me more than measuring the effects of causes. An evaluation masters the second, but only hints at the first. The hardest and most rewarding work is the theoretical and investigative work that comes with uncovering the underlying rhythms and rules of human behavior.

The best advice I can give to aspiring researchers: apply to PhD programs that will give you the best all-round training in as many different tools and pools of knowledge as possible.

Also: use your schooling time to tech up in formal theory and statistics (plus qualitative and comparative methods if you are a political scientist). Once you are finished, you won’t have time to acquire these skills. From the day you finish your PhD, it is a slow but steady descent into technical obsolescence.

If you’re interested in becoming a professional evaluator, rather than an academic, my advice changes little. It is now possible to be an evaluation consultant in much the same way that you can be an accountant or a lawyer–a highly specialized professional, with interesting and rewarding work. A PhD helps, but I don’t think it’s requisite. Yet I would still make the same case: you will be a better consultant, manager, and professional if you have a broad range of skills and knowledge.

If your goal is to improve the delivery of aid, and truly advance development, many more skills and knowledge are involved than the randomized evaluation. See here for more. But in short: a well-identified causal impact that arrives two years after the program does not performance management make.

For aspiring professionals, a masters in statistics with an MBA or MPA may be preferable, along with plenty of experience on the ground–preferably working inside a developing country government, not an aid agency. If nothing else, you may need a different set of skills if the fad ever fades.

I’m going to see if some of the other academic bloggers want to weigh in on this debate. Owen, Dani, Shanta, Nancy, Tyler, Brad: can I tempt you? Other bloggers welcome–I will post links here.

15 Responses

  1. Hi, Chris. I happen to read this blog when I was googling ‘how to get experience in policy evaluation’. Having spent an year in one of the most poor districts in India for a year, I now, genuinely would like to work on improving development policies in my country. I have some knowledge of policy evaluation methods and I have worked with few government institutions but I wouldn’t claim it to be a fruitful experience. I would really like to gain experience in policy evaluation. Could you suggest some posibilities to connect with people working in India in Policy/ Program evaluation? to work as intern/ research associate etc. to begin with?

  2. For those interested in pursuing advanced level training in impact assessment, the University of East Anglia School of International Development (where I am a PhD candidate) offers both a Masters and Short Professional course:

    “The MSc Impact Evaluation for International Development degree offers familiarisation with and skills in the basics of modern evidence-based policy-making and impact evaluation, including the contexts and practices of evaluation, research design and data production for evaluation, and basic and more advanced methods of quantitative and qualitative analysis.

    “The course has been designed for students who are interested in designing and implementing development projects and programmes and/or in researching development effectiveness, and who need to develop and enhance their skills for undertaking high-quality rigorous impact evaluations.”

  3. First off, Chris, thank you so much for posting this. In fact, I am an evaluation consultant in the int’ development field and have done some baseline studies for an impact evaluation and taken short courses. I was considering a PhD in economics or an add’l Master’s to give me the tools and knowledge I need to apply a RCT to education development projects. As you state, it is the fad now, and to survive as consultants we must confirm, plus I have a personal interest. Do you suggest I pursue a Master’s in Public Policy rather than a PhD. Or if there a specific PhD program that covers both RTCs and other methods that could advance my consulting career. Thank so much for your advice!

  4. Pingback: On data-driven policy and the continuing rationalization of governance « The Widening Gyre

  5. I’d be curious if you have any related thoughts for current PhD students considering undertaking an RCT as part of their dissertation. How feasible is that from your perspective, and what are the limiting factors (assuming, for starters, that you aren’t already plugged into the JPAL-Harvard-MIT network)?

  6. “uncovering the causes of effects excites me more than measuring the effects of causes”

    Isn’t there a lot of economic theory out there that doesn’t hold water? Isn’t the use of randomized trials one effective tool to test theory?

    “The hardest and most rewarding work is the theoretical and investigative work that comes with uncovering the underlying rhythms and rules of human behavior.”

    What other “investigative work” are you referring to that can help unravel the causalities like RCT?

  7. Chris B writes:

    “But to be honest, uncovering the causes of effects excites me more than measuring the effects of causes. An evaluation masters the second, but only hints at the first.”

    Isn’t the point of your Impact Evaluation 2.0 DFID presentation that impact evaluation methodologies can be improved and used to uncover causes behind effects?

  8. I can’t speak much to international development, but I do know that there are a lot of incentives against real evaluation in domestic programs, chiefly because of funding and time constraints. I discussed the issue here and here in the context of grant writing.

    I know little about how evaluations work in developing countries, but I imagine the incentive problems are not so different.

  9. From a medical point of view… Among the principal limitations of most RCTs is the population studied. The population studied typically differs systematically from the population that will receive the treatment, for myriad reasons. If there are treatment x person effects — and there almost always are — the difference matters. The point is obvious, but everyone in the business has huge incentives to forget it. I am currently running an RCT, but I recognize that any serious attempt to understand a treatment has to be addressed from many methodological perspectives.

  10. A separate, but related, question: if you are in grad school already and are trying to apply to various places for funding, what recommendations would you have for someone interested in evaluation of development policy who needs funding? What are the dos and don’ts? To whom should, or to which foundations, should one apply? What have your experiences been like with getting funding for REs or other evaluation projects?

    For whatever reason, skills related to ‘drawing up a funding proposal’, or ‘knowing who to apply to’ often seems to be a skill you are meant to have as a grad student, but which you are seemingly meant absorb from profs, etc in academic osmosis, but I haven’t seen many explicit guides, say ‘getting academic funding for the n3wb’, or ‘an idiot’s guide to project funding’.

    The problem, in my mind, is that better connected professors result in better connected grad students which results in perpetualization of particular students in particular schools getting funding, rather than a greater transparency and liberalization of ‘funding knowledge’. This could be complete speculation, but in my rather anecdotal observations it seems to be an accurate depiction of some funding processes I’ve observed.

  11. “The hardest and most rewarding work is the theoretical and investigative work that comes with uncovering the underlying rhythms and rules of human behavior.”

    Amen. I just wish I had the time and resources to approach these questions from all angles. I’m working on a B Sc in health right now, but I’m shoehorning as much psych, history, geography, and other things into five years as I can. There are so many interesting academic/career possibilities out there; the decisions I have to make require closing many doors.

    It is better, I suppose, to have too many choices than too few. Most people are not so lucky to make these “hard” choices.

  14. A post that I think will be helpful to many aspiring graduate students, as it has been to me. Thank you.

  15. Chris, can you share with us some interesting research questions you have that a randomized evaluation (plus some process monitoring) cannot help answer?