Guest Blogger: Eric Green is an Associate at the Population Council and a collaborator on the WINGS impact evaluation in Uganda.

Julian beat me to posting about the new working paper on impact evaluation by Michael Clemens and Gabriel Demombynes. Both authors blogged about the paper here (MC) and here (GD). The paper discusses the costs and benefits of “careful” impact evaluations through a case study of a well-known initiative, the Millennium Village Project (MVP). The authors define “careful” (or “rigorous”) impact evaluation as:

the measurement of a policy‘s effects with great attention to scientifically distinguishing true causal relationships from correlations that may or may not reflect causal relationships, using ―well-controlled comparisons and/or natural quasi-experiments (Angrist and Pischke 2009: xii).

Michael and Gabriel conclude that the current MVP evaluation plan does not allow for a “rigorous” evaluation of project impacts and suggest modifications to help overcome several methodological hurdles going forward.

I appreciate the authors’ focus on the need to understand impact before scaling-up massive programs like the MVP. Here in New York we have a local example of the MVP impact debate taking place several blocks from Columbia University’s Earth Institute: the Harlem Children’s Zone (HCZ). If you don’t know about HCZ, just turn on your TV. HCZ founder Geoffrey Canada, who really seems like a true innovator, is featured in an American Express Member’s Project ad that sums up the HCZ approach: “to help kids in a sustained way, starting as early in their lives as possible, and to create a critical mass of adults around them who understand what it takes to help children succeed.”

Just like the MVP model, HCZ-inspired change does not come cheap. But we should not expect it to. As Mr. Canada is quoted as saying in a recent NYTimes article:

‘You could, in theory, figure out a less costly way of working with a small number of kids, and providing them with an education,’ Mr. Canada said. ‘But that is not what we are attempting to do. We are attempting to save a community and its kids all at the same time.’

It’s an ambitious and laudable goal, just like MVP. But is it ‘working’?

The short story is that it’s probably too soon to tell. But this might not stop the policymaking process. As discussed in the article, the federal government is looking to replicate this model elsewhere. Back in 2007, Senator Obama made the following remarks about HCZ:

And we have to focus on what actually works . .  . And it is working . .  . And if we know it works, there’s no reason this program should stop at the end of those blocks in Harlem.

A Brookings Institution report references this quote and suggests that the picture is not so clear cut.

Doing what works depends on evidence not instincts. There is no evidence that the HCZ influences student achievement through neighborhood investments. There is considerable evidence that schools can have dramatic effects on the academic skills of disadvantaged children without their providing broader social services. Improving neighborhoods and communities is a desirable goal in its own right, but let’s not confuse it with education reform.

Clearly an interesting addition to the broader conversation about how we know ‘what works.’ Have your say in the comments.