Comments on: Am I actually sticking up for the Millennium Villages?

By: Gabrielle Blocher

Gabrielle Blocher — Thu, 28 Jan 2010 19:03:18 +0000

I am managing a team trying to do this the right way in neighboring Kenya for Nuru International. We have been on the ground for just over a year and have a similarly modeled five-year exit plan. Last year, we collected baseline data, then 8 months later, brought in 3rd part evaluators to measure our progress toward our goals for all of our metrics. This year we are beginning scaling efforts and we hope to use those efforts as a means to begin conducting randomized comparisons of our interventions with neighboring communities. It IS hard, but we are trying to start-out the right way, despite the stumbling that we’ve done.

By: The Millenium Villages Controversy « The Unended Development Quest

The Millenium Villages Controversy « The Unended Development Quest — Tue, 20 Oct 2009 21:16:53 +0000

[...] Yet Chris Blattman says we shouldnâ€™t ask for rigor that isnâ€™t feasible. [...]

By: David Levine

David Levine — Mon, 19 Oct 2009 04:14:10 +0000

Millennium Villages reports: “By the end of 2008, the MVP… served over 400,000 people in 14 sites in ten countries.” http://www.millenniumvillages.org/docs/MVP_Annual_Report_2008.pdf

There would have been no problem finding credible comparison groups (even if not randomized) for a sample of 14 clusters with multiple villages and 400,000 people. MVP has foregone a useful opportunity by not collecting baseline and follow-up data on nearby comparison communities. That opportunity would have added credibility to their claims of improvement and would have modelled how to bulid learning into more development projects.

By: Another anonymous!

Another anonymous! — Fri, 16 Oct 2009 14:20:11 +0000

Also big fail on my part for being anonymous since google helpfully shows my picture!

By: Another anonymous!

Another anonymous! — Fri, 16 Oct 2009 14:19:05 +0000

Just a point on the design of MVP. I believe the aim was for these projects to “demonstrate” what can be done with multiple interventions – but I think this means demonstrate in terms of visibility rather than statistically. In this sense they have probably achieved their aim – and for this their design was adequate.

I agree with Chris that one very useful thing would be for them to document and share their lessons learned from implementation along the way to help inform other programmes about the practical challenges they faced and how they might be overcome – or what surprising things they learned along the way.

By: The original anonymous

The original anonymous — Thu, 15 Oct 2009 20:18:39 +0000

Certainly, with big effects, randomization is not that important.

But then again, with careful planning, randomization is not that hard or expensive.

I believe a project claiming to be a leader in the field should have led by example.

By: Anonymous

Anonymous — Thu, 15 Oct 2009 19:40:58 +0000

I think I’m with you on this one. This is a big enough push that maybe one might define failure as anything short of overwhelming change that dwarfs *every* plausible counterfactual. In which case, there’s no need to approximate one.

In 15 years, if the Millennium Villages look like La Jolla, California, and the neighboring villages are all empty because people have voted with their feet, then you really would have to be a pedant to claim the MVP wasn’t responsible for that in every relevant-to-the-real-world sense.

On the other hand, if 15 years from now the argument is whether we can causally attribute to the MVP whatever small differences are observed from whatever crude approximation of the counterfactual a reasonable person might be inclined to point to, well, then the very fact that the question is out there means the project failed.

By: The headless heart « Aid Thoughts

The headless heart « Aid Thoughts — Thu, 15 Oct 2009 15:08:36 +0000

[...] Chris Blattman suggests that testing the impact of the Millenium Villages might be infeasible. [...]

By: Ryan

Ryan — Thu, 15 Oct 2009 15:01:39 +0000

You are right that it would be extremely expensive to test the big push theory, but that doesn’t mean that they shouldn’t be testing each piece of the push independently and, where sample size allows it, their interactions. Is Busia, Kenya, not an example of an MV-like area that is essentially getting a big push, but testing each component along the way? Why not have Millennium locations or sub-locations and randomize across villages, schools, households, and individuals within these areas? Idle speculation that the sample size would be too small is unproductive.

By: Anonymous

Anonymous — Thu, 15 Oct 2009 14:18:22 +0000

I guess what you are saying is that the MV project is so badly designed from the start that it cannot be randomly evaluated. I think that is an indictment of the project not a defense. It is very hard to argue that proper evaluation would not have been possible if conceived from the start.

And I disagree completely about sample size to test big push theories. Presumably these effects will be huge (big push) and cumulative over time (take off). We actually don’t need large samples to detect these huge effects.

Frankly, there is no excuse for a project as visible, important and expensive as this not to have thought more carefully about evaluation from the outset. Least of all when its sponsors will use it to make strongly worded policy recommendations.