Comments on: Should the Millennium Villages be randomly evaluated?

By: Henk J.Th. van Stokkom

Henk J.Th. van Stokkom — Fri, 26 Mar 2010 18:16:05 +0000

Having visited Sauri/Kenya in november 2006 and also a Millennium Village close to Zomba/Malawi in October 2009, I can see the enthusiasm of people involved. Changes being achieved.

But can it be done without Sachs, Jolie and the world watching?

I asked a Sauri citizen. He immediately shared my doubt on that but said that he was happy with everybody watching and supporting….. I would if was living in Sauri.
But can Jack and Jill in rural wherever do it?
Can they get local government moving on healthcare, electricity, roads and more.
Not without lots of millennium money and everybody watching.
There is no research needed to figure that out.

The problem with MV is also that even if it ‘works’, can it be scaled up ?

http://vanstokkom.blogspot.com/2007/03/rapid-victories-against-extreme-poverty.html

By: Andy Reid

Andy Reid — Mon, 22 Mar 2010 01:13:17 +0000

Unrelated, but I suspect you will dislike this latest initiative of Invisible Children. Who needs bednets, or deworming pills? Sounds like a waste of money, not like concerts!

http://www.kickstarter.com/projects/750755607/take-away-shows-in-uganda-with-polyphonic-spree

By: Michael Clemens

Michael Clemens — Mon, 22 Mar 2010 01:07:51 +0000

Thanks for this Chris. I really agree with the big point you’ve argued before on this blog, that “Evaluation 2.0″ requires all kinds of iterative and evolving learning, much of which is difficult to base on randomized evidence but can still be done rigorously. That said, I don’t agree that the “fundamental premise” of the MVs is untestable.

The fundamental premise of the MVs is that a certain package of local interventions — together, as a package — can break village clusters out of poverty traps and place them on the “ladder of development”. That premise is testable and can be either supported or rejected without knowing which elements of the package were or were not responsible for a given change observed in the villages.

What the MVP seeks to scale up is not different elements of the package in different places — schools in one place, fertilizer in other places — but rather what they seek to scale up is application of the package as a package Whether or not the *package* has placed village clusters on a long-term trajectory of economic growth is the evaluation question of interest for the MVs because that is their premise.

If I understand correctly, you’re making the good point that we would learn a lot more about village-level development interventions if we could test the effectiveness of individual interventions in isolation against the effectiveness of bundles (packages) of individual interventions. I’m sure you’re absolutely right when you say that such complex tests are probably infeasible with the sample sizes of the MVP, and when you say that such tests would be more revealing than a zero-one test of whether a certain package is capable of lastingly freeing villages from poverty traps.

But that doesn’t mean that such a zero-one test of the package per se is uninteresting or can’t be done. The MVP is making strong claims about the package, and testing what sort of lasting effects are attributable to the package per se is 1) possible with realistic sample sizes and 2) highly policy-relevant given the prominence and cost of the MVP.

By: Prabhas

Prabhas — Sun, 21 Mar 2010 17:43:54 +0000

Indeed, +1 Amanda.

“A gazillion dollars in aid produces good outcomes but they don’t last.” or “… don’t produce good” are still viable answers. That’s the fear, and thats what randomized evaluations can answer.

By: Amanda (big fan of mclem)

Amanda (big fan of mclem) — Sun, 21 Mar 2010 05:55:22 +0000

what evidence do you have that ‘a gazillion dollars in aid and lots of government attention produces good outcomes’? are we even confident in this?

you make a good point that subjecting the MVP to a RCT is hard. i agree that any kind of eval using a comparison group would be a step in the right direction. if this is really a ‘new approach to fighting poverty’ as the slogan suggests, we need to know if this does any good, and if the interventions merit the cost. (i know you know that)

but here are two things going for a RCT of MVP: 1) even evaluating an interaction of some of the interventions would shed light on this approach/help in understanding the ‘big push.’ what if the eval interacted, say, five of the MVP interventions? that to me is better than a matching/qual eval (not saying that qual shouldnt be done too). maybe this is what you are suggesting w ‘testing manageable claims’? 2) some of the best minds in development economics do RCTs. these people care about what is happening with a ‘gazillion’ in aid dollars, and i am confident a meeting of smart minds could come up w sensible approaches to deal with the big push challenge.

i hope the MVP team responds to you and michael. (i havent given up on sachs yet.)

By: edawg

edawg — Sat, 20 Mar 2010 15:15:30 +0000

I think your argument is a sound one, Chris. But aside from the specific question about randomized evaluation, it seems like there’s a valid broader critique of the MVs that they don’t seem to have been designed for any rigorous evaluation at all. Shouldn’t arguments like yours and Clemmons’ and Freschi’s have been aired at the inception of this massive effort, rather than at this late point in the game? After all, these things are largely publicly funded.

By: Ryan

Ryan — Sat, 20 Mar 2010 12:31:32 +0000

You are certainly right that we aren’t going to be detecting interesting results with N=13 villages. But I don’t think that we would have to measure the marginal impact of the individual interventions to learn anything useful; the individual components have already been measured elsewhere. The great opportunity of the MVs, then, would be to test packages of components. This would require assembling a package that you think will have interesting interactions, say fertilizer, savings products, and irrigation pumps. You could then phase in the packages, randomizing by individual, enterprise, school, location, etc. This would be somewhat counter to the big push idea that everyone gets everything at the same time, but everyone would get something, and everyone would get everything eventually. External validity may be low if the control is also part of an MV, so randomly selected control villages getting no services might be helpful as well. There may not be the power to test interactions between packages, but you could compare the package to existing evaluations of the components and come to a conclusion about that particular delivery module. This may not produce cutting-edge economics research, but at the very least there is a lot the MV movement itself could learn from program evaluations of groups of services as it scales up.

By: Sam Gardner

Sam Gardner — Sat, 20 Mar 2010 07:01:44 +0000

There is a the “placebo effect” in aid interventions: Attention from the international community has an effect that is sometimes more important than the effect of the added funding or expertise. I don have data to back this up, only some circumstantial evidence, but I have seem it time and again: under the spotlight, communities, administrations come up with their own solutions, get their act together and find a spring in their step. Money and consultants can be the oil for the chain, but the motor is this aid placebo effect.
With the Millennium villages, the spotlight is quite strong.