Today we ran a very public randomization for our field experiment in Uganda. Village and parish leaders gathered for a presentation and meeting, and after introductions and some discussion, everyone lined up to take turns selecting villages into treatment and control. Despite some hiccups, a big success.
The trouble with randomized experiments, some fear, is the randomization. In some instances I’d agree. Here, the leaders we met seemed to be pleased that aid was finally being allocated transparently and fairly.
In this case the lottery was particularly easy. There is no permanent control group, merely a Phase 1 and Phase 2. The control villages will receive the program in 18 months time (a reasonable consolation). The NGO is running at max capacity just to fit in all the Phase 1 people, and the leaders seemed to like the fact that no favorites were played.
Even so, there were collective cries and laughs when one parish’s villages would all get Phase 1, and another would get nearly all to Phase 2. The only truly sorrowful cry came from the blogger/econometrician at the table, as he saw some of his statistical power drift away.
Ah, to be able to block and randomize on a computer…
But some level of blocking/stratification is possible even with a public lottery right? The fairness principle can be pushed one level down and you can say that we'll make sure all parishes get a proportional number of villages in Phase 1! It should work – though it's potentially more baskets. Did you consider it?
Also, the bigger risk in public randomization is substitution bias – since a district-level government may decide to prioritize other spending to the control group – and the fairness criteria would support that. Is this a problem for you? Can you at least measure it to make sure it's not a problem for the analysis?
Good job regardless!
According to Gary King, pairwise randomization is where you get the most leverage because non-compliance means you only through out one pair rather than having your entire pool potentially contaminated. However, it's far harder to sell the norms necessary to do pairwise randomization publicly.
I've had a number of conversations with people in the NGO world about randomization and they just don't like it, it seems deeply unfair to them. They think that way even though they don't have the capacity necessary to scale up to all localities at once, but (in these conversations) they argue that other norms are more important. I'm wondering if the NGOs are more bothered by the randomization than are the constituencies receiving the services …
When you don't have enough resources to get *everyone*, rando is really helpful–the villagers I worked with in Ghana laughed and cried based on the number they drew, but in the end we never once heard anyone cry foul. This is a big deal in places where people think things are generally rigged.