Chris Blattman

Chris Blattman

Twitter Facebook-f Envelope Rss
Search
Close this search box.

Miguel-Kremer versus Cochrane: The battle of the meta-analyses

The latest salvo in the worm wars brings out big guns:

There is consensus that the relevant deworming drugs are safe and effective, so the key question facing policymakers is whether the expected benefits of MDA exceed the roughly $0.30 per treatment cost. The literature on long run educational and economic impacts of deworming suggests that this is the case. However, a recent meta-analysis by Taylor-Robinson et al. (2015) (hereafter TMSDG), disputes these findings. The authors conclude that while treatment of children known to be infected increases weight by 0.75 kg (95% CI: 0.24, 1.26; p=0.0038), there is substantial evidence that MDA has no impact on weight or other child outcomes. We update the TMSDG analysis by including studies omitted from that analysis and extracting additional data from included studies, such as deriving standard errors from p-values when the standard errors are not reported in the original article. The updated sample includes twice as many trials as analyzed by TMSDG, substantially improving statistical power. We find that the TMSDG analysis is underpowered: it would conclude that MDA has no effect even if the true effect were (1) large enough to be cost-effective relative to other interventions in similar populations, or (2) of a size that is consistent with results from studies of children known to be infected.
…Applying either of two study classification approaches used in previous Cochrane Reviews (prior to TMSDG) also leads to rejection at the 5% level.
…Under-powered meta-analyses (such as TMSDG) are common in health research, and this methodological issue will be increasingly important as growing numbers of economists and other social scientists conduct meta-analysis.

A new NBER paper by Croke, Hicks, Hsu, Kremer, and Miguel (ungated version here).

I think the most confident prediction that can be made is this: a new meta-analysis by TMSDG or someone will argue something different, a response analysis will find statistical flaws, and so on, until it doesn’t matter anymore.

I say “until it doesn’t matter anymore” because I recently spoke to a large charitable organization who basically said, “yes we could fund more long term studies looking at the economic impacts of deworming, but by the time the results arrived in 10 years it wouldn’t matter anymore because we’re probably going to eliminate worms in those areas in 10 years anyways.” Hm.

At current publishing speeds, we can expect at least 10 more battling meta-analyses before then. I’m sitting on the sidelines from here on out. Pass the popcorn.

31 Responses

  2. Given that eliminating worms is agreed to have some positive effect and that we live in a low interest rate world, eliminating worms from an area entirely is likely to have much larger net present value than the individual level studies show because you also then eliminate all future worm infections for zero marginal cost.

  4. The same fate happened to the impact of microfinance in Bangladesh debate. The whole thing ended up an archaic debate as to whether it is appropriate to discard outliers, log-linear versus linear functional form, least-square versus maximum likelihood, and local versus global maxima. In the meantime, microfinance marches on in Bangladesh

Why We Fight - Book Cover
Learn More
Subscribe to Blog

Recent Posts