Academic research: Use with caution

That’s the recomendation from Holden at GiveWell, after a long look at microlending debate between Mark Pitt and critics David Roodman and Jonathan Morduch.

Holden gives excellent advice. An excerpt:

Never put too much weight on a single study. If nothing else, the issue of publication bias makes this an important guideline…

Strive to understand the details of a study before counting it as evidence. Many “headline claims” in studies rely on heavy doses of assumption and extrapolation…

If a study’s assumptions, extrapolations and calculations are too complex to be easily understood, this is a strike against the study. Complexity leaves more room for errors and judgment calls, and means it’s less likely that meaningful critiques have had the chance to emerge…

If a study does not disclose the full details of its data and calculations, this is another strike against it – and this phenomenon is more common than one might think…

Context is key. We often see charities or their supporters citing a single study as “proof” of a strong statement (about, for example, the effectiveness of a program).

I could not have said it better. The post is worth reading in full.

Publication and confirmation bias are horrifically rampant. And data are seldom available. I am struggling to obtain 4-year old data for a replication myself, at the moment.

Two changes I would consider:

1. Journals should require submission of replication data and code files with final paper submissions, for posting on the journal site. (The Journal of Conflict Resolution is one of the few major political science or economics journals I know that does so faithfully.)

2. PhD field and method courses ought to encourage replication projects as term assignments. (Along with encouragements to diplomacy–something new scholars are slow to learn, to their detriment.)

Other suggestions for the profession?