Replication data (and the perils thereof)

IPA and J-PAL have a new data replication archive. Most or all of my published papers make available full, original survey datasets in addition to the paper replication data, and I’m starting to archive all the data in the new IPA/JPAL dataverse. Here is one. Highly recommended.

I do wonder a little about perverse incentives for people replicating the paper, however. Here is a reply from Stefan Dercon and coauthors to a 3ie-sponsored replication exercise.

In this reply, we explain why we welcome the principle of replication studies. We document how we have approached cooperation with the replicator but also express disappointment in how this process seems to have operated in practice; specifically the extent to which it created incentives to go beyond replication until methods and data were found that yielded different results.

We are glad to note that in terms of pure replication, our results are confirmed beyond a minor coding error that did not matter for either the results or their interpretation. We are disappointed, however, that the replication study is selective in reporting our own cautious discussion on method and robustness in both our original AJAE paper, and a subsequent paper in Journal of Development Studies. We quote our own papers on how we already addressed a number of the concerns raised in this study and why we judge these innovations as being difficult to consider as ‘superior’ both in principle and in the way they are applied.

The study places considerable weight on the robustness of our results on agricultural extension but ignores that we have highlighted as much in both papers before. We are not convinced that there is much value added in the part of the study that investigates robustness rather than just replicability.

Berk Ozler has commented previously, and I thank the Development Impact blog for the links.

Update: Great thoughts on replication etiquette by 3ie’s Annette Brown

26 Responses