Chris Blattman

Search
Close this search box.

Worm wars continued (but not by me)

gummy-worms-1538377

Michael Clemens and Justin Sandfur at CGD weigh in:

Suppose a chemistry lab claimed that when it mixed two chemicals, the mixture rose in temperature by 60 degrees. Later, a replication team reviewed the original calculations, found an error, and observed that the increase in temperature was only 40 degrees.

It would be strictly correct for the replication team to announce, “We fail to replicate the original finding of 60 degrees.” That’s a true statement by itself, and it doesn’t fall within the strict purview of a pure replication to do additional tests to see whether the mix rose by 30 degrees, or 40 degrees, or whatever.

But it in this situation it would be excessive to claim that replication “debunks the finding of a rise in temperature,” because the temperature certainly did rise, by a somewhat different amount. This is basically what’s happened with the deworming replication, as we’ll explain.

I haven’t seen many non-economist responses, other than Stéphane Helleringer’s comments on this blog. Have I missed them or they don’t exist?

3 Responses

  1. Reposting an observation I made on Clemens & Sandefur:

    Without quarreling with the main thrust of the argument, I think there is a logical flaw in the analogy. If the question is whether or not combining two chemicals raises the temperature, then replication has been achieved. However, if the question is what is the most cost-effective way to raise the temperature, and other, equally-cheap methods are known to raise the temperature by 50 degrees, then the quantitative difference in the results of the replication experiment is significant. And surely that is what all the comparative philanthropy rankings are all about! Having said that, I doubt that the evidence in favour of other interventions to support improved educational outcomes is any more robust.

    (Apologies for cross-posting, but it seemed relevant.)

  2. Additionally, as far as chemistry is concerned, any replication team making those kinds of statements would pretty much permanently lose all credibility as scientists – their professional careers would be significantly damaged, if not completely destroyed.

    As far as I can tell, these sort of reputational consequences are restricted to the hard sciences. Feel free to correct me if I’m wrong.

  3. As a professional scientist, I would have to say that it would be much more than “excessive” to claim that the replication “debunks the finding of a rise in temperature”. That statement would be both false and maliciously deceptive. Additionally, the statement “We fail to replicate the original finding of 60 degrees.” without a further explanation that the temperature rise was replicated, but the magnitude of the rise was not, while strictly correct, would also be maliciously deceptive, albeit not to the same degree.