Ioannidis says most published research findings are false. This is plausible in his field of medicine where it is easy to imagine that there are more than 800 false hypotheses out of 1000. In medicine, there is hardly any theory to exclude a hypothesis from being tested. Want to avoid colon cancer? Let’s see if an apple a day keeps the doctor away. No? What about a serving of bananas? Let’s try vitamin C and don’t forget red wine. Studies in medicine also have notoriously small sample sizes. Lots of studies that make the NYTimes involve less than 50 people – that reduces the probability that you will accept a true hypothesis and raises the probability that the typical study is false.
Alex Tabarrok breaks down the Ioannidis argument and applies it to economics. Economics may do better, he says, because our sample sizes are larger and there is more theory to guide what we do.
Agreed, but I am not sure theory is used often enough. The number of theory-less papers that cross my desk, searching for cross-national correlates of poverty or conflict, makes me despondent. The number of famous papers that fall apart when you replicate the results with three more years of new data furthers my skepticism. I don’t know if I believe a single cross-national result any longer.
Alex’s bit is worth reading in full–two or three times, to wrap your brain around it.