Writing in the New Yorker, Jonah Lehrer highlights the slipperiness of empiricism:
all sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable.
…In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.
The culprit? Not biology. Not adaptation to drugs. Not even prescription to less afflicted patients. Rather, it’s scientists themselves.
Journals reward statistical significance, and too many academics massage or select results until the magical two asterisks are reached.
But more worrisome is that much of the problem might be more unconscious: a profession-wide tendency to pay attention to, pursue, write up, publish, and cite unusually large and statistically significant findings.
Social science is well behind natural science and medicine in registering experiments and replicating results. For me, it’s sobering to see that, having accomplished these feats, the harder sciences are still finding most of their cherished results disappear.
Social science suffers less from medicine’s problem of small sample sizes and gazillions of small experiments. There are limitations, however, we share. Take the classic Ioannidis piece, Why Most Published Research Findings Are False:
a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance.
Interest piqued? Alex Tabarrok on Ioannidis here.