This is how to recognize bad science


Rather, my message is that this noisy, N = 41, between-person study never had a chance. The researchers presumably thought they were doing solid science, but actually they’re trying to use a bathroom scale to weigh a feather—and the feather is resting loosely in the pouch of a kangaroo that is vigorously jumping up and down.

Andrew Gelman critiquing a Kahneman psychology study. Via one of my favorite blogs.

Health, psychology, and exercise studies are, in my experience, the worst kangaroo-jumping feather-weighers.

One of the most important things you can do as a consumer of science is also the simplest: check the N. If you are reading a peppy New York Times study about how coffee makes you live longer, or exercise doesn’t help you lose weight, more often than not the study had 30 subjects and you can simply ignore the information you just received.

If you want to go a little bit further, you could also say “is this the only outcome I care about?”