Photo of the day: How researchers really work

This more or less captures my suspicions about half the papers I read these days:

From a sensible rant against the practice.

I think the authors go too far, however. Sub-groups are not only useful, they are essential. Take any economic development randomized trial. The subgroup pattern actually let us test competing theories of poverty.

The essential bit, as I see it:

  • Specify the theory, the subgroups, and the data beforehand, then show all the results.
  • Withhold judgment until it’s replicated in another study.
  • Or, if another study is unlikely (say, because it’s a rare natural expeirment) raise your power (expand your sample size or the frequency of measurement), split the sample randomly, do the subgroup analysis on one half, and test it on the other.

Maybe Andy Gelman or Gary King will blog and tell me if I’m crazy or not.

Hat tip to Eric Greeen.

One thought on “Photo of the day: How researchers really work

  1. Hi, Chris. Replication, getting more data, raising your power, etc. are all good ideas, but in the meantime you have to make do with the data you have. If the alternative to subgroup analysis is to assume, implicitly or explicitly, that effects are constant across groups, then you have to decide whether that’s an assumption you’re comfortable with. The ultimate solution, I think, is some partial pooling–that is, multilevel modeling.