This more or less captures my suspicions about half the papers I read these days:
I think the authors go too far, however. Sub-groups are not only useful, they are essential. Take any economic development randomized trial. The subgroup pattern actually let us test competing theories of poverty.
The essential bit, as I see it:
- Specify the theory, the subgroups, and the data beforehand, then show all the results.
- Withhold judgment until it’s replicated in another study.
- Or, if another study is unlikely (say, because it’s a rare natural expeirment) raise your power (expand your sample size or the frequency of measurement), split the sample randomly, do the subgroup analysis on one half, and test it on the other.
Maybe Andy Gelman or Gary King will blog and tell me if I’m crazy or not.
Hat tip to Eric Greeen.