The publication bias problem (and the redemption of Blattman?)

After many years in the file drawer, I’m putting out a paper as penance.

Eight years ago, finishing a master’s program, I wrote a paper that made a simple argument:

  • Most countries, from Sweden to India, have produced and exported the same handful of commodities for more than a century–the commodities they were essentially “born” with.
  • Some commodities are more price volatile than others, and for most countries this price volatility is simply given to them.
  • The countries with the most price volatile commodities have grown the most slowly over history. Volatility is a huge predictor of growth.

For my PhD I decided I would figure out why. My hypothesis: it’s all about political instability. States come to depend on easily taxed commodities for revenues–fuels, minerals and tree crops. When prices plunge, so does the ability to buy off or repress opponents.

I dug around in the fifth basement of Harvard’s and Berkeley’s libraries for more price and export data, and I carefully built a new database. My dissertation committee was enthusiastic.

At long last, regressions were run and… no result. No relationship between price shocks and conflict, even in the most generous scenarios. I shrugged and thought, “Well, so much for that.” My committee said, “Huh, what about that child soldiering project we told you not to do?” And off I went on my career as micro-conflict man.

In the meantime, lots of papers that did see an impact of economic shocks on conflict or instability did get published. The conventional wisdom grew: Rising incomes made the state more attractive to rebels as a prize, and falling incomes made it easier to recruit rebels. No matter that these two ideas ran in apparently opposite directions.

Meanwhile, I met other academics that had run the same regression as me. Famous ones you have probably heard of. Their reaction was the same as mine: “Oh, I found that result,” several said, “but I’m worried there’s nothing there because my data have problems, and the specification wasn’t quite right. So I left it out of the paper. I’ve been meaning to get back to that.”

Let’s follow a simple decision rule: run your regressions with inevitably imperfect data and models. If you get the theoretically predicted result (any of them), publish. If not, wait and look into your data and empirical strategy more.

The result? As in the natural sciences, most published research findings are probably false.

After several guilty years, I brushed off my data with co-author Sami Bazzi, and set about the hundred small tasks and tests needed to believe something might not be true.

We finally have a working paper: Economic Shocks and Conflict: The (Absence of?) Evidence from Commodity Prices.

I will only attest: It is very, very difficult to argue and write up a null result.

The abstract:

One of the most influential ideas in the study of political instability is that income shocks provoke conflict. “State prize” theories argue that higher revenues increase incentives to capture the state. “Opportunity cost” theories argue that higher prices decrease individual incentives to revolt. Both mechanisms are central to leading models of state development and collapse. But are they well-founded? We examine the effects of exogenous commodity price shocks on conflict and coups, and find little evidence in favor of either theory. Evidence runs especially against the state as prize. We do find weak evidence that the intensity of fighting falls as prices rise—results more consistent with the idea that revenues augment state capacity, not prize-seeking or opportunity cost. Nevertheless, the evidence for any of these income-conflict mechanisms is weak at best. We argue that errors and publication bias have likely distorted the theoretical and empirical literature on political instability.

In the spirit of transparency, Sami and I have posted replication data. Fire away!