The publication bias problem (and the redemption of Blattman?)

After many years in the file drawer, I’m putting out a paper as penance.

Eight years ago, finishing a master’s program, I wrote a paper that made a simple argument:

  • Most countries, from Sweden to India, have produced and exported the same handful of commodities for more than a century–the commodities they were essentially “born” with.
  • Some commodities are more price volatile than others, and for most countries this price volatility is simply given to them.
  • The countries with the most price volatile commodities have grown the most slowly over history. Volatility is a huge predictor of growth.

For my PhD I decided I would figure out why. My hypothesis: it’s all about political instability. States come to depend on easily taxed commodities for revenues–fuels, minerals and tree crops. When prices plunge, so does the ability to buy off or repress opponents.

I dug around in the fifth basement of Harvard’s and Berkeley’s libraries for more price and export data, and I carefully built a new database. My dissertation committee was enthusiastic.

At long last, regressions were run and… no result. No relationship between price shocks and conflict, even in the most generous scenarios. I shrugged and thought, “Well, so much for that.” My committee said, “Huh, what about that child soldiering project we told you not to do?” And off I went on my career as micro-conflict man.

In the meantime, lots of papers that did see an impact of economic shocks on conflict or instability did get published. The conventional wisdom grew: Rising incomes made the state more attractive to rebels as a prize, and falling incomes made it easier to recruit rebels. No matter that these two ideas ran in apparently opposite directions.

Meanwhile, I met other academics that had run the same regression as me. Famous ones you have probably heard of. Their reaction was the same as mine: “Oh, I found that result,” several said, “but I’m worried there’s nothing there because my data have problems, and the specification wasn’t quite right. So I left it out of the paper. I’ve been meaning to get back to that.”

Let’s follow a simple decision rule: run your regressions with inevitably imperfect data and models. If you get the theoretically predicted result (any of them), publish. If not, wait and look into your data and empirical strategy more.

The result? As in the natural sciences, most published research findings are probably false.

After several guilty years, I brushed off my data with co-author Sami Bazzi, and set about the hundred small tasks and tests needed to believe something might not be true.

We finally have a working paper: Economic Shocks and Conflict: The (Absence of?) Evidence from Commodity Prices.

I will only attest: It is very, very difficult to argue and write up a null result.

The abstract:

One of the most influential ideas in the study of political instability is that income shocks provoke conflict. “State prize” theories argue that higher revenues increase incentives to capture the state. “Opportunity cost” theories argue that higher prices decrease individual incentives to revolt. Both mechanisms are central to leading models of state development and collapse. But are they well-founded? We examine the effects of exogenous commodity price shocks on conflict and coups, and find little evidence in favor of either theory. Evidence runs especially against the state as prize. We do find weak evidence that the intensity of fighting falls as prices rise—results more consistent with the idea that revenues augment state capacity, not prize-seeking or opportunity cost. Nevertheless, the evidence for any of these income-conflict mechanisms is weak at best. We argue that errors and publication bias have likely distorted the theoretical and empirical literature on political instability.

In the spirit of transparency, Sami and I have posted replication data. Fire away!

9 thoughts on “The publication bias problem (and the redemption of Blattman?)

  1. Bravo! Very exciting that you are willing to speak up about these null results. So important for the social aspect of social sciences– since so many people in power base policy on what gets published. Way to bring science back to the social sciences, and kick out some of the politricks!

  2. Matt Lange and Hrag Balian have a paper on the null effect of state strength on conflict. Their explanation of the null finding is that the relationship is marked by effects pointing in each direction, which cancel out in the aggregate. The empirics are ugly, but the idea is nice, and marks another all-too-rare example of a null finding actually published. The paper is in Studies in Comparative International Development from 2009 or so.

  3. States have two other ways to get money in their coffers: International aid and international debt. I am not sure to which extent you can actually control these two variables, or to which extent you have already done so, but perhaps that could impact results (as a curiosity, Norway saves large part of its petroleum income. As a result, as petroleum prices plunged in 2009, Norway’s public expenditures increased. The exact opposite from what you would expect if the country had weak institutions.)

    Furthermore, I believe most commentators would support the idea that changes in government social programmes, as a relatively direct result of the current reduced supply of international debt, do lead to social unrest in a range of countries – from OWS to Athen to Tahrir. And, as another case study, it may appear as the stability of Malawi currently rests on the government’s ability to provide the usual fertilizer subsidies, which is challenging due to recent donor cutbacks.

  4. One more thing that just crossed my mind: As Dani Rodrik reminds us in his latest book: In advanced economies, government, as a share of GDP, grows with increased exposure to trade (and, remember, the price shocks you are talking about mainly concerns tradables). This could indicate that the link between export price shocks and instability is real, but that governments around the world have found ways to reduce or even eliminate it. Hence, we should expect the data to give no certain answer. (states may reduce the effect of the shock by borrowing, by receiving international aid, by having strong social safety nets, or simply by saving in good times and spend in bad times. And probably a whole range of other factors that I can not think about. And as I know actually read you paper, I probably realize that you have already thought of all this :D )