Don Green and Allison Sovey have a new reader’s guide to instrumental variables. It is much needed.
For the uninitiated, instruments are God’s gift to causal identification. Like so many miracles, they are not always what their greatest believers want you to believe. If a journal sends me one more referee assignment where the author grossly misuses instruments, blog rampages will ensue.
The authors try to put an optimistic spin on what is still (at least in political science) a sad state of affairs:
…it is clear that the percentage of articles that provide some justi cation for the choice of instruments has risen substantially. Articles falling under the “Experiment,” “Natural Experiment,” “Theory,” “Lag,” and “Reference” categories have all risen overtime. Collectively, the articles in these categories have increased from a low of 14% between 1991-1996 to 56% in the most recent period.
…The percentage of articles reporting fi rst stage results increases from a low of 7% between 1991-1996 to 33% between 2003-2008. In absolute terms, there is still much room for improvement, and only a fraction of those who report fi rst-stage results assess statistically whether instruments are weak or whether overidentifying restrictions are satis fied.
The article offers clear guidelines for judging, reporting and refereeing instruments. Most interesting, however is the take-down of the growing use of rainfall as a valid instrument:
…the reason using rainfall as an instrument is intuitively appealing is that we think of rainfall as patternless. It appears that rainfall growth is systematically related to a range of other observable variables, and therefore we have to assume we have just the right covariates in order to isolate the random component of rainfall.
That is the sound of a dozen top papers crumbling?
6 Responses
What is interesting is that Miguel et al in fact show that future rainfall growth is not correlated with current economic growth, controlling for lagged and current rainfall growth (see their Table 2). In contrast, Green and Sovey do find that future rainfall is correlated with current growth, and one difference seems to be that they don’t control for current and lagged rainfall growth (their Table 3). Here’s a (potentially far-fetched) explanation that would still support Miguel et al’s interpretation of the results (at least if I’m interpreting all these results correctly): lagged rainfall growth causes current economic growth. Economic growth is positively correlated over time (think this is plausible in annual data); rainfall growth is negatively correlated over 2-year periods (not sure this is plausible at all). Hence, you could get a negative correlation between rainfall growth in t+1 and economic growth in t (Green and Sovey), but once you control for rainfall growth in t-1 this correlation disappears (Miguel et al). Like I said, not sure this is at all correct or plausible — just a thought.
Thanks for elaborating.
I don’t think the evidence against rainfall is damning. I think it does raise questions deserving investigation. The burden of proof is always on the instrument. In this case, the correlation with lagged growth is troubling. Other coefficients are sometimes significant, and in many cases quite large even if not always significant. These are not necessarily random, and indeed our null hypothesis with an instrument should not be that any significant correlations are randomly generated. With more and better cross-country data, correlations with rainfall could be better tested. unfortunately the authors don’t do this.
It’s worth noting that when instruments are weak that very small violations of validity can explode in bias. that is the reason for being conservative.
The non-independence concern likewise needs to be addressed. I doubt it would change results much, though standard errors would probably get larger.
I think it’s entirely possible the instrument is valid (note the use of question marks in the post) but the discipline benefits from scrutiny of influential papers. MSS were ahead of their time with this paper, but much has been learned about instruments since.
Yes, can you expand on why that’s such a take-down?
Correction, I meant to say it’s purely random correlation between the IV and other variables, not spurious correlation (which would be a problem for the IV).
Chris, did you look at Table 3 which claims to show “rainfall growth is systematically related to a range of other observable variables”? All I see on the table is a bunch of insignificant coefficients (okay, 1 significant coefficient out of many) and sign changes on some variables across specifications. This is spurious correlation between the IV and other variables — there is no “pattern.”