The Quarterly Journal of Political Science has a policy of in-house review of all statistical code. This is expensive and hard to do. Is it worth it?
According to Nick Eubank, an unambiguous yes:
Of the 24 empirical papers subject to in-house replication review since September 2012, only 4 packages required no modifications.
Of the remaining 20 papers, 13 had code that would not execute without errors, 8 failed to include code for results that appeared in the paper, and 7 failed to include installation directions for software dependencies.
13 (54 percent) had results in the paper that differed from those generated by the author’s own code. Some of these issues were relatively small — likely arising from rounding errors during transcription — but in other cases they involved incorrectly signed or mis-labeled regression coefficients, large errors in observation counts, and incorrect summary statistics.
You might think to yourself, “that would never happen in the very top journals”, or “that’s less likely among statisticians and economists. While I think expertise might reduce errors, I’m not so sure. More senior people with more publications tend to rely more on research assistants. And, speaking from personal experience, I’ve found major problems in important people’s code before.
158 Responses
Ouch. A very good idea would be for folks to run their replication programs by a “devil’s advocate” RA or faculty member before submission, who specifically seeks out errors– maybe promise a free dinner for every mistake they find in the final paper? When I worked in econ consulting, we had to turn over replication programs for every number, figure, and table in our expert reports to the opponents’ expert, and they would set their RAs to work night and day looking for errors in our code. If the replication didn’t even run, well I shudder to think what our bosses would say, but even modest errors could be catastrophic for our experts’ credibility. To prevent this, we ‘double-did’ every calculation: a second analyst, working mostly in the dark, would start from the same raw data and replicate the analysis, often in a different stats package or just in Excel, then check every number for equality down to a certain number of digits. We would submit nothing to court that hadn’t been audited thus.
RT @millerpolsci: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/AN94Y…
RT @csabatino: @ahalterman What happens when a very good political science journal checks the statistical code of its submissions? http://t…
RT @csabatino: @ahalterman What happens when a very good political science journal checks the statistical code of its submissions? http://t…
RT @csabatino: @ahalterman What happens when a very good political science journal checks the statistical code of its submissions? http://t…
@ahalterman What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/LWAGBO6w6T
What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/AN94YgM7km
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/CxHYOrhSJs
In-house code review by Quarterly Journal of Political Science. Relatively small (private) cost, big (public) gains. http://t.co/JnW3WCPf2j
What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/LBHPbS7cp9
RT @BrendanNyhan: MT @cblatts QJPS checks the code of its empirical papers. Only 4/24 did not have errors, some quite large http://t.co/H95…
RT @BrendanNyhan: MT @cblatts QJPS checks the code of its empirical papers. Only 4/24 did not have errors, some quite large http://t.co/H95…
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
I absolutely believe this. I actually have a friend who worked on a grasslands study that was published and, years later, realized that her calculations had all been done wrong…luckily, she had the conscience to contact her PI and tell him about the mistake so that he could fix it. But I think you are onto something with the connection between PIs relying on grad students and research assistants for too much…maybe they will be more hesitant to put their names on work that others did if they know it will be more closely checked.
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @tedsvo: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/nIVheZpFIW
What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/2cPHPLLB9J
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
“What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/nIVheZpFIW
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
“Of the 24 empirical papers subject to in-house replication review since September 2012, only 4 packages required… http://t.co/Jtgrx9aFUz
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/YOCyWolqeD
RT @fix_frarent: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/nykT…
This is bad – @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/VmqkcLQXpl
RT @davdittrich: What happens when a very good political science journal checks the statistical code of its submissions?
http://t.co/jsFX3u…
What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/9otzUH6xjR
The Quarterly Journal of Political Science reviews all statistical code in-house. They nearly always catch problems: http://t.co/g5vfBJRS6n
RT @tjvision: What happens when a … journal checks the statistical code of its submissions? http://t.co/UCkfqPW6Uq h/t @LibSkrat
Wouldn’t it be amazing if replication code and data (appropriately anonymized and secured) were made available to reviewers? Reviewers currently rely on some combination of trust and intuition to determine what analyses were actually executed and reported. This creates at least two problems: they suggest bad “fixes” for problems that don’t exist on the one hand and fail to catch major errors on the other!
RT @davdittrich: What happens when a very good political science journal checks the statistical code of its submissions?
http://t.co/jsFX3u…
RT @davdittrich: What happens when a very good political science journal checks the statistical code of its submissions?
http://t.co/jsFX3u…
RT @davdittrich: What happens when a very good political science journal checks the statistical code of its submissions?
http://t.co/jsFX3u…
RT @kcranstn: #rrhack RT @tjvision: a journal checks the statistical code of its submissions http://t.co/rHGVCBfIzZ h/t @LibSkrat
RT @davdittrich: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/jsFX3u…
What happens when a very good political science journal checks the statistical code of its submissions?
http://t.co/jsFX3uIL9s
RT @davdittrich: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/jsFX3u…
RT @drunkeynesian: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/5z…
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @kcranstn: #rrhack RT @tjvision: a journal checks the statistical code of its submissions http://t.co/rHGVCBfIzZ h/t @LibSkrat
RT @kcranstn: #rrhack RT @tjvision: a journal checks the statistical code of its submissions http://t.co/rHGVCBfIzZ h/t @LibSkrat
It absolutely happens among economists. “Replication in Empirical Economics” by Dewald, Thursby, and Anderson showed that (at least in 1986) this was true. They even had one author try to help them to get his regressions to work and they couldn’t do it working together. Data also was found to have been screwed up badly by transcription error, and about 2/3 of people who were “required” to provide the data that they used failed to do so on repeated request.
Its a really eye-opening and disheartening read.
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @ChingunAnderson: What happens when a very good #polisci journal checks the #statistical code of its submissions? http://t.co/P6vWu8gRqe…
RT @ChingunAnderson: What happens when a very good #polisci journal checks the #statistical code of its submissions? http://t.co/P6vWu8gRqe…
RT @ChingunAnderson: What happens when a very good #polisci journal checks the #statistical code of its submissions? http://t.co/P6vWu8gRqe…
RT @ChingunAnderson: What happens when a very good #polisci journal checks the #statistical code of its submissions? http://t.co/P6vWu8gRqe…
RT @ChingunAnderson: What happens when a very good #polisci journal checks the #statistical code of its submissions? http://t.co/P6vWu8gRqe…
RT @davdittrich: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/jsFX3u…
RT @davdittrich: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/jsFX3u…
RT @davdittrich: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/jsFX3u…
What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/jsFX3uIL9s
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @ElizabethPisani: Biomedical papers likely worse, but journals don’t let reviewers see raw data or code. Time for #opendata @cblatts: h…
RT @gwbstr: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/1RZYuLetkW
What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/B3DNFLb1Cb
What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/1RZYuLetkW
need of the hour. journals create an escrow style facility specifically for depositing results and code for papers given provisional acceptance from empirical submission for in-house replication
RT @LibSkrat: Bookmark: What happens when a very good political science journal checks the statistical code of its submissions? -… http://t…
RT @raulpacheco: RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http:…
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/9oBTPqhAPW
Good insight by @cblatts into the need for detailed peer review and replication for quantitative #academic articles http://t.co/uHqDDk9axP
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @MichaelEddy: Top journal checks statistical code of papers & finds 20 of 24 have flaws including incorrectly signed coefficients http:/…
RT @MichaelEddy: Top journal checks statistical code of papers & finds 20 of 24 have flaws including incorrectly signed coefficients http:/…
Top journal checks statistical code of papers & finds 20 of 24 have flaws including incorrectly signed coefficients http://t.co/aPJJSn7TA5
@Timlagor @cblatts Me too.
Bookmark: What happens when a very good political science journal checks the statistical code of its submissions? -… http://t.co/uc1fAZlUBH
Bookmark: What happens when a very good political science journal checks the statistical code of its submissions? -… http://t.co/XLtXjtbneW
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @Simon_the_Pratt: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/5b…
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
RT @sanderwagner: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/plL…
What happens when a very good political science journal checks the statistical code of its submissions?: The Q… http://t.co/o7qakHBHL3
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @drunkeynesian: “What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/5z…
“What happens when a very good political science journal checks the statistical code of its submissions?” http://t.co/5zy5AjGV7K
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
Remember: Everything is bullshit. Everyone is bullshitting.
http://t.co/NRUgjWcSr6
@cblatts @geekyisgood I find the faith in expertise of economists rather naive.
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
What happens when a very good #polisci journal checks the #statistical code of its submissions? http://t.co/P6vWu8gRqe @PolSciReplicate
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
Not surprising… // What happens when a very good political science journal checks the code of its submissions? http://t.co/lF3xBVR2ow
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/YKh20c8CcL
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
Scary stuff MT @cblatts: What happens when a very good poli sci journal checks the stats code of its submissions? http://t.co/XDoswxxTok
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
.@cblatts It blows my mind that all journals don’t do this. RT: QJPS checks the code of its empirical papers. http://t.co/JRkmoWbLaF
“@cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/Qxdqn9kSeP” / wow
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: QJPS checks the code of its empirical papers. Only 4% did not have errors, some quite large. http://t.co/dw5xLl0Jlb
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
“@cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/3g8X419olm”
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
“@cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/bH4c27SLPy”
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6
RT @cblatts: What happens when a very good political science journal checks the statistical code of its submissions? http://t.co/qoGLmxjGB6