….unfortunately, there have been far too many cases where the quantitative analysis of those numbers has been flawed, causing doubt about the authors’ interpretation and uncertainty about the result. Furthermore, it is not realistic to expect that a technical reviewer, chosen for her or his expertise in the topical subject matter or experimental protocol, will also be an expert in data analysis.
For that reason, with much help from the American Statistical Association, Science has established, effective 1 July 2014, a Statistical Board of Reviewing Editors (SBoRE), consisting of experts in various aspects of statistics and data analysis, to provide better oversight of the interpretation of observational data.
…I have been amazed at how many scientists have never considered that their data might be presented with bias. There are fundamental truths that may be missed when bias is unintentionally overlooked, or worse yet, when data are “massaged.” Especially as we enter an era of “big data,” we should raise the bar ever higher in scrutinizing the analyses that take us from observations to understanding.
This is an important move. I would love to see medical journals do the same, where I think the problems are greater and the consequences for human welfare more immediate.
At the same time, if your research mainly deals with numbers, then I think it is time to expect people with substantive expertise to become better statisticians. They need this not only to produce better work, but to be effective users of what their peers produce. This cannot simply be exported to a committee in the top journal.
Raising the refereeing bar is going to get the incentives right, which is a step in the right direction. But something will need to change in graduate admissions requirements and training.
In particular, I think that a 21st century undergraduate degree in social science ought to require fluency in statistics. It’s such a fundamental part of science, medicine, social science, and even reading the newspaper. But even the Columbia’s and Yale’s of the world don’t impress this on their undergraduates, let alone require it. I’m a big supporter of the liberal arts education, but on the margin I’d substitute a couple courses in the humanities for statistics and causal inference.