Isn’t it ironic?
A recent article by the Open Science Collaboration (a group of 270 coauthors) gained considerable academic and public attention due to its sensational conclusion that the replicability of psychological science is surprisingly low. Science magazine lauded this article as one of the top 10 scientific breakthroughs of the year across all fields of science, reports of which appeared on the front pages of newspapers worldwide.
We show that OSC’s article contains three major statistical errors and, when corrected, provides no evidence of a replication crisis.
Indeed, the evidence is consistent with the opposite conclusion — that the reproducibility of psychological science is quite high and, in fact, statistically indistinguishable from 100%. The moral of the story is that meta-science must follow the rules of science.
A new article by Gilbert, King, Pettigrew, and Wilson.
This excerpt is rather amazing:
For example, many of OSC’s replication studies drew their samples from different populations than the original studies did. An original study that measured American’s attitudes toward African Americans was replicated with Italians, who do not share the same stereotypes;
an original study that asked college students to imagine being called on by a professor was replicated with participants who had never been to college;
and an original study that asked students who commute to school to choose between apartments that were short and long drives from campus was replicated with students who do not commute to school.
What’s more, many of OSC’s replication studies used procedures that differed from the original study’s procedures in substantial ways: An original study that asked Israelis to imagine the consequences of military service was replicated by asking Americans to imagine the consequences of a honeymoon;
an original study that gave younger children the difficult task of locating targets on a large screen was replicated by giving older children the easier task of locating targets on a small screen;
an original study that showed how a change in the wording of a charitable appeal sent by mail to Koreans could boost response rates was replicated by sending 771,408 e-mail messages to people all over the world (which produced a response rate of essentially zero in all conditions).
While the fact that this was detected and reported on quickly is (in some sense) evidence of science working, it seems that something is broken about the review and referee process that the original paper got through the publication process.
Update: The original group of authors don’t like the replication of their replication. See Andrew Gelman’s excellent and skeptical discussion. I have read none of these closely enough to have an opinion.
150 Responses
my wife needed TSP-1 this month and learned about a great service that has lots of form templates . If people are looking for TSP-1 also , here’s a
http://goo.gl/RK7X3K
The original authors suggest these characterizations of the replications were misleading
http://retractionwatch.com/2016/03/07/lets-not-mischaracterize-replication-studies-authors/
Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/tDOZNR0bBW
Science is hard https://t.co/fl7eYMCE5D
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/UhODixgFNz
RT @busbyj2: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/SVEF6dBNGB
RT @busbyj2: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/SVEF6dBNGB
Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/SVEF6dBNGB
RT @VictoriaMonro: “Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate.”
https://t.co/1bl3…
“Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate.”
https://t.co/1bl3tz0gep
“Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate.” https://t.co/fMhZ6VaOPD
C’mon, the title of the post is misleading! It’s not a new article, it’s a comment on the original replication paper. And the comment does not try to replicate the study, it simply re-analyzes the results. BTW, using a single, inadequate, metric for replication (with erroneous interpretations of a CI and its properties) and comparing different studies as if they belong to the same reference class.
They do bring some important issues to attention, like the validity of some replications that seems to drift away from the originals. But their conclusion – high reproducibility – is overreaching.
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate.: Isn’t it … https://t.co/Q2pxs0vita
?! “…has re-analyzed the paper about re-analyzing papers, and they say that it’s wrong.” via https://t.co/eBbfYm3nnx https://t.co/hXfWuUUSx8
No Evidence for a Replicability Crisis in Psychological Science https://t.co/iEaoYCL73f https://t.co/5fd9ghZKZ6
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
So meta https://t.co/gEXqcndjUB
RU4Real
https://t.co/QRtRkb22kY
@erikphoel
I have some reading to do methinks
This week’s Science Mag also has this new article on replicability in econ lab experiments:
http://science.sciencemag.org/content/early/2016/03/02/science.aaf0918
“We find a significant effect in the same direction as the original study for 11 replications (61%); on average the replicated effect size is 66% of the original. The reproducibility rate varies between 67% and 78% for four additional reproducibility indicators, including a prediction market measure of peer beliefs.”
Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/0x5vr7hRAT
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @chloewitty: This is some quality bit of psychological research https://t.co/rBUxw3dvfR
RT @chloewitty: This is some quality bit of psychological research https://t.co/rBUxw3dvfR
This is some quality bit of psychological research https://t.co/rBUxw3dvfR
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
Looool || Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/icXvREPqxW
RT @freakonometrics: thanks @cblatts for pointing out that the study showing most psychology studies don’t replicate doesn’t replicate http…
This ! “Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate” — https://t.co/nJ3goMQ3kU
Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/NT3SXfaJT8
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @StevenVdWalle: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/C1IXVKz…
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. – Chris Blattman https://t.co/ma6InS3zSV
RT @po_rbl: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/tGvGuxjv3q @ro…
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @StevenVdWalle: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/C1IXVKz…
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @po_rbl: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/tGvGuxjv3q @ro…
Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. – Chris Blattman https://t.co/NfjXlvaP3V
RT @StevenVdWalle: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/C1IXVKz…
Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/C1IXVKzitm
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
replication crisis is indistinguishable from 100% replication: “meta-science must follow the rules of science” @peez https://t.co/59n2tDatZg
#Psychology https://t.co/WlQKRplOz7 Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. – …
Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. – Chris Blattm… https://t.co/CF7AF4wu7y
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @freakonometrics: thanks @cblatts for pointing out that the study showing most psychology studies don’t replicate doesn’t replicate http…
RT @freakonometrics: thanks @cblatts for pointing out that the study showing most psychology studies don’t replicate doesn’t replicate http…
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
“Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate.” https://t.co/sSpGvieT7A
I would say design diff within a certain degree verifies our inference of concepts from operationalized experiments. psychologists deal with concepts not procedures.
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
Ah, a replication crisis replication crisis.
https://t.co/fMP3Q3taqI
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
@cblatts @Noahpinion Couldn’t some of the new applications of a finding, in a sense, be considered some form of “replication”, e.g. Italians
@cblatts @adamgurri That’s not even close to an accurate statement.
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
@cblatts Repliception.
.@cblatts Hoooooooly crap, that is some sloppy “replication”.
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
On the other hand, if the effect is so narrow as to only work in very specific populations under narrow conditions, is it worth reporting on to begin with?
Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/YDCXRpO3ti
RT @Laurenz_ml: Very true: ‘The moral of the story is that meta-science must follow the rules of science.’ https://t.co/iBFwqPEQgy
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
@cblatts @moebius_strip Is it like the study that showed 85% of statistics are made up?
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
Very true: ‘The moral of the story is that meta-science must follow the rules of science.’ https://t.co/iBFwqPEQgy
Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/iBFwqPEQgy
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0
RT @cblatts: Remember the study showing most psychology studies don’t replicate? Turns out it doesn’t replicate. https://t.co/a6tTKU36t0