Famous and influential psychology studies that fail replication

While most authors of the original research argue for reasons why the replication failed, they seem to do so in order to defend themselves, and against suspicion of poor research in the first part. (or that which worse is) When reading such defenses, it is easy to forget an important point of replication; namely that even if the original findings are correct; they may be limited to a stricter set of conditions that initially believed.

Erik (HASH) Hersman

To illustrate: two extracts from the article:

“John Bargh the lead author of the 1996 classic criticised the replication attempt and outlined a number of reasons why it might have failed, including that the replication researchers may have included too many ageing-related words thereby making the priming effect too obvious. “

“In her response Kathleen Vohs highlighted that over ten years, 165 studies from 18 countries have documented psychological effects of money reminders, and that perhaps one factor influencing the results is how important money is to the participants in question – she and her colleagues had tested students at the University of Chicago, an institution renowned for its economics scholars, whereas Rohrer had mostly tested people online via Amazon’s Mechanical Turk and The University of California. “

The first argues for a “Goldilocks” amount of priming, the second speculates that the effect may only be valid for a certain type of participant who hold specific set of values.

The reply to the lack of replication of ego-depletion is also worth reading (link in article) where they criticize the replication method, and for not using a sufficiently depleting treatment.

The controversy regarding the “power pose”, where a large scale replication failed to find the effect, was met with criticism from Amy Cuddy and her team, who cite 33 other studies that have found the effect. However, a forthcoming study of these 33 studies show the effect size to be very small. Rather than pick a side, and get defensive, perhaps it would be more constructive to consider:

• Are there confounders? Are there contextual variables that are more important than the actual powerpose?
• Are there moderators / suppressors that could explain what situations / individuals it has an effect?

A final conclusion may then be that while these effects may occur, in given situations; they are a far cry from general principles.

Update: June 2017

The powerpose paper has garnered a lot of attention, as an example for much that is wrong in the world of social science. In this article, James Coyne goes through a lot of the weaknesses to the actual paper, but also how the review processes was not rigorous enough, and how post publication review was stifled. The article is worth reading.

It is also worth nothing that while Amy Cuddy and her adviser and mentor Susan Fiske defend the study tooth and nail, the first author Dana Carney takes a definite stance against the findings, outlined in a letter, which can be found here: http://faculty.haas.berkeley.edu/dana_carney/pdf_My%20position%20on%20power%20poses.pdf

In this letter she admits to what is now coined as P-hacking, and concludes:

Where do I Stand on the Existence of “Power Poses”

1. I do not have any faith in the embodied effects of “power poses.” I do not think the effect is real.

2. I do not study the embodied effects of power poses.

3. I discourage others from studying power poses.

4. I do not teach power poses in my classes anymore.

5. I do not talk about power poses in the media and haven’t for over 5 years (well before skepticism set in)

6. I have on my website and my downloadable CV my skepticism about the effect and links to both the failed replication by Ranehill et al. and to Simmons & Simonsohn’s p-curve paper suggesting no effect. And this document.