I have had my first graduate class in Methodology, and have thus been primed to focus on potential flaws in research design. No wonder then, that when an article called: “Making it all up – The behavioral science Scandal” appeared in my facebook timline, I clicked, read and nodded my agreement to most of the claims. http://www.weeklystandard.com/articles/making-it-all_1042807.html?nopager=1
Much of the focus is on experiments in behavioral psychology, and some of the central points are:
THOS FLUHARTY
- The samples are often small and college students who are induced, either by money, credit or a sense of obligation, to participate. These are not necessarily representative of any population but college students.
- Researchers are no better, as most teams and even fields, are made up of very homogeneous people.
- Alternative explanations abound
- The conclusion is often assumed: a difference is found, but the conclusion of WHY there was a difference is merely assumed.
- There is little direct replication, (as in duplication), most are extensions, that have a strong vested interest in the original study being correct.
- Of the duplication performed by ” Reproducibility Project”, 2 out of 3 studies did not show the purported effects
- One study examined a landmark study showing that college students who are primed to think of old people, walk slower. They found that the priming effect was present, but only in that when researchers knew what they were looking for, they found it. No difference was found in the students.
- There are measurement flaws, such as stop-watches used by researchers who want to find a result, vs. infrared timing, which has no vested interest.
- Negative results are rarely publishable.
There are many great examples in the article, so I recommend reading it! The takeaway is that a robust study design can make it harder to get started, and reduce the probability of finding real and interesting findings, but one thing going for it, is that the findings will be relevant and more likely true; saving the researcher from potential disgrace further down the road.
All to say, the article casts strong doubt on the effect of priming, so I am left wondering: was I indeed primed, or do I just want to think I was.