This page is moving to a new website.

This is one of the best articles I have ever read in the popular press about the complexities of the research process.

This article by Susan Dominus covers some high profile research by Amy Cuddy. She and two co-authors found that your body language not only influences how others view you, but it influences how you view yourself. Striking a “power pose” meaning something like a “legs astride or feet up on a desk” can improve your sense of power and control and these subjective feelings are matched by physiological changes, Your testosterone goes up and your cortisol goes down. Both of these, apparently, are good things.

The research team publishes these findings in Psychological Science, a prominent journal in this field. The article receives a lot of press coverage. Dr. Cuddy becomes the public face of this research, most notably by garnering an invitation to give a TED talk and does a bang-up job. Her talk becomes the second most viewed TED talk of all time.

But there’s a problem. The results of the Psychological Science publication do not get replicated. One of the other two authors expresses doubt about the original research findings. Another research team reviews the data analysis and labels the work “p-hacking”.

The term “p-hacking” is fairly new, but other terms, like “data dredging” and “fishing expedition” have been around for a lot longer. There’s a quote attributed to the economist Robert Coase that is commonly cited in this context, “If you torture the data long enough, it will confess to anything.” I have described it as “running ten tests and then picking the one with the smallest p-value.” Also relevant is this XKCD cartoon.

If p-hacking is a real thing (and there’s some debate about that), then it is a lot more subtle than the quotes and cartoon mentioned above. You can find serious and detailed explanations at a FiveThirtyEight web article by Christie Aschwanden and this 2015 PLOS article by Megan Head et al.

If p-hacking is a problem, then how do you fix it? It turns out that there is a movement in the research world to critically examine existing research findings and to see if the data truly supports the conclusions that have been made. Are the people leading this movement noble warriors for truth or are they shameless bullies who tear down peer-reviewed research in non-peer-reviewed blogs?

I vote for “noble warriors” but read the article and decide for yourself what you think. It’s a complicated area and every perspective has more than one side to it.

One of the noble warriors/shameless bullies is Andrew Gelman, a popular statistician and social scientist. He comments extensively about the New York Times article on his blog, which is also worth reading as well as many comments that others have made on his blog post. It’s also worth digging up some of his earlier commentary about Dr. Cuddy. Continue reading →