This is a non-technical discussion of the difference between effectiveness and efficacy (two easily confused terms) in the context of vaccination. Short answer: efficacy is a measurement under ideal circumstances while effectiveness is a measurement in a “real-world” setting.

Christopher Harrison. Is vaccine effectiveness (VE) different from vaccine efficacy? Frontline Medical News, February 27, 2018. Available at https://www.pm360online.com/is-vaccine-effectiveness-ve-different-from-vaccine-efficacy/.

I’ve done a lot of work with Evidence-Based Health, but one big and largely unsolved problem is how to get health care professionals to change their practices once the evidence for these changes becomes obvious. If no one changes in the face of evidence, then all the effort to produce and critically appraise the evidence becomes worthless. A new field, implementation science, has been developed to get at methods to encourage the adoption of new evidence-based practices. This paper outlines how implementation science is supposed to work and offers two real world examples of implementation science studies.

Bauer, Mark S., Damschroder, Laura, Hagedorn, Hildi, Smith, Jeffrey, Kilbourne, Amy M. An introduction to implementation science for the non-specialist, BMC Psychology (2015, Sep 16),
3(1): 32. Available at https://doi.org/10.1186/s40359-015-0089-9.

I generally shy away from Philosophical debates, but I did discuss a Postmodern critique of Evidence Based Medicine a while back. When one of my more intellectual friends posted a link to a commentary on Postmodernism on the Existential Comics web site, I had to take a look. I think I did a pretty good job of summarizing Postmodernism without stereotyping it, but maybe I’m setting my standards too low if I try to compete with a comic strip. You can judge for yourself.

Philosophy News Network: Postmoderism Special Report. Available at http://existentialcomics.com/comic/218.

This is one of the best articles I have ever read in the popular press about the complexities of the research process.

This article by Susan Dominus covers some high profile research by Amy Cuddy. She and two co-authors found that your body language not only influences how others view you, but it influences how you view yourself. Striking a “power pose” meaning something like a “legs astride or feet up on a desk” can improve your sense of power and control and these subjective feelings are matched by physiological changes, Your testosterone goes up and your cortisol goes down. Both of these, apparently, are good things.

The research team publishes these findings in Psychological Science, a prominent journal in this field. The article receives a lot of press coverage. Dr. Cuddy becomes the public face of this research, most notably by garnering an invitation to give a TED talk and does a bang-up job. Her talk becomes the second most viewed TED talk of all time.

But there’s a problem. The results of the Psychological Science publication do not get replicated. One of the other two authors expresses doubt about the original research findings. Another research team reviews the data analysis and labels the work “p-hacking”.

The term “p-hacking” is fairly new, but other terms, like “data dredging” and “fishing expedition” have been around for a lot longer. There’s a quote attributed to the economist Robert Coase that is commonly cited in this context, “If you torture the data long enough, it will confess to anything.” I have described it as “running ten tests and then picking the one with the smallest p-value.” Also relevant is this XKCD cartoon.

If p-hacking is a real thing (and there’s some debate about that), then it is a lot more subtle than the quotes and cartoon mentioned above. You can find serious and detailed explanations at a FiveThirtyEight web article by Christie Aschwanden and this 2015 PLOS article by Megan Head et al.

If p-hacking is a problem, then how do you fix it? It turns out that there is a movement in the research world to critically examine existing research findings and to see if the data truly supports the conclusions that have been made. Are the people leading this movement noble warriors for truth or are they shameless bullies who tear down peer-reviewed research in non-peer-reviewed blogs?

I vote for “noble warriors” but read the article and decide for yourself what you think. It’s a complicated area and every perspective has more than one side to it.

One of the noble warriors/shameless bullies is Andrew Gelman, a popular statistician and social scientist. He comments extensively about the New York Times article on his blog, which is also worth reading as well as many comments that others have made on his blog post. It’s also worth digging up some of his earlier commentary about Dr. Cuddy.

Susan Dominus. When the Revolution Came for Amy Cuddy. The New York Times Magazine (October 18, 2017). Available at https://www.nytimes.com/2017/10/18/magazine/when-the-revolution-came-for-amy-cuddy.html.

I’m teaching a class (or running a journal club or giving a seminar) on research design (or evidence based medicine or statistics) and I’d like to find an example of a research study that use bad statistical analysis.

And there’s always a flood of responses back. But if I were less busy, I’d jump into the conversation and say “Stop! Don’t do it!” Here’s why.

Sharing bad examples have several problems. They encourage black-and-white thinking versus a more nuanced interpretation. They also can breed fear and/or cynicism in your students. The alternative to bad examples are to pick examples based on other criteria than how bad or good the statistical analysis is, or to pair bad examples with good examples to show the right way to do research.

Black-and-white thinking. An important critical thinking skill is deciding when a problem is so serious that you should totally disregard that study.  Think of this as the fatal flaw approach to critical thinking. It’s easy to find fatally flawed studies. The problem is that the number and frequency of fatal flaws is far outnumbered by non-fatal flaws.

For example, lack of blinding is a frequent limitation of research, but it rarely is a fatal flaw. If it were, you’d have to disregard almost all of the published research on surgery interventions. I do worry about unblinded studies, but I worry about lots of things. It is a combination of weaknesses that make a study unpersuasive, and I normally don’t let one particular type of weakness dominate my thinking.

Most flaws are not fatal, but rather they decrease the persuasiveness of a research study, unless counterbalanced by other strengths.

Many researchers are already pre-disposed to think in terms of black-and-white. Your job is to increase their sense of nuance and get them to think about shades of gray. If you give them a clearly flawed study, they’ll lose their sense of nuance.

Fear. You and I both know that Statistics is not a difficult thing to learn if you take the time to learn it well. There are subtleties and you need an eye for detail. But the sort of people that are in your journal club have already mastered far more difficult skills, such as inserting a feeding tube that goes down into the stomach rather than into the lungs.

Put yourself in your student’s shoes. You’re going to show them a study that has a least on the face of it looks normal. Meaning that it appears in a reasonably prominent journal and with authors with reasonable credentials. Then you’re going to reveal something to them that they probably didn’t know before. After all, you’re the teacher and they’re the students. And that revelation will demolish the study’s conclusions. This might spook them. Even if they understand the fatal flaw after you explain it (not a slum dunk by any means), they will start worrying about what other previously unrecognized fatal flaws are out there that they are unaware of.

They’ll probably heed your message that you should always consult with a statistician before running a study. But they’ll take it a bit farther and be fearful of making a critical appraisal of a study without checking with a statistician first. They’ll be afraid about spending a lot of time outlining the “trivial” features of a study, but because they missed noticing the one fatal flaw, all of that effort was wasted.

Cynicism. The flip side of fear is cynicism. It may be easy to lie with Statistics, so the saying goes, but it is even easier to lie without them. The problem is that your students are likely to remember only the first half of this statement. When you show fatally flawed studies, they’ll probably start thinking that Statistics is a powerful weapon. If it can destroy this study, maybe it can destroy any study. Every study has a hidden fatal flaw, that once revealed by an experienced Statistician will devastate the conclusions. This is the path to a cynical disregard of all published research.

Alternatives. So what would I recommend instead of bad examples? I argue that you should choose examples, not based on how bad (or good) they are, but rather on how interesting they are to your students. If you are a Statistician, you may need some help here with someone who is familiar with the medical expertise and background of your students.

Then tally the strengths and weaknesses of the study. If your students are able to recognize half or more of the ones that you notice, they are doing very well. Even if the weaknesses strongly outweigh the strengths, avoid excessively negative comments. The limitations of a study mean that you should hold off on acting on the recommendations made by the authors until they can provide more persuasive arguments through a better research study. Then talk about what the better research study would be. What additional steps could the authors take to add more strengths or remove some of the existing weaknesses.

Now you probably won’t heed my advice. It’s too much fun to trash other people’s research. But here’s an alternative. Find a study that is fatally flawed and pair it with a study of the same topic that is fairly persuasive. These studies could both make the same claim or they could make contradictory claims. It doesn’t matter. Contrast what one researcher team did well to what the other team did not (or could not).

If you don’t take the time to pair your bad example with a good example, don’t be surprised when your students revert to black-and-white thinking, or if they become cynical or fearful.

selection bias through randomization (sequence generation and allocation concealment); confounding through design or analysis; performance bias through fidelity to the protocol, avoidance of unintended interventions, patient or caregiver blinding and clinician or provider blinding; detection bias through outcome assessor and data analyst blinding and appropriate statistical methods; detection/performance bias through double blinding; attrition bias through intention-to-treat analysis or other approaches to accounting for dropouts; and reporting bias through complete reporting of all prespecified outcomes.

The general finding was that failure to use these bias reduction approaches tended to exaggerate treatment effects, but the magnitude and precision of these exaggerated effects was inconsistent.

Nancy D Berkman, P Lina Santaguida, Meera Viswanathan, and Sally C Morton. The Empirical Evidence of Bias in Trials Measuring Treatment Differences. Available at http://www.ncbi.nlm.nih.gov/books/NBK253181/.

This page has moved to a new website.

Here’s a research article tackling the same problem of changing outcome measures after the data is collected. Apparently, this occurs in 66 of the 226 papers reviewed here or almost 30% of the time. The interesting thing is that whether this occurred or not was independent of whether paper was accepted. So journal editors are missing an opportunity here to improve the quality of the published literature by demanding that researchers abide by the choices that they made during trial registration.

van Lent M, IntHout J, Out HJ. Differences between information in registries and articles did not influence publication acceptance. Journal of clinical Epidemiology. 68(9): 2015 (September); 1059-1067.

One of the many problems with medical publications is that researchers will choose which outcomes to report based on their statistical significance rather than their clinical importance. This can seriously bias your results. You can easily avoid this potential bias by specifying your primary and secondary outcome measures prior to data collection. Apparently, though, some researchers will change their minds after designating these outcome variables and fail to report on some of the outcomes and/or add new outcomes that were not specified prior to data analysis. How often does this occur? A group of scientists at the Centre for Evidence-Based Medicine at the University of Oxford are trying to find out.

Goldacre B, Dale A, Milosevic I, Mahtani K, Powell-Smith A, Drysdale H, Hartley P, slade E, Heneghan C. The COMPare Project. Available at http://compare-trials.org/.