I’m making a webinar presentation in April for The Analysis Factor. I’ve done this several times in the past. The talk in April will be on tests of equivalence and non-inferiority, a topic which I have covered briefly in my newsletter. I thought I’d share a first draft of the abstract here on my blog.

Tests of equivalence and non-inferiority.

Statistics is, to a large extent, a science of comparison. You are trying to find if one group is bigger, faster, or smarter than another. You do this by setting up a null hypothesis that your two groups have equal means or proportions and an alternative hypothesis that one group is “better” than the other. Even if you pretend to be dispassionate, you hope, quietly and secretly, that the data you collect will end up rejecting the null hypothesis. But there are times when you want to show not that one group is better than the other, but that the two groups are equivalent. This is impossible, by conventional wisdom. You can’t prove a negative. Well, you can prove a negative, but it takes more work. In this presentation, you’ll review the framework of the traditional hypothesis test and compare it to the framework of the equivalence test, and its one-sided cousin, the non-inferiority test. You’ll learn how to specify a range of equivalence or a margin of non-inferiority, where those values define differences that are small enough to be tolerable. You’ll also identify critical issues in the design of an equivalence or non-inferiority study that can make or break your experiment.

Here’s a revised version:

Tests of Equivalence and Non-Inferiority with Steve Simon

Statistics is, to a large extent, a science of comparison. You are trying to test whether one group is bigger, faster, or smarter than another.  You do this by setting up a null hypothesis that your two groups have equal means or proportions and an alternative hypothesis that one group is “better” than the other. The test has interesting results only when the data you collect ends up rejecting the null hypothesis. But there are times when the interesting research question you’re asking is not about whether one group is better than the other, but whether the two groups are equivalent.  For example, at which point do second-language speakers make as few grammar errors as native speakers?  Or at which point does a new drug with fewer side effects work just as well as the standard treatment?

Finding equivalence is impossible, by conventional wisdom. You can’t prove a negative. (Well, you can prove a negative, but it takes more work.)  There is an entirely different hypothesis testing framework that allows for this kind of comparison.  In this webinar, we’ll review the framework of the traditional hypothesis test and compare it to the framework of the equivalence test and its one-sided cousin, the non-inferiority test.  You’ll learn how to specify a range of equivalence or margin of non-inferiority, where those values define differences that are small enough to be tolerable.  You’ll also identify critical issues in the design of an equivalence or non-inferiority study that can make or break your experiment.

Here’s what I submitted.

I have to disagree with one comment. “Never attempt to describe results that fail to achieve significance at the a priori threshold for statistical significance, such as suggesting that the results ‘approached significance’ or displayed a ‘trend towards significance.’” There is a belief in some circles that a p-value of 0.04 tells you something that is radically different than a p-value of 0.06. Both represent findings that should be reported, but treated with caution. I would much rather put my faith in a p-value of 0.06 that has a solid mechanistic explanation than one of 0.04 that appears to defy any scientific rationale. Similarly, I would trust a p-value of 0.06 that was associated with other closely related outcome measures that did achieve statistical significance than a p-value of 0.04 that was surrounded by other similar outcome measures that failed to achieve statistical significance.

Context is critical in interpretation of p-values. Unfortunately most scientists do not allow the context of a finding to enter into the discussion of borderline p-values (either the 0.06 p-value or the 0.04 p-value) out of a fear of violating some sacrosanct edict of research conduct.

It seems that every scientist has a p-value receptor in their brain. It stimulates the pleasure center when it encounters a p-value less than 0.05 and it stimulates the pain center when it encounters a p-value greater than 0.05. But no matter what the value, the p-value receptor also shuts down all other areas of the scientist’s brain once it encounters a p-value of any size. And arguing that p-values on the “wrong” side of an arbitrary Type I error rate of 0.05 should be discussed only as a negative result encourages this sort of unthinking approach to p-values.

At a minimum, look at the confidence interval for a “negative” finding. If it includes the null value but also includes values that are considered clinically important, then you should describe the result as being one that warrants further study with a larger sample size.

Other than this one complaint, I think it is a very good article.

Dear Professor Mean, I ran a statistical test in SPSS and got a p-value of .000. I re-ran the same data in Microsoft Excel and got a p-value of 3.9433E-9. I know from scientific notation that this is the same as 0.0000000039433. Why are these numbers different?

Before I say too much I have to point out that many statisticians hate, hate, hate Microsoft Excel. There are good reasons for this. But for your analysis, it might be okay.

The discrepancy that you are seeing is due to rounding. When SPSS reports a p-value that prints as .000, it means that when you round to three decimal places it is closer to .000 than it is to .001. So you know the p-value is smaller than .0005. A p-value of 0.0000000039433 is also smaller than .0005. So the two p-values are consistent.

As a general rule, I round any p-value smaller than .001 up to .001. Some people think that very small p-values should be reported in all their microscopic glory because they make the results seem more impressive. It’s almost like a competition. Ha! My p-value has more zeros in front of it than your p-value. But beyond three decimal places, the p-value is extremely sensitive to even very minor departures from the underlying assumptions.

There are a few exceptions to this in Physics and Genetics, but reporting more precision in your p-value than you really need marks you as an amateur. So be a reasonable person and report the p-value as .001.

Some people might prefer p <.001 and I won’t complain about that. But never, never, never use scientific notation on very small p-values to give your p-value a false sense of precision.

I’m attending a great short course on non-inferiority trials and the speaker provided a key reference of historical interest. This reference is the one that got the Statistics community interested in the concept of non-inferiority. The full text is behind a paywall, but you can look at the abstract. A footnote is a paper, Dunnett and Gent 1977, (also trapped behind a paywall) addressed this problem earlier.

Blackwelder WC. “Proving the null hypothesis” in clinical trials. Control Clin Trials. 1982 Dec;3(4):345-53. Abstract available at http://www.sciencedirect.com/science/article/pii/0197245682900241.

I found an interesting research study that shows what happens inside the brains of  scientists as they view statistical graphs of the type commonly used in peer-reviewed research. I don’t have the citation in front of me, but it was published in a very prominent research journal. Here’s a brief summary of the research.

A research team took ten scientists and placed them inside an fMRI. The fMRI shows which parts of your brain are active as your brain processes different types of information. The scientists were shown a variety of graphs taken from actual peer-reviewed publications.

As you might expect, the part of your brain that activates first when you are presented an image of a graph is your visual cortex. For most graphs, this was quickly followed by an activation of the parietal lobe, the part of your brain responsible for numerical computations.

But some graphs showed a different pattern. If the graph included a p-value, activation of the visual cortex is followed by activation of an area of your amygdala that is as yet poorly understood. The research team called this portion of the amygdala the p-value receptor.

If your p-value receptor is activated and the p-value is larger than 0.05, the p-value receptor sends strong signals to the pain centers of the brain. This is clearly an adaptive behavior. Scientists who routinely produce p-values larger than 0.05 will not survive and reproduce.

If the p-value receptor is activated and the p-value is 0.05 or smaller, the p-value receptor sends strong signals to the pleasure centers of the brain. Again this is an adaptive behavior. But the interesting finding is that there is a dose response effect. The p-value receptor produces about the same level of pleasure stimulation for p-values of 0.05, 0.04, 0.03, and 0.02. But p-values of 0.01 show an increase in stimulation that becomes even strong for p-values of 0.0099 and smaller. Perhaps there is some pattern associated with p-values that have two zeros to the right of the decimal place that is stronger then a p-value with just a single zero.

The scientists also examined the effect of p-values reported in scientific notation. There was an increase in latency when the p-value receptor is fed a p-value in scientific notation. This probably represents an attempt to decode the scientific notation. But p-values in scientific notation with exponents of -4 or smaller showed an eventual spike in activation of the pleasure centers of the brain that are comparable to those achieved during orgasm.

The research also noted a second important effect of the p-value receptor. Once the p-value receptor is stimulated, the entire cereberal cortex, the portion of your brain associated with logic and complex thinking, is immediately shut down. This insures that a scientist’s brain will focus only on the pleasure or pain associated with the p-value and will ignore the power of the study, the magnitude of the treatment effect, and other unimportant issues.

The researchers suggest that statisticians who want to earn more consulting income and insure repeated business should do their best to produce only p-values that stimulate the pleasure centers of the brain.

One of the most confusing aspects of medical research is the difference between non-inferiority and superiority trials. This article explains in simple terms what the two type of trials are. Then it covers the desire of many researchers to switch from a non-inferiority trail to a superiority trial or vice versa. In general, if you would like to make the claim of superiority if the data justifies it, or to fall back on a claim of non-inferiority if you must, you are best off designing a high quality non-inferiority trial. The extra methodological rigor and the typically larger sample sizes that come with a non-inferiority trial make the transition from a non-inferiority hypothesis to a superiority hypothesis much smoother than the reverse. A high quality non-inferiority trial includes pre-specifying the margin of non-inferiority, demonstrating adequate power for the non-inferiority hypothesis, and justifying that the control group has demonstrated efficacy in previous trials. You need to show sufficient methodological rigor in your research design to establish that a non-inferiority finding is not just caused by an insensitive research design. Finally, you need to consider a “per protocol” analysis for the non-inferiority hypothesis, but switch to an “intention to treat” analysis for the superiority hypothesis.

Committee for Proprietary Medicinal Products (CPMP). Points to consider on switching between superiority and non-inferiority. Br J Clin Pharmacol. 2001 Sep; 52(3): 223–228. doi: 10.1046/j.0306-5251.2001.01397-3.x. Available at http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2014556/.

Trafimow D and Marks M (2015) Editorial. Basic and Applied Psychology, 37(1). DOI: 10.1080/01973533.2015.1012991. Available at: http://www.tandfonline.com/doi/full/10.1080/01973533.2015.1012991

Randall Munroe. P-Values. Available at: http://xkcd.com/1478/.