A comment about the phrase “trend towards efficiency” on the Statistical Consulting Section discussion board raised a lot of interesting commentary. The phrase refers to a setting where the p-value is not small enough to allow you to claim statistical significance, but still was close enough to 0.05 to be worth commenting on. Most of responses were fairly negative and stressed that we need to refuse to sign off on any report of publication using that phrase. I posted a response that differed from the others. Here’s the gist of what I said. Continue reading

# Tag Archives: Hypothesis testing

# Recommended: False-Positive Psychology

This article was recommended at a talk I attended. The full title says it all. Continue reading

# PMean: Tests of equivalence and non-inferiority

I’m making a webinar presentation in April for The Analysis Factor. I’ve done this several times in the past. The talk in April will be on tests of equivalence and non-inferiority, a topic which I have covered briefly in my newsletter. I thought I’d share a first draft of the abstract here on my blog. Continue reading

# PMean: The unthinking approach to borderline p-values

I ran across a nice discussion of how to write the results section of a research paper, but it has one comment about the phrase “trend towards significance” that I had to disagree with. So I wrote a comment that they may or may not end up publishing (note: it did look like the published my comment, but it’s a bit tricky to find).

Here’s what I submitted. Continue reading

# PMean: A p-value of .000

Dear Professor Mean, I ran a statistical test in SPSS and got a p-value of .000. I re-ran the same data in Microsoft Excel and got a p-value of 3.9433E-9. I know from scientific notation that this is the same as 0.0000000039433. Why are these numbers different? Continue reading

# Recommended: Proving the null hypothesis in clinical trials

I’m attending a great short course on non-inferiority trials and the speaker provided a key reference of historical interest. This reference is the one that got the Statistics community interested in the concept of non-inferiority. The full text is behind a paywall, but you can look at the abstract. A footnote is a paper, Dunnett and Gent 1977, (also trapped behind a paywall) addressed this problem earlier. Continue reading

# PMean: Looking inside the brains of scientists

I found an interesting research study that shows what happens inside the brains of scientists as they view statistical graphs of the type commonly used in peer-reviewed research. I don’t have the citation in front of me, but it was published in a very prominent research journal. Here’s a brief summary of the research. Continue reading

# Recommended: Points to consider on switching between superiority and non-inferiority

One of the most confusing aspects of medical research is the difference between non-inferiority and superiority trials. This article explains in simple terms what the two type of trials are. Then it covers the desire of many researchers to switch from a non-inferiority trail to a superiority trial or vice versa. In general, if you would like to make the claim of superiority if the data justifies it, or to fall back on a claim of non-inferiority if you must, you are best off designing a high quality non-inferiority trial. The extra methodological rigor and the typically larger sample sizes that come with a non-inferiority trial make the transition from a non-inferiority hypothesis to a superiority hypothesis much smoother than the reverse. A high quality non-inferiority trial includes pre-specifying the margin of non-inferiority, demonstrating adequate power for the non-inferiority hypothesis, and justifying that the control group has demonstrated efficacy in previous trials. You need to show sufficient methodological rigor in your research design to establish that a non-inferiority finding is not just caused by an insensitive research design. Finally, you need to consider a “per protocol” analysis for the non-inferiority hypothesis, but switch to an “intention to treat” analysis for the superiority hypothesis. Continue reading

# Recommended: Editorial (Basic and Applied Social Psychology)

Recommended does not always mean that I agree with what’s written. In this case, it means that this is something that is important to read because it offers an important perspective. And this editorial offers the perspective that all p-values and all confidence intervals are so fatally flawed that they are banned from all future publications in this journal. The editorial goes further to criticize most Bayesian methods because of the problems with the “Laplacian assumption.” The editorial authors have trouble with some of the ambiguities associated with creating a non-informative prior distribution that is, a prior distribution that represents a “state of ignorance.” They will accept Bayesian analyses on a case by case basis. Throwing out most Bayesian analyses, all p-values, and all confidence intervals makes you wonder what they will accept. They suggest larger than typical sample sizes, strong descriptive statistics (which they fail to define), and effect sizes. They believe that by “banning the NHSTP will have the effect of increasing the quality of submitted manuscripts by liberating authors from the stultified structure of NHSTP thinking thereby eliminating an important obstacle to creative thinking.” It’s worth debating this issue, though I think that these recommendations are far too extreme. Continue reading

# Recommended: P-Values

Randall Munroe, author of the xkcd comic strip, will often comment on Statistics. This cartoon shows how p-values are typically interpreted. Continue reading