Rutterforda C, Taljaard M, Dixon S, Copas A, Eldridge S. Reporting and methodological quality of sample size calculations in cluster randomized trials could be improved: a review. Journal of Clinical Epidemiology 68 (2015) 716e723. Available at http://www.ncbi.nlm.nih.gov/pubmed/25523375.

This page has moved to a new website.

Austin PC , Steyerberg EW. The number of subjects per variable required in linear regression analyses. Journal of Clinical Epidemiology 2015, 68(6):627-636. Available at http://europepmc.org/abstract/med/25704724.

First set up a column in SPSS that represents a counter for each row. It should go from 1 to 10,000, but for the sake of simplicity, I am only showing the first ten rows. Now, you don’t have to type the numbers 1 to 10,000. Open an Excel spreadsheet, put 1 in the top leftmost cell, 2 in the cell beneath it, highlight both cells, and then extend down to 10,000. Import this file into SPSS.

Now, we want to compute the power of a test of association in a two by two table where the treatment group has 200 patients and a probability of success of 60% and the control group has 100 patients and a probability of success of 50%. We want a one-sided test at an alpha level of 0.05. First simulate the number of successes in the treatment group as binomial with n=200 and p=0.6.

Next simulate the number of success in the control group as binomial with n=100 and p=0.5.

Calculate the difference in proportions

and the standard error.

Then calculate for each row whether you exceeded the 0.95 percentile of a standard normal distribution.

Here’s what the first few rows of your simulation look like.

Now tally your results.

Hey look at that. You rejected the null hypothesis exactly 50% of the time. That means that your power is 50%. Re run this simulation with twice as many subjects in each group. What do you get?

The approach that I most often take with power calculation for a test of association for a two by two table is to treat it as a test of two independent proportions. Here’s an example. You are collecting data on the proportion of women who are still successfully breast feed their newborn child at six months of age. The child is six months old, not the mother. The mothers are randomly assigned to either a treatment group that receives special counseling or a control group. Both groups of mothers are encouraged to breast feed, but the treatment group has a more intensive intervention. The latest statistics seem to indicate that the probability of breast feeding at six months for the typical mother in the United States is around 50%. The number is much lower if you look at exclusive breastfeeding at six months.

The intervention is expensive and would only be worth adopting if the rate of breast feeding increased from 50% to 60%. You want to sample a total of 300 mothers, with twice as many assigned to the treatment group than the control group. Normally this is a bit inefficient, but there are other objectives of the research that require a larger number of women in the treatment group.

Here’s what the two by two table would look like under the ideal setting of no sampling error

.

You want to test the hypothesis that the proportion of mothers breast feeding at six months is equal in the treatment and control groups. You plan to test this hypothesis using a two-sided test with an alpha level of 0.05.

Under the null hypothesis, the test statistic,

is approximately a normal distribution with mean zero and standard deviation 1 where

is the estimate of the proportion pooled across both groups. If the alternative hypothesis is true, then test statistic is approximately normal, but the mean is

and the standard deviation is

Notice that the hats disappear in these formulas because we are talking about population parameters and not statistics.

For a one sided test, the power would be

where Z* is not a standard normal, but rather the normal distribution described above. to calculate this probability, you need to standardize by subtracting the mean and dividing by the standard deviation.

Eeeek! It does simplify a bit. The left hand side is a now a standard normal. And you can cancel a few things on the right hand side.

It’s still a bit messy but manageable. If you wanted to get a sample size that provides you with a specific amount of power, you set that messy thing on the right side of the inequality equal to the 1-beta percentile of a standard normal distribution and solve for the sample size. There are actually two sample sizes here, so you need to assume that the sample sizes of the two groups are equal or perhaps that they vary in a certain ratio (2 to 1 in the example above). I won’t show all the details, but like everything else here,  it is more tedious than difficult.

Dupont WD, Plummer WD. PS: Power and Sample Size Calculation. Available at http://biostat.mc.vanderbilt.edu/wiki/Main/PowerSampleSize.

There are many ways to validate. I’m guessing here, but I suspect that you want to compare your algorithm, which is simple, cheap, or fast, to a gold standard measure of diabetes. The gold standard is something that has been around for a while and is well trusted by doctors, but it may be a lot more expensive or time consuming that what you are proposing.

Establishing validity in this framework is typically done by establishing that your sensitivity and specificity are large enough. You want to select a sample size so that the confidence intervals for sensitivity and specificity are reasonably narrow. A key statistic here is the proportion of patients in your sample that will have diabetes according to your gold standard.

Psychologists use terms like “criterion validity” or “predictive validity” in this case, though I am always a bit unclear on their terminology. That’s probably more of a limitation on my intellectual capacity than a criticism of their definitions.

Note that there is no “power” involved in this calculation. The reason for this is that validity is not something that is easily reduced to a simple hypothesis test.

If you want more details, I talk about sample sizes needed for a study of a diagnostic test at http://www.pmean.com/04/SampleSizeDiagnostic.html

Benjamin DK Wood and Eric Djimeu, Requiring fuel gauges: A pitch for justifying impact evaluation sample size assumptions. Evidence Matters blog. October 17, 2014. Available at http://blogs.3ieimpact.org/requiring-fuel-gauges-a-pitch-for-justifying-impact-evaluation-sample-size-assumptions/

Sample size of 12 per group rule of thumb for a pilot study. Steven A. Julious. Pharmaceut. Statist. 2005; 4: 287–291. Abstract available at http://onlinelibrary.wiley.com/doi/10.1002/ps

I was not told how many patients they expected to get in the treatment group, but there are three possible choices that you could investigate as a starting point for further discussion, a treatment group of equal size to the control group, a treatment group twice as large as the control group, and a treatment group half the size of the control group. The last choice is a hard sell to an IRB that has been a bit skeptical in the past of some of my other sample size justifications, so let’s concentrate on the first two choices.

The outcome measure has a standard deviation of 11 based on previously published research. The researcher did not specify a minimum clinically important difference, but in a previous project has suggested a value of 15. I think we can do a bit better than that for this study.

I want to show screen shots of a program called Piface, that is available for free at Dr. Russ Lenth’s website. Here’s the splash screen.

Dr. Lenth’s son, Brian,  did the artwork here. After you click OK, you get the following window.

Since the outcome is continuous and since there is no pairing or matching between the control group and the treatment group, an independent (two sample) t-test is worth looking at. If we want to consider something later on that involves matching or risk adjustment, we would have to look at a more complex model. Select the “Two sample t test” under “Type of analysis”,

and then click on the “Run dialog” button. This is what you get.

The default option in Piface is to use sliders, which are very nice for “what if” calculations. But you can switch from sliders to direct data entry by clicking on the small gray box above and to the right of each slider.

Let’s start with both standard deviations (sigma1 and sigma2) being equal to 11. There is no strong belief that the treatment data will be substantially more variable or substantially less variable than the control data. Also set both sample sizes (n1 and n2) for now to 30. I’m not sure what will work here, but let’s try 15 for the minimum clinically important difference (true difference in means). Once you enter the 15 and click on OK, you get the following window.

Whoa! That’s 99.94% power. Generally, anything this large means that we have a bit of “wiggle room” and can afford to either decrease the overall sample size or decrease the minimum clinically important difference. At this point, I would opt for the latter, but this is something you should discuss with the client. So what if we wanted to be able detect differences of size 10 rather than 15. What would this do to our power? Replace the 15 in the true difference of mean entry field and you get this.

We have 93% power. That’s still very good. You might ask yourself, what difference in means could we detect with 80% power. There’s a general comfort level among many researchers with a power of 80%, though I’ve seen some hints that 90% power may make more sense in some contexts. Nobody really thinks this through very carefully, and if you wanted to do this well, you’d estimate the cost of a false positive research finding, the cost of a false negative research finding, and try to balance your beta error (1-power) and your alpha error. No one expects this level of detail, though, unless you are talking about a really expensive clinical trial.

To get the difference in means that you could detect with 80% power, change the “Solve for” field to “Diff of means” and put 0.80 in the “Power” field.

Now, let’s see what we could get at 90% power.

Now, at this point, we could look at 30 in the control group and 60 in the treatment group, but I suspect that this would be overkill. We already can detect what I might consider pretty small changes in the outcome variable. This is where you need to have more discussions with your client. I would recommend that we write up something like the following:

With a sample of 30 historical controls and 30 patients in the [treatment] group, we would have 93% power for detecting a 10 unit shift in [the outcome variable]. This is based on a two sample t-test with a common standard deviation of 11 (based on [cite your reference here]) and a two-sided alpha level of 0.05.

I try to write these statements with enough detail that another statistician could look at it and reproduce my calculations. In this example, it means that you mention

1. the statistical test being used
2. the standard deviations
3. the sample sizes
4. the minimum clinically important difference
5. the alpha level
6. whether the test is one sided or two sided, and
7. the power

Why didn’t I use the 90% power or 80% power scenarios? It’s just my personal preference. I would rather have the minimum clinically important difference be a nice round number than have the power be a nice round number. But there is nothing wrong with presenting any of these scenarios. Just substitute in the appropriate values.

Now further discussions would be warranted here. I’m guessing that the researchers would be very happy with a research design that could detect a difference as small as 10 units, mostly because I have worked with them on similar projects before. I’m also assuming that getting 30 treatment patients is easy (the number of control patients is fixed at 30 because it is a historical control group). These are pretty big assumptions. I don’t have problems with making assumptions like this because it offers them something to react to, but I would never publish a sample size justification like this without first confirming these assumptions.

Further work is needed here. Perhaps they were thinking of 20 patients in the experimental group, and this might be reasonable, depending on what they were using as their minimum clinically important difference. But hopefully we’ve made some progress.

Note that I could have tried to pin them down more at the beginning. This researcher was a bit vague about the minimum clinically important difference and didn’t really mention whether a sample size for the experimental group equal to, larger than, or smaller than the control group was what they were looking for. But when you are doing this sort of thing by email, asking them to pin things down precisely before you do any sample size justification will slow down the process versus giving them what you think is a plausible scenario and letting them make adjustments to that scenario. There’s a risk here that your client will take your scenario as fixed and mandated by the needs of the statistics, but I know this client well enough that I don’t have to worry about this.

I haven’t presented these results yet, so I don’t have a report on what the reaction will be. If I’m way off on my assumptions, I may write a new blog entry or add to this blog entry.

Bristol University | Centre for Multilevel Modelling | MLPowSim. Available at: http://www.bristol.ac.uk/cmm/software/mlpowsim/