Dear Professor Mean, I’m helping a student with a piece of work and he described systematic reviews as being reliable. I’m troubled by this on a number of fronts.

Your student, I suspect, is using the term “reliable” in a generic sense meaning “trustworthy” or “persuasive.” The real question, I suspect, is not whether systematic reviews are reliable, but rather are they valid?

There are a few empirical studies that could possibly help resolve the question of validity.

Jacques LeLorier, Genevieve Gregoire, Abdeltif Benhaddad, Julie Lapierre, Francois Derderian. Discrepancies between Meta-Analyses and Subsequent Large Randomized, Controlled Trials. N Engl J Med. 1997;337(8):536-542.

What these authors did was compare a large randomized trial to an earlier published meta-analysis. The results were so-so, but read the article yourself and decide.

Ian Shrier, Jean-Francois Boivin, Robert Platt, et al. The interpretation of systematic reviews with meta-analyses: an objective or subjective process? BMC Medical Informatics and Decision Making. 2008;8(1):19.

This study presented a series of meta-analyses and found that different people had different interpretations of identical results.

Laura Rosen, Michal Ben Noach, Elliot Rosenberg. Missing the forest (plot) for the trees? A critique of the systematic review in tobacco control. BMC Medical Research Methodology. 2010;10(1):34.

This study looked at how much inconsistency there was in how different researchers conducted their meta-analyses and found that there were a lot of problems.

These empirical studies are not such a great endorsement, perhaps, but you have to put things in perspective. Compared to what preceded it, the expert editorial, a systematic overview is a vast improvement. Nothing makes this point better than the following satirical article.

A Caveman. The invited review – or, my field, from my standpoint, written by me using only my data and my ideas, and citing only my publications. J Cell Sci. 2000;113(18):3125-3126.

Does the systematic overview deserve a high spot in the hierarchy of evidence? You could argue that a systematic overview is no better than an observational study. One analogy that I cite is that a systematic overview is like a multi-center study where every center gets to choose its own protocol.

But another argument is that a systematic overview, by synthesizing multiple clinical trials, has to be better than any single clinical trial. If you have multiple clinical trials with potentially conflicting interpretations, how do you handle this in an evidence based fashion without a systematic overview? Do you chose the single study that has the best quality score or the one with the largest sample size and ignore everything else? That seems like a recipe for disaster.

Systematic overviews also should have better generalizability. If you find ten studies, each conducted in different geographical location, and they all say more or less the same thing, then you have better proof that the research will generalize to your geogrpahic setting than you would with any single study.

Another big advantage of systematic overviews is that they represent a repeatable and reproducible form of research. If you lay out your methods carefully, anyone else could repeat your work and get the same results. They can tweak your analysis if they don’t like your search strategy or your inclusion/exclusion criteria, or your method of pooling results.

Systematic overviews are also self-critical. If there is a limitation due to publication bias, heterogeneity, or lack of good quality studies, the tools of a systematic overview will identify those limitations.

Finally, most of the empirical evidence on the value of blinding, concealed allocation, and other aspects of clinical trials comes from systematic overviews. If you say that you believe in blinding but you don’t believe in systematic overviews, then your belief in blinding must be faith based and not evidence based.

Most statisticians that I know put very little faith in systematic overviews, but I am generally positive on them, especially on the systematic overviews conducted by the Cochrane Collaboration. They’re not perfect, but nothing in evidence based medicine is perfect.

Rahul Mhaskar, Benjamin Djulbegovic, Anja Magazin, Heloisa P. Soares, and Ambuj Kumar. Published methodological quality of randomized controlled trials does not reflect the actual quality assessed in protocols.J Clin Epidemiol. 2012 Jun; 65(6): 602–609. doi: 10.1016/j.jclinepi.2011.10.016. Available at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3637913/

First things first, you should talk to a medical librarian. A librarian can help you in a lot of ways above and beyond what a statistician can help you with. I’m going to focus on the statistical issues associated with a systematic overview, but will also comment where appropriate on issues that the librarian can help you with.

You may have seen the term meta-analysis instead of systematic overview. These terms are used interchangeably, which is not surprising because they are so similar. There is a technical distinction, however. Meta-analysis is the statistical combination of quantitative information from multiple research studies. A systematic overview is a rigorous review of the peer-reviewed literature. You must have a systematic overview in order to run a meta-analysis. But a systematic overview will not always lead to a quantitative combination of information.

The word “systematic” in systematic overview is not there by accident. The systematic overview is intended to be a reproducible form of research, so you need to prepare a research protocol that describes all of the steps that you plan to take. In many ways, the protocol for a systematic overview resembles the protocol for a prospective clinical trial. In particular, the research papers in your systematic overview are like the patients in a clinical trial, except that you don’t have to read an informed consent statement to a research paper.

First, define in clear and unambiguous terms what types of research studies you are looking for in your systematic overview. These are like the inclusion and exclusion criteria that you see in a prospective clinical trial. Tread cautiously here. Cast your net too widely and you are likely to have difficulty with heterogeneity, the statistical mixing of apples and oranges. But a little bit of heterogeneity is actually a benefit. If you study an intervention across a wide range of implementations, a wide range of patient populations, and a wide range of research designs, and the results are reasonably consistent, you have found evidence of a robust intervention that works consistently no matter what.

If you are thinking about inclusion and exclusion criteria for research studies, you might start first by considering the type of intervention or exposure. Make sure that you define your boundaries clearly. You may, for example, be interested in weight loss programs. Would you want to include programs that are dietary only, or is some combination of diet and exercise okay? How about weight loss programs that include some type of drug or some type of surgery? You need to think in terms of both inclusions and exclusions.

You may also want to place restrictions on the types of patients that are being studied. Do you want to look only at moderately ill patients, only at severely ill patients, or any combination? Do you want to limit the ages of the patients being studied? Do you want to restrict the geographical boundaries (e.g., studies in the United States only)? Do you want to limit where patients receive their treatments (e.g., outpatient only)?

Think about how far back you want to go. Do you want studies only in the past decade or would earlier studies be okay?

Finally, think about any restrictions that you might place on the research design? Do you want only blinded studies, only randomized studies, only prospective studies? Do you care if the researchers used a surrogate outcome? Do you want to look only at a certain duration of follow-up time (e.g., no short term studies)?

You may decide on other exclusions as well. Remember that you want the systematic overview to be reproducible, so define your inclusion and exclusion criteria in sufficient detail that another reasonably intelligenet person would end up picking the same studies that you did.

In your protocol, explain where you will find your research papers. There’s a lot of debate in the research community about how rigorous you need to make your literature search. Most people recommend that you review several bibliographic databases, not just PubMed. There are many unpublished research studies, and you need to identify how you will try to find these studies as well. A medical librarian can help you design a reasonably comprehensive literature search strategy. One factor that you may overlook, if you don’t consult with a librarian first, is how to save your search strategy. A saved search strategy ensures the reproducibility of your work and also allows you to refine your search strategy as needed.

There are sensitive searches and specific searches and you need to balance these appropriately. A sensitive search is one that, if a study is out there, your search is likely to find it. You get a more sensitive search by looking at multiple bibliographic databases, hand searching selected journals, sending out requests to prominent researchers in this area, and reviewing clinical trial registries. You also improve sensitivity by including a wide range synonyms and using MeSH terms in your search. How much effort do you want to put in here? That’s a tough call. A sensitive search does reduce the risk of bias in your systematic overview. There’s a fair amount of empirical data to support the notion your systematic overview will be overly optimistic if you leave out papers that are harder to find: because they didn’t make it into a peer-reviewed study, because they were published in a journal not indexed by the major databases, because they were written in a language other than English, and so forth. So your efforts to improve sensitivity will pay off in producing a systematic overview that is more authoritative.

But too much of this might be overkill. There’s a saying that the good is the enemy of the great and I tend to fall on the side of the good rather than the great. The effort you spend producing an ever more exquisitely sensitive search is effort that could have been spent producing a second or even a third good systematic overview.

The particular topic you want to research may be an important consideration here. Certain types of therapies, such as complementary and alternative medicine are likely to have lots of research in obscure journals and a lot of research that was conducted but never published. So for an area like this, improving your sensitivity may be really important.

The flip side of the coin is a specific search. A specific search does not produce a lot of false positives: papers that don’t really meet your inclusion criteria. It’s nearly impossible to produce a highly specific search unless you are willing to live with mediocre sensitivity. So get used to the fact that you’ll be reviewing a lot more papers than the ones that meet your inclusion and exclusion criteria. Your librarian can help some here. In particular, if you are looking for certain factors associated with the research design, such as randomized studies only, there are some filters that have been tested and shown to work reasonably well.

Because you aren’t likely to get a highly specific search, it’s okay to screen some studies out, based just on the abstract. The abstract is often a highly imperfect summary of a research study, but for the purposes of deciding which studies obviously need to be excluded, the abstract does a pretty good job. Keep two things in mind, though. First, if the paper does not have an abstract, you have to read the entire paper. Second, if there is any uncertainty in whether a paper meets your inclusion criteria based just on the abstract, you have to read the full paper.

Keep track of the number of papers that you have identified and the number which were excluded based just on a review of the abstract.

Now for the papers that didn’t obviously get excluded based just on the abstract, you need to get some help. First, you’ll need to translate any papers not written in English. More importantly, you need to have a second person read these papers independently from you. Your goal is reproducibility and one of the best ways to establish reproducibility is to demonstrate that two independent reviewers both came to the same conclusions about whether a paper should be included or excluded. Track the number of papers that you and your friend disagreed on. You can resolve this disagreement through the use of a third party to adjudicate the dispute. If the number of disagreements is too high, then rethink your inclusion and exclusion criteria and write them with more precision.

All along the way, you have to make sure that a study is not double -counted. Some duplicates come because you used multiple bibliographic databases, and these are not too hard to find and remove. The bigger problem is when researchers like a study so much that they publish it several times in several different locations. There are sometimes legitimate reasons for this: one paper looks at the primary outcomes and another looks at the secondary outcomes. One paper might look at the short term results and a paper on the long term results appears later. Sometimes a paper appears originally in the native language and then gets republished in an English language journal. In a multi-center study, one paper might focus on the summary across all centers and another might look at a single center because that center conducted additional research and collected additional data above and beyond what the other centers did. When the reasons for multiple publications are legitimate, the authors will normally cite the related papers.

What is more worrisome is covert duplicate publications, where researchers publish their research results twice without letting anyone know. This might be an effort to pad someone’s resume, but it represents an abuse of the free reviews provided by the editors and peer reviewers. Tracking covert duplicate publications is not easy. Sometimes (but not always) you can deduce this by looking at the author lists, or reviewing some of the details of the studies.

If you end up counting some of the studies twice, you are likely to produce an overly optimistic bias, because the studies with very striking results are more likely to have duplicate publications.

Now that you have all the papers in hand that satisfy your inclusion and exclusion criteria, you need to build a database from these papers. Recruit your friend again here, because the process of pulling data from research papers is not as clear and unambiguous as you might think. So both you and your friend read the paper independently and fill your database independently. If you can demonstrate that you and your friend produce pretty much the same data, then you have established that the systematic overview is reproducible. Get a third party to adjudicate the few differences that you do have.

If you disagree a lot, then you have to start over again with more precise definitions of what you want. This will really test your friendship and you might want to spring for a fancy dinner after all the work is done.

So what do you put in your database? First, you need information about the studies themselves. In particular, you want information on the interventions, the patients being studied, and the type of research design. Focus on information from the studies that are likely to be sources of heterogeneity. If your intervention is defined quite broadly, for example, then you need to categorize which interventions are evaluated in which studies.

Many systematic overviews use quality scores (the Jadad score is one example), and you and your friend should apply the quality scores independently. Does your friend like flowers? Maybe a dozen roses in addition to that fancy dinner is in order here.

Your database needs information from all the relevant outcome measures. You can decide in advance that you will exclude certain outcome measures, of course, but even with these measures excluded, you will shocked at how many different ways that researchers can measure the same thing.

The two most common types of outcome measures are continuous outcome measures and binary outcome measures. For continuous outcome measures, you need to record six numbers: the mean, standard deviation, and sample size in the treatment group and the mean, standard deviation, and sample size in the control group. For binary outcome measures, you need to record four numbers: the number of events and the total sample size in the treatment group and the number of events and the total sample size in the control group. Not every study, of course, will have every outcome measure, so develop a missing value code.

There are lots of complications that can occur, and you should consult with a statistician about how to handle these cases. For example, some studies may not report a standard deviation. Sometimes you can infer what the standard deviation would be based on other measures of variability, but do this carefully. Some studies might use paired patients and some might use unpaired patients. There’s no easy way to combine paired and unpaired data. Some studies rely on survival curves, and for these studies, you should try to get access to the original data sets.

If you don’t have these complications, the standard meta-analysis is not that difficult. Even so, you should probably use specialized software for this. The two basic choices for analyses are a fixed effects meta-analysis and a random effects meta-analysis. I strongly prefer the random effects meta-analysis, because it mirrors closely how you might analyze a multi-center clinical trial. (I often joke that meta-analysis is a multi-center trial where each center gets to use its own protocol.)

There are measures of heterogeneity that you can use, and funnel plots that some researchers like to see as an assessment of publication bias. But there is a fair amount of criticism of both measures of heterogeneity and funnel plots. When you’re writing your protocol, it’s best to mention that you will calculate these things because you can always leave them off in your publication.

The one graph that is universally used is the forest plot and it is actually a pretty good plot. One final choice that you will face is with binary outcome measures. For these measures, you can combine them as odds ratios, or as relative risks, or as absolute risks. This choice is trickier than you might think because sometimes the statistic that you use for a binary outcome can create heterogeneity. In general, I like to combine odds ratios, but there are sometimes good arguments to be made for combining relative risks.

The one thing that difference in meta-analysis is that subgroup analysis is strongly encouraged. If you’re familiar at all with other types of research design, the research community is generally hostile towards subgroup analysis. There are good reasons for this, but in meta-analysis, one of your biggest concerns is heterogeneity. If you can show that the results across various subgroups are reasonably consistent, then you have established that heterogeneity does not exist (or perhaps it exists but does not produce any problems). If you find that some subgroups are different than others then you have removed the problem of heterogeneity by focusing on smaller, more homogeneous subsets. You can also think of subgroup analysis as a type of sensitivity analysis. One of the strengths of the forest plot, by the way, is that it easily accommodates subgroup analyses. That’s probably the reason it is so popular.

So what subgroups should you look at. This may depend on what types of studies you find, but prior to the study, try to imagine potential sources of heterogeneity. Is there a lot of variation in patient populations? Are there a broad range of research designs that are commonly used? Is the duration of follow-up quite different? Areas where you might expect to see heterogeneity are areas that you should plan subgroup analyses around. Then allow for the possibility of conducting several post hoc subgroup analyses based on what you find during data collection.

I wish you the best of luck on your systematic overview. They are a lot of work, but they are also very important.

This is the third in a series of blog entries: So you’re thinking about research. The two entries are:

PMean: So you’re thinking about a pilot study, and

PMean: So you’re thinking about a retrospective chart review.

This page has moved to a new website.

There is no consensus in the research community on how to analyze an ordinal outcome measure. Some people use a non-parametric test (e.g., Mann-Whitney test). Others use ordinal logistic regression. Still others will treat the ordinal variable as if it were interval/ratio and apply a t-test. Finally, some will use a cut-point to create a binary outcome.

The latter two approaches, by the way, have some serious issues, but that doesn’t stop people from using them.

The non-parametric test leaves you out in the cold because there is no effect size associated with a non-parametric test. You might be able to cobble something together, but you’d probably need access to individual patient data.

The ordinal logistic regression produces an odds ratio and you can probably get those odds ratios into your meta-analysis software. Be careful, because the standard error for the odds ratio in ordinal logistic regression is not the same as it would be for the odds ratio from a two by two table.

The t-test is pretty easy to handle, as you have a mean and standard deviation for each group.

Of course, with no consensus among researchers, you will probably have at least one study choosing each of the different analysis strategies, making it next to impossible to pool the results.

Howick J, Mebius A. In search of justification for the unpredictability paradox. Trials. 2014;15(1):480. doi:10.1186/1745-6215-15-480. Available at http://www.trialsjournal.com/content/15/1/480/abstract.