This page is moving to a new website.

Dear Professor Mean, I’m helping a student with a piece of work and he described systematic reviews as being reliable. I’m troubled by this on a number of fronts.

Your student, I suspect, is using the term “reliable” in a generic sense meaning “trustworthy” or “persuasive.” The real question, I suspect, is not whether systematic reviews are reliable, but rather are they valid?

There are a few empirical studies that could possibly help resolve the question of validity.

Jacques LeLorier, Genevieve Gregoire, Abdeltif Benhaddad, Julie Lapierre, Francois Derderian. Discrepancies between Meta-Analyses and Subsequent Large Randomized, Controlled Trials. N Engl J Med. 1997;337(8):536-542.

What these authors did was compare a large randomized trial to an earlier published meta-analysis. The results were so-so, but read the article yourself and decide.

Ian Shrier, Jean-Francois Boivin, Robert Platt, et al. The interpretation of systematic reviews with meta-analyses: an objective or subjective process? BMC Medical Informatics and Decision Making. 2008;8(1):19.

This study presented a series of meta-analyses and found that different people had different interpretations of identical results.

Laura Rosen, Michal Ben Noach, Elliot Rosenberg. Missing the forest (plot) for the trees? A critique of the systematic review in tobacco control. BMC Medical Research Methodology. 2010;10(1):34.

This study looked at how much inconsistency there was in how different researchers conducted their meta-analyses and found that there were a lot of problems.

These empirical studies are not such a great endorsement, perhaps, but you have to put things in perspective. Compared to what preceded it, the expert editorial, a systematic overview is a vast improvement. Nothing makes this point better than the following satirical article.

A Caveman. The invited review – or, my field, from my standpoint, written by me using only my data and my ideas, and citing only my publications. J Cell Sci. 2000;113(18):3125-3126.

Does the systematic overview deserve a high spot in the hierarchy of evidence? You could argue that a systematic overview is no better than an observational study. One analogy that I cite is that a systematic overview is like a multi-center study where every center gets to choose its own protocol.

But another argument is that a systematic overview, by synthesizing multiple clinical trials, has to be better than any single clinical trial. If you have multiple clinical trials with potentially conflicting interpretations, how do you handle this in an evidence based fashion without a systematic overview? Do you chose the single study that has the best quality score or the one with the largest sample size and ignore everything else? That seems like a recipe for disaster.

Systematic overviews also should have better generalizability. If you find ten studies, each conducted in different geographical location, and they all say more or less the same thing, then you have better proof that the research will generalize to your geogrpahic setting than you would with any single study.

Another big advantage of systematic overviews is that they represent a repeatable and reproducible form of research. If you lay out your methods carefully, anyone else could repeat your work and get the same results. They can tweak your analysis if they don’t like your search strategy or your inclusion/exclusion criteria, or your method of pooling results.

Systematic overviews are also self-critical. If there is a limitation due to publication bias, heterogeneity, or lack of good quality studies, the tools of a systematic overview will identify those limitations.

Finally, most of the empirical evidence on the value of blinding, concealed allocation, and other aspects of clinical trials comes from systematic overviews. If you say that you believe in blinding but you don’t believe in systematic overviews, then your belief in blinding must be faith based and not evidence based.

Most statisticians that I know put very little faith in systematic overviews, but I am generally positive on them, especially on the systematic overviews conducted by the Cochrane Collaboration. They’re not perfect, but nothing in evidence based medicine is perfect.