Statistical Evidence. Chapter 5. Do the Pieces Fit Together? Systematic Overviews and Meta-analysis.

5.0 Introduction

When there are multiple research studies evaluating a new intervention, you need to find a way to assess the cumulative evidence of these studies. You can do this informally, but medical researchers now use a formal process, known as meta-analysis. Meta-analysis involves the quantitative pooling of data from two or more studies. More recently another term, systematic overview, has come into favor. A systematic overview involves the careful review and identification of all research studies associated with a topic, but it may or may not end up pooling the results of these studies. So meta-analysis represents a subset of all the systematic overviews. I tend to use the older term, meta-analysis, partly because I'm stubborn, but partly because I am interested in the quantitative aspects of this type of research. But most of my comments apply more broadly to systematic overviews.

Case study: Declining sperm counts

In 1992, the British Medical Journal published a controversial meta-analysis. This study (Carlsen 1992) reviewed 61 papers published from 1938 and 1991 and showed that there was a significant decrease in sperm count and in seminal volume over this period of time. For example, a linear regression model on the pooled data provided an estimated average count of 113 million per ml in 1940 and 66 million per ml in 1990.

Several researchers (Olsen 1995; Fisch 1996) noted heterogeneity in this meta-analysis, a mixing of apples and oranges. Studies before 1970 were dominated by studies in the United States and particularly studies in New York. Studies after 1970 included many other locations including third world countries. Thus the early studies were United States apples. The later studies were international oranges. There was also substantial variation in collection methods, especially in the extent to which the subjects adhered to a minimum abstinence period.

The original meta-analysis and the criticisms of it highlight both the greatest weakness and the greatest strength of meta-analysis.

Meta-analysis is the quantitative pooling of data from studies with sometimes small and sometimes large disparities. It doesn't always make sense to pool these studies. Think of it as a multi-center trial where each center gets to use its own protocol and where some of the centers don't bother sending you their data. This is meta-analysis at its worst.

On the other hand, the strength of meta-analysis is that it lays all the cards on the table. Sitting out in the open are all the methods for selecting studies, abstracting information, and combining the findings. Meta-analysis allows objective criticism of these overt methods and even allows replication of the research.

Contrast this to an invited editorial or commentary that provides a subjective summary of a research area. Even when the subjective summary is done well, you cannot effectively replicate the findings. Since a subjective review is a black box, the only way, it seems, to repudiate a subjective summary is to attack the messenger.

Do the pieces fit together? What to look for.

When you are examining the results of a meta-analysis, you should ask the following questions:

Were apples combined with oranges? Heterogeneity among studies may make any pooled estimate meaningless.

Were some apples left on the tree? An incomplete search of the literature can bias the findings of a meta-analysis.

Were all of the apples rotten? The quality of a meta-analysis cannot be any better than the quality of the studies it is summarizing.

Did the pile of apples amount to more than just a hill of beans? Make sure that the meta-analysis quantifies the size of the effect in units that you can understand.

[Remainder of this material deleted out of respect for the publisher's copyright.]

Creative Commons License This work is licensed under a Creative Commons Attribution 3.0 United States License. It was written by Steve Simon on 2005-05-29, edited by Steve Simon, and was last modified on 2017-06-15. Send feedback to ssimon at cmh dot edu or click on the email link at the top of the page. Category: Statistical evidence