Meta-analysis: Its strengths and limitations

Author and Disclosure Information

ABSTRACTNowadays, doctors face an overwhelming amount of information, even in narrow areas of interest. In response, reviews designed to summarize the large volumes of information are frequently published. When a review is done systematically, following certain criteria, and the results are pooled and analyzed quantitatively, it is called a meta-analysis. A well-designed meta-analysis can provide valuable information for researchers, policy-makers, and clinicians. However, there are many critical caveats in performing and interpreting them, and thus many ways in which meta-analyses can yield misleading information.


  • Meta-analysis is an analytical technique designed to summarize the results of multiple studies.
  • By combining studies, a meta-analysis increases the sample size and thus the power to study effects of interest.
  • There are many caveats in performing a valid meta-analysis, and in some cases a meta-analysis is not appropriate and the results can be misleading.



The amount of information generated in medical research is becoming overwhelming, even for experienced researchers. New studies are constantly being published, and clinicians are finding it nearly impossible to stay current, even in their own area of specialty.

To help make sense of the information, we are seeing more and more review articles that pool the results of multiple studies. When certain principles are followed and the data are quantitatively analyzed, these reviews are called meta-analyses. A PubMed search of the word “meta-analysis” in the title yielded 1,473 articles in the year 2007.

Combining available information to generate an integrated result seems reasonable and can save a considerable amount of resources. Nowadays, meta-analyses are being used to design future research, to provide evidence in the regulatory process,1 and even to modify clinical practice.

Meta-analysis is powerful but also controversial—controversial because several conditions are critical to a sound meta-analysis, and small violations of those conditions can lead to misleading results. Summarizing large amounts of varied information using a single number is another controversial aspect of meta-analysis. Under scrutiny, some meta-analyses have been inappropriate, and their conclusions not fully warranted.2,3

This article introduces the basic concepts of meta-analysis and discusses its caveats, with the aim of helping clinicians assess the merits of the results. We will use several recent meta-analyses to illustrate the issues, including a controversial one4 with potentially far-reaching consequences.


The main objectives of a meta-analysis are to:

  • Summarize and integrate results from a number of individual studies
  • Analyze differences in the results among studies
  • Overcome small sample sizes of individual studies to detect effects of interest, and analyze end points that require larger sample sizes
  • Increase precision in estimating effects
  • Evaluate effects in subsets of patients
  • Determine if new studies are needed to further investigate an issue
  • Generate new hypotheses for future studies.

These lofty objectives can only be achieved when the meta-analysis satisfactorily addresses certain critical issues, which we will discuss next.


Four critical issues need to be addressed in a meta-analysis:

  • Identification and selection of studies
  • Heterogeneity of results
  • Availability of information
  • Analysis of the data.


The outcome of a meta-analysis depends on the studies included. The critical aspect of selecting studies to be included in a meta-analysis consists of two phases. The first is the identification phase or literature search, in which potential studies are identified. In the second phase, further criteria are used to create a list of studies for inclusion. Three insidious problems plague this aspect of meta-analysis: publication bias and search bias in the identification phase, and selection bias in the selection phase. These biases are discussed below.

Publication bias: ‘Positive’ studies are more likely to be printed

Searches of databases such as PubMed or Embase can yield long lists of studies. However, these databases include only studies that have been published. Such searches are unlikely to yield a representative sample because studies that show a “positive” result (usually in favor of a new treatment or against a well-established one) are more likely to be published than those that do not. This selective publication of studies is called publication bias.

In a recent article, Turner et al5 analyzed the publication status of studies of antidepressants. Based on studies registered with the US Food and Drug Administration (FDA), they found that 97% of the positive studies were published vs only 12% of the negative ones. Furthermore, when the nonpublished studies were not included in the analysis, the positive effects of individual drugs increased between 11% and 69%.

One reason for publication bias is that drug manufacturers are not generally interested in publishing negative studies. Another may be that editors favor positive studies because these are the ones that make the headlines and give the publication visibility. In some medical areas, the exclusion of studies conducted in non-English-speaking countries can increase publication bias.6

To ameliorate the effect of publication bias on the results of a meta-analysis, a serious effort should be made to identify unpublished studies. Identifying unpublished studies is easier now, thanks to improved communication between researchers worldwide, and thanks to registries in which all the studies of a certain disease or treatment are reported regardless of the result.

The National Institutes of Health maintains a registry of all the studies it supports, and the FDA keeps a registry and database in which drug companies must register all trials they intend to use in applying for marketing approval or a change in labeling. “Banks” of published and unpublished trials supported by pharmaceutical companies are also available (eg, The Cochrane collaboration ( keeps records of systematic reviews and meta-analyses of many diseases and procedures.


Next Article:

Related Articles