Meta-analysis: Its strengths and limitations
ABSTRACTNowadays, doctors face an overwhelming amount of information, even in narrow areas of interest. In response, reviews designed to summarize the large volumes of information are frequently published. When a review is done systematically, following certain criteria, and the results are pooled and analyzed quantitatively, it is called a meta-analysis. A well-designed meta-analysis can provide valuable information for researchers, policy-makers, and clinicians. However, there are many critical caveats in performing and interpreting them, and thus many ways in which meta-analyses can yield misleading information.
KEY POINTS
- Meta-analysis is an analytical technique designed to summarize the results of multiple studies.
- By combining studies, a meta-analysis increases the sample size and thus the power to study effects of interest.
- There are many caveats in performing a valid meta-analysis, and in some cases a meta-analysis is not appropriate and the results can be misleading.
META-ANALYSIS VS LARGE RANDOMIZED CONTROLLED TRIALS
There is debate about how meta-analyses compare with large randomized controlled trials. In situations where a meta-analysis and a subsequent large randomized controlled trial are available, discrepancies are not uncommon.
LeLorier et al6 compared the results of 19 meta-analyses and 12 subsequent large randomized controlled trials on the same topics. In 5 (12%) of the 40 outcomes studied, the results of the trials were significantly different than those of the meta-analysis. The authors mentioned publication bias, study heterogeneity, and differences in populations as plausible explanations for the disagreements. However, they correctly commented: “this does not appear to be a large percentage, since a divergence in 5 percent of cases would be expected on the basis of chance alone.”6
A key reason for discrepancies is that meta-analyses are based on heterogeneous, often small studies. The results of a meta-analysis can be generalized to a target population similar to the target population in each of the studies. The patients in the individual studies can be substantially different with respect to diagnostic criteria, comorbidities, severity of disease, geographic region, and the time when the trial was conducted, among other factors. On the other hand, even in a large randomized controlled trial, the target population is necessarily more limited. These differences can explain many of the disagreements in the results.
A large, well-designed, randomized controlled trial is considered the gold standard in the sense that it provides the most reliable information on the specific target population from which the sample was drawn. Within that population the results of a randomized controlled trial supersede those of a meta-analysis. However, a well conducted meta-analysis can provide complementary information that is valuable to a researcher, clinician, or policy-maker.
CONCLUSION
Like many other statistical techniques, meta-analysis is a powerful tool when used judiciously; however, there are many caveats in its application. Clearly, meta-analysis has an important role in medical research, public policy, and clinical practice. Its use and value will likely increase, given the amount of new knowledge, the speed at which it is being created, and the availability of specialized software for performing it.30 A meta-analysis needs to fulfill several key requirements to ensure the validity of its results:
- Well-defined objectives, including precise definitions of clinical variables and outcomes
- An appropriate and well-documented study identification and selection strategy
- Evaluation of bias in the identification and selection of studies
- Description and evaluation of heterogeneity
- Justification of data analytic techniques
- Use of sensitivity analysis.
It is imperative that researchers, policy-makers, and clinicians be able to critically assess the value and reliability of the conclusions of meta-analyses.
