Setting the bar for accepting positive findings

The preceding letter was referred to Dr Woolf who responded as follows:

Dr Green recommends mammography but believes the absolute benefit is small, an arguable point. If, as reported, it reduces mortality by 21%,¹ the absolute benefit is sizable for a disease that claims 40,000 lives annually.²

I share Dr Green’s concerns about the dangers of advocacy bias—evidence-based medicine originated largely to address this bias—but I disagree that advocacy should drive the intensity of critical appraisal. Evidence should be rejected if there are explicit reasons to consider it untrue. These include flaws in design, conduct, or reporting that offer a competing explanation for observed outcomes. Such flaws are either apparent in publications or hidden from view. If they are apparent, the evidence is suspect regardless of advocacy bias.

If they are hidden from view, no evidence of bias exists. One engages in speculation to question such studies, putting patients at risk if the guesswork is wrong. Dr Green considers research less trustworthy when advocacy bias is pronounced, even when observable flaws are minor. This entails 3 assumptions: that researchers feel pressured by intense advocacy to produce positive results, that they succumb to this pressure and skew data accordingly, and that this bias is not always discernible in publications. These conditions occur occasionally, but Dr Green considers them pervasive, sufficiently so that he advocates a different standard for critical appraisal when advocacy bias is pronounced.

These assumptions are, to my knowledge, unproved. The hypothesis speaks poorly of researchers, positing that they lack the discipline to ignore advocacy and focus on science over self-interest. A preference for positive results has been demonstrated in industry-sponsored research and journal articles. But that analogy has its limits. Although negative results dampen drug sales and readership, they do not necessarily harm researchers, who often achieve notoriety by proving the ineffectiveness of treatments.

Advocacy bias has disrupted mammography guideline development for years, but there is no evidence that it influenced the original trials. There is little documentation that the investigators, many of them epidemiologists, were biased. No advocacy movement existed in the 1960s when the first trial was launched³; even in the 1970s few people advocated testing asymptomatic women.⁴ Negative findings did not stifle the careers of Miller and colleagues,⁵ whose Canadian trial infuriated advocates by suggesting that mammography could be harmful. Even if advocacy bias tainted some trials, the sheer number of studies reporting the same effect suggests that mammography, and not research bias, explains the results.

Those who think poorly of researchers might believe otherwise—that the 21% difference represents a systematic 35-year effort by investigative teams in 4 countries to skew 8 trials in favor of mammography. People are entitled to their suspicions, but being overcautious can harm patients if solid evidence is ignored. We should judge research on its merits, not on the indiscretion of advocates.

Stephen H. Woolf, MD
Virginia Commonwealth University
Fairfax

REFERENCES

• Kerlikowske K, Grady D, Rubin SM, Sandrock C, Ernster VL. Efficacy of screening mammography: a meta-analysis. JAMA 1995; 273:149-54.
• American Cancer Society. Cancers facts and figures 2000. Atlanta, Ga: American Cancer Society; 2000.
• Shapiro S. Periodic screening for breast cancer: The Health Insurance Plan Project and its sequelae, 1963-1986. Baltimore, Md: Johns Hopkins University Press; 1988.
• Cochrane AL, Holland WW. Validation of screening procedures. Br Med Bull 1971; 27:3-8.
• Miller AB, Baines CJ, To T, Wall C. Canadian National Breast Screening Study, 1: breast cancer detection and death rates among women aged 40 to 49 years. Can Med Assoc J 1992; 147:1459-76.