Of mites and men: Reference bias in narrative review articles

Data analysis. For reviews with positive intervention recommendations, the supporting references to randomized trials were compared with all those randomized trials that might potentially have been known to the authors—that is, published 2 years or more prior to the publication date of the review.¹

If, for example, a review recommended acaricides and quoted 3 trials that claimed a significant effect of this intervention and 1 trial that did not, and if 8 trials were available at the time but only 3 favored acaricides, this would be a positively biased selection of references, since the proportion of positive trials in the reference list, 3/4, was greater than among all the available trials, 3/8.¹

We did not judge whether the significance testing was correctly done but noted whether the trial authors had reported 1 or more significant results favoring the intervention. We then used a sign test, for positive vs negative selection, to determine whether there was significant reference bias.

Results

The Medline search identified 302 abstracts, of which 151 were clearly of no relevance and 5 were abstracts of the index Cochrane review. Of the remaining 146, 6 were excluded because of the language; 140 were read in full, and a further 70 were excluded for other reasons. This left 70 reviews that all had intervention recommendations (positive, neutral, or negative); 63 of them (90%) positively recommended physical interventions and 30 (43%) recommended chemical interventions (TABLE).

The reviews were published between 1971 and 2002 (see the APPENDIX); they were all narrative in regard to interventions—ie, had no methods section or search strategy, whereas 1 review was systematic in regard to economy.¹⁰ Forty-six reviews (66%) contained references to a least 1 randomized trial (range 1–20, median 2, interquartile range 1–3), 4 of these to chemical interventions only. In total there were 162 trial references out of 777 possible (21%). The most quoted trial ⁹ had 22 citations.

Thirty-nine of the 46 reviews with trials (85%) also had references to non-randomized controlled studies that had been excluded from the Cochrane review. The authors rarely distinguished between the 2 designs but emphasized equally, or some times even more, the results obtained with nonrandomized studies. The most quoted nonrandomized study¹¹ was referred to 25 times.

TABLE
Reviews and treatment recommendations

Citations to trials in reviews

Since all reviews (TABLE) had an opinion on physical interventions, and since most trials were of physical interventions, we restricted the analysis to the 38 reviews with a positive recommendation for physical interventions and omitted 4 with a neutral or negative recommendation for physical interventions. We excluded 2 trial reports of physical interventions because they were published in Italian¹² and Dutch¹³; none of the reviews quoted the Italian trial and 1 quoted the Dutch trial.

There was significant bias towards a positive selection of references in the reviews that recommended physical interventions; 10 reviews were neutral in this respect, whereas 27 reviews had a positive selection and 1 a negative selection (P=2 x 10^–8). Conversely, the 4 reviews that did not recommend physical interventions all had a negative selection of references.

Discussion

Intervention recommendations in narrative reviews of house dust mites and asthma do not reflect the available, reliable evidence. Ninety percent of the reviews positively recommended physical interventions, although the Cochrane review of 28 trials on the topic⁶ failed to find an effect of physical and chemical interventions.

Cochrane reviews are also to some extent subjective, but trials published after the latest update of the Cochrane review on mites have provided support to its negative findings. Of note is a recently published trial of 732 patients who were allergic to mites.¹⁴ It was a double-blind, placebo-controlled trial conducted in Britain to investigate the effect of physical intervention measures (allergen-impermeable covers for mattress, pillow, and quilt). Although the interventions reduced mite allergen levels after 6 months, peak flow, medication use, and asthma symptoms were very similar in the 2 groups. During the next 6 months there was a planned reduction of inhaled corticosteroids, and also after this period, the outcomes were very similar—the morning peak flow was 431 L/min in both groups. The authors concluded that the intervention seemed clinically ineffective, in accordance with the findings of the Cochrane review.¹⁴