Of mites and men: Reference bias in narrative review articles

The narrative reviews were heavily biased by a selection of trial references that supported the opinions of the authors. When this was not the case, some authors—instead of ignoring negative results—used them as evidence that more rigorous intervention measures are needed for a positive intervention effect.

Physical interventions may need to be applied repeatedly before the reduction in allergen levels is sufficient to be effective. However, the lack of effect was also apparent in the subgroup of trials with long treatment duration or follow-up. And, if interventions were effective, one would expect to see at least some effect also in short-term trials.

Some of the authors of narrative reviews used long trial duration as a positive selection criterion for what they believed were relevant trials, but most often disregarded whether the trials were of acceptable quality in other respects. For example, the most popular randomized trial ran for a year⁹ and was quoted by 22 of the 35 reviews with trial references where it could have been quoted (63% citation rate). However, this trial had only 7 patients per group in the analysis, the outcome was not a clinical outcome but the result of a histamine tolerance test, and the part of the trial that yielded a significant result was not blinded. Furthermore, this result was probably not statistically significant. The authors claimed a significant result with nonparametric testing (P<.05), but when we repeated the test, using their raw data as depicted in a graph, we got P=.07; and in the comparison with the placebo group that was used in the Cochrane review of the trials,⁶ we got P=.15 for the geometric mean.

This trial⁹ is not only the most quoted in our sample of reviews, it is also the most quoted of our 28 included trials in the Science Citation Index as of September 17, 2003, where it had received 209 citations. We find it disturbing that 7 children in a nonblinded trial with no clinical outcomes have been so influential for the intervention recommendations over the past 10 years.

A survey of review articles on the relation between diet and heart disease³ showed that authors often used nonrandomized studies to create a false impression of consensus by referring to them together with the randomized trials if their results supported the authors’ opinion and ignoring them if they did not. We also found a high citation rate for nonrandomized studies. In fact, the most popular nonrandomized controlled study was only surpassed in citation rate by the small randomized trial just discussed. It was a physical intervention study,¹¹ quoted by 25 of the 45 reviews with trial references where it could have been quoted (56% citation rate). This study was usually referred to without comment on its methodology. It had only 10 patients per group and its duration was only 1 month. The likely explanation for the high quotation rate for this study is that it had a very positive outcome for the intervention. It therefore seems that the reported results were more important than the study duration as a criterion for selection of studies for quotation.

The 24 reviews we excluded from statistical analysis because they had no references to randomized trials generally had strong intervention recommendations and argued that a reduced allergen level will lead to patient improvement. The authors often based this claim on uncontrolled studies in which the patients had been relocated to a dry, high-altitude location in the Alps with good clinical effect, but the authors failed to appreciate that the kind of allergen reduction achieved in these studies currently appears impossible to obtain in the patients’ homes.

Authors of review articles can be expected to stay abreast with new trials and to be familiar with systematic reviews within their field of expertise. However, the only systematic review^6-8 in this area was quoted in just 6 of 12 reviews published at least 2 years later than the Cochrane review. Only 1 review¹⁵ did not recommend physical intervention because of the findings in the Cochrane review; another review¹⁶ disregarded the Cochrane review with the reasoning that few of the included trials obtained a reduction in allergen levels (although the Results section of the Cochrane review explained that the results were very similar in the subgroup of trials that reported successful mite reduction⁶).

Of the other 4 reviews that quoted the Cochrane review, the first¹⁷ mentioned it only in the introduction; the second¹⁸ claimed that the methods investigated were out of date; the third¹⁹ noted that an expert panel reviewing the same articles had come to a different conclusion than the Cochrane review; and the fourth²⁰ stated that the complexity and heterogeneity of the interventions limited the utility of a simple statistical summary of the data as presented in the Cochrane review. Interestingly, all 4 reviews recommended physical intervention. It therefore seems that the assets of a systematic approach to the literature based on a predefined protocol, and applying rigorous methodology, was little understood.