Do drug treatment POEMs report data in clinically useful ways?

These numbers are rarely found in journal articles and, when present, rarely appear in abstracts, tables, or graphs, where the busy clinician looks to find information quickly. Our study sought to estimate the percentage of drug treatment articles published in major medical journals that provide a calculated ARR, NNT, or number needed to harm (NNH), as demonstrated in TABLE 1.

TABLE 1
How to calculate RRR, ARR, and NNT

Methods

Pilot study
We first performed a pilot study that retrospectively reviewed all drug therapy articles published in the Journal of the American Medical Association (JAMA) from April 1, 2008, through April 1, 2009. Its purpose was to ensure concurrence in data gathering, rule out any measurement bias, and refine the analysis tool.

We applied an algorithmic approach to the review and used an Excel spreadsheet as a record-keeping tool, giving each article an abbreviated name and recording the issue, year, volume, and page numbers. We excluded case reports, review articles that were not systematic reviews or meta-analyses, letters, and editorials. We also excluded articles on cancer chemotherapy because, although family physicians need to have a working knowledge of antineoplastic drugs, they do not routinely prescribe them. Moreover, family physicians rarely write the first prescription for such a drug.

We reviewed the drug treatment articles to determine relevance—that is, whether they met POEMs criteria: addressed a question that most family medicine doctors encounter in a typical 6-month period, measured an outcome that family physicians and patients care about—such as morbidity, mortality, quality of life, or effect on clinical events—and had the potential to change clinical practice.⁸ Articles that met all 3 criteria were included in our analysis; articles that did not were recorded but excluded from further examination.

We analyzed articles that met relevance criteria for validity and clinical usefulness. We assessed validity based on whether the article was a randomized, controlled, double-blinded trial and whether allocation was concealed, follow-up was complete, information was analyzed on an intention-to-treat basis, and the results were statistically significant. We assessed clinical usefulness by recording whether the study reported RRR, ARR, NNT, or NNH, and if so, whether it recorded the information in free text, including the abstract, or in a graph or chart.

Review of articles in 7 journals
After the pilot study, we reviewed articles in JAMA and 6 other journals during a 6-month period from April 1, 2008 through September 30, 2008. We applied the same algorithmic analysis as in the pilot study to drug therapy articles in Archives of Internal Medicine (Arch Intern Med), BMJ, New England Journal of Medicine (N Engl J Med), Lancet, Obstetrics and Gynecology (Obstet Gynecol), and Pediatrics. We met regularly and settled disagreements about relevance, validity, or clinical usefulness by re-reviewing the article.

We recorded the total number of drug therapy articles for each journal, then tallied the number of POEMs and the total number of calculations of RRR, ARR, NNT, and NNH in articles that met POEMs criteria. We used these numbers to determine the percentage of POEMs and POEMs with clinically useful information for each journal.

Results

We identified a total of 995 articles in the 7 journals during the 6-month study period. Of these, 24 (2.4%) were classified as drug therapy POEMs and 6 (0.6%) were POEMs with clinically useful information.

The journals that published the most POEMs were Arch Intern Med, N Engl J Med, BMJ, and JAMA. Arch Intern Med had the highest percentage (33%), followed by JAMA (20%), and BMJ (17%) (TABLE 2).

When we analyzed the POEMs for clinical usefulness based on whether they provided calculated ARR, NNT, or NNH, only 3 journals published POEMs with clinically useful information: JAMA published the most (15%), followed by BMJ (9%) and Obstet Gynecol (4%) (FIGURE).

FIGURE
Percentage of POEMs with clinically useful information*