Appraising the Evidence Supporting Choosing Wisely® Recommendations
Despite the growing enthusiasm surrounding the Choosing Wisely® campaign, little is known regarding the evidence underlying these recommendations. We extracted references for all 320 recommendations published through August, 2014, including the 10 adult and pediatric recommendations published by the Society for Hospital Medicine. We then categorized each item by evidence strength, and then assessed a sample of referenced clinical practice guidelines (CPGs) using the validated Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument. Among all recommendations, 70.3% cited CPGs, whereas 22.2% cited primary research as their highest level of evidence. Moreover, 7.8% cited case series, review articles, editorials, or lower quality data as their highest level of evidence. Hospital medicine recommendations were more likely to cite CPGs (90%) as their highest level of evidence. Among the sampled CPGs, the median overall score obtained using AGREE II was 54.2% (IQR 33.3%-70.8%), whereas among hospital medicine-referenced CPGs, the median overall score was 58.3% (IQR 50.0%-83.3%). These findings suggest that Choosing Wisely® recommendations vary in terms of evidence strength.
© 2018 Society of Hospital Medicine
As healthcare costs rise, physicians and other stakeholders are now seeking innovative and effective ways to reduce the provision of low-value services.1,2 The Choosing Wisely® campaign aims to further this goal by promoting lists of specific procedures, tests, and treatments that providers should avoid in selected clinical settings.3 On February 21, 2013, the Society of Hospital Medicine (SHM) released 2 Choosing Wisely® lists consisting of adult and pediatric services that are seen as costly to consumers and to the healthcare system, but which are often nonbeneficial or even harmful.4,5 A total of 80 physician and nurse specialty societies have joined in submitting additional lists.
Despite the growing enthusiasm for this effort, questions remain regarding the Choosing Wisely® campaign’s ability to initiate the meaningful de-adoption of low-value services. Specifically, prior efforts to reduce the use of services deemed to be of questionable benefit have met several challenges.2,6 Early analyses of the Choosing Wisely® recommendations reveal similar roadblocks and variable uptakes of several recommendations.7-10 While the reasons for difficulties in achieving de-adoption are broad, one important factor in whether clinicians are willing to follow guideline recommendations from such initiatives as Choosing Wisely®is the extent to which they believe in the underlying evidence.11 The current work seeks to formally evaluate the evidence supporting the Choosing Wisely® recommendations, and to compare the quality of evidence supporting SHM lists to other published Choosing Wisely® lists.
METHODS
Data Sources
Using the online listing of published Choosing Wisely® recommendations, a dataset was generated incorporating all 320 recommendations comprising the 58 lists published through August, 2014; these include both the adult and pediatric hospital medicine lists released by the SHM.4,5,12 Although data collection ended at this point, this represents a majority of all 81 lists and 535 recommendations published through December, 2017. The reviewers (A.J.A., A.G., M.W., T.S.V., M.S., and C.R.C) extracted information about the references cited for each recommendation.
Data Analysis
The reviewers obtained each reference cited by a Choosing Wisely® recommendation and categorized it by evidence strength along the following hierarchy: clinical practice guideline (CPG), primary research, review article, expert opinion, book, or others/unknown. CPGs were used as the highest level of evidence based on standard expectations for methodological rigor.13 Primary research was further rated as follows: systematic reviews and meta-analyses, randomized controlled trials (RCTs), observational studies, and case series. Each recommendation was graded using only the strongest piece of evidence cited.
Guideline Appraisal
We further sought to evaluate the strength of referenced CPGs. To accomplish this, a 10% random sample of the Choosing Wisely® recommendations citing CPGs was selected, and the referenced CPGs were obtained. Separately, CPGs referenced by the SHM-published adult and pediatric lists were also obtained. For both groups, one CPG was randomly selected when a recommendation cited more than one CPG. These guidelines were assessed using the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument, a widely used instrument designed to assess CPG quality.14,15 AGREE II consists of 25 questions categorized into 6 domains: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence. Guidelines are also assigned an overall score. Two trained reviewers (A.J.A. and A.G.) assessed each of the sampled CPGs using a standardized form. Scores were then standardized using the method recommended by the instrument and reported as a percentage of available points. Although a standard interpretation of scores is not provided by the instrument, prior applications deemed scores below 50% as deficient16,17. When a recommendation item cited multiple CPGs, one was randomly selected. We also abstracted data on the year of publication, the evidence grade assigned to specific items recommended by Choosing Wisely®, and whether the CPG addressed the referring recommendation. All data management and analysis were conducted using Stata (V14.2, StataCorp, College Station, Texas).
