The Impact of Checklists on Inpatient Safety Outcomes: A Systematic Review of Randomized Controlled Trials

30-day Mortality

A random-effects model pooling data from the 3 studies that reported data for 30-day all cause mortality suggested a significant reduction with use of a checklist (OR 0.60, 95% CI, 0.41-0.89; P = 0.01, I² = 0.0%, P = 0.573).

Study Quality

Supplementary File 5 (Table 5) summarizes the quality assessment of the 9 studies. The clarity of description for each intervention varied. All studies reported inclusion/exclusion criteria and randomization procedures. Three studies indicated that outcome assessors were blinded to intervention allocation;^32,34,36 while this was unclear in 2 studies.^21,30 Three studies reported baseline characteristics.^21,30,36 Two studies reported power calculations;^33,37 however, one study had a sample size that was less than that required to achieve the target power.³⁷ The Jadad scores ranged from 1to 5.

This systematic review identified 9 RCTs that examined the impact of a checklist on patient safety outcomes in hospitalized patients. The studies employed checklists with different purposes and elements and measured different patient safety outcomes. The methodological quality of the included studies was moderate. In aggregate, the results suggest that checklists may be effective at improving patient safety outcomes, but the small number of moderate quality studies and the heterogeneity of interventions and outcome measures suggests that there is an urgent need for further evaluation.

The most important observation from our systematic review is the paucity of high quality evidence evaluating checklists’ impact on patient safety outcomes in acute inpatient care. The implementation of checklists is increasingly common as they are relatively low cost to develop and implement, and intuitively make sense. This is particularly true in an era of increasing efforts to standardize care as a means for improving quality and minimizing cost (ie, previous systematic reviews cite 38 unique studies).³⁹ However, implementation of an inadequately tested checklist risks unintended consequences (eg, inefficient resource utilization).¹⁸ The small number of RCTs identified might be owing to quality improvement efforts traditionally focusing on ‘real life’ applicability over rigorous research methodology.⁴⁰ The translation of evidence into clinical practice is known to be slow;⁴¹ however, these more rigorous methodologies reduce the risk of biases and generate high-quality evidence, which help to fulfill the necessity to identify best practices while avoiding these unintended consequences.

The studies varied both in the approaches used to develop checklists and in the number of items included (ranging from 2 to 54). What is the optimal method for developing a checklist and how does this impact their effectiveness?⁴² The answers to these questions are not known. However, this review highlights some important issues to consider when developing a checklist. As the number of items or complexity of a task increases, our ability to efficiently perform the task without aid decreases.^43-45 As such, a well-designed checklist should detail explicit instructions on the what, where, when, and how of a given task in a fashion that ensures a consistent accuracy for completing the work.⁵ It is recommended that construction of a checklist follow the principles of human factors engineering: engage stakeholders and human factors experts in the design; are developed based on user needs and realities; list items in order of importance; are concise and subgroup sections of checklists by task or chronological order; ensure usability and evaluate potential negative consequences (eg time to complete); are pilot tested and validated before implementation; are updated as needed based the on generation of new findings or changes in operational procedures.⁴⁶ These general principles of human factors engineering⁴⁶ provide a practical approach for the development and evaluation of a checklist. In addition, standardization of operational definitions (ie, process, outcome, compliance) is important for study replication and robust meta-analyses.

Checklists used in aviation are perhaps best known¹² and the evidence of their effectiveness is derived from the attribution of aviation errors to incomplete checklists.¹² Although more recently implemented in medicine, checklists have the potential to guide the successful completion of complex tasks in healthcare.⁷ Systematic reviews of observational studies have been conducted for specific checklists (eg, WHO Surgical Safety Checklist) and for select patient populations (eg, surgical patients), and the number of included studies ranges from 7-27 (n = 38 unique studies).^15,16,18,19 For example, Gillespie et al. in a systematic review and meta-analysis reported the implementation of Surgical Safety Checklists to be associated with a reduction in postoperative complications (relative risk [RR] 0.63, 95% CI, 0.58-0.72, P = < 0.001), but not mortality (RR 1.03, 95% CI, 0.73-1.4, P = 0.857).¹⁹ Similarly, Treadwell et al. reported in a systematic review of Surgical Safety Checklists that while data are promising, more evaluation of their impact on clinical outcomes is needed.¹⁸ These recommendations are nicely illustrated by Urbach et al.’s²⁰ and O’Leary et al.’s⁴⁷ evaluations of the mandatory adoption of Surgical Safety Checklists across all hospitals in Ontario, Canada, which respectively demonstrated no significant reductions in 30-day perioperatively conplications for both adult (OR 0.97, 95% CI, 0.90-1.03, P = 0.29) and pediatric (AOR 1.01, 95% CI, 0.90-1.14, P = 0.9) patients. These data not only highlight the need for further evaluation of checklists but are also a reminder that checklists and their associated implementation strategies are complex interventions for which there may be important differences between the efficacy reported in clinical trials and the effectiveness reported in implementation studies.⁴⁸ This all suggests that if checklists are to be effective in improving patient safety, process evaluations of implementation⁴⁹ and realist reviews of published studies⁵⁰ may be important to determine optimal approaches for implementation. We believe that, based on the limited currently available evidence, there is urgency for further robust evaluations of checklists before their widespread implementation. If effective, they should be widely implemented. If ineffective, they should be abandoned to minimize unintended consequences and inefficient use of resources.

There are 4 primary limitations to this review that should be considered when interpreting the findings. First, the RCT design is not the study design employed by most quality improvement initiatives.⁴⁰ While some quality improvement experts may argue that an RCT design is insufficiently flexible for applied settings, it does minimize the risk of biased assessments of intervention effectiveness. Second, our search strategy included an RCT filter. The filter helped restrict the number of citations to be reviewed (n = 11,225) but could have resulted in improperly indexed studies being excluded. To guard against this risk, we used the validated Cochrane Highly Sensitive Search Strategy for Identifying Randomized Trials,²⁴ reviewed reference lists of citations included in the review, and solicited suggestions for missing studies from quality improvement experts. Third, our review was restricted to hospitalized patients. Although the studies evaluated commonly reported safety outcomes across patients with diverse clinical conditions, care settings, and providers that broadly reflect hospital-based care, evaluations of checklists in additional patient and provider groups are needed (eg, hospitalists). Furthermore, the effectiveness of checklists for improving patient safety outcomes in outpatients is important; however, the organizational and patient characteristics of these 2 settings (hospitalized vs outpatient) are sufficiently different to warrant separate systematic reviews. Finally, owing to the heterogeneity of the checklists used and outcomes measured, we were unable to perform a robust meta-analysis. Heterogeneity, combined with the small number of studies identified in our search, prevented us from applying statistical methods to assess for publication bias. This limitation of our systematic review highlights an important gap in the literature and emphasizes the importance of additional primary research to evaluate checklists.

In summary, we identified few RCTs that examined checklists designed to improve patient safety outcomes. The small number of existing studies suggests that checklists may improve patient safety outcomes; however, these observations were not reported for all outcomes examined and the studies were heterogeneous and of limited methodological quality. There is an urgent need for high-quality evaluations of the effectiveness of patient safety checklists in inpatient healthcare settings to substantiate their perceived benefits.