Beyond Reporting Early Warning Score Sensitivity: The Temporal Relationship and Clinical Relevance of “True Positive” Alerts that Precede Critical Deterioration
BACKGROUND: Clinical deterioration is difficult to detect in hospitalized children. The pediatric Rothman Index (pRI) is an early warning score that incorporates vital signs, laboratory studies, and nursing assessments to generate deterioration alerts.
OBJECTIVES: (1) Evaluate the timing of pRI alerts and clinicians recognizing deterioration or escalating care prior to critical deterioration events (CDEs) and (2) determine whether the parameters triggering alerts were clinically related to deterioration.
DESIGN: CDEs are unplanned transfers to the intensive care unit with noninvasive ventilation, tracheal intubation, and/or vasopressor infusion in the 12 hours after transfer. Using one year of data from a large freestanding children’s hospital without the pRI, we analyzed CDEs that would have been preceded by pRI alerts. We (1) compared the timing of pRI alerts to time-stamped notes describing changes in patient status and orders reflecting escalations of care and (2) identified score component(s) that caused alerts to trigger and determined whether these were clinically related to CDE etiology.
RESULTS: Fifty CDEs would have triggered pRI alerts if the pRI had been in use (sensitivity 68%). In 90% of CDEs, the first clinician note reflecting change in patient status and/or the first order reflecting escalation of care preceded the first pRI alert. All of the vital sign and laboratory components of the pRI and 51% of the nursing components were clinically related to the etiology of the CDE.
CONCLUSIONS: Evidence that clinicians were aware of deterioration preceded pRI alerts in most CDEs that generated alerts in the preceding 24 hours.
© 2018 Society of Hospital Medicine.
METHODS
Patients and Setting
This retrospective cross-sectional study was performed at Children’s Hospital of Philadelphia (CHOP), a freestanding children’s hospital with 546 beds. Eligible patients were hospitalized on nonintensive care, noncardiology, surgical wards between January 1, 2013, and December 31, 2013. The CHOP Institutional Review Board (IRB) approved the study with waivers of consent and assent. A HIPAA Business Associate Agreement and an IRB Reliance Agreement were in place with PeraHealth to permit data transfer.
Definition of Critical Deterioration Events
Critical deterioration events (CDEs) were defined according to an existing, validated measure7 as unplanned transfers to the ICU with continuous or bilevel positive airway pressure, tracheal intubation, and/or vasopressor infusion in the 12 hours after transfer. At CHOP, all unplanned ICU transfers are routed through the hospital’s rapid response or code blue teams, so these patients were identified using an existing database managed by the CHOP Resuscitation Committee. In the database, the elements of CDEs are entered as part of ongoing quality improvement activities. The time of CDE was defined as the time of the rapid response call precipitating unplanned transfer to the ICU.
The Pediatric Rothman Index
The pRI is an automated acuity score that has been validated in hospitalized pediatric patients.2 The pRI is calculated using existing variables from the electronic health record, including manually entered vital signs, laboratory values, cardiac rhythm, and nursing assessments of organ systems. The weights assigned to continuous variables are a function of deviation from the norm.2,8 (See Supplement 1 for a complete list of variables.)
The pRI is integrated with the electronic health record and automatically generates a score each time a new data observation becomes available. Changes in score over time and low absolute scores generate a graduated series of alerts ranging from medium to very high acuity. This analysis used PeraHealth’s standard pRI alerts. Medium acuity alerts occurred when the pRI score decreased by ≥30% in 24 hours. A high acuity alert occurred when the pRI score decreased by ≥40% in 6 hours. A very high acuity alert occurred when the pRI absolute score was ≤ 30.
Development of the Source Dataset
In 2014, CHOP shared one year of clinical data with PeraHealth as part of the process of deciding whether or not to implement the pRI. The pRI algorithm retrospectively generated scores and acuity alerts for all CHOP patients who experienced CDEs between January 1, 2013, and December 31, 2013. The pRI algorithm was not active in the hospital environment during this time period; the scores and acuity alerts were not visible to clinicians. This dataset was provided to the investigators at CHOP to conduct this project.
Data Collection
Pediatric intensive care nurses trained in clinical research data abstraction from the CHOP Critical Care Center for Evidence and Outcomes performed the chart review for this study. Chart abstraction comparisons were completed on the first 15 charts to ensure interrater reliability, and additional quality assurance checks were performed on intermittent charts to ensure consistency and definition adherence. We managed all data using Research Electronic Data Capture.9
