Beyond Reporting Early Warning Score Sensitivity: The Temporal Relationship and Clinical Relevance of “True Positive” Alerts that Precede Critical Deterioration
BACKGROUND: Clinical deterioration is difficult to detect in hospitalized children. The pediatric Rothman Index (pRI) is an early warning score that incorporates vital signs, laboratory studies, and nursing assessments to generate deterioration alerts.
OBJECTIVES: (1) Evaluate the timing of pRI alerts and clinicians recognizing deterioration or escalating care prior to critical deterioration events (CDEs) and (2) determine whether the parameters triggering alerts were clinically related to deterioration.
DESIGN: CDEs are unplanned transfers to the intensive care unit with noninvasive ventilation, tracheal intubation, and/or vasopressor infusion in the 12 hours after transfer. Using one year of data from a large freestanding children’s hospital without the pRI, we analyzed CDEs that would have been preceded by pRI alerts. We (1) compared the timing of pRI alerts to time-stamped notes describing changes in patient status and orders reflecting escalations of care and (2) identified score component(s) that caused alerts to trigger and determined whether these were clinically related to CDE etiology.
RESULTS: Fifty CDEs would have triggered pRI alerts if the pRI had been in use (sensitivity 68%). In 90% of CDEs, the first clinician note reflecting change in patient status and/or the first order reflecting escalation of care preceded the first pRI alert. All of the vital sign and laboratory components of the pRI and 51% of the nursing components were clinically related to the etiology of the CDE.
CONCLUSIONS: Evidence that clinicians were aware of deterioration preceded pRI alerts in most CDEs that generated alerts in the preceding 24 hours.
© 2018 Society of Hospital Medicine.
To study the value of alerts labeled as “true positives,” we restricted the dataset to CDEs in which acuity alert(s) within the prior 72 hours would have been triggered if the pRI had been in clinical use at the time.
To identify the clinical relationship between pRI and CDE, we reviewed each chart with the goal of determining whether the preceding acuity alerts were clinically associated with the etiology of the CDE. We determined the etiology of the CDE by reviewing the cause(s) identified in the note written by rapid response or code blue team responders or by the admitting clinical team after transfer to the ICU. We then used a tool provided by PeraHealth to identify the specific score components that led to worsening pRI. If the score components that worsened were (a) consistent with a clinical change as opposed to a documentation artifact and (b) an organ system change that was plausibly related to the CDE etiology, we concluded that the alert was clinically related to the etiology of the CDE.
We defined documentation artifacts as instances in nursing documentation in which a finding unrelated to the patient’s acute health status, such as a scar, was newly documented as abnormal and led to worsening pRI. Any cases in which the clinical relevance was unclear underwent review by additional members of the team
To determine the temporal relationship among pRI, CDE, and clinician awareness or action, we then sought to systematically determine whether the preceding acuity alerts preceded documented evidence of clinicians recognizing deterioration or escalation of care. We made the a priori decision that acuity alerts that occurred more than 24 hours prior to a deterioration event had questionable clinical actionability. Therefore, we restricted this next analysis to CDEs with acuity alerts during the 24 hours prior to a CDE. We reviewed time-stamped progress notes written by clinicians in the 24 hours period prior to the time of the CDE and identified whether the notes reflected an adverse change in patient status or a clinical intervention. We then compared the times of these notes with the times of the alerts and CDEs. Given that documentation of change in clinical status often occurs after clinical intervention, we also reviewed new orders placed in the 24 hours prior to each CDE to determine escalation of care. We identified the following orders as reflective of escalation of care independent of specific disease process: administration of intravenous fluid bolus, blood product, steroid, or antibiotic, increased respiratory support, new imaging studies, and new laboratory studies. We then compared the time of each order with the time of the alert and CDE.
RESULTS
During the study period, 73 events met the CDE criteria and had a pRI alert during admission. Of the 73 events, 50 would have triggered at least one pRI alert in the 72-hour period leading up to the CDE (sensitivity 68%). Of the 50 events, 39 generated pRI alerts in the 24 hours leading up to the event, and 11 others generated pRI alerts between 24 and 72 hours prior to the event but did not generate any alerts during the 24 hours leading up to the event (Figure).
