Patient Safety Indicator-12 Rarely Identifies Problems with Quality of Care in Perioperative Venous Thromboembolism

Utilizing AHRQ version 5 software for PSI-12, our institution identified 154 cases of perioperative VTE. Most of these cases were a pulmonary embolism, occurred within a week of admission, and were associated with surgical specialties that portend a higher risk of VTE. Very few of these cases were deficient in guideline-directed VTE prophylaxis, and several cases had associated factors such as trauma and thrombocytopenia that may have appropriately influenced the decision to use mechanical only prophylaxis. Sixteen percent of pharmacologic VTE prophylactic doses were refused or held for a procedure, a known but rarely quantified influence on rates of pharmacologic prophylaxis. The use of patient-level data and adjudication of the clinical decision that affected the administration of VTE prophylaxis is a major strength of this work. In all, our data raise several questions about the accuracy of PSI-12 in identifying preventable postoperative VTE, especially as it is utilized as a marker for pay-for-performance measures in addition to identifying further areas for research and improvement.

Our results align with previous studies that suggest that PSI-12 is an inaccurate measure of performance quality. A study by Bilimoria et al. in 2013 noted that surveillance biases associated with PSI-12, showing that hospitals with higher compliance to appropriate VTE prophylaxis paradoxically had worse outcomes.¹⁰ Furthermore, Blay et al. recently published a study evaluating PSI-90 scores for hospitals with and without the VTE measure included. Their results indicated that larger hospitals (teaching hospitals, level I trauma centers, etc) caring for sicker patients were noted to improve by 8%-25% when the VTE measure was removed.¹¹ Similarly, our data indicate that the PSI-12 may not be an accurate measure of quality performance, as only 6% of cases were noted to be deficient in appropriate guideline-directed prophylaxis. Even when accounting for the refusal of doses of pharmacologic prophylaxis, this figure only increased to 11%.

Procedures included in the current PSI-12 algorithm also vary in the risk they pose to developing VTE. For example, the current version of PSI-12 includes surgical Medicare Severity Diagnosis Related Groups (MS-DRGs) for procedures with a high risk of VTE such as orthopedic, abdominal, or thoracic procedures.^14,15, However, it is worth noting that procedures such as tracheostomy and ocular surgery are also included.¹⁴ The variation in risk for development of VTE is reflected in the Caprini score, a perioperative risk stratification tool. These latter procedures only contribute one point, whereas trauma would add five points each to the total score, and make the patient a high VTE risk from the procedure alone.¹⁶ With regard to PSI-12, in theory, scores could vary significantly between centers even if the quality of care is the same, based on the volume and risk of procedures performed.

While most of our cases of VTE occurred within higher risk surgical subspecialties, 15 (10%) of our cases were within the clinical specialties of interventional radiology, cardiology, gastroenterology, and bone marrow transplant. One procedure of notable conflict includes bronchoalveolar lavage (BAL), which is included as an operating room procedure per the current version of PSI-12 software,¹⁷ but is no longer recognized by CMS as a surgical MS-DRG for reimbursement. The PSI-12 observed-to-expected rates take the DRG and comorbidity codes into consideration, but which DRG is selected does not always reflect the procedure type or the risk of VTE associated with the procedure.

Regarding the type of VTE event, our data revealed that PE was the predominant event, accounting for 62% of cases. This is different compared with other studies, such as the population-based studies in which PE and DVT account for approximately 40%-42% of events, respectively, and approximately 15% with concurrent PE and DVT. ^9,18 Borzecki et al. however, noted similar rates to our study, with 55% as PE only, 38% as DVT only, and 8% had both PE and DVT. This study also found a positive correlation between PSI-12 rates and VTE imaging rates, such that if more CT scans were completed, more PEs could be found.¹⁹ Our data identified 16 of the 97 cases of PE were subsegmental PE, a subset of VTE whose clinical significance has been questioned.²⁰ Excluding these cases brings the percentage closer to 52%. Also, screening ultrasounds are not performed at our institution. Asymptomatic or minimally symptomatic DVTs could be missed.

Further, this study also highlights that the reliability of PSI-12 is dependent on accurate documentation and coding. This is most evident when reviewing studies that evaluated the positive predictive value (PPV) of this measure.^9,12,21 A study of 28 Veteran’s Affairs hospitals from 2003 to 2007 used the AHRQ version 3 software to assess the PPV of PSI-12. Out of the 112 cases flagged by the AHRQ PSI software, only 48 were true events of postoperative DVT, yielding a PPV of only 43%. False-positive results were primarily patients with VTE present on admission and cases that were diagnosed after admission but prior to the index procedure. They also noted that coding inaccuracies were present in 38% of cases.⁹ Similarly, Henderson et al. conducted a retrospective review of 112 postsurgical discharges noting a PPV of 54%, with most false-positives resulting from superficial clots identified by PSI-12 related to coding ambiguity.¹² Our data similarly showed false-positive results as seven cases were excluded based on chart review and documentation correction, and 4.5% were preprocedural. However, there is some discordance with previous studies, as our data yields a PPV of 91% for VTE when accounting for preprocedural events as well as those excluded based on chart review. One explanation is an improvement in documentation strategies and coding, as our institution has adopted strategies to review cases and clarify documentation prior to billing to improve accuracy, and inclusion of a checkbox to indicate that a diagnosis was present on admission. Another possibility is improved accuracy with ICD-10 coding, as shown in a paper by Quan et al. that found a 79% PPV of PSI-12, which improved to 89% when cases present on admission were excluded.²¹ Lastly, we used newer versions of the AHRQ software, which could have improved the accuracy of detection. Despite these improvements in PSI-12 identifying true cases of postoperative VTE, as our data show, there is still much to be desired in terms of identifying issues with the quality of care and inclusion of PSI-12 in pay-for-performance.

Our study has several limitations. As a retrospective review, a major limiting factor is that data obtained are subject to the accuracy of documentation within the provider and nursing notes. As such, we focused on broad topics such as VTE events, the type of VTE event, and timing of the event in relation to admission and procedure. We actively review cases at discharge prior to billing to correct documentation if required. However, a prolonged hospital stay could lead to a review of the case weeks to months later. A real-time alert for a VTE event in a patient with surgery could improve documentation.

Further, although mechanical only prophylaxis was present on chart review, it is impossible to know both the rate of compliance with this method and whether it contributed to some events. Lastly, our study evaluated primarily the PPV of PSI-12. We did not directly evaluate the negative predictive value of PSI-12, which could mean there are cases of preventable VTE that are missed entirely.

Despite these limitations, our goals of evaluating the validity of PSI-12 and identifying areas for improved measurement and techniques for DVT prophylaxis were met. As previously discussed, our data suggest that PSI-12 has several limitations in identifying quality of care issues with the prevention of DVT. For example, PSI-12 included many cases in which pharmacologic prophylaxis was appropriately not given. Approximately 50% of cases occurred in patients who were thrombocytopenic (platelets < 100,000), and 29% of events occurred in trauma patients. In addition, 33% of cases identified were within three days of the procedure, which, in cases of high-bleeding risk procedures, is an appropriate time to refrain from pharmacologic anticoagulation.¹³ In cases such as these, it may be worth evaluating alternative forms of mechanical only prophylaxis, or other strategies to mitigate this risk.

This study also highlights areas for further research. Many of the VTE events in this study occurred while patients were treated with appropriate prophylaxis, as other studies have also shown.^22,23 Gangireddy et al. looked at demographic and clinical information surrounding postoperative VTE and found several other risk factors that incur a higher risk of VTE, including steroid use, infections, and myocardial infarction.²⁴ Perhaps future studies could target these groups as potential points for increased prophylactic strategies. As noted by Lau et al. appropriate VTE prophylaxis involves risk stratification, ordering the appropriate prophylaxis by clinicians, patient acceptance of prescribed therapy, and nursing administration of prescribed therapy.²⁵ As exemplified by our data, despite appropriate prophylaxis being ordered, eight events were associated solely with the refusal of pharmacologic prophylaxis. An ideal VTE metric would identify patients with a VTE who had a defective VTE prophylactic process,²⁵ but the cost associated with manual data abstraction may limit the inclusion of a process metric into pay-for-performance. Additional research also needs to identify whether modifications to PSI-12 can improve its accuracy and predictive value, such as the exclusion of VTE prior to a procedure, modifications to what counts as a procedure, or utilizing markers to assess adherence to VTE prophylactic rates. These points are especially important to consider if PSI-12 is to remain as one of the key factors in pay-for-performance outcome measures. Lastly, understanding the effectiveness and patient adherence to oral VTE prophylactic regimens could also decrease the rates of refusal of VTE prophylaxis.

Overall, VTE remains a large contributor to morbidity and mortality among hospitalized surgical patients. While there is utility in PSI-12 as a screening tool to identify these events and potential areas for improved processes to decrease VTE, its usefulness for detection of true quality issues and its utilization in pay-for-performance is questionable. The universally high rates of VTE prophylaxis question the need for a VTE prophylactic metric. However, if these metrics are going to continue to be used in determining payments, modifications to the current processes and algorithms are needed to improve accuracy in identifying issues in quality of care.