POINT: SCIP is both efficacious and effective.

The Surgical Care Improvement Project (SCIP) was a national campaign that set out to reduce surgical mortality and morbidity by 25% by 2010 through recommendations in targeted areas: wound infections, perioperative MIs, and venous thromboembolism. The recommendations have become pay-for-performance measures. There are seven in the area of infectious disease to reduce surgical site infections. There is one measure for reducing perioperative MI: Continue beta-blockers (for patients who are on them) in the perioperative period. For venous thromboembolism prevention, give prophylaxis within 24 hours before to 24 hours after surgery.

It’s key to understand the difference between efficacy and effectiveness. I think we would all agree that the SCIP measures have efficacy. Efficacy trials determine whether an intervention produces the expected result under ideal circumstances. Effectiveness trials measure the degree of beneficial effect under "real world" clinical conditions. The problem with effectiveness trials is that those real-world conditions may change the effect, or they might just change the ability to measure the effect.

I believe that the SCIP measures have proven efficacy because they all are based upon randomized controlled trials that were identified by systematic reviews amenable to meta-analysis. All of these measures are Level 1 recommendations, based on the highest forms of evidence. The studies that Dr. Barash uses to criticize SCIP measures are cohort studies. They do not randomize. There may be unknown confounding variables.

There have been effectiveness trials that show that the SCIP measures do work. One showed a 27% decrease in surgical site infections, another showed a 62% decrease in surgical site infections, and a third showed a 39% decrease in surgical site infections.

Perhaps the strongest endorsement of efficacy of the SCIP measures comes from Dr. Kaveh G. Shojania, who has written several reviews of the efficacy of medical interventions. This guy is like Mikey from the old Life cereal commercials ... he hates everything. He said there were 11 patient safety practices rated most highly in terms of strength of the evidence, and 3 are SCIP measures: appropriate use of prophylaxis to prevent venous thromboembolism in patients at risk, use of perioperative beta-blockers in appropriate patients, and appropriate use of antibiotic prophylaxis in surgical patients (AHRQ Publication No. 01-E058).

Several trials published by pretty good researchers in reputable journals show a lack of effectiveness of SCIP measures. Even those researchers admit to the efficacy of SCIP measures. The lead investigator of the best effectiveness trial, a retrospective cohort study, wrote, "There are several explanations as to why we did not observe an association between timely antibiotic administration and surgical site infection (SSI). The first is that timely antibiotic administration does not diminish SSI risk. This is an unlikely interpretation. There are numerous randomized controlled trials and observational studies that demonstrate the efficacy of prophylactic antibiotics in reducing SSI for various surgical procedures" (Ann. Surg. 2011;254:494-9).

A separate retrospective cohort study showed a decrease in surgical site infection only if two or more SCIP recommendations were followed (JAMA 2010;303:2479-85). Shocking – if you give the wrong antibiotic at the right time, it might not work.

Another retrospective cohort study found no association with adjusted complications and SCIP compliance. Hospitals in the lowest compliance group had patients in lower-income ZIP codes and lower unadjusted complication rates. So, poor people go home and don’t come back, perhaps because of payment considerations. The study didn’t have enough patients; it also used measures that don’t apply to SCIP (Arch. Surg. 2010;145:999-1004).

SCIP did not design these measures for pay-for-performance programs. The intent was to decrease perioperative complications by 25% by 2010. When you start changing the baseline with pay-for-performance, it doesn’t work. In a study by Hawkins et al., the authors tested the hypothesis that documented compliance with antibiotic prophylaxis guidelines on a pediatric surgery service does not reflect adherence to guidelines as intended. In a 7-week observational study of elective pediatric surgical cases, adherence was evaluated for appropriate administration, type, timing, weight-based dosing, and redosing of antibiotics. Prophylactic antibiotics were administered appropriately in 141 of 143 cases (99%). Of 100 cases in which antibiotic prophylaxis was indicated, compliance was documented in 100% of cases in the electronic medical record; but only 48% of cases adhered to all five guidelines. Lack of adherence was due primarily to dosing or timing errors.

The SCIP measures, however, are based on best evidence. They are tightly linked with the desired outcomes. They are measurable and effectible, as demonstrated in multiple randomized controlled trials. Studies of effectiveness have had variable results due to methodological flaws.

Dr. Lagasse is a professor of anesthesiology and director of quality management at the Yale School of Medicine, New Haven, Conn. He is on the steering committee for the Surgical Care Improvement Project. He reported having no financial disclosures.

COUNTERPOINT: Studies have not shown effectiveness.

When it was created, SCIP did not reflect reality. SCIP started at the U.S. Department of Veterans Affairs, which conducted a 10-year study. They found a 25% relative risk reduction, but that was only a 0.8% absolute risk reduction for the incidence of complications, a drop from about 3.1% to about 2.3% (Arch. Surg. 2002;137:20-27).

It would be great to have randomized controlled trials on the effectiveness of SCIP, but it’s not happening. We’re going to have to go by high-fidelity observational trials, which according to a number of researchers in the field have the same impact as randomized controlled trials.

One study of 35,543 patients in 44 hospitals found a whopping 27% reduction in surgical site infections, but that was only a 0.6% absolute reduction, from about 2.5% to about 1.9% (Am. J. Surg. 2005;190:9-15). There was no significant difference between groups.

Another study showed improved compliance with SCIP measures, but no change in surgical site infection rate (Dis. Colon Rectum 2010;53:24-30). This is the theme in study after study after study.

A 2008 study enrolled 9,195 patients undergoing colorectal, orthopedic, or vascular surgery and looked at SCIP compliance vs. surgical site infection. The SCIP rate correlated with the hospital case mix. If you look at the SCIP rate in terms of antibiotic timing, SCIP is not significant. The study basically showed that variables other than timely antibiotic administration are affecting surgical site infection rates (J. Am. Coll. Surg. 2008;206:814-19).

Hospital performance on process measures may not be a good marker of surgical site infection or the outcome we’re looking at, according to another study, which reported that unmeasured effects may have a larger impact than the measured effects (Health Serv. Res. 2008;43:1464-84).

There is a randomized controlled trial that randomized patients to strict control with the SCIP measures or routine treatment at the hospital. The SCIP-treated patients had nearly twice the incidence of surgical site infections as the patients receiving standard treatment. The authors concluded that combining each of the SCIP factors into one big category doesn’t necessarily work (Arch. Surg. 2011;146:263-9).

Should we be evaluating outcome measures with performance measures (e.g., percent timely antibiotic administration) to determine whether they work or not? One editorial evaluated eight articles with data on 31,448 patients, looking just at antibiotic administration within 1 hour of surgery, a SCIP measure. It found a higher infection rate if antibiotics were administered within 30 minutes of incision (JAMA 2010;303:2527-2528).

There was no significant difference in another study between standard of care and SCIP for venous thromboembolism (Am. J. Surg. 2012;204:591-97). The authors wrote that there is no convincing evidence that improvements in compliance are associated with better outcomes. We see this time and time again.

When people find that SCIP is not working, they turn to other measures to reduce surgical site infection. The Comprehensive Unit-based Safety Program (CUSP) is targeted at a specific problem that a specific hospital is having in managing infections. It’s not coming from Washington; it’s based at the hospital. One study showed that following CUSP, there was a significant reduction in surgical site infections despite the fact that previous to that there was 95% compliance with SCIP standards (J. Am. Coll. Surg. 2012;215:193-200). SCIP was working, but it wasn’t affecting outcome.

Dr. Lagasse and I interpret one key study very differently. He abstracts a sentence from a Limitations section of the study and makes a sweeping generalization out of context. But the study showed no relationship between facility adherence to SCIP and the surgical site infection rate. The authors concluded, "Policies regarding continued SCIP measurement and reporting should be reassessed" (Ann. Surg. 2011;254:494-99).

The largest SCIP study to date from a single entity involved 32,459 patients in the Veterans Affairs medical system. Overall, antibiotics were administered within 28 minutes of surgical incision. Once they adjusted for confounders, they found no significant relationship between surgical site infection and the SCIP measures (JAMA 2013;148:649-57). No one has proven that giving antibiotics within 60 minutes of surgical incision gives you a lower infection rate.

SCIP measures divert resources and divert clinical care. They obscure the nuances of care. They may harm the hospital and the provider, and they raise unnecessary legal risk if an antibiotic is not given within 60 minutes of incision.

Dr. Barash is a professor of anesthesiology at Yale University, New Haven, Conn. He reported having no financial disclosures.

These are excerpts from a debate at the annual meeting of the American Society of Anesthesiologists.

Point/Counterpoint – Are SCIP measures efficacious?