CHICAGO – Risk scoring systems are increasingly being used to assign an indication for therapy and to guide reimbursement, but they are rarely sufficiently calibrated and discriminatory to direct individual patient decisions, according to Dr. Anelechi Anyanwu.

"Risk models are designed as tools to define risks of a population and not risks in the individual patient," Dr. Anyanwu said at the Heart Valve Summit 2011.

Dr. Anyanwu, a cardiothoracic surgeon at Mount Sinai Medical Center in New York City, ran through a checklist of what’s missing in today’s risk scoring systems, starting with the absence of all relevant risk factors. Models can adjust only for known risk factors or things that can be measured easily and objectively, but don’t adjust for unproven, unknown, or rare risk factors. "The reality is that most of the variance in outcome is not explained," he said.

Even such known and probable risk factors as poor lung reserve, unclampable aorta, cirrhosis, frailty, radiation, and socioeconomic status are not included in the scoring systems, but almost certainly influence outcome. A study of 230 patients undergoing cardiac surgery after thoracic radiation found worse in-hospital mortality, respiratory complications, and 4-year survival among those receiving extensive radiation vs. variable or tangential radiation – even though the extensively irradiated group was the youngest at 51 years vs. 64 years and 72 years, respectively (J. Thorac. Cardiovasc. Surg. 2007;133:404-13).

Differential Effect of Risk

Risk systems also don’t allow for the fact that risk factors can vary, or their relevance can disappear over time like the once-worrisome variable of 90% stenosis of the left main artery during primary isolated coronary artery bypass grafting (CABG), Dr. Anyanwu said.

The effect of the same risk fact can also differ in various patient subsets. For example, the EuroSCORE identifies a creatinine level of more than 200 mmol/L as a risk factor, but assumes similar risk for the subsets of creatinine clearance 30 mL/min, creatinine clearance 50 mL/min, and dialysis patients.

"Risk systems are imprecise and need further development to allow for more robust prediction."

Risk systems also assume equal weighting in all geographic regions, he said. The EuroSCORE Study Group reported that the predictive value of its scoring system among the six largest national samples in its database varied from "excellent" in Finland (area under the ROC curve 0.87) to "good" in Spain (AUC 0.74) (Eur. J. Cardiothorac. Surg. 2000;18:27-30).

Certainty

Scoring systems can never be 100% specific, Dr. Anyanwu reminded the audience. Quite often the probability range is broad, with imprecision greater if subjective or poorly defined predictor variables have been included in the model.

A high severity score cannot indicate absolute poor risk, and low severity scores cannot guarantee against poor outcome. "At the end of the day, outcome is always uncertain," he said.

Gaming the System

Patient selection is a powerful risk modification tool that cannot be captured by risk models. The more selective a center or surgeon is, the less the effect a specific risk factor will have, Dr. Anyanwu said. A low ejection fraction of less than 30% denotes poor prognosis in aortic valve replacement (AVR), and is accounted for in all risk systems. But not all low ejection fractions are the same, and surgeons can game the system by cherry-picking the better ones, he said.

He cited a study among 81 consecutive patients with symptomatic, calcified low-flow/low-gradient aortic stenosis that reported an operative mortality of 67% in the subgroup of patients with a mean pressure gradient of 20 mm Hg or less, compared with 16% in patients with an mean pressure gradient of more than 20 mm Hg. Moreover, if a surgeon opted not to perform CABG at the time of the AVR, operative mortality was just 10%, compared with 53% for patients in whom CABG was performed at the time of AVR (J. Am. Coll. Cardiol. 2009;53:1865-73).

Other Considerations

Numerous systems show varied outcome by different geographic regions, centers, and surgeons, and by surgical volume. In one study, the risk-adjusted odds of death among patients undergoing first-time elective surgery for mitral regurgitation was 0.71 at centers with an annual volume of 36-70 mitral operations, 0.74 at centers performing 71-140 mitral operations, and just 0.48 at the highest volume centers doing more than 140 mitral operations per year (Circulation 2007;115:881-7).

Despite such variations, risk scores will predict the same outcome for patient groups that clearly have different outcomes in practice, Dr. Anyanwu said at the meeting, sponsored by the American Association for Thoracic Surgery, American College of Cardiology Foundation, and The Society of Thoracic Surgeons.

He suggested that a good risk system has to be recalibrated frequently, and that further study is needed to allow for incorporation of nonpatient factors in risk prediction.

"Risk systems are imprecise and need further development to allow for more robust prediction," he concluded.

Dr. Anyanwu reported no relevant conflicts of interest.

Risk Assessment Systems Found Flawed, Imprecise