Inside the operating room—balancing the risks and benefts of new surgical procedures

RANDOMIZED TRIALS VS CASE SERIES: A TIME AND PLACE FOR EACH

Even as we recognize these challenges specific to surgical evaluation, we are still left with the task of determining when a randomized controlled trial is appropriate and when a case-control series may suffice.

There are three broad sets of circumstances in which a randomized trial is essential:

• For preventive procedures, ie, when the operation is done to reduce the potential for a future adverse event. An example would be evaluating carotid endarterectomy to reduce the potential for stroke in asymptomatic patients with 60% or greater stenosis. Only a randomized trial could have shown a difference in favor of endarterectomy over aspirin plus best medical therapy.
• To compare a procedure with alternative medical or surgical interventions. I would argue that laparoscopic surgery should have been introduced with randomized trials, as it begs one to suspend judgment and accept that small incisions are invariably and de facto better than a large incision.
• For trials in oncology, where the outcome depends on long-term results, such as survival or time to recurrence. Examples would include comparisons of surgery alone versus surgery plus chemotherapy for prevention of cancer recurrence.

Similarly, there are several scenarios in which a case-control series is appropriate and adequate:

• When no alternative therapy exists. Falling into this category, in my view, are lung transplant, which we introduced successfully at the University of Toronto in 1983, and lung volume reduction surgery, which we introduced in 1993.
• When the natural history of the condition is well documented and the impact of the intervention is obvious.
• When the magnitude of the procedure’s effect is measurable, significant, and expected.

RANDOMIZED TRIALS IN SURGERY

Advantages of randomized trials

Randomized clinical trials confer a number of advantages. They eliminate bias. They ensure a balance between treatment groups in terms of known or unknown prognostic factors. And, importantly, they have a major impact on payors.

A tale of two Medicare payment decisions

The impact of clinical trials on payors is exemplified by the contrasting stories of two procedures: transmyocardial laser revascularization and lung volume reduction surgery.

Transmyocardial laser revascularization (TMR) involves the creation of channels in the myocardium with a laser to relieve angina. Although TMR is a dubious intervention with no physiologic rationale (similar to internal mammary artery ligation for angina5) and no proven improvement in life expectancy (only a reduction in pain), it was approved for reimbursement by Medicare because it was investigated in a randomized trial.⁶ However, the “randomized trial” was not truly a randomized investigation because the control patients received only medical therapy and did not go to the operating room to receive a sham operation.⁶ Despite this flaw, the perceived authority of the trial was sufficient to influence Medicare.

In contrast, Medicare refused to pay for lung volume reduction surgery until it was subjected to a randomized trial, despite the fact that the procedure had produced tremendous benefit in hundreds of patients at multiple centers who otherwise could not have achieved such benefit. Only after $50 million was spent on a randomized controlled trial, the National Emphysema Treatment Trial (NETT),^7–9 did Medicare agree to pay for lung volume reduction surgery. The trial showed that over 5 years, the procedure was associated with significant improvements in life expectancy, exercise tolerance, and quality of life, but the study took 8 years to conduct and by then it was a bit too late, as detailed in the following section.

NETT: A case study in how a trial can be counterproductive

Lung volume reduction surgery is an operation based on the recognition that the crippling effects of emphysema are hyperinflation of the chest, flattening of the diaphragm, and inability to move air in and out of the chest. The notion that the chest can be reconfigured in the patient with emphysema by removing the distending overinflated emphysema led us to develop the volume reduction operation.

The NETT was initiated by Medicare, and the protocol denied compassionate crossover of patients.⁷ In an attempt to establish clinical equipoise, surgeons who participated were not allowed to perform any volume reduction operations on non-Medicare patients or on Medicare patients not enrolled in the trial. After 2 years of slow patient enrollment, the clinical trial committee, in an effort to increase enrollment, eliminated the original entrance criteria specifying certain degrees of hyperinflation and diffusing capacity. An excess of mortality was discovered 2 years later in a subgroup randomized to volume reduction surgery;⁸ not surprisingly, further analysis showed that the excess mortality was largely confined to patients who would have been excluded based on the original entrance criteria. This is a matter of public record but was never acknowledged in published reports of the trial. Final 5-year NETT results showed that in patients with upper lobe emphysema, lung volume reduction surgery improved survival, increased exercise capacity, and improved quality of life.⁹ By the trial’s completion, however, the procedure’s reputation had been tarnished irreparably by bad publicity from the deaths attributable to the misguided changes to the original eligibility criteria.

Disadvantages of randomized trials

The NETT exemplifies many of the drawbacks of randomized trials in surgery, particularly the need to wait long periods while they are being conducted. During the 8 years in which the NETT was ongoing, the number of lung volume reduction operations declined, with the typical center performing fewer than 6 cases per year, on average. That limitation is certainly not conducive to the development of a new procedure for a disabling condition in patients with no ready alternative.

Other disadvantages of randomized trials in surgery are their considerable expense and the fact that they often are not generalizable and often are not appropriate. Moreover, when they are flawed, randomized trials propagate, sometimes for decades, misleading information that is nonetheless considered “authoritative.” For instance, lung cancer kills more men and women in the United States than the next three cancers combined, yet, on the basis of a flawed randomized trial,¹⁰ the American Cancer Society advises smokers to wait for symptoms before undergoing chest radiography, instead of recommending annual screening chest radiography. This is a major reason why two-thirds of lung cancer cases are discovered too late to save the patient.

‘Better to know nothing than to know what ain’t so’

Indeed, this potential for randomized clinical trials, when flawed, to propagate misleading information makes the perceived authoritativeness of randomized trials both an advantage and a disadvantage. As Berger and colleagues noted a few years ago, overuse of randomized trials for evaluating emerging operations could have led to the demise of heart transplantation, mechanical circulatory assist devices, cardiac valve procedures, coronary bypass grafting, and repair of congenital lesions.¹¹

For this reason, one of our responsibilities when reading the literature and conducting studies is not just to answer unanswered questions but to question unquestioned answers. As 19th-century humorist Josh Billings put it, “It’s better to know nothing than to know what ain’t so.”

A PERSONAL PERSPECTIVE

In my view, health care providers should restrict the application of new procedures to a limited number of centers of excellence that have appropriate resources and experience. Those centers should be required to document and report specified information regarding morbidity, mortality, and objective measures of outcome; if they do not comply, they should lose the privilege of doing such research. The data should be reviewed by an independent, nongovernmental scientific panel. In this way, the procedure can be offered to appropriate patients, insurers and patients can be protected against abuse, and the necessary data can be collected for objective analysis.