But how many people died? Health outcomes in perspective
WHY STUDIES LOOK ONLY AT SOME OUTCOMES
There are many reasons why researchers favor examining some outcomes over others, but there is no clear justification for ignoring overall mortality. Overall mortality should routinely be examined in large population studies of diet and supplements and in trials of medications27 and cancer screening.
With regard to large observational studies, it is hard to understand why some would not include survival analyses, unless the results would fail to support the study’s hypothesis. In fact, some studies do report overall survival results,28 but others do not. The omission of overall survival in large data-set research should raise concerns of multiple hypothesis testing and selective reporting. Eating peaches as opposed to grapefruit may not be associated with differences in rates of all-cause mortality, myocardial infarction, pneumonia, or lung cancer, but if you look at 20 different variables, chances are that one will have a P value less than .05, and an investigator might be tempted to report it as statistically significant and even meaningful.
Empirical studies support this claim. One group found that for 80% of ingredients randomly selected from a cookbook, there existed Medline-indexed articles assessing cancer risk, with 65% of studies finding nominally significant differences in the risk of some type of cancer.29
An excess of significant findings such as this argues that significance-chasing and selective reporting are common in this field and has led to calls for methodologic improvements, including routine falsification testing30 and up-front registration of observational studies.31
WHY ALL OUTCOMES MATTER
Healthy people do not care about some outcomes; they care about all outcomes. Some patients may truly have unique priorities (quality of life vs quantity of life), but others may overestimate their risk of death from some causes and underestimate their risk from others, and practitioners have the obligation to counsel them appropriately.
For instance, a patient who watches a brother pass away from pancreatic adenocarcinoma may wish to do everything possible to avoid that illness. But often, as in this case, fear may surpass risk. The patient’s risk of pancreatic cancer is no different than that in the general population: the best data show32 an odds ratio of 1.8, with a confidence interval spanning 1. As such, pancreatic cancer is still not among his five most likely causes of death.
Some patients may care about their bone mineral density or cholesterol level. But again, physicians have an obligation to direct patients’ attention to all of the outcomes that should be of interest to them.
OBJECTIONS TO INCLUDING ALL OUTCOMES
There are important objections to the argument I am presenting here.
First, including all outcomes is expensive. For studies involving retrospective analysis of existing data, looking at overall mortality would not incur additional costs, only an additional analysis. But for prospective trials to have statistical power to detect a difference in overall mortality, larger sample sizes or longer follow-up might be needed—either of which would add to the cost.
In chemoprevention trials, the rate of incident cancer has been called the gold standard end point.33 To design a thrifty chemoprevention study, investigators can either target a broad population and aim for incident malignancy, or target a restricted, high-risk population and aim for overall mortality. The latter is preferable because although it can inform the decisions of only some people, the former cannot inform any people, as was seen with difficulties in interpreting the Prostate Cancer Prevention Trial and trials showing reduced breast cancer incidence from tamoxifen, raloxifene, and exemestane.
In large cancer screening trials, the cost of powering the trial for overall mortality would be greater, and though a carefully selected, high-risk population can be enrolled, historically this has not been popular. In cancer screening, it is a mistake to contrast the costs of trials powered for overall mortality with those of lesser studies examining disease-specific death. Instead, we must consider the larger societal costs incurred by cancer screening that does not truly improve quantity or quality of life.34
The recent reversal of recommendations for prostate-specific antigen testing by the United States Preventive Services Task Force35 suggests that erroneous recommendations, practiced for decades, can cost society hundreds of billions of dollars but fail to improve meaningful outcomes.
The history of medicine is replete with examples of widely recommended practices and interventions that not only failed to improve the outcomes they claimed to improve, but at times increased the rate of all-cause mortality or carried harms that far outweighed benefits.36,37 The costs of conducting research to fully understand all outcomes are only a fraction of the costs of a practice that is widely disseminated.38
A second objection to my analysis is that there is more to life than survival, and outcomes besides overall mortality are important. This is a self-evident truth. That an intervention improves the rate of overall mortality is neither necessary nor sufficient for its recommendation. Practices may improve survival but worsen quality of life to such a degree that they should not be recommended. Conversely, practices that improve quality of life should be endorsed even if they fail to prolong life.
Thus, overall mortality and quality of life must be considered together, but the end points that are favored currently (disease-specific death, incident cancer, diabetes mellitus, myocardial infarction) do not do a good job of capturing either. Disease-specific death is not meaningful to any patient if deaths from other causes are increased so that overall mortality is unchanged. Furthermore, preventing a diagnosis of cancer or diabetes may offer some psychological comfort, but well-crafted quality-of-life instruments are best suited to capture just how great that benefit is and whether it justifies the cost of such interventions, particularly if the rate of survival is unchanged.
Preventing stroke or myocardial infarction is important, but we should be cautious of interpreting data when decreasing the rate of these morbid events does not lead to commensurate improvements in survival. Alternatively, if morbid events are truly avoided but survival analyses are underpowered, quality-of-life measurements should demonstrate the benefit. But the end points currently used capture neither survival nor quality of life in a meaningful way.
WHEN ADVISING HEALTHY PEOPLE
Looking at all outcomes is important when caring for patients who are sick, but even more so for patients who are well. We need to know an intervention has a net benefit before we recommend it to a healthy person. Overall mortality should be reported routinely in this population, particularly in settings where the cost to do so is trivial (ie, in observational studies). Designers of thrifty trials should try to include people at high risk and power the trial for definite end points, rather than being broadly inclusive and reaching disputed conclusions. Research and decision-making should look at all outcomes. Healthy people deserve no less.
