Do HCAHPS doctor communication scores reflect the communication skills of the attending on record? A cautionary tale from a tertiary-care medical service

STATISTICAL ANALYSIS

Physician characteristics were summarized with standard descriptive statistics. Pearson correlation coefficients were computed between HCAHPS and 4HCS scores. All analyses were performed with RStudio (Boston, MA). The Pearson correlation between the averaged HCAHPS and 4HCS scores was also computed. A correlation with a P value less than 0.05 was considered statistically significant. With 28 physicians, the study had a power of 88% to detect a moderate correlation (greater than 0.50) with a 2-sided alpha of 0.05. We also computed the correlations based on the subgroups of data with patients seen by providers for less than 50%, 50% to less than 100%, and 100% of LOS. All analyses were conducted in SAS 9.2 (SAS Institute Inc., Cary, NC).³⁶

RESULTS

There were 31 physicians who met our inclusion criteria. Of 29 volunteers, 28 were observed during 3 separate inpatient encounters and made up the final sample. A total of 1003 HCAHPS survey responses were available for these physicians. Participants were predominantly female (60.7%), with an average age of 39 years. They were in practice for an average of 4 years (12 were in practice more than 5 years), and 9 were observed on a teaching rotation.

The means of the overall 4HCS scores per observation were 17.39 ± 2.33 for the first, 17.00 ± 2.37 for the second, and 17.43 ± 2.36 for third bedside observation. The mean 4HCS scores per observation, broken down by habit, appear in Table 1. The ICC among the repeated scores within the same physician was 0.81. The median number of HCAHPS survey returns was 32 (range = [8, 85], with mean = 35.8, interquartile range = [16, 54]). The median overall HCAHPS doctor communication score was 89.6 (range = 80.9-93.7). Participants scored the highest in the respect subdomain and the lowest in the explain subdomain. Median HCAHPS scores and ranges appear in Table 2.

Because there were no significant associations between 4HCS scores or HCAHPS scores and physician age, sex, years in practice, or teaching site, correlations were not adjusted. Figure 2A and 2B show the association between mean 4HCS scores and HCAHPS scores by physician. There was no significant correlation between overall 4HCS and HCAHPS doctor communication scores (Pearson correlation coefficient 0.098; 95% confidence interval [CI], -0.285, 0.455). The individual habits also were not correlated with overall HCAHPS scores or with their corresponding HCAHPS domain (Table 3).

For 325 patients, 1 hospitalist was present for the entire LOS. In sensitivity analysis limiting observations to these patients (Figure 2C, Figure 2D, Table 3), we found a moderate correlation between habit 3 and the HCAHPS respect score (Pearson correlation coefficient 0.515; 95% CI, 0.176, 0.745; P = 0.005), and a weaker correlation between habit 3 and the HCAHPS overall doctor communication score (0.442; 95% CI, 0.082, 0.7; P = 0.019). There were no other significant correlations between specific habits and HCAHPS scores.

DISCUSSION

In this observational study of hospitalist physicians at a large tertiary care center, we found that communication skills, as measured by the 4HCS, varied substantially among physicians but were highly correlated within patients of the same physician. However, there was virtually no correlation between the attending physician of record’s 4HCS scores and their HCAHPS communication scores. When we limited our analysis to patients who saw only 1 hospitalist throughout their stay, there were moderate correlations between demonstration of empathy and both the HCAHPS respect score and overall doctor communication score. There were no trends across the strata of hospitalist involvement. It is important to note that the addition of even 1 different hospitalist to the LOS removes any association. Habits 1 and 2 are close to significance in the 100% subgroup, with a weak correlation. Interestingly, Habit 4, which focuses on creating a plan with the patient, showed no correlation at all with patients reporting that doctors explained things in language they could understand.

Development and testing of the HCAHPS survey began in 2002, commissioned by CMS and the Agency for Healthcare Research and Quality for the purpose of measuring patient experience in the hospital. The HCAHPS survey was endorsed by the National Quality Forum in 2005, with final approval of the national implementation granted by the Office of Management and Budget later that year. The CMS began implementation of the HCAHPS survey in 2006, with the first required public reporting of all hospitals taking place in March 2008.^37-41 Based on CMS’ value-based purchasing initiative, hospitals with low HCAHPS scores have faced substantial penalties since 2012. Under these circumstances, it is important that the HCAHPS measures what it purports to measure. Because HCAHPS was designed to compare hospitals, testing was limited to assessment of internal reliability, hospital-level reliability, and construct validity. External validation with known measures of physician communication was not performed.⁴¹ Our study appears to be the first to compare HCAHPS scores to directly observed measures of physician communication skills. The lack of association between the 2 should sound a cautionary note to hospitals who seek to tie individual compensation to HCAHPS scores to improve them. In particular, the survey asks for a rating for all the patient’s doctors, not just the primary hospitalist. We found that, for hospital stays with just 1 hospitalist, the HCAHPS score reflected observed expression of empathy, although the correlation was only moderate, and HCAHPS were not correlated with other communication skills. Of all communication skills, empathy may be most important. Almost the entire body of research on physician communication cites empathy as a central skill. Empathy improves patient outcomes^{1-9,13-14,16-18,42} such as adherence to treatment, loyalty, and perception of care; and provider outcomes^10-12,15 such as reduced burnout and a decreased likelihood of malpractice litigation.

It is less clear why other communication skills did not correlate with HCAHPS, but several differences in the measures themselves and how they were obtained might be responsible. It is possible that HCAHPS measures something broader than physician communication. In addition, the 4HCS was developed and normed on outpatient encounters as is true for virtually all doctor-patient coding schemes.⁴³ Little is known about inpatient communication best practices. The timing of HCAHPS may also degrade the relationship between observed and reported communication. The HCAHPS questionnaires, collected after discharge, are retrospective reconstructions that are subject to recall bias and recency effects.^44,45 In contrast, our observations took place in real time and were specific to the face-to-face interactions that take place when physicians engage patients at the bedside. Third, the response rate for HCAHPS surveys is only 30%, leading to potential sample bias.⁴⁶ Respondents represent discharged patients who are willing and able to answer surveys, and may not be representative of all hospitalized patients. Finally, as with all global questions, the meaning any individual patient assigns to terms like “respect” may vary.

Our study has several limitations. The HCAHPS and 4HCS scores were not obtained from the same sample of patients. It is possible that the patients who were observed were not representative of the patients who completed the HCAHPS surveys. In addition, the only type of encounter observed was the initial visit between the hospitalist and the patient, and did not include communication during follow-up visits or on the day of discharge. However, there was a strong ICC among the 4HCS scores, implying that the 4HCS measures an inherent physician skill, which should be consistent across patients and encounters. Coding bias of the habits by a single observer could not be excluded. High intra-class correlation could be due in part to observer preferences for particular communication styles. Our sample included only 28 physicians. Although our study was powered to rule out a moderate correlation between 4HCS scores and HCAHPS scores (Pearson correlation coefficient greater than 0.5), we cannot exclude weaker correlations. Most correlations that we observed were so small that they would not be clinically meaningful, even in a much larger sample.