Validating an instrument for selecting interventions to change physician practice patterns
A Michigan Consortium for Family Practice Research study
- OBJECTIVES: The goal of this study was to develop a psychometric instrument that classified physicians’ response styles to new information as seekers, receptives, traditionalists, or pragmatists. This classification was based on specific combinations of 3 scales: (a) belief in evidence vs experience as the basis of knowledge, (b) willingness to diverge from common or previous practice, and (c) sensitivity to pragmatic concerns of practice. The instrument will help focus efforts to change practice more accurately.
- STUDY DESIGN: This was a cross-sectional study of physician responses to a psychometric instrument. Paper-and-pencil survey forms were distributed to 3 waves of physicians, with revision for improved internal consistency at each iteration.
- POPULATION: Participants were 1393 primary care physicians at continuing education events in the Midwest or at primary care clinic sites in the Veterans’ Health Administration system.
- OUTCOMES MEASURED: Internal consistency was measured by factor analysis with orthogonal rotation and Cronbach’s alpha.
- RESULTS: A total of 1287 usable instruments were returned (106, 1120, and 61 in the 3 iterations, respectively), representing approximately three fourths of distributed forms. Final scale internal consistencies were a = 0.79, b = 0.74, and c = 0.68. The patterns of scores on the 3 scales were consistent with the predictions of the theoretical scheme of physician types. The “seeker” type was the rarest, at fewer than 3%.
- CONCLUSIONS: It is possible to reliably classify physicians into categories that a theoretical framework predicts will respond differently to different interventions for implementing guidelines and translating research findings into practice. The next step is to demonstrate that the classification predicts physician practice behavior.
TABLE 2
Factor analysis solutions
| Eigenvalues by number of factors in solution | ||||
|---|---|---|---|---|
| Iteration | 1 | 2 | 3 | 4 |
| 1 | 2.88 | 1.67 | 1.44 | 1.23 |
| 2 | 1.95 | 1.20 | 0.809 | 0.387 |
| 3 | 3.35 | 2.31 | 1.60 | 0.821 |
TABLE 3
Scale interpretations
| Scale | Interpretation | Questions (on iteration 3)* |
|---|---|---|
| 1 | Evidence–experience | 1, 3, 9, 12, 16, 17 |
| 2 | Nonconformity | 2, 5, 7, 11, 13, 15 |
| 3 | Practicality | 4, 6, 8, 10, 14 |
| *See Figure. | ||
TABLE 4
Scale internal consistencies
| Cronbach α at each iteration | |||
|---|---|---|---|
| Iteration | Evidence-experience | Nonconformity | Practicality |
| 1 | 0.63 | 0.61 | 0.54 |
| 2 | 0.70 | 0.59 | 0.48 |
| 3 | 0.79 | 0.74 | 0.68 |
TABLE 5
Scale scores by physician type, second iteration
| Physician type | Evidence vs experience (range, 5–25) | Nonconformity (range, 4–20) | Practicality (range, 4–20) |
|---|---|---|---|
| Seekers | Extreme evidence end: ≥20 | High: >12 | Not high: ≤14 |
| Receptives | Toward evidence end: ≥15 | Moderate: ≤12 | Not high: ≤14 |
| Traditionalists | Toward experience end: <15 | Variable | Not high: ≤14 |
| Pragmatists | Variable | Variable | High: >14 |
TABLE 6
Scale scores by physician type, third iteration (depicted in the Figure)
| Physician type | Evidence vs experience (range, 6–30) | Nonconformity (range, 6–30) | Practicality (range, 5–25) |
|---|---|---|---|
| Seekers | Extreme evidence end: ≥22 | High: >18 | Not high: ≤14 |
| Receptives | Toward evidence end: ≥18 | Moderate: ≤18 | Not high: ≤14 |
| Traditionalists | Toward experience end: <18 | Variable | Not high: ≤14 |
| Pragmatists | Variable | Variable | High: >14 |
FIGURE Psychometric Instrument
Results
For the first, second, and third iterations, we received 106, 1120, and 61 instruments respectively that were completed in usable form. At every stage of the instrument’s development, factor analysis showed that a 3-factor model fit best. The eigenvalues declined rapidly when there were more than 3 factors (Table 2), showing that additional factors would not improve the solution.
Orthogonal rotation and interpretation of the questions making up each factor produced 3 psychologically meaningful scales (Table 3) corresponding closely to our theoretical model; the same 3 scales emerged at each iteration. The scales are named similarly to the theory above: evidence–experience, practicality, and nonconformity. The Cronbach α for each scale at each iteration is presented in Table 4.
Using the above-described classification scheme (with specific cutoffs detailed in Tables 5 and 6), the 1181 physicians who completed the instrument in the second and third iterations were classified as follows: 2.5% seekers; 57.0% receptives; 12.6% traditionalists; and 27.9% pragmatists. Different cutoff values would yield somewhat different percentages, but seekers are very few using any reasonable value.
Discussion
These results are consistent with the theoretical construct of 3 factors underlying our physician classification scheme and demonstrate that those factors can be measured on scales with reasonable internal consistency. The data are consistent with the theory on which the instrument was developed. Not all possible combinations of the 3 factors exist, which is consistent with the 4-types theory depicted in Table 1. For example, there should be no physicians who are strongly evidence-based and strongly conformist, and that combination does not occur. However, there are physicians who are strongly evidence-based and strongly nonconformist (the seekers). Few physicians selected either extreme for any factor, but with the exception of nonconformity, a broad range existed across all of the factors.
These findings show that physicians differ in their attitudes toward new information about the effectiveness or appropriateness of clinical strategies, and that those differences are measurable and quantifiable. Quantifying those differences was a major step forward in testing our theoretical framework for selecting effective practice change strategies.
The next step is to demonstrate external validity by showing that differences in physician behavior are consistent with demonstrable differences in attitudes. Such a study is underway at this writing. A trial of practice change interventions guided by the categorization scheme should be carried out subsequently.
The categories we propose do not reflect bimodal distributions of attributes; physicians are distributed relatively uniformly all along the 3 scales. The categories are useful descriptors, not absolute pigeonholes.
The results suggest to us that there is fertile ground for applied psychometrics and cognitive science research related to changing clinical practices. Such work may help illuminate the murky results of practice change intervention and guideline implementation studies to date. Further cognitive research about our own theoretical framework is likely to identify factors and complexities that we have not yet addressed.
· Acknowledgments ·
The authors thank Mark Ebell, MD, MS for his assistance in revising the instrument; Judith Zemencuk and Bonnie Boots-Miller of the Ann Arbor Veterans Administration Health Services Research and Development offices for their assistance in distributing the instrument to and collecting data from Veterans Administration physicians; Janice Klos for her help in gathering data from Michigan Academy of Family Practice member physicians; Van Harrison, PhD and his staff for their help in enlisting the participation of physicians at Michigan CME events; and of course, the Veterans Administration, Michigan Academy of Family Practice, and Michigan physicians who graciously completed instruments for this project.
