Validating an instrument for selecting interventions to change physician practice patterns
A Michigan Consortium for Family Practice Research study
- OBJECTIVES: The goal of this study was to develop a psychometric instrument that classified physicians’ response styles to new information as seekers, receptives, traditionalists, or pragmatists. This classification was based on specific combinations of 3 scales: (a) belief in evidence vs experience as the basis of knowledge, (b) willingness to diverge from common or previous practice, and (c) sensitivity to pragmatic concerns of practice. The instrument will help focus efforts to change practice more accurately.
- STUDY DESIGN: This was a cross-sectional study of physician responses to a psychometric instrument. Paper-and-pencil survey forms were distributed to 3 waves of physicians, with revision for improved internal consistency at each iteration.
- POPULATION: Participants were 1393 primary care physicians at continuing education events in the Midwest or at primary care clinic sites in the Veterans’ Health Administration system.
- OUTCOMES MEASURED: Internal consistency was measured by factor analysis with orthogonal rotation and Cronbach’s alpha.
- RESULTS: A total of 1287 usable instruments were returned (106, 1120, and 61 in the 3 iterations, respectively), representing approximately three fourths of distributed forms. Final scale internal consistencies were a = 0.79, b = 0.74, and c = 0.68. The patterns of scores on the 3 scales were consistent with the predictions of the theoretical scheme of physician types. The “seeker” type was the rarest, at fewer than 3%.
- CONCLUSIONS: It is possible to reliably classify physicians into categories that a theoretical framework predicts will respond differently to different interventions for implementing guidelines and translating research findings into practice. The next step is to demonstrate that the classification predicts physician practice behavior.
As we emphasized in our original formulation, our categorizations refer to trait, not state; that is, the categories describe general response tendencies, not moment-to-moment clinical decision making. It is incorrect to say that a physician responds as a seeker in one instance and a pragmatist in another, or that the same person shows traditionalist responses to one topic and receptive responses to another. (Most actual clinical behavior is, of necessity, pragmatic most of the time.)
We hypothesize that these physician response styles represent various combinations of 3 underlying factors:
- Extent to which scientific evidence, rather than clinical experience and authority, is perceived as the best source of knowledge about good practice (evidence vs experience).
- Degree of comfort with clinical practices that are out of step with the local community’s practices or the recommendations of leaders (nonconformity).
- Importance attached to managing workload and patient flow while maintaining general patient satisfaction (practicality).
Not all possible combinations of the 3 factors exist, and some combinations are behaviorally indistinguishable—that is, they produce the same response style. The manner in which these 3 factors define the 4 types of physicians is shown in Table 1. In this paper we report the results of 3 iterations in the development of a psychometric instrument to measure these factors.
TABLE 1
Hypothesized factor loading by physician type
| Physician type | Evidence vs experience | Nonconformity | Practicality |
|---|---|---|---|
| Seekers | Extreme evidence end | High | Not high |
| Receptives | Toward evidence end | Moderate | Not high |
| Traditionalists | Toward experience end | Variable | Not high |
| Pragmatists | Variable | Variable | High |
Methods
To test the hypothesized relationship between physician category and response to practice change interventions, we needed to develop an instrument for assessing physicians on the underlying 3 attributes so that, based on those attributes, we could subsequently place them in the 4 information response categories. We created several questions addressing each of our hypothesized factors and refined them for clarity. The question pool was further refined in consultation with active practitioners serving on commissions and committees of the American Academy of Family Practice, who represented a variety of nonacademic perspectives on clinical practice and learning. An 18-item psychometric instrument was prepared and pilot tested on a convenience sample of 112 family physicians in Iowa and Michigan who were participating in other research projects.
The results of that pilot test were used to prepare a second version, which was tested with 328 physicians at a regional CME conference and 889 physicians with the national Veterans Health Administration system for a total of 1217. The sample comprised 234 family physicians; 848 internists; 29 obstetrician/gynecologists; 27 general practitioners; 24 emergency physicians; and a small number of general surgeons, pediatricians, psychiatrists, and other specialists. The results from the second version guided the preparation of the third (Figure), which was tested on a sample of 64 family physicians at 2 CME events.
Because of the free-choice manner in which the instruments were distributed, it was not possible to calculate an exact response rate; however, the total number of participants equaled slightly more than 75% of the total number of instruments distributed.
To refine the instrument at each iteration, we began with a factor analysis using the principalcomponents method and orthogonal varimax rotation. The eigenvalues from the factor analysis were used to determine the number of factors in the optimum solution. The instrument’s questions were assigned to these factors based on the factor on which they loaded most heavily in the rotated solution. Cronbach α was calculated for each factor scale. At each iteration, questions loading less than 0.35 on all factors in the rotated varimax solution were dropped. Questions loading on 2 factors were revised for clearer wording in the subsequent draft. New questions were added to factor scales on which too few questions were loading. All analyses were performed using Intercooled Stata 7.0 statistical software (Stata Corp, College Station, TX) on a Linux workstation.
The results of the factor analysis were compared with the theory after the second and third iterations. Physicians were scaled on the 3 factors by summing the responses to the items of each scale, with strongly agree (SA) = 5 and strongly disagree (SD) = 1 (reversing the numbers for items phrased in the opposite manner). Normalization (adjusting scores to account for scales that included more items, resulting in larger maximum scores) was considered but rejected, because normalized scores proved more confusing than unequal scales when the results were presented to audiences.
We used the scale scores to classify the physicians into the 4 types (seeker, receptive, traditionalist, and pragmatist). We performed the factor analyses and interpretations as described in Tables2, 3, and 4, then translated the hypothesized relationships in Table 1 into specific calculations as shown in Tables 5 and 6 (for the second and third iterations, respectively). The chosen cutoff points were necessarily somewhat arbitrary; to prove them optimal, we must complete an external validation study of the physicians’ behavior vs their scale scores, which is now underway. The current data address the instrument’s development and internal consistency.
