Process Evaluation of a Tailored Multifaceted Approach to Changing Family Physician Practice Patterns and Improving Preventive Care
The PFs worked with all physicians and allied health staff in the practice. They provided management support to practices and followed a quality improvement framework similar to that proposed by Leininger and coworkers.25 For each practice the PFs were to: (1) present baseline preventive performance rates, (2) facilitate the development of a practice policy for preventive care, (3) assist in setting goals and desirable levels of performance, (4) assist in the development of a written plan for implementing preventive care, (5) assist in the development and adaptation of tools and the strategies to implement the prevention plan, (6) facilitate meetings to assess progress and modify the plan if necessary, and (7) conduct chart audits to measure the impact of the changes made. The intervention period lasted 18 months and ended in December 1998.
The Figure is the program logic model describing each of the 7 intervention component strategies and the associated work activities, outputs, and short-term and long-term objectives associated with each component. It served as a framework for the evaluation of the intervention.26-28 The logic model allowed us to look inside the black box of the intervention29,30 by linking implementation activity to outcomes, and provided a framework to explore which elements worked and why.
Intervention Outcomes
The goal of the intervention was to increase the performance of 8 preventive maneuvers supported by evidence as appropriate and decrease the performance of 5 preventive maneuvers supported by evidence as inappropriate.1 An absolute change over time of 11.51% in preventive care performance in favor of intervention practices was found (F=19.29 [df=1,43], P<.0001. In other words, the intervention practices improved preventive performance by 36% going from 31% of eligible patients having received preventive care to 43% while the control practices remained at 32%.1
Methods
Research Questions
There were 2 objectives to our process evaluation: to document the extent to which the intervention was implemented with fidelity and to gain insight into how facilitation worked to improve preventive performance. The process evaluation was designed to answer questions concerning: (1) the time involved to deliver intervention components, (2) the quality of the delivery of intervention components, and (3) physician satisfaction with the intervention components. Quality was assessed by examining the scope or range of delivery of the intervention components and by analyzing the feedback received from practices on the usefulness of the intervention components.
Setting
The intervention arm of the trial included 22 practices with 54 physicians (Table 1). All health service organizations (HSOs) in Southwestern Ontario were approached to participate in the study. HSOs are primary care practices reimbursed primarily through capitation rather than fee for service. A total of 46 of the 100 primary care practices were recruited (response rate=46%). At follow-up only one intervention practice was lost, because the entire practice had moved. Intervention and control group practices did not differ significantly on any of the measured demographic characteristics (Table 2). Complete details on practice recruitment and attrition rates are published elsewhere.1
The practices covered a geographic area where the greatest distance between any 2 practices was more than 600 kilometers. PFs were assigned practices within a specific region of this geographic area. They arranged times to visit and work with intervention practices and traveled by car between visits to practices. PFs worked independently at their residences and corresponded with the project team through electronic mail regularly and quarterly with scheduled meetings.
Data Collection Tools
Each intervention practice was visited regularly by the same nurse facilitator who documented her activities and progress on 2 structured forms known as the weekly activity sheet and the monthly narrative report. Weekly activity sheets noted the number of hours spent on both on-site and offsite activities. Monthly narrative reports provided detailed information on the number of visits to a practice, the activities within each practice, the outcomes of those activities, the number of participants in meetings, and the plan for the following month. The activities in the narrative reports were summarized by intervention component to provide a cumulative overview of all intervention activity within a practice.
Also during the intervention, semistructured telephone interviews of participating physicians were conducted by 2 physician members of the project team at 6 months and 17 months. Participating physicians were asked what they had been happy and unhappy with and their ideas of improvement. Close-ended questions measured overall satisfaction with the intervention. The interview at 17 months also asked physicians if they would agree to have a nurse facilitator continue to visit their practice if funding were found.
At the end of the intervention, the PFs conducted interviews with each of the physicians identified as the primary contact in the intervention practices to solicit feedback on their experience. Physicians in both the intervention and control arm were sent a questionnaire by mail to report any changes that had taken place in their practice over the preceding 18 months.
