The Design and Evaluation of the Comprehensive Hospitalist Assessment and Mentorship with Portfolios (CHAMP) Ultrasound Program

Statistical Analysis

Demographics of the included hospitalist population and pre and post 3-day assessments, including knowledge score, skills score, confidence in ultrasound use, confidence in volume management, and quality of physical exam, were summarized. Values for all assessment variables are presented as percentages. Confidence scores were reported as a percentage of the Likert scale (eg, 4/5 was reported as 80%). Skills and written examinations were expressed as percentages of items correct. Data were reported as median and interquartile range or means and standard deviation based on variable distributions. Differences between pre- and postvalues for 3-day course variables were assessed by using 2-sample paired Wilcoxon signed rank tests with a 95% confidence level.

For the subset of hospitalists who also completed the 1-day course, pre and post 1-day course assessments, including skills score, confidence in ultrasound use, confidence in volume management, and quality of physical exam, were summarized. Differences between pre- and postvalues for 1-day assessment variables were assessed by using 2-sample paired Wilcoxon signed rank tests with a 95% confidence level.

For hospitalists who completed both the 3-day and 1-day courses, the change in course assessments, including skills score, confidence in ultrasound use, confidence in volume management, and quality of physical exam, was assessed by summarizing the change from post 3-day metrics to pre 1-day metrics (Table 2). The differences between these 2 assessments were evaluated by using 2-sample paired Wilcoxon signed rank tests with a 95% confidence level. Changes in skills score from post 3-day assessment to pre 1-day assessment were also compared for hospitalists completing any of the portfolio and those completing none, and for hospitalists attending any monthly scanning sessions and those who did not attend any, by using analysis of variance and Scheffe tests.

Multiple linear regression was performed with the change in skills assessment score from postcompletion of the 3-day course to precompletion of the 1-day course as the dependent variable. Hospitalists were split into 2 age groups (30-39 and 40-49) for the purpose of this analysis. The percent of monthly scanning sessions attended, age category, timing of 1-day course, and percent portfolio were assessed as possible predictors of the skills score by using simple linear regression with a P = .05 cutoff. A final model was chosen based on predictors significant in simple linear regression and included the percent of the portfolio completed and attendance of monthly scanning sessions.

RESULTS

Demographics

Of the 56 3-day course participants, 53 had complete data (Table 3). Three participants with incomplete data completed most of the course but left prior to postcourse assessments and were excluded from the analysis. Twenty-three hospitalists also completed the 1-day in-person course. Seven hospitalists completed the 1-day course 3 months after the initial course, 8 completed it at 6 months, and 8 completed it at 12 months. Completed portfolios required 164 approved video images. Fifteen of the 23 hospitalists at the 1-day course have started and are working towards completion of the online portfolio, while 9 of the 23 participated in the monthly scanning sessions.

3-Day In-Person Course

For the 53 hospitalists who completed skills-based assessments, performance increased significantly after the 3-day course. Knowledge scores also increased significantly from preassessment to postassessment. Self-reported confidence ratings for ultrasound use, confidence in volume management, and quality of physical exam all increased significantly from preassessment to postassessment (Table 2).

Refresher Training: 1-Day In-Person Course

Because the refresher training was encouraged but not required, only 25 of 53 hospitalists, 23 with complete data, completed the 1-day course. For the 23 hospitalists who completed skills-based assessments before and after the 1-day course, mean skills scores increased significantly (Table 2). Self-reported confidence ratings for ultrasound use, confidence in volume management, and quality of physical exam all increased significantly from preassessment to postassessment (Table 2).

Monthly Scanning Sessions and Portfolio Development

The skills retention from initial course to refresher course by portfolio completion and monthly scanning sessions is shown in Table 2. Multiple regression analysis showed that for every 10% increase in the percent of monthly sessions attended, the mean change in skills score was 3.7% (P = .017), and for every 10% increase in the percent of portfolio completed, the mean change in skills score was 2.5% (P = .04), showing that both monthly scanning session attendance and portfolio completion are significantly predictive of skills retention over time.

Final Assessments

Four providers met mastery at initial attempt. No providers to date have needed remediation. Many others are going through different stages of the process and are expected to attain mastery in a short period of time.