The Potential for Artificial Intelligence Tools in Residency Recruitment

Specific Tools to Consider

There are some tools that are publicly available for programs and applicants to use that rely on AI.

In collaboration with ERAS and the Association of American Medical Colleges, Cortex powered by Thalamus (SJ MedConnect Inc)(https://thalamusgme.com/cortex-application-screening/) offers technology-assisted holistic review of residency and fellowship applications by utilizing natural language processing and optical character recognition to aggregate data from ERAS.

Tools also are being leveraged by applicants to help them find residency programs that fit their criteria, prepare for interviews, and complete portions of the application. Match A Resident (https://www.matcharesident.com/) is a resource for the international medical graduate community. As part of the service, the “Learn More with MARai” feature uses AI to generate information on residency programs to increase applicants’ confidence going into the interview process.¹⁷ Big Interview Medical (https://www.biginterviewmedical.com/ai-feedback), a paid interview preparation system developed by interview experts, utilizes AI to provide feedback to residents practicing for the interview process by measuring the amount of natural eye contact, language used, and pace of speech. A “Power Word” score is provided that incorporates aspects such as using filler words (“umm,” “uhh”). A Pace of Speech Tool provides rate of speaking feedback presuming that there is an ideal pace to decrease the impression that the applicant is nervous. Johnstone et al¹⁸ used ChatGPT (https://chat.openai.com/auth/login) to generate 2 personal statements for anesthesia residency applicants. Based on survey responses from 31 program directors, 22 rated the statements as good or excellent.¹⁸

Ethnical Concerns and Limitations of AI

The potential use of AI tools by residency applicants inevitably brings forth consideration of biases, ethics, and current limitations. These tools are highly dependent on the quality and quantity of data used for training and validation. Information considered valuable in the holistic review of applications includes unstructured data such as personal statements and letters of recommendation, and incorporating this information can be challenging in ML models, in contrast to discrete structured data such as grades, test scores, and awards. In addition, MLAs depend on large quantities of data to optimize performance.¹⁹ Depending on the size of the applicant pool and the amount of data available, this can present a limitation for smaller programs in developing ML tools for residency recruitment. Studies evaluating the use of AI in the residency application process often are from single institutions, and therefore generalizability is uncertain. The risk for latent bias—whereby a historical or pre-existing stereotype gets perpetuated through the system—must be considered, with the development of tools to detect and address this if found. Choosing which data to use to train the model can be tricky as well as choosing the outcome of interest. For these interventions to become more resilient, programs need to self-examine what defines their criteria for a successful match to their program to incorporate this data into their ML studies. The previously described models in this overview focused on outcomes such as whether an applicant was invited to interview, whether the applicant was ranked, and whether the applicant matriculated to their program.^10,11 For supervised ML models that rely on outcomes to develop a prediction, continued research as to what outcomes represent resident success (eg, passing board certification examinations, correlation with clinical performance) would be important. There also is the possibility of applicants restructuring their applications to align with goals of an AI-assisted search and using AI to generate part or all of their application. The use of ChatGPT and other AI tools in the preparation of personal statements and curriculum vitae may provide benefits such as improved efficiency and grammar support.²⁰ However, as use becomes more widespread, there is the potential increased similarity of personal statements and likely varied opinions on the use of such tools as writing aids.^21,22 Continued efforts to develop guidance on generative AI use cases is ongoing; an example is the launch of VALID AI (https://validai.health/), a collaboration among health systems, health plans, and AI research organizations and nonprofits.²³

Final Thoughts

Artificial intelligence tools may be a promising resource for residency and fellowship programs seeking to find meaningful ways to select applicants who are good matches for their training environment. Prioritizing the holistic review of applications has been promoted as a method to evaluate the applicant beyond their test scores and grades. The use of MLAs may streamline this review process, aid in scheduling interviews, and help discover trends in successful matriculants.