Lightning Talk

Evaluation of an IPE Program Using Natural Language Processing Methods

Sunday, August 21, 2022, 10:30 am - 11:30 am CDT
evaluation

Natural language processing (NLP), a branch of artificial intelligence (AI), provides a means for computers to explore written narrative. Widely adopted in a broad range of contexts, including education, it may offer efficient processes to examine qualitative based interprofessional education (IPE) outcomes. AI can be used to explore narrative at many different levels (individual, program, institution). We applied NLP at the programmatic level to explore and understand the narrative data produced during one longitudinal community-based experiential interprofessional learning experience. This program evaluation study paired a common NLP method with an exploratory analytical procedure, topic modeling and correspondence analysis, to analyze learner reflections from the IPE experience. Reflective statements from student cohort years 2020-2021 were analyzed, with 565 students consented to provide data. Statements were evaluated using R and RStudio, employing topic modeling and correspondence analysis techniques. Topic modelling elicited the 20 most prevalent topics, including words reflecting the IPE experience’s broad learning goals, such as “team,” “patient,” “respect”, and “plan”. Correspondence analysis provided evidence of lexiconic similarities between student narratives from the dental, medicine, nursing, pharmacy, physical therapy, and veterinary medicine programs. Clinical dietetics, health administration and clinical health psychology narratives were dissimilar with other programs. Patterns of concordance among terms provided evidence of word groupings that similarly reinforced the program’s goals and objectives; words such as team, patient and communication were closely grouped. NLP provided a novel and efficient method to augment our program evaluation processes, rapidly identifying themes and information important for demonstrating value and enhancing future curriculum. This presentation addresses Building the IPE Case by examining how the IPE Case can be supported through use of AI tools. Attendees will be able to describe how AI can be used to better understand IPE impact on learners and other stakeholders for program evaluation and research purposes.