P4926 - Lost in Translation: ChatGPT's Digestive Dialogue in English vs Spanish

Advait Suvarnakar, BS, Harjit Singh, MD, MBA, Srikar Reddy, MD, Shervin Shafa, MD
MedStar Georgetown University Hospital, Washington, DC

Introduction: With the burgeoning role of Artificial Intelligence (AI) in healthcare, particularly in gastroenterology, the utilization of ChatGPT has emerged as a significant tool in explaining gastrointestinal (GI) conditions to the ordinary patient, and helping them make informed medical decisions. While the validity of ChatGPT in the English language has been extensively studied, its efficacy in other languages remains underexplored. We aimed to bridge this gap by examining the validity of using ChatGPT in Spanish-speaking populations.

Methods: We created a single set of 14 questions after a Google Trends search of most common search terms related to GI pathologies. After translating the questions into Spanish, 12 English-Spanish fluent GI attendings were surveyed to assess the accuracy of ChatGPT’s (version 4.0) responses. Average responses were evaluated using a 10 point Likert scale (10 being the most accurate) to gauge differences in accuracy and reliability between the 2 language versions (Table 1). The usage of the Likert scale as well as independent surveyors helped decrease potential bias and validate our study.

Results: Our study revealed notable disparities between the English and Spanish versions of ChatGPT. The English responses demonstrated higher accuracy compared to the Spanish translations (73.3% vs. 40%, p < 0.032), and provided more medically reliable information (43% vs. 25%, p < 0.014). The observed differences underscore the importance of linguistic nuances and cultural context in optimizing AI applications for effective communication with patients across different languages.

Discussion: This study contributes to the limited literature exploring language variations in chatbot systems. The findings highlight the necessity for further research and development in adapting AI tools like ChatGPT for diverse linguistic populations, particularly in the medical domain. Addressing these disparities is crucial to ensure equitable access to accurate healthcare information for non-English-speaking communities. Future studies should delve deeper into linguistic challenges and explore strategies for enhancing the linguistic proficiency and cultural sensitivity of AI-driven healthcare solutions.


Disclosures:


Advait Suvarnakar, BS, Harjit Singh, MD, MBA, Srikar Reddy, MD, Shervin Shafa, MD. P4926 - Lost in Translation: ChatGPT's Digestive Dialogue in English vs Spanish, ACG 2024 Annual Scientific Meeting Abstracts. Philadelphia, PA: American College of Gastroenterology.