Raj Mohan Ram Mohan, MD, Sai Reshma Magam, MD, Sai Greeshma Magam, MD, Melvin Joy, MD, Dilman Natt, MD, Leeza Pannikodu, MD, Winghang Lau, MD, Abhishek Tadikonda, MD, Jiten Desai, MD, Krishnaiyer Subramani, MD, Paul Mustacchia, MD, MBA Nassau University Medical Center, East Meadow, NY
Introduction: The integration of chatbots in gastroenterology presents a promising frontier for enhancing patient care through artificial intelligence. This study delves into their ability to navigate nuanced medical and ethical scenarios, crucial for sensitive healthcare domains. By rigorously evaluating chatbots like ChatGPT and Google GEMINI, we aim to determine their effectiveness in interpreting complex cases, ensuring they meet the high standards required for ethical decision-making and accurate medical advisement in real-world clinical settings.
Methods: The study assessed the medical and ethical reliability of 2 gastroenterology chatbots, ChatGPT, and Google GEMINI, through a questionnaire evaluated by 5 independent reviewers. Responses were rated on a 1-10 Likert scale. We analyzed both chatbots' accuracy, reliability, and user interface using a dataset from various questionnaire categories, calculating the mean and standard deviation. Independent t-tests evaluated statistical differences. Missing data were imputed using column means. Visualizations such as bar charts and scatter plots highlighted metric comparisons.
Results: The study evaluated the performance of CHAT GPT 4o and GOOGLE GEMINI across 3 criteria: accuracy, reliability, and user interface, based on data from various questionnaire categories. Descriptive statistics showed that CHAT GPT 4o had higher mean scores in accuracy (7.80 vs. 3.90), reliability (7.76 vs. 3.67), and user interface (7.99 vs. 3.51). T-tests confirmed these differences as statistically significant with p-values less than 0.05 for all criteria (accuracy: =2.53×10−32p=2.53×10−32, reliability: 𝑝=9.38×10−28p=9.38×10−28, and user interface: 𝑝=2.00×10−48p=2.00×10−48). These results suggest that CHAT GPT 4o is more accurate, reliable, and user-friendly than GOOGLE GEMINI, indicating its potential suitability for supporting gastroenterologists.
Discussion: The study's results underscore the superior performance of CHAT GPT 4o over GOOGLE GEMINI in all evaluated criteria—accuracy, reliability, and user interface. The statistical significance of these outcomes highlights the potential of advanced chatbots to support gastroenterologists effectively. However, while ChatGPT shows promise in understanding complex medical and ethical scenarios, continuous monitoring and iterative updates will be crucial to address any evolving challenges and ensure the technology remains a reliable adjunct in clinical decision-making.
Figure: ChatGPT Vs. Google GEMNI on ethical and medical practice in the field of Gastroenterology
Note: The table for this abstract can be viewed in the ePoster Gallery section of the ACG 2024 ePoster Site or in The American Journal of Gastroenterology's abstract supplement issue, both of which will be available starting October 27, 2024.
Disclosures:
Raj Mohan Ram Mohan indicated no relevant financial relationships.
Sai Reshma Magam indicated no relevant financial relationships.
Sai Greeshma Magam indicated no relevant financial relationships.
Melvin Joy indicated no relevant financial relationships.
Dilman Natt indicated no relevant financial relationships.
Leeza Pannikodu indicated no relevant financial relationships.
Winghang Lau indicated no relevant financial relationships.
Abhishek Tadikonda indicated no relevant financial relationships.
Jiten Desai indicated no relevant financial relationships.
Krishnaiyer Subramani indicated no relevant financial relationships.
Paul Mustacchia indicated no relevant financial relationships.
Raj Mohan Ram Mohan, MD, Sai Reshma Magam, MD, Sai Greeshma Magam, MD, Melvin Joy, MD, Dilman Natt, MD, Leeza Pannikodu, MD, Winghang Lau, MD, Abhishek Tadikonda, MD, Jiten Desai, MD, Krishnaiyer Subramani, MD, Paul Mustacchia, MD, MBA. P1490 - Chatbots and Ethical Medical Practice: A Detailed Analysis of Chat GPT and Google GEMINI in Gastroenterology, ACG 2024 Annual Scientific Meeting Abstracts. Philadelphia, PA: American College of Gastroenterology.
