P2125 - The Role of Generative Language Systems in Increasing Patient Awareness of Colon Cancer Screening

Daryl Ramai, MD, MPH, Msc¹, Marcello Maida, MD², Amy Oxentenko, MD³, Jonathan Leighton, MD³, David Hass, MD⁴, Yuichi Mori, MD⁵, Mario Dinis-Ribeiro, MD⁶, Cesare Hassan, MD⁷, Marianna Arvanitakis, MD⁸, Monique T.. Barakat, MD⁹, Raf Bisschops, MD¹⁰, Kathryn Byrne, MD¹¹, Evelien Dekker, MD¹², John Fang, MD¹¹, Lorenzo Fuccio, MD¹³, Paraskevas Gkolfakis, MD¹⁴, Vicente Lorenzo-Zúñiga, MD¹⁵, Andrew Ofosu, MD¹⁶, Franco Radaelli, MD¹⁷, Jessica Stout, DO¹¹, Luigi Ricciardiello, MD¹⁸, Matt Rutter, MD¹⁹, Konstantinos Triantafyllou, MD²⁰, Antonio Facciorusso, MD, PhD^21
1University of Utah Health, Salt Lake City, UT; ²University of Enna "Kore", Enna, Lombardia, Italy; ³Mayo Clinic, Rochester, MN; ⁴Yale New Haven Health, Hamden, CT; ⁵University of Oslo, Blindern, Oslo, Norway; ⁶Portuguese Institute of Oncology, Porto, Porto, Portugal; ⁷Humanitas Clinical and Research Hospital, Rozzano, Lombardia, Italy; ⁸Hôpital Erasme, Brussels, Brussels Hoofdstedelijk Gewest, Belgium; ⁹Stanford Health Care, Stanford, CA; ¹⁰University Hospitals, Leuven, Brussels Hoofdstedelijk Gewest, Belgium; ¹¹University of Utah, Salt Lake City, UT; ¹²Amsterdam University Medical Center, Amsterdam, Zuid-Holland, Netherlands; ¹³University of Bologna, Bologna, Marche, Italy; ¹⁴Konstantopoulio-Patision General Hospital of Nea Ionia, Athens, Ioannina, Greece; ¹⁵La Fe University and Polytechnic Hospital, Valencia, Comunidad Valenciana, Spain; ¹⁶University of Cincinnati, Cincinnati, OH; ¹⁷Valduce Hospital, Como, Lombardia, Italy; ¹⁸University of Bologna, Bologna, Emilia-Romagna, Italy; ¹⁹North Tees and Hartlepool NHS Foundation Trust, Stockton-on-Tees, England, United Kingdom; ²⁰Attikon University General Hospital, Athens, Ioannina, Greece; ²¹University of Foggia, Foggia, Puglia, Italy

Introduction: Despite numerous efforts, adherence to colorectal cancer (CRC) screening remains suboptimal. Artificial intelligence (AI)-assisted chatbots such as ChatGPT (Chat Generative Pretrained Transformer) are emerging as a revolutionary tool that has the potential to provide educational support to patients through an accessible question–answer system. This study aims to evaluate the effectiveness of chat GPT in answering patients' questions about CRC screening, with the ultimate goal of enhancing patients' awareness and adherence to national screening programs worldwide.

Methods: Three experts created a list of 15 questions on CRC screening that were posed to ChatGPT4. Thereafter, the answers were rated by 20 gastroenterology experts and 20 non-experts in three domains (accuracy, completeness, and comprehensibility). The answers were also rated by 100 gastroenterology patients in three dichotomic domains (completeness, comprehensibility and trustability).

Results: According to expert rating, the mean accuracy score on a scale ranging from 1 to 6 was 4.8±1.1, with 10/15 questions receiving a mean rating ³5 points. The average completeness score on a scale ranging from 1 to 3 was 2.1±0.7, with 12/15 questions receiving a mean rating ³2 points. Mean comprehensibility score on a scale from 1 to 3 was 2.8±0.4, with all questions receiving a rating ³2.5 points. Overall, accuracy (4.8±1.1 vs 5.6±0.7, P< 0.001) and completeness (2.1±0.7 vs 2.7±0.4, P< 0.001) scores were significantly lower for expert compared to non-expert, while comprehensibility was comparable among the two groups (2.7±0.4 vs 2.8±0.3, P=0.546). Patients rated all questions as complete, comprehensible and trustable in 97 to 100% of cases.

Discussion: ChatGPT shows good performance with the potential to enhance awareness about CRC and improve screening outcomes. In the future, generative language systems may be further improved after proper training in accordance with scientific evidence and current guidelines.



Disclosures:


Daryl Ramai, MD, MPH, Msc¹, Marcello Maida, MD², Amy Oxentenko, MD³, Jonathan Leighton, MD³, David Hass, MD⁴, Yuichi Mori, MD⁵, Mario Dinis-Ribeiro, MD⁶, Cesare Hassan, MD⁷, Marianna Arvanitakis, MD⁸, Monique T.. Barakat, MD⁹, Raf Bisschops, MD¹⁰, Kathryn Byrne, MD¹¹, Evelien Dekker, MD¹², John Fang, MD¹¹, Lorenzo Fuccio, MD¹³, Paraskevas Gkolfakis, MD¹⁴, Vicente Lorenzo-Zúñiga, MD¹⁵, Andrew Ofosu, MD¹⁶, Franco Radaelli, MD¹⁷, Jessica Stout, DO¹¹, Luigi Ricciardiello, MD¹⁸, Matt Rutter, MD¹⁹, Konstantinos Triantafyllou, MD²⁰, Antonio Facciorusso, MD, PhD²¹. P2125 - The Role of Generative Language Systems in Increasing Patient Awareness of Colon Cancer Screening, ACG 2024 Annual Scientific Meeting Abstracts. Philadelphia, PA: American College of Gastroenterology.