P3858 - Are ChatGPT Cancer Screening Recommendations Biased in the Presence of Screening Guideline Incongruence?

Adebolanle Ayinde, MD, MPH¹, Joseph Atarere, MD, MPH², Olachi Egbo, MD³, Hiep Nguyen, DO¹, Onyema Chido - Amajuoyi, MD, MPH^4
1CHRISTUS Health | Texas A&M School of Medicine, Longview, TX; ²MedStar Union Memorial Hospital, Baltimore, MD; ³Aurora Medical Center, Oshkosh, WI; ⁴MD Anderson Cancer Center, Longview, TX

Introduction: Health information is crucial to shaping positive health behaviors within the general population, with many turning to the internet for initial guidance. Artificial intelligence (AI) models like ChatGPT, offer potential for delivering accurate health information. However, the variability in colorectal cancer (CRC) screening guidelines from different organizations can create confusion even for AI. This study examines whether ChatGPT’s CRC screening recommendations are appropriate and free from bias, given the multiple, sometimes incongruent, guidelines.

Methods: Guidelines from societies such as the American Cancer Society, U.S. Preventive Services Task Force, American College of Gastroenterology were reviewed. Eight questions on CRC screening among average and high-risk populations were developed, written in simple language and posed to ChatGPT by three independent reviewers. Responses were compared with existing societal guidelines and graded based on appropriateness and Bias.

Results: The responses provided by ChatGPT to all 8 questions on CRC screening in average and high-risk populations corresponded with at least one of the existing societal guidelines in terms of recency and content. Therefore, all responses were graded as appropriate. The responses to all 8 questions also included recommendations from at least two (2) of the professional bodies earlier mentioned. Therefore, all responses were graded as unbiased. The inter-rater reliability of the grading internal medicine physicians who participated in this study was 100%.

Discussion: To our knowledge, this study is the first study to evaluate how ChatGPT handles questions about cancer screening in the settling of societal guideline incongruence. ChatGPT provided appropriate and unbiased responses, aligning with existing societal guidelines for average and high-risk individuals. It recommended initiating screening at age 45 for average risk and earlier for high-risk individuals, but emphasized individualized discussions with healthcare providers. ChatGPT incorporated recommendations from diverse bodies, indicating neutrality and potential to reduce guideline confusion. ChatGPT’s ability to provide reliable information emphasizes its potential in disseminating accurate health data. While promising, limitations include model variations and readability concerns. By addressing the challenges and capitalizing on the strengths of AI, we can enhance the dissemination of accurate health information and support informed health decisions.


Disclosures:


Adebolanle Ayinde, MD, MPH¹, Joseph Atarere, MD, MPH², Olachi Egbo, MD³, Hiep Nguyen, DO¹, Onyema Chido - Amajuoyi, MD, MPH⁴. P3858 - Are ChatGPT Cancer Screening Recommendations Biased in the Presence of Screening Guideline Incongruence?, ACG 2024 Annual Scientific Meeting Abstracts. Philadelphia, PA: American College of Gastroenterology.