P1520 - Suboptimal Adherence to The (CONSORT-AI) Extension Guideline Threatens Transparency and Reproducibility of Randomized Trials in Gastroenterology: A Systematic Review and Meta-Analysis

Mohanad Awadalla, MD¹, Maryam R Hussain, MD, MPH², Chris Labaki, MD³, Tasnim Ahmed, MD³, Tia Andrade, MD³, Mark Enrik Geissler, ³, Rachel Hicklen, ⁴, Tyler M. Berzin, MD³, Faisal Ali, MD^5
1Beth Israel Deaconess Medical Center, Boston, MA; ²University of Texas Medical Branch, Houston, TX; ³Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA; ⁴University of Texas MD Anderson Cancer Center, Houston, TN; ⁵University of Texas, Houston, TX

Introduction: The Consolidated Standards of Reporting Trials–Artificial Intelligence (CONSORT-AI) Extension Guideline was devised to establish reporting standards for AI-RCTs. The adherence of AI trials to CONSORT-AI in Gastroenterology (GI) has not been well studied, and is the topic of this study.

Methods: We conducted a systematic review of Ovid Embase and Ovid MEDLINE to identify randomized control trials (RCTs) on the utility of AI in GI in adults. Data on trial demographics and AI system use were obtained. Definitions for CONSORT-AI extension guideline subdomains that pertain to reporting standards for AI clinical trials were defined a priori. Adherence to these reporting standards was assessed at a trial level. A total of 13 domains were assessed. A CONSORT-AI score was devised to assess adherence, with the highest score of 13 representing adherence to all 13 domains assessed.

Results: A total of 53 RCTs were identified (Table 1). The mean CONSORT-AI score was 9.2 (±2.6). No trials adhered to all 13 domains of CONSORT-AI that were assessed (Figure1). The least adherence was noted with information regarding code access/availability and reporting/analysis of performance errors (17% each), and handling of poor quality and/or missing data (36%). In contrast, 100% adherence was noted for two domains pertaining to the clinical context and provision of background information.<br><br>The mean total CONSORT-AI score was 8.1 ± 4.1 for studies focused on the colon, 9.6 ± 1.3 for studies focused on the upper GI tract, and 6.7 ± 3.1 for studies of diagnostic imaging and sedation in endoscopy. Reporting/analysis of performance errors and reporting of code access/availability remained critically low across all GI disciplines for which RCTs currently exist, with the lowest adherence in studies focused on the upper GI tract (0% each from 7 RCTs). Similarly, information on handling poor data was also limited across all GI disciplines, with the lowest adherence in diagnostic imaging and sedation studies in endoscopy. A complete lack of adherence (0% across the body of evidence) to CONSORT-AI by studies focused on the colon was not seen.

Discussion: AI trials are failing to report information on code accessibility, handling of poor quality and/or missing data, and reporting of analyses and handling of performance errors; these domains are critical for transparency and reproducibility of AI technology within the field of GI. Strategies that improve reporting of these domains in publications of AI trials should be explored.



Disclosures:


Mohanad Awadalla, MD¹, Maryam R Hussain, MD, MPH², Chris Labaki, MD³, Tasnim Ahmed, MD³, Tia Andrade, MD³, Mark Enrik Geissler, ³, Rachel Hicklen, ⁴, Tyler M. Berzin, MD³, Faisal Ali, MD⁵. P1520 - Suboptimal Adherence to The (CONSORT-AI) Extension Guideline Threatens Transparency and Reproducibility of Randomized Trials in Gastroenterology: A Systematic Review and Meta-Analysis, ACG 2024 Annual Scientific Meeting Abstracts. Philadelphia, PA: American College of Gastroenterology.