Yale Researchers Use Large Language Models to Detect Gastrointestinal Bleeding < Yale School of Medicine

Timely and accurate identification of gastrointestinal bleeding is key to guide appropriate medical management and is also necessary to ensure correct coding and reimbursement for the condition. A team of Yale researchers, led by Dennis L. Shung, MD, MHS, PhD, assistant professor of medicine (digestive diseases) in collaboration with Hua Xu, PhD, vice chair for research and development (biomedical informatics and data science) and assistant dean for biomedical informatics, demonstrated the effectiveness of a large language model (LLM)-based pipeline in identifying overt gastrointestinal bleeding in 1,108 patients seen between 2014 and 2023 in the Yale-New Haven Health System. The study, “Detection of Gastrointestinal Bleeding With Large Language Models to Aid Quality Improvement and Appropriate Reimbursement,” was published in Gastroenterology.

To conduct the study, researchers used more than 17,000 nursing notes from the electronic health record (EHR) and longitudinal data on vital signs and laboratory measures to train a machine learning model to detect recurrent bleeding. They then validated this detection model on more than 500 patients at four different hospitals in the Yale-New Haven Health System. Additionally, the researchers collaborated with Hamita Sachar, MD, associate professor of medicine (digestive diseases) and associate chief of gastroenterology for digestive health, and Ohm M. Deshpande, MD MHA, associate chief population health officer and vice president, clinical finance for Yale-New Haven Health, to develop an algorithm for appropriate reimbursement coding and validated this with expert coder input.

The study’s results showed a high accuracy of the model in detecting melena, hematochezia, and hematemesis, identifying recurrent bleeding, and catching more than 98% of cases (sensitivity: 98.4%). The algorithm increased the average per-patient reimbursement from $1,299 to $3,247.

Discussing the important implications of the work, Shung says, “We present a concrete use case for local large language models to perform complex tasks that have specific value to healthcare systems. By automating the extraction of abstract clinical concepts from longitudinal electronic health record data, our local large language model pipeline enables calculation of meaningful quality measures and appropriate complexity capture in patients with acute gastrointestinal bleeding.”

Other Yale authors include: Neil Zheng, MD; Vipina Keloth, PhD; Daniel Kats, MD; Darrick Li, MD, PhD; Hua Xu, PhD; and Loren Laine, MD. The research team also included Kisung You from the City University of New York Baruch College.

Zheng, N. S., Keloth, V. K., You, K., Kats, D., Li, D. K., Deshpande, O., Sachar, H., Xu, H., Laine, L., & Shung, D. L. Detection of Gastrointestinal Bleeding With Large Language Models to Aid Quality Improvement and Appropriate Reimbursement. Gastroenterology. Published online September 18, 2024. doi:10.1053/j.gastro.2024.09.014

The research reported in this news article was supported by the Yale School of Medicine Fellowship for Medical Student Research funded through the National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases (awards T35DK104689 and K23 DK125718). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.