A Large Language Model Assistant for Summarizing Hepatology Referral Documents

Clinical Snapshot

Key Findings

• P Value: P < 0.001 for time reduction

• Effect Size: Adjusted mean triage time: 37.2 seconds (AI) vs 94.2 seconds (original)

• Primary Outcome: 60% reduction in triage time with LLM summaries

• Nnt Or Sensitivity: Accuracy rate: 94.6% (IQR: 86.5%-97.3%)

• Confidence Interval: Not clearly reported

Clinical Application

Requires integration with electronic health records and staff training on LLM outputs Would need adaptation for Australian referral systems, privacy regulations, and TGA approval for clinical decision support tools Gastroenterology and hepatology specialists managing referral triage

Abstract

INTRODUCTION: To use a large language model (LLM) to create accurate and useable summaries of patient records for clinicians triaging new hepatology referrals. METHODS: We developed a comprehensive list of data elements required to triage a new hepatology referral and engaged in an iterative prompt engineering process to instruct an LLM to extract relevant data from patient referral documents. The final prompt was used on 50 original patient records from June to July 2025 to generate corresponding artificial intelligence (AI) summaries, which were assigned to 2 providers to review and triage according to their usual process. We assessed time to triage original vs AI files and accuracy of the AI files. A linear mixed-effects model was used to determine an adjusted time ratio comparing the time to triage AI files vs original files. RESULTS: AI-generated summaries were significantly shorter than original files (median [interquartile range] 2 [2-3] vs 23 [10.2-38.8] pages, P < 0.001). AI summaries had high accuracy (median [interquartile range]: 94.6% [86.5%-97.3%]) with a low hallucination rate. Use of the AI summaries led to a 60% reduction in triage time (adjusted mean triage time of 37.2 seconds for AI files vs 94.2 seconds for original files, P < 0.001). DISCUSSION: The use of an LLM led to significantly reduced document length, maintained an appropriate level of accuracy, and led to a significant decrease in clinician time to review the patient record. Future steps involve creating a fully automated workflow that is integrated into the electronic health record for widespread use.

References

1. 1.Shroff, H., Shankar, A., Baron, A., Yoxheimer, L., & Fix, O. K. (2026). A Large Language Model Assistant for Summarizing Hepatology Referral Documents. The American Journal of Gastroenterology. https://doi.org/10.18653/v1/2025.findings-emnlp.1128