Feasibility of retrieval-augmented generation for large language models with Japanese input in radiotherapy

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: 7.2% to 16.1% improvement across examinations

• Primary Outcome: Examination accuracy improvement with RAG enhancement

• Nnt Or Sensitivity: Medical Physicist: 84.6% ± 1.5% vs 77.0% ± 2.6%; Radiologist: 82.1% ± 1.1% vs 74.9% ± 2.0%; Radiation Oncologist: 71.7% ± 4.5% vs 55.6% ± 4.4%

• Confidence Interval: Not reported

Clinical Application

Technically feasible but requires substantial infrastructure and knowledge base development Limited direct applicability due to language-specific design, but methodology could inform development of similar systems for Australian radiotherapy training and practice support Radiotherapy professionals and trainees requiring decision support or educational assistance

Abstract

Large language models (LLMs) have recently gained attention for their potential. However, concerns remain regarding their reliability due to limitations such as hallucinations and insufficient domain-specific knowledge. Retrieval-augmented generation (RAG) has emerged as a promising approach, enabling LLMs to reference external knowledge sources and generate accurate outputs. We aimed to clarify the potential of RAG-enhanced LLMs with Japanese input in the field of radiotherapy. This was assessed by evaluating performance on three certification examinations in Japan: the Japanese Medical Physicist Examination, the Japanese Board Examination for Radiologists, and the Japanese Board Examination for Radiation Oncologists. In this study, we constructed a RAG system named Rad-Hub, consisting of a Japanese Radiotherapy Knowledge Database (JRKD) and a retrieval framework built on Microsoft Azure. The JRKD was populated with 32 Japanese radiotherapy textbooks and clinical guidelines. We assessed its utility by inputting all multiple-choice questions from the three examinations into ChatGPT-4o, both with and without Rad-Hub, and recording the answers. They were then compared with reference answers determined by experienced medical physicists and radiation oncologists. Rad-Hub improved accuracy across all examinations. Accuracy increased from 77.0% ± 2.6% to 84.6% ± 1.5% in the Medical Physicist examination, from 74.9% ± 2.0% to 82.1% ± 1.1% in the Radiologist examination, and from 55.6% ± 4.4% to 71.7% ± 4.5% in the Radiation Oncologist examination. Performance gains ranged from 7.2% to 16.1%. These findings highlight the potential of RAG-enhanced LLMs, particularly ChatGPT-4o with Rad-Hub, for integration into radiotherapy applications, such as educational and clinical decision assistance.

References

1. 1.Takahashi, Y., Kadoya, N., Arai, K., Tanno, H., Tanaka, S., Katsuta, Y., Hoshino, T., Harada, H., Omata, S., Yamamoto, T., Umezawa, R., Yasui, K., Hayashi, N., & Jingu, K. (2026). Feasibility of retrieval-augmented generation for large language models with Japanese input in radiotherapy. Journal of Radiation Research. https://doi.org/10.1093/jrr/rrag019