JADE: jawbone lesion diagnosis and decision supporting system

Clinical Snapshot

Key Findings

• P Value: Significant improvement for GPT-5 with RAG augmentation: P = .002 (Bonferroni-corrected McNemar's test); No significant differences among standalone models or among RAG-based models versus ORAD

• Effect Size: RAG-GPT-5: 20/25 correct (80%); RAG-Claude Sonnet 4.5: 18/25 (72%); RAG-DeepSeek R1: 17/25 (68%); RAG-Gemini 2.5 Flash: 15/25 (60%); Standalone LLMs: 9–13/25 (36–52%); ORAD: 17/25 (68%)

• Primary Outcome: Diagnostic accuracy of JADE (RAG-augmented LLMs) versus standalone LLMs and ORAD across 25 jawbone lesion validation cases

• Nnt Or Sensitivity: Sensitivity, specificity, and likelihood ratios not reported. Accuracy only: RAG-GPT-5 80% vs standalone GPT-5 range 36–52%. Intra-model stability: RAG-GPT-5 mean 0.90 ± 0.11. Response times: 3–10 seconds across configurations

• Confidence Interval: Not reported for any accuracy estimate

Clinical Application

Cloud-based architecture with 3–10 second response times suggests practical workflow integration is technically feasible. However, regulatory approval, data governance compliance, integration with existing PACS/RIS infrastructure, and clinician training requirements are unaddressed. The system requires structured clinical input encoding, which may add workflow burden. In Australia, clinical decision support systems used in diagnostic radiology are subject to TGA regulation as Software as a Medical Device (SaMD) under the Therapeutic Goods Act 1989, with classification dependent on intended use and risk level. JADE in its current proof-of-concept form would not meet TGA requirements for clinical deployment. The Royal Australian and New Zealand College of Radiologists (RANZCR) and the Australian Dental Association (ADA) have not issued specific guidance on LLM-based diagnostic support in oral radiology. PBS reimbursement is not applicable to diagnostic software tools. Australian oral and maxillofacial radiology practice, which relies on CBCT and panoramic radiography for jawbone lesion assessment, could theoretically benefit from such decision support, but independent Australian validation data are entirely absent. Potentially applicable to oral and maxillofacial radiologists, oral surgeons, and dentomaxillofacial specialists assessing jawbone lesions requiring differential diagnosis. However, the current evidence base is insufficient to support clinical deployment — this remains a proof-of-concept tool requiring prospective external validation in representative clinical populations.

Abstract

OBJECTIVES: To develop and evaluate JADE, a proof-of-concept retrieval-augmented generation (RAG) diagnostic assistive system, designed to enhance large language model (LLM) reasoning for jawbone lesion assessment. This study examined whether RAG improves diagnostic accuracy and stability compared with standalone LLMs and ORAD, a supervised learning-based system. METHODS: JADE was developed as a cloud-based RAG system integrating an expert-curated oral radiology database embedded using text-embedding-3-large and indexed in Qdrant Cloud. Structured clinical inputs were encoded as prioritized vector queries. Hybrid semantic and keyword-based retrieval appended relevant evidence to prompts for differential diagnosis generation across LLM backbones. Performance was evaluated in 25 validation cases and compared with GPT-5, Claude Sonnet 4.5, DeepSeek-R1, Gemini 2.5 Flash, their RAG configurations, and ORAD. Diagnostic accuracy was analyzed using Cochran's Q test, with post-hoc McNemar's tests and Bonferroni correction. Intra-model stability and response time were assessed. RESULTS: RAG-GPT-5 achieved the highest diagnostic accuracy (20/25), followed by RAG-Claude Sonnet 4.5 (18/25), RAG-DeepSeek R1 (17/25), and RAG-Gemini 2.5 Flash (15/25). Standalone models achieved 9-13/25 correct diagnoses, ORAD achieved 17/25. No significant differences were observed among standalone models or RAG-based models and ORAD. A significant improvement was observed for GPT-5 when integrated with RAG (P = .002). RAG configurations showed higher intra-model stability, with RAG-GPT-5 achieving mean stability of 0.90 ± 0.11. Mean response times ranged from 3 to 10 s. CONCLUSIONS: JADE improved diagnostic accuracy and stability compared with standalone LLMs, underscoring value of RAG reasoning in jawbone lesion assessment and marking the first RAG application in dentomaxillofacial radiology.

References

1. 1.Baseri Saadi, S., Ver Berne, J., Cavalcante Fontenele, R., Claes, P., & Jacobs, R. (2026). JADE: jawbone lesion diagnosis and decision supporting system. Dentomaxillofacial Radiology. https://doi.org/10.4317/medoral.26824