A unified multimodal framework for chest X-ray retrieval and disease prediction for clinical decision support

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Macro AUROC of 0.95

• Primary Outcome: Disease classification across 22 diagnostic categories

• Nnt Or Sensitivity: Macro F1-score of 0.71; sensitivity and specificity not reported

• Confidence Interval: Not reported

Clinical Application

Requires DICOM-format images and associated reports; computational requirements not specified Would need TGA approval as medical device software; integration with Australian PACS systems required; compliance with Australian privacy regulations essential Patients undergoing chest X-ray examination, though specific population characteristics unclear

Abstract

Recent advances in medical imaging and natural language processing enable new opportunities for automated diagnostic support. Chest X-rays (CXRs) remain the most common imaging modality for screening pulmonary, cardiovascular, and systemic diseases; however, the growing volume of studies and free-text reports can overwhelm clinicians. We propose a unified multimodal retrieval and prediction framework that jointly leverages DICOM-format CXRs and radiology reports by projecting visual and textual features into a shared semantic space. Trained with contrastive and multi-label objectives, the system supports disease classification, case-based retrieval, and explainable AI. Experiments on the Open-I dataset demonstrate strong performance, achieving a Macro AUROC of 0.95 and Macro F1-score of 0.71 across 22 diagnostic categories. The retrieval module attains high ranking quality (MRR and nDCG >0.93) with sub-millisecond query latency. Quantitative explainability analysis further shows strong agreement between attention- and gradient-based attribution maps (Pearson ρ≈0.92), supporting trustworthy clinical decision-making.