Artificial Intelligence in Occupational Health Surveillance: Evaluating AI-Assisted ILO Classification of Radiographs of Pneumoconioses

Clinical Snapshot

Key Findings

• P Value: Significant differences between models (McNemar's test)

• Effect Size: Balanced accuracy 60.8%-70.3% across four AI models

• Primary Outcome: Balanced accuracy for pneumoconiosis classification

• Nnt Or Sensitivity: Sensitivity 35.5%-54.9%, Specificity 84.6%-86.2%

• Confidence Interval: Bootstrap 95% confidence intervals provided

Clinical Application

Current performance insufficient for independent use; potential as screening tool requires validation Relevant to Australian mining and industrial sectors; would need validation against local radiological practices and integration with existing occupational health surveillance programs Workers at risk of pneumoconiosis requiring chest radiograph surveillance

Abstract

BACKGROUND: Pneumoconioses remain an important occupational health issue, particularly in low- and middle-income countries. The International Labour Organization (ILO) Classification standardizes chest radiograph interpretation but requires trained readers and is affected by inter-reader variability. This study evaluated whether generative multimodal artificial intelligence (AI) models can approximate ILO-based diagnostic reasoning. METHODS: Eighty-two chest radiographs from the official NIOSH B Reader syllabus were analysed using four AI systems (GPT-4o, GPT-5, MedGemma-4B, MedGemma-27B). Each image was evaluated with a standardized prompt based on the 2022 revised ILO guidelines using deterministic settings. Model outputs were mapped to ILO codes and compared with the official answer keys of the ILO Standard Radiograph Set used for B Reader training and examination. Performance metrics included balanced accuracy, sensitivity, specificity, precision, and Matthews correlation coefficient (MCC). Bootstrap 95% confidence intervals, McNemar's test, and Cohen's κ assessed performance variability and agreement. RESULTS: All four AI models showed moderate diagnostic performance, with balanced accuracy ranging from 60.8% to 70.3%. Sensitivity remained limited (35.5%-54.9%), while specificity was consistently high (84.6%-86.2%). MedGemma-27B performed best for small opacities, GPT-5 for pleural abnormalities and for technical quality. Large opacities and rare findings were systematically under-detected. Statistical comparisons showed significant differences between models, although agreement patterns were broadly similar. CONCLUSION: All AI models partially followed structured ILO radiographic criteria but did not achieve expert-level performance, confirming that they cannot replace certified B Readers. Larger, real-world datasets are needed to assess their potential clinical utility as supportive tools in occupational health surveillance programs.

References

1. 1.Baldassarre, A., Padovan, M., Palla, A., Quercia, A., Leonori, R., Dugheri, S., Mucci, N., & Traversini, V. (2026). Artificial Intelligence in Occupational Health Surveillance: Evaluating AI-Assisted ILO Classification of Radiographs of Pneumoconioses. La Medicina del lavoro. https://doi.org/10.23749/mdl.2026.18371