Safety design guidelines for clinician-AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation
Clinical Snapshot
PICO Framework
| P — Population | Clinicians interacting with Artificial Intelligence-enabled Computer-Aided Diagnosis (CAD) systems in clinical diagnostic workflows |
| I — Intervention | System-Theoretic Process Analysis (STPA)-derived safety design guidelines operationalised within a safety-oriented GUI framework integrating multiple explainability methods and interactive mechanisms |
| C — Comparator | Existing regulatory and risk management frameworks for AI-enabled CAD systems (no active comparator group; framework development and conceptual analysis study) |
| O — Outcomes | Identification of critical hazards (e.g., automation bias, misinterpretation of AI explanations); formulation of actionable safety design guidelines; development of a quantitative explainability consistency indicator for detecting unreliable or ambiguous AI explanations |
Bottom Line
This paper from the Fraunhofer Institute for Cognitive Systems presents a methodologically structured approach to safety design for clinician-AI interaction in computer-aided diagnosis, using System-Theoretic Process Analysis (STPA) to identify hazards and derive actionable design guidelines. The work addresses a genuine and underserved problem: the inadequacy of current regulatory frameworks in managing the human-AI interface in clinical diagnostics. The identification of automation bias and misinterpretation of AI explanations as primary hazard categories is clinically credible and consistent with the broader patient safety literature. The proposed explainability consistency metric is conceptually innovative. However, this remains a theoretical framework paper. There is no empirical clinical validation, no patient outcome data, no usability testing with clinicians, and no quantitative precision metrics. The expert derivation process lacks transparency. For Australian clinicians and health technology governance teams, this framework offers a useful conceptual scaffold for structuring AI safety assessments and informing TGA SaMD submissions, but it cannot be adopted as clinical guidance without prospective validation in real-world settings. It is best regarded as a starting point for institutional AI governance work rather than evidence-based clinical practice guidance.
Key Findings
P Value: Not reported
Effect Size: Not applicable — no empirical effect size reported; this is a framework development study
Primary Outcome: Derivation of a set of actionable, STPA-grounded safety design guidelines for clinician-AI interaction in CAD systems, addressing hazards including automation bias and XAI misinterpretation
Nnt Or Sensitivity: Not applicable — no diagnostic accuracy, NNT, or hazard ratio data presented. The proposed explainability consistency metric is described as a quantitative indicator but no sensitivity/specificity values for detecting unreliable explanations are reported
Confidence Interval: Not reported
Clinical Application
The framework is conceptually implementable but requires substantial institutional resources for operationalisation: dedicated human factors expertise, iterative GUI prototyping, XAI method integration, and ongoing safety monitoring infrastructure. Feasibility in smaller health services or primary care settings is not addressed. The STPA methodology requires specialist training not routinely available in clinical teams. Australia's Therapeutic Goods Administration (TGA) has published guidance on Software as a Medical Device (SaMD) and AI/ML-based medical devices, requiring post-market surveillance and risk management aligned with ISO 14971. The STPA-based framework described in this paper could theoretically complement TGA SaMD regulatory submissions by providing structured hazard documentation. However, the paper does not reference TGA guidance, the Australian Digital Health Agency's AI frameworks, or RACGP digital health standards. Australian health services considering AI-enabled CAD deployment should treat this framework as a supplementary methodological resource requiring local contextualisation, regulatory alignment with TGA requirements, and validation within Australian clinical workflows before adoption. The PBS does not currently fund AI-assisted diagnostic services as a standalone item, and MBS item numbers for AI-augmented diagnostics remain limited and evolving. Clinicians in any specialty using AI-enabled CAD systems for diagnostic decision support, particularly in radiology, pathology, dermatology, and cardiology where CAD tools are most mature. Most immediately relevant to clinical informaticists, health technology assessment teams, and clinical AI governance committees.
Abstract
Ensuring the safety of Artificial Intelligence-enabled Computer-Aided Diagnosis systems is critical because diagnostic errors can have serious consequences for patient care. However, existing regulatory and risk management frameworks often do not sufficiently address the complex socio-technical interactions between clinicians and Artificial Intelligent systems, leaving key human-centered safety challenges underexplored. This paper presents a systematic and human-centered approach to deriving safety design guidelines for clinician-Artificial Intelligence interaction in Computer-Aided Diagnosis systems using System-Theoretic Process Analysis. Through this analysis, we identify critical hazards associated with clinician-Artificial Intelligence collaboration, including automation bias on system recommendations, and misinterpretation of explanations. Based on the identified unsafe control actions, we formulate a set of actionable and traceable safety design guidelines that promote transparency, coherent explanations, and calibrated trust in Artificial Intelligence-assisted decision-making. To bridge safety analysis and system design, the proposed guidelines are operationalized within a Computer-Aided Diagnosis interaction framework. The framework includes a safety-oriented Graphical User Interface that integrates multiple explanation methods and interactive mechanisms to promote clinician engagement. Furthermore, we introduce a safety-oriented evaluation approach that uses consistency across multiple explanation methods as a quantitative indicator of potentially unreliable or ambiguous explanations. By linking System-Theoretic Process Analysis, interaction design, and explainability evaluation, this work provides a unified and reusable framework for improving the safety and reliability of Artificial Intelligence-driven Computer-Aided Diagnosis systems.
References
- 1.Hagiwara, Y., Fitch, K., & Trapp, M. (2026). Safety design guidelines for clinician-AI interaction in computer-aided diagnosis systems using system-theoretic framework with explainability validation. Scientific Reports. PubMed ID: 42527433.
Related Research
JMIR research protocols
Evaluation and Comparison of Latent Health Risk Prediction Models for Clinical Triage: Protocol for a Mixed Methods Study
5 July 2026
Medical image analysis
Medical hierarchical image classification via dual-geometry image-text learning
4 July 2026
Medical physics
Gamma Knife treatment planning using knowledge-based reinforcement learning
4 July 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service