Development and evaluation of a technology-enhanced simulation to measure physician decision making in trauma triage

Clinical Snapshot

Key Findings

• P Value: High-volume perceptual sensitivity: p=0.025; Low-volume perceptual sensitivity: p=0.39; Decisional threshold: p=0.18

• Effect Size: Spearman rho = 0.46 (high-volume physicians, ≥10 severely injured patients/year); rho = -0.26 (low-volume physicians); decisional threshold correlation rho = 0.22 (all physicians)

• Primary Outcome: Correlation between simulation-derived perceptual sensitivity (SDT d') and real-world EHR-derived perceptual sensitivity among ED physicians

• Nnt Or Sensitivity: Internal consistency (Cronbach's alpha): 0.80–0.86 across similar case types; Physicians completed mean 20/26 cases (77% completion rate); Mean time per case: 2.6 minutes (SD 1.8); Mean decisions per case: 2.6 (SD 1.7)

• Confidence Interval: High-volume perceptual sensitivity: 95% CI 0.07–0.76; Low-volume perceptual sensitivity: 95% CI -0.87–0.50; Decisional threshold: 95% CI -0.12–0.51

Clinical Application

The simulation is designed to be low-cost and scalable, with a mean completion time of approximately 52 minutes (20 cases × 2.6 minutes). Online delivery supports remote administration. However, the 33% enrolment rate in a research context raises questions about feasibility for routine quality assurance or credentialing purposes. Further development and validation in larger, more diverse samples is required before clinical deployment. Australian relevance is currently limited. Australian trauma systems operate under jurisdictional trauma networks (e.g., the Major Trauma Service in Victoria, the NSW Trauma Network) with distinct triage criteria and retrieval pathways that differ from the US American College of Surgeons field triage framework used in this simulation. The RACGP and Australasian College for Emergency Medicine (ACEM) have not endorsed SDT-based simulation tools for performance assessment. No PBS or TGA implications apply. However, the underlying methodological framework — using SDT to distinguish diagnostic accuracy from decisional bias in clinical triage — is conceptually transferable and could inform future Australian instrument development, particularly in the context of rural and remote ED physician performance assessment where trauma volume is low and guideline adherence is variable. ED physicians practising in non-trauma centre settings who evaluate injured patients, particularly those with sufficient clinical volume (≥10 severely injured patients/year) to generate meaningful real-world performance benchmarks

Abstract

PURPOSE: Variable implementation of clinical practice guidelines causes preventable morbidity and mortality. The paucity of valid and reliable measures of physician performance impedes efforts to improve implementation. The objective of this study was to evaluate a low-cost, scalable simulation method to evaluate physician non-adherence to guidelines, using trauma triage in Emergency Departments (EDs) as an archetypal clinical problem. METHODS: We created an online simulation that mimicked the task environment of a non-trauma center ED. We recruited a sample of ED physicians, asked them to use the simulation, and obtained the electronic health records of injured patients treated by these physicians in the prior 3 years. We used signal detection theory, a behavioral science method, to analyze triage performance. The method quantified the influence of 2 determinants of non-adherence: perceptual sensitivity (diagnostic accuracy) and decisional threshold (revealed preferences for false positive/negative decisions). We collected evidence of the simulation's response process, internal structure, content validity, and relations with other variables. RESULTS: Among 180 invited physicians, 60/180 (33%) enrolled and 45/60 (75%) participated; 38/45 (84%) had accessible electronic records. Physicians completed an average of 20/26 simulation trauma cases, spending 2.6 minutes/case (SD 1.8), and making 2.6 decisions/case (SD 1.7). Responses to similar types of simulation cases were consistent (Cronbach's alpha 0.80-0.86). Users reported strong content validity. Perceptual sensitivity on the simulation and in real-life correlated moderately well among physicians who evaluated ≥10 severely injured patients/year (rho 0.46, 95% CI 0.07-0.76, p = 0.025), but not lower volumes (rho -0.26, 95% CI -0.87-0.50, p = 0.39); decisional thresholds were uncorrelated (rho 0.22, 95% CI -0.12-0.51, p = 0.18). CONCLUSIONS: The technology- and behavioral science-enhanced simulation demonstrated evidence of validity across multiple domains, with mixed findings for relations with other variables. With further refinement, it offers a promising avenue for studying physician decision making in trauma triage, with possible application to other clinical domains.