AI-generated clinical summaries: errors and susceptibility to speech and speaker variability

Clinical Snapshot

Key Findings

• P Value: Phonological impairment significantly reduced recognition (p<0.001)

• Effect Size: No significant differences for personality types (all p>0.05) or accents (patients p=0.851, doctors p=0.980)

• Primary Outcome: Error rates across communication variables

• Nnt Or Sensitivity: Median error rates: extraversion 3.5, Chinese/Indian accented doctors 3.0

• Confidence Interval: Not reported

Clinical Application

Requires clinician verification and performance monitoring systems Relevant for Australian primary care given diverse patient populations and TGA considerations for AI medical devices Primary care settings considering AI scribe implementation

Abstract

OBJECTIVES: To evaluate whether variability in patients' communication style (personality), international English-accents (human and synthetic) and speech impairments affects the accuracy of a Clinical AI Scribe (CAIS) and identify where performance degrades to inform pre-deployment validation and monitoring. METHODS: We conducted simulated primary-care consultations using trained actors. For personality types, four scenarios were enacted, each with five patient-personality types. For accents, transcripts of consultations were used to generate combinations of seven accents across five scenarios. The CAIS produced summaries that were compared with transcripts, and errors classified as omissions, factual inaccuracies or hallucinations. For speech impairments, public recordings representing five profiles were transcribed and word-recognition accuracy calculated. RESULTS: Personality types showed no statistically significant differences in errors (all p>0.05). Extraversion had the highest total errors (median 3.5). Across accents, comparisons were non-significant for both patient and doctor voices (patients: p=0.851; doctors: p=0.980). Omissions predominated, with low rates of hallucinations and factual inaccuracies. Omissions were slightly higher for Chinese-accented and Indian-accented doctors (both medians 3.0). Conversely, speech impairments differed: cleft palate and vowel disorders were near-perfect, whereas phonological impairment markedly reduced recognition (p<0.001). DISCUSSION: Operationally, CAIS deployment should include clinician-in-the-loop verification, subgroup performance monitoring (accents, impairments) and predefined 'switch-off' criteria for severe phonological patterns. High-quality synthetic voices are a pragmatic proxy for accent testing when balanced corpora are unavailable. CONCLUSIONS: Under controlled conditions CAIS performance was broadly stable across communication styles and most accents, but vulnerable to specific speech characteristics, particularly phonological impairment, in this single-system simulation study.

References

1. 1.Draper, T. C., Leake, J., Cox, T., Lamb-Riddell, K., Johns, B. E., McCormick, J., Trowell, S., Kiely, J., & Luxton, R. (2026). AI-generated clinical summaries: errors and susceptibility to speech and speaker variability. BMJ Health & Care Informatics. https://doi.org/10.1136/bmjhci-2025-101918