Pronunciation assessment in foreign language learning: Reliability and scoring bias in human-generative AI evaluation

Clinical Snapshot

Key Findings

• P Value: Statistical significance of score differences confirmed via paired-samples t-tests (p values not specified in abstract)

• Effect Size: Not reported in abstract; direction of bias consistently favours AI leniency (AI scores higher than human raters across all eight subcomponents)

• Primary Outcome: Gen-AI demonstrated moderate inter-rater reliability with human raters across most pronunciation subcomponents, with highest agreement in fluency and lowest in individual phonemes

• Nnt Or Sensitivity: ICC values not numerically reported in abstract; described qualitatively as 'moderate' for most components. No sensitivity/specificity or diagnostic accuracy metrics applicable to this reliability study design.

• Confidence Interval: Not reported for any estimate

Clinical Application

Gen-AI pronunciation assessment is technically feasible as a supplementary formative tool given its scalability and availability. However, the systematic leniency bias identified means AI scores cannot substitute for human rater judgements in high-stakes summative assessment without calibration. Implementation requires identification of a validated, named AI platform and establishment of score adjustment protocols. In the Australian context, this study has relevance to ELICOS (English Language Intensive Courses for Overseas Students) providers, university English language centres, and IELTS/PTE preparation programmes. The Australian Skills Quality Authority (ASQA) and TEQSA regulate language programme quality, and any AI-assisted assessment tool would need to demonstrate validity and fairness across the diverse L1 backgrounds of Australia's international student cohort (predominantly Mandarin, Hindi, Arabic, Vietnamese speakers). The TGA is not applicable here as this is an educational rather than medical device. RACGP guidelines are not relevant. The finding of systematic AI leniency bias is particularly pertinent for programmes where pronunciation scores influence visa-related English proficiency determinations. Foreign language learners in structured educational programmes where pronunciation is formally assessed; most directly applicable to EFL/ESL learners in higher education settings with access to Gen-AI tools

Abstract

This study examines the reliability and scoring bias of generative AI (Gen-AI)-based pronunciation assessment compared with human raters, addressing whether AI-generated scores can be trusted in real educational settings. Sixty students participated in a 12-week program. A total of 180 pronunciation samples were evaluated across eight subcomponents (individual phonemes, stress, rhythm, intonation, linking, reduction, fluency, and clarity) by three standardized human raters and Gen-AI using the same 7-point rubric. Quantitative analyses (intraclass correlation coefficients, paired-samples t-tests, and Pearson correlations) assessed reliability and bias, while semi-structured interviews with raters provided explanatory qualitative insights. Gen-AI demonstrated moderate reliability with human raters across most components, showing the highest agreement in fluency and the weakest in individual phonemes. However, Gen-AI consistently assigned significantly higher scores than human raters across all subcomponents. Qualitative findings revealed that discrepancies originated from Gen-AI's limited discriminative power, systematic flaws in handling missing data, decontextualized scoring approach, lack of sensitivity to L1 interference, and inability to interpret pragmatic context. While Gen-AI cannot fully replace human expertise, it can function as a complementary tool for formative assessment and autonomous practice. A hybrid assessment model integrating Gen-AI's efficiency with human raters' contextual and interpretive insights is recommended for effective foreign language pronunciation education.

References

1. 1.Choi, W. (2026). Pronunciation assessment in foreign language learning: Reliability and scoring bias in human-generative AI evaluation. PLOS ONE. https://doi.org/10.1371/journal.pone.0354603