Comparing Large Language Models and Traditional Machine Translation Tools for Translating Medical Consultation Summaries: Quantitative Pilot Feasibility Study
Clinical Snapshot
PICO Framework
| P — Population | Medical consultation summaries in English requiring translation to Arabic, Chinese (simplified), and Vietnamese |
| I — Intervention | Large Language Models (GPT-4o, Llama-3.1, Gemma-2) for medical translation |
| C — Comparator | Traditional machine translation tools (Google Translate, Microsoft Bing Translator, DeepL) and professional interpreter translations |
| O — Outcomes | Translation accuracy measured by BLEU, Character-level F-score, and METEOR metrics; feasibility and safety for healthcare contexts |
Bottom Line
This pilot study provides preliminary evidence comparing translation tools for medical summaries in Australian healthcare contexts. While traditional machine translation tools showed better surface-level performance, large language models demonstrated advantages in semantic similarity for some languages. The study's major limitation is its small scale (only two simulated summaries), which severely limits generalizability. Critical gaps include lack of clinical accuracy assessment, patient safety evaluation, and real-world validation. The findings suggest potential utility but emphasize the essential need for human oversight and domain-specific evaluation. Clinicians should await larger, more comprehensive studies before implementing these tools in practice. The research direction is promising for addressing healthcare equity in multicultural Australia, but substantial further work is needed to establish safety and efficacy standards.
Key Findings
P Value: Not reported
Effect Size: Traditional MT tools outperformed LLMs on surface-level metrics
Primary Outcome: Translation performance varied across languages, tools, and summary complexity
Nnt Or Sensitivity: LLMs showed relative strengths in semantic similarity for Vietnamese and Chinese
Confidence Interval: Not reported
Clinical Application
Limited by small sample size and simulated nature; requires further validation in real clinical settings Highly relevant to Australian healthcare given focus on Arabic, Chinese, and Vietnamese - the most common non-English languages. Aligns with cultural safety requirements and TGA digital health guidelines Healthcare providers serving culturally and linguistically diverse populations in Australia
Abstract
BACKGROUND: Translation of medical consultation summaries is essential for equitable health care communication in culturally and linguistically diverse populations. While machine translation (MT) tools and large language models (LLMs) are widely accessible, their feasibility and safety for health care contexts remain underexplored. OBJECTIVE: This pilot study investigates the feasibility and limitations of using LLMs and traditional MT tools to translate medical consultation summaries from English into the most common languages other than English spoken in Australia-Arabic, Chinese (simplified written form), and Vietnamese. METHODS: Two simulated summaries-a simple patient-facing summary and a complex clinician-oriented interprofessional letter-were translated using 3 LLMs (GPT-4o, Llama-3.1, and Gemma-2) and 3 MT tools (Google Translate, Microsoft Bing Translator, and DeepL). Translations were benchmarked against professional third-party interpreter translations using Bilingual Evaluation Understudy, Character-level F-score, and Metric for Evaluation of Translation with Explicit Ordering metrics. RESULTS: The translation performance varied across languages, tools, and summary complexity when assessed using automatic evaluation metrics. Traditional MT tools outperformed LLMs on surface-level metrics, while LLMs showed relative strengths in semantic similarity for Vietnamese and Chinese. Arabic translations improved with complex input, suggesting morphological advantages. The metric-based evaluation highlighted feasibility but also risks, particularly in Chinese clinical contexts. CONCLUSIONS: This pilot study provides formative evidence of opportunities and limitations in applying artificial intelligence translation for health care communication. Findings underscore the importance of human oversight; domain-specific evaluation metrics; and further formative and clinical research to guide the safe, equitable use of artificial intelligence translation tools.
References
- 1.Li, A., Zhou, W., Hoda, R., Bain, C., & Poon, P. (2026). Comparing Large Language Models and Traditional Machine Translation Tools for Translating Medical Consultation Summaries: Quantitative Pilot Feasibility Study. JMIR Formative Research. https://doi.org/10.2196/85169
Related Research
The Journal of rheumatology
Examining the Role of Wearables in Inflammatory Arthritis Care: A Narrative Literature Review
3 Aug 2026
Acta obstetricia et gynecologica Scandinavica
ChatGPT in urogynecology: Comparing large language model responses to human experts
3 Aug 2026
Sleep medicine
Concordance of wearable device sleep metrics with patient-reported sleep quality: A systematic review
3 Aug 2026
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service