Textual overlap rather than domain alignment: A comparative study of fine-tuning strategies for specialised machine translation with large language models

Clinical Snapshot

Key Findings

• P Value: Not statistically significant on Test Sets A and C across continuous metrics; statistically significant BLEU pass-rate and metric advantages on Test Set B for both fine-tuned models

• Effect Size: Cohen's dz reported for paired comparisons; specific values not available in abstract. FPFT achieved highest scores on Test Set B; PEFT provided a computationally efficient alternative with comparable directional gains.

• Primary Outcome: Fine-tuning benefit is strongly conditioned by textual overlap between training and test data, not by broad domain similarity. On Test Set B (sampled from fine-tuning corpus), both FPFT and PEFT substantially outperformed the base model across all four metrics. On Test Sets A (unseen in-domain) and C (semantically related, non-fine-tuned domain), neither fine-tuning strategy produced statistically significant or consistent advantages over the base model.

• Nnt Or Sensitivity: Not applicable (computational linguistics study; no NNT, sensitivity, specificity, or hazard ratio relevant)

• Confidence Interval: Not reported in available data

Clinical Application

The finding that fine-tuning gains are driven by textual overlap rather than domain similarity has practical implications for any organisation considering fine-tuning LLMs for specialised translation: deployment data must closely resemble training data for fine-tuning to confer benefit. PEFT is identified as a computationally efficient alternative to FPFT when overlap conditions are met. No direct relevance to Australian PBS, TGA, or RACGP clinical guidelines. Indirect relevance exists for Australian health services considering LLM-based translation tools for multilingual patient communication, clinical documentation translation, or health literacy resources in Chinese-speaking communities. The findings caution against assuming that domain-level fine-tuning alone will improve translation quality in deployment contexts that differ textually from training data — a consideration relevant to any Australian health technology procurement involving LLM translation capabilities. Not directly applicable to a clinical patient population. Potentially relevant to developers of clinical NLP tools, health translation services, or medical document translation systems considering LLM fine-tuning strategies.

Abstract

General-purpose large language models (LLMs) may struggle in specialised machine translation, but the conditions under which fine-tuning improves translation performance remain unclear. This study compares full-parameter fine-tuning (FPFT) and parameter-efficient fine-tuning (PEFT) for Chinese-English political discourse translation using a purpose-built corpus and the Qwen3-14B model. Translation performance was assessed on three 50-item test sets using BLEU, ROUGE-L F1, METEOR, and BERTScore F1, together with BLEU pass-rate likelihood-ratio G2 tests, paired t-tests, and paired Cohen's dz for item-level score differences. The results reveal a clear contrast between unseen in-domain evaluation, maximum-overlap benchmarking, and semantically related but non-fine-tuned evaluation. On Test Set A and Test Set C, neither fine-tuning strategy produced a statistically significant BLEU pass-rate advantage over the base model, and paired tests across the continuous metrics did not show consistent fine-tuning gains. On Test Set B, which was sampled from the fine-tuning corpus, both fine-tuned models substantially outperformed the base model across all four metrics, with FPFT achieving the highest scores and PEFT providing a more computationally efficient alternative. These findings indicate that textual overlap between training and deployment data, rather than broad domain similarity alone, strongly conditions the observed benefit of fine-tuning. The study offers an empirically grounded framework for selecting fine-tuning strategies in specialised machine translation.

References

1. 1.Yang, L., Zhu, J., & Zhang, Z. (2026). Textual overlap rather than domain alignment: A comparative study of fine-tuning strategies for specialised machine translation with large language models. PLoS ONE. https://doi.org/10.1371/journal.pone.0352256