Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 3 appraisals
PloS one
Textual overlap rather than domain alignment: A comparative study of fine-tuning strategies for specialised machine translation with large language models
General-purpose large language models (LLMs) may struggle in specialised machine translation, but the conditions under which fine-tuning improves translation performance remain unclear. This study compares full-parameter fine-tuning (FPFT) and parameter-efficient fine-tuning (PEFT) for Chinese-English political discourse translation using a purpose-built corpus and the Qwen3-14B model. Translation performance was assessed on three 50-item test sets using BLEU, ROUGE-L F1, METEOR, and BERTScore F1, together with BLEU pass-rate likelihood-ratio G2 tests, paired t-tests, and paired Cohen's dz for item-level score differences. The results reveal a clear contrast between unseen in-domain evaluation, maximum-overlap benchmarking, and semantically related but non-fine-tuned evaluation. On Test Set A and Test Set C, neither fine-tuning strategy produced a statistically significant BLEU pass-rate advantage over the base model, and paired tests across the continuous metrics did not show consistent fine-tuning gains. On Test Set B, which was sampled from the fine-tuning corpus, both fine-tuned models substantially outperformed the base model across all four metrics, with FPFT achieving the highest scores and PEFT providing a more computationally efficient alternative. These findings indicate that textual overlap between training and deployment data, rather than broad domain similarity alone, strongly conditions the observed benefit of fine-tuning. The study offers an empirically grounded framework for selecting fine-tuning strategies in specialised machine translation.
22 July 2026
Read appraisal →PloS one
Stylistic analysis of translated languages: A perturbation-based XAI deep learning framework
Text classification using traditional machine learning techniques has been used in natural language processing (NLP) tasks to distinguish translated from non-translated languages, with high accuracy scores indicating the distinctive style of translated languages. While deep learning (DL) has demonstrated impressive performance in terms of representation learning and capturing nuanced patterns in natural language data, DL models act as black boxes, making their results difficult to interpret. This study addresses this issue by demonstrating an explainable AI (XAI) DL framework in a case study of United Nations (UN) meetings. The framework consists of three stages: i) train a variational autoencoder (VAE) combined with BERT embeddings converted from translated and non-translated texts; ii) utilize the majority vote from three classifiers selected from a stacked ensemble to classify the VAE's latent representations; iii) implement a perturbation-based XAI method to interpret the DL model's decisions. The results indicate that the VAE-based model effectively distinguishes the two text types, with accuracy scores above 0.8. The XAI analysis reveals that interpreting the VAE-based model's decision uncovers stylistic differences between the two text types beyond superficial lexical and syntactic features. This proof-of-concept study demonstrates the potential of the XAI DL framework in other NLP studies that aim to analyze style.
9 July 2026
Read appraisal →PloS one
Extracting user profile via large language models and ontologies
Extracting user-specific profiles that include general personal information, such as hobbies, occupation, or age, is a valuable asset for systems like recommendation engines or personalized chatbots. Currently, most approaches moved toward exploiting the capabilities of large language models (LLMs) on rich languages while less-resourced languages still present significant opportunities for exploration and improvement. In this research, we present a multi-step approach for profile extraction from dialogue systems in Persian as a use case. Through an extensive set of experiments and analyses on various models, we show that LLMs struggle with these languages due to limited language-specific resources and complex linguistic structures. To address this, we propose a hybrid method that combines techniques such as slot filling, in-context learning, and ontology-based inference. Our final results demonstrate a significant improvement over current state-of-the-art models, including LLMs with few-shot examples and even their fine-tuned version. Our method achieves an F-score of 90.46, outperforming GPT-4o and Llama-3-70B by an absolute difference of 17.08 and 24.51 respectively. Our system can also detect inconsistencies in presented information in which our performance substantially exceeds the best performance of GPT-4o and Llama-3-70B with an accuracy of 92%. It is 21% absolutely better than GPT-4o and 47% better than Llama.
13 May 2026
Read appraisal →