Research Appraisals
Evidence-based critical appraisals of the latest medical research, systematically evaluated using Oxford CEBM methodology.
Showing 20 appraisals
PLoS computational biology
Twelve quick tips for AI-assisted coding in science.
While AI coding tools have demonstrated potential to accelerate software development, their use in scientific computing raises critical questions about code quality and scientific validity. In this paper, we provide twelve practical tips for AI-assisted coding that balance the capabilities of AI with the demands of scientific and methodological rigor. We address how AI can be leveraged strategically throughout the development cycle with four key themes: problem preparation and understanding, managing context and interaction, testing and validation, and code quality assurance and iterative improvement. These principles serve to emphasize maintaining human agency in coding decisions, establishing robust validation procedures, and preserving the domain expertise essential for methodologically sound research. These tips are intended to help researchers harness AI's transformative potential for faster software development while ensuring that their code meets the standards of reliability, reproducibility, and scientific validity that research integrity demands.
29 July 2026
Read appraisal →Behavior research methods
Measuring complex constructs in large-scale text with computational social mixed methods
A growing convergence between social science and machine learning enables, in principle, large-scale analyses of complex social phenomena through text. Yet, approaches leveraging supervised text classification based on human-annotated data for statistical analysis often treat conceptual validity and technical performance as separate challenges, impairing measurement quality. We provide guidelines to bridge this gap in what we call computational social mixed methods pipelines across three stages: data annotation, model training, and statistical analysis. Building on best practices and our own methodological innovations, such as "Iterative Annotation" and "Training on Confident Examples", we address recurring pitfalls like unbalanced training data or stagnant model performance. We also discuss when large language models constitute a viable alternative to transformer-based classifiers. Using a case study on countering online hate, we illustrate how consequently integrating social science and machine learning expertise improves the validity and comparability of computational social science.
24 July 2026
Read appraisal →PloS one
"We'll have to see how it works": An interview study to understand collaborative practices in interdisciplinary artificial intelligence and healthcare research.
Developing artificial intelligence (AI) algorithms for healthcare is a collaborative effort, bringing data scientists, clinicians, patients and other stakeholders together. By understanding AI as 'sociotechnical' where the social and the technical nature of the work and the models are inseparable, we explore the AI development workflow and how stakeholders navigate the challenges and tensions of sharing and generating knowledge across disciplines. We conducted an inductive thematic analysis of 13 semi-structured interviews with participants in early stages of AI-in-healthcare research consortia in the UK. Our findings identify that participants needed to adapt both the tools used for sharing and the information communicated according to their audience, particularly when working with those with a clinical or patient perspective. We identify the novelty of participating in AI research, how AI knowledge is shared, and the inclusion of clinician and patient stakeholder perspectives as key areas within collaborative AI practices in healthcare. These findings highlight that bringing AI into the mix can introduce new obstacles to interdisciplinary work.
24 July 2026
Read appraisal →BMJ health & care informatics
Benchmarking large language models for de-identification of electronic health record notes
OBJECTIVES: The rapid evolution of large language models (LLMs) and their growing application in clinical text processing have created an urgent need for reliable de-identification mechanisms. While LLMs show promise in identifying sensitive health information (SHI), their capabilities require rigorous evaluation. This study aims to conduct a comprehensive benchmarking analysis of various LLM-based, traditional rule-based and hybrid de-identification methods. METHODS: Our benchmark analysis used five datasets (i2b2-2006, MIMIC-2008, i2b2-2014, i2b2-2016 and OpenDeID v1) from different countries. We developed three baseline and eight LLM-based models. The experimental setup encompassed nine different settings using various combinations of training and testing sets to assess model robustness and cross-dataset performance. RESULTS: In the baseline models, the approach trained on the combined corpus of all five datasets (setting 3) significantly outperformed the other settings, achieving a strict F1 micro-average score of 0.8172. Regarding LLM-based models, the supervised fine-tuning approach using the same combined configuration (setting 9) achieved the highest performance with a strict F1 score of 0.9447. DISCUSSION: The harmonisation of corpora ensured standardised data formatting and SHI management across five diverse datasets, highlighting the necessity for uniform categorisation to enhance the reliability of de-identification results. CONCLUSIONS: Our findings indicate that while fine-tuned LLMs offer superior accuracy, the observed performance variability across heterogeneous electronic health record sources poses significant technical challenges. Real-world implementation must address these inconsistencies to overcome the ethical and technical hurdles associated with deploying LLMs for handling sensitive health data.
23 July 2026
Read appraisal →Supportive care in cancer : official journal of the Multinational Association of Supportive Care in Cancer
Patient experiences of tissue donation and digital consent support in primary craniospinal tumour research.
PURPOSE: Requests for tissue donation for research are often made at times of heightened vulnerability, particularly around diagnosis and surgery. This study explored patient experiences of tissue donation discussions, perspectives on consent, and the acceptability of digital decision support in primary craniospinal tumour research. METHODS: A UK national online cross-sectional survey was conducted with 50 adults with a primary brain tumour or spinal sarcoma. The survey was developed with patient and public involvement; six patient contributors reviewed the initial questionnaire before launch. Descriptive statistics summarised closed responses, and open-text comments were grouped descriptively to contextualise quantitative findings. Reporting was informed by STROBE guidance. RESULTS: Just over half of participants reported being invited to donate tissue for research (26/50, 52%). Respondents strongly preferred tissue donation to be discussed at or after a clinic appointment, and none selected the day of surgery as the preferred time. Among invited respondents, most reported that information was easy to understand (22/26, 85%), that they had an opportunity to ask questions (23/25, 92%), and that they had sufficient time to consider the decision (23/26, 88%). Sixteen of 26 invited respondents (62%) discussed the decision with family or friends; among invited respondents who had not done so, 7/10 (70%) would have liked the opportunity. Interest in a secure digital adjunct was high (46/49, 94%). CONCLUSION: Overall experience was generally positive, but the data identify specific, practical opportunities to strengthen consent support in rare craniospinal tumour pathways, including appropriate timing, clear and revisitable information, opportunities for question-asking, and resources that support family-inclusive decision-making.
19 July 2026
Read appraisal →Philosophical transactions. Series A, Mathematical, physical, and engineering sciences
Predictability-Computability-Stability workflow for veridical data science in the age of artificial intelligence
Data science is a pillar of artificial intelligence (AI), which is transforming nearly every domain of human activity, from the social and physical sciences to engineering and medicine. While data-driven findings in AI offer unprecedented power to extract insights and guide decision-making, many are difficult or impossible to replicate. A key reason for this challenge is the uncertainty introduced by the many choices made throughout the data science life cycle (DSLC). Traditional statistical frameworks often fail to account for this uncertainty. The Predictability-Computability-Stability (PCS) framework for veridical (truthful) data science (VDS) offers a principled approach to addressing this challenge throughout the DSLC. This paper presents an updated and streamlined PCS workflow, tailored for practitioners and enhanced with guided use of generative AI (GenAI). We include a running example to display the PCS framework in action and conduct a related case study that showcases the uncertainty in downstream predictions caused by judgement calls in the data cleaning stage. This article is part of the theme issue 'Statistical workflow'.
30 May 2026
Read appraisal →BMJ open
Assessment of patient preferences in digital health trials: procedure development and implementation in a preference-based trial
OBJECTIVES: To develop and evaluate a standardised, fully automated assessment procedure for patient treatment preferences suitable for use in randomised controlled trials (RCTs) with remote or digital health interventions. DESIGN: Multiphase developmental study followed by implementation in a fully randomised preference trial (FRPT). SETTING: University and outpatient cancer care settings in Switzerland; implementation in RCTs with a remote digital health intervention. PARTICIPANTS: 50 medical students for pilot testing; 11 patients with cancer for qualitative testing; 124 adult patients with cancer experiencing fatigue or sleep problems in an FRPT with a remote digital health intervention. PRIMARY AND SECONDARY OUTCOME MEASURES: Primary outcome: feasibility and applicability of a standardised digital preference assessment procedure. SECONDARY OUTCOMES: distribution and strength of treatment preferences, reasons for preference. RESULTS: A procedure with three items for assessing treatment preferences was developed and refined based on pilot testing, stakeholder meetings and patient interviews. The stepwise procedure ensured that study participants with an initially unclear preference were finally able to report their preference. The majority of students (91.7%) indicated their preference already in the first assessment step. Similar in the RCT phase, 110 of 124 participants (88.7%) indicated their preference in the first step. Preference drivers included interest, perceived ease and conceptual appeal in addition to expected treatment benefits. CONCLUSIONS: Our fully automated standardised preference assessment with three items appears feasible in trials with digital health interventions. Future trials with two active treatment conditions should consider implementation of such a preference assessment to evaluate non-specific treatment effects driven by the match of patients' preferences with the respective treatment. TRIAL REGISTRATION NUMBERS: DRKS00035596; DRKS00035541; SNCTP000006209.
28 May 2026
Read appraisal →Studies in health technology and informatics
Anonymization of Electronic Health Records by the Use of Dimensionality Reduction Techniques
Electronic health records (EHR) mostly consist of highly sensitive data, whose sharing is highly regulated due to privacy and security concerns. To address privacy protections, de-identification and anonymization techniques offer a more secure way of data transmission to ensure patient's privacy. Principal components analysis (PCA) has emerged as an anonymization technique. The applicability of more diverse and complex dimensionality reduction (DR) techniques such as Factor Analysis of Mixed Data (FAMD), or Autoencoder (AE), remains largely unexplored. The objective of this study is an investigation into the potential of FAMD and AE as anonymizers for EHR. The goal of this study is to apply advanced DR methods on three EHR datasets of varying sizes. Subsequently, a supervised prediction task is performed on the anonymized EHR using an artificial neural network (ANN), and the performance of the prediction task is compared to the performance of the original, non-anonymized EHR. The findings indicate that the use of FAMD and AE in the anonymization of EHR results in a comparable performance to that of the original, non-anonymized EHR in a downstream prediction task. This study thus indicates the potential of advanced DR as an anonymization technique for EHR, thereby underscoring the necessity for further investigation.
24 May 2026
Read appraisal →Lupus science & medicine
Can disease activity and treatment responses be captured by a core set of domains in SLE clinical trials? An analysis of phase III belimumab trial data.
OBJECTIVE: Outcome measures used in SLE randomised controlled trials (RCTs) have impeded trial success and drug approval. In an ongoing global project to develop a novel outcome measure for SLE RCTs, a 'core set' of domains with which to determine treatment response was chosen via consensus. We investigated the impact of restricting disease activity assessment to this core set on responder classification in RCTs. METHODS: Using pooled baseline and week 52 data from two phase III SLE RCTs (Belimumab in Subjects with Systemic Lupus Erythematosus (BLISS)-52, NCT00424476; BLISS-76, NCT00410384), we divided items from the British Isles Lupus Assessment Group (BILAG) disease activity index into core and non-core sets. The core set included 30 of 86 BILAG items across five domains: mucocutaneous (rash, alopecia, mucosal ulcers), haematological (thrombocytopenia, haemolytic anaemia), arthritis, serositis and nephritis. We analysed baseline disease activity and treatment response at week 52. RESULTS: We analysed 1526 patients. At baseline, the core set captured the majority of disease activity captured using the full BILAG (1426/1526 (93.5%) of patients). Cutaneous vasculitis, non-haemolytic anaemia, leucopenia, dyspnoea and fatigue were the most common non-core set manifestations present. At week 52, about 1278/1426 (89.5%) patients demonstrated concordant treatment responses measured with the core set only versus all BILAG items. Discordance was again driven by cutaneous vasculitis, non-haemolytic anaemia, leucopenia and fatigue. CONCLUSIONS: Limiting the classification of treatment response to a core set of domains has the potential to simplify SLE RCT endpoints without a significant negative impact on patient inclusion or responder classification.
23 May 2026
Read appraisal →Chinese medical journal
Estimating minimal, moderate, and marked clinically important differences for patient reported outcome measure in randomised controlled trial
BACKGROUND: Patient-reported outcome measures (PROMs) are increasingly used in clinical trials, yet thresholds that define clinically important difference (CID) within patients and treatment effects beyond placebo remain inconsistently reported, limiting its utility in clinical research and practice. Using chronic pelvic pain and its core PROM as a worked example, the study aims to present a structured framework for CID thresholds that separates within-group change from between-group difference in treatment effect, each further stratified as minimal, moderate, and marked. METHODS: We used data from a multicenter randomized controlled trial of acupuncture vs . sham acupuncture in 440 men with chronic prostatitis (CP)/chronic pelvic pain syndrome (CPPS) conducted from October 2017 to April 2019. National Institutes of Health Chronic Prostatitis Symptom Index (NIH-CPSI) served as the PROM and the Patient Global Impression of Change (PGI-C) as the anchor. CIDs were estimated using a linear mixed-effects model to assess changes in NIH-CPSI scores across PGI-C categories. RESULTS: At the end of treatment, 12.3% (51/414) participants reported no change, 34.1% (141/414) slight improvement, 32.1% (133/414) moderate improvement, and 19.6% (81/414) marked improvement; and 1.9% (8/414) reported worsening. The estimated Clinically important within-group difference (CIDs within-group ) in NIH-CPSI total score was 4.6 points for minimal, 6.0 points for moderate, and 7.3 points for marked improvement. The corresponding Clinically important between-group difference (CIDs between-group ) were 1.4, 2.7, and 4.1 points, respectively. CONCLUSIONS: In this methodological study, we proposed a pragmatic and scalable operational framework to derive thresholds for minimal, moderate, and marked CIDs. By explicitly distinguishing CIDs within-group and CIDs between-group , the approach clarifies two frequently conflated terms that serve distinct purposes in interpretation and decision-making. This framework offers benchmarks to strengthen inference on treatment response, trial design, and guideline decision thresholds.
21 May 2026
Read appraisal →International journal of medical informatics
Harmonizing patient-reported outcome measures for nasal complaints using traditional and machine learning methods
BACKGROUND: Nasal obstruction measurement instruments are widely used in the field of nasal surgery. There are various scales that measure nasal obstruction and they differ regarding the number of items, their wording, and the type of response options. In order to pool the data and analyze it together, it is necessary to harmonize it so that we can compare participants' nasal obstruction scores irrespective of instrument they filled in. Data harmonization is still not used in the field of nasal obstruction assessment. THE AIM: The aim of this study was to find the best harmonization method in terms of predicting the scores on a target instrument based on the scores from another instrument as precise as possible in the case of four different nasal complaints instruments. A method was sought to find a transformation of scores on the NOSE, Utrecht-Q and SCHNOS that makes them equivalent to ENFAS scores. METHODS: A total of 1324 unique patients completed all four measurement instruments. We tried linear equating, Item Response Theory (IRT), and the following machine learning methods: linear regression, random forest regression, support vector machine regression, and neural network. We used the root-mean-square error (RMSE) of differences between predicted and observed scores to evaluate the quality of harmonization in 5-fold cross-validation. RESULTS: The ML methods gave overall the best results (the lowest RMSEs) and outperformed IRT (which is considered as a common choice for data harmonization in psychometrics). CONCLUSION: The ML methods led to the best quality of the results, confirming their strong potential for data harmonization. This study shows that next to linear equating and IRT that are commonly used for data harmonization, we can also use ML methods for the same purpose and, by doing so, to even increase the quality of the harmonization in certain use cases.
6 May 2026
Read appraisal →Journal of medical systems
Automated data extraction from electronic medical records for pragmatic clinical trials
Data collection in randomized trials is expensive and labor intensive. With the rise in ongoing pragmatic trials, the use of electronic medical records (EMR) as a source of data has increased. Although potentially faster and cheaper, EMR use can lead to errors. Therefore, to ensure accurate data collection and to avoid systematic errors we performed a study comparing automated data extraction (ADE) with manual data extraction (MDE). We performed a retrospective cohort study to compare the accuracy of ADE using Structured Query Language with MDE by blinded physicians from our EMR. We tested the interrater agreement and intraclass correlation coefficient of clinical baseline data and outcomes of a random sample of 30 patients admitted to the ICU, on mechanical ventilation, requiring opioids for analgosedation for an upcoming pragmatic clinical trial. Key data compared included, but not limited to, patient's demographics, laboratory and vital signs, daily morphine milligram equivalent (MME), days alive and free of mechanical ventilation, days alive and free of hospitalization, days alive and free of ICU, days alive and free of vasopressors, and death. Among 238 patients screened over 1-month period, 72 fulfilled inclusion criteria and 30 were randomly selected to be included in the evaluation. We blindly collected 1320 baseline data, 2160 categorical outcomes and 705 continuous outcomes for a total of 4185 data points. The intraclass correlation coefficient and the Cohen's Kappa were perfect or almost perfect for all data, including outcomes such as daily MME, days alive and free of mechanical ventilation, days alive and free of ICU and days alive and free of hospital with p < 0.001. Among all rechecked data, the ADE was correct in 53 (77.9%) of cases, while MDE in 15 (22.1%). The inaccurate data collected by ADE accounted for 0.36% of the total data-points. The performance of ADE had almost perfect agreement for all outcomes and when rechecking for disagreements, it was more accurate than MDE.
5 May 2026
Read appraisal →The British journal of ophthalmology
Use of patient-reported outcomes in ophthalmology clinical trials between 2014 and 2023.
BACKGROUND/AIMS: Patient-reported outcomes (PROs) evaluate health and functional status, and PRO measures (PROMs) are standardised tools for measuring PROs. Together, they provide valuable insights into treatment efficacy, safety and practicality not captured by traditional clinical endpoints. This cross-sectional analysis with a systematic search component aims to investigate the use, interpretation and reporting of PROs and PROMs in ophthalmic randomised controlled trials (RCTs). METHODS: Ophthalmic RCTs published in the top 10 highest impact factor ophthalmic and medical journals between 2014 and 2023 were systematically reviewed. The frequency of PRO inclusion and adherence to Consolidated Standards of Reporting Trials (CONSORT) PRO guidelines was assessed. The relationship between PRO utilisation and study-level and journal-level characteristics was explored with multivariable regression. RESULTS: Among 9436 records screened, 333 RCTs met eligibility criteria. Of these, 87 (26.1%) included PROs, and 28 (8.4%) used them as primary outcomes. Most studies (83/87, 95.4%) leveraged PROMs, with ophthalmology-specific tools predominating (73.5%). Minimal important differences (MIDs) were rarely used (2.3%) for PRO interpretation. At least 8/13 CONSORT PRO Extension items were reported in 33.3% studies, and trials with primary PRO endpoints had better adherence (p<0.001). PRO utilisation was less likely in trials with lower 5-year journal impact factor (adjusted OR (aOR) 0.99, 95% CI 0.98 to 1.00, p=0.037) and pharmaceutical compared with health service interventions (aOR 0.15, 95% CI 0.03 to 0.67, p=0.013). CONCLUSIONS: PRO integration and interpretation remain limited in high-impact ophthalmic RCTs, despite offering a meaningful adjunct to objective endpoints. Future trials should adopt validated and condition-specific PROMs and establish MIDs to enhance interpretation.
26 Apr 2026
Read appraisal →Journal of the American College of Cardiology
Extension Reports in Clinical Trials: A Systematic Review and Consequent Guidance
BACKGROUND: Following the primary publication of a clinical trial, questions of longer-term efficacy and safety remain unanswered. Investigators often have sequel publications based on extended follow-up; yet, no guidelines exist for such extension reports. OBJECTIVES: The purpose of this analysis was to assess the characteristics of extension reports of cardiovascular trials published in 7 high-impact medical journals. METHODS: We searched MEDLINE to identify cardiovascular trial extension papers published between January 2019 and October 2023 in 7 high-impact medical journals. Extension reports were defined as those presenting additional follow-up data after publication of the primary results. Papers were screened by 2 reviewers before extracting data using a standard proforma. RESULTS: After screening 1,388 papers, 69 extension reports across 64 trials were identified. A variety of cardiovascular topics were covered, coronary heart disease (n = 20) being the most common. Interventions were medical devices, drugs, surgery, or treatment strategies. Duration of follow-up varied considerably, most common being an original 1-year report (n = 20) with extension at 2, 3, 5, or 10 years. Nearly all extension reports had a superiority hypothesis (n = 68). Most extensions (n = 53) examined the same primary outcome as the original publication. There were also 7 nonrandomized open-label extensions, which suffered from the lack of a concurrent control arm. In 18 reports, treatment crossovers were permitted during the extension, and in 16 cases, alternatives to analysis by intention to treat were utilized. One-half of the extension studies were prespecified (n = 38), and only 10 had dedicated protocols or statistical analysis plans. No study used multiplicity corrections to control type I error across the totality of evidence. Landmark analyses were common (n = 39) which, while descriptively useful, were not based on the original randomized groups. Considerable loss to follow-up (>10%) occurred in 10 of 50 extension reports with clear reporting of this issue. For some trials, the conclusions regarding treatment effects were altered by the extended follow-up. In a previous survey of cardiovascular trials published in the New England Journal of Medicine, Lancet, and Journal of the American Medical Association in 2019, we identified a total of 23 of 84 trials involving extensions in some form. CONCLUSIONS: This first systematic investigation into the reporting of extension studies found a great diversity in practice, noting several issues that need more adequate consideration. Considerations are provided on what trialists should consider when planning, conducting, and reporting extension studies.
25 Apr 2026
Read appraisal →Trials
The wood and the trees: estimands in cluster randomised trials.
BACKGROUND: The estimand framework was introduced into guidance on good clinical practice to address a variety of shortcomings and ambiguities in the reporting of trials, including the use of terms such as "intention to treat" and the handling of non-adherence to treatment. The framework was primarily grounded in individually randomised trials, and some thorny issues still cloud understanding of its application to cluster randomised trials. ESTIMANDS IN CLUSTER RANDOMISED TRIALS: This commentary addresses some of the challenges in thinking about the estimands behind cluster randomised trials. These challenges include informative cluster size-the possibility that a treatment effect may be modified by the size of the cluster. In particular, we consider the perspectives of different actors-a population-level decision maker or politician, a cluster manager, and a patient-and examine possible estimands for each, and how they differ. CONCLUSIONS: In the cluster randomised trial context, the estimand framework can be complex to navigate. Different perspectives lead to different estimands. We caution against abandoning careful statistical modelling. This is particularly true in the presence of informative cluster size, where modelling any interaction between cluster size and treatment effect could be useful from a number of perspectives.
22 Apr 2026
Read appraisal →Journal of medical Internet research
Artificial Intelligence for Identifying Patient-Reported Outcome and Experience Measures in Oncology: Retrospective Cross-Sectional Study Using ClinicalTrials.gov
BACKGROUND: Traditional health care systems have evolved to increasingly recognize patients' perspectives as key to improving the quality of care, especially in oncology. Hence, patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) can enhance patient-health care provider communication while facilitating individualized care. This tailored approach improves patient outcomes and highlights the need for research methods to account for variability in patient experiences across diverse sociodemographic and clinical backgrounds while proposing artificial intelligence (AI) to automate and accelerate the identification of PROMs and PREMs. OBJECTIVE: This study aimed to estimate the proportion of clinical studies including PROMs or PREMs, using either a traditional expert-based identification method or an AI-enriched approach. METHODS: In a retrospective cross-sectional study using the ClinicalTrial.gov database, we focused on oncology studies between 2012 and 2022. Two methods were used to identify PROMs and PREMs: (1) a traditional expert-based method, where an algorithm identified PROMs and PREMs from a list of 346 oncology-specific PROMs and PREMs (extracted from the PROQOLID database, Mapi Research Trust) and/or 11 PROM- and PREM-specific terms, and (2) an AI-enriched method using a bidirectional encoder representations from transformers model, trained on 2399 outcomes labeled by experts. To evaluate algorithm performance, results were compared with expert decisions. Studies were classified by PROM/PREM use, analyzed using logistic regression to identify drivers, and a named entity recognition model identified frequently used measures. RESULTS: A total of 24,491 studies were included. According to the traditional expert-based algorithm, 7549 (31%) studies used at least one PROM or PREM, as compared to 8029 (33%) studies identified by the AI-enriched algorithm, increasing from 2012 to 2022 (P<.001). With 90% (95% CI 88%-92%) accuracy, the AI-enriched algorithm outperformed (P<.001) the expert-based algorithm (84%, 95% CI 82%-86%) in identifying PROMs and PREMs. Breast and digestive cancers accounted for nearly 50% of all oncology studies using PROMs and PREMs, with the European Organization for Research and Treatment of Cancer Quality of Life Questionnaire-Core 30 being the most frequently used. As expected, trials in phases 2 to 4 more frequently included PROMs and PREMs than preclinical or early phase 1 studies (odds ratio [OR] 1.8, 95% CI 1.1-2.8 for phase 2; OR 3.6, 95% CI 2.3-5.8 for phase 3; and OR 2.6, 95% CI 1.6-4.4 for phase 4). In observational studies, cross-sectional and prospective studies incorporated more PROMs and PREMs than retrospective studies (OR 4.6, 95% CI 3.3-6.4 and OR 3.2, 95% CI 2.5-4.1, respectively). CONCLUSIONS: Our study shows that an AI-enriched algorithm outperforms traditional expert methods in identifying PROMs and PREMs in oncology. Combining expert-labeled data with AI enables scalable, automated trial monitoring, supports efficient research, informs stakeholders, enhances patient-centered decisions, and can be extended to other diseases and databases.
17 Apr 2026
Read appraisal →Journal of medical Internet research
Beyond Missingness: Systematizing Methods for Comprehensive Data Fitness Assessment in Clinical Research
Secondary use of clinical data offers unprecedented opportunities to rapidly conduct large-scale research and improve patient care. However, incomplete understanding of data quality requirements for a study often causes significant delays in executing analyses and validating results. Current practice has largely followed 2 paths. First, multi-institutional networks have developed general data quality programs, but these are typically tied to unique network characteristics and do not address study-specific requirements well. Second, models have been proposed to formalize the requirements for data fitness analyses without extending to the methods needed to meet these requirements. More recently, tools have been developed to conduct cohort-centric screening, focusing on generally applicable structural checks such as missingness or facial implausibility. These provide a first level of information but incompletely capture the fitness requirements of an analysis. In turn, investigators conduct per-study exploratory analyses, but these efforts are typically ad hoc and partially reported, which can hinder reproducible science and delay advances in patient care. Analogously to advances over the past decade in data modeling and reproducible analytics, there is a need for a more systematic, capable approach to study-specific data quality assessment (SSDQA). We discuss such a model, which guides improved SSDQA design and implementation, including metadata for consistent annotation and reporting of data quality assessment results. The model integrates theoretical principles of data quality testing with pragmatic considerations of application to clinical data, providing a consistent approach to specifying data quality assessment checks. Additionally, it proposes to regularize check application through a standard set of options. The SSDQA model builds on current practice, providing a path toward more complete, sound, and reproducible assessments. These characteristics foster multidisciplinary collaboration to identify data quality issues that, in turn, inform decisions about study design and provide important context that has a bearing on adoption of results.
17 Apr 2026
Read appraisal →Computers in biology and medicine
Forecasting-based biomedical time-series data synthesis for open data and robust AI
The limited data availability due to strict privacy regulations and significant resource demands severely constrains biomedical time-series AI development, which creates a critical gap between data requirements and accessibility. Synthetic data generation presents a promising solution by producing artificial datasets that maintain the statistical properties of real biomedical time-series data without compromising patient confidentiality. While GANs, VAEs, and diffusion models capture global data distributions, forecasting models offer inductive biases tailored for sequential dynamics. We propose a framework for synthetic biomedical time-series data generation based on recent forecasting models that accurately replicates complex electrophysiological signals such as EEG and EMG with high fidelity. These synthetic datasets can be freely shared for open AI development and consistently improve downstream model performance. Numerical results on sleep-stage classification show up to a 3.71% performance gain with augmentation and a 91.00% synthetic-only accuracy that surpasses the real-data-only baseline.
16 Apr 2026
Read appraisal →Journal of medical Internet research
The Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB) for the Evaluation of Medical Large Language Model Question-Answer Studies: Development and Pilot Validation
BACKGROUND: Despite the transformative potential of large language models (LLMs) in health care, the rapid development of these tools has outpaced their rigorous evaluation. While artificial intelligence-specific reporting guidelines have been developed to address standardized reporting of artificial intelligence studies, there is currently no specific tool available for risk of bias assessment of LLM question-answer (QA) studies. Existing risk-of-bias tools for medical research are not well suited to the unique challenges of evaluating LLM-QA studies, which creates a critical gap in assessing their safety and effectiveness. OBJECTIVE: This study aims to develop the Alberta Quality Assessment Tool: Risk of Bias (AQAT:RoB) for LLM-QA studies to systematically evaluate the validity and risk of bias in LLM-QA studies. METHODS: We conducted 2 literature reviews. The first was on quality assessment tools for LLM-QA studies, and the second was on LLM-QA studies, which informed the first draft of the AQAT:RoB. The draft AQAT:ROB was further refined through a prespecified iterative process of modified Delphi, consensus meeting, and validation. The first Delphi process occurred between May 1 and May 20, 2025, and the first consensus meeting was held on May 22. The first round of validation was completed by 4 evaluators, who were not part of the consensus meeting, on 16 randomly selected studies. As this first round of validation surpassed our a priori threshold of ≥80% agreement and a Cohen κ of ≥0.61 between evaluators, no further rounds of development and validation were undertaken. A second Delphi process occurred between February 20 and February 23, 2026, to vote on postpilot changes in response to peer review. RESULTS: The AQAT:RoB consists of 5 high-level domains (Questions, Reference Answers, LLM Answers, Evaluators, Outcomes). These domains are subdivided into 9 subdomains. Each subdomain includes at least one "Support for Judgment" and at least one "Type of Bias" and is to be rated "low," "high," or "unclear" for risk of bias. A pilot evaluation was completed by internal validators who were not part of the consensus discussion and were asked to complete the AQAT:RoB form for each assigned study. Each of the 16 studies was evaluated by 2 evaluators independently. Pilot validation showed a percent agreement of 86.1% and a Cohen κ of 0.70 between assessors. CONCLUSIONS: The AQAT:RoB demonstrates promising initial reliability for assessing the validity or risk of bias in LLM-QA studies. The tool will benefit from future refinements, external validation, and periodic updates to keep pace with evolving technology.
10 Apr 2026
Read appraisal →Philosophical transactions. Series A, Mathematical, physical, and engineering sciences
The need for verification in artificial intelligence-driven scientific discovery
Artificial intelligence (AI) is transforming the practice of science. Machine learning (ML) and large language models (LLMs) can generate hypotheses at a scale and speed far exceeding traditional methods, offering the potential to accelerate discovery across diverse fields. However, the abundance of hypotheses introduces a critical challenge; without scalable and reliable mechanisms for verification, scientific progress risks being hindered rather than advanced. In this article, we trace the historical development of scientific discovery, examine how AI is reshaping established practices for discovery and review the principal approaches, ranging from data-driven methods and knowledge-aware neural architectures to symbolic reasoning frameworks and LLM agents. While these systems can uncover patterns and propose candidate laws, their scientific value ultimately depends on rigorous and transparent verification, which we argue must be the cornerstone of AI-assisted discovery. This article is part of the discussion meeting issue 'Symbolic regression in the physical sciences'.
10 Apr 2026
Read appraisal →