Model and Task-Aware Test-Time Scaling Strategies for Large Language and Vision-Language Models in Medicine: Evaluation Study

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Not reported quantitatively in the abstract; directional findings only — reasoning models showed 'significant performance gains' on complex tasks; non-reasoning models showed rapid accuracy saturation

• Primary Outcome: Benchmark accuracy across textual and multimodal medical QA tasks under three test-time scaling conditions (token budget increase, sequential scaling, parallel scaling)

• Nnt Or Sensitivity: Not applicable (benchmarking study); no NNT, sensitivity, specificity, or hazard ratio reported. Key comparative finding: parallel scaling outperformed sequential scaling on easier tasks; extended sequential scaling or increased token budgets were superior for complex problem-solving

• Confidence Interval: Not reported

Clinical Application

The findings are directly relevant to the configuration of AI inference systems in clinical settings. The recommendation to match scaling strategy to task complexity (parallel for simpler tasks, sequential/extended budget for complex reasoning) is operationally actionable for AI deployment teams. However, the computational cost implications of extended sequential scaling in time-sensitive clinical environments require further evaluation. Australian relevance is moderate. The Australian Digital Health Agency (ADHA) and state health departments are actively evaluating LLM integration into clinical workflows, including My Health Record and clinical decision support. The TGA's Software as a Medical Device (SaMD) regulatory framework would apply to any clinically deployed LLM-based tool. The finding regarding vulnerability to misleading authority prompts has direct implications for Australian medicolegal frameworks and clinical governance. RACGP and specialist colleges considering AI-assisted clinical decision support should note that medically fine-tuned models may underperform on procedural/calculation tasks — relevant to medication dosing and clinical scoring tools. PBS and formulary-specific performance of these models in Australian contexts has not been evaluated. Clinicians, health informaticians, and health system administrators considering deployment of LLM or VLM-based clinical decision support tools; medical AI developers designing inference pipelines for clinical applications

Abstract

BACKGROUND: Test-time scaling has emerged as a promising method to enhance the reasoning capabilities of large language models (LLMs) and vision-language models (VLMs) during inference without additional training. While foundational studies established scaling paradigms in general domains, their applicability to the unique complexities of medical AI remains underexplored. OBJECTIVE: This study aims to conduct a comprehensive investigation of test-time scaling in the medical domain. We evaluate the impact of scaling across different model sizes and task complexities. Furthermore, we seek to identify domain-specific bottlenecks and assess model robustness against user-driven perturbations, such as misleading clinical authority. METHODS: This study evaluated a diverse set of general and medical-specific LLMs and VLMs. Experiments used five textual medical benchmarks comprising over 5500 questions and two multimodal benchmarks comprising 7000 samples. Performance was measured under three scaling conditions: increasing token budgets, iterative sequential scaling, and parallel scaling. Robustness was tested by embedding misleading hints with varying tones and levels of simulated clinical expertise into prompts. RESULTS: For nonreasoning LLMs, accuracy saturated quickly, with token usage often remaining under 500 tokens regardless of budget increases. Reasoning models demonstrated significant performance gains on complex tasks as token budgets increased. Notably, we identified distinct domain-specific behaviors. First, current VLMs showed a structural bottleneck in integrating visual clues and experienced limited benefit from token expansion. Second, medically fine-tuned LLMs excelled in clinical question answering but exhibited degraded scaling efficiency on calculation tasks compared to general-domain models. This reflects a disparity between qualitative clinical alignment and procedural logic. Third, while optimal scaling improved robustness, models exhibited a cognitive vulnerability by readily abandoning correct reasoning when confronted with misleading expert physician hints. Regarding scaling strategies, parallel scaling outperformed sequential scaling on easier tasks. Conversely, extended sequential scaling or increased budgets proved essential for complex problem-solving. CONCLUSIONS: Test-time scaling rules from general domains do not perfectly translate to medical AI. Longer reasoning is not universally beneficial. Concise reasoning with parallel scaling is optimal for simpler tasks. An extended chain of thought via sequential scaling or increased budgets is required for complex problems. Furthermore, safe clinical deployment requires addressing fundamental vision-language alignment, balancing clinical and procedural reasoning, and mitigating vulnerabilities to perceived clinical authority.

References

1. 1.Oh, G., Kim, S., Park, S., & Kim, B.-H. (2026). Model and task-aware test-time scaling strategies for large language and vision-language models in medicine: Evaluation study. Journal of Medical Internet Research. https://doi.org/10.2196/90693