Benchmarking Large Language Models and Prompt Engineering Strategies in Microsatellite Instability Cancers: Evaluation Study

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: RAG strategy showed most substantial improvement in complex decision-making tasks

• Primary Outcome: LLM accuracy significantly decreased in open-ended vs multiple-choice formats, demonstrating 'scaffolding effect'

• Nnt Or Sensitivity: RAG reduced high-risk fabrications while increasing safer refusals, creating safety-utility trade-off

• Confidence Interval: Not reported

Clinical Application

Requires significant technical infrastructure for RAG implementation and ongoing knowledge base curation Relevant to Australian oncology practice; aligns with Cancer Australia guidelines for MSI testing and TGA considerations for clinical decision support tools Oncologists, pathologists, and clinical teams managing MSI cancer patients

Abstract

BACKGROUND: The reliability of general-purpose large language models (LLMs) for complex clinical tasks in specialized domains such as microsatellite instability (MSI) cancers remains critically uncharacterized. The absence of a domain-specific benchmark to evaluate and guide the optimization of their capabilities across diverse clinical tasks poses unevaluated risks to patient safety. OBJECTIVE: This study aimed to develop and validate Microsatellite Instability Cancer Benchmark (MSIC-Bench), a novel, two-tiered benchmark for MSI cancer, covering both consensus and frontier knowledge. Using this framework, we aimed to systematically assess LLM performance across various prompting strategies, identify task-specific weaknesses, and reveal effective pathways for performance improvement. METHODS: We developed MSIC-Bench, a 511-question benchmark derived from clinical guidelines and a curated knowledge base. Three state-of-the-art LLMs (GPT-4o, [OpenAI], Gemini 2.5 Pro [Google], and Claude Opus 4 [Anthropic]) were evaluated using 4 prompting strategies, including vanilla, chain-of-thought, reflection of thoughts, and retrieval-augmented generation (RAG), under both multiple-choice and open-ended modalities. Performance was assessed on accuracy, safety (honesty), error composition, and token usage. RESULTS: LLMs demonstrated a significant "scaffolding effect," with accuracy dropping substantially in open-ended scenarios. For non-RAG strategies, the primary failure mode was an internal knowledge deficit. The integration of RAG proved to be the most effective intervention. A domain-aligned RAG strategy not only significantly improved accuracy in complex decision-making tasks but also fundamentally shifted the system's primary bottleneck from knowledge deficits to retrieval failures. In terms of safety, RAG induced a favorable shift from high-risk fabrications to safer refusals, though this introduced a safety-utility trade-off in the form of false refusals. Notably, our hybrid-RAG configuration, which combined both knowledge sources, demonstrated the most robust and generalizable performance across all tasks. CONCLUSIONS: Current LLMs lack the specialized knowledge required for reliable application in MSI oncology. A well-designed RAG architecture is the pivotal intervention to address this gap. However, its success is not automatic; it transforms the nature of system failure, making retrieval precision and knowledge base quality the new critical determinants of performance and safety. Our findings establish a clear directive for developing trustworthy clinical artificial intelligence: focus must shift toward optimizing the retrieval component and curating high-quality, comprehensive knowledge sources. MSIC-Bench provides a robust framework to guide these future efforts.

References

1. 1.Zhang, Y., Song, J., Bi, C., Zheng, X., Xu, Z., Cao, D., & Shen, B. (2026). Benchmarking Large Language Models and Prompt Engineering Strategies in Microsatellite Instability Cancers: Evaluation Study. Journal of Medical Internet Research. https://doi.org/10.2196/88614