Developing and Validating a Coding Scheme for Clinical Reasoning in History Taking Using Generative AI-Based Virtual Patients: Systematic Text Condensation Approach

Clinical Snapshot

Key Findings

• P Value: Significant positive correlations reported but specific p-values not provided

• Effect Size: High interrater reliability κ≥0.85 for all coding categories

• Primary Outcome: Development of 12-behavior coding scheme across 3 dimensions (clinical reasoning, information gathering, social interaction)

• Nnt Or Sensitivity: Coding scheme sensitivity: captures higher-order cognitive strategies effectively

• Confidence Interval: Not reported in abstract

Clinical Application

High feasibility with increasing GenAI-VP adoption; requires trained coders and technological infrastructure Relevant to Australian medical schools implementing virtual patient technology; aligns with RACGP clinical reasoning competencies and AMC graduate outcome standards Medical students in undergraduate clinical training programs using virtual patient technology

Abstract

BACKGROUND: Effective history taking helps clinicians identify key symptoms and form accurate hypotheses. Generative artificial intelligence (GenAI)-based virtual patients (VPs) are increasingly used to simulate and practice history taking. However, there is currently no straightforward approach to effectively identify students' clinical reasoning activities during these interactions, which limits the ability to provide instructional feedback. OBJECTIVE: This study aims to develop and validate a coding scheme to identify medical students' history-taking behaviors during interactions with GenAI-based VPs. METHODS: Second-year medical students (N=210) participated in 5 history-taking cases with GenAI-based VPs, yielding 1030 dialogues. Researchers applied the systematic text condensation method to these dialogue data from cases 1 to 4 to inductively develop a coding scheme and validate coding consistency. Subsequently, the dialogue data from case 5 were used to assess the correlation between the students' history-taking behaviors and their academic performance, including diagnostic accuracy, history-taking checklist scores, clinical knowledge test scores, and postencounter form scores. RESULTS: A coding scheme comprising 12 behaviors across 3 dimensions-clinical reasoning behaviors, information gathering behaviors, and social interaction behaviors-was developed with high interrater reliability (κ≥0.85). The correlation analysis revealed that key clinical reasoning behaviors, such as summarizing and integrating and logical organization, showed significant positive correlations with multiple performance metrics, underscoring their importance in fostering clinical competence. In contrast, information gathering behaviors such as specifying symptoms and routine question were associated with clinical knowledge and thoroughness of history taking but were less predictive of diagnostic accuracy. CONCLUSIONS: This study developed a reliable, theory-informed coding scheme that can identify students' questioning behaviors during history taking with GenAI-based VPs. The scheme effectively captures higher-order cognitive strategies and provides valuable insights into the development of clinical reasoning in medical students. This approach offers a scalable and efficient way to integrate real-time feedback into future medical education, fostering personalized learning and advancing competency-based assessments in clinical training.

References

1. 1.Chen, N., Tang, L., Liu, Y., Lin, C., Li, Z., Shi, C., Xia, M., Gasevic, D., Gasevic, D., Zheng, J., Fan, Y., & Li, X. (2026). Developing and validating a coding scheme for clinical reasoning in history taking using generative AI-based virtual patients: Systematic text condensation approach. JMIR Medical Education. https://doi.org/10.2196/84347