Assessment of CDASI scoring by a multimodal large language model: a comparative study with expert assessors.
Clinical Snapshot
PICO Framework
| P — Population | Published dermatomyositis (DM) cases (n=30) with available clinical images, retrospectively analysed |
| I — Intervention | Multimodal large language model (Claude v3.5 Sonnet) scoring the Cutaneous Dermatomyositis Disease Area and Severity Index (CDASI) using structured prompting based on CDASI definitions |
| C — Comparator | Two expert rheumatologists independently scoring CDASI activity and damage domains |
| O — Outcomes | Agreement between LLM and expert raters measured by intraclass correlation coefficients (ICC) for CDASI activity and damage domains; evaluation time per case |
Bottom Line
This small retrospective study (n=30 published cases) evaluated whether Claude v3.5 Sonnet, a multimodal large language model, could score the CDASI — a validated dermatomyositis skin activity instrument — with accuracy comparable to expert rheumatologists. The LLM achieved good agreement for global CDASI activity (ICC 0.71) and near-perfect agreement for hand-related domains (periungual ICC 1.0), but only fair agreement for damage features such as poikiloderma (ICC 0.47, 95% CI 0.10–0.73). Evaluation time was reduced by approximately 92%. However, the study has critical limitations: the sample is literature-derived and almost certainly unrepresentative of real-world DM spectrum; the sample size is too small to yield stable ICC estimates (note the very wide confidence intervals); blinding procedures are not confirmed; and the LLM prompts are not published. These findings are hypothesis-generating rather than practice-changing. Clinicians should not deploy LLM-assisted CDASI scoring outside a research context without prospective validation in consecutive patients, standardised image protocols, and regulatory clearance. Physician oversight remains essential, particularly for damage domain assessment.
Key Findings
P Value: Not reported in abstract
Effect Size: Global CDASI activity: ICC 0.71 (good agreement); Global CDASI damage: ICC 0.41 (fair agreement); Periungual changes: ICC 1.0; Global hand score: ICC 0.95; Erythema: ICC 0.61; Scaling: ICC 0.57; Erosions: ICC 0.57; Poikiloderma: ICC 0.47
Primary Outcome: Intraclass correlation coefficient (ICC) between Claude v3.5 Sonnet and expert rheumatologists for CDASI activity and damage domain scoring across 30 published DM cases
Nnt Or Sensitivity: Sensitivity/specificity not applicable (continuous agreement study). Evaluation time: LLM 42 seconds vs. expert 8.4 minutes per case (~92% reduction). ICC benchmarks: <0.50 poor, 0.50–0.75 moderate, 0.75–0.90 good, >0.90 excellent (Koo & Mae, 2016)
Confidence Interval: Global activity: 95% CI 0.45–0.86; Damage: 95% CI 0.25–0.67; Poikiloderma: 95% CI 0.10–0.73; Global hand: 95% CI 0.91–0.97; Periungual: 95% CI 1.0–1.0
Clinical Application
Technically feasible in settings with access to multimodal LLM platforms and standardised clinical photography. However, real-world implementation requires: standardised image acquisition protocols, validated structured prompts (not yet published), integration with clinical workflows, and robust physician oversight — particularly for damage domain scoring. The 92% time reduction is clinically attractive for high-volume monitoring but must be weighed against accuracy limitations in a rare, complex disease. Dermatomyositis is a rare autoimmune condition managed predominantly by rheumatologists and dermatologists in Australia. CDASI is not routinely used in all Australian centres, and there are no specific PBS or TGA regulatory frameworks governing LLM-assisted clinical scoring tools at present. The TGA's Digital Health regulatory pathway would apply to any commercial deployment of such a tool as a Software as a Medical Device (SaMD). RACGP and RACP guidelines do not currently address LLM-assisted disease activity scoring. Australian rheumatology centres with DM expertise (e.g., major teaching hospitals) may find this technology of interest for research purposes, but clinical adoption would require prospective Australian validation data, ethics approval for image processing, and compliance with the Australian Privacy Act 1988 regarding patient image data. Telehealth expansion in Australia may increase the relevance of remote, image-based disease activity monitoring tools. Adult patients with confirmed dermatomyositis requiring serial CDASI activity and damage assessment in rheumatology or dermatology settings. Results are most applicable to cases with prominent hand involvement (periungual changes, Gottron's papules) where LLM performance was strongest. Caution is warranted for chronic damage features (poikiloderma) where agreement was poor to fair.
Abstract
Large language models (LLMs) with multimodal capabilities may support automated assessment of cutaneous disease activity in dermatomyositis (DM). We evaluated the performance of Claude v3.5 Sonnet in scoring the Cutaneous Dermatomyositis Disease Area and Severity Index (CDASI) compared with expert rheumatologists. Thirty published DM cases with available clinical images were retrospectively analyzed. Two expert rheumatologists independently scored CDASI activity and damage domains. Claude v3.5 Sonnet assessed the same images using structured prompting based on CDASI definitions. Agreement was evaluated using intraclass correlation coefficients (ICCs) with a two-way random-effects model for absolute agreement. The LLM demonstrated better agreement for activity than for damage assessment. Global CDASI activity showed good agreement with expert raters (ICC 0.71, 95% CI 0.45-0.86), whereas damage reliability was lower (ICC 0.41, 95% CI 0.25-0.67). Agreement was moderate for activity-related domains including erythema (ICC 0.61), scaling (ICC 0.57), and erosions (ICC 0.57), while lower concordance was observed for chronic damage-related features such as poikiloderma (ICC 0.47, 95% CI 0.10-0.73). Hand assessments demonstrated the strongest performance, particularly for periungual changes (ICC 1.0, 95% CI 1.0-1.0) and global hand scores (ICC 0.95, 95% CI 0.91-0.97). The model also markedly reduced evaluation time compared with clinicians (42 s vs. 8.4 min per case), corresponding to an approximately 92% reduction in scoring time. Multimodal LLMs demonstrated promising agreement with expert raters for CDASI activity assessment and markedly improved efficiency, although lower reliability in damage domains reinforces the continued need for physician oversight.
This content is for educational purposes for healthcare professionals only and does not constitute clinical advice. Clinical decisions should be based on individual patient assessment, current guidelines, and appropriate specialist consultation. Editorial Standards · Privacy Policy · Terms of Service