Multimodal deep learning fusion for automatic pain detection in cancer patients

Clinical Snapshot

Key Findings

• P Value: Not reported; no formal statistical significance testing between architectures described

• Effect Size: Bimodal autoencoder: mean accuracy approximately 80% across folds. Pairwise transformer with cross-modal attention: similar performance (exact figure not reported in abstract). Multimodal models performed comparably to the strongest unimodal modality (text) while showing improved balance across modalities.

• Primary Outcome: Binary classification of pain vs. non-pain states in cancer patients using multimodal deep learning fusion of facial, acoustic, and textual features

• Nnt Or Sensitivity: Sensitivity and specificity not reported. Balanced error rate mentioned as a metric but not quantified in the abstract. No NNT applicable to this diagnostic/detection study design.

• Confidence Interval: Not reported

Clinical Application

Clinical implementation is not currently feasible. The system requires video recording infrastructure, real-time audio capture, and transcript generation in clinical settings — logistically complex in standard oncology or palliative care workflows. Model retraining for English-language populations would be required. Regulatory approval as a medical device (Software as a Medical Device, SaMD) would be necessary before clinical deployment. The proof-of-concept nature, small sample, and absence of external validation preclude clinical use at this stage. No direct applicability to current Australian clinical practice. The TGA would classify an automated pain detection system as a SaMD requiring conformity assessment before clinical use. There is no current PBS listing or RACGP guideline recommendation for AI-based pain assessment tools. The RACGP and ANZCA pain management guidelines emphasise validated self-report tools (NRS, VAS, BPI) and observer-rated scales (PAINAD for cognitively impaired patients) as gold standards. Australian oncology practice, governed by Cancer Australia and COSA guidelines, would require prospective clinical trials with English-speaking, culturally diverse Australian populations before any such technology could be considered for integration. Privacy considerations under the Australian Privacy Act 1988 and My Health Records Act 2012 would apply to video/audio data collection. Conceptually applicable to adult oncology patients requiring pain assessment, particularly those with communication difficulties (e.g., post-operative, cognitively impaired, or non-verbal patients). Current model is validated only in Italian-speaking cancer patients at a single Italian tertiary centre.

Abstract

INTRODUCTION: Since pain is a multidimensional and subjective experience, pain assessment remains challenging. With advances in artificial intelligence (AI), automatic pain assessment (APA) systems offer a valuable opportunity for objective pain evaluation. However, most approaches focus on a single modality. In this proof-of-concept study, exploring multimodal fusion strategies in a controlled experimental setting, we present a deep learning framework for multimodal fusion that combines facial, acoustic, and textual information to improve APA in cancer patients. METHODS: A multimodal dataset was created from video-recorded interviews with oncologic patients. In Phase I, audio, video, and transcripts were segmented at the sentence level and temporally aligned using the Eudico Linguistic Annotator (ELAN) to ensure frame-level correspondence across modalities. In Phase II, modality-specific features were extracted: Facial Action Units from OpenFace, acoustic descriptors (MFCCs, chroma, spectral contrast, and Mel-spectrogram) from a dedicated speech-processing pipeline, and sentence-level textual embeddings from ITA-BERT. During training, the most effective analytical strategy was chosen through knowledge transfer approaches. The ELAN-assisted annotation pipeline streamlined expert labeling. Two architectures were implemented and compared: bimodal autoencoder fusion models and a transformer-based model with pairwise cross-modal attention. These models were trained and evaluated using subject-independent and stratified 5-fold cross-validation. To address the lack of independence between segments, a strictly subject-independent cross-validation strategy was adopted. RESULTS: Knowledge transfer using pretrained large-scale models outperformed traditional feature-based approaches and was applied to multimodal pain detection. Multimodal models achieved performance comparable to the strongest unimodal modality (text), while showing improved balance across modalities, suggesting potential complementary effects. Both multimodal architectures demonstrated high accuracy in distinguishing between pain and non-pain classes. The bimodal autoencoder achieved stable results across folds, with a mean accuracy of about 80 % and balanced error distribution. The pairwise transformer with cross-modal attention achieved similar performance, with smooth training and validation loss curves. No evident divergence between training and validation loss curves was observed across folds, suggesting stable behavior within the cross-validation setting. However, subject-level overfitting cannot be excluded given the limited sample size. CONCLUSION: Multimodal fusion enhances system robustness by integrating complementary signals. Despite limitations and the need for improvement, multimodal deep learning strategies can support the detection of observable pain-related expressions.

References

1. 1.Cascella, M., Mariani, F., Barberio, D., Crispo, A., Ottaiano, A., Bimonte, S., Franci, G., Sabbatino, F., Piazza, O., & Cutugno, F. (2022). Multimodal deep learning fusion for automatic pain detection in cancer patients. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) (pp. 1–4). IEEE. https://doi.org/10.1109/EMBC48229.2022.9871990