EAC-Agent: A deep learning framework for multimodal emotion-aware conversational agent with contextual response generation

Clinical Snapshot

Key Findings

• P Value: Not reported

• Effect Size: Superior performance compared to existing models (specific comparisons not quantified)

• Primary Outcome: Emotion recognition accuracy: 76.27% (IEMOCAP), 67.57% (MELD)

• Nnt Or Sensitivity: BLEU scores: 0.31 (IEMOCAP), 0.30 (MELD); ROUGE-L scores: 0.45 (IEMOCAP), 0.44 (MELD)

• Confidence Interval: Not reported

Clinical Application

Requires significant computational resources and technical expertise for implementation Would require TGA evaluation for therapeutic applications, alignment with RACGP digital health guidelines, and consideration of privacy regulations under Australian healthcare frameworks Healthcare organisations implementing conversational AI for patient interaction, mental health support, and customer service

Abstract

Conversational agents, commonly referred to as chatbots, have become an integral part of various important applications, such as customer support and virtual assistants in enterprise-level solutions. Despite their widespread use, current solutions predominantly rely on a single modality input, primarily text, which limits their ability to fully understand the correct emotions of users and respond accordingly. The increasing demand for multimodal inputs in conversation such as text, audio, and videos, highlights the need for a more comprehensive approach. Existing conversational agents face significant challenges in generating emotionally aware responses, as they lack the ability to effectively handle emotion embeddings, leading to limitations in emotional accuracy and contextual appropriateness. Focusing on these challenges, this research work leads to a novel multimodal approach which incorporates features from text, audio, and visual. The EAC-Agent proposed a sequence-to-sequence model with transformer, along with pre-trained embeddings such as GloVe. The self and cross-modal attention on text, audio and visual is used to generate a more emotionally intelligent response. EAC-Agent is validated through comparison with existing techniques on two benchmark datasets. The obtained results demonstrate superior performance in emotion classification and response generation. The proposed model achieves an accuracy of 76.27% on IEMOCAP and 67.57% on MELD for emotion recognition from multimodal user inputs. In addition, the emotion-aware response generation module shows clear improvements, with perplexity values of 39.01 and 42.30, BLEU scores of 0.31 and 0.30, and ROUGE-L scores of 0.45 and 0.44 on IEMOCAP and MELD, respectively. EAC-Agent demonstrates clear superiority over existing models and holds great promise for applications in customer service, healthcare, and other areas requiring empathetic and contextually appropriate interactions.

References

1. 1.Jamil, S., Ali, T., Nawaz, A., & Shahid, A. (2026). EAC-Agent: A deep learning framework for multimodal emotion-aware conversational agent with contextual response generation. PLoS One. https://doi.org/10.1371/journal.pone.0346770