Lightweight audio-visual sentiment analysis with dynamic mouth-state modeling and multimodal LLM benchmarking

Information from the abstract

Sentiment analysis from video streams requires effective modeling of both temporal dynamics and multimodal cues. This paper presents a lightweight multimodal CNN-LSTM framework, designed for low-latency operation on live video streams, that classifies individuals' affective states as negative, neutral, or positive by jointly exploiting audio and visual information. The proposed system consists of four lightweight deep neural network models that fuse sequences of audio and visual features into a unified temporal stream, enabling the accumulation of sentiment evidence over time for robust prediction. To further enhance discriminative power, dynamic mouth state classification is incorporated, allowing the model to better capture subtle visual cues related to speech and sentiment. Experiments conducted on the Ryerson Audio-Visual Database of Emotional Speech (RAVDESS) demonstrate that the proposed approach achieves a test accuracy of 84.89% on the three-class sentiment recognition task, competitive with substantially larger audio-visual architectures, while using an order of magnitude fewer parameters and maintaining low-latency operation. In addition, we provide a comprehensive comparison with multiple multimodal large language models (LLMs), evaluated under diverse input configurations and prompting strategies. The results offer practical insights into the strengths and limitations of contemporary LLMs for audio-visual sentiment analysis and highlight the continued effectiveness of compact, task-specific neural architectures for streaming audio-visual input.

Why this record is monitored

This record has an Impact Signal of 70/100 based on recency, source, collaboration, and bibliographic signals. It prioritizes monitoring and is not a judgment of research quality.

Related topics: Emotion and Mood Recognition · Sentiment Analysis and Opinion Mining · Speech and Audio Processing