EMONET: EFFICIENT MULTIMODAL DEEP FUSION FOR EMOTION RECOGNITION IN SOCIAL MEDIA VIDEOS

Abstract

With the eruption growth of social media video sharing platforms, emotion recognition in social media videos has emerged as an important research field in AI, Affective Computer Science, and Multimodal Deep Learning. The multimodal information encoded in social media videos is vast it includes facial expressions, speech signals, text of captions, gestures, and other context cues in which they are perceived by the human listener reflecting human states of mind. Yet, conventional machine learning techniques using hand designed features fail to give satisfactory performance in the real world. In this regard, the aim of this study is to alleviate those limitations with the introduction of a new efficient multimodal deep learning model for social media videos emotion recognition: EmoNet. A hybrid architecture: ConFusionNet and an Attention-based Multimodal Fusion Transformer (AMFT) is proposed to combine all the three modalities: visual, audio, and textual in the architecture. The three streams, plurality recorded as a whole, use two different types of deep learning networks: Convolutional Neural Networks (CNNs) are used for spatial feature extraction, and Audio and Textual Streams process speech and transcript data in their respective deep sequential learning environments. Evaluations done on standard data sets with TensorFlow on a GPU-capable machine. The proposed EmoNet model outperformed the other CNN-RNN and attention-based models and was more accurate compared to the existing models, with precision, recall and F1 score metrics. Results help show that when looking for emotion recognition in real world social media context, efficient multimodal fusion and contextual attention models can significantly benefit the recognition performance.