MULTIMODAL SENTIMENT ANALYSIS USING TRANSFORMER-BASED FUSION

Abstract

The rapid growth of multimodal content on social media has reshaped how opinions and emotions are expressed, with users increasingly relying on both text and visual elements such as memes, images, and graphics. Traditional sentiment analysis methods, which focus solely on text, often fail to capture the full meaning conveyed in multimodal data. This limitation is particularly pronounced in low-resource contexts, where labeled datasets are scarce and language diversity complicates analysis. The central research question addressed in this work is: How can multimodal sentiment classification be performed effectively under limited data conditions while maintaining computational efficiency?

To answer this, we propose a multimodal sentiment analysis framework that integrates textual and visual features using deep learning. Our approach leverages pretrained BERT and ResNet-50 encoders to extract rich representations, which are fused through a Transformer-based attention mechanism to capture cross-modal interactions. Unlike prior methods that rely on either simple concatenation or overly complex architectures, our model balances simplicity and effectiveness, making it suitable for low-resource datasets such as MVSA-Single.

The experimental evaluation demonstrates that the proposed model achieves 57.56% accuracy, with weighted and macro F1 scores of 0.5687 and 0.5593, respectively, and an AUC-ROC of 0.7539. Error analysis reveals that positive sentiment is de-tected reliably, while negative sentiment remains challenging due to subtle cues. Ablation studies confirm the importance of the fusion module, showing clear improvements over unimodal and simple fusion baselines. Overall, our framework provides a practical and efficient solution for multimodal sentiment classification, bridging the gap between performance and deployability in real-world, low-resource scenarios.