Spatio-Temporal Deepfake Video Detection via 3D Convolutional Neural Networks and Vision Transformers

Abstract

Recent advances in generative artificial intelligence have enabled the creation of highly realistic deepfake videos, raising significant concerns regarding misinformation, identity fraud, and the erosion of trust in digital media. Existing detection methods typically rely on either spatial artifact analysis or temporal motion modeling in isolation, limiting their ability to capture the full range of manipulation cues present in synthetic video content. This paper proposes a hybrid spatio-temporal deepfake detection framework that sequentially integrates an 18-layer 3D Convolutional Neural Network (R3D-18) with a Vision Transformer (ViT-Base/16). The R3D-18 branch first extracts motion-aware spatiotemporal embeddings, capturing irregularities such as unnatural blinking, facial jitter, and interframe flickering. These embeddings are then encoded as a token sequence and processed by the Vision Transformer, enabling global spatial reasoning over representations that are already temporally informed. To address the severe 10:1 class imbalance present in the Celeb-DF v2 dataset, the framework employs Focal Loss, class-weighted training, and weighted random sampling. Evaluated on Celeb-DF v2, the standalone R3D-18 and ViT-Base/16 branches achieve AUC-ROC scores of 0.9163 and 0.8061, respectively, while the proposed hybrid model achieves an AUC-ROC of 0.9226 and an F1-score of 0.9081, outperforming MesoNet, XceptionNet, CNN+ViT, and GenConViT. A crossdataset evaluation on FaceForensics++ further demonstrates generalization, achieving an average AUC-ROC of 0.7117 across four manipulation categories without any fine-tuning on the target dataset. These results demonstrate the effectiveness of sequential spatio-temporal integration for robust and generalizable deepfake detection