STATE-AWARE HUMAN ACTIVITY RECOGNITION USING MULTI-MODAL TIME-SERIES PHYSIOLOGICAL AND MOTION SIGNALS FOR MONITORING FATIGUE, STRESS, AND CHRONIC HEALTH CONDITIONS IN PAKISTAN

Abstract

The aim of this study is to examine the effectiveness of a state aware HAR framework that incorporated multiple types of physiological signals (HR, HRV, PPG, and skin temperature) and motion (Accelerometer and Gyroscopes). Physiological, Motion, and Video datasets were collected from 120 Adult Subjects in Punjab and Sindh under both controlled (n=80) and Free-Living (n=120) research conditions, resulting in 8720 hours of data and 512340 usable time windows; all datasets were processed through Butterworth and Adaptive filtering methods, followed by early and late stage fusing of multimodality data. Model performance was evaluated using a subject independent test and 10-fold cross validation, wherein Hybrid CNN-LSTM models achieved Mean Accuracy and F1 Score of 94.8% (F1=94.8%). On the other hand, state aware ensemble models were able to recognize Stress with Mean Accuracy and F1 Score of 91.2% (F1=91.1%) and Fatigue with Mean Accuracy and F1 Score of 89.7% (F1=89.7%). Chronic Physiological Indicators related to Hypertension and Post-COVID Fatigue were identified at Mean Accuracy = 87.5% (F1 = 87.6%). Pre-processing techniques utilized to reduce Signal Noise by 78%, increase Valid HRV Extraction to 92%, and Fusion between Multi-model modalities (i.e., Acquiring from various cultural sources) improved reductions in average cultural-based Variability of approximately 65%. Overall usability analysis showed a high, (82.4 + 8.6) System Usability Scale score and that 85% of respondents expressed a Willingness to use.