Monitoring In-Home Emergency Situation and Preserve Privacy using Multi-modal Sensing and Deep Learning

Videos and images are commonly used in home monitoring systems. However, detecting emergencies in-home while preserving privacy is a challenging task concerning Human Activity Recognition (HAR). In recent years, HAR combined with deep learning has drawn much attention from the general public. Besides that, relying entirely on a single sensor modal-ity is not promising. In this paper, depth images and radar presence data were used to investigate if such sensor data can tackle the challenge of a system's ability to detect abnormal and normal situations while preserving privacy. The recurrence plots and wavelet transformations were used to make a two-dimensional representation of the presence radar data. Moreover, we fused data from both sensors using data-level, feature-level, and decision-level fusions. The decision-level fusion showed its superiority over the other two techniques. For the decision-level fusion, a combination of the depth images and presence data recurrence plots trained first on convolutional neural networks (CNN). The output was fed into support vector machines, which yielded the best accuracy of 99.98%.