An efficient and lightweight multiperson activity recognition framework for robot-assisted healthcare applications

Aging is inevitably associated with a decline in physical abilities and can pose challenges to the social lives of elderly individuals. In long-term care facilities, group exercise is instrumental for keeping elderly residents physically and socially healthy. Accommodating these needs in elderly care can be challenging due to staff shortages and other lacking resources. A robotic exercise coach could be helpful in such contexts. Intelligent human–robot interaction requires accurate and efficient human activity recognition. Several solutions focusing on human activity recognition in healthcare robotics have been proposed. However, multiperson activity recognition remains a challenging task in case of using vision-based or wearable sensors data, and past research has mainly focused on single-person rather than multiperson or group activity recognition. Moreover, the existing state-of-the-art methods for activity recognition mainly use heavyweight Convolutional Neural Network (CNN) models to achieve good accuracy. However, these models have certain drawbacks, such as requiring significant computational resources, higher memory and storage needs, and slower inference times. Another challenge is the limited number of publicly available datasets containing few activities for physical activity recognition. In this work, we propose a lightweight, deep learning-based, multiperson activity recognition system for group exercise training of elderly persons. Considering the limited publicly available datasets, we curated a new dataset named the Routine Exercise Dataset (RED), comprising 19 routine exercise activities recommended for elderly persons. The RED dataset has 14,440 samples collected from 19 participants and is one of the most extensive datasets of its kind. We evaluated our proposed activity recognition method based on proposed feature extraction modules and a one-dimensional multilayer long short-term memory network on 16 datasets, including 10 publicly available benchmark activity recognition datasets, an RED dataset, a publicly available dataset combined with RED dataset, and four noise-corrupted RED datasets. The results indicate the efficiency of the proposed method for real-time activity recognition compared to the state-of-the-art methods. The proposed method achieved F1-scores of 98.64%, 97.95%, and 99% on large-scale datasets named UESTC RGB-D, NTU RGB+D, and RED, respectively. We also developed a Robot Operating System (ROS)-based application to deploy our proposed system in a social robot and test it in real-life scenarios. Keywords: Human activity recognition; Multiperson activity recognition; Exercise recognition; Robot-assisted rehabilitation; Virtual coaches; Eldercare