Occlusion-Aware Human Mesh Model-Based Gait Recognition

Contributed by Prof. Chi Xu, based on the IEEEXplore® article, “Occlusion-Aware Human Mesh Model-Based Gait Recognition”, published in the IEEE Transactions on Information Forensics and Security in January 2023, and the SPS Webinar, "Towards Robust Gait Recognition Integrated with Human Mesh Model”, available on the SPS Resource Center.

Gait recognition is a popular biometric that recognizes people from their unique gait features, including the body shape and walking posture characteristics, utilizing machine learning for gait video processing. The gait has distinct advantages over other biometrics (e.g., face), such as long-distance capture without subject cooperation and applicability to low-resolution images. Therefore, gait recognition is considered to have great potential in applications that use CCTV footage, such as surveillance, forensics, and criminal investigation. However, partial occlusion of the human body caused by obstacles or a limited camera field of view often occurs in surveillance videos, which affects the performance of gait recognition in practice.

Existing methods for gait recognition against occlusion require a bounding box or the height of a full human body as a prerequisite, which is unobserved in occlusion scenarios. Unlike the above-mentioned appearance-based approaches to occlusion handling with a prerequisite, model-based approaches (e.g., ModelGait [1]) have the potential to handle occlusion without a prerequisite in a more natural manner. Therefore, we propose an occlusion-aware model-based gait recognition method that works directly on gait videos under occlusion without the above-mentioned prerequisite [2].

Occlusion-Aware Model-Based Gait Recognition Framework Against Occlusion Without a Prerequisite

The proposed occlusion-aware model-based gait recognition framework is shown in Fig. 1. Given a gait video with occlusion, we crop the non-occluded body parts using a square bounding box for each frame, and then resize the cropped images to a unified image size while maintaining the aspect ratio. We use a sequence encoder to estimate the 3D human mesh (i.e., the body shape and pose parameters), and global rotation and camera parameters for each cropped input image. Thereafter, we alleviate the intra-subject variation of the estimated SMPL model parameters induced by various occlusion patterns using an occlusion attenuation module. Because intra-subject pose variation should be computed in the same phase (i.e., gait stance), we first estimate the phase sequence of an input gait video and then synchronize the phases among sequences. Next, we introduce the camera parameters as a cue for the occlusion patterns, and transform the shape and pose parameters to more occlusion-independent parameters. Finally, we feed the 3D joint locations obtained from the occlusion-independent pose parameters and shape parameters averaged over frames into a recognition module. To ensure a trade-off between model estimation and recognition accuracy, a unified loss is used to optimize the entire framework in an end-to-end manner.

State-of-the-Art Performance on Occlusion Data

We evaluated the proposed method using OU-MVLP [3], which is the world’s largest gait dataset with wide view variations. We prepared various types of occlusions to simulate occlusion scenarios that often occur in real life. Specifically, two occlusion scenarios are considered: fixed occlusion ratio and changing occlusion ratio in a sequence (i.e., the proportion of occluded body height changed). The fixed occlusion ratio simulates a scenario such as a subject in the side view being occluded by a relatively long obstacle (e.g., flower bed), or the front view being occluded by an object moving with the subject (e.g., a large suitcase). The changing occlusion ratio simulates a scenario such as a subject in the front view being occluded because of the limited camera field of view, or the side view being occluded by an obstacle at a certain angle to the walking direction (e.g., a billboard). Compared with existing state-of-the-art gait recognition methods, the proposed method achieved superior performance by about 15% rank-1 identification rate and 2% equal error rate in the identification and verification scenarios, respectively.

Conclusion

To address the issue of occlusion in gait recognition, we introduce an occlusion-aware model-based gait recognition method that operates without prerequisites. Given an occluded gait sequence, we estimate the SMPL models directly from the input images by incorporating the occlusion attenuation module, and further use the models to extract the shape and pose features for the recognition task. Experiments on simulated occlusion illustrated the effectiveness of the proposed method.

References:

[1] X. Li, Y. Makihara, C. Xu, Y. Yagi, S. Yu, and M. Ren, “End-to-end model-based gait recognition,” in Proc. Asian Conf. Comput. Vis. (ACCV), Nov. 2020, pp. 1–17, doi: https://doi.org/10.1007/978-3-030-69535-4_1.

[2] C. Xu, Y. Makihara, X. Li, Y. Yagi, “Occlusion-aware Human Mesh Model-based Gait Recognition,” IEEE Transactions on Information Forensics and Security, Vol. 18, pp. 1309-1321, Jan. 2023, doi: https://dx.doi.org/10.1109/TIFS.2023.3236181.

[3] N. Takemura, Y. Makihara, D. Muramatsu, T. Echigo, and Y. Yagi, “Multi-view large population gait dataset and its performance evaluation for cross-view gait recognition,” IPSJ Trans. Comput. Vis. Appl., vol. 10, no. 1, pp. 1–14, 2018, doi: https://doi.org/10.1186/s41074-018-0039-6.