MV-SSM: Multi-View State Space Modeling for 3D Human Pose Estimation

August 2025

MV-SSM: Multi-View State Space Modeling for 3D Human Pose Estimation

Authors:

Aviral Chharia, Wenbo Gou, and Haoye Dong
Conference Paper
Proceedings of (CVPR) Computer Vision and Pattern Recognition

Abstract:

While significant progress has been made in single-view 3D human pose estimation, multi-view 3D human pose estimation remains challenging, particularly in terms of generalizing to new camera configurations. Existing attention-based transformers often struggle to accurately model the spatial arrangement of keypoints, especially in occluded scenarios. Additionally, they tend to overfit specific camera arrangements and visual scenes from training data, resulting in substantial performance drops in new settings. In this study, we introduce a novel Multi-View State Space Modeling framework, named MV-SSM, for robustly estimating 3D human keypoints. We explicitly model the joint spatial sequence at two distinct levels: the feature level from multi-view images and the person keypoint level. We propose a Projective State Space (PSS) block to learn a generalized representation of joint spatial arrangements using state space modeling. Moreover, we modify Mamba's traditional scanning into an effective Grid Token-guided Bidirectional Scanning (GTBS), which is integral to the PSS block. Multiple experiments demonstrate that MV-SSM achieves strong generalization, outperforming state-of-the-art methods: +10.8 on AP25 (+24%) on the challenging three-camera setting in CMU Panoptic, +7.0 on AP25 (+13%) on varying camera arrangements, and +15.3 PCP (+38%) on Campus A1 in cross-dataset evaluations.
@conference{Chharia-2025-148843,
author = {Aviral Chharia And Wenbo Gou And Haoye Dong},
title = {MV-SSM: Multi-View State Space Modeling for 3D Human Pose Estimation},
booktitle = {Proceedings of (CVPR) Computer Vision and Pattern Recognition},
year = {2025},
month = {August},
pages = {11590-11599},
publisher = {IEEE},
keywords = {Human Pose Estimation, Mamba, State Space Model, Multi-view Understanding},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.