ControllerPose: Inside-Out Body Capture with VR Controller Cameras

April 2022

ControllerPose: Inside-Out Body Capture with VR Controller Cameras

Authors:

Karan Ahuja, Vivian Shen, Cathy Fang, Nathan Riopelle, Andy Kong, and Chris Harrison

Abstract:

We present a new and practical method for capturing user body pose in virtual reality experiences: integrating cameras into handheld controllers, where batteries, computation and wireless communication already exist. By virtue of the hands operating in front of the user during many VR interactions, our controller-borne cameras can capture a superior view of the body for digitization. Our pipeline composites multiple camera views together, performs 3D body pose estimation, uses this data to control a rigged human model with inverse kinematics, and exposes the resulting user avatar to end user applications. We developed a series of demo applications illustrating the potential of our approach and more leg-centric interactions, such as balancing games and kicking soccer balls. We describe our proof-of-concept hardware and software, as well as results from our user study, which point to imminent feasibility.
@conference{Ahuja-2022-135215,
author = {Karan Ahuja And Vivian Shen And Cathy Fang And Nathan Riopelle And Andy Kong And Chris Harrison},
title = {ControllerPose: Inside-Out Body Capture with VR Controller Cameras},
booktitle = {Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems},
year = {2022},
month = {April},
publisher = {ACM},
keywords = {body pose, homography, VR},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.