Videos as space-time region graphs

September 2018

Videos as space-time region graphs

Conference Paper
Proceedings of (ECCV) European Conference on Computer Vision

Abstract:

How do humans recognize the action "opening a book" ? We argue that there are two important cues: modeling temporal shape dynamics and modeling functional relationships between humans and objects. In this paper, we propose to represent videos as space-time region graphs which capture these two important cues. Our graph nodes are defined by the object region proposals from different frames in a long range video. These nodes are connected by two types of relations: (i) similarity relations capturing the long range dependencies between correlated objects and (ii) spatial-temporal relations capturing the interactions between nearby objects. We perform reasoning on this graph representation via Graph Convolutional Networks. We achieve state-of-the-art results on both Charades and Something-Something datasets. Especially for Charades, we obtain a huge 4.4% gain when our model is applied in complex environments.
@conference{Wang-2018-113279,
author = {Xiaolong Wang And Abhinav Gupta},
title = {Videos as space-time region graphs},
booktitle = {Proceedings of (ECCV) European Conference on Computer Vision},
year = {2018},
month = {September},
pages = {413 - 431},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.