Temporal Dynamic Graph LSTM for Action-Driven Video Object Detection

October 2017

Temporal Dynamic Graph LSTM for Action-Driven Video Object Detection

Authors:

Yuan Yuan, Xiaodan Liang, Xiaolong Wang, Dit-Yan Yeung, and Abhinav Gupta
Conference Paper
Proceedings of (ICCV) International Conference on Computer Vision

Abstract:

In this paper, we investigate a weakly-supervised object detection framework. Most existing frameworks focus on using static images to learn object detectors. However, these detectors often fail to generalize to videos because of the existing domain shift. Therefore, we investigate learning these detectors directly from boring videos of daily activities. Instead of using bounding boxes, we explore the use of action descriptions as supervision since they are relatively easy to gather. A common issue, however, is that objects of interest that are not involved in human actions are often absent in global action descriptions known as "missing label". To tackle this problem, we propose a novel temporal dynamic graph Long Short-Term Memory network (TD-Graph LSTM). TD-Graph LSTM enables global temporal reasoning by constructing a dynamic graph that is based on temporal correlations of object proposals and spans the entire video. The missing label issue for each individual frame can thus be significantly alleviated by transferring knowledge across correlated objects proposals in the whole video. Extensive evaluations on a large-scale daily-life action dataset (i.e., Charades) demonstrates the superiority of our proposed method. We also release object bounding-box annotations for more than 5,000 frames in Charades. We believe this annotated data can also benefit other research on video-based object recognition in the future.
@conference{Yuan-2017-113306,
author = {Yuan Yuan And Xiaodan Liang And Xiaolong Wang And Dit-Yan Yeung And Abhinav Gupta},
title = {Temporal Dynamic Graph LSTM for Action-Driven Video Object Detection},
booktitle = {Proceedings of (ICCV) International Conference on Computer Vision},
year = {2017},
month = {October},
pages = {1819 - 1828},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.