Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces

August 2020

Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces

Authors:

Senthil Purushwalkam, Tian Ye, Saurabh Gupta, and Abhinav Gupta
Conference Paper
Proceedings of (ECCV) European Conference on Computer Vision

Abstract:

In this paper, we focus on the task of extracting visual correspondences across videos. Given a query video clip from an action class, we aim to align it with training videos in space and time. Obtaining training data for such a fine-grained alignment task is challenging and often ambiguous. Hence, we propose a novel alignment procedure that learns such correspondence in space and time via cross video cycle-consistency. During training, given a pair of videos, we compute cycles that connect patches in a given frame in the first video by matching through frames in the second video. Cycles that connect overlapping patches together are encouraged to score higher than cycles that connect non-overlapping patches. Our experiments on the Penn Action and Pouring datasets demonstrate that the proposed method can successfully learn to correspond semantically similar patches across videos, and learns representations that are sensitive to object and action states.
@conference{Purushwalkam-2020-126741,
author = {Senthil Purushwalkam And Tian Ye And Saurabh Gupta And Abhinav Gupta},
title = {Bounce and Learn: Modeling Scene Dynamics with Real-World Bounces},
booktitle = {Proceedings of (ECCV) European Conference on Computer Vision},
year = {2020},
month = {August},
pages = {262 - 278},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.