Batteries, camera, action! Learning a semantic control space for expressive robot cinematography

May 2021

Batteries, camera, action! Learning a semantic control space for expressive robot cinematography

Authors:

Rogerio Bonatti, Arthur Bucker, Sebastian Scherer, Mustafa Mukadam, and Jessica Hodgins
Conference Paper
Proceedings of (ICRA) International Conference on Robotics and Automation

Abstract:

Aerial vehicles are revolutionizing the way film-makers can capture shots of actors by composing novel aerial and dynamic viewpoints. However, despite great advancements in autonomous flight technology, generating expressive camera behaviors is still a challenge and requires non-technical users to edit a large number of unintuitive control parameters. In this work, we develop a data-driven framework that enables editing of these complex camera positioning parameters in a semantic space (e.g. calm, enjoyable, establishing). First, we generate a database of video clips with a diverse range of shots in a photo-realistic simulator, and use hundreds of participants in a crowd-sourcing framework to obtain scores for a set of semantic descriptors for each clip. Next, we analyze correlations between descriptors and build a semantic control space based on cinematography guidelines and human perception studies. Finally, we learn a generative model that can map a set of desired semantic video descriptors into low-level camera trajectory parameters. We evaluate our system by demonstrating that our model successfully generates shots that are rated by participants as having the expected degrees of expression for each descriptor. We also show that our models generalize to different scenes in both simulation and real-world experiments.
@conference{Bonatti-2021-126972,
author = {Rogerio Bonatti And Arthur Bucker And Sebastian Scherer And Mustafa Mukadam And Jessica Hodgins},
title = {Batteries, camera, action! Learning a semantic control space for expressive robot cinematography},
booktitle = {Proceedings of (ICRA) International Conference on Robotics and Automation},
year = {2021},
month = {May},
keywords = {Cinematography, UAV, Motion Planning, Machine Learning},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.