Learnable Embedding Space for Efficient Neural Architecture Compression

May 2019

Learnable Embedding Space for Efficient Neural Architecture Compression

Authors:

Shengcao Cao, Xiaofang Wang, and Kris M. Kitani
Conference Paper
Proceedings of (ICLR) International Conference on Learning Representations

Abstract:

We propose a method to incrementally learn an embedding space over the domain of network architectures, to enable the careful selection of architectures for evaluation during compressed architecture search. Given a teacher network, we search for a compressed network architecture by using Bayesian Optimization (BO) with a kernel function defined over our proposed embedding space to select architectures for evaluation. We demonstrate that our search algorithm can significantly outperform various baseline methods, such as random search and reinforcement learning (Ashok et al., 2018). The compressed architectures found by our method are also better than the state-of-the-art manually-designed compact architecture ShuffleNet (Zhang et al., 2018). We also demonstrate that the learned embedding space can be transferred to new settings for architecture search, such as a larger teacher network or a teacher network in a different architecture family, without any training.
@conference{Cao-2019-118813,
author = {Shengcao Cao And Xiaofang Wang And Kris M. Kitani},
title = {Learnable Embedding Space for Efficient Neural Architecture Compression},
booktitle = {Proceedings of (ICLR) International Conference on Learning Representations},
year = {2019},
month = {May},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.