Multi-fidelity Bayesian Optimisation with Continuous Approximations

August 2017

Multi-fidelity Bayesian Optimisation with Continuous Approximations

Authors:

K. Kandasamy, G. Dasarathy, J. Schneider, and B. Poczos
Conference Paper
Proceedings of (ICML) International Conference on Machine Learning

Abstract:

Bandit methods for black-box optimisation, such as Bayesian optimisation, arc used in a variety of applications including hyper-parameter tuning and experiment design. Recently, multi-fidelity methods have garnered considerable attention since function evaluations have become increasingly expensive in such applications. Multi-fidelity methods use cheap approximations to the function of interest to speed up the overall optimisation process. However, most multi-fidelity methods assume only a finite number of approximations. On the other hand, in many practical applications, a continuous spectrum of approximations might be available. For instance, when tuning an expensive neural network, one might choose to approximate the cross validation performance using less data N and/or few training iterations T. Here, the approximations are best viewed as arising out of a continuous two dimensional space (iV, T). In this work, we develop a Bayesian optimisation method, BOCA, for this setting. We characterise its theoretical properties and show that it achieves better regret than than strategies which ignore the approximations. BOCA outperforms several other baselines in synthetic and real experiments.
@conference{Kandasamy-2017-119741,
author = {K. Kandasamy And G. Dasarathy And J. Schneider And B. Poczos},
title = {Multi-fidelity Bayesian Optimisation with Continuous Approximations},
booktitle = {Proceedings of (ICML) International Conference on Machine Learning},
year = {2017},
month = {August},
volume = {70},
pages = {1799 - 1808},
}
Copyright notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.