CEREBRO: Efficient and Reproducible Model Selection on Deep Learning Systems

Artificial Neural Networks (ANNs) are revolutionizing many machine learning (ML) applications. But there is a major bottleneck to wider adoption: the pain of model selection. This empirical process involves exploring the ANN architecture and hyper-parameters, often requiring hundreds of trials. Alas, most ML systems focus on training one model at a time, reducing throughput and raising costs; some also sacrifice reproducibility. We present our vision of CEREBRO, a system to raise ANN model selection throughput at scale and ensure reproducibility. CEREBRO uses a novel parallel execution strategy we call model hopper parallelism. We discuss the research questions in building CEREBRO and present promising initial empirical results.