Discretization-aware architecture search

The search cost of neural architecture search (NAS) has been largely reduced by differentiable architecture search and weight-sharing methods. Such methods optimize a super-network with all possible edges and operations, and determine the optimal sub-network by discretization, i.e., pruning off operations/edges of small weights. However, the discretization process performed on either operations or edges incurs significant inaccuracy and thus the quality of the architecture is not guaranteed. In this paper, we propose discretization-aware architecture search (DA(2)S), and target at pushing the super-network towards the configuration of desired topology. DA(2)S is implemented with an entropy-based loss term, which can be regularized to differentiable architecture search in a plug-and-play fashion. The regularization is controlled by elaborated continuation functions, so that discretization is adaptive to the dynamic change of edges and operations. Experiments on standard image classification benchmarks demonstrate the effectiveness of our approach, in particular, under imbalanced network configurations that were not studied before. (C) 2021 Elsevier Ltd. All rights reserved.