Synthetic Petri Dish: A Novel Surrogate Model for Rapid Architecture Search (Paper Explained)

Neural Architecture Search is usually prohibitively expensive in both time and resources to be useful. A search strategy has to keep evaluating new models, training them to convergence in an inner loop to find out if they are any good. This paper proposes to abstract the problem and extract the essential part of the architecture to be optimized into a smaller version and evaluates that version on specifically custom learned data points to predict its performance, which is much faster and cheaper than running the full model. OUTLINE: 0:00 - Intro & High-Level Overview 1:00 - Neural Architecture Search 4:30 - Predicting performance via architecture encoding 7:50 - Synthetic Petri Dish 12:50 - Motivating MNIST example 18:15 - Entire Algorithm 23:00 - Producing the synthetic data 26:00 - Combination with architecture search 27:30 - PTB RNN-Cell Experiment 29:20 - Comments & Conclusion Paper: https://arxiv.org/abs/2005.13092 Code: https://github.com/uber-research/Synthetic-Petri-Dish Abstract: Neural Architecture Search (NAS) explores a large space of architectural motifs -- a compute-intensive process that often involves ground-truth evaluation of each motif by instantiating it within a large network, and training and evaluating the network with thousands of domain-specific data samples. Inspired by how biological motifs such as cells are sometimes extracted from their natural environment and studied in an artificial Petri dish setting, this paper proposes the Synthetic Petri Dish model for evaluating architectural motifs. In the Synthetic Petri Dish, architectural motifs are instantiated in very small networks and evaluated using very few learned synthetic data samples (to effectively approximate performance in the full problem). The relative performance of motifs in the Synthetic Petri Dish can substitute for their ground-truth performance, thus accelerating the most expensive step of NAS. Unlike other neural network-based prediction models that parse the structure