Neural networks are sensitive to hyper-parameter and architecture choices.
Automated Machine Learning (AutoML) is a promising paradigm for automating
these choices. Current ML software libraries, however, are quite limited in
handling the dynamic interactions among the components of AutoML. For example,
efficientNAS algorithms, such as ENAS and DARTS, typically require an
implementation coupling between the search space and search algorithm, the two
key components in AutoML. Furthermore, implementing a complex search flow, such
as searching architectures within a loop of searching hardware configurations,
is difficult. To summarize, changing the search space, search algorithm, or
search flow in current ML libraries usually requires a significant change in
the program logic. In this paper, we introduce a new way of programming AutoML
based on symbolic programming. Under this paradigm, ML programs are mutable,
thus can be manipulated easily by another program. As a result, AutoML can be
reformulated as an automated process of symbolic manipulation. With this
formulation, we decouple the triangle of the search algorithm, the search space
and the child program. This decoupling makes it easy to change the search space
and search algorithm (without and with weight sharing), as well as to add
search capabilities to existing code and implement complex search flows. We
then introduce PyGlove, a new Python library that implements this paradigm.
Through case studies on ImageNet and NAS-Bench-101, we show that with PyGlove
users can easily convert a static program into a search space, quickly iterate
on the search spaces and search algorithms, and craft complex search flows to
achieve better results.