Learning Hierarchical Control Programs

Machine learning is, in a sense, data-driven programming: the execution of coded programs is replaced by the evaluation of learned models. In this talk, I will propose leveraging decades of hard-earned programming wisdom by imposing a hierarchical structure on deep-learnable models. I will introduce Parametrized Hierarchical Procedures (PHP), a neural network architecture with improved data efficiency, interpretability, and reusability "” in analogy to similar benefits for human coders who adhere to the procedural programming paradigm. This line of work focuses on two application domains, robot control and algorithmic program induction, in which the control program interacts with physical environments and memory structures, respectively, through a sequence of API calls to readers (sensors) and writers (motors). We collect data demonstrating correct interaction, and use it to learn a program, represented as a PHP, that can perform the task.

I will present three learning methods for PHPs: (1) hierarchical behavior cloning, which can utilize data consisting of complete execution traces of the intended program; (2) exact inference, which can discover program structure that is latent in the data; and (3) variational inference, which can learn a more expressive class of models. I will demonstrate the superior data efficiency and generalization of these methods in learning several robotic and algorithmic tasks. Finally, I will discuss the need and opportunity to reuse data and models between related tasks, and show how hierarchical models can enhance reusability. In current work, we scale this up to a cloud service for sharing data and models between large libraries of control procedures.
Roy Fox is a postdoc at UC Berkeley AI Research (BAIR), working with Ion Stoica in the RISELab and with Ken Goldberg in the AUTOLAB. His research interests include reinforcement learning, dynamical systems, information theory, and robotics. His current research focuses on data-driven discovery of hierarchical control structures in deep reinforcement and imitation learning of robotic tasks. Roy has a MSc in Computer Science with Moshe Tennenholtz at the Technion, and a PhD in Computer Science with Naftali Tishby at the Hebrew University. He was an exchange PhD student with Larry Abbott and Liam Paninski at Columbia University, and a research intern at Microsoft Research.