On Building General-Purpose Home Robots

Abstract

The concept of a “generalist machine” in homes — a domestic assistant that can adapt and learn from our needs, all while remaining cost-effective — has long been a goal in robotics that has been steadily pursued for decades. In this talk, I will present our recent efforts towards building such capable home robots. First, I will discuss how large, pretrained vision-language models can induce strong priors for mobile manipulation tasks like pick-and-drop. But pretrained models can only take us so far. To scale beyond basic picking, we will need systems and algorithms to rapidly learn new skills. This requires creating new tools to collect data, improving representations of the visual world, and enabling trial-and-error learning during deployment. While much of the work presented focuses on two-fingered hands, I will briefly introduce learning approaches for multi-fingered hands which support more dexterous behaviors and rich touch sensing combined with vision. Finally, I will outline unsolved problems that were not obvious initially, which, when solved, will bring us closer to general-purpose home robots.

Bio

Lerrel Pinto is an Assistant Professor of Computer Science at NYU. His research focuses on machine learning for robots. He received a Ph.D. degree from CMU after which he did a Postdoc at UC Berkeley. His research on robot learning has received the best paper awards at ICRA 2016 and RSS 2023, and finalist at IROS 2019, and CoRL 2022. Lerrel has received the Packard Fellowship and was named a TR35 innovator under 35 for 2023. Several of his works have been featured in popular media such as The Wall Street Journal, TechCrunch, MIT Tech Review, Wired, and BuzzFeed among others. His recent work can be found on www.lerrelpinto.com.