Title: On Building General, Zero-Shot Robot Policies

Abstract:
In this talk, I will give perspectives on how large models have changed robotics, and why there is still fundamental research to be done. The main focus of the discussion is how we can achieve generalization in robotics. More traditional methods from Task and Motion Planning (TAMP) are capable of solving complex sequential manipulation problems while generalizing over a wide range of initial scene configurations. However, those methods make assumptions that limit their generalization in real world scenarios. In the first half of the talk, I will discuss how (small) machine learning methods can be integrated into TAMP to address some of these shortcomings. In the second half of the talk, I will then explain how the rise of large models has transformed these previous findings. In particular, I will present PaLM-E, a large vision-language model for embodied decision making, RT-2, a vision-language-action model that connects large models to low-level robot actions, and Aloha Unleashed, a recipe to push the boundary of robot dexterity. Finally, I will sort all these developments into a larger picture of how the future of robotics research could look like.

свернуть