Forestier, Sébastien, Mollard, Yoan, Oudeyer, Pierre-Yves (2017) Intrinsically motivated goal exploration processes with automatic curriculum learning. [ Ссылка ]
2nd rank at Demonstration Awards, NIPS 2016, Barcelona, Spain, December 6th, 2016.
Open-source code: [ Ссылка ]
Abstract:
Intrinsically motivated spontaneous exploration is a key enabler of autonomous lifelong learning in human children. It enables the discovery and acquisition of large repertoires of skills through self-generation, self-selection, self-ordering and self-experimentation of learning goals. We present an algorithmic approach called Intrinsically Motivated Goal Exploration Processes (IMGEP) to enable similar properties of autonomous learning in machines. The IMGEP algorithmic architecture relies on several principles: 1) self-generation of goals as fitness functions; 2) selection of goals based on intrinsic rewards; 3) exploration with incremental goal-parameterized policy search and exploitation of the gathered data with a batch learning algorithm; 4) systematic reuse of information acquired when targeting a goal for improving towards other goals.
We present a particularly efficient form of population-based IMGEP that uses a modular representation of goal spaces as well as intrinsic rewards based on learning progress.
We show how IMGEPs automatically generate a learning curriculum within several experimental setups including a real humanoid robot that can explore multiple spaces of goals with several hundred continuous dimensions.
While no particular target goal is provided to the system, this curriculum allows the discovery of skills that act as stepping stone for learning more complex skills, e.g. nested tool use.
We show that learning diverse spaces of goals with intrinsic motivations is more efficient for learning complex skills than only trying to directly learn these complex skills.
Python/Explauto Tutorial on a simulated environment: [ Ссылка ]
Other references:
Forestier S, Oudeyer P-Y. 2016. Modular Active Curiosity-Driven Discovery of Tool Use. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). [ Ссылка ]
NIPS demo poster: [ Ссылка ]
=== This project was conducted within a larger long-term research program at the Flowers lab on mechanisms of lifelong learning and development in machines and humans. This research program has in particular lead to a series of novel intrinsically motivated learning algorithms working on high-dimensional real robots and opening new perspectives in cognitive sciences. Papers providing this broader context are:
GOTTLIEB, Jacqueline et OUDEYER, Pierre-Yves. Towards a neuroscience of active sampling and curiosity. Nature Reviews Neuroscience, 2018, vol. 19, no 12, p. 758-770.
Oudeyer, P-Y. and Smith. L. (2016) How Evolution may work through Curiosity-driven Developmental Process, Topics in Cognitive Science, 1-11. [ Ссылка ]
Baranes, A., Oudeyer, P-Y. (2013) Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robots, Robotics and Autonomous Systems, 61(1), pp. 49-73.
[ Ссылка ]
Oudeyer P-Y. and Kaplan F. (2007) What is intrinsic motivation? A typology of computational approaches. Frontiers in Neurorobotics, 1:6. [ Ссылка ]
=== Links
* Poppy Project: an open-source 3D printed low-cost humanoid robotic platform that allows non-roboticists to quickly set up and program robotic experiments.
[ Ссылка ]
* This video is available for download here (CC-BY-NC): [ Ссылка ]
A music-free and subtitle-free CC-BY version is available here: [ Ссылка ]
If you reuse these videos, please cite the NIPS 2016 poster:
Forestier, S., Mollard, Y., Caselli, D., Oudeyer, P-Y. (2016) Autonomous exploration, active learning and human guidance with open-source Poppy humanoid robot platform and Explauto library, NIPS 2016 demonstration, [ Ссылка ].
