Model-Based and Model-Free Replay Mechanisms for Reinforcement Learning in Neurorobotics - Massi, Elisa; Barthélemy, Jeanne; Mailly, Juliane; Dromnelle, Rémi; Canitrot, Julien; Poniatowski, Esther; Girard, Benoît; Khamassi, Mehdi (Frontiers in Neurorobotics, 2022)
Experience replay is widely used in AI to bootstrap reinforcement learning by enabling an agent to remember and reuse past experience. Classical techniques include shuffled-, reversed-ordered- and prioritized-memory buffers, which have different properties and advantages depending on the nature of the data and problem. Interestingly, recent computational neuroscience work has shown that these techniques are relevant to model hippocampal reactivations recorded during rodent navigation. Nevertheless, the brain mechanisms for orchestrating hippocampal replay are still unclear. In this paper, we present recent neurorobotics research aiming to endow a navigating robot with a neuro-inspired reinforcement learning architecture (including different learning strategies, namely model-based and model-free, and different replay techniques). We illustrate through a series of numerical simulations how the specificities of robotic experimentation (e.g., autonomous state decomposition by the robot, noisy perception, state transition uncertainty, non-stationarity) can shed new lights on which replay techniques turn out to be more efficient in different situations. Finally, we close the loop by raising new hypotheses for neuroscience from such robotic models of hippocampal replay.