Realistic Actor-Critic: A framework for balance between value overestimation and underestimation - Li, Sicen; Tang, Qinyun; Pang, Yiming; Ma, Xinmeng; Wang, Gang (Frontiers in Neurorobotics, 2023)
This paper proposes a reinforcement learning framework to enhance the exploration-exploitation trade-off by learning a range of policies concerning various confidence bounds. The underestimated values provide stable updates but suffer from inefficient exploration behaviours. On the other hand, overestimated values can help the agent escape local optima, but it might cause over-exploration on low-value areas and function approximation error accumulation. Algorithms have been proposed to mitigate the above contradiction. However, we lack an understanding of how the value bias impact performance and a method for efficient exploration while keeping value away from catastrophic overestimation bias accumulation. In this paper, we 1) highlight that agents could over-explore low-value states due to inflexible under-overestimation trade-off in the fixed hyperparameters setting, and it is a particular form of the exploration-exploitation dilemma; 2) propose a unified framework called Realistic Actor-Critic(RAC), which employs Universal Value Function Approximators (UVFA) to simultaneously learn policies with different value confidence-bond with the same neural network, each with a different under-overestimation trade-off. These allow us to perform directed exploration without over-exploration using the upper bounds while still avoiding overestimation using the lower bounds. Through carefully designed experiments, We empirically verify that RAC achieves 10x sample efficiency and 25% performance improvement compared to Soft Actor-Critic in the most challenging Humanoid environment. All the source codes are available at https://github.com/ihuhuhu/RAC.