Autonomous maneuver decision-making method based on reinforcement learning and Monte Carlo tree search - Zhang, Hongpeng; Zhou, Huan; Wei, Yujie; Huang, Changqiang (Frontiers in Neurorobotics, 2022)
Autonomous maneuver decision-making methods for air combat often rely on human knowledge, such as advantage functions, objective functions or dense rewards in reinforcement learning, which limits the decision-making ability of unmanned combat aerial vehicle to the scope of human experience and result in slow progress in maneuver decision-making. Therefore, a maneuver decision-making method based on deep reinforcement learning and Monte Carlo tree search is proposed to investigate whether it is feasible for maneuver decision-making without human knowledge or advantage function. To this end, Monte Carlo tree search in continuous action space is proposed and neural networks guided Monte Carlo tree search with self-play is utilized to improve the ability of air combat agent. It starts from random behaviors and generates samples consist of states, actions and results of air combat through self-play without using human knowledge. These samples are used to train the neural network, and the neural network with greater winning rate is selected by simulations. Then, repeat the above process to gradually improve the maneuver decision-making ability. Abundant simulations are conducted to verify the effectiveness of the proposed method, and the kinematic model of missile is used in simulations instead of missile engagement zone to test whether the maneuver decision-making method is effective or not. The simulation results of fixed initial state and random initial state show that the proposed method is efficient and can meet the real-time requirement.