The UAV one-on-one confrontation scenario is shown in Figure 1 (Yang et al., 2019).

In Figure 1, v B ⃗ and P B ⃗ ( x B , y B , z B ) denote the velocity vector and position vector of the UAV, respectively, while v R ⃗ and P R ⃗ ( x R , y R , z R ) represent the velocity vector and position vector of the target, respectively. α represents the relative azimuth angle.

Defining D ⃗ as the relative distance of the UAV with respect to the target—where the direction represents the UAV pointing toward the target and the magnitude is given by d — D ⃗ and d can be calculated by Equations 1, 2 (Li B. et al., 2022) D ⃗ = D R ⃗ − D B ⃗ , (1) d = D ⃗ 2 . (2)

Thus, the relative azimuth angle α can be calculated as Equation 3 (Li B. et al., 2022): α = arccos D ⃗ × v B ⃗ / D ⃗ 2 ⋅ v B ⃗ 2 . (3)

Generally, the operational capability of a UAV is constrained by the capabilities of its weapon system (Luo et al., 2022). A schematic showing the UAV attacking and locking onto the target is presented in Figure 2.

In Figure 2, υ max denotes the maximum off-axis emission angle of the UAV weapon system, D max a stands for the maximum firing distance, and t min lock indicates the minimum locking time. Moreover, the UAV wins the confrontation when it locks onto the target for more than t min lock seconds within the UAV firing range. Specifically, the victory adjudication condition can be described as Equation 4: d ≤ D max a α ≤ υ max t i n ≥ t min lock , (4) where t i n represents the time the UAV locks onto the target.

Based on the points discussed above, the objective of this paper is to design an intelligent algorithm that enables UAVs to make autonomous decisions to achieve victory adjudication conditions in advance, based on environmental and situational information.

The UAV maneuver decision-making system is comprised of three components: the situation assessment module, the maneuver decision-making module, and the flight drive module. The situation assessment module obtains relative situational information from the environment and determines whether the conditions for adjudicating victory are met. If not, the maneuver decision module provides the maneuver command based on the situation assessment result. Afterward, the flight drive module updates the UAV state and provides feedback to the environment. This cycle would continue until one side achieves victory through adjudication. The interactive procedure is shown in Figure 3.

The maneuver action of the UAV is designed as a discrete set by NASA, which only considers several basic maneuvers, including uniform flight, accelerated flight, decelerated flight, left turn, right turn, forward climb, and forward dive (Huang et al., 2018). To further describe the maneuvering behavior of the UAV with continuous state in actual aerial confrontations, this paper establishes a state transition update mechanism based on the UAV motion model. The schematic diagram of the UAV motion model is shown in Figure 5 (Guo et al., 2023).

In Figure 5, v , ψ , γ , and ϕ represent the velocity, yaw angle, flight path angle, and roll angle, respectively.

The following particle motion model (Guo et al., 2023) of UAVs is considered in this paper. x ̇ = v ⁡ cos ⁡ γ ⁡ sin ⁡ ψ y ̇ = v ⁡ cos ⁡ γ ⁡ cos ⁡ ψ z ̇ = v ⁡ sin ⁡ γ v ̇ = g n x − sin ⁡ γ ψ ̇ = − g n z ⁡ sin ⁡ ϕ v ⁡ cos ⁡ γ γ ̇ = g v n z ⁡ cos ⁡ ϕ − cos ⁡ γ , (5) where x , y , and z denote the position of the UAV in the inertial coordinate system. x ̇ , y ̇ , and z ̇ denote the projections of velocity v in the x -axis, y -axis, and z -axis, respectively. n x and n z denote the normal overload and tangential overload of the UAV, respectively. g is the acceleration of gravity.

According to Equation 5, the state of the UAV is affected by normal overload n x , tangential overload n z , and roll angle ϕ . Thus, the 3-DOF model mentioned above is established as a flight drive module in this paper. The new state of the UAV is calculated in real-time based on the current state and control input. The UAV motion model-driven state transition update mechanism is shown in Figure 6.

As common knowledge dictates, there are four possible scenarios in an aerial confrontation, depending on the relative positions of the UAV and the target: the trailing side holds the advantage, the pursued side is at a disadvantage, and an equilibrium state is reached when both sides are flying either toward or away from each other. To enable UAVs to reach positions with more favorable environmental conditions, this paper considers the instantaneous aerial situation between the UAV and the target, along with the victory/defeat adjudication model, as the basis for reward and punishment signals.

The angle reward r α is defined as Equation 6 (He et al., 2020): r α = 1 − α B + α R 2 π , d ≤ D max a 1 − α B + α R 2 π e − d − D max a 2 2 D max a 2 , d > D max a , (6) where α B and α R represent the relative azimuth angle of the UAV and the target, respectively.

When the target is within the attack range of the UAV, the distance reward of the UAV to the target r B → R is defined as Equation 7 (He et al., 2020): r B → R = 5 , d ≤ D max a a n d α B < υ max a n d α R < π / 2 0 , e l s e . (7)

Similarly, when the UAV is within the attack range of the target, the distance reward of the target to the UAV r R → B is defined as follows (He et al., 2020): r R → B = 5 , d ≤ D max a a n d α R < υ max a n d α B < π / 2 0 , e l s e . (8)

Invoking Equations 8, 9, the distance reward r d is defined as follows (He et al., 2020): r d = r B → R − r R → B . (9)

In addition, to ensure the UAV flight safety and avoid collisions, the height reward r h is defined as Equation 10 (Li et al., 2021): r h = − 10 , 1 k m < h < 12 k m a n d d > 200 m 0 , e l s e , (10) where h represents the flight height of the UAV.

In summary, the total reward R is obtained by Equation 11: R = r α + r d + r h . (11)

Due to the uncertainties in the UAV model and the complexity and flexibility of the actual aerial confrontation environment, the UAV and the target may not be able to reach the desired position after maneuvering during the aerial confrontation.

To describe the uncertainty of aerial confrontation, we assume that the state deviation Δ s after a maneuver action follows a discrete empirical distribution Ω ̂ constructed from observations. The nominal distribution Ω ̂ can be indirectly observed through the historical data samples O H i , where i ∈ { 1,2 , … , N } , and the current data samples O C j , where j ∈ { 1,2 , … , M } .

Under the uniform distribution over the N historical data samples and M current data samples, we have Ω ̂ = 1 N + λ M ∑ i = 1 N ξ O H i + λ N + λ M ∑ i = 1 N ξ O C j , (12) In Equation 12, ξ O H i denotes the Dirac point mass at the i th historical data sample O H i , ξ O C j denotes the Dirac point mass at the j th current data sample O C j , and λ > 1 is the proportional parameter, representing that the current data samples are more valuable.

Moreover, to describe the true distribution of target state deviation Δ s , the Wasserstein distance is considered to measure the distance between any two distributions. Lemma 1 defines the distance.

Lemma 1 (Wasserstein distance (Esfahani and Kuhn, 2018)): The Wasserstein distance between any probability distributions Θ and the nominal distribution Ω ̂ is defined as Equation 13: W p Θ , Ω ̂ = inf π ∈ Π Θ , Ω ̂ ∫ Ξ × Ξ Δ s − Δ s ′ π d Δ s , d Δ s ′ , (13) where ⋅ is a norm and Π ( Θ , Ω ̂ ) is the set of all joint probability distributions of Δ s and Δ s ′ with the marginals Θ and Ω ̂ , respectively.

According to Lemma 1, the true distribution of the state deviation Δ s is defined in a set as Equation 14: Λ Ω ̂ = Δ s ∈ P Ξ : W p Θ , Ω ̂ ≤ η k , (14) where Ξ is a closed set containing all possible values of Δ s , P ( Ξ ) is the family of all probability distributions supported on Ξ , and η k is the maximum Wasserstein distance.

On the basis of this, the state deviation Δ s is considered to ensure accuracy in this paper. The schematic diagram is shown in Figure 7.

The TD3 algorithm is a novel form of deep reinforcement learning algorithm, founded on the policy gradient algorithm and the actor–critic (AC) framework (Cheng et al., 2021; Duan et al., 2022). The proposed UAV model-driven state transition update mechanism in this paper is suitable and can be applied in continuous state and action spaces.

The TD3 algorithm consists of six networks, namely, the actor network π μ , the critic 1 network Q θ 1 , the critic 2 network Q θ 2 , the target actor network π μ ′ , the target critic 1 network Q θ 1 ′ , and the target critic 2 network Q θ 2 ′ . The network parameters for each network are μ , θ 1 , θ 2 , μ ′ , θ 1 ′ , and θ 2 ′ . Based on the two critic networks’ structure, the TD3 algorithm selects the smaller Q value to alleviate overestimation.

In each episode, TD3 selects an action a ∼ π ϕ ( s ) + ϵ , ϵ ∼ N ( 0 , σ ) , using exploration noise, and observes reward r and new state s ′ , where ϵ denotes the noise added to the output of the policy network to enhance the stability of the algorithm, as specified in Equation 15 (Fujimoto et al., 2018). a ̃ ← π φ ′ s ′ + ε , ε ∼ clip N 0 , δ ̃ , − c , c . (15)

Based on the structure of the two critic networks, the TD3 algorithm selects the minimum between the two estimates of Q θ 1 and Q θ 2 to avoid overestimation. The objective function of the TD3 algorithm is defined as Equation 16 (Fujimoto et al., 2018): y ← r + γ min i = 1,2 Q θ i ′ s ′ , a ̃ . (16)

The TD-error e T D of Q θ 1 and Q θ 2 is defined as Equation 17 (Fujimoto et al., 2018): e T D = y − Q θ i s , a ∣ θ i . (17)

The critics can be updated as Equation 18 (Fujimoto et al., 2018): θ i ← argmin θ i N − 1 ∑ y − Q θ i s , a 2 . (18)

The actor network is updated via the deterministic policy gradient as Equation 19 (Fujimoto et al., 2018): ∇ μ J μ = N − 1 ∑ ∇ a Q θ 1 s , a a = π μ s ∇ μ π μ s . (19)

The target networks are updated through a slow-moving update rate τ , following Equations 20, 21. Specifically, θ i ′ ← τ θ i + 1 − τ θ i ′ , (20) μ ′ ← τ μ + 1 − τ μ ′ . (21)

Due to the high dynamic and high-dimensional complexity in an actual aerial confrontation environment, the TD3 algorithm cannot effectively manage uncertainty and model the policy function and value function using a fully connected neural network. By adding a special gate structure to RNN, the LSTM network has a positive impact on processing time series data, thus enhancing the efficiency and effectiveness of the training algorithm (T. Ergen, 2018). It is considered that the state information of UAVs and targets in actual aerial confrontations exhibits time series characteristics. In this paper, an LSTM network is utilized to extract features from high-dimensional aerial confrontation situations with uncertainty. This structure aims to output valuable perceptual information and advance representation learning for sequential sample data. The policy and value functions are jointly approximated using a fully connected neural network. The TD3–LSTM algorithm architecture diagram is shown in Figure 8.

The general structure of the LSTM policy network is shown in Figure 9.

The structural parameters of the LSTM policy network are provided in Table 1.

The general structure of the LSTM value network is presented in Figure 10.

The structural parameters of the LSTM value network are shown in Table 2.

The initial position and attitude information of the UAV and the target are randomly initialized, and the target follows a strategy of uniform-speed straight flight. The confrontation trajectory of the UAV and the target in scenario 1 is depicted in Figure 11.

The real-time reward curve of the UAV and the target in scenario 1 is depicted in Figure 12.

On the basis of Figures 11, 12, the UAV gains a height advantage by climbing and then dives toward the target after reaching a specific altitude to gain velocity and angle advantages. This maneuver forces the target into the attack zone, leading to eventual triumph.

Figure 13 shows the cumulative reward curve of the UAV and the target.

In the early stages, the cumulative reward curve fluctuates because the UAV is unable to learn any effective strategies, leading to crashes or defeats in confrontations. With further training, the UAV learned effective maneuvers, developed an attack posture, locked onto the target, and gradually increased its cumulative reward value until convergence. It should be noted that the cumulative reward value may fluctuate slightly during the later stages of training due to uncertainty considerations. However, this variation will not impact the eventual acquisition of effective maneuvering strategies.

In scenario 2, the target employs the circle maneuver strategy. The trajectory of the UAV and the target during the confrontation is illustrated in Figure 14.

The real-time reward curve of the UAV and the target in scenario 2 is presented in Figure 15.

According to Figures 14, 15, the UAV climbs to gain a height advantage before diving toward the target to lock onto it for the first time. However, the target continues to circle due to the insufficient lock time. The UAV then ascends to gain a height advantage, locks onto the target again, and maintains the lock until it wins the confrontation.

Figure 16 illustrates the cumulative reward curve of the UAV and the target in scenario 2.

Similarly, the UAV failed to learn any useful strategies at the beginning of the training. Once the UAV learns efficient maneuvers, the cumulative reward value gradually increases until convergence. The cumulative reward value also fluctuates slightly in the late training period due to the effect of uncertainty.

In scenario 3, the target selects maneuvers according to expert rules. The confrontation trajectory of the UAV and the target in scenario 3 is illustrated in Figure 17.

The real-time reward curve of the UAV and the target in scenario 3 is shown in Figure 18.

From Figures 17, 18, the UAV and the target climb simultaneously to gain a height advantage during the initial phase. Next, both the UAV and the target choose a turning strategy to prevent being locked onto each other. Finally, the UAV performs a somersault maneuver to circle behind the target, completes the lock, and triumphs over the confrontation.

Figure 19 shows the cumulative reward curve of the UAV and the target.

Based on the results in Figure 19, the confrontation process is more intensive since the target has a specific maneuvering strategy. Because the UAV lacks environmental cognition, it is unable to develop effective strategies, leading to the UAV acting with high penalty values in the initial stages of training. The trend in the cumulative reward curve change indicates that the convergence speed is relatively slow, and the curve fluctuates sharply, reflecting the difficulty and complexity of aerial confrontation.

In scenario 4, the target selects the optimal maneuvers based on the current situation using a genetic algorithm. The confrontation trajectory of the UAV and the target in scenario 4 is shown in Figure 20.

The real-time reward curve of the UAV and the target in scenario 4 is shown in Figure 21.

As observed from Figures 20, 21, both the UAV and the target climb simultaneously, aiming to gain a height advantage at the start. Next, the target continues to climb in an attempt to gain a vantage point. The UAV takes the opportunity and maneuvers around the rear of the target during this process. Finally, the UAV completes the lock and wins the confrontation.

Figure 22 shows the cumulative reward curve of the UAV and the target.

As shown in Figure 22, the target selects the optimal maneuver based on the genetic algorithm optimization results under the current situation. The UAV has not yet learned the corresponding strategy and is unable to defeat the target in the early stages of training. The cumulative reward curve fluctuates sharply in the negative area within a certain number of iterations. As the UAV continues to learn the maneuvering strategy, it gradually becomes able to defeat the target. Compared with the three other scenarios, the convergence speed of the cumulative reward curve is slower, and the curve fluctuates sharply in scenario 4.