Reinforcement learning has been increasingly applied to robotic navigation tasks. Early RL-based navigation used discrete controllers or small state spaces, but modern DRL uses neural networks to handle high-dimensional inputs (e.g., images or LIDAR). For instance, Tai et al. (2017) demonstrated a mapless navigation planner trained end-to-end with asynchronous DDPG using a sparse 10-dimensional laser scan and relative target position; their planner transferred from simulation to a real robot without explicit mapping.

More recent works address exploration and sample efficiency in mapless navigation. For example, Yadav et al. (2023) employed Soft Actor-Critic (SAC) with curriculum learning and dual prioritized replay for mobile robot navigation, highlighting the challenge of sparse rewards. Ruan et al. (2022) introduced a curiosity-based intrinsic motivation combined with a temporal cognition module for visual navigation, suggesting that self-generated rewards can improve exploration. Li et al. (2023) combined a human-designed gap-detection planner with a DRL agent, effectively incorporating prior knowledge to accelerate learning in complex scenes. These approaches emphasize the value of intrinsic rewards or hybrid methods to overcome DRL limitations in navigation.

Parallel to DRL, classical navigation methods remain widely used. Techniques such as potential fields (Khatib, 1986) and the Dynamic Window Approach (DWA) (Fox et al., 1997) perform reactive obstacle avoidance given a map or sensor data, but often require careful design and can get stuck in local minima. Hybrid methods have been explored (Liu et al., 2024): proposed TD3-DWA, combining TD3 with DWA by treating DWA parameters as tunable by the policy. Our work avoids the use of traditional planners such as DWA or A*, relying instead on learning-based policies guided by goal-relative information and onboard sensing.

Our baseline algorithm is Twin Delayed DDPG (TD3) (Fujimoto et al., 2018), an off-policy actor-critic method for continuous control. TD3 addresses function-approximation error in DDPG (Lillicrap et al., 2016) by three tricks: (1) using two independent Q-networks and taking the minimum for target computation (clipped double-Q learning); (2) adding clipped noise to the target policy (policy smoothing regularization); and (3) delaying actor updates relative to critic updates. TD3 typically outperforms vanilla DDPG and some on-policy methods due to reduced overestimation and variance. Several works have since extended or optimized TD3 for robotics.

Actor-critic methods like TD3, DDPG, and SAC (Haarnoja et al., 2018) have been favored in robotics for handling continuous actions and sample reuse via replay. SAC adds entropy maximization for exploration stability (Haarnoja et al., 2018). Asynchronous variants like A3C/A2C (Mnih et al., 2016) also apply to navigation, but rely on on-policy updates. In our work we build on TD3 due to its strong performance and off-policy efficiency, while incorporating additional modules to tackle exploration in unknown environments.

Intrinsic reward strategies have proven effective in improving exploration when extrinsic rewards are sparse. A common approach is to train a predictor network and use its error as a curiosity signal (Pathak et al., 2017). For instance, Pathak et al. (2017) defined curiosity as the error in predicting the consequence of actions in a learned feature space, encouraging exploration of novel states. Others have used prediction uncertainty or state-counts for intrinsic rewards. In line with this, our Improved TD3 uses the error of a learned state-transition predictor (operating in latent space) as an intrinsic reward, thus motivating the agent to visit states where its model is poor.

Several recent studies have extended the TD3 framework for mobile robot navigation. Jeng and Chiang (2023) introduced survival-based penalty shaping in TD3 to improve convergence and reduce collision. Yang et al. (2024) added an intrinsic curiosity module (ICM) and randomness-enhancement to help the agent escape sparse-reward local optima. Kashyap and Konathalapalli (2025) compared TD3 against DDPG and DQN on a TurtleBot3 platform using ROS2 and LiDAR, showing that TD3 offered the best performance.

Neamah and Mayorga Mayorga (2024) proposed Optimized TD3 (O-TD3) with prioritized replay and parallel critic updates, achieving high success in human-crowded scenarios. Raj and Kos (2024) implemented dynamic delay updates, Ornstein–Uhlenbeck noise, and curriculum transfer learning on top of TD3. Huang et al. (2024) developed LP-TD3, which fuses LSTM memory, prioritized experience replay, and intrinsic curiosity rewards, significantly improving convergence and generalization.

We introduce a learned latent encoding of the state to capture useful features. Let the original state vector be s ∈ R n (e.g., LIDAR readings plus robot pose). An encoder network E ψ ( s ) produces a latent embedding z s = E ψ ( s ) ∈ R d (we use d = 256 ). Concretely, E ψ is a feedforward network with two hidden layers of size h enc = 256 and nonlinear activations (e.g., ReLU), followed by an AvgL1Norm layer normalizing the output. Formally: z s = A v g L 1 N o r m σ W 3 σ W 2 σ W 1 s , where σ is the ReLU activation, W ( i ) are weight matrices, and AvgL1Norm normalizes the vector.

Given z s and an action a ∈ R m , we also define a predictor network that estimates the next latent state. Specifically, we compute z ̂ s ′ = P ψ ( z s , a ) via two hidden layers of size 256 and ReLU activation: z ̂ s ′ = P ψ z s , a = W 5   σ W 4 z s ; a + b 5 , with input [ z s ; a ] ∈ R d + m concatenation. This predictor shares weights with the encoder beyond the first layer, ensuring P and E use the same latent basis. The next latent of the actual next state is z s ′ = E ψ ( s t + 1 ) .

The actor (policy) network is modified to condition on the latent: instead of taking only s , it first encodes s to z s and then computes the action. In practice we implement a split architecture: the actor first processes s through a hidden layer, concatenates the resulting features with z s , then applies two more layers and a tanh output layer. The critic networks likewise take ( s , a ) , but also incorporate z s and z s a = P ψ ( z s , a ) as additional inputs: each Q-network processes [ z s , z s a ] alongside [ s , a ] (with appropriate layers). This way, the critic value depends on both the raw state-action and the learned latent embedding.

We incorporate an intrinsic reward r t int to encourage exploration of states that are hard to predict. After sampling a transition ( s t , a t , s t + 1 ) from the replay buffer, we compute the latent encodings z t = E ψ ( s t ) and z t + 1 = E ψ ( s t + 1 ) , and the predicted next latent z ̂ t + 1 = P ψ ( z t , a t ) .

We define the raw prediction error as e t = ‖ z ̂ t + 1 − z t + 1 ‖ 2 .

Let the running maximum be M t = max M t − 1 ,   e t , M 0 = ε .

We set ε = 1 0 − 8 . The normalized intrinsic reward is r ̃ t int = e t M t + ε ∈ 0,1 , and the total reward used for TD target computation is R t = r t ext + β   r ̃ t int , β = 0.1 , where r t ext is the environment reward. To prevent intrinsic bonuses from destabilizing learning when the predictor is initially inaccurate, we employ two controls. First, we scale the bonus by a small constant β = 0.1 , keeping it modest relative to extrinsic rewards. Second, we normalize the prediction error e t = ‖ z ̂ t + 1 − z t + 1 ‖ 2 by a running maximum M t = max ( M t − 1 ,   e t ) with M 0 = ε . The normalized curiosity r ̃ t int = e t / ( M t + ε ) ∈ [ 0,1 ] bounds the intrinsic term β   r ̃ t int within [ 0 , β ] , avoiding large spikes early in training and preventing the intrinsic component from dominating the learning signal.

While the intrinsic reward shapes exploration, the encoder–predictor must itself be learned. At each training step, we update the encoder parameters ψ to minimize the prediction error. Specifically, we compute the loss for a sampled batch: L enc = E ‖ z ̂ t + 1 − z t + 1 ‖ 2 2 + λ MMD   M M D z t , z ̂ t + 1 , where M M D ( ⋅ , ⋅ ) is the maximum mean discrepancy between the distribution of encoded states z t and predicted states z ̂ t + 1 (using a Gaussian kernel) (Gretton et al., 2012). We use λ MMD = 1.0 . The MMD term acts as a disentanglement or regularization loss: it encourages the latent distribution of ( s , a ) transitions to match the marginal latent distribution of states, promoting consistency. In practice, the code computes MMD by sampling pairwise kernel differences. The encoder loss is then backpropagated and the encoder parameters ψ updated by Adam. We use a learning rate of 3 × 1 0 − 4 for the encoder.

After computing the intrinsic reward and updating the encoder, we update the critics. We first form the target Q-value using the combined reward: a t + 1 ′ = π ϕ ′ s t + 1 , Q target = R t + γ min i = 1,2 Q θ i ′ s t + 1 , a t + 1 ′ , z t + 1 , z ̂ t + 1 , where γ = 0.99 is the discount factor. The critic networks Q θ 1 , Q θ 2 are then trained to regress to Q target . We use standard MSE loss and update θ 1 , θ 2 via Adam with learning rate 3 × 1 0 − 4 .

We update the actor every p o l i c y _ f r e q = 2 critic updates. Specifically, we compute the actor loss as the negative Q-value under the current critic: L actor = − E s ∼ B Q θ 1 s , π ϕ s , z s , z ̂ s , where z s = E ψ ( s ) and z ̂ s = P ψ ( z s , π ϕ ( s ) ) for on-policy action. This encourages the policy to choose actions with high predicted Q-value. We take a gradient step on ϕ with learning rate 3 × 1 0 − 4 .

We use soft updates for the target networks every training step: θ i ′ ← τ θ i + 1 − τ θ i ′ , ϕ ′ ← τ ϕ + 1 − τ ϕ ′ , with τ = 0.005 as is standard in TD3.

Algorithm 2 provides the high-level training loop. For clarity, we summarize in words.

Intrinsic/extrinsic reward: Each transition yields extrinsic reward plus a scaled curiosity bonus based on normalized latent prediction error.

During training, we periodically evaluate the current policy (without exploration noise) over several episodes to measure performance. We plot the mean total reward per evaluation epoch to track learning progress. After training, we report: (1) Success Rate: fraction of episodes reaching the goal; (2) Collision Rate: fraction of episodes ending in collision (Timeouts are treated as failures and counted under collisions); (3) Average Time: mean steps to reach goal (for successes).

Figure 1 shows the evaluation rewards over training epochs. The shaded regions show individual episode rewards, and the solid lines are running averages. We see that Improved TD3 quickly attains higher rewards and stabilizes around a larger mean. This indicates that the intrinsic rewards and encoding help the agent learn a more effective navigation policy.

To assess the learning behavior of the proposed Improved TD3 agent, we analyzed several key training metrics using TensorBoard visualizations. Comparing ITD3 with a baseline TD3 (without the encoder–predictor) confirms these advantages: ITD3’s evaluation reward curve rises more quickly and stabilizes with a narrower confidence band, indicating faster learning and reduced variance. This aligns with the expectation that compact latent representations facilitate sample-efficient value approximation. These include the Q-values, target Q-values, Q-max, loss, and their corresponding target counterparts.

To evaluate the effectiveness of our Improved TD3 (ITD3) framework, we conducted extensive testing. The metrics used include success rate, collision rate and average time to goal as shown in Table 2.

The Improved TD3 agent achieves a success rate of 93.14%, indicating reliable goal-reaching behavior across diverse and previously unseen environments. The collision rate is as low as 6.84%, showing that the agent learns to avoid obstacles effectively. Moreover, the average number of steps to reach the goal is reduced to 12.91, suggesting faster convergence and efficient navigation.

These improvements stem from our enhancements to the standard TD3 framework, including latent state encoding, intrinsic curiosity-driven rewards, and MMD-based regularization. Together, they enable the robot to explore more intelligently, learn more efficiently, and generalize better across different scenarios.

Figure 5 shows the top view of the Gazebo Simulation Environment, in which the black circle under the spotlight is the robot, and the white circles are the obstacles. The walls and the surroundings also act as obstacles. Figure 6 is the rviz of the same Gazebo environment depicted in Figure 5. Here, the red arcs represent the LIDAR scan of the robot. We can visualise the presence of obstacles through this laser scan, as shown by the green curves in Figure 6. The sensor data indicates four obstacles clearly, as shown in Figure 5, while the remaining two are already accounted for but not directly visible, as they are positioned behind the other two. (In total, there are six obstacles, but due to their alignment, two are hidden behind others, which is why the robot only senses four directly).

Figure 7 demonstrates that the robot successfully moved toward the goal location, marked by the green dot, while avoiding all obstacles. The red line represents the path taken by the robot. The curves along this path indicate the presence of obstacles that the robot avoided while navigating successfully toward the goal. The other red lines depict the previous trajectories that the robot took. A detailed observation of the laser scan reveals that the robot has effectively sensed the obstacles within its range, including the walls and surrounding structures. Figure 8 demonstrates that the robot has successfully evaded all the obstacles in its path while moving towards the goal location. The policy learned to drive the Robot forward while avoiding collisions. The presence of intrinsic reward during training helped the agent learn to maintain broad coverage with its sensors, encouraging diverse exploration and helping avoid local traps such as corners or walls. Across many trials, the Improved TD3 robot consistently navigates effectively. These qualitative observations align with the quantitative gains: improved exploration leads to faster and safer goal-reaching.

We evaluated our Improved TD3 (ITD3) against the state of the art by comparing key metrics reported in related work, including training efficiency, success and collision rates and cumulative reward. For example, Jeng and Chiang (2023) introduced a survival-penalty reward shaping in an end-to-end TD3/DDPG framework. They report that TD3 converges faster and more stably than DDPG, yielding a higher task success rate in evaluation. In their parking and maze scenarios, TD3 achieved a markedly higher success rate while keeping collisions low, demonstrating the benefit of their collision-penalty reward shaping.

Yang et al. (2024) focus on intrinsic rewards. By integrating an Intrinsic Curiosity Module (ICM) and a Randomness-Enhanced Module (REM) with TD3, they report that their ICM + REM-TD3 method achieved an 83.5% success rate out of 1000 test episodes–significantly higher than baseline DRL methods–with only 3 episodes exceeding the time limit. This corresponds to a collision rate of only about 16.2% (compared to 23.1% for vanilla TD3 in their Table 1), and a substantial reduction in average episode length. These results indicate that the intrinsic reward mechanism accelerated learning and exploration, improving success and reducing collisions. (Yang et al., 2024). also report the performance of an A3C baseline, which achieved a 62.5% success rate with a 32.4% collision rate in their Table 1. Compared to TD3 and TD3-based intrinsic-reward variants, A3C shows noticeably weaker navigation and poorer obstacle-avoidance capability, indicating limited exploration efficiency under sparse-reward conditions.

Kashyap and Konathalapalli (2025) compare TD3, DDPG, and DQN on a TurtleBot3 platform in static and dynamic obstacle courses. They report that TD3 yields the most efficient navigation overall. Quantitatively, TD3 reduced travel time by roughly 14%–27% relative to DDPG and 28%–55% relative to DQN, while also cutting collision counts per episode by similar factors. In other words, TD3 achieved the highest success rates and average rewards across diverse scenarios, leveraging ROS2 and LiDAR integration for robust perception. The combination of high success and low collisions in these tests underscores the robustness of TD3 under more complex, sensor-rich settings.

Neamah and Mayorga Mayorga (2024) propose an optimized TD3 tuned for crowded human–robot interaction. They report exceptionally high performance: after training for 12,000 episodes, their method achieved a 92% success rate in evaluation, requiring on average only 772 steps (about 11 s) to reach each target. By contrast, their baseline DQN only achieved 64% success. These results highlight that a well-tuned TD3 can learn very efficient, collision-free navigation policies even in highly dynamic human-populated environments.

In addition to TD3-based methods, other reinforcement learning algorithms have also been applied to mapless navigation. On-policy methods such as A3C (Mnih et al., 2016) and PPO (Schulman et al., 2017) learn directly from raw sensor data and have demonstrated success in similar ROS-based environments, but typically require more training samples and careful reward shaping. For instance, in a ROS 2 + Gazebo setup with LiDAR inputs, a PPO-trained TurtleBot3 achieved an 82% success rate after extensive training (Schulman et al., 2017), Similarly, A3C has been used for end-to-end navigation (Mirowski et al., 2017), but its sample-inefficient on-policy updates often result in moderate performance unless combined with auxiliary objectives or curricula. Off-policy Soft Actor-Critic (SAC) (Haarnoja et al., 2018) tends to be more sample-efficient; however, SAC alone can struggle with sparse rewards in navigation. Recent extensions like SAC-LSTM (Zhang and Chen, 2023) (incorporating memory) report success rates of 89% in highly dynamic scenarios), highlighting that augmenting SAC with memory and exploration modules improves performance. These comparisons suggest that our ITD3’s combination of intrinsic motivation and representation learning yields competitive or better performance than A3C, PPO, and SAC in similar mapless navigation tasks, with faster convergence and higher success rates.

Other recent works likewise report strong performance. For instance, Raj and Kos (2024) used a DQN-based DRL controller and demonstrated high target-reaching success rates and reward gains compared to PPO on complex mazes. Huang et al. (2024) apply an improved TD3 with LSTM and curiosity modules to mapless inspection tasks, reporting performance improvements (e.g., higher success and reward) over prior baselines (they cite “curiosity-driven” exploration). Taken together, these benchmarks show that our Improved TD3 (ITD3) consistently matches or exceeds the success and efficiency of recent mapless navigation methods: all report success rates in the 70%–90% range with low collision rates, and our results fall at the high end of this range. In parallel to navigation, related perception-driven architectures have improved robustness in thermal or cross-modal settings. Deep-IRTarget leverages dual-domain spatial and frequency cues to enhance thermal target localization (Zhang et al., 2022), while DFANet preserves modality-specific IR/RGB information before attention-based fusion, yielding more reliable downstream representations (Zhang et al., 2024). Further, cognition-driven structural-prior modeling has been shown effective for instance-dependent label noise, where STMN aligns transition matrices with human-plausible structural constraints to suppress invalid label transitions (Zhang et al., 2025). These findings reinforce the broader value of structured representation learning and prior-guided regularization, which conceptually motivate our use of compact latent encoding with regularization and intrinsic learning signals. In addition to scalar metrics, we note key architectural enhancements across these works. Many use advanced reward shaping: for example, Jeng and Chiang (2023) employ a “survival penalty” function to combat sparse rewards, enabling collision-free paths. Yang et al. (2024) integrate a curiosity-driven ICM and a randomness-enhancement (REM) module to provide dense intrinsic rewards and encourage exploration. Although the encoder–predictor adds modest complexity, the benefits of compressing the observation space (improved sample efficiency and stability) and the auxiliary predictive signal justify its inclusion. Representation-learning suggests that an optimal latent space should be low-dimensional yet informative; our architecture adheres to this principle. The latent dimension d = 256 was selected to balance information retention and computational efficiency. According to state representation learning principles, the representation space should be constrained to a low dimensionality while remaining sufficiently informative. Dimensions that are too high risk redundancy and overfitting, whereas overly compressed spaces omit critical features. Our experiments indicated that d = 256 achieves this balance for the sensor modalities considered. Kashyap and Konathalapalli (2025) uses middleware (ROS2) and LiDAR sensors to improve perception and robustness. Huang et al. (2024) augment TD3 with recurrent memory (LSTM) and curiosity-based rewards. Collectively, these innovations (survival penalties, intrinsic rewards, sensor fusion, memory models) are aimed at improving convergence and final policy quality. In our design, we incorporate an adaptive reward scheme to stabilize learning. By comparison with these, our method achieves better training efficiency and navigation performance.