The robotic platform utilized in this study is the bipedal wheel-legged robot, Diablo (Direct Drive Tech, China), with simulations conducted in the Isaac Gym environment (Hwangbo et al., 2019; Lee et al., 2020; Kumar et al., 2021; Nahrendra et al., 2023). The overall control framework comprises a high-level decision controller and a low-level PD controller (Figure 1A). The high-level decision controller computes the target positions for each joint of the robot at a given time, which are then be scaled and transmitted to the low-level PD controller. The PD controller calculates the joint torques required to track these target positions, thereby driving the physics simulation.

We initially trained the robot using a standard reinforcement learning approach. The objective was to achieve multi-gait capabilities within a single decision-making network. In this section, we employ the PPO algorithm (Schulman et al., 2017) to train the robot, and both the Actor and Critic consist of a neural network with dimension of [512, 256, 128].

Based on the original PPO framework, we introduced a MoE mechanism in the Actor. The algorithm is illustrated in Figure 1B. The new algorithm maintains consistency with the original PPO algorithm in terms of state space, action space, and reward function. For the new algorithm that incorporates the MoE, the Actor is composed of a gating network with dimensions of [128, 64] and two expert networks sized [512, 256, 128]. At each time step, the final action output of the Actor corresponds to the expert network with the highest weight.

For a bipedal wheel-legged robot, the learning process begins with mastering fundamental equilibrium on planar surfaces, followed by tracking velocity commands via rolling locomotion. Subsequently, the robot negotiates slopes and overcomes vertical obstacles on complex terrains using leg-lifting gaits. Ultimately, it masters dynamically switching between rolling and leg-lifting modes. To this end, we designed a progressive training framework to induce these skills sequentially. Central to this approach is the two-phase training pipeline illustrated in Figure 2A, where each phase features distinct terrain compositions and complexity levels tailored to specific skills. The first phase consists exclusively of flat ground, slopes, and vertical stairs. An inclination threshold is defined such that slopes exceeding this value are rendered as vertical stairs in the simulation. This process evolves the initial policy π 0 into an obstacle-capable policy π 1 . The second phase involves transferring the policy π 1 to a comprehensive terrain set for re-training. To increase terrain complexity, Phase 2 incorporates wave terrain and discrete terrain into the training set, with Perlin noise added to introduce irregular surface perturbations. This phase serves as a comprehensive training ground for hybrid locomotion switching and dynamic balancing. Building upon the results of Phase 1, this process culminates in the final generalized policy π 2 .

Prior research demonstrates that curriculum learning—where task difficulty is incrementally increased—is a highly effective strategy for training complex locomotion policies. Accordingly, we propose an adaptive terrain curriculum designed to facilitate effective learning for bipedal wheel-legged robots in unstructured environments. At the onset of training, robots are uniformly distributed across the lowest difficulty tier of each terrain type, and the terrain difficulty level is dynamically updated based on the robot’s real-time performance. If the robot successfully traverses more than half the terrain length within a fixed time, it is deemed to have mastered the current difficulty. Consequently, the terrain level is incremented upon the next reset. Conversely, the difficulty level is decremented, allowing the robot to consolidate basic skills on simpler terrain. To mitigate catastrophic forgetting, robots that reach the maximum difficulty level are randomly reassigned to intermediate terrains.

Since the Actor in MoE-enhanced algorithm contains two networks of size [512, 256, 128], its total parameter count is approximately twice that of the standard PPO baseline. To exclude the impact of increased network capacity, this study introduces an additional large PPO baseline with a similar parameter count to MoE-enhanced algorithm. The large PPO baseline has a network size of [768, 384, 192], corresponding to scaling each layer of the Actor in the standard PPO baseline by a factor of 1.5. In both training phases, all three algorithms are trained with identical hyperparameters, including reward term weights, terrain settings, randomization parameters, and total training iterations. In the first phase, no Perlin noise is applied to the terrain, while in the second phase, Perlin noise is added to introduce irregular surface disturbances, as shown in Figure 3A. Both phases contain 6,000 training iterations, and the average rewards are compared, with results presented in Figures 3B,C, respectively. In the first phase, no obvious training collapse is observed for the three algorithms. Before around 1,600 iterations, their reward values and convergence speeds are nearly identical. After 1,600 iterations, the reward curves diverge significantly: the reward of the MoE-enhanced algorithm rises steadily, while that of the standard PPO baseline declines continuously. Although the reward of the large PPO baseline does not drop obviously, it shows no upward trend. By the end of the first phase, the reward of the MoE-enhanced algorithm is significantly higher than the other two baselines. Initially, the terrain mainly includes flat ground, slopes, and low stairs with small differences in dynamic properties. For low stairs, the robot can even traverse them via pure rolling, with dynamics similar to normal flat-ground locomotion and slope traversal. As the training progressed and stair heights increased, reaching a maximum height of 14 cm, which is much larger than the wheel radius, the robot can only overcome such obstacles through jumping or leg-lifting stepping, whose dynamics differ greatly from pure rolling. Training with a single network easily causes gradient update interference: after learning leg-lifting or jumping, the robot partially forgets pure rolling; similarly, improving pure rolling leads to partial forgetting of obstacle-crossing skills. Thus, the policy learned by a single network is a compromised solution. When facing higher stairs, its performance is insufficient, leading to stagnant or even severely degraded rewards.

In the second phase, Perlin noise with an amplitude of 6 cm is introduced into the terrain, and terrain types are extended to full terrain. Under these conditions, both the standard PPO baseline and large PPO baseline exhibit severe training instability: reward curves fail to converge, and training collapse frequently occurs. The primary cause is that Perlin noise increases terrain surface unevenness, leading to large fluctuations in the robot’s state observations at each time step. The policy continuously chases changing optimal behavior, resulting in reward volatility. If the frequency of terrain variations exceeds the adaptation speed of the policy, training collapse occurs. For single-network algorithms, limited learning capacity causes a trade-off dilemma, making stable training infeasible on rough terrain and preventing a robust final policy. In contrast, the MoE-enhanced algorithm maintains favorable reward curves throughout entire training process. In the first phase, its policy gradually adapts to increasing stair heights, with consistent reward rises. In the second phase, since the two expert networks’ skills were specialized in the first phase, with each expert responsible for distinct gait tasks, stable locomotion can be achieved by alternating between different experts even on rough terrain surfaces, thereby ensuring training stability. To further verify MoE-enhanced algorithm stability, we trained all three algorithms with different random seeds. The results are shown in Supplementary Figure S1 of the Supplementary Material. The results demonstrate that the MoE-enhanced algorithm performs stably across different seeds, whereas the single-network standard PPO and large PPO baselines exhibit significant variations across seeds and struggle to learn a stable policy. Overall, these results preliminarily validate the inherent limitations of single-network structures when handling multi-mode locomotion tasks in complex environments. The performance improvement of the MoE-enhanced algorithm arises primarily from its intrinsic mechanism, rather than the increase in total network parameters.

The variation in the average weights of the experts during the training process of the MoE-enhanced algorithm was extracted in Figure 3D. It is evident that in the first phase, there was a substantial difference in the call frequencies between the two expert networks; however, this disparity significantly decreased in the second phase. The primary reason for this change can be attributed to the relatively simple terrain in the first phase, which consisted solely of flat surfaces, slopes, and vertical stairs without any added Perlin noise. In this scenario, the use of the leg-lifting gait was required only for navigating the vertical stairs. When encountering vertical stairs, the robot needed to roll a certain distance on the step before transitioning to the next one, which meant that calls to the expert responsible for the leg-lifting gait were infrequent, occurring only during the transition from one stair to the next, while the majority of the time involved pure rolling. Consequently, during the first phase of training, the call frequency for the leg-lifting expert (Expert 2 in the figure) was significantly lower than that of the expert responsible for pure rolling (Expert 1), leading to a notable difference in their average weights as the number of training epochs increased. In contrast, the second phase introduced discrete terrains that required frequent alternation between pure rolling and the leg-lifting gait, along with the addition of Perlin noise, which exacerbated the irregularity of the terrain surface. This heightened the demand for the leg-lifting gait. Under these terrain conditions, the call frequency for Expert 2 in the second phase noticeably increased to accommodate the undulating ground. As the number of training epochs in the second phase increased, the difference in average weights between the two experts significantly diminished.

The locomotion policy derived from the MoE-enhanced algorithm was tested for ascending stairs, with the results presented in Figure 3E. When faced with stairs of a vertical height of 12 cm, the robot successfully executed an ideal leg-lifting gait. The motion breakdown was analyzed from both the frontal and lateral views, as shown in Figure 3E. With a speed command set at 0.6 m/s, the robot took approximately 0.4 s to clear the step. During the ascent, the robot first lifted its left wheel onto the next step before bringing up the right wheel, thereby effectively utilizing the leg-lifting gait to complete the stair-climbing task. In general, for bipedal robots encountering vertical obstacles, common motion policies include jumping or employing a leg-lifting gait. However, in this study, due to the incorporation of a significant leg-lifting reward, the resulting obstacle negotiation policy was exclusively the leg-lifting gait. Utilizing only the leg-lifting gait for obstacle negotiation effectively addresses several issues associated with jumping, such as problems related to body stability and the impacts and vibrations that may affect the robot’s internal components.

A specially designed testing environment is established to test the obstacle negotiation success rates of the two policies (Figure 4A), which included three types of terrain: rough terrain, discrete terrain, and vertical stairs. Each terrain type was set within a square area, with the rough terrain generated using Perlin noise at an amplitude of 6 cm, and the maximum vertical heights for the discrete terrain and vertical stairs set between 6 and 12 cm. During the testing process, a successful trial required the robot to remain upright throughout, not exit the terrain boundaries, and reach the endpoint. Each terrain with distinct difficulty was measured repeatedly, and 500 robots were used in each single test for statistical analysis. The results are shown in Figures 4B,C. From the results, the locomotion policy trained by the MoE-enhanced algorithm significantly outperforms the standard PPO baseline and the large PPO baseline. In terrains of all difficulty levels, its success rate exceeds 50% across all velocity commands. For the standard PPO baseline, insufficient obstacle-crossing performance can be clearly observed, especially under low velocity commands. Even in the simplest terrain with an obstacle height of 6 cm, its success rate remains low. In the terrain with a height of 12 cm, it performs poorly across all velocity commands. The large PPO baseline also exhibits a considerable performance gap compared with the MoE-enhanced algorithm, and this gap becomes increasingly pronounced as terrain difficulty rises. In particular, when the maximum vertical obstacle height reaches 12 cm, the success rate of the large PPO baseline is generally below 50%. Overall, these results further demonstrate that the superiority of the MoE-enhanced algorithm mainly stems from the inherent mechanism of MoE, and is not significantly affected by the specific amount of network parameters.

Based on this experiment, we further conducted ablation experiments to analyze the effects of MoE and phased training. We established an additional baseline trained directly for 12,000 iterations without phasing. Similarly, we evaluate the two training schemes using two metrics: the average training reward and the obstacle-crossing success rate on terrains of different heights. First, with respect to the average reward, training was performed with four different random seeds, and the reward results were statistically analyzed, as shown in Supplementary Figure S2 in the Supplementary Material. It can be clearly observed that the MoE-enhanced algorithm achieves higher reward values under phased training than under non-phased training. Subsequently, the locomotion policies obtained by the two training methods were tested for obstacle-crossing success rate, with the results presented in Supplementary Figure S3 in the Supplementary Material. The results show that the performance difference between the two policies is not significant on relatively simple terrains. When the maximum vertical height of obstacles reaches 12 cm, the success rate of the policy trained with phased training is significantly higher than that of the non-phased policy. This validates that phased training can improve the upper bound of obstacle-crossing performance. Combined with the comparison results against the PPO algorithm using a single network, it can be concluded that in the MoE-enhanced algorithm proposed in this study, the mechanism of MoE itself plays a more dominant role, while phased training serves as an auxiliary approach to further improve the algorithm’s performance.