The robot we used in this paper is the wheel-legged robot Ollie. This robot was self-developed by Tencent Robotics X and was first released in both the plenary talk and the research paper (Wang et al., 2021) at ICRA 2021. Ollie (mechanical model shown in Figure 2) is designed using eight driving motors: four motors on the hips, two motors on the tail and two motors on the wheels. The five-bar linkage mechanism is used so that each leg is actuated by two motors on the base, without actuators on the knees. Based on the mechanical design, each five-bar leg is able to move along the x and z-axis in the leg plane, which keeps perpendicular to the robot base. Except for two-wheel balancing, the robot has the ability to switch flexibly between two-wheel and three-wheel modes. Moreover, the attitude of the base can be adjusted by changing the height of each leg, i.e., distance between the center of the wheel and the base in the leg plane. With an optimal mechanical design of legs, the robot can stride over obstacles, jump, and flip, demonstrating its strong ability to realize dynamic movements (Ackerman, 2021). The total weight of Ollie is around 15 kg. The height of the robot is between 0.33 and 0.7 m.

In order to keep Ollie moving with agile and flexible body poses at the same time of wheel balancing, the whole-body control (WBC) is used. The robotic control framework is shown in Figure 3. See details of the whole-body dynamics model and optimization formulation in the authors' previous publication (Zhang et al., 2022).

Before the WBC module, an individual controller is designed to generate the reference torques τ^des for the balancing task due to the nonminimum-phase property of the wheel-legged robot. Conventionally, references of the balancing task in the WBC are usually generated by controller design based on a simplified model (Klemm et al., 2020; Murtaza et al., 2020) or by a manual tuning controller (Zambella et al., 2019). To overcome the aforementioned model uncertainties and inaccuracy of model parameters and external disturbances (formally stated in Section 2.3), the AOOR is used to calculate balancing task references.

Other task references, such as attitudes and height of the robot, are obtained from the motion generation and classic controllers.

The problem mentioned in Zhang et al. (2022) is that it is hard for such a wheel-legged robot to keep standing completely still in different working conditions. In order to make the robot stand still, the center of mass (CoM) and the middle of wheels should be vertical. Therefore, the accurate estimation of the CoM position is crucial for model-based controllers, but the deviation from the expected position is inevitable in practice due to the nonuniform mass distribution and the assembly errors. This is presented in the sagittal plane in Figure 2, where the CoM of the floating base is assumed to be consistent with the geometric center but they are different in practice.

In the balancing task, the pitch angle θ¯, pitch angular velocity θ¯˙ and the robot's linear velocity x¯˙ are regulated to zero by controlling the motor torque τ at the wheel. The balancing system can be expressed in the linear form

where ζ¯=[θ¯ θ¯˙ x¯˙]T, A ∈ ℝ^n×n and B ∈ ℝ^n×m (n = 3, m = 1). It must be highlighted that, due to the uncertain factors of the robotic system, it is hard to model an accurate expression of A and B, and they are not used in this paper.

Particularly, considering the deviation of the CoM, the real pitch angle is the subtraction of the measured value θ and an offset Δθ, i.e., θ¯=θ-Δθ (Note that subtraction is used to keep consistent with the formulation in Section 3), while the real value and the measured value are equal for the other two states (θ¯˙=θ˙, x¯˙=ẋ). Then, the dynamic model (Equation 1) is equivalent to

where ζ=[θ,θ˙,ẋ]T, D ∈ ℝ^n×q (q = 1) and D = A[−Δ, θ, 0, 0]^T.

In addition, to stop the robot at a fixed position, ẋ is chosen as the output to be regulated to zero, and the error is

where C = [0 0 1].

In this paper, we consider the balancing task in the WBC framework with the system described in Equations (2) and (3) with the unknown parameters A, B, and Δθ. The objective is to stabilize the system by optimal control adaptively and to regulate the output to zero.

The standard state feedback control of the system (Equation 1) is

when (A, B) is stabilizable, where K ∈ ℝ^n×1 is a stabilizing feedback gain. The control law can be rewritten in the equivalent form for the system (Equation 2) as

where L = KZ + T with Z = [Δθ 0 0]^T. In the control law (Equation 5), the first term regulates the state ζ, and the second term compensates the uncertainty D. Then, the original system (Equation 1) is asymptotically stable (limt→∞ζ¯=0) if the unknown Z and T are solvable by the regulator equation (see the proof in Huang, 2004):

Due to the specific underactuated dynamics of the wheel-legged robot, there exists a unique solution of the regulation equation. It is self-evident because the steady-state must be ζ = [Δθ, 0, 0] for the standstill condition, and the torque is then fixed. However, for other types of robots, there may exist multiple solutions, and an optimization problem can be designed to solve the optimal solution (Gao and Jiang, 2016).

In this paper, the optimality is represented by solving the following constrained optimization problem for the state feedback control.

where Q ∈ ℝ^n×n is positive-definite and R ∈ ℝ is positive. The optimal solution is represented by the optimal feedback gain

where the positive-definite P^* is the solution of the algebraic Riccati equation (ARE) (Lewis et al., 1995).

In the data-driven approach, the main objective is to remove the dependency on the model parameter A, B and D, and determine the optimal solution (K^*, P^*) and (Z, T).

To determine the optimal feedback gain K^*, by utilizing the collected data ζ(t) and τ(t), the optimization problem (Equation 7) is solved iteratively. Particularly, the dependency on A is removed by the Lyapunov equation

where j is the index of iteration. Meanwhile, the dependency on B is removed by the gain iteration

In addition, D is set as a variable to be determined in each iteration. When K and P converge, the results can be proved as the optimal solutions K^* and P^* (Kleinman, 1968).

To determine the regulation parameters (Z, T), a basis of Z is defined as Z^=[1 0 0]T such that Z=αZ^ with a scalar α and the second equation in the regulator (Equation 6) is satisfied. Notice that the construction of the basis of Z is slightly different from Gao and Jiang (2016) in this particular application on Ollie. Then, by defining a Sylvester map S:ℝⁿ↦ℝⁿ by S(Z) = −AZ, the first equation in the regulation equation is rewritten as

where B is represented by Equation (10) using K^* and P^* from the previous step, and D is solved in the previous step.

Then, if S(Z^) is available, the pair (Z, T) can be computed by solving the following linear equation

Finally, by setting L = K^*Z + T, the optimal output regulation law in Equation (5) is attained.

The proposed AOOR is provided in Algorithm 1, with the related matrices defined in Equations (13) and (14), where the notation vec() denotes the vectorization, and ⊗ denotes the Kronecker product. It is suggested to check the conditioning number of Ψ and Ψ^ to avoid potential numerical problems caused by ill-conditioned matrix. The general idea of the proof follows from Gao and Jiang (2016), but the AOOR-based control is applied for the specific problem on the balance augmentation of wheel-legged robots. Due to the page limit, the detailed proof is omitted.

In the experiment, the control framework in Figure 3 is realized by the CPU PICO-WHU4 at a frequency of 1 k Hz. The rotation of the floating base is measured by the onboard IMU at a frequency of 400 Hz. The experimental data are saved at a frequency of 1 k Hz. The rotation of each actuated joint is measured by the motor encoder. The reference velocity and the reference height of the robot are set by the operator using a remote controller.

We train the data-driven optimal output regulation controller for Ollie in four scenarios to deal with uncertainties in model and disturbances. In our previous work (Cui et al., 2021; Zhang et al., 2022), the robot kept balanced, but it could not stand still, which was mentioned as a limitation. Hence, in the first two scenarios, we deal with the asymptotic stability problem in our previous works where the robot is at the height 0.33 m (Figure 4A) and the height 0.5 m (Figure 4B). Furthermore, we invoke two more challenging scenarios. In the third scenario, we mount an eccentric load of 3.5 kg in the front of the floating base (Figure 4C). As the floating base is about 11 kg, the heavy load would change the equilibrium point significantly. In the fourth scenario, we train the robot on a slope (Figure 4D). It is clear that extra torque is required to compensate for the component of gravity in the direction of the slope, which in turn implies that the equilibrium point changes. Notice that no information of the load (e.g., mass, shape, location) and the slope (e.g., gradient) is used in the training process, and they are treated as unknown disturbances.

During the training, the initial controller gain K₀ = [−74 − 26 − 10] is used, which is close to the resultant controller obtained from the VI training in Zhang et al. (2022). As the controller is stabilizing, the robot can move forward and backward repeatedly according to the remote control and the exploration noise β(t) = 1.6sin(8πt) + 0.8cos(12πt) is used to trigger more informative data. Training data of 6 s are collected for each experiment with the sampling interval 0.015 s. It means that, in Algorithm 1, t₀ = 0, t_s = 6 s, and t_k+1 − t_k = 0.015 s. Considering the sampling rate of 1 k Hz, there are 400 elements in each matrix in Equations (13) and (14). As suggested in Jiang and Jiang (2017), it is a good practice to guarantee the full rank condition in Alglorithm 1 by collecting data for which every matrix in Equations (13) and (14) has elements more than twice as many as the required rank. Hence, the full rank condition would be satisfied easily in this work.

The training data when the robot is at the height 0.33 m and when the robot is at the height 0.5 m on the slope are shown in Figures 5, 6 respectively as an example, where the oscillation implies the exploration noise. Other training data are given in the accompanying video.

Finally, by setting Q = diag(900, 400, 150), R = 1 in Algorithm 1, the control parameters of the four training converge to

by the stopping criterion ϵ = 10⁻⁴ in 19, 18, 16, and 21 iterations, respectively (Figure 7). It should be highlighted that the fast convergence enables Algorithm 1 to be implemented online. Moreover, the robustness of the ADP-based algorithm guarantees the convergence to a small neighborhood of the optimal solution when noisy data are used (Pang et al., 2022).

The tuning of the parameter Q and R follows the standard process of LQR design. It is noticed in Zhang et al. (2022) and its accompanying video that the motor input is oscillating, which is caused by the overlarge regulation on the pitch angle of the robot. In this paper, the first diagonal term of Q is reduced accordingly. Moreover, by manual testing, the convergence of the algorithm is most sensitive on the third diagonal term of Q, whose feasible range is from 90 to 280. In addition, the second diagonal term of Q should be larger than 180, and the first diagonal term can be any positive value. These results indicate the robustness of the convergence of Algorithm 1 on the parameters Q and R.

In the accompanying video, the trained controllers are updated to the robot online after the data collection. Here, we compare the initial controller and the result controller of AOOR directly to avoid the effect on the visual sense caused by the movement and exploration noises in the training process.

In the first experiment, the robot is at the height 0.33 m, with experimental data given in Figure 8. The initial controller K₀ is used at first, and the experimenter uses the remote controller to force the robot to stand still. Then, after releasing the remote controller, the robot moves forward continuously in the first 2 s, which is indicated by the red line in Figure 8C. Then, by enabling the updated controller with Ka* and L_a at t = 2 s, the robot decelerates sharply and stays still in the following 6 s, which is indicated by the blue line in Figure 8C. In order to realize the fast regulation, the motor input is a little oscillating, but the effects on the pitch angle (Figure 8A) and the linear velocity (Figure 8C) are acceptable. The still image of the testing process is shown in Figure 9A. Combining the still image and the testing data, it is clear that the average velocity when using the AOOR is much slower than the initial case, reflected by the smaller displacement during a longer time. The small displacement during the AOOR period is caused by the convergence from the initial velocity to zero, which is naturally inevitable.

The testing processes when the robot is at the height 0.5 m and with the load are similar. By the testing data (Figures 10, 11), the robot decelerates and stands still (ẋ → 0) after the AOOR is enabled. The small displacement, including the deceleration phase, is demonstrated in Figures 9B, C respectively.

For the testing on the slope, the method is similar. The difference is that the AOOR is used in the beginning to compare with the initial control latter. The reason is that, if the initial control is used in the beginning, the robot may leave the slope quickly before enabling the AOOR. The outstanding regulation performance is indicated by the data (Figure 12) together with the still image (Figure 9D).

The comparison of the controllers are summarized in Table 1. The much smaller displacement within the longer time by the AOOR indicates the strong regulation. The nonzero displacement includes the deceleration phase after enabling the AOOR, and any slight error in the regulation term L may also cause a small displacement in the steady state.

The AOOR is applied when the robot is balancing a rolling cylindrical water bottle on the top. A PD controller is used to tune the floating base by the feedback of the bottle position (s_b) and velocity (ṡ_b) with respect to the center of the floating base in the x direction by the following laws

where q₍₅₎ denotes the pitch angle of the floating base of the robot. Hence, the controller aims to regulate the ball at the target position (sb*=0) with the PD gain kpball, kdball, and to keep the floating base horizontal (q₍₅₎ = 0) wit the PD gain kpbase, kdbase. The signs depend on the positive directions of the predefined coordinates. In addition, the state s_b is updated by a self-developed tactile sensor (Zhao et al., 2022) by Tencent Robotics X.

First, with the initial controller, the robot keeps moving forward as shown in Figure 13. In consequence, when the bottle is placed on the top, the overlarge relative velocity between the robot and the bottle is a destructive initial condition for the bottle balancing task. As shown in the figure, the control is not converging, and the bottle drops in 2 s.

Next, with the AOOR, the robot almost stands still, so the ignorable relative velocity provides an ideal initial condition for the bottle balancing task. As shown in Figure 14, the bottle is controlled to the middle of the top. Combining with the mobility over complex terrains, this result shows the possibility to apply the wheel-legged robot to deliver goods for which the contact with the robot is not closed.