Due to advantages such as repeatability, accuracy, and lack of physical fatigue, autonomous systems have been increasingly utilized to perform tasks that are dull, dirty, or dangerous. Autonomy in safety-critical applications such as autonomous driving and unmanned flight relies on the ability to synthesize safe controllers. To improve robustness to parametric uncertainties and changing objectives and models, autonomous systems also need the ability to simultaneously synthesize and execute control policies online and in real time. This paper concerns reinforcement learning (RL), which has been established as an effective tool for safe policy synthesis for both known and uncertain dynamical systems with finite state and action spaces [see, e.g., Sutton and Barto (1998); Doya (2000)].

RL typically requires a large number of iterations due to sample inefficiency [see, e.g., Sutton and Barto (1998)]. Sample efficiency in RL can be improved via model-based reinforcement learning (MBRL); however, MBRL methods are prone to failure due to inaccurate models [see, e.g., Kamalapurkar et al. (2016a,b, 2018)]. Online MBRL methods that handle modeling uncertainties are motivated by complex tasks that require systems to operate in dynamic environments with changing objectives and system models. Accurate models of the system and the environment are generally not available due to sparsity of data. In the past, MBRL techniques under the umbrella of approximate dynamic programming (ADP) have been successfully utilized to solve reinforcement learning problems online with model uncertainty [see, e.g., Modares et al. (2013); Kiumarsi et al. (2014); Qin et al. (2014)]. ADP utilizes parametric methods such as neural networks (NNs) to approximate the value function and the system model online. By obtaining an approximation of both the value function and the system model, a stable closed-loop adaptive control policy can be developed [see, e.g., Vamvoudakis et al. (2009); Lewis and Vrabie (2009); Bertsekas (2011); Bhasin et al. (2012); Liu and Wei (2014)].

Real-world optimal control applications typically include constraints on states and/or inputs that are critical for safety [see, e.g., He et al. (2017)]. ADP was successfully extended to address input constrained control problems in Modares et al. (2013) and Vamvoudakis et al. (2016). The state-constrained ADP problem was studied in the context of obstacle avoidance in Walters et al. (2015) and Deptula et al. (2020), where an additional term that penalizes proximity to obstacles was added to the cost function. Since the added proximity penalty in Walters et al. (2015) was finite, the ADP feedback could not guarantee obstacle avoidance, and an auxiliary controller was needed. In Deptula et al. (2020), a barrier-like function was used to ensure unbounded growth of the proximity penalty near the obstacle boundary. While this approach results in avoidance guarantees, it relies on the relatively strong assumption that the value function is continuously differentiable over a compact set that contains the obstacles in spite of penalty-induced discontinuities in the cost function.

Control Barrier Function (CBF) is another approach to guarantee safety in safety-critical systems [see e.g., Ames et al. (2017)], with recent applications to the safe reinforcement learning problems [see e.g., Choi et al. (2020); Cohen and Belta (2020); Marvi and Kiumarsi (2021)]. Choi et al. (2020) have addressed the issue of model uncertainty in safety-critical control with an RL-based data-driven approach. A drawback of this approach is that it requires a nominal controller that keeps the system stable during the learning phase, which may not be always possible to design. In Marvi and Kiumarsi (2021), the authors develop a safe off-policy RL scheme which trades-off between safety and performance. In Cohen and Belta (2020), the authors develop a safe RL scheme in which the proximity penalty approach from Deptula et al. (2020) is cast into the framework of CBFs. While the control barrier function results in safety guarantees, the existence of a smooth value function, in spite of a nonsmooth cost function, needs to be assumed. Furthermore, to facilitate parametric approximation of the value function, the existence of a forward invariant compact set in the interior of the safe set needs to be established. Since the invariant set needs to be in the interior of the safe set, the penalty becomes superfluous, and safety can be achieved through conventional Lyapunov methods.

This paper is inspired by a safe reinforcement learning technique, recently developed in Yang et al. (2019), based on the idea of transforming a state and input constrained nonlinear optimal control problem into an unconstrained one with a type of saturation function, introduced in Graichen and Petit (2009), and Bechlioulis and Rovithakis (2009). In Yang et al. (2019), the state constrained optimal control problem is transformed using a barrier transformation (BT), into an equivalent, unconstrained optimal control problem. A learning technique is then used to synthesize the feedback control policy for the unconstrained optimal control problem. The controller for the original system is then derived from the unconstrained approximate optimal policy by inverting the barrier transformation. In Greene et al. (2020), the restrictive persistence of excitation requirement in Yang et al. (2019) is softened using model-based reinforcement learning (MBRL), where exact knowledge of the system dynamics is utilized in the barrier transformation.

One of the primary contributions of this paper is a detailed analysis of the connection between the transformed dynamics and the original dynamics, which is missing from results such as Yang et al. (2019), Greene et al. (2020), and Yang et al. (2020). While the stability of the transformed dynamics under the designed controllers is established in results such as Yang et al. (2019), Greene et al. (2020), and Yang et al. (2020), the implications of the behavior of the transformed system on the original system are not examined. In this paper, it is shown that the trajectories of the original system are related to the trajectories of the transformed system via the barrier transformation as long as the trajectories of the transformed system are complete.

While the transformation in Yang et al. (2019) and Greene et al. (2020) results in verifiable safe controllers, it requires exact knowledge of the system model, which is often difficult to obtain. Another primary contribution of this paper is the development of a novel filtered concurrent learning technique for online model learning, and its integration with the barrier transformation method, to yield a novel MBRL solution to the online state-constrained optimal control problem under parametric uncertainty. The developed MBRL method learns an approximate optimal control policy in the presence of parametric uncertainties for safety critical systems while maintaining stability and safety during the learning phase. The inclusion of filtered concurrent learning makes the controller robust to modeling errors and guarantees local stability under a finite (as opposed to persistent) excitation condition.

In the following, the problem is formulated in Section 2 and the BT is described and analyzed in Section 3. A novel parameter estimation technique is detailed in Section 4 and a model-based reinforcement learning technique for synthesizing feedback control policy in the transformed coordinates is developed in Section 5. In Section 6, a Lypaunov-based analysis is utilized to establish practical stability of the closed-loop system resulting from the developed MBRL technique in the transformed coordinates, which guarantees that the safety requirements are satisfied in the original coordinates. Simulation results in Section 7 demonstrate the performance of the developed method and analyze its sensitivity to various design parameters, followed by a comparison of the performance of the developed MBRL approach to an offline pseudospectral optimal control method. Strengths and limitations of the developed method are discussed in Section 8, along with possible extensions.

The following parameter estimator design is motivated by the subsequent Lyapunov analysis, and is inspired by the finite-time estimator in Adetola and Guay (2008) and the filtered concurrent learning (FCL) method in Roy et al. (2016). Estimates of the unknown parameters, θ ̂ ∈ R p , are generated using the filter Y ̇ = y ( s ) , Y f ≤ Y f ̄ , 0 , otherwise , Y ( 0 ) = 0 , (8) Y ̇ f = Y T Y , Y f ≤ Y f ̄ , 0 , otherwise , Y f ( 0 ) = 0 , (9) G ̇ f = G ( s ) u , Y f ≤ Y f ̄ , 0 , otherwise , , G f ( 0 ) = 0 , (10) X ̇ f = Y T ( s − s 0 − G f ) , Y f ≤ Y f ̄ , 0 , otherwise , X f ( 0 ) = 0 , (11) where s 0 = b x 1 0 ; … ; b x n 0 , and the update law θ ̂ ̇ = β 1 Y f T ( X f − Y f θ ̂ ) , θ ̂ ( 0 ) = θ 0 , (12) where β ₁ is a symmetric positive definite gain matrix and Y f ̄ is a tunable upper bound on the filtered regressor Y _f .

Equations 7–12 constitute a nonsmooth system of differential equations z ̇ = h ( z , u ) = h 1 ( z , u ) , Y f ≤ Y f ̄ , h 2 ( z , u ) , otherwise , (13) where z = [ s ; vec ( Y ) ; vec ( Y f ) ; G f ; X f ; θ ̂ ] , h 1 ( z , u ) = [ F ( s ) + G ( s ) u ; vec ( y ( s ) ) ; vec ( Y T Y ) ; G ( s ) u ; Y T ( s − s o − G f ) ; β 1 Y f T ( X f − Y f θ ̂ ) ] , and h 2 ( z , u ) = [ F ( s ) + G ( s ) u ; 0 ; 0 ; 0 ; 0 ; β 1 Y f T ( X f − Y f θ ̂ ) ] . Since ‖Y _f ‖ is non-decreasing in time, it can be shown that (13) admits Carathéodory solutions.

Lemma 2. If ‖Y _f ‖ is non-decreasing in time then (13) admits Carathéodory solutions.

Proof. see Lemma 2 in Supplementary Material.

Note that (9), expressed in the integral form Y f ( t ) = ∫ 0 t 3 Y T ( τ ) Y ( τ ) d τ , (14) where t 3 : = inf t { t ≥ 0 | ‖ Y f ( t ) ‖ ≤ Y f ̄ } , along with (11), expressed in the integral form X f ( t ) = ∫ 0 t 3 Y T ( τ ) s ( τ ) − s 0 − G f ( τ ) d τ , (15) and the fact that s(τ) − s ⁰ − G _f (τ) = Y(τ)θ, can be used to conclude that X _f (t) = Y _f (t)θ, for all t ≥ 0. As a result, a measure for the parameter estimation error can be obtained using known quantities as Y f θ ̃ = X f − Y f θ ̂ , where θ ̃ ≔ θ − θ ̂ . The dynamics of the parameter estimation error can then be expressed as θ ̃ ̇ = − β 1 Y f T Y f θ ̃ . (16)

The filter design is thus motivated by the fact that if the matrix Y f T Y f is positive definite, uniformly in t, then the Lyapunov function V 1 ( θ ̃ ) = 1 2 θ ̃ T β 1 − 1 θ ̃ can be used to establish convergence of the parameter estimation error to the origin. Initially, Y f T Y f is a matrix of zeros. To ensure that there exists some finite time T such that Y f T ( t ) Y f ( t ) is positive definite, uniformly in t for all t ≥ T, the following finite excitation condition is imposed.

Assumption 3. There exists a time instance T > 0 such that Y _f (T) is full rank.

Note that the minimum eigenvalue of Y _f is trivially non-decreasing for t ≥ t ₃ since Y _f (t) is constant ∀t ≥ t ₃. Indeed, for t ₄ ≤ t ₅ ≤ t ₃, Y f ( t 5 ) = Y f ( t 4 ) + ∫ t 4 t 5 Y T ( τ ) Y ( τ ) d τ . Since Y _f (t ₄) is positive semidefinite, and so is the integral ∫ t 4 t 5 Y T ( τ ) Y ( τ ) d τ , we conclude that λ _min (Y _f (t ₅)) ≥ λ _min (Y _f (t ₄)), As a result, t↦λ _min (Y _f (t)) is non-decreasing. Therefore, if Assumption 3 is satisfied at t = T, then Y _f (t) is also full rank for all t ≥ T. Similar to other MBRL methods that rely on system identification [see e.g., Chapter 4 of Kamalapurkar et al. (2018)], the following assumption is needed to ensure boundedness of the state trajectories over the interval [0, T].

Assumption 4. A fallback controller ψ : R n × R ≥ 0 → R q that keeps the trajectories of (7) inside a known bounded set over the interval [0, T), without requiring the knowledge of θ , is available.

If a fallback controller that satisfies Assumption 4 is not available, then, under the additional assumption that the trajectories of (7) are exciting over the interval [0, T), such a controller can be learned online, while maintaining system stability, using model-free reinforcement learning techniques such as Bhasin et al. (2013); Vrabie and Lewis (2010), and Modares et al. (2014).

Remark 5. While the analysis of the developed technique dictates that a different stabilizing controller should be used over the time interval [0, T), typically, similar to the examples in Sections 7.1 and 7.2, the transient response of the developed controller provides sufficient excitation so that T is small (in the examples provided in Sections 7.1 and 7.2 , T is the order of 10^–5 and 10^–6 , respectively), and a different stabilizing controller is not needed in practice.

Lemma 1 implies that if a feedback controller that practically stabilizes the transformed system in (7) is designed, then the same feedback controller, applied to the original system by inverting the BT, also achieves the control objective stated in Section 2.1. In the following, a controller that practically stabilizes (7) is designed as an estimate of the controller that minimizes the infinite horizon cost. J ( u ( ⋅ ) ) : = ∫ 0 ∞ r ( ϕ ( τ , s 0 , u ( ⋅ ) ) , u ( τ ) ) d τ , (17) over the set U of piecewise continuous functions t↦u(t), subject to (7), where ϕ(τ, s ⁰, u (⋅)) denotes the trajectory of (7), evaluated at time τ, starting from the state s ⁰ and under the controller u (⋅), r (s, u) := s ^T Qs + u ^T Ru, and Q ∈ R n × n and R ∈ R q × q are symmetric positive definite (PD) matrices ¹ .

Assuming that an optimal controller exists, let the optimal value function, denoted by V ∗ : R n × R q → R , be defined as V ∗ ( s ) ≔ min u ( ⋅ ) ∈ U t , ∞ ∫ t ∞ r ( ϕ ( τ , s , u t , τ ( ⋅ ) ) , u ( ⋅ ) ) d τ , (18) where u _I and U I are obtained by restricting the domains of u and functions in U I to the interval I ⊆ R , respectively. Assuming that the optimal value function is continuously differentiable, it can be shown to be the unique positive definite solution of the Hamilton-Jacobi-Bellman (HJB) equation [see, e.g., Theorem 1.5 of Kamalapurkar et al. (2018)]. min u ∈ R q ∇ s V ( s ) F ( s ) + G ( s ) u + s T Q s + u T R u = 0 , (19) where ∇ ⋅ : = ∂ ∂ ⋅ . Furthermore, the optimal controller is given by the feedback policy u(t) = u ^∗(ϕ(t, s, u _[0,t))) where u ∗ : R n → R q defined as u ∗ ( s ) ≔ − 1 2 R − 1 G ( s ) T ( ∇ s V ∗ ( s ) ) T . (20) Remark 6. In the developed method, the cost function is selected to be quadratic in the transformed coordinates. However, a physically meaningful cost function is more likely to be available in the original coordinates. If such a cost function is available, it can be transformed from the original coordinates to the barrier coordinates using the inverse barrier function, to yield a cost function that is not quadratic in the state. While the analysis in this paper addresses the quadratic case, it can be extended to address the non-quadratic case with minimal modifications as long as s↦r(s, u) is positive definite for all u ∈ R q .

In the following theorem, the stability of the trajectories of the transformed system, and the estimation errors W ̃ c , W ̃ a , and θ ̃ are shown.

Theorem 8. Provided Assumptions 3, 4, and 7 hold, the gains are selected large enough based on (46) - (49) in Supplementary Material, and the weights θ ̂ , W ̂ c , Γ, and W ̂ a are updated according to (12), (29), (30), and (31), respectively, then the estimation errors W ̃ c , W ̃ a , and θ ̃ and the trajectories of the transformed system in (7) under the controller in (32) are locally uniformly ultimately bounded.

Proof. See Theorem 8 in Supplementary Material.

Using Lemma 1, it can then be concluded that the feedback control law u ( t ) = ψ b ( a , A ) ( x ( t ) ) , t , 0 < t < T , u ̂ b ( a , A ) ( x ( t ) ) , W ̂ a ( t ) , t ≥ T , (34) applied to the original system in (1), achieves the control objective stated in Section 2.1.

To demonstrate the performance of the developed method, two simulation results, one for a two-state dynamical system, and one for a four-state dynamical system corresponding to a two-link planar robot manipulator, are provided. ²

This paper develops a novel online safe control synthesis technique which relies on a nonlinear coordinate transformation that transforms a constrained optimal control problem into an unconstrained optimal control problem. A model of the system in the transformed coordinates is simultaneously learned and utilized to simulate experience. Simulated experience is used to realize convergent RL under relaxed excitation requirements. Safety of the closed-loop system, expressed in terms of box constraint, regulation of the system states to a neighborhood of the origin, and convergence of the estimated policy to a neighborhood of the optimal policy in transformed coordinates is established using a Lyapunov-based stability analysis.

While the main result of the paper states that the state is uniformly ultimately bounded, the simulation results hint towards asymptotic convergence of the part of the state that corresponds to the system trajectories, x (⋅). Proving such a result is a part of future research.

Limitations and possible extensions of the ideas presented in this paper revolve around two key issues: (a) safety, and (b) online learning and optimization. The barrier function used in the BT to address safety can only ensure a fixed box constraint. A more generic and adaptive barrier function, constructed, perhaps, using sensor data is a subject for future research.

For optimal learning, parametric approximation techniques are used to approximate the value functions in this paper. Parametric approximation of the value function requires selection of appropriate basis functions which may be hard to find for the barrier-transformed dynamics. Developing techniques to systematically determine a set of basis functions for real-world systems is a subject for future research.

The barrier transformation method to ensure safety relies on knowledge of the dynamics of the system. While this paper addresses parametric uncertainties, the BE method could potentially result in a safety violation due to unmodeled dynamics. In particular, the safety guarantees developed in this paper rely on the relationship (Lemma 1) between trajectories of the original dynamics and the transformed system, which holds in the presence of parametric uncertainty, but fails if a part of the dynamics is not included in the original model. Further research is needed to establish safety guarantees that are robust to unmodeled dynamics (for a differential games approach to robust safety, see Yang et al. (2020)).