The forward kinematics, which contains the transformation between the joint angle ϕ(t)∈ℝ^m of a robot and the end-effector position and posture s(t)∈ℝ⁶, can be expressed as follows:

where f(·) is the non-linear mapping related to the structure of the robot. In view of strongly non-linear and redundant characteristics of f(·), it is difficult to obtain the desired angle information directly from the desired end-effector information s_d(t), i.e., s(t) = s_d(t). By taking the time derivative of both sides of Equation (1), one can deduce

where ϕ˙(t) denotes the joint velocity; s˙(t) covers the joint velocity and translational velocity of the end-effector; Jro=∂f(ϕ(t))/∂ϕ(t)∈ℝ6×m stands for the robot Jacobian matrix. Owing to the physical properties of manipulators, output control signals based on design formulas and intelligent calculations may not be suitable for the normal operation of real robots. Therefore, to ensure the protection of the robot, it is crucial to take into account the following joint restrictions:

where ϕ⁻, ϕ˙-, and ϕ¨- signify the lower bounds of joint angle, joint velocity, and joint acceleration, respectively; ϕ⁺, ϕ˙+ and ϕ¨+ denote the upper bounds of joint angle, joint velocity, and joint acceleration, respectively. Utilizing the special conversion techniques (Zhang and Zhang, 2012; Xie et al., 2022), the joint restrictions would be integrated into the acceleration level as ϕ¨∈γ, where γ = {g∈ℝ^m, γ⁻ ≤ g ≤ γ⁺} is the safe range of joints with γ⁻ and γ⁺ denoting the lower bound and upper bound of γ, respectively. In detail, the i-th elements of γ⁻ and γ⁺ are designed as

where i = 1, 2, 3, ⋯ , m; μ>0 and ν>0 are designed to select the feasible region for different levels; θ_i is the margin to ensure that the acceleration has a sufficiently large feasible region (Xie et al., 2022). Then, a brief introduction to the visual servoing system is presented as follows. Regarding visual servoing tasks, the number of features determines the complexity of a visual servoing system. Simply considering a visual servoing system with one feature, a miniature camera is mounted on the end-effector of the manipulator and moves with the end-effector. Figure 2 illustrates the geometric transformation in different coordinate systems. Three-dimensional space with O_ca as the original point and [X, Y, Z] as the coordinate axis is called the camera system with the internal coordinate point q = [x, y, z]^T. Relatively, with O_im as the center point, the image system is the two-dimensional space with the projection pixel point of q being [px,py]T and the pixel coordinate being p=[pu,pv]T∈ℝ2. According to the similar triangle, it can be readily obtained in the study by Zhang et al. (2017) and Zhang and Li (2018):

and

with l standing for the focal length of the camera; u₀ and v₀ denoting the pixel coordinate of principle point; and [ax,ay]T standing for the conversion scale. Based on Equations (3, 4), the image Jacobian matrix Jim(p,z)∈ℝ2×6 is defined using the following relationship (Liang et al., 2018):

where p˙ stands for the movement velocity of the pixel coordinate and

with

For the sake of convenience, Equations (2, 5) can be combined as follows:

with J=Jim(p,z)Jro∈ℝ2×m defined as the visual Jacobian matrix. The relationship between joint space and image space is established directly by Equation (6) at the velocity level. Taking the time derivatives of both sides of Equation (6) generates

where J˙ is the time derivative of 𝒥; ϕ¨ denotes the joint acceleration; and p¨ stands for the movement acceleration of the pixel coordinate. When it comes to a complicated situation with more features, the above analyses still hold under the requirements of appropriate dimensions. It is worth noting that a single feature is analyzed as an example for simple illustration. When the number of features increases, the principle of coordinate transformation remains unchanged along with the increase in dimension.

However, unknown conditions may exist in the robot visual servoing system, such as focal length changes or robot modifications. In this regard, it could not control the robot accurately to execute the IBVS task based on J˙. Hence, motivated by this issue, a data-driven learning algorithm is designed as follows. To begin with, a virtual IBVS system is established, incorporating the virtual visual Jacobian matrix J̄∈ℝ2×m and the following relationship:

where p˙̄∈ℝ2 is the virtual pixel velocity determined by the virtual robot and ϕ˙ is the joint velocity measured in real time from the robot. Beyond dispute, the goal of the data-driven learning algorithm is to guarantee that p˙̄ can rapidly converge to the real pixel velocity p˙. Thereout, an error function is devised as ℓ=||p˙̄-p˙||22/2, where ||·||₂ is the Euclidean norm of a vector. On the basis of the gradient descent method (Stanimirovic et al., 2015) to minimize the error function along the negative gradient direction, one can get

where J̄˙ is the time derivation of J̄; δ>0 denotes the coefficient that controls the convergence rate. Hereinafter, J̄˙ and J̄ are used to replace the calibrated parameter J˙ and 𝒥 to deal with the unknown situations. This method directly explores the relationship between joint space and image space without the utilization of J˙ and 𝒥. It is worth highlighting that Equation (8) does not involve real structural information and estimates structural information from the joint velocity ϕ˙ and velocity of the pixel coordinate p˙ measured by sensors, which belongs to the core idea of the data-driven learning algorithm.

It is worth pointing out that there are few acceleration-level robot control schemes for dealing with IBVS problems. None of the existing acceleration-level solutions take joint constraints into account (Keshmiri et al., 2014; Anwar et al., 2019). In this regard, considering joint constraints, acceleration control, and visual servoing kinematics, the AIVS scheme is constructed as a quadratic programming problem, taking the following form:

where p_d denotes the desired pixel coordinate. As a result, the goal of AIVS scheme (9) is to make the end-effector track the desired pixel point. In addition, according to robot Jacobian matrix J_ro and the image Jacobian matrix J_im, the visual Jacobian matrix 𝒥 and its time derivative J˙ are determined by the structure and parameters of the robot and the parameter settings inside the camera. Hence, if there are any changes in the internal parameters or structures, leading to an unknown state, the accuracy of 𝒥 and J˙ may be compromised, potentially leading to a decline in performance. In contrast to velocity-level visual servoing schemes (Hashimoto et al., 1991; Malis et al., 2010; Zhang et al., 2017; Liang et al., 2018; Van et al., 2018; Zhang and Li, 2018; Xu et al., 2019; Li et al., 2020), the proposed AIVS scheme (9) offers two advantages. First, it utilizes joint acceleration as the control signal, resulting in continuous joint velocities. This helps mitigate the issues associated with excessive and discontinuous joint velocities. Second, AIVS scheme (9) takes into account the equality and inequality constraints at the acceleration level. This allows for a more comprehensive consideration of constraints, expanding the range of applications.

For the AIVS scheme (9), the pseudoinverse method is applied to generate the relevant RNN solution (Cigliano et al., 2015; Li et al., 2020). Primarily, as reported in the study by Zhang and Zhang (2012) and Xie et al. (2022), one can readily extend pixel coordinate error p−p_d into the acceleration level by neural dynamics method (Liufu et al., 2024) as

where the design parameter α>0 and β>0; p˙d and p¨d are the desired velocity and the desired acceleration of the pixel coordinates, respectively. It is worth pointing out that the desired pixel coordinates p_d is a constant, thus p˙d=p¨d=0. As a result, Equation (10) can be rearranged as

Substituting Equation (11) into Equation (9b), it could be obtained:

In light of the pseudoinverse method, the joint acceleration can be minimized with the following formula:

where superscript ^† denotes the pseudoinverse operation of a matrix with 𝒥^† = 𝒥^T(𝒥𝒥^T)−1. It is deserved to note that Equation (12) is employed in the study by Keshmiri et al. (2014) and Anwar et al. (2019) to generate the acceleration command for a manipulator. However, the research in the study by Keshmiri et al. (2014) and Anwar et al. (2019) does not consider joint constraints of the manipulator. To address this problem, the RNN corresponding to the AIVS scheme (9) is derived as

where projection function 𝒫_γ(c) = argmin_b∈γ||b−c||₂. Furthermore, theoretical analyses regarding the convergence of RNN (13) are presented as follows.

Theorem 1: The pixel error ξ = p−p_d driven by AIVS scheme (9) assisted with RNN (13) globally converges to a zero vector.

Proof: According to Equations (7, 13), one has

Due to the fact that p_d is a fixed feature, error function ξ¨ can be readily derived as

By considering the projection function, a substitution matrix H is designed to replace 𝒫_γ(·), leading to

of which

and

By matrix decomposition, structural analyses of matrix JℋJ†=[a11,a12;a21,a22∈ℝ2×2 are given as follows:

where L=H. In this regard, matrix 𝒥H𝒥^† can be viewed as the product of two positive definite matrices. It is evident that the eigenvalues of 𝒥H𝒥^† are greater than zero and det(𝒥H𝒥^†) = det(𝒥LL^T𝒥^T)det((𝒥𝒥^T)−1)>0 with det(·), denoting the determinant of a matrix. According to the properties of the diagonal elements of the matrix, it can be concluded that the diagonal elements of 𝒥H𝒥^† are greater than zero (a₁₁>0, a₂₂>0). Furthermore, Equation (12) can be rewritten as

and further we get

and

which can be regarded as a perturbed second-order constant coefficient differential equation with respect to ξ. In conclusion, pixel error ξ is able to converge exponentially. To illustrate the steady state of the system (Equation 14), further derivations continue to be given. As the pixel error decreases, all joint properties return to the interior of joint constraints. In this sense, joint properties, i.e., ϕ¨, ϕ˙ and ϕ, are all inside the joint limits with h_i = 1. Therefore, Equation (14) can be reorganized as

It is worth mentioning that Equation (15) can be regarded as a second-order constant coefficient differential equation with regard to ξ. Moreover, the solutions of Equation (15) can be segmented into three subcases on account of different settings of α and β, given the original state ξ(0) = p−p_d.

Subcase I: As for α²−4β>0, the characteristic roots could be obtained simply as R1=(-α+α2-4β)/2 and R2=(-α-α2-4β)/2 with real number R₁≠R₂. Therefore, one can readily deduce

with D1=α/(2α2-4β)+1/2 and D2=1/2-α/(2α2-4β)

Subcase II: As to α²−4β = 0, calculating characteristic roots generates R₁ = R₂ = −α/2. Hence, it can be readily obtained:

Subcase III: As to α²−4β < 0, we get two complex number roots as R₁ = ζ+iη and R₂ = ζ−iη. Accordingly, it is evident that

The above three subcases indicate that the pixel error ξ = p−p_d converges to zero over time globally. The proof is complete.

Based on the virtual visual Jacobian matrix, a DAIVS scheme (8) would be designed as

It is a remarkable fact that the DAIVS scheme does not involve the visual structure of the real robot. Instead, the virtual visual Jacobian matrix J̄ conveys the transformation relationship between the joint space and image space to deal with possible unknowns in the structure of the robot system. Compared with acceleration-level visual servoing schemes (Keshmiri et al., 2014; Anwar et al., 2019), the proposed DAIVS scheme offers two distinct advantages. First, it prioritizes the safety aspect by considering joint limits. Second, the DAIVS scheme takes into account the uncertainty of the robot vision system and employs the virtual visual Jacobian matrix for robot control, enhancing the fault tolerance ability. The existing acceleration-level visual servoing schemes (Keshmiri et al., 2014; Anwar et al., 2019) cannot accurately implement visual servoing tasks when the Jacobian matrix lacks precision. Furthermore, combining Equations (8, 13) generates

It is worth pointing out that the RNN (16) is divided into the inner cycle and outer cycle, i.e., the learning cycle and control cycle. Subsystem (Equation 16a), which can be viewed as the outer cycle, mainly generates the control signal to adjust the joint properties via virtual visual Jacobian matrix J̄. In return, inner cycle (Equation 16b) with learning ability can explore the relationship between end-effector motion and joint motion, thus producing virtual visual Jacobian matrix J̄ to simulate the movement process of real robots. From a control point of view, the inner cycle (Equation 16b) must converge faster than the outer cycle (Equation 16a). In this sense, δ≫α is a necessary condition for the normal operation of the system.

Note that both RNN (13) and RNN (16) involve the use of pseudo-inverse operations. As a result, various existing methods can be employed to mitigate singularity issues, such as the damped least squares method. Specifically, 𝒥^† can be calculated via 𝒥^† = 𝒥^T(𝒥𝒥^T+hI)−1 with h being a tiny constant and I being an identity matrix. The additional item hI ensures that all eigenvalues of 𝒥𝒥^T+hI are never zero during the inversion process, thereby preventing singular issues. In addition, RNN (16) relies on the virtual visual Jacobian matrix and estimates the real Jacobian matrix using Equation (16b). This enables a robust handling of the visual system's uncertainty. However, RNN (13) relies on the real visual Jacobian matrix, leading to potential inaccuracies in the robot control process.

The learning and control performance of the proposed DAIVS scheme aided with RNN (16) are proved by the following theorem.

Theorem 2: The Jacobian matrix error E=J̄-J and pixel error ξ = p−p_d produced by RNN (16) converges to zero, given a large enough δ.

Proof: The proof is segmented into two parts: (1) proving learning convergence; (2) proving control convergence.

Part 1: Proving learning convergence. Design the i-th system of Jacobian matrix error as Ei=J̄i-Ji (i = 1, 2) where J̄i and 𝒥_i denote the i-th row of J̄ and 𝒥 and set the Lyapunov candidate Vi=(J̄i-Ji)(J̄i-Ji)T. Calculating the time derivative of 𝒱_i leads to

where p˙i represents the i-th element of p˙, and Π(ϕ˙ϕ˙T) denotes the least eigenvalue of matrix ϕ˙ϕ˙T. When the manipulator is tracking the feature, the value of Π(ϕ˙ϕ˙T) is always greater than zero. In this case, we substitute Ei=J̄i-Ji into the above equation, resulting in the following expression:

For further analysis, we consider three cases based on the above equation:

Combining the above three cases, it can be summarized that limt→+∞||Ei||2≤||J°i||2/δΠ(ϕ°ϕ°T). Furthermore, it can be deduced that the Jacobian matrix error E=J̄-J produced by RNN (16a) globally approach zero, given a sufficiently large value of δ.

Part 2: Proving control convergence.

According to the proof in Part 1, we take advantage of the LaSalle's invariant principle (Khalil, 2001) again to conduct the convergence proof on Equation (16b). In other words, the following formula is provided by replacing J̄ and J̄˙ with 𝒥 and J˙:

which is equivalent to Equation (13). In consequence, the proof on the convergence of the pixel error p−p_d in Equation (17) has been discussed in Theorem 1 and thus omitted here. The proof is complete.

In this subsection, in order to prove the feasibility of the AIVS scheme (9), four simulations with different initial position states of the Franka Emika Panda manipulator are conducted to trace one desired feature with results shown in Figure 3. Simply design α = 10 and β = 10. It would be readily discovered from Figure 3A that four test examples from four different directions are straightforward to successfully pursue the desired pixel. With test 4 as an example, detailed joint data and pixel errors are shown in Figure 3B through Figure 3F, which illustrate that the joint angle, joint velocity, and joint acceleration are all kept inside the joint limit and that the pixel error can converge to zero within 5 s. The above descriptions well verify the validity of the proposed AIVS scheme (9) in the case of the known visual servoing Jacobian matrix to solve the visual servoing problem at the acceleration level.

This subsection indicates the feasibility and capability of the pixel error convergence of the DAIVS scheme aided with the RNN (16) by providing simulation results, as shown in Figure 4. Furthermore, we choose δ = 2 × 10⁴, α = 10 and β = 40. Notably, the virtual visual Jacobian matrix is exploited with random initial values, instead of the real visual Jacobian matrix to facilitate system operation. The end-effector of the robotic arm is oriented toward the object, as shown in Figure 4A. In addition, the joint acceleration is shown in Figure 4B, which is confined to the joint limit and maintain the normal operation. As shown in Figure 4C, the Franka Emika Panda manipulator successfully traces the desired feature with pixel error converging to zero and maintaining the order of 10⁻² pixel. As for the learning ability, Figure 4D illustrates that the virtual robot manipulator can learn the movement of the real robot manipulator with the learning error approaching to zero in 0.05 s and maintaining the order of 10⁻⁴ pixel/s. In short, the simulation results in Figure 4 highlight the simultaneous learning and control ability of RNN (16).

This subsection offers simulation comparison results between the proposed schemes aided with the corresponding RNNs and the IBVS method presented in the study by Zhang and Li (2018). In this regard, the RNN provided in the study by Zhang and Li (2018) is shown as

where parameters κ₁>0 and κ₂>0 determine the rate of error convergence. It is worth pointing out that the IBVS method in the study by Zhang and Li (2018) assisted with RNN (18) is constructed from the viewpoint of the velocity level, and that, RNN (18) requires exact structural information 𝒥 to maintain the normal operation.

In the first place, simulations are conducted on the Franka Emika Panda manipulator for IBVS task with Figures 5A–D synthesized by RNN (13) and Figures 5E–H synthesized by RNN (18). Notably, the results in Figure 5 are carried out on the premise of known structural information 𝒥 with parameters κ₁ = κ₂ = 2, α = 10, and β = 10. As shown in Figures 5A, E, the manipulator's end-effector is controlled to point toward the object. In Figure 5B, the joint acceleration generated by RNN (13) is safely confined within the joint limits, while the joint acceleration generated by RNN (18) exists a sudden change of ~38 rad/s² in Figure 5F, which may cause damage to the robot. Furthermore, in contrast to Figure 5G, the joint velocity shown in Figure 5C is smaller and exhibits smoother changes, making it more suitable for real-world scenarios. Figures 5D, H demonstrate that both RNN (13) and RNN (18) are able to quickly propel pixel errors to zero. Therefore, it is concluded from the above results that AIVS scheme (9) aided by RNN (13) is able to guarantee a better safety performance when controlling the manipulator.

Beyond that, in the case of the unknown visual system, corresponding comparison simulations are driven by the DAIVS scheme aided with the RNN (16) and the IBVS method in the study by Zhang and Li (2018) assisted with RNN (18). The results are shown in Figure 6 with parameters κ₁ = κ₂ = 2, α = 10, β = 40, and δ = 2 × 10⁴. To simulate the unknown visual system, J̄ in Equation (16) and 𝒥 in Equation (18) are random matrices of constants with the absolute value of each element < 100. Figures 6A, B well embody that, when encountering unknown structural information, the DAIVS scheme assisted with RNN (16) controls the Franka Emika Panda manipulator to preferably complete IBVS task with the pixel error converging to zero. Nevertheless, the generated joint velocity in Figure 6C changes dramatically within the joint limit in a mess. Even worse, the pixel error driven by RNN (18) does not converge and maintain a diffused state, which indicates the failure of the IBVS task. In conclusion, the proposed DAIVS scheme is able to deal with the unknown structural information in the robot system and fulfill the visual servo control with simultaneous learning and control performance.

Furthermore, comparison results among different existing approaches (Hashimoto et al., 1991; Keshmiri et al., 2014; Zhang et al., 2017; Van et al., 2018; Zhang and Li, 2018; Anwar et al., 2019; Li et al., 2020; Zhu et al., 2022) for visual servoing of robot manipulators are presented in Table 1. It is worth emphasizing that, compared with the prior art, the proposed RNN (13) and RNN (16) are the first acceleration-level work, considering the multiple levels of joint constraints, and RNN (16) is the first study to dispose the unknown situations in the robot visual system with simultaneous learning and control ability. As a result, the above two points are the innovative contributions of this study.