The use of Unmanned Aerial Vehicles (UAVs) towards autonomous task completion has received increased attention in the past decade. Indicative tasks include aerial manipulation in Gassner et al. (2017), Gkountas and Tzes (2021), Nguyen and Alexis (2021), Li et al. (2021), surveillance in Bisio et al. (2021), Tsoukalas et al. (2021), as well as Simultaneous Localization and Mapping (SLAM) in Papachristos et al. (2019a), Tsoukalas et al. (2020), Dang et al. (2019) or inspection in Papachristos et al. (2016), Steich et al. (2016), Bircher et al. (2015); these works necessitate the collaborating agents to exchange their relative pose (position and orientation).

High frequency measurements from the inherent Inertial Measurement Unit (IMU) within each UAV are filtered using an Extended-Kalman-Filter (EKF) for attitude estimation (Abeywardena and Munasinghe, 2010) in Flight Control Units (FCUs). Migrating from the attitude to the altitude estimation necessitates the use of additional onboard sensors. Among these, the Global Positioning System GPS/GNSS (Qingbo et al., 2012) typically feeds at 5Hz positioning data to the FCU. In GPS-denied indoor environments, other sensing modalities are deployed for estimating the drone position within a swarm. These include LiDaR sensing (Kumar et al., 2017; Tsiourva and Papachristos, 2020; Yang et al., 2021), RSSI measurements (Yokoyama et al., 2014; Xu et al., 2017; Shin et al., 2020), RF-based sensing (Zhang et al., 2019; Shule et al., 2020; Cossette et al., 2021) and visual methods (Lu et al., 2018).

An indicative setup of two drones bearing imaging modalities is showcased in Figure 1 for evaluating the proposed technique in an experimental real-world leader-follower scenario.

In this work, we assume the existence of multiple UAV agents, each bearing an imaging modality without zooming capabilities, while their cameras are assumed to be mounted at the front of the vehicle. Having already introduced several methods for pose detection in Section 1, this research work focuses and extends the use of the ArUco fiducial markers framework (Romero-Ramirez et al., 2018). The identification process consists of a filtering process with local adaptive thresholding for edge extraction. This is followed by a contour extraction and a polygonal approximation to identify the rectangular markers. Then a size based elimination and marker code extraction is carried out, to be followed by marker identification and marker pose estimation. When applied to visual object pose tracking, assuming object distances of less or equal to 5m, this framework can provide a robust and accurate result of position and orientation (Xavier et al., 2017).

Each UAV is carries the fiducial marker arrangement similar to the one in Tsoukalas et al. (2018). This so called RhombicubOctahedron (RhOct) arrangement is comprised of squares and isosceles triangles as its facets and has an overall weight of 75g. The utilization of this Archimedean-solid, depicted in Figure 2 (left), allows for concurrent observation of these fiducial markers, as shown in Figure 2 (right).

The truncated RhOct comprises of 13 squares and 4 triangle facets and has an area of A r = 13 ( a × a ) + 4 1 2 a × a 5 4 , where a is the square facet’s edge length. The planar-rollover of the 17 facets with their associated markers are presented in Figure 2 (right). Marker 0 corresponds to the one that has axis Z _c = Z ₀ passing thru its center as shown in Figure 2, and the angles θ _i and ϕ _i for each marker correspond to the horizontal and vertical axes, respectively.

This configuration is expected to improve the pose estimation accuracy when more than one marker is detected. Additionally, it ensures robustness in occlusion and lighting conditions, as the system can identify the pose given only a subset of the markers. Such scenarios are depicted in Figure 3. In Figure 3C, the RhOct is partially occluded from the view, however its upper markers are still detected, allowing for the pose estimation process to conclude. Similarly, in Figure 3D two frames are shown with significant varying lighting conditions compared, and yet the RhOct’s pose can be extracted.

The algorithm for pose estimation is initialized by generating each fixed marker’s configuration on the RhOct. The marker identifying number and its corner geometry on the solid are obtained from a configuration file. The marker’s corner geometry, in this case refers, to the coordinates of its four corners with respect to the axis system X _c , Y _c , Z _c of Figure 2. Subsequently the online pose extraction is carried out for every new image acquisition using the following steps: 1) Detect fiducial (ArUco) markers in the image, 2) For each marker discovered extract the pixel coordinates of each corner, 3) Feed all pixel coordinates to a Perspective N–point solver of OpenCV library (Bradski, 2000) for extracting the RhOct’s pose with respect to the camera axis system. The algorithm encountered in the EPnP-formulation (Lepetit et al., 2009) was used to solve Perspective N-points (PnP) problem (Quan and Lan, 1999), due to its recursive nature which starts from the previously encountered solution. This algorithm is a significant enhancement to (Tsoukalas et al., 2018), where the center of the solid was extracted from each individual marker and then averaged, leading to ambiguity in individual marker pose extraction (Oberkampf et al., 1996). To address this, the algorithm simultaneously passes all marker corners (in pixels) to the solvePnP function, which in turn calculates the global optimal pose satisfying the initial fixed geometry of the RhOct.

Additional improvements include changes in the marker arrangement for more accuracy and improved detection frequency using a Region-of-Interest (RoI) approach. The adopted dictionary was “ARUCO_MIP_16h3” since this provides 4 × 4 markers which are easier to detect at larger distance than the default ArUco 6 × 6 ones. The markers at the square (triangular) facets were set to 48.75 (22.5)mm to accommodate space for the detection method. This method improved the accuracy, reduced jittering due to the sub-pixel corner refinement.

The adopted recursive RoI implementation assumes that a pose solution from the previous iteration exists, and the subsequent image is only cropped at a 1280 × 720 size around the center pixel of that solution and then passed to the pose detection algorithm. This implementation can handle 30 FpS in an i7-CPU implementation and the resulting algorithm is summarized in Figure 4.

Owing to inherent delays in the relative pose computation of the leader demands the inference of the maximum allowable lag, so that proper control parameters can be applied to the follower in order to efficiently maintain the leader within its Field-of-View (FoV).

The dynamic model of a quadrotor with mass m forms a set of nonlinear ODEs Sabatino, 2015. Let o p = x , y , z , ϕ , θ , ψ T ∈ R 6 the vector containing position and the Euler angle orientation of the quadrotor in earth frame. Let its derivative be o ̇ p = u , υ , ω , p , q , r T ∈ R 6 containing the linear and angular velocities with respect to body frame. Let the UAV’s inertia matrix with respect to its center of gravity be diagonal I = I x 0 0 0 I y 0 0 0 I z ∈ R 3 x 3 . The input forces and torques to the system are assumed proportional to the square rotational speed of the rotors as in Eq. 1, where l is the distance of any rotor from the drone center, b the thrust factor and d the drag factor. u = u 1 , … , u 4 T = f t , τ x , τ y , τ z T ∈ R 4 , w h e r e f t = b Ω 1 2 + Ω 2 2 + Ω 3 2 + Ω 4 2 τ x = b l Ω 3 2 − Ω 1 2 τ y = b l Ω 4 2 − Ω 2 2 τ z = d Ω 2 2 + Ω 4 2 − Ω 1 2 − Ω 3 2 . (1)

Let the gyroscopic moments and any ground effect phenomena be absent, then the dynamic model of the quadrotor can be described as in x ̇ = f x + ∑ i = 1 4 g i x u i , w h e r e (2)

x = x , y , z , ϕ , θ , ψ , x ̇ , y ̇ , z ̇ , p , q , r T ∈ R 12 , and c ( ⋅ ) = △ cos ( ⋅ ) , s ( ⋅ ) = △ sin ( ⋅ ) , and t ( ⋅ ) = △ tan ( ⋅ ) . f x = x ̇ y ̇ z ̇ q s ϕ c θ + r c ϕ c θ q c ϕ − r s ϕ p + q s ϕ t θ + r c ϕ t θ 0 0 g I y − I z I x q r I z − I x I y p r I x − I y I z p q , a n d g i x = 0 1 × 12 T , i = 1 , … , 4 (3) g 1,7 T = − 1 m s ϕ s ψ + c ϕ c ψ s θ , g 1,8 T = − 1 m c ψ s ϕ − c ϕ s ψ s θ (4) g 1,9 T = − 1 m c ϕ c θ , g 2,10 T = 1 I x , g 3,11 T = 1 I y , g 4,12 T = 1 I z . (5)

Linearizing around a hovering condition x ° = x ‐ ° , y ‐ ° , z ‐ ° , 0 1 × 9 T and u ° = m g , 0,0,0 T to account for the weight of the quadrotor, where x = x° + Δx and u = u° + Δu. Δ x ̇ = Δ x ̇ , Δ y ̇ = Δ y ̇ , Δ z ̇ = Δ z ̇ , Δ ϕ ̇ = Δ ϕ ̇ , Δ θ ̇ = Δ θ ̇ , Δ ψ ̇ = Δ ψ ̇ Δ x ̈ = − g θ , Δ y ̈ = g ϕ , Δ z ̈ = 1 m Δ u 1 , Δ ϕ ̈ = 1 I x Δ u 2 , Δ θ ̈ = 1 I y Δ u 3 , Δ ψ ̈ = 1 I z Δ u 4 .

These equations can be written in a compact form where the notation I _N (0 _N ) refers to an identity (zero) N × N matrix. Δ x ̇ 12 × 1 = A 12 × 12 Δ x 12 × 1 + B 12 × 4 Δ u 4 × 1 . (6) = 0 6 I 6 0 3 0 − g 0 g 0 0 0 0 0 0 3 0 3 0 6 Δ x + 0 8 × 4 1 m 0 0 0 0 1 I x 0 0 0 0 1 I y 0 0 0 0 1 I z Δ u , (7)

The controllers for the altitude and attitude, as shown in Figure 5 within the Ardupilot framework are adopted in this study. For the z-axis a P-differentiator converts vertical position error to vertical velocity; this is subsequently converted to desired vertical acceleration through a cascade P-controller and finally through a PID-controller to motor output. For the xy-plane control a gain differentiator coverts the x and y position error to reference velocity, followed by a velocity PID which converts the velocity error to roll (pitch) when x (y) is referred. Correspondingly this amounts to a = ϕ (θ) when x f r ( y f r ) is used. This reference roll (pitch) is sent to the attitude controller which is a second velocity P-differentiator along with a feedforward term which generates the desired roll (pitch) rate. A second cascade PID-controller is used to generate the necessary torques along the x (y) axes. The yaw-controller is similar to the attitude component of the roll and pitch components, as shown in Figure 5.

The four inputs f _t , τ _x , τ _y and τ _z are then used in Eq. 1 to compute the Ω _i , i = 1, … , 4 which are provided in the ESCs to power up the brushless motors of the quadrotor.

The resulting closed loop stable system takes the torm Δ x ̇ = A c Δ x + B c z f r x f r y f r ψ f r , (8) where the subscript “f” indicates the follower quadrotor.

In a leader/follower configuration the deployment strategy and, thus, reference paths dynamically change upon changes to the position of the leader. Having obtain a dynamic model for a single drone, namely the follower’s dynamic behaviour, the potential coupling between a dynamically moving leader and the follower’s reference path strategy allows for the generation of a single linear model for the multi-agent UAV configuration, that is used for assessing the system stability. It should be noticed that for a single leader/follower UAV configuration this coupling is not necessary.

In a leader/follower configuration we use the subsript “l” to distinguish the leader from the follower quadrotor. The leader quadrotor performs a motion while the follower attempts to follow the leader’s motion at a certain distance in each axis, or x _l −x _f = c _x , y _l −y _f = c _y and z _l −z _f = c _z where c _x , c _y and c _z are constants and c 2 = c x 2 + c y 2 + c z 2 . It should be noted that the presented controllers are continuous in nature, whereas in reality they are implemented with a 400Hz rate on a Pixhawk autopilot. This generates minor inherent delay in the system response, which in most cases can be neglected. However, in a real world scenario, other inherent time delays in pose estimation can affect the follower’s tracking performance of the leader. These delays can be attributed to sensing modality refresh rates and communication (if needed) of the relative poses within collaborating aerial agents.

In this work the use of an imaging modality for computing the relative pose between the leader and follower, means that the information does not need to be communicated between agents. Still, the refresh rate for the controller output cannot exceed the refresh rate of the imaging modality. In our case this is 30Hz, however for high resolution cameras of 12 mega-pixel it can be as low as 5Hz, which might affect tracking performance in the case of aggressive attitude motions from a leader. This scenario is of similar nature with communicating GPS sensed positioning between agents in outdoor environments for relative positioning. Since these sensors are bounded to a 4Hz frequency, the position of a leader will be communicated to a follower at 250msec for processing the calculations. Coupled to the data RF-transmission delay this interval can easily increase up to 350msec (excluding any lost packets). The follower will then need to compute x f r : x l − x f ≃ c x ( y f r : y l − y f ≃ c y ) [ z f r : z l − z f ≃ c z ] and direct it to its controller shown in Figure 5.

Hence there is an inherent varying delay in the feedback path since the vector x f r ( t ) , y f r ( t ) , z f r ( t ) T is computed after h seconds, where h ∈ [0, 350msec]. Subsequently, there exists a motivation to assess such delays when operating multi-agent systems. We should note that despite h(t) is time varying, we can use polling to constrain it at a sampling period (in the experimental section h ∈ 0.25 , 1 , 1.25 seconds. The controller employed by the follower is a simple MIMO P-controller, and we should note that the yaw is regulated to zero and there is no need to infer the leader’s relative yaw; this is due to the used RhOct that can be viewed from any yaw angle of the leader. The employed controller is z f r ( t ) x f r ( t ) y f r ( t ) ψ f r ( t ) = K z 0 0 0 0 K x 0 0 0 0 K y 0 0 0 0 0 z l ( t − h ) − z f ( t − h ) x l ( t − h ) − x f ( t − h ) y l ( t − h ) − y f ( t − h ) ψ f ( t − h ) + c z c x c y 0 . (9)

The follower quadrotor’s time-delayed dynamics is expressed as Δ x ̇ f = A f Δ x f ( t ) + A 1 , f Δ x f ( t − h ) + B f c z c x c y 0 . (10)

This system is asymptotically stable for any constant delay h: 0 < h < h ^max of there exist matrices p > 0, Q > 0 and Z > 0 such that (Gouaisbaut and Peaucelle, 2006; Xu and Lam, 2008) P A f + A f T P + Q − Z P A 1 , f + Z h max A f T Z A 1 , f T P + Z − Q − Z h max A 1 , f T Z h max Z A f h max Z A 1 , f − Z < 0 (11)

An exhaustive search is needed to compute the maximum delay that is allowed under the time-delayed leader/follower configuration. It should be noted: 1) that the provided results are sufficient and higher values of h ^max can still provide a stable system, and 2) the provided analysis is valid for near hovering conditions by the follower forcing a smooth trajectory [s(ϕ) ≃ ϕ and s(θ) ≃θ, or ϕ, θ ≤ 10°]. Furthermore the inherent assumption is that the leader remains within the FoV of the follower’s camera and the marker can be viewed with clarity.

In the following experimental study, the efficiency and accuracy of the suggested scheme is evaluated using two identical drones in a leader/follower formation.

In experimental leader/follower UAV-configurations, there is need to transmit the relative pose between these UAVs. Rather than relying on the individual FCUs and an RF-transmission scheme, fiducial markers are attached in each UAV. Using an onboard camera, these markers are accurately detected and the relative pose can be inferred within 30msec, as long as the leader is within the follower’s FoV and at a distance up to 4m. The maximum theoretical latency time is compared to the experimental one, while the FoV can be extended: 1) to cover the complete space if spherical cameras are used (Holter et al., 2021), and 2) focus in multiple agents.