To implement a remote operation setup, we use a 6 Degree of Freedom (DoF) UR5 Robotic Arm (Universal Robots) that can perform complex end-effector trajectories (see Figure 1).

The robot is placed in front of the workbench used to perform the experiments and it is directly controlled by a “server” workstation via TCP/IP. A remote “client” machine can then achieve teleoperation of the robot by connecting to the server through a graphical user interface (GUI) client program. The communication is implemented ad hoc for these experiments via sockets, and appropriate sensing delays from the server to the client can therefore be induced at will. The operator can access visual feedback corresponding to a pressure map detected by a distributed tactile sensor array placed at the tip of the end-effector. In the proposed study, we restrict the robot’s movements to vertical motions and translations. In every trial, the end-effector is placed at a random height, between 0.5 and 2.5 cm over either a soft or a hard dummy: the two dummies are both 8 cm³ cubes but the hard one is 3D printed with ABS, whereas the soft one is obtained by silicone casting Ecoflex 00-10 into a 3D printed mold. The starting height is randomized to avoid possible biases due to the adaptation of the subjects to the trials. Every subject is asked to remotely control the robot and to make contact with one of the dummies, also randomly selected in each trial, and to determine if the dummy is soft or hard. All subjects undergo a calibration trial in which they are made to control the robot to touch both dummies while knowing the correct answer. After the first trial, the subjects are then to perform the experiment 10 times for each condition. The different conditions include different introduced delays and different control modalities: in this work we tested delays of 0, 2 and 4 s jointly with four different control modalities (HC, PHC, PRC and PHRC), which will be described in Section 2.2.1, for a total of 120 trials for subject. The experiments have been performed block-wise, so all 10 trials for a given condition are completed before changing the condition itself. The conditions are tested in the following order: concerning the modalities, HC first, followed by PHC, PRC and PHRC, and for each modality the 0 s added delay has been tested first, followed by 2 and 4 s. A total of five subjects with no previous experience of remote task teleoperation or technology-assisted navigation have taken part in the experimental session, overall resulting in 600 trials. Among the participants, two subjects have a background in engineering and have previously interacted with robotic platforms, but not similar to the follower-leader architecture employed in this study. Given the length of the overall experiment (roughly 3 h) under the constant supervision of the authors, the study has been limited to five people.

The sensor used to collect haptic data is a hexagonal capacitive sensor array with seven tactile elements, or “taxels”, providing high sensitivity and spatial distribution over the surface of the sensor. The sensor provides measurement with a resolution of 16 bits corresponding to a variation of capacitance proportional to the pressure acting on top of the sensor. Details of the specific sensor and its fabrication have been previously reported (Scimeca et al., 2018; Maiolino et al., 2013). The collected data are normalized with respect to the maximum and minimum of each individual taxel and spatially calibrated to a visual representation of the pressure map showed to the user (see Figure 2).

The control architecture is composed of a leader, a follower, and two communication channels: one from the leader to the follower and one from the follower to the leader. The implemented architecture falls in the category of unilateral teleoperation systems, due to the absence of a kinesthetic feedback: the haptic data are visualized as an haptic map in the GUI rather than used directly as haptic feedback. The leader sends to the follower the motor commands according to the selected control modality, and receives the delayed sensor’s information.

The predictive modalities of operation investigated assume the possibility of achieving an accurate predictive system of the tactile sensor response. As mentioned in Section 1, many solutions have been proposed for accurate prediction, including deep learning architectures, however, these are beyond the scope of this work. In these experiments, we assume near-perfect prediction of the robot position, given the user control inputs, under delayed conditions. We thus use a third, non-delayed, channel from the follower to the leader, which allows the sensor prediction system to observe near-real-time robot positions. This, in turn, allows us to observe more rigorously only the effects of sensor delay on the human response, and to dissociate the human response outcomes from the quality of the robot position prediction under delay conditions.

In the following sub-sections, we will first analyse the leader and its control modalities and later describe the follower and how it reacts to the motor commands.

As introduced in the previous section, all subjects undergo blocks of 10 trials for each delay level (0, 2 and 4 s) and each control modality. In all the modalities, the sensor data is delayed by a fixed and constant Δt. We mainly want to study the effects of the magnitude of fixed delays, and as a result we do not consider time-varying delay conditions. The only difference between the 10 trials is the initial starting point of the end-effector, which is randomly generated in a range of 0.5–2.5 cm from the object: in order to avoid user’s adaptation to the task, the starting point is randomized, thus the users do not know the distance that they have to cover before entering in contact with the dummy’s surface. First, we analyse how the different modalities can control the trajectory of the end-effector. Figure 5 shows examples of how the three possible inputs can control the motion, all in a 0 s added delay condition, and how different control modalities result in different achieved trajectories. In the case of HC and PHC, the only control input are the pressed keys K ^t . As it can be seen in sub-figure (A), that creates a jerky trajectory because the robot stops every time the user is not pressing any key, given that l T t is ignored for these modalities. Since the starting height is unknown, the subject tends to stop every few seconds in every delay condition to avoid sudden impact. Therefore, the subject is able to stop soon after the contact with the object, and answer usually within 20 s. Conversely, PRC shows exactly the opposite trends: the trajectories are a lot smoother, but the end-effector reaches a much lower position before the internal model is able to process enough data in order to correctly update the model change the target position l T t . In this modality, K ^t is ignored and not taken into consideration. Finally, in PHRC, both inputs are accounted, as explained in Eq. 8. The magnitude of ri depends on how much data are gathered about l T t , therefore it increases as the end-effector spends time in l T t . At the same time, the data gathered by the sensor can shift the l T t , resetting ri to a new value. It can be noticed that the trajectory is initially very similar to the sub-figure (A) because ri is close to 0. Then, as ri increases, the trajectory rapidly converges to the target position, with little oscillations. When the l T t changes, around 20 s, ri drops, but it rapidly recovers as soon as enough data are collected from the new target.

Next, we are interested in how the introduced delay affects the different modalities, so we introduce a delay of 2 and 4 s to the communication channel from follower to leader. Figure 6 shows some trials taken from the experimental data. In all the graphs, it is shown the trajectory of the end-effector of the robot and a proxy of the delayed feedback showed to the user, obtained by performing the sum of all taxels’ values present in Tx ^t−Δt . The sum of the taxels has been selected as a metric because it summarizes effectively all the haptic information seen by the user. At 0 s delay all four modalities have similar behaviors: the end-effector stops when the contact is detected and then it undergoes small adjustments to reach the correct height at which is possible to recognize the touched object without the sensor’s saturation nor detachment.

In the case of HC, adding delay causes the formation of “chattering”: the users are able to see that they have touched too late, therefore they do not stop the end-effector in time and they saturate the sensor’s readings. Then, trying to move back to the correct height, they are once again unable to stop at the correct height and the end-effector moves away from the object, causing the sensor’s signal to drop back to 0. Chattering is an example of indirect instability, since it is originated by the user overcompensating the perceived errors due to the delay added to the visual representation of the haptic map. Compared with the 2 s delay case, the 4 s delay has longer and deeper oscillations, and the elapsed time before the answer is very high in both cases when compared with the 0 s case. Concerning the PHC, we can still observe some oscillations, but their magnitude is strongly reduced: it is evident that the users are able to use the information about the predicted output to minimizing overshooting, thus reducing chattering and giving an answer faster.

Next, the PRC shows almost no chattering because it does not rely on the user’s input, but the delay directly affects the update timing of l T t : both in the 2 and 4 s case the target position is only updated after 2 and 4 s after the previous one is reached, respectively. This could cause potentially dangerous contacts if the end-effector and the object exchange high forces for a prolonged time: this is evident in the 4 s delay case, in which we can see the end-effector almost “vibrating” at its lowest point, indicating that it was trying to go further but it was stopped by the environment. We also know that such a contact saturated the sensor’s signal because l T t is changed as a consequence of it. Moreover, it can also be noticed that this modality does not allow any oscillation, once the final l T t is determined: in all the remotely controlled modalities with no delay the users use a small oscillation as a strategy to successfully complete the task. However, this modality showcases much faster trials’ time, likely due to the absence of chattering. Finally, in PHRC we can still observe a little oscillation, but it is mixed with the target following behavior proper of PRC. This modality is shown to be slightly slower than PRC, but it allows to some extent small movements and deviations from the target position, thus ensuring to some extent the user’s freedom of motion.

To further validate our hypothesis, we have analyzed the percentage of cases in which we can detect chattering, defining chattering as the presence of at least one detachment from the dummy after initial contact is made. Figure 7 shows the cumulative percentage computed among all trials and all subjects. It is clear that chattering is common in remotely control modalities, especially HC, but can be strongly reduced by introducing autonomy. Note that the modalities are listed from the least autonomous, HC, on the left, to the most autonomous, PRC, on the right. Therefore, autonomy in the system can be considered a valid answer to strongly limit such behavior. To the right, it can be noticed that introducing any delay strongly increases the chattering, but going from 2 to 4 s does not produce additional chattering: as already discussed, this further increase in delay only produces greater and wider oscillations, not affecting the presence of chattering, but its severity.

The main results shown in Figure 8 are the accuracy, to the left, and the decision time, to the right. The decision time is calculated from the first contact made with the dummy to the moment in which the operator gives the answer, in order to avoid any bias related to the random initial height. The accuracy plot illustrates that there is a clear cut between HC and all the other modalities as the introduced delay increases: at 0 s delay, HC’s accuracy is 90%, but it rapidly decreases when the delay is introduced. The PHC also decreases in performance as the introduced delay increases, but less severely. Since both the fully remotely operated modalities have noticeably decreasing trends as the delay increases, we can infer that the delay compromises the stability of the feedback loop relying on the human action. On the other hand, the other two modalities seem not to be affected by delay as much, especially in the case of PRC, in which the accuracy stays between 80 and 85%. The PHRC seems to increase its performance as the delay increases: on one side, it could be inferred that PHRC is better exploited with delay, on the other side this may be due either to the small sample size or to the fact that every subject undergoes 0, 2 and 4 s delay trials in this specific order, thus they may find the control modality not intuitive at first, but learn and become better over time. We believe that PHRC is able to achieve better results than PRC because the user is able to slightly tune the motion of the end effector, thus exploring the contact zone and gather more useful information. Similarly, the decision time of all four modalities is within a 3 s difference when looking at the no-delay condition, but there is a clear difference among them as the delay increases. In high delay conditions, the four modalities are largely separated: HC is the slowest, followed by PHC and PHRC, and the fastest is PRC. We believe that this is due to the fact that both PHRC and PRC are effective in keeping the end-effector at the right range so as to avoid sensor saturation and contact detachment, thus not wasting time in regions with no information. Overall, HC and PHC are more sensitive and they lose stability when the delay is introduced, whereas PRC and PHRC show a much more robust behavior, showing that complete or partial automation is able to contrast the unstable behavior of delayed human-in-the-loop teleoperation.

Next, Figure 9 shows some more metrics useful to assess the performance: the deepest point reached during contact and the total average trial time. The deepest reached point is a proxy for how strong the interaction with the environment has been during the trial. Ideally, it is not needed to reach deep in the dummy in order to achieve good discrimination, and going too deep can produce high forces that could damage the robot or the environment. The plot illustrates that only the shared control mode seems to be affected by the delay added to the system. The autonomous mode always reaches deeper than the other modalities and the two fully remote-controlled modes show the best results, having a light interaction in all cases. Even when the delay is introduced, the subjects are able to avoid deep contacts by moving slower, thus they do not reach deeper than necessary in the dummies, but, as previously shown, they compromise the task execution’s speed. Shared control approaches the autonomous values as the delay increases, likely because the control architecture tends to trust the robot more than the remotely given motor commands. Concerning the approaching speed, the fastest modality is PRC and it is completely not affected by the delay: once the target position is set, the end-effector will approach the phantom at a constant speed v _a , regardless of the sensor’s data. The other three modalities are quite close to each other for the 0 s case, but as the delay increases it is shown that PHRC tends to go toward the PRC, PHC decreases slightly and then stabilizes, and HC tends to decrease its performance. Once again, PHRC approaches PRC as the delay increases, likely due to the increased trust toward the internal model. While working in high delay conditions, if the user is left in charge of the motion, the motion will result jerky and oscillating, thus collecting data points for more height levels than a smooth trajectory: this additional data can make the internal model much more precise, thus increasing ri ^t and consequently approaching the trajectories of the PRC.

More in detail, Figure 10 illustrates the precision of the internal model in all the different conditions. The distance between prediction and real data is calculated as follows: D = ∫ 0 t e n d ∑ j = 1 J ( x ̂ j t − x j t ) 2 d t t e n d ∀ x j t ∈ T x t ∧ x ̂ j t ∈ T x ̂ t (9) where t _end is the overall duration of the trial. The data show that the internal model is able to give a better prediction in PHRC than in the other modalities, especially as the introduced delay increases. The increasing trend of ri with respect to delay further supports that the introduction of delay makes the model more reliable and thus makes the robot behaving more autonomously (see Eq. 8). Moreover, both too little and too much autonomy seem to result in lower performance upon the introduction of delay. For PHC, the prediction’s accuracy decreases as the delay increases because the model is trying to predict the sensor’s output for wider and more chattering trajectories, whereas, in PRC, it decreases because the sensor gathers data from a lower number of different levels. Compared with PHC, PHRC allows voluntary small oscillations that can gather data from adjacent levels, increasing the overall quality of the model, and thus the average value of ri.

To further study how the interaction differs among soft and hard dummy, the average height while interacting with the environment and the interaction forces are reported in Figure 11. Due to the different stiffness of the two dummies, the end-effector reaches deeper when interacting with the soft dummy in all cases. When comparing trials belonging to the same dummy, it is evident that during PRC the subjects tend to interact more with the surface of the dummy, reaching deeper. Concerning the soft dummy, it is also clear that the magnitude of the delay affects the performance, likely due to the slower updating speed of the internal model. When analyzing the interaction forces, we focused on the maximum registered taxels’ value and the average momentum. For each trial, the average momentum is computed using the average taxels’ value to calculate the impulse of the trial, later averaged using the total time of each experiment, as follows: M = ∫ 0 t e n d ∑ j = 1 J x j t d t t e n d ∀ x j t ∈ T x t (10)

Both metrics show that PHRC is able to achieve on average softer interactions, exchanging less force with the environment and thus achieving a safer performance. Even if the interaction force is not considered as a main result, it is preferable to maintain it as low as possible, as long as it does not negatively affect the main performance metrics.

The high variance shown is mostly due to inter-subject, as opposed to intra-subject, variability. This is explained by both the starting baseline and the general ability to recognize the two different objects, which largely varies from person to person.