Learning from demonstration enables programming of a variety of robotic tasks in a way that is intuitive to a non-expert end-user. Previously, this method was utilized in an industrial context to teach robot based assembly tasks e.g., peg-in-hole insertion task, as well as polishing and grinding tasks. However, most existing approaches are applied on artificial or simplified benchmarks, and often do not consider the tight tolerances and process requirements with regards to the applied force that occur in a real manufacturing scenario. They often require multiple task demonstrations in order to learn task constraints or achieve robust task executions. Lastly, they often do not encompass the full pipeline of demonstration—encoding—execution from an end-user point of view.

In this paper we present a method for fast programming of a force-controlled industrial gluing task from a single user demonstration. We propose the use of adaptive admittance control during the demonstration phase to enable the user to kinesthetically move the robot more intuitively by providing a better control response, allowing them to provide more accurate task demonstrations.

From a user demonstration of the task, we obtain information on the kinematic trajectory of the robot. This allows us to then encode the motion in a motion primitive that allows both direct replay and generalization of the task. By detecting the contact state of the robot and estimating the surface normal, we can segment the task into phases of unconstrained (free-air) and constrained (in-contact) movement. Simultaneously, we estimate the constrained axes of the robot along the trajectory. This allows us to formulate a parallel position/force controller for the subsequent execution of the task, such that we can either imitate the forces applied during the demonstration or allow the user to specify a constant contact force, as is typical in industrial applications.

We evaluate our system on a benchmark that is closely based on a real industrial gluing task related to printed circuit board assembly, as shown in Figure 1, showing that our method can ensure constant and safe interaction forces and suitable trajectories in a task with low contact-force requirements.

With the evolution of collaborative robots, new user-friendly efficient and flexible ways for automating industrial tasks which were previously done with specialized automation equipment are possible (Gašpar et al., 2020). Furthermore, methods such as Learning by demonstration (LbD) (Billard et al., 2008) and force control can be exploited, thanks to the integrated sensors and controllers of the collaborative robot systems.

In robotic learning by demonstration scenarios, robot trajectories that describe the kinematic information are recorded (Deniša and Ude, 2015). On the other hand, for tasks associated with industrial robotics e.g., assembly (Abu-Dakka et al., 2015) or polishing (Gams et al., 2013), dynamic data is equally important. These two data sets lay the basis for in-contact execution of robot tasks, where the learned trajectory cannot be replayed but has to be adaptable to changes in the environment. To facilitate this requirement, in a lot of works, robot trajectories are represented with Dynamic Movement Primitives (Ijspeert et al., 2013; Ude et al., 2014). With this framework, both kinematic and dynamic trajectories can be represented in a unified manner, and can be enhanced with additional properties such as modulation and time scaling.

The benefits of the DMP framework can be efficiently exploited for in-contact tasks, when coupling the DMP in a force-based LbD scenario, as shown in Koropouli et al. (2012) and Kormushev et al. (2011). In the work of Rozo et al. (2013) a two stage force LbD approach was presented, where in the first stage they recorded the positions and orientations of the desired movement and in the subsequent stage the corresponding forces and torques.

Simultaneous control of force and position for teaching and execution tasks is physically impossible (Stramigioli, 2001), and therefore these approaches all have to make compromises between prioritising force control or position control. In literature, classical position/force controllers are still adopted for dealing with this challenge. One of the early works in this field was presented by Asada and Izumi (1989). The measured data was used to automatically program the position and force set points for the hybrid force control framework (Raibert and Craig, 1981) used in Cartesian space. Furthermore, simultaneous position and force control frameworks in the LbD literature primarily focus on selecting the dimension of the constraint frame on which the force or position control is applied (Peternel et al., 2017) and the selection of the best control frame (Ureche et al., 2015).

The main motivation for constraint selection methods is that the dimension of the constraint frame that is exposed to a high variance in force and low variance in position should favor force control, and vice versa (Ureche et al., 2015). Constraint frames are usually chosen manually based on the task specifications Raibert and Craig (1981). Common choices include the tool frame (Kronander and Billard, 2014; Peternel et al., 2017) and the surface normals (Deng et al., 2016; Conkey and Hermans, 2019). The latter can also be estimated, e.g. from the velocity of the contact point (Karayiannidis et al., 2014; Sloth and Iturrate, 2021). Conkey and Hermans (2019) present a framework for learning a dynamic constraint frame aligned to the direction of desired force using CDMPs and force control to ensure that the force is aligned in the normal plane with the direction of the movement. In (Kober et al., 2015), multiple demonstrations of a task are used to segment a series of separate movement primitives, each associated with a specific discrete frame of reference. The transitions between each of the primitives—and therefore compliant frames—are then separately addressed. Kumar and Rani (2021) present a new hybrid force/position control approach for time-varying constrained re-configurable manipulators for physical human robot interaction with environment of unknown stiffness. A review of hybrid position/force controllers is presented in Ortenzi et al. (2017).

An alternative to hybrid force/position control is impedance control with varying parameters. By changing the stiffness parameters of the impedance controller, the interaction forces can be adjusted between the robot and the environment (Buchli et al., 2011). Stiffness adaptation can be learned from demonstration in multiple ways, and typically requires multiple demonstrations or iterative corrections of the path. In Lee et al. (2015), the mean and covariance information of the trajectories from multiple demonstrations of a task are used as an approximation of the desired stiffness along different regions of the path. Kronander and Billard (2012, 2014) allow the user to iteratively re-execute the task and apply external disturbances on the robot such that it deviates from the demonstrated path. The magnitude of these deviations is then used analogously to stiffness. Pastor et al. (2011) introduced a method for real-time adaptation of demonstrated trajectories for task exertions, depending on the measured sensory data. They developed an adaptive regulator for learning and adaptation of demonstrated motion, where actual and learned force feedback was utilized. Schindlbeck and Haddadin (2015) presented a unified impedance approach for safe online parameter adaptation, based on passivity and energy tanks. Furthermore, a review of online adaptation of impedance control parameters for human robot interaction is presented in (Ficuciello et al., 2015; Müller et al., 2018) and in-contact execution tasks are presented by Abu-Dakka and Saveriano (2020). To adapt the controller parameters one can estimate the compliance of the interaction between robot and environment. This approach is taken in Santos and Cortesão (2018), where the perceived stiffness is estimated and used for controller gain scheduling.

For tasks where impedance control cannot be utilized because of the physical limitation of the used systems, admittance control (Siciliano et al., 2009) can be used for teaching and execution of tasks in contact with the environment. A variable admittance controller, for safe human-robot interaction, where the admittance control parameters were adapted based on the passivity criterion was presented by Ferraguti et al. (2019). Furthermore, variable admittance control can be adopted in human-robot cooperation tasks by means of learning (Dimeas and Aspragathos, 2014, 2015). Finally, Li et al. (2020) consider the robot to be in contact with both human and environment, similar to the condition considered in this paper.

It is important to ensure the stability and guarantee satisfactory performance despite varying controller gains and environmental compliance. Elegant stability conditions are set up in Müller et al. (2019) for a physical human-robot system with delays in the human reaction. The system considered in this paper has varying parameters in addition to input delays; thus, the methods presented in Briat (2014) may be used for the stability analysis and performance specification.

Recently, automated gluing applications have become very popular with the development of new industrial machines and tools, which can be efficiently utilized by collaborative robots. The majority of the work focuses on defining the robot trajectories based on the CAD description of the industrial object (Castelli et al., 2021). Other approaches, utilize simple Cartesian space industrial manipulators coupled with computer vision (Pagano et al., 2020). The vision system enables localization and reconstruction of the shape of objects to which glue should be applied. The gathered information is used to construct a motion trajectory that can be executed with the specialized manipulators, achieving the desired industrial performance.

Given a gluing or similar in-contact processing or dispensing task, our objective is to develop a full pipeline comprising the demonstration, encoding and execution that enables a robust and high-quality execution of the task with as little setup as possible required from the end-user. In order to make the approach more universally deployable in an industrial setting, we propose to learn from a single user demonstration, while exploiting the semantics of the task to adapt the robot controllers.

A key characteristic of the addressed task is that it is composed of free-air approach and retraction motions, but the actual dispensing or gluing consists in an in-contact motion whereby the manipulator is partially constrained by the environment. In particular, this constraint will be given by a surface normal during contact. In addition to this, in the proposed learning from demonstration scenario using kinesthetic teaching, the human is also in contact with the robot, thus partially constraining its motion as well.

An illustration of how these concepts fit into our system pipeline is shown in Figure 2. From an initial approach motion consisting only of the robot in contact with the human, the human will then move the robot into contact with the target surface and perform a sliding motion for the actual dispensing. This will correspond to a state where the entire system dynamics are comprised of the robot, the human and the environment. Finally, a retraction motion will move the robot away from the surface and into a final state where the dynamics will be given by the robot and human. If we consider the subsequent autonomous execution of the task, the dynamics will be the same, with the exception of the human’s contribution.

We consider our main contribution to be our controller design for kinesthetic teaching of processing tasks. We propose an adaptive controller architecture that takes into account the different constraints of the task at different phases of the execution. In particular, our approach to demonstration comprises the following:

1) We learn continuous kinematic and dynamic task constraints from a single demonstration.

In addition to our controller for demonstration, we also propose a second controller for task execution. This controller adopts a similar structure to the one used for teaching, albeit with a few modifications, and exhibits the following characteristics:

1) We use parallel position/force control to guarantee that both the kinematic and dynamic targets of the task are met.

The purpose of this section is to describe the proposed approach to obtain high-quality robotic gluing based on learned behaviors from kinesthetic teaching. It is challenging to learn the gluing behavior, as it is a task that requires contact between robot and environment; thus, both the motion and environment geometry and dynamic parameters should be learned from a single demonstration. As it is chosen to use only one demonstration for learning the task, the parameters estimated from the demonstration should be improved from executions of the task. We describe the entire process of 1) Programming by demonstration, 2) Encoding of robot behavior, and 3) Execution of desired gluing control; however, the main focus of the paper is the demonstration phase, as this is the novel contribution of the paper.

We propose a unified controller structure for the kinesthetic teaching and execution of tasks with interaction between robot and environment. Figure 3 is a block diagram of the controller that runs during demonstration. The controller is an adaptive admittance controller, where the adaptation is conducted based on the surface normal to the environment when the robot and environment is in contact. This ensures stability and improves the quality of the demonstrated behaviors. A similar control structure is used for the execution, as shown in Figure 5; however, a parallel position/force controller is used instead of the admittance controller.

In order to evaluate the performance of our proposed system, we constructed a benchmark from a real-world use-case provided by a partner company. The original use-case involves gluing electronic components to a PCB. For our evaluation, we 3D-printed a replica of the PCB with the attached components. Similarly, we substituted the original gluing dispenser tip with a 3D-printed model. Both the PCB and gluing tip mock-ups can be observed in Figure 6. Note that, due to being 3D-printed, our mock-ups actually introduce more challenges from a control perspective than the original components. Printing imperfections on the surface of the PCB mock-up increase friction and introduce high frequency components in the forces that the parallel position/force controller must then address. The 3D-printed tip is also stiffer, which can challenge the stability of the system when compared to the more compliant true gluing tip.

We used a Universal Robots UR10e collaborative robot manipulator for our experiments. The robot is equipped with an internal force-torque (FT) sensor at the wrist 3 joint, close to the tool flange. All force-torque measurements and control were performed using the output of this sensor. The controllers were implemented in Python 3, using the ur_rtde ¹ library for communication with the robot, both for streaming data and for sending commands to the internal robot controller. Our Python control application and the internal robot controller both run at a frequency of 500 Hz.

The following sections will evaluate our full system pipeline, from a demonstration of a task, as shown in Figure 1, to its segmentation, encoding and subsequent execution. We will present results for each of the system components previously described in Section 4. Note that all figures in the following sections are expressed in robot base frame, that is, with the z-axis normal to the table surface and its positive direction pointing out of the table, while the x − y plane coincides with the table plane.

In the previous section, we evaluated our system on a benchmark inspired by a real industrial gluing use case, and analyzed the performance of the different system components both during a task demonstration and during the subsequent execution of the learned primitive. During demonstration, we showed that the adaptive damping in the admittance controller can assist the user in accomplishing their intended motion, while at the same time guaranteeing a stable interaction despite a changing environmental stiffness. Similarly, we showed that our surface normal estimator works satisfactory and allows the admittance controller to change the structure of its gain matrices, such that the normal direction maintains a higher damping to ensure stability, while the damping in the other directions is reduced to assist the user. During execution, we showed that our controller is able to track positional targets while applying the target force profile in the normal direction. We also showed the system’s ability to reduce impact forces by using DMP goal modulation.

However, there are many factors that were not fully considered and that could be improved upon in future iterations of the system.

When it comes to our adaptive admittance controller for demonstration, our small-scale usability evaluation in Section 5.1.2 shows a clear trend that our controller enables users to provide both more accurate and more efficient demonstrations of a path-constrained task when compared to Freedrive, as implemented on Universal Robots manipulators. This was verbally confirmed by the participants following the experiment. The subjects also reported that demonstrating was considerably more physically taxing using Freedrive, whereas our adaptive admittance scheme assisted in completing the task, making it easier to keep contact with the surface and to maintain a smooth continuous motion. As a full usability evaluation falls beyond the scope of this paper, backing this up with, e.g. the NASA-TLX (Hart and Staveland, 1988) or SUS (Brooke et al., 1996) tests is left for future work.

In gluing and dispensing tasks, it is often imperative for an optimal process outcome that the feed-rate be kept constant throughout the surface of application. While approaches that are able to adjust the feed-rate of the gluing gun to compensate for changes in robot speed exist ² , by far the majority of applications use constant feed-rate gluing guns and therefore require constant robot movement speed. With our approach, a skilled demonstrator can with some practice provide a near-constant speed demonstration, as is close to the case with the y-axis of Figure 15. However, it would be preferable to normalize execution speed in the encoding of the task to allow for imperfect demonstrations. Some approaches to DMP speed-scaling have been presented (Nemec et al., 2018a,b), typically through modulation of the time-constant, τ, with an additional term ν(x) that can be encoded as a mixture of RBFs. This method was considered for our system, but we were unable to achieve better performance than was demonstrated with it, as it did not result in constant speed. Alternative modulation schemes for DMP-speed will be investigated in future work.

The performance of the surface normal estimator is imperative for obtaining a good performance of the admittance controller used during the demonstration. It is of importance to both obtain fast convergence and little noise on the estimate. As seen in Figure 9, the convergence of the estimator might be improved; however, the parameters were chosen as a compromise between convergence rate and noise on the estimate. The surface normal estimation could potentially be improved by using a sliding model observer as presented in Hasan and Husain (2009). An analysis of the stability of the interconnection between the adaptive controllers and the surface normal estimator was omitted, as the faster dynamics of the controller when compared to those of the gain adaptation according to the surface normal estimator preclude instability. Note that the performance of the surface normal estimator still enables tracking of curved surfaces, as shown in Figure 8, despite its dynamics being slower than those of the controller.

In applications relying on force control it is important to ensure that force limits are not surpassed. Especially when relying on programming by demonstration there is a high chance that the nominal behavior will not comply with force constraints. Thus, it is relevant to modify the position/force controller such that it guarantees compliance with force constraints. This can e.g. be ensured by set invariance control as shown in Polverini et al. (2017). High forces will likely occur in the transition between free and constrained motions if small positional uncertainties are present. Therefore, it could be beneficial to adapt the reference path according to the measured force as proposed in De Wit and Brogliato (1997) with an adaptive approach. This would be an alternative approach to the goal modulation proposed in Section 4.4.1.

As our system relies heavily on the performance of several parameter estimators, particularly the surface normal estimator, a possible improvement would be to allow for repetitive update of our parameter estimates, for instance through the use of Iterative Learning Control (ILC) (Wang et al., 2009). Such methods have already shown good performance in learning from demonstration systems for peg-in-hole tasks (Nemec et al., 2013; Abu-Dakka et al., 2015).

In this paper, we have presented a full system for learning from demonstration of gluing and dispensing tasks that encompasses the full pipeline of demonstration, encoding and execution.

During the demonstration, our system is designed to assist the user and provide a non-obtrusive kinesthetic teaching experience. By adapting the damping of an admittance controller according to a force-velocity law, we aid the user during acceleration or deceleration. Furthermore, when contact against the target surface is detected, a surface normal estimation algorithm is used to adapt the gain matrices of the controller to guarantee higher damping and preserve stability in the normal direction, while allowing lower damping in the other directions. Unlike previous approaches in the literature, we do not assume diagonal gain matrices. Rather, we allow the whole eigenstructure to change. This results in lower user forces and better adaptation to the user’s intention.

During the encoding, we segment the task based on the initial contact into an approach phase and a process phase. Each of these are encoded as Dynamic Movement Primitives. Similarly, the surface normal estimate is encoded as Radial Basis Functions and synchronized with the process DMP phase system.

During the execution, a parallel position/force controller is used to meet both a position target and a force target, which is calculated based on a user-specified force magnitude applied on the encoded normal direction. The encoded normal is again used to vary the eigenstructure of the force controller gain matrices so as to apply force control in the contact direction and position control otherwise. Finally, we are able to reduce impact forces during the transition from approach to process by slowly modulating the goal of the approach DMP.

We have evaluated and analyzed our system on a benchmark based on a real industrial PCB gluing application and verified the suitability of our proposed methods, showing robust execution performance capable of meeting both the kinematic and dynamic requirements of the task. We have also shown that the surface normal estimator works satisfactorily on curved geometries, and that the adaptive controller used for demonstration results in faster and more accurate executions compared to a standard controller implemented on a collaborative robot.