Recent progresses in active prosthesis control for the upper limb include the introduction of novel control approaches (Scheme and Englehart, 2011; Jiang et al., 2014a; Amsuess et al., 2016), sensor types and sensor fusion algorithms (Weir et al., 2003; Dosen et al., 2010; Cipriani et al., 2014; Ortenzi et al., 2015; Nissler et al., 2016), surgical techniques (Kuiken et al., 2004; Aszmann et al., 2015), as well as advanced hardware (Cipriani et al., 2011; Grebenstein et al., 2011; Catalano et al., 2014). Nonetheless, the impact of these advances towards improving the experience of the everyday end user is still limited. The discrepancy between myoelectric solutions which academia develops and promotes, and the systems available on the market is indeed substantial. This issue has been previously discussed (e.g., Hill et al., 2009; Jiang et al., 2012; Farina and Aszmann, 2014) and relates to the conditions in which new methods are tested.

The necessity for testing prosthetic solutions in a greater number of amputees than currently done is a widely recognized problem. Moreover, the tests used often fail to include clinically relevant metrics. Performance metrics prevalent in laboratory research may be poorly associated to the clinical outcome, as noted previously (Simon et al., 2011; Jiang et al., 2014b; Ortiz-Catalan et al., 2015). In this perspective article, we support these arguments to further substantiate the relevance of this problem.

Transferring myoelectrical systems developed in the laboratory to clinical settings is a challenge that requires multidisciplinary efforts. Clinical tests, although not ideal, offer the most realistic prediction of the system performance in the daily use. These tests account for several of the challenges that laboratory-based assessment methodologies tend to neglect. For example, noiseless laboratory-based evaluation platforms fail to account for the end effector loads, poor socket fitting and sweating.

Here, we briefly introduce the evaluation methods regularly applied for prosthetics use, with a focus on offline approaches and some selected clinical measures. Moreover, we provide experimental data on seven conventional myoelectric users. The literature review and the experimental data are limited to the primary aim of providing our view on assessment procedures for myocontrol and suggestions for their improvement.

Laboratory-based techniques and tests for measuring the performance in controlling a myoelectric interface are numerous and, in case of offline techniques, have been mainly derived or adapted from the machine learning literature. On the other hand, initially, clinicians have mostly adapted established hand and arm impairment assessment tools to the evaluation of functional recovery with prostheses. However, in recent years, new clinical measures have been introduced to specifically target the amputee patient population.

We provide data on amputees that compare the accuracy estimated offline, for one of the classic control schemes developed over the past decades, with clinical scores. These data serve the purpose of representatively supporting the need for clinical tests for myocontrol developments. Therefore, the experiment and results do not aim at providing general conclusions on all myocontrol schemes and evaluation methods but rather at exemplifying the view presented in this perspective article.

Seven male transradial myoelectric users agreed to participate. They were all fit with custom-made sockets and with the Michelangelo hand (Ottobock Healthcare GmbH, Austria) with additional wrist rotation and flexion/extension units. The study was performed in accordance with the recommendations of the local ethics board of the Medical University of Vienna (Ethics Commission number 1044/2015), with written informed consent from all subjects. Subjects were fully briefed on the study protocol and possible adverse effects in presence of a clinical staff. All given consents are in accordance with the Declaration of Helsinki. All involved participants were transradial amputees with previous experience in using commercially available prosthetic devices. Before participation in the experiment medical state of each participant has been checked by the clinical staff.

The control of the prosthesis was based on the common spatial pattern (CSP) based classifier, as described by Amsuess et al. (2016). The EMG signals were recorded with 8 bipolar surface electrodes (Otto Bock raw signal electrodes 13E200 = 50AC). The control system allowed the subjects to access seven prosthetic functions—wrist flexion/extension, wrist pronation/supination, hand open, pinch, and key grip. All the motions were recorded in three arm positions (relaxed, fully extend arm in front of the ipsilateral shoulder, and fully extended arm across the contralateral shoulder) and at three forces (30%, 60% and 90% relative to the EMG level at maximum voluntary contraction force) while wearing the full prosthetic fitting. For offline accuracy assessment, the classifier was trained by data collected in only one arm position and tested against the remaining two data sub-sets. The average of the three scores was the reference performance of the subject. The entire data set was used for training the same CSP classifier that allowed execution of the B&B and SHAP tests. These particular clinical tests were chosen since they cover a wide range of assessment goals while being entirely objective. Additionally, these two tests have been widely recognized and familiar to academic and industry-based developers as well as clinical experts.

The performance scores in both offline and clinical tests are presented in Figure 1. The offline classification accuracies are slightly lower than in other studies (Ahsan et al., 2010; Liu et al., 2013) because of the different arm positions used for training and testing as well as the full prosthetic fitting which is not usual in offline evaluation studies. Although with these choices we have presumably maximized the prediction capacity of offline indexes for clinical scores, still the clinical scores did not strongly correlate with the offline performance measures. For example, there were two patients who achieved a similar SHAP score just below 40 but with very different classification accuracies of <70% and >85% (Figure 1A). Similarly, two patients who had similar classification accuracies of 70%–75% had SHAP scores of 27 and 47 (Figure 1A). The B&B test requires less skill to be performed than the SHAP. However, the B&B score was even less associated to the offline classification than the SHAP (Figure 1B). For example, subjects with an offline accuracy >95% performed very differently in this test (Figure 1B). Furthermore, when considering strictly the hand movements—hand open, fine pinch and key grip—that are primarily used for this test, the mismatch between this test and offline performance was even more substantial. This was observed consistently in all patients but it is shown representatively for only two patients in Figure 2. For these patients, the average classification rate across the three hand motions was 89% and 79% whereas the transferred blocks (score of the B&B) were 5 and 12, respectively.

When the offline evaluation was performed by using data collected without wearing the prosthesis and tested on the same arm position as the training, as more commonly done in laboratory tests (e.g., Englehart et al., 1999; Hargrove et al., 2009; Li et al., 2010; Ortiz-Catalan et al., 2014b), the resulting offline classification rates were high and comparable to those reported in the literature (>90% on average). However, once fully fitted, the majority of patients were unable to successfully conclude the clinical evaluations without retraining, suggesting that the classic offline evaluation procedure performed in several research studies, even though indicative, does not necessarily vouch for superior clinical performance.

Abandonment rates among upper limb myoelectric prosthetic users are still very high (Burrough and Brook, 1985; Glynn et al., 1986; Østlie et al., 2012). At the same time, research efforts have provided several new solutions for myocontrol that have been proven to be highly functional strictly under laboratory conditions. The limited transfer from research to real world applications likely depends on an insufficient level of evaluation procedures.

Using novel prototypes of myoelectric systems in daily life would provide the ultimate assessment, but this strategy would often require official certification by notified bodies, which often goes beyond the possibilities of academic development. The COAPT system (Coapt LLC, 2016) is one of the first systems that has reached this level of testing. Clinical evaluations at earlier stages are a compromise between laboratory conditions and real-life tests. Although not perfect, clinical tests are closer to the conditions of interest for the users than offline assessments or online tests using virtual prostheses which provide valuable, but not always sufficiently transferable scores. Here, we have presented an example of this dissociation on a small sample of amputees and focusing on offline metrics, for demonstration purposes. We have compared clinical scores with offline indexes of performance extracted in the most realistic offline conditions (patients wearing a prosthesis, training and test sets obtained on different arm postures). Despite these conditions rarely being met in the offline studies, the prediction capacity for clinical outcome was not strong. On the other hand, when the offline indexes were obtained in more common laboratory conditions without the prosthesis and for the same arm posture for test and training, the clinical information they provided was minimal (indeed with this training, once fitted with the prosthesis patients could not even finish the clinical tests without re-training). Further extrapolating, it is obvious that an offline analysis performed in these simple conditions and, in addition, on able-bodied individuals instead of patients, is of rather poor clinical value. While we are fully aware that in the initial evaluation of a new myocontrol scheme the strict laboratory tests on healthy individuals are valuable and needed for assessing the basic algorithmic working principles, there is also the need to make efforts in continuing the evaluations of promising algorithms in clinically-relevant settings (and to further develop clinical tests that fully represents the functional benefits). We believe that the evaluation stages after the laboratory level have had so far a slower progress, and less academic interest, with respect to the proposal of new algorithms.

Considering the discrepancy presented in the literature (Jiang et al., 2014b; Ortiz-Catalan et al., 2015) and further supported here, it seems necessary that novel myoelectric systems that passed laboratory testing are then fully clinically evaluated for assessing their performance. For this purpose, researchers and clinicians should jointly devise a standardized testing framework for quantitatively and qualitatively assessing the performance of upper limb prosthetic devices and their users to boost the process of commercialization and, as a consequence, availability for the patients. This need does not only relate to the feed-forward control aspects, on which we focused here, but also to fully closed-loop systems that include sensory feedback integration (Gonzalez and Yu, 2009; Jorgovanovic et al., 2014; Ortiz-Catalan et al., 2014a).

IV, DF and OCA: substantial contributions to the conception; IV, ADR, SA, DF and OCA: design of the work; IV, ADR, TH, AS and SA: the acquisition; IV, ADR, TH, AS, SA, DF and OCA: analysis; IV, ADR, TH, DF and OCA: interpretation of data for the work; IV, ADR, DF, OCA: drafting the work and revising it critically for important intellectual content . The major writing of the report was completed by IV, DF and OCA. Final approval of the version to be published was given by all authors.

This work was supported by the European Union’s Horizon 2020 research and innovation program under grant agreement number 687795 (project INPUT) and by the Christian Doppler Research Foundation of the Austrian Federal Ministry of Science, Research and Economy.

The handling Editor declared a past collaboration with one of the authors DF and states that the process nevertheless met the standards of a fair and objective review. The other authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.