This study was carried out in accordance with the recommendations of the Northwestern University Institutional Review Board with written informed consent from all subjects. Following IRB approval, 10 able-bodied subjects (7 male, 3 female; 23–29 years, 160–193 cm, 54–95 kg) completed the experiment. Before walking, subjects were instrumented bilaterally with wearable sensors to measure lower limb muscle activity and joint and limb kinematics. EMG signals were recorded using bipolar surface electrodes (DE2.1; Delsys, Boston, MA, USA) from the same seven muscles in each leg: tibialis anterior (TA), medial gastrocnemius (MG), soleus (SOL), vastus lateralis (VL), rectus femoris (RF), biceps femoris (BF), and semitendinosus (ST). These muscles were chosen because they are in part responsible for hip and knee flexion/extension and ankle plantarflexion/dorsiflexion, movements that are commonly assisted by wearable devices. They are also relatively easy to target when facing the subject from in front and behind and are similar to muscle sites used by Sylos-Labini et al. (2014). Electrode placement was guided by the Surface ElectroMyoGraphy for the Non-Invasive Assessment of Muscles (SENIAM, seniam.org) standards. We palpated to locate the muscle belly and oriented the electrode along the primary fiber direction (Kendall et al., 2005), and verified placement by having subjects perform maximum voluntary contractions. The muscle sites were prepared by removing excess hair and the skin was cleaned by mildly scrubbing with an alcohol wipe. Sensors were attached to the skin with a double-sided adhesive. Signals were amplified by 1000x, hardware band-pass filtered between 20 and 450 Hz (Bagnoli 16, Delsys), and sampled at 1 kHz.

Joint kinematic signals (sagittal plane only) were recorded using electrogoniometers (SG150; Biometrics Ltd, Newport, UK) placed along the knee and ankle and sampled at 500 Hz. 6-DOF (tri-axial accelerometer and gyroscope) IMU's were placed bilaterally on the subjects' thigh (below RF EMG electrode) and shank (adjacent to TA EMG electrode) and sampled at 500 Hz (MPU-9250; Invensense, San Jose, CA, USA). All signals were simultaneously recorded with a custom 16-bit data acquisition device that permits multi-rate sampling. We chose these wearable sensors because they are analogous to sensors commonly embedded in prostheses, orthoses, and exoskeletons (such as joint position encoders and shank/thigh IMU's) and are more easily integrated with existing device-based sensorization in a hypothetical hybrid system. Other force and interaction torque sensors such as load cells and strain gauges were excluded because they are not as relevant for our device-agnostic approach and are more difficult to integrate if not already embedded in the device.

To facilitate integration with our data acquisition software, all wearable sensors were used in a tethered setup; as a drawback, fully instrumenting one leg took up to an hour. The full instrumentation setup is shown for a representative subject in (Figure 1 top, middle). Prior to data collection, the goniometers were calibrated while the subject was in the upright standing position. In an experimental session, each subject completed approximately 25 repetitions of a circuit consisting of walking on level ground (LW), ascending/descending a ramp with a 10° slope (RA/RD), and ascending/descending a four-step staircase (SA/SD) step-over-step using a data collection procedure previously described in Young et al. (2014b). These activities were chosen because they encompass the main types of terrain likely encountered in community ambulation. Subjects were instructed to freely transition between activity modes at their self-selected speed while the experimenter labeled the true locomotor intent of the subject using a key fob.

Heel contact and toe off gait events for each leg were reliably identified by finding peaks in the low-pass filtered (1st order Butterworth, 6 Hz) sagittal plane angular velocity of the shank segment using a dual-minima method similar to (Jasiewicz et al., 2006; Maqbool et al., 2016). Briefly, the largest peaks in angular velocity were first used to identify mid-swing events. Toe off and heel contact events were identified by searching for peaks before and after each mid-swing event, respectively. Gait initiation and termination strides and trials during which the subject paused, stumbled, or tripped were excluded. For each subject, there were 530 ± 46 heel contact events and 536 ± 45 toe off events for each leg (mean ± standard deviation). EMG signals were first high-pass filtered (6th order Butterworth) at 20 Hz to attenuate motion artifact (De Luca et al., 2010). EMG signals were then notch-filtered (6th order Butterworth, 6 Hz width) at 60, 180, and 300 Hz (based on spectral analysis) to remove ambient interference. Goniometer and IMU signals were low-pass filtered (6th order Butterworth) at 10 and 25 Hz, respectively. Because we did not use foot-mounted IMU's, joint velocities could not be estimated using inertial signals only for the ankle. For consistency, we indirectly estimated joint velocities for both the knee and ankle by taking the centered-difference numerical derivative of the low-pass filtered joint position signals instead and added these velocities to the goniometer channels.

All signals were segmented into 300 ms analysis windows before each identified heel contact or toe off gait event (one window/event). For each window, we extracted features previously used in intent recognition for online control of a powered knee-ankle prosthesis. Features for goniometer and IMU signals included the mean, standard deviation, maximum, minimum, initial, and final values (6 features/channel) (Varol et al., 2010). Features for EMG signals included the mean absolute value (MAV), waveform length, number of zero crossings and slope sign changes, and the coefficients of a sixth-order autoregressive model (10 features/channel) (Huang et al., 2005; Hargrove et al., 2008). These heuristic features were chosen because they can be computed efficiently on an embedded system and concisely capture the general shape of mechanical signals and the frequency content of EMG signals. Bilaterally, there were a total of 22 sensors (14 EMG, 4 goniometer, 4 IMU) and 46 channels (14 EMG, 8 goniometer, 24 IMU). The feature dimensionality for all ipsilateral and bilateral signals was 166 and 332, respectively (Figure 1, bottom).

For each subject, we evaluated leg- and mode/phase-specific classifiers (e.g., right heel contact, left toe off) for several sensor sets to compare their offline error rates (Figure 2). We used a mode-specific classification scheme previously developed for powered leg prosthesis control (Young and Hargrove, 2016), which achieves lower error rates by encoding domain knowledge about the allowable transition(s) from each mode. Briefly, 20 total classifiers were trained to encompass all combinations of the four gait events (right/left heel contact or toe off) and five locomotor activities (level ground, ramp ascent/descent, stair ascent/descent). The total number of steps used to train each classifier (combined between legs for all subjects) is reported in Table 1. The appropriate classifier for each prediction was selected based on the activity just before the gait event (i.e., incoming activity based on the key fob label). The possible outputs for each classifier (i.e., predicted activities) only included remaining in the current locomotor activity or transitioning to another allowable mode (e.g., in stair ascent mode, remaining in stair ascent or transitioning back to level walking but excluding stair descent and ramp ascent/descent). The error rate was defined as the proportion of incorrectly classified gait events in the testing set and was computed by averaging across legs and gait events for each subject. Errors were also categorized as steady-state or transitional. We performed randomized 10-fold cross-validation for each subject for 12 different sensor sets (Figure 1, bottom) using the steps collected from all circuits completed during the experimental session. The ipsilateral side was defined as the side on which the gait event was identified, which could have been either the leading or trailing leg. Sensors were divided into four modality groups: EMG only (EMG), goniometer only (GONIO), IMU only (IMU), or all combined (ALL). Sensors were also divided into three laterality groups: ipsilateral (I), contralateral (C), or bilateral (B).

Linear discriminant analysis (LDA) has emerged as a convenient a priori choice of classifier for intent recognition for the control of upper- and lower-limb prostheses because it provides a good compromise between classification accuracy and computational efficiency (Hargrove et al., 2007; Scheme and Englehart, 2011). Other commonly used classifiers include support vector machines (SVM) and artificial neural networks (ANN), which can represent more complex, non-linear decision boundaries and may be more appropriate for modeling transitions. The sensor set containing all bilateral signals contains more features than previously used in intent recognition for lower-limb prostheses so we also assessed the effect of classifier type on error rates for ipsilateral and bilateral sensor sets containing all modalities. For all classifiers, the feature data were normalized to have zero mean and unit variance. For LDA, the input dimensionality was further reduced using principal components analysis (PCA) to preserve 95% of the total variance and the prior for each classifier was set to be equiprobable. Hyperparameters for SVM (one-vs-one, linear kernel, C = 10 using the scikit-learn Python package) and ANN (one hidden layer with 10 units, hyperbolic tan activation function, stochastic gradient descent with momentum, adaptive learning rate initialized to 0.1 using the scikit-learn Python package) were chosen based on pilot data.

We performed repeated measures ANOVA for LDA classifiers with error rate as the response variable, modality and laterality (ipsilateral and bilateral only) as fixed within-subject factors, and subject as a random factor. We expected some modalities would benefit more from bilateral information so we included an interaction term. Post-hoc comparisons (paired t-test) with Bonferroni correction were conducted on statistically significant factors. We also used paired t-tests to compare ipsilateral and contralateral sensor sets. We performed repeated measures ANOVA for the combined sensor set with error rate as the response variable, laterality and classifier as fixed within-subject factors, and subject as a random factor. We expected more complex classifiers to perform worse for higher dimensionality data so we included an interaction term. Post-hoc comparisons (paired t-test) with Bonferroni correction were conducted on statistically significant factors.

To determine the optimal number and type of sensors to instrument on the contralateral leg, we performed sequential forward selection for each subject to choose the sensors which minimized overall error rate with LDA classification (10-fold cross-validation), beginning with all ipsilateral sensors as the baseline and ending with all bilateral sensors. We chose not to identify the bilaterally optimal sensor combination (i.e., beginning with the empty set) because we were primarily interested in the effect of adding contralateral sensors. After each iteration, all features associated with the selected sensor were added to the existing feature set and the selected sensor was removed from the set of remaining sensors. The composition of the sensor set after each iteration was recorded. We performed repeated measures ANOVA with error rate as the response variable, iteration as a fixed within-subject factor, and subject as a random factor. We also performed post-hoc comparisons (paired t-test) between iterations using a Bonferroni correction.

Subjects' bilateral neuromechanical signals were distinguishable based on locomotor activity and mostly consistent between legs and trials throughout the experimental session. Representative data depicting all sensors except SOL, RF, and BF (similar to MG, VL, and ST, respectively) are shown in Figure 4 (EMG), Figure 5 (goniometer), and Figure 6 (IMU) for one representative circuit. Qualitatively, unique patterns of activation in the feature space (Figure 7) of certain channels aligned closely with different activities and their associated transitions (only mean value features shown for overlapping 300 ms windows).

There was a significant interaction effect between modality and laterality (p = 5.53 × 10⁻⁷). Simple main effects analysis showed that overall error rates for classifiers using bilateral sensors were significantly reduced compared to their unilateral counterparts for almost all modality groups and machine learning algorithms (Figure 8, Table 2). There was a significant effect of classifier (p = 2.22 × 10⁻¹⁶) for the combined sensor sets without interaction between classifier and laterality (p = 0.32). Steady-state and transitional error rates were also significantly reduced when using bilateral sensors for all modality groups and machine learning algorithms with the exception of steady-state errors with SVM and ANN for the fused sensor set (ALL) (Table 2). There was generally no significant difference between unilateral (ipsilateral versus contralateral) single modality sensor sets for overall, steady-state, or transitional error rates; however, contralateral sensors had significantly higher transitional error rates with LDA and overall and transitional error rates with ANN for the fused sensor set (Table 2). For unilateral sensor sets, the error rate of IMU was lower than EMG or GONIO and decreased with sensor fusion; the lowest average overall error rate (1.43 ± 0.24%) was achieved by the LDA classifier using all bilateral sensors. The random effect of subject was significant (p < 10⁻¹⁰) and the overall error rate using all bilateral sensors with LDA ranged from 0.52 to 2.78%. Interestingly, there was no significant difference between overall error rates for either GONIO(B) and ALL(I) (p = 0.08) or IMU(B) and ALL(I) sensor sets (p = 0.26). For ALL(I), there was no significant difference between classifiers; however for ALL(B), the overall error rate of LDA was significantly lower than ANN (p = 3.02 × 10⁻⁵) but not different from SVM (p = 0.11).

The optimal collection of additional contralateral sensors after each iteration varied between subjects but on average goniometer and IMU sensors were preferentially selected before EMG sensors (Figure 9, top). One contralateral lower leg kinematic sensor (i.e., ankle goniometer or shank IMU) was selected within the first two iterations for every subject; additionally, at least one goniometer sensor (i.e., ankle or knee) was selected within the first two iterations for all but two subjects. Overall (p = 1.27 × 10⁻⁴), steady-state (p = 2.35 × 10⁻³), and transitional (p = 2.54 × 10⁻³) error rates were significantly reduced from baseline after only one additional contralateral sensor (Figure 9, bottom). After four iterations, the error rate plateaued and even increased slightly when approaching all bilateral sensors.

Adding kinematic information from the non-prosthesis side modularly and consistently reduced offline steady-state and transitional error rates for both the baseline delayed/merged classifier and the more generic classifier (dagger symbol) (Table 3). Sensor fusion with non-prosthesis side sensors yielded the greatest relative improvement for the heel contact level walking and ramp/stair descent classifiers. For the more generic classifier which used both additional IMU's (Row 7), transitional error rates were slightly better than baseline (Row 1) but overall error rates were still higher because the toe off classifier performed worse. By using prosthesis signals only for the toe off classifier only (Row 8), the performance of the more generic classifier improves and matches the baseline classifier (Row 1).

Our protocol was nearly identical to Young et al. (2014a) and Spanias et al. (2015) but involved able-bodied subjects, no device, and wearable sensors. Using all ipsilateral sensors in this setup, we achieved average transitional error rates (5.94 ± 0.84%) that were roughly half those reported by Spanias et al. (2015) and Young et al. (2014a); steady-state error rates (1.25 ± 0.19%) were comparable. As expected, error rates were higher for transitions, which had fewer training examples and generally more variability. Although we used a more generic mode-specific classification scheme compared to Simon et al. (2017) by neither merging level walking and ramp ascent classes nor adding special classifier configurations (e.g., predict transitions between stairs and level ground during mid-swing or mid-stance), we achieved error rates approaching the state-of-the-art for unilateral-informed intent recognition control of a powered leg prosthesis. Because the addition of only one contralateral sensor (preferentially kinematic) significantly reduced error rates, our results suggest that substantial improvements in controllability may be achievable with minimal instrumentation of the contralateral limb. Unexpectedly, the error rate increased slightly instead of plateauing after the addition of four sensors. Although these consistent increases were not statistically significant, they suggest that the extraneous sensors may not only be redundant but also detrimental for intent recognition because they contribute to model overfitting and/or introduce undesirable sensor drift. From a practical standpoint, the total time required for instrumenting the subject would also be substantially reduced by using an optimized subset of sensors instead of all sensors. Using delayed transitions (i.e., windows starting 210 ms before the gait event and ending 90 ms after the gait event) has significantly reduced intent recognition errors for prosthesis control because delayed windows span both the onset of and continuation of movement (Simon et al., 2017). However, although offline error rates would likely have decreased further we chose not to implement this delay for the sake of clarity and generalizability to non-prosthesis applications for which such delays may not be a desirable tradeoff for accuracy.

We showed that unilateral (ipsilateral or contralateral) sensor sets performed comparably for single modalities, which had previously never been demonstrated for intent recognition control strategies. Thus, our results suggest that using signals from the non-affected leg (which alternates between serving as the leading and the trailing leg) for control and perhaps signals from the affected side for gait segmentation could be suitable for assisting individuals with severe unilateral impairment given proper training on how to perform transitions. Therefore, we believe high performance intent recognition control systems could still be realizable for a range of assistive devices with simple sensorization by integrating with wearable sensors and modifying a generic intent recognition control architecture. Our findings also showed that linear discriminant analysis (LDA) can perform as well as, if not better than, more complex algorithms such as support vector machines (SVM) and artificial neural networks (ANN) on a feature set with higher dimensionality (up to 22 sensors, 332 features) than we have previously used in an intent recognition framework for lower-limb prosthesis control (Spanias et al., 2015). To our knowledge, no other studies have used simultaneously recorded bilateral lower-limb neuromechanical signals from able-bodied individuals in an intent recognition control framework; therefore, these results for predicting locomotor activities for unimpaired individuals freely walking without a device also help to establish classification accuracy benchmarks for this framework.

Our offline study with an above-knee amputee subject also lays a foundation for hybrid setups combining device-embedded and wearable sensors by demonstrating the feasibility and success of using bilateral sensor fusion for intent recognition control of an assistive device for a clinical population. Although bilateral sensor fusion achieved modest reductions in offline error rate for the state-of-the-art classifier, the clinical significance of these improvements remains unknown without an online analysis with more subjects walking in settings more representative of home use (longer sessions, more variability in terrain, no clinician to check alignment or supervise, etc.). We also demonstrated that a more generic classifier using bilateral sensor fusion could match the state-of-the-art classifier, thus showing that bilateral sensor fusion presents an alternative set of tradeoffs (instrumenting the non-prosthesis side versus delayed transitions and no unique assistance mode for ramp ascent) for reducing error rates which may be preferable for some subjects. Aside from intent recognition, other applications of neuromechanical sensor fusion have included volitional control of ankle position for below-knee amputees (Au et al., 2005, 2008; Kannape and Herr, 2014) and stumble detection and classification (Zhang et al., 2011a). Beside neuromechanical sensor fusion, vision-based environmental sensing has also shown potential to improve control in an intent recognition framework (Zhang et al., 2011b; Krausz et al., 2015). Although we have focused on bilateral neuromechanical sensor fusion for intent recognition, there are many other potential applications of bilateral sensor fusion including measuring balance, controlling stumble recovery mechanisms, modulating a reference trajectory (e.g., estimating slope of an incline based on the leading leg), and estimating walking speed. There is also potential to use other modalities such as soft capacitive stretch sensors and vision in bilateral intent recognition systems.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The reviewers JM and FOB and handling Editor declared their shared affiliation.