The first approach involves creating more detailed models. For instance, studies have developed precise models of the lumbar spine, addressing areas that are not adequately represented by the base model of OpenSim (Christophy et al., 2012; Raabe and Chaudhari, 2016). Tools such as NMSBuilder developed by G. Valente et al. have also been introduced to aid researchers in developing customized OpenSim models (Valente et al., 2017). However, there are inherent limitations to the accuracy that is achievable in simulations owing to challenges in replicating the complexity of the human body. In addition, measurements such as CT scans are required to capture individual differences accurately, which may not be feasible for all users.

The second approach focuses on optimizing the model parameters. OpenSim moco (Dembia et al., 2021) introduces functions to calibrate parameters such as the muscle length and maximum tension using individual user data. Although this improves the personalization, discrepancies between the estimated muscle activity and measured EMG data persist. Valente et al. proposed robust optimization methods to address uncertainties in muscle models (Valente et al., 2014). However, parameter optimization alone cannot resolve fundamental modeling errors in musculoskeletal models.

In contrast to previous studies that focused only on optimizing the parameters or models within the OpenSim framework, this study introduces the NEM2E framework. By addressing temporal and spatial modeling errors separately, the STDR-Net compensates for these errors using a Seq2Seq model with an attention mechanism. This approach has the potential to mitigate the limitations of conventional methods and improve the accuracy of muscle activity estimation.

NEM2E framework and STDR-Net.

The proposed NEM2E framework was validated using two publicly available datasets (Liu et al., 2008): published in https://simtk.org/projects/mspeedwalksims and the datasets used in the literature (Hamner and Delp, 2013) published in https://simtk.org/projects/nmbl_running. The contents of the dataset are shown in Table 1 These datasets contain the estimated muscle activation results for which individual body parameter tuning and other standard tuning has already been performed in the respected studies.

The parameters of the STDR-Net were trained using a dataset, where the simulated muscle activation from OpenSim served as the input and the muscle activation obtained from the measured EMG served as the output. The squared error was used as the loss function and the optimization process was carried out using the Adam algorithm (Kingma and Ba, 2015).

For all conditions, the STDR-Net was configured with 20 units corresponding to the number of muscle actuators in OpenSim for the encoder (input layer), 700 units for each intermediate layer, and 20 units for the decoder (output layer). Training was performed with a batch size of 1 and 700 epochs. The data were standardized by thinning the input and output data to 20 samples each. For the temporal attention models, the input dimension corresponded to the single dimension of the target muscle.

The performance of the proposed model was compared with the baseline muscle activity estimations of OpenSim. All EMG measurements in each dataset (running data: 16 muscles, walking data: 5 muscles) were verified. The names and abbreviations of each muscle are as follows: the soleus (soleus), biceps femoris long head (bi), semimembranosus (sem), tibialis anterior (tib), rectus femoris (rect), gluteus maximus (glmax1, glmax2, and glmax3), gluteus medius (glmed1, glmed2, and glmed3), gastrocnemius medial head (gasmed), gastrocnemius lateral head (gaslat), vastus lateralis (vaslat), and vastus medialis (vasmed) muscles. These names follow the column labels used in publicly available OpenSim muscle activity datasets and may differ from standard anatomical abbreviations. All data were segmented into individual gait cycles. A leave-one-out cross-validation approach was used, with data blocked by subject for both the walking and running datasets.

The accuracy of the model was assessed using the root mean squared error (RMSE) between the model output y ̂ and measured EMG data y , which is calculated using the following (Equation 1): E = 1 N ∑ i = 0 N y i − y ̂ i 2 , (1) where N = 20 is the number of samples. Assuming that the error in the refined muscle activity obtained by the proposed method is E pro and the error in the estimated muscle activity obtained from OpenSim is E OpenSim , the error improvement rate E r is calculated as E r = E pro E OpenSim − 1 . (2)

Because each subject has data for four different speeds, the average of the four RMSE values is treated as the representative value for each subject. The null hypothesis H 0 : E r = 0 was tested using a T-test. The effect size is calculated using Cohen’s d in the following (Equation 3). d = E r ave − μ 0 E r std , (3) where, E r ave and E r std denote average and standard deviation of E r , respectively. μ 0 is hypothesized population mean.

The 95% confidence interval (CI) is calculated using the following (Equation 4). C I = E r ave ± t α 2 , n − 1 E r std n , (4) where, t α 2 , n − 1 represents the two-tailed t-value of α 2 in the t-distribution with n-1 degrees of freedom. n is sample number. α is 0.05. Three input data patterns were defined for validation.

• One muscle, quarter cycle (One-Quarter): Sequence data for a quarter cycle of one muscle of the target muscle.

One-Entire tested the temporal distortion compensation. Because the proposed framework emphasizes the importance of utilizing the entire sequence data to address temporal distortion, the effect of using only a portion of the sequence as input was examined in the One-Quarter model. All-Entire assessed the spatial distortion compensation by incorporating all muscle activity data. Temporal attention (Figure 2a) was used for One-Entire and One-Quarter, whereas spatial attention (Figure 2b) was applied to All-Entire. Tukey’s multiple comparison tests were performed for each muscle to analyze significant differences among the three input patterns.

Figure 3 shows representative results from the NEM2E framework. The results for the One-Entire condition in the walking dataset are shown in (a) and the results for the All-Entire condition in the running dataset are shown in (b). The blue lines indicate simulated muscle activity from OpenSim, the red lines show the refined output from the STDR-Net, and the black lines represent measured EMG data. The horizontal axis denotes the gait cycle (right heel strike to left heel strike), whereas the vertical axis represents muscle activation. In all cases, the refined muscle activation (red) closely matched the measured EMG (black), demonstrating the ability of the framework to improve upon the baseline OpenSim estimates (blue).

Tables 2 and 3 summarize the statistical results for the improvement rates in the estimation accuracy for the walking and running datasets, respectively. ∗ indicates that the null hypothesis H 0 : E r = 0 is rejected in the estimation error improvement rate for each muscle calculated by Equation (2). In the walking data, significant improvements in accuracy were observed for the soleus, tib, and rect under the All-Entire condition. No significant improvements were observed for bi and sem under any conditions. In the running dataset, significant improvements were evident for all muscles in the All-Entire condition. Under the One-Quarter condition, only 4 out of the 16 muscles showed significant improvement, and under the One-Entire condition, 10 out of the 16 muscles demonstrated significant improvement.

Figure 4 show the improvement rates in the estimation error for the walking and running datasets. A value of 0 represents the baseline accuracy, whereas negative values indicate improvement. In the walking dataset, there were no significant differences between conditions. Conversely, in the running data, significant differences were observed between One-Quarter and All-Entire in 13 out of 16 muscles as well as between One-Entire and All-Entire in bi and glmax2.

In the running dataset, significant improvements were observed in all muscles under the All-Entire condition. However, under the One-Entire condition, no significant improvements were observed in 6 out of the 16 muscles. Furthermore, in the walking dataset, significant improvements in the accuracy of the gastrocnemius muscle were observed under all conditions. The rectus femoris muscle showed improvements only under the All-Entire condition, suggesting that the spatial attention mechanism in this condition plays a crucial role in enhancing muscle activity estimation. Furthermore, under the One-Quarter condition, significant improvements were observed in only 5 out of 21 muscles across both running and walking data, indicating the need to consider the entire sequence rather than applying window processing to the sequence data. Moreover, as the models in this study were trained only on running and walking data, shortcut learning may have occurred, in which average muscle activity patterns were learned that are dependent on the movement duration. Then, for the One-Entire model trained on the soleus walk data, which showed strong improvement results in the validation, the improvement rate of the refined results was tested with a T-test by entering 32 randomly frequency- and phase-changed data, which was the same number of test trials in the walk dataset of the original paper. The average improvement rate (- indicates improvement and + indicates degradation) was +0.48, with a p-value of 0.001, showing significant degradation. Thus, the results demonstrate that shortcut learning, which is dependent on the operating time, did not occur.

Furthermore, Figure 5 shows the loss trends for the training and validation data across each learning trial. In the running data, as shown in Figure 5a, the validation loss converged, whereas, in the walking data, as shown in Figure 5b, the loss temporarily increased without decreasing and then converged. For the running data, the validation loss demonstrates a stable relationship with the training loss, consistently decreasing alongside it. This trend suggests that overfitting is unlikely for this dataset, even with the relatively high number of epochs (700). The observed generalization across individuals further indicates that the model is successfully learning the underlying patterns for this specific sequence task without memorizing the training data.

Conversely, for the walk data, the validation loss exhibits an increasing trend from early epochs. However, the validation loss remains stagnant and elevated from the outset, suggesting that the intrinsic complexity or inherent noise within this dataset may hinder the model’s ability to achieve substantial learning improvements from the beginning. This observation implies that for the walk data, increasing the dataset size is considered more critical for improving model performance and generalization than simply extending the training duration or adjusting architectural parameters.

Figure 6 depicts two representative examples of temporal attention weights in the One-Entire condition. In (I), the attention mechanism primarily focuses on the central part of the input sequence to refine the initial portion of the output waveform of the model. Conversely (II) highlights attention on the latter part of the input sequence, corresponding to the refinement of the second half of the output waveform.

The temporal attention results (Figure 6) reveal that the model allocated higher weights to data near the center of the input sequence when refining the muscle activity. This suggests that the temporal distortions in the estimations of OpenSim can be effectively learned and corrected by the attention mechanism. However, the lack of improvement in the One-Quarter condition for certain muscles, such as the biceps femoris long head, underscores the need for longer temporal sequences to capture the full distortion. The fact that the attention was focused on the peak shift in the attention weight results suggests that the model learned as intended, correcting for temporal bias rather than movement duration.

Figure 7 depicts the spatial attention results for the tibialis anterior in the All-Entire condition. Each skeleton visualizes the gait cycle motions at 0%, 50%, and 100%, with the top 10 muscles having the highest attention weights. At 0% and 100%, corresponding to the heel strike phase, similar muscle groups with high attention weights were identified. At 50%, additional trunk and hip muscles were selected, reflecting a shift in the coordination required during the swing phase.

The spatial attention revealed that the top 10 muscles with the highest attention weights for the tibialis anterior, which exhibited improved accuracy only in the All-Entire condition, varied at 0%, 50%, and 100% of the gait cycle. This variation may be explained by differences in the active muscle groups during the stance and swing phases of gait.

In addition, the top 10 muscles with the highest attention weights included muscles from the trunk, hip, and lower back on the opposite leg, in addition to the peripheral muscles of the target muscle. The peripheral muscles may have complemented the information obtained through muscle synergy during gait. Regarding the involvement of the hip, pelvis, and trunk muscles of the opposite leg, previous studies have reported that the trailing limb angle is related to ankle moments during gait (Hsiao et al., 2016). This suggests that ankle torque is not solely generated by muscle activity in the target leg, but also involves coordinated actions with the trunk and opposite leg. Owing to the significant number of muscles associated with the hip and trunk, these regions likely act as a counterweight for ankle movement. The results suggest that incorporating information from these muscles improved the accuracy of the muscle activity estimations by accounting for the broader coordination required during gait.