A brain-computer interface (BCI) enables communication and control over computer applications and external devices directly from brain activity (Yao et al., 2017), and can improve the quality of life and independence of people with motor disabilities (Joadder and Rahman, 2017). Motor imagery (MI) can provide an intuitive mapping of direction between BCI interfaces and control commands better than other existing systems (i.e., steady-state visually evoked potential– and event-related potential–based BCI systems) because the required MI tasks would be closely associated with commands to control the external device (Batula et al., 2014). However, owing to the poor MI-BCI performance achieved thus far, this technique is a long way from providing interfaces that interact with external devices in applications in daily life. The performances of BCI systems feature a significant user-dependent difference; users can be categorized as either “good” or “poor.” Thus, one of the critical issues in MI-BCI studies is to improve performance and narrow the gap between good and poor performers. In addition, some users do not reach a sufficient level of accuracy (less than 70%) when trying to control an external device using a BCI system, a phenomenon called “BCI illiteracy” (Allison and Neuper, 2010; Yao et al., 2018) or “BCI inefficiency” (Shu et al., 2018; Zhang et al., 2019a). Studies revealed that 15–30% of users have BCI inefficiency and are unable to generate the proper brain rhythms even after BCI training (Leeuwis et al., 2021). Previous studies found that 55.6% (Lee et al., 2019) and 42.9% (Meng and He, 2019) of BCI-naïve participants were unable to achieve a 70% performance across one and three training sessions, respectively. BCI inefficiency is a significant problem that warrants research effort if these systems are to be useful in the future (Maskeliunas et al., 2016). Recent studies have tried to improve the classification performance of BCI inefficiency subjects using the deep learning method (i.e., convolutional neural network) because they cannot produce stronger contralateral ERD/ERS activity (Zhang et al., 2019b; Stieger et al., 2021; Tibrewal et al., 2021).

BCI research aimed at improving performance can focus either on technological factors (i.e., improvements in algorithms for feature extraction and classification) or on human factors (i.e., factors affecting how well a person generates quality EEG patterns). The latter has been relatively less studied (Penaloza et al., 2018), but many studies have attempted to improve the performance of MI-BCIs using a human-factors approach. This was done by assisting performance of the experimental task by providing visual feedback of the following kinds: (1) MI task results (i.e., direction, gauge bar) (Mousavi et al., 2017; Lukoyanov et al., 2018), (2) virtual reality environment (Skola et al., 2019; Choi et al., 2020), (3) neuro-feedback (Ono et al., 2018; Meekes et al., 2019), and (4) realistic visual feedback designed to induce a sense of embodiment (Alimardani et al., 2018) or a virtual-reality embodiment (Škola and Liarokapis, 2018). Other studies have demonstrated enhancement of MI-BCI performance resulting from improvements in training methods by use of haptics (Wu et al., 2017; Grigorev et al., 2021) and electric-stimulation feedback (Bhattacharyya et al., 2019). Haptic feedback can affect the modulation of beta (Wu et al., 2017) and mu (Grigorev et al., 2021) rhythms in the left or right sensorimotor cortex during hand movement. Another study reported the use of auditory feedback to improve MI performance by modulating sensorimotor rhythms (McCreadie et al., 2013). Lastly, studies have shown improvements in MI performance through multimodal feedback, including haptic and visual feedback (Lukoyanov et al., 2018; Wang et al., 2019), electrical stimulation, and visual feedback (Bhattacharyya et al., 2019).

Many previous studies have made efforts to improve MI performance by using visual, haptic, electrical, and auditory feedback, and some have reported improvements (Ladda et al., 2021). However, training using sensory stimulation with tangible objects has not been considered. MI is a challenging technique that often requires high concentration and extended training from users (Royer et al., 2010), and a substantial percentage of participants do not achieve good accuracy even after training (Guger et al., 2003). In this context, a hybrid training approach that combines two or more training methods can potentially improve the poor performance of some users by improving the consistency of the MI features detected by the BCI (Ramos-Murguialday et al., 2012). We focused on revising the training method to improve performance in poorly performing users by combining MI and somatosensory attentional orientation (SAO). SAO and MI are cross-modal mental tasks independent of exogenous stimuli and have different neurophysiological origins, namely the somatosensory and motor cortices, respectively (Yao et al., 2017). We believe that hybrid training methods can improve the performance of users compared to conventional methods that use only signals recorded over the motor cortex. Thus, this study explored a new hybrid imagery method to improve MI performance that employed three MI classes: left hand, right hand, and right foot. This approach involved training in the use of the sensations from tangible objects (i.e., a hard and rough ball) and motor execution (ME), where participants were instructed to imagine them at the same time.

The average accuracy for all participants was 62.29 ± 16.96% for MI and 68.88 ± 14.04% for SMI. A paired t-test revealed a significant difference in the classification accuracy of MI and SMI (Shapiro-Wilk test, p = 0.560), with the performance of SMI better than that of MI (t₁₂ = −2.669, p = 0.019, Cohen’s d = 0.423, with small effect size). Averaging over all participants, SMI achieved a 68.88% classification accuracy, 6.59% larger than that of MI, as shown in Table 1.

Figure 5 summarizes the classification performances of the good and poor performers in MI and SMI conditions. In our study, good performers were defined as those having more than 70% accuracy, a definition adopted from a previous study (Zhang et al., 2019a). In the poor performer group, the average classification performance on the SMI task was 10.73% larger than that on the MI task (62.18 ± 11.1% vs. 51.45 ± 10.27%). In contrast, the good performer group showed a slight reduction (0.86%) in performance on the SMI task compared to the MI task (80.93 ± 10.19% vs. 81.79 ± 4.91%).

We used heat maps to plot the covariance matrix pattern in MI and SMI conditions to observe the relationship among the 11 channels. For this, we conducted repeated paired t-tests on the components of the covariance matrix, using a Bonferroni-corrected p-value of 0.00008 (p = 0.01/121). For good performers, we isolated nine significant differences in the covariance matrix pattern between MI and SMI conditions. For poor performers, however, we found 30 significant differences in the connectivity among channels in the covariance matrices between MI and SMI conditions, as shown in Figure 6.

Table 2 shows the cross-modality classification results, where all eight (2^3) possible combinations for the two different modalities (MI and SMI) and for the three classes are presented. All combinations for each left hand, right hand, and right foot are as follows: MMM, SSS, SMM, SSM, SMS, MSM, MMS, and MSS. M and S mean motor image and somatosensory motor image, respectively, and the abbreviations presented in each condition are mapped in the order of the left hand, right hand, and right foot. For example, MSM condition is defined as a three-class analysis, with MI, SMI, and MI corresponding to left hand, right hand, and right foot, respectively. Classification performance of each pair of imagery tasks for all participants was 69.37–78.13% [72.13 ± 9.71% (SMM), 70.41 ± 7.85% (SSM), 78.13 ± 10.48% (SMS), 74.62 ± 10.87% (MSM), 69.37 ± 9.78% (MMS), and 75.62 ± 11.35% (MSS)], with SMS cross-modality achieving the best performance. The performance of the SMS condition was, on average, 15.84 and 9.25% greater than that of the MMM and SSS conditions, respectively. In the good performer group, classification performance of each pair of imagery tasks was 76.67–86.98% [83.23 ± 6.10% (SMM), 76.67 ± 6.04% (SSM), 86.98 ± 6.70% (SMS), 85.84 ± 5.49% (MSM), 79.90 ± 3.34% (MMS), and 86.05 ± 9.71% (MSS)]. The SMS condition achieved the best performance, which was, on average, 5.19 and 6.05% greater than in the MMM and SSS conditions, respectively. In the poor performer group, classification performance of each pair of imagery tasks was 63.52–73.22% [66.89 ± 5.54% (SMM), 66.93 ± 6.46% (SSM), 73.22 ± 8.85% (SMS), 68.39 ± 7.63% (MSM), 63.52 ± 6.84% (MMS), and 69.83 ± 7.35% (MSS)]. The SMS condition achieved the best performance, which was, on average, 21.76 and 11.04% greater than in the MMM and SSS conditions, respectively. Performances after optimized selection, which isolates the individual highest performance across cross-modalities, was on average 1.73, 2.23, and 1.45% higher in all participants, the good performer group, and the poor performer group, respectively.

Table 3 shows the results of interviews reported by subjects after each experimental paradigm. To question 1, the four good performers answered that the MI task was more helpful. They answered that the MI was more familiar than SMI as they had done it previously, and SMI was more confusing than MI. On the other hand, the eight poor performers answered that the SMI task was more helpful. Among them, four subjects reasoned that SMI felt more comfortable while performing kinesthetic MI and other three answered that tangible objects helped enable consistent imagery. In addition, subjects 10 (poor performer) and 11 (good performer) reported no significant difference between the two paradigms.

This work proposes a unique approach to improving MI-BCI performance for three classes by using a tangible object-based training method to enhance MI. Approaches to improving BCI performance are divided into technological factors (i.e., improving the algorithm) and human factors (i.e., improving the EEG patterns of the users) (Penaloza et al., 2018). Technological factors may have reached their limit in enhancing performance. In addition, the MI-BCI approach may have limitations in generating quality EEG patterns from users (Blankertz et al., 2006; Tangermann et al., 2012). This work proposes a new hybrid method of imagery that combines motor execution and somatosensory sensation from a tangible object to improve the MI performance of poorly performing users. The three classes showed an average accuracy in the SMI condition 6.59% larger than in the MI condition (68.88% vs. 62.29%, p < 0.05). In poor performers, SMI achieved an average classification performance 10.73% larger than did MI (62.18% vs. 51.45%). Good performers did not show a significant difference between the MI and SMI conditions. This work confirmed that methods using hybrid modalities (i.e., combining MI and SAO) can be useful alternatives to elementary modalities in improving the three-class BCI performance of poor performers.

The investigation of the effect of hybrid modality was motivated by how well a person can generate quality EEG patterns. First, this study attempted to achieve quality EEG patterns by enabling consistent MI using tangible objects. We confirmed that participants have different methods of MI for the task of imagining hand clenching. For instance, some imagined performing hand clenching (i.e., kinesthetic imagery) while others imagined watching a hand clenching (i.e., visual imagery). The participants performed tasks with their preferred method of MI and reported difficulties in using non-preferred methods. Inconsistent MI methods can adversely affect the quality of EEG patterns in participants (Neuper and Pfurtscheller, 2009; Wolpaw and Wolpaw, 2012). According to post-experiment interviews, participants in the poor-performer group who had experienced the hybrid modality were able to perform consistent MI by imagining the sensation of a tangible object (i.e., the imagination of hand clenching). This approach is considered to have a positive impact on generating quality EEG patterns as it enables participants to perform consistent MI. Furthermore, previous studies have reported significant differences in MI patterns depending on the force and speed of hand clenching (Fu et al., 2017; Geng and Li, 2020; Ortega et al., 2020). Here, we presented feedback to participants on the force and speed of hand and foot movements based on EMG signals (target: 50% of MVC) so that these parameters are stabilized across MI trials. If the force and movement speed are not controlled, the participant performs the MI task based on user characteristics. Movement was controlled in this study because it is a major factor that can affect the procedure for hypothesis testing.

Second, the SAO and MI modalities in hybrid BCI should enhance classification performance compared to that of the motor-cortex EEG only. SAO from the somatosensory cortex and MI from the motor cortex have distinctive neurophysiological origins (Yao et al., 2017). This study confirmed that both the somatosensory and motor cortices can be influenced by hybrid MI using tangible objects (i.e., a rough and hard ball), which leads to improved MI performance by generating quality EEG patterns. In the analysis of the feature covariance matrix, the SMI condition in the poor-performer group improved the connectivity among the 11 channels relative to the MI condition. These results support the idea that hybrid-modality imagery using tangible objects can generate quality EEG patterns. However, we found no significant changes in the good performer group. All participants underwent under the MI-BCI paradigm. The good-performer group was defined as participants who achieved high performance in the MI paradigm, and the poor performer group was defined as those who did not. According to the post-experiment interviews, the good performer group found the established MI method more familiar than SMI and found it difficult to imagine two modalities at once. The poor-performer group, however, found that SMI using a tangible object could help them to imagine more specifically and consistently than in the MI condition. In the classification results, the poor performer group showed better performance in the SMI condition than in the MI condition. The good performer group revealed no significant difference among the two paradigms. We thus believe that training methods to enable specific and consistent imagination can lead to the generation of quality EEG patterns and contribute to improving the performance of the poor performer who has previously reported difficulties with MI methods.

A limitation of this study is recruitment of only healthy subjects. MI-BCIs are commonly used in medical applications such as restoring motor functions in stroke patients. Because the stroke patients with loss of motor movement are unable to make voluntary movements, the established training process uses the passive movement (PM) by external devices instead of ME. A previous study reported that PM (i.e., the execution of a movement by an external agency without voluntary control) and MI induce similar EEG patterns over the motor cortex (Arvaneh et al., 2017). Many previous studies have tried using MI-based BCI control with passive motion for stroke patients, and significant positive results have been confirmed (Arvaneh et al., 2017; Cantillo-Negrete et al., 2018; Lu et al., 2020). Because a tangible object can be applied to a hand or foot when the patient performs a passive movement using an external agent, the proposed method may have a positive effect on stroke patients. This limitation requires further validation in future studies.

Previous research reported that a hybrid modality (MI and SAO with a vibration burst) achieved an average classification accuracy for classes 7.70% larger than did MI and 7.21% larger than did SAO (86.1% vs. 78.4% and 78.9%, respectively) (Yao et al., 2017). Another study showed that the hybrid modality group achieved an accuracy 11.13% higher than did the MI group and 10.45% higher than did the selective sensation group, using a vibration stimulus for the two classes (Yao et al., 2014). Moreover, prior studies have reported that hybrid modalities using somatosensory stimuli can improve MI-BCI performance. These studies applied MI and SAO stimuli to both hands (i.e., L-MI, R-SAO, R-MI, L-SAO). However, our study differs in that participants were instructed to simultaneously imagine MI and SAO stimuli on the hands and feet. Ours is an intuitive approach in which movements and sensations based on tangible objects (somatosensory signals) can be simultaneously imagined in the motor imagery procedure. The past studies did not employ intuitive imagining of two modalities simultaneously because vibration stimuli and motor actions are independent. However, the proposed method of this study allows intuitive imagining of two modalities because the motor action and the somatosensory stimulus (i.e., a tangible object) are dependent. For example, the users are able to imagine feeling the sense of a tangible object in the hand when their hand is grasping. The proposed hybrid imagery is consistent with the human mental model, and intuitive processes designed according to this mental model show high efficiency in behavior or action (Young, 2008). Our work demonstrated relatively good performance compared to previous studies, and can contribute to improving the performance of poorly performing users. This study has the limitation that the small sample size was too small to allow generalization of the results, requiring further validation in future studies with a larger sample.

This study compared performance in cross-modality conditions such as MMM, SSS, SMM, SSM, SMS, MSM, MMS, and MSS, involving mixtures of MI (“M”) and SMI (“S”) conditions on different body parts. (The order employed in the above three-letter codes is left hand, right hand, right foot.) In the poor performer group, the SMS cross-modality had the best performance, which was 21.76% larger than in MMM and 11.04% larger than in SSS (78.9%). The conditions with different modalities on the two hands (i.e., SMS, MSM, and MSS) had higher performance than did conditions with the same modality on the hands (i.e., MMM, SSS, SSM, and MMS). We believe that cross-modality combinations of MI and SMI help users to distinguish their hands better in imagination than do single modalities. A previous study reported that the cross-modalities of left-MI, right-SAO (84.1%), and right-MI and left-SAO (84.9%) showed a higher performance than having the same modality on the two hands, i.e., left-MI and right-MI (78.4%) and left-SAO and right-SAO (78.9%) (Yao et al., 2017). MI and SMI were conducted in two separate sessions with different trainings in mind. Thus, the results obtained from combining the experimental data of the two separate sessions may differ from the experimental results of a single session. The purpose of this study is to confirm the possibility of cross-modality, which will be verified through future research. Future studies are needed on cross-modality imagery using various somatosensory stimuli, such as tangible objects that feel rough and hard, rough and soft, or smooth and soft, or with vibration and electric stimuli.

In addition, this work placed the arrow on the sides of the display indicating the cue for the MI task. The arrows could be placed closer to the center of the screen rather than on the sides, to reduce eye movement and eye movement artifacts in the EEG. The neuro-feedback is also presented on the side screen, which can cause the EEG artifacts by eye movement. These issues need to be improved to minimize the effect of the artifacts in further study. In conclusion, this study proposes a unique hybrid-MI using a somatosensory stimulus to improve the MI-BCI performance of poorly performing users. The hybrid modality enabled consistent MI and activation of both the somatosensory and motor cortex using a rough and hard ball as the stimulus (SAO). This hybrid modality led to a significantly improved three-class MI-BCI performance in poor performers. In addition, we here confirmed that the cross-modality combination of MI and SMI performed better than did a single modality. However, only one type of somatosensory stimulus was assessed in this study. We believe that combinations of the different types of somatosensory stimuli alluded to above may significantly improve BCI performance and should be further investigated in future. The hybrid modality proposed in this study can help poor performers improve MI-BCI performance and BCI literacy, which can contribute to the practical use and uptake of BCI.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

This study was approved by the Ethics Committee of the Korea Institute of Science Technology in Seoul, South Korea (Approval No. 2020-021). The patients/participants provided their written informed consent to participate in this study.

SP: conceptualization, methodology, the experiment, and writing—original draft. JH: investigation, visualization, data analysis, and the experiment. D-HK: data analysis, data curation, and the experiment. LK: conceptualization, writing—review, and editing, and supervision. All authors contributed to the article and approved the submitted version.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.