Neurofeedback (NF) is a technique that consists in feeding back information to an individual about his/her brain activity in real-time in order for him/her to learn to better control an aspect of it (Hammond, 2011; Birbaumer et al., 2013). Specific changes in emotional, cognitive, or behavioral functions are expected to occur along with NF training, which makes NF a promising tool for brain rehabilitation of patients with neurological or psychiatric disorders (Birbaumer et al., 2009; Sulzer et al., 2013; Linden, 2014; Linden and Turner, 2016) and for peak performance training of healthy subjects (Vernon, 2005; Gruzelier, 2014a,b; Scharnowski and Weiskopf, 2015). From the advent of NF in the early sixties up to now, most NF studies have relied on the use of EEG for measuring the brain activity. While EEG is inexpensive and benefits from a high temporal resolution (order of the millisecond), it is particularly sensitive to noise and lacks specificity because of its low spatial resolution (order of the centimeter) and the fact that source localization from EEG suffers from an ill-posed inverse problem (Baillet et al., 2001; Grech et al., 2008). Combining EEG with other modalities could allow to extract richer information about the ongoing brain activity and therefore enable to develop more efficient NF protocols. In the recent years, fMRI, functional near-infrared spectroscopy (fNIRS), and magnetoencephalography (MEG) have started to be exploited for the purpose of NF (Sudre et al., 2011; Masahito Mihara et al., 2012; Sulzer et al., 2013; Thibault et al., 2016). Most noteworthy, the field of fMRI-NF has grown exponentially during the last 10 years (Sulzer et al., 2013) and contributed in reviving NF research. However, if fMRI-NF gives access to the self-regulation of deep brain regions at high spatial resolution (order or the millimeter), it is at the price of a low temporal resolution (order of the second). In the context of NF, combining EEG and fMRI enables to return to a subject two signals at the same time, one that contains temporally fine information about brain oscillations and one that contains spatially fine information about specific brain regions. The simultaneous combination of EEG-NF and fMRI-NF was introduced for the first time by Zotev et al. (2014) who made a proof-of-concept application of this new type of NF in the training of emotional self-regulation. In their pioneering work, the authors hypothesized that bimodal EEG-fMRI-NF could be more efficient than EEG-NF or fMRI-NF performed alone. However, to our knowledge, this hypothesis has not been studied so far.

As NF, brain computer interfaces (BCI) also rely on a closed-loop that exploits brain activity in real-time. Traditional BCIs, so called assistive BCIs, are intended for the purpose of control and communication of an external object, while NF is intended to allow an individual to acquire control over him/herself (Wyckoff and Birbaumer, 2014; Perronnet et al., 2016). This distinction has tended to fade since the development of restorative/rehabilitative BCIs which target brain rehabilitation applications like NF (Soekadar et al., 2011; Chaudhary et al., 2016; Perronnet et al., 2016). In the BCI community, the field of hybrid BCI has recently emerged (Pfurtscheller et al., 2010; Amiri et al., 2013), the term “hybrid” referring to a multimodal combination of sensors. A hybrid BCI is defined as the combination of two BCIs or of at least a BCI and another system such as another biofeedback system (like an electromyogram for example; Pfurtscheller et al., 2010). They can be designed to work simultaneously or sequentially. Their purpose is mostly to augment the usability and/or the performance of the BCI. Most of the hybrid BCIs combining two BCIs that have been proposed in the literature relied only on EEG paradigms, but some hybrid BCI combining EEG and fNIRS have also been proposed (Fazli et al., 2012; Buccino et al., 2016) and have shown enhanced performance. These encouraging results suggest that using hybrid approaches for NF could enhance the efficiency of unimodal approaches.

Protocols targeting motor imagery (MI) patterns are one of the most studied applications of NF and BCI research because of their potential for stroke rehabilitation (Sharma et al., 2006). To this day, such NF protocols have been explored with all existing NF modalities: EEG (Birbaumer et al., 2009; Shindo et al., 2011; Silvoni et al., 2011; Ramos-Murguialday et al., 2013; Pichiorri et al., 2015), fMRI (Sitaram et al., 2012; Liew et al., 2015; Linden and Turner, 2016), and even the most recent ones fNIRS (Mihara et al., 2013; Kober et al., 2014) and MEG (Foldes et al., 2015). A recent study by Zich et al. (2015) showed that MI-based EEG-NF allowed subjects not only to generate stronger EEG response at the motor electrodes, but also that the BOLD activity observed in the sensorimotor regions in simultaneous fMRI recordings was higher during MI with EEG-NF as compared with MI training alone. Interestingly enough, the study revealed that the contralateral activity in EEG and fMRI were correlated while the laterality patterns were not. The finding that EEG and fMRI signatures of MI are not redundant suggests a potential for bimodal EEG-fMRI-NF. To our knowledge no bimodal MI-based EEG-fMRI-NF protocol has already been proposed. The authors (Zich et al., 2015) particularly stressed the need of conducting an exhaustive comparison of unimodal and bimodal neurofeedback in order to understand the specific contribution of each modality: “only a systematic within-subject comparison using simultaneous EEG-fMRI data acquisition and providing fMRI-based feedback, EEG-based feedback and a hybrid feedback based on both modalities, would provide exact information about the validity of each recording modality.”

In the present work, we introduce an MI-based EEG-fMRI-NF protocol and compare its effects with EEG-NF and fMRI-NF by looking at the MI-related EEG and fMRI activity patterns elicited in each NF condition. Additionally, we introduce a hybrid EEG-fMRI-NF metaphor that integrates both EEG and fMRI signals simultaneously in a single bi-dimensional feedback. We assume that in this way, subjects are more likely to perceive the NF task as one regulation task instead of two and that this might relieve their cognitive load (Gaume et al., 2016).

For the first time, we compared the effects of unimodal EEG-NF and fMRI-NF with bimodal EEG-fMRI-NF in order to assess the potential added value of bimodal NF over unimodal NF. We tested our hypotheses (cf. Section Working Hypotheses) by looking at the level of MI-related EEG and fMRI activations during each NF conditions. Motor activations as revealed by post-hoc fMRI analysis were significantly higher during EEG-fMRI-NF than during EEG-NF (see Figure 6). This results partly validated our “direct NF effect” hypothesis and could mean that EEG-fMRI-NF specifically triggered more fMRI activations than EEG-NF because feedback from fMRI was provided. But it could also mean that bimodal EEG-fMRI-NF was more engaging than unimodal EEG-NF because subject had to control the feedback in the vertical and horizontal dimension. To disentangle whether EEG-fMRI-NF is more specific or simply more engaging than EEG-NF, one could use a one-dimensional EEG-fMRI-NF feedback that would mix both EEG and fMRI feature in a single gauge and compare it directly to EEG-NF. Alternatively, to rule out the engaging factor, one could also compare EEG-fMRI-NF with EEG-shamfMRI-NF in which sham fMRI-NF would be provided together with real EEG-NF. Post-hoc EEG activations did not show any significant differences between the different NF conditions. This can be due to the fact that EEG is noisier than fMRI, especially in the MR environment, but it is also possible that it was hard for participants to maintain the μ desynchronization throughout the 20 s of the NF blocks. The 20 s block design was chosen mainly in consideration of the fMRI modality in order to account for the hemodynamic delay. MI-based EEG-NF/-BCI tasks are usually much shorter, around 4 s length (Pfurtscheller and Neuper, 2001). The electrophysiology of continuous MI is still not fully understood. Though continuous MI is thought to induce a succession of ERDs it can be hard to observe a continuous desynchronization throughout the duration of the continuous MI (Jeon et al., 2011; Rimbert et al., 2015) This highlights the difficulty of designing the task specifically for bimodal EEG-fMRI-NF given the different spatio-temporal dynamics of EEG and fMRI. Interestingly, the specific effect of NF in the three NF conditions can be confirmed by the significant difference in the level of post-hoc fMRI and EEG motor activations between the NF runs and the MI_pre and MI_post runs which were done without NF (see Figures 6, 8). Despite the somehow limited number of subject in our study, these results support our “generalized NF effect” hypothesis. Further work with more subjects should be conducted to even enforce this outcome. In the seminal work on EEG-fMRI-NF (Zotev et al., 2014), the authors studied a protocol of positive emotion induction with feedback from frontal EEG asymmetry in the beta band and from left amygdala BOLD. As in this related work, we found similar value ranges of the EEG and fMRI features and similar variability. We were however not able to observe significant differences between the three NF conditions by directly looking at the EEG and fMRI laterality features (see Figures 5, 7). Lateralization of activity in motor regions is known to be an indicator of good motor imagery (Marchesotti et al., 2016). Also in stroke rehabilitation, best results are usually obtained when the recovery happens in the ipsi-lesional hemisphere rather than in the contra-lesional one and NF based on laterality indices could allow to promote this kind of recovery (Chiew et al., 2012; Rehme et al., 2012). However, laterality features are hard to interpret and may have been too hard to regulate significantly in a single session for participants who were not trained to MI before. Regarding the EEG laterality index and given the spatial proximity of the chosen electrode locations C1 and C2, one could wonder if they could be influenced by the same sources. Such sources would be situated in cortices close to the inter-hemispherical midline such as feet sensorimotor area. However, the activity of the hand sensorimotor area is quite far from the midline, so the activity measured by a contralateral electrode will be far stronger than that of an ipsilateral electrode. Given that the neurofeedback was based on a laterality index, there is no chance that activating common sources such as with feet imagination would allow to control the neurofeedback. However, we do admit that C3 and C4 are more common locations for hand movements and might lead to better results. One could also consider computing a CSP filter on calibration data which would allow to define the spatial filtering for the EEG feature at the individual level. Regarding the fMRI laterality index, the right motor ROI was defined approximately by mirroring the left motor ROI. This was done mainly in order not to add more time to the already long experimental protocol. Given the size of the ROI (18 × 18 × 12 mm³), there is high chance that the mirror ROI would lie in the right primary motor cortex. However, we admit that it would be better to use a functional localizer to define the right motor ROI.

Regarding the “compromise NF effect,” our results did not allow us to get any preliminary insight into our speculations. More experiments with longer NF training and more subjects are needed to confirm the rest of the “direct NF effect” and the “compromise NF effect” assumptions. We can note that in our study, the signal-to-noise ratio (SNR) was the same in unimodal and bimodal NF conditions as EEG and fMRI were simultaneously acquired throughout the whole experiments to assess the cross-modality effects. However, when doing unimodal EEG-NF or fMRI-NF without simultaneous EEG and fMRI recordings, SNR should be better than the one of bimodal EEG-fMRI-NF. This could reinforce the “compromise NF effect.” Artifacts occurring during simultaneous EEG-fMRI are a major limitation of EEG-fMRI-NF (Zotev et al., 2014). The BCG artifact and motion artifacts from the subject or the environment (vibrations from helium pump and ventilation) are particularly hard to correct. The development of new methods for correcting these artifacts is an ongoing topic of research, but few options are available for online correction (Allen et al., 1998, 2000; Krishnaswamy et al., 2016; Mayeli et al., 2016; Wu et al., 2016; van der Meer et al., 2016). Interestingly, a recent approach consists in using the EEG not only as a brain imaging modality but also as a motion sensor to correct for motion artifact (Jorge et al., 2015; Wong et al., 2016). Another important aspect of the EEG analysis is the choice of the reference. In this work we used the standard reference FCz as it was proven to be efficient for motor imagery (Choi et al., 2006). But regarding the fact that the potential of FCz is non-zero, it would be interesting in the future to consider using another reference such as the common average reference (CAR) or reference electrode standardization technique (REST; Yao, 2001).

Though the NF features change between the consecutive NF runs and between each pair of NF conditions was not significant, the EEG and fMRI laterality features had asymmetric tendency (see Figures 5, 7). Throughout the consecutive NF runs, EEG laterality tended to improve while fMRI laterality tended to worsen. Besides, participants reported on average that the fMRI dimension was easier to control than the EEG dimension, so it is possible that they have put more effort (however not necessarily more attention as they reported) on controlling the EEG dimension. This could explain the learning tendency observed on the EEG laterality feature at the price of a decrease on the fMRI laterality feature. Putting these observations together suggests that during bimodal NF, one feature could be more regulated than the other, possibly the one that is harder to control. We should note however that our study was conducted at a single-session level and that the asymmetric change of the features that we observed could actually be part of a learning scenario in which subjects would by example first learn to regulate one feature, then the other one and eventually manage to regulate both simultaneously. Interestingly, this decrease of performance on NF features was also observed in related works (Zotev et al., 2014) though both on EEG and fMRI features. This decrease of performance can also be explained as being part of the U-shaped learning curve (Carlucci and Case, 2013; Gaume et al., 2016): by trying new regulation strategies, the cognitive load of the subject suddenly increases and results in a temporary loss of performance. However, it is not yet known how this applies to bimodal NF. Our results thus open interesting questions on how subjects learn to regulate a bimodal NF and on how to define the EEG and fMRI features so that they are complementary enough. The assessment of this complementarity can be based on studies and methods investigating the coupling between BOLD and EEG signal (Formaggio et al., 2010; Yuan et al., 2010; Dong et al., 2014; Murta et al., 2015, 2016; Yin et al., 2016) which generally report that BOLD is negatively correlated with low-frequency EEG bands (α, β) and positively correlated with high-frequency EEG bands (γ). Besides these questions on the learning mechanisms and the inner definition of the features, our observations also raise the issue of whether the two NF signals should be made discriminable or not by the feedback metaphor. Indeed, if the subject was not able to discriminate between both signals, he/she might be less likely to control one signal more than the other.

Feedback design is an important aspect of a neurofeedback protocol and the optimal form of feedback for unimodal NF is still an ongoing topic of research (Cohen et al., 2016). Though the traditional thermometer metaphor (Sitaram et al., 2007) can appear boring for subjects, it has the advantage of being easy to understand. In their pioneering work, Zotev et al. (2014) have naturally extended the thermometer feedback to the bimodal NF case. We introduced a novel metaphor for EEG-fMRI-NF that integrates both signal into one single feedback in order for the subject to more easily perceive the bimodal NF task as one single regulation task. Though we did not compare our integrated metaphor with a non-integrated one, most of our participants reported that it felt like they had one task to do during bimodal NF. Having two separate feedbacks to control and thus two separate targets to achieve could increase the cognitive load, which is an important aspect of the NF process (Gaume et al., 2016). Integrating both NF signals in one single feedback can be a way to relieve the cognitive load of the subject. One of the difficulty in combining both NF signals in a single feedback is that EEG and fMRI do not have the same sampling rate. In the present study, the fact that the update rates of the EEG and fMRI dimensions were different might have been disturbing for the participants. Indeed, five participants found that fMRI update rate was slow (16 times slower than EEG). Bringing the EEG and fMRI update rates closer is therefore advisable for future experiments. However, for fMRI, the update rate is constrained by the TR, which cannot be brought much below 1 s. One way to prevent the subject from being disturbed by the different update rates of the two modalities could be to mix the two NF signals in a feedback that would not allow the subject to discriminate between the two signals, like a one-dimensional feedback. Besides the representative advantage of using an integrated feedback metaphor, we believe that it makes it possible to define a truly integrated NF target that would reward brain patterns defined from both modalities. There is different level of “integration” of EEG and fMRI data. In our study, we integrated the two neurofeedback signals in one feedback metaphor in order to provide a bimodal neurofeedback. A more advanced way to provide an integrated bimodal feedback could be to use EEG-fMRI integration methods (Jorge et al., 2013; Sulzer et al., 2013), such as fMRI-informed EEG analysis, EEG-informed fMRI analysis, or EEG-fMRI fusion. However, despite the wide range of existing methods, these methods are mostly designed for offline use and there is no prospect yet of doing this integration online. In the framework of EEG-fMRI-NF, one could benefit from using these methods offline to study the effects of neurofeedback, guide the choice of the NF features, learn priors for a reconstruction model, learn a predictive (Meir-Hasson et al., 2013) or a coupling model.

It is important to stress that in our experiment unimodal and bimodal NF targets were different. The EEG-fMRI-NF target was probably “harder” to reach than the EEG-NF or the fMRI-NF target, as subjects needed to regulate EEG and fMRI simultaneously to reach the target. Thus, by directly integrating the EEG and fMRI NF signals without any fancy fusion technique, brain patterns defined this way from both modalities should already be more specific than those defined from one modality alone. Future experiments involving more subjects and other cognitive tasks will allow to characterize more precisely how EEG and fMRI are modulated in different unimodal and bimodal NF conditions. Eventually, the use of offline EEG-fMRI integration techniques should help understand how to define bimodal EEG-fMRI-NF protocol that make the most of both modalities for therapeutic applications such as stroke, depression, and other psychiatric and neurological disorders.

We have found that MI-related hemodynamic and electrophysiological activity are modulated both during unimodal EEG-NF and fMRI-NF and during bimodal EEG-fMRI-NF. Notably, we found that MI-related hemodynamic activity was higher during EEG-fMRI-NF than during EEG-NF, unlike fMRI-NF. This result suggests that EEG-fMRI-NF could be more specific or more engaging than EEG-NF alone. We have also observed that during EEG-fMRI-NF one modality could be more regulated than the other, suggesting the existence of self-regulating processes that would be proper to bimodal NF training. Taken together, our results pave the way to novel combinations of EEG and fMRI modalities for more effective neurofeedback approaches.

This study was carried out in accordance with the recommendations of Comité de Protection des Personnes “Ouest V” Rennes with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the Comité de Protection des Personnes “Ouest V” Rennes.

LP, AL, CB, MM, EB, FL, and MC designed the study and the unimodal and bimodal neurofeedback protocols. MM developed the real-time EEG-fMRI neurofeedback platform. LP and MM conducted the experiments. LP performed the data analysis. LP wrote the majority of the manuscript. All the authors read, revised, and approved the final manuscript.

This work has received a French government support granted to the CominLabs excellence laboratory and managed by the National Research Agency in the “Investing for the Future” program under reference ANR-10-LABX-07-01. It was also financed by Brittany region under HEMISFER project, and the National Research Agency with the REBEL project and grant ANR-15-CE23-0013-01. MRI data acquisition was supported by the Neurinfo MRI research facility from the University of Rennes I. Neurinfo is granted by the European Union (FEDER), the French State, the Brittany Council, Rennes.