All brain signals were non-invasively recorded. A custom 3D printed (formlabs Form2, Formlabs Inc., Somerville, MA 02143, USA) holder made of flexible resin (formlabs RS-F2-FLGR-02) was used to align the fNIRS and EEG sensors to approximately target similar cortical areas (Figure 1B). The sensor layout was configured to result in 12 hybrid EEG-fNIRS recording locations per hemisphere. These locations were homogeneously spread with a 3 cm separation creating a grid. Each hemispherical grid was centered around the corresponding 10-20 system C3 and C4 location.

fNIRS signals were recorded using a NIRScout system (NIRx Medizintechnik GmbH, Berlin, Germany). We used a total of 12 optodes per hemisphere (10 sources and eight detectors in total) sampling at 12.5 Hz. An optode is a source-detector pair 3 cm apart from each other (allowing light to reach an approximate 1.5 cm depth into the skull). fNIRS sources and sensors were laid out to result in 12 optodes. The sources and sensors were symmetrically laid around C3 and C4 positions according to the International 10-20 system leaving an inter-optode distance of 3 cm (Figure 1B). Two wavelengths (wl₁ = 760 nm, wl₂ = 850 nm) continuous functional near-infrared spectroscopy (fNIRS) was used to obtain optical absorption densities that were transformed to oxy-hemoglobin [HbO] and deoxy-hemoglobin concentrations [HbR] using the modified Beer-Lambert Law (Cope et al., 1988). The raw optical densities are also provided in the data set.

An ActiChamp amplifier (BrainProducts, Berlin, Germany) operating at 4 kHz (running software BrainVision, v1.20.0801) was used to record EEG. Twelve EEG sensors per hemisphere were placed in between each sensor-detector fNIRS pair overlaying the region measured by that optode (Figure 1B). The reference in our setup corresponds to the standard Cz 10-20 position (Nomenclature, 1991; Klem et al., 1999). The signals were down-pass filtered and downsampled to 1 kHz in the data set.

To enable EEG and fNIRS sensors to record cortical activity from the same cortical locations, we used a non-standard sensor arrangement covering the bilateral motor cortex (Figure 1B). We used a custom sensor holder 3D printed in flexible resin that for every recording channel allowed the EEG sensor to sit on top of the cortical area targeted by a corresponding fNIRS source-detector pair. Namely, for every fNIRS source-detector pair, an EEG electrode was placed in between. Each holder consisted of a 5 × 5 grid of circular holes whose centers were 1.5 cm apart allowing the required 3 cm separation between fNIRS source-detector pairs with an EEG sensor occupying a hole in between. The positions in the 5 × 5 grid marked in red in Figure 1B correspond to the physical location of EEG electrodes and the approximate recording areas of EEG electrodes and fNIRS source-detector pairs. Physical locations of fNIRS sources and detectors are marked, respectively by “S” and “D” in Figure 1B. A total of 12 recording sites were used per hemisphere due to the limitation of space to fit the multi-modal sensors together. A holder was placed on each hemisphere and held in position using elastics. The disposition of every sensing point is symmetrical to the scalp mid-line, and each grid is centered in the corresponding C3 or C4 site depending on the side so that the central hole overlaid the respective central position. C3 and C4 were located per subject following the 10-20 standard (Nomenclature, 1991; Klem et al., 1999), i.e., at a 20% of the distance between the pre-auricular points passing over the top of the head from the mid-line. The spherical coordinates of the standard positions are provided in the data set.

We recorded the grip force and muscular electrical activity to represent motor behavior during the task.

Bilateral bipolar surface electromyography (EMG) was recorded over the longitudinal axis (+ distal and − proximal) of the muscle belly of the flexor digitorum superficialis (4 kHz, on the Aux channels of the BrainVision ActiChamp) placed in the anterior and posterior forearm faces (Figure 1B). Before electrode placement, the skin was cleaned with abrasive pads and alcohol to eliminate dead skin cells and fat impact on electrical recording quality. EMG signals were down-pass filtered and downsampled to 1 kHz in the data set.

The dynamic gripping task was conducted using a continuously recorded grip force transducer (PowerLab 4/25T, ADInstruments, Castle Hill, Australia) sampling at 1 kHz. The signal was also used to provide real-time visual display feedback to the subject and the target force level they were asked to produce (Figure 1A). The Maximum Voluntary Contraction (MVC) force for each subject's hand was measured and computed using the same transducer. The recorded force signals, in Volts, were down-pass filtered and downsampled to 50 Hz in the published data set.

To complete the picture provided by the data, we recorded potential confounds in the brain and motor behavior signals of interest. The pulse and breathing rate have an impact at the body level on the concentration of hemoglobin and therefore can have an impact on brain and scalp levels of hemoglobin concentration. Whereas pulse is easily removed in the fNIRS analysis band (0.01–0.25 Hz), the breathing rate can overlap with it (Pinti et al., 2019). Sources of muscular electrical activity can spread to the EEG sensors and include eye muscles and skeletal muscles. Thus, we consider electrooculography (EOG), which also carries information on blinks, and breathing as pure confounds. However, EMG might also leak into the EEG sensors and can carry confound information at the same time it provides behavioral information.

The utils package only depends on the public python packages, h5py, numpy, scipy, scikitlearn, matplotlib, which need to be installed. The notebook presentation.ipynb contains a thorough explanation and examples of how the tools in utils can be used to process the data and depends on jupyter. A conda environment.yml file is provided with all dependencies to facilitate installation. All together make the data readily available to exploit, i.e., Data Readiness Level C (Lawrence, 2017).

The data set file structure follows a tree-like organization in three levels. In the first and third levels, the data set contains meta-data in string format that can be accessed via the attributes of the level. The first level is the data set itself and the shared attributes across subjects measures, e.g., sampling frequencies and units, and other information such as the channel grid disposition and a template of hybrid sensor spherical coordinates over the scalp. In the second level, the data set is organized in one group per subject indexed by their anonymized ID (i.e., 14 groups with keys A to N) and contain no attributes. In the third level, each group contains a subgroup for each measure (e.g., keys frc for force and eeg for EEG) containing the data in numeric format and an attribute called events containing the times at which a timestamp was received during the recording (e.g., “relax,” “left-hand,” “right-hand”) and the “begin” and “end” timestamps indicating the beginning and end of the recording session, also numeric. In particular, the events group contains a second numeric attribute MVC containing the maximum voluntary contraction value for each hand.

The following preprocessing was applied to the raw signals in the data set to obtain the brief analysis in Figure 2, which can be reproduced using the companion notebook. After preprocessing all signals, epochs were extracted from 5 s before the “Go” instruction to 25 s after.

After preprocessing, signals were correlated to characterize a “brain and behavior” or neurobehavioral correlation (EEG, fNIRS, EMG, and force in Figure 2D) and a “confounds” correlation structure (EEG, fNIRS, EOG, and breathing in Figure 2E). EEG and EMG underwent additional filtering and spectral density computation similarly to that used for the EMG plots. For EEG the mu (8–12 Hz), beta (12–30 Hz), and gamma (60–125 Hz) bands spectral densities were extracted. Then, signals were downsampled to the lowest sampling frequency in the data set, i.e., 12.5 Hz, and cropped between −1 and 1 s (“Go” cue at t = 0 s) to focus on the onset of motor activity in the case of the “brain and behavior” correlation and between −1 and 21 s for the “confounds.” The signals were finally normalized. For each hand condition, the “brain and behavior” correlation (Figure 2D) was computed over the appended observations corresponding to the t = [−1, 1] s crops for all combinations of the left and right hemisphere mu, beta, and gamma EEG power bands and HbO and HbR, the right- and left-arm EMG and force. For the brain signals, channels 4 and 16 (Figure 1B) were selected as representative of the corresponding hemisphere activity. A similar process was used for the “confounds” correlation for the t = [−1, 20] s. A significance level α = 0.001 is set and only significant correlation values are shown.

There is a very small correlation between brain signals and confounds (<0.03 in absolute values, not shown due to lack of significance α = 0.001, Figure 2E). This suggests that the confounds do not interfere with the recorded signals after applying the standard preprocessing pipeline to the raw data.

The key observations of our data can be found for the “brain and behavior” correlation (Figure 2D). For considering the interaction between brain signals (fNIRS and EEG) and task measure (force) we need to bear in mind that the time scales of fNIRS and EEG are very different (seconds vs. milliseconds). In the EEG domain, it is known that motor activity onsets are reflected in EEG power features. In our data, we observe significant strong anti-correlation (r ≈ −0.3, p < 0.001) between the EEG power in the mu band and the force which indicates that passing from a resting state (high mu power) to motor activity (low mu power) is properly captured by these spectral features when we look at a time window from −1 to 1 s around the Go cue. This decrease in power, also known as mu event-related desynchronization (mu-ERD), is due to the desynchronization of neuronal activity (Pfurtscheller et al., 2006), and it shows that the EEG is aligned with the force, helping to further confirm the validity of our data set.

However, when the correlation is computed focusing on the 20 s of the task (from 0 s before the Go cue to 20 s after), the EEG mu-ERD is not significant (p> 0.001). Only the beta band (12–30 Hz), known to be synchronized with motor activity (Kristeva-Feige et al., 2002), appears with a low level positive correlation (r ≈ 0.04−0.05, p < 0.001). The lack of mu-ERD and force correlation during the 20 s of continuous contraction/relaxations might be a consequence of the velocity at which the sequential contractions/relaxations were executed, not leaving enough time to the motor cortex to reach a synchronized equilibrium state before it was desynchronized again. Furthermore, this can also indicate that mu frequencies (8–12 Hz) are not fast enough to track this kind of subtle phase changes and a justification to develop more precise algorithms or feature extractors as suggested by Paek et al. (2019).

In the fNIRS domain ±1 s around the “Go” cue, we also observe a typical HbO/HbR anticorrelation (r ≈ 0.15, p < 0.001) (Jasdzewski et al., 2003; Huppert et al., 2006) in the structure with higher magnitudes present for the contralateral hemisphere to the hand used. The HbO/HbR anticorrelation is stronger in the right hemisphere for the left hand although also present in the left hemisphere, and it is stronger in the right hemisphere. This suggests that the dominant right hand may engage the left hemisphere while the non-dominant left hand engages both hemispheres with a preference for the contralateral one. HbO and HbR also show significant correlation with the force (r ≈ 0.2−0.3, p < 0.001). In particular, HbR, which is more specific than HbO (Hirth et al., 1997), also shows higher anticorrelation with the force for the contralateral hemisphere.

Finally, there is a strong significant correlation (r ≈ 0.7, p < 0.001) between the EMG power envelopes of the active hands and the force which supports the synchronization of the different devices used to record these measures.

This brief analysis aims to validate the data set and present some of its features. We encourage the community to develop algorithms to better understand the rich temporal relationships between brain signals spanning very different time scales and physical origins present in the data set with BCI and neuroimaging purposes.