Based on the ADS1299 chip from Texas Instruments, we designed an eight-channel high-precision signal amplifier with a sampling rate of 250 Hz. The frequency passband of the amplifier is from 0.15 Hz to 200 Hz for simultaneous acquisition of EEG and EOG. Based on the standard position in the 10–20 system, the forehead (Fpz) and left mastoid electrode (A1) were selected as the reference electrode and right leg drive electrode (Figure 1A), respectively, and five channels (PO3, PO4, Oz, O1, and O2) in the occipital region were selected to acquire EEG data. In this paper, a bipolar lead was used to acquire EOG data, and the acquisition electrodes were placed as shown in Figure 1B, with four electrodes placed on the vertical and horizontal axes of the eye for measuring eye movements. The ground electrode for EEG signal acquisition was used as the positive electrode for the vertical EOG signals.

Sinusoidal stimulation can not only effectively solve the problem of screen refresh rate limitation but also alleviate the problem of experimental fatigue in SSVEP (Jia et al., 2010). Furthermore, the phase information can enhance the discrimination of SSVEP at similar frequencies. Based on the above advantages, this paper adopts a periodic sinusoidal visual stimulation paradigm combining frequency and phase information to induce SSVEP (Chen et al., 2014, 2015b). As shown in Figure 2, the interface has a total of 20 buttons, and the interface is divided into five regions (up, down, left, right, and middle), and four buttons are evenly distributed in each region. Each button flashes at a different frequency (8–15.8Hz in 0.4 Hz intervals) and phase (0–1.5π in 0.5π intervals). The sinusoidal stimulus sequence Sti(n) used to induce SSVEP is as follows:

The stimulus sequence Sti(n) is from 0 to 1 (0 is black and 1 is white), which is modulated by frequency and phase, and FPS is the screen refresh rate. Where m is the index of the interface region, and Δf₁ (0.5 Hz) is the frequency interval of different regions; k is the index of the target in the same region; Δf₂ (2 Hz) is the frequency interval, and Δθ (0.5 π) is the phase interval of the same region.

Specifically, the designed paradigm of this paper includes two stages: the SSVEP stimulus stage and the eye movement stimulus stage. In the first stage of the SSVEP stimulus, 20 buttons start flashing simultaneously (for 1.1 s), and the subject stares at the button of his/her choice. The second stage is the eye movement stimulus. When the 1.1S sinusoidal stimulation ended, 16 buttons in four regions (top left, bottom left, top right, and bottom right) in the GUI will move in different directions. When the button starts moving, the subject continues to follow the target with his/her eyes, and the corresponding eye movement occurs. As shown in Figure 2, the left GUI shows the initial position at the beginning of the button movement, and the right GUI shows the position at the end of the button movement. The blue arrow indicates the direction of button movement. Each button moves at a constant speed, gradually moving from the initial position to the target position, with a total time of 0.3 s. The button stays for 0.2 s after reaching the target position. When the eye movement stimulus stage is completed, the position of each button is suddenly reset back to its initial position, which ensures that the initial position of each button remains the same. At the end of a single experiment the border color change of the characters analyzed by the system turns red for 1S. The method of using 20 buttons divided into five regions evenly distributed so that the frequency gap between two buttons in the same region is 2 Hz, and the phases are not the same. The design of button movement in the GUI can induce eye movements and group the buttons by analyzing the EOG signals.

Figure 3 illustrates the stimulus process during a representative trial of the online experiment, as well as an example of EEG and EOG signals processing and final decision method. Considering that the visual system has a response delay, the EEG data for the first 130 ms are discarded during SSVEP analysis (Di Russo and Spinelli, 1999). The eye movement signals will have obvious peaks or valleys when the eye movement is stimulated. Experiments have found that some peaks or valleys disappearing tails may occur during rest periods. In order to improve the accuracy of eye movement analysis, we collect EOG signals during stimulation time and rest time to analyze eye movement direction comprehensively. Specifically, each trial consisted of a 1.6 s stimulus and a 0.5 s rest period (the time from the end of a trial stimulus to the start of the next trial). The following paper will describe the signal processing and decision-making process in detail.

CCA is a multivariate statistical method that is used to analyze the correlation that exists between two sets of multidimensional variables. The CCA method can be used for SSVEP detection in multichannel EEG signals (Lin et al., 2006). Compared with the CCA method, the FBCCA method uses harmonic information more effectively when detecting SSVEP (Chen et al., 2015a). The FBCCA method consists of three main steps. Firstly, the input EEG data is decomposed into N sub-band components by a band-pass filter set. Then the CCA method is performed on each of the N sub-band components. Finally, the correlation coefficients of the N sub-band components are weighted and averaged to obtain the overall correlation coefficient value of each stimulus frequency, and the largest correlation coefficient value corresponding to the frequency is selected as the final recognition result. In this study, this paper evaluates the performance of both CCA and FBCCA methods for SSVEP identification. In this study, the performance of CCA and FBCCA methods for SSVEP recognition in online experiments is first compared, and then the offline analysis compares the classification results of FBCCA and CCA methods with different window lengths. In this paper, harmonic components of N sub-bands (N = 3 in this study) are extracted from the EEG signals X using Butterworth infinite impulse response (IIR) bandpass filters. These sub-band filters have the same upper bound frequency (77 Hz) but different lower bound frequencies. For the nth sub-band component X_n, the lower bound frequency is n×7 Hz.

At the end of the SSVEP stimulus, the buttons in the four regions will move, and the subjects continue to follow the selected target with their eyes, and the corresponding eye movements occur. The EOG signals were recorded for 1S (250 sampling points) segments after the end of the SSVEP stimulus, and the EOG signals were filtered to the 1–10 Hz range using a 3rd-order Butterworth bandpass filter. As shown in Figure 4, the amplitude change of the channel signal corresponding to eye movement is greater than the amplitude variation of the channel signal corresponding to no eye movement (Barea et al., 2012). The root mean square (RMS) of the EOG vertical channel and horizontal channel was calculated by the following formula to determine which channel has eye movement:

H_rms is the RMS of the EOG horizontal channel; V_rms is the RMS of the EOG vertical channel; x_i is the amplitude of the EOG signals at the i-th sampling point of the corresponding channel; N are the number of sampling points. As shown in Figure 5, the EOG signals in the channel corresponding to the eye movement signal has obvious time-domain waveform features. The waveform features of EOG are calculated by the following formula:

H_d and V_d is the peak-to-peak values of the channels. H_t and V_t is the time of the peak minus the time of the trough of the channel.

The following three Criterions were used to make decisions about eye movement direction (Figure 6).

Criterion I: H_d < = thresholdv and V_d < = thresholdv, thresholdv is the threshold value of the peak-to-peak value.

Criterion II: HrmsVrms>=thresholdd or VrmsHrms>=thresholdd, thresholdd is the threshold value of the channel RMS difference.

As shown in Figure 6, if Criterion I is satisfied, no eye movement occurs, and the subject stares at the middle four buttons (the value of the index m of the interface region is 5). If Criterion I is not satisfied, continue to analyze the direction of eye movement by Criterion II. If Criterion II is satisfied, continue to judge the specific eye movement direction through H_t or V_t. If both Criterion I and Criterion II are not satisfied, the eye state cannot be determined.

This paper evaluates the online performance of SSVEP through CCA and FBCCA methods, and analyzes whether the eye move and the direction of movement through waveform analysis. Decisions on the final output were made using the following methods (Figure 7).

Case 1: When the eye movement features satisfy Criterion I or Criterion II, the eye state is effectively analyzed. The 20 stimulus targets were grouped and selected according to the results of no eye movement or the direction of specific eye movement. The template signal of CCA was composed of four stimulus frequencies of the grouped results.

Case 2: None of the eye movement features satisfies Criterion I or Criterion II, indicating that the eye state is not effectively analyzed. The template signals composed of all stimulus frequencies were directly selected for CCA analysis and FBCCA analysis.

The spelling system designed in this paper consisted of a laptop computer with an extended display, and the designed visual stimulation was presented on the extended display with a screen refresh rate of 60 Hz. Subjects were asked to sit in front of the laptop display, with the distance between the display and the subject kept at about 40 cm.

In this experiment, we need to test the online performance of the hybrid BCI system based on SSVEP and EOG-base eye movement proposed in this paper. Before each stimulus, the background color of the character to be selected changes to red at break time (0.5 s), which is the symbol to indicate which button is the target button. The order of selection is from the character "1" to "j." Each experiment contains six sets of stimuli, each requiring the subject to select each character in a specified order. He/she was required to gaze at the flashing character and then complete the specified eye-movement actions according to the GUI cues. In a complete trial, each character on the screen was selected six times (120 pre-specified characters).

We choose to use accuracy, ITR (bits/min) to evaluate the performance of the system. The ITR represents the amount of information output by the system per unit time and is calculated as follows:

Where M is the number of characters. P is the accuracy, and T is the average response time. Each experiment included 1.6 s stimulation and 0.5 s rest time, so T was fixed at 2.1 s.

The single-modality part is first analyzed, including the results of the EOG-based eye movement classification accuracy and the SSVEP classification accuracy for each experiment. In EOG eye movement recognition, the thresholdv is used to determine whether eye movements occur. We analyze the relationship between the number of incorrect judgments and the thresholdv. As shown in Figure 9A, the number of misjudgments first decreases and then increases as thresholdv increases. The role of the thresholdd is to determine the direction of eye movements. When the value of thresholdd is 1, corresponding to a simple hybrid system, the EOG signal must analyze the direction of eye movements to determine the region of the button. Figure 8 shows the recognition accuracy of EOG eye movements at this time, with results of 97.42, 97.33, 98.00, 92.00, and 96.67% (mean 96.28%). When the value of thresholdd is >1, the hybrid system uses the decision-making method based on SSVEP and EOG proposed in the previous section (Figure 7). The region of the button is only confirmed when the eye state is effectively analyzed. We analyze the relationship between the classification accuracy of the hybrid system and the thresholdd. As shown in Figure 9B, the classification accuracy of system first increases and then decrease as thresholdd increases. Based on the results of the above analysis, the value of thresholdv chosen in this paper is 84 and the value of thresholdd is 1.56. Figure 10 shows the SSVEP classification accuracy based on different methods.The above results show that each eye movement direction of the EOG and each frequency of the SSVEP can be well-identified, confirming the validity of the single modality identification parameter selection.

After EEG combined with EOG to form a hybrid BCI, compared with the single-modality BCI, the classification accuracy of each group was greatly improved (Figure 10). For those subjects with relatively low eye movement classification accuracy (< 94%), the classification accuracy can be further improved by using the decision-making method based on SSVEP and EOG proposed in this paper (Figure 11). The hybrid BCI system corresponds to the case where the value of thresholdd is 1. The hybrid BCI system using decision-making method based on SSVEP and EOG corresponds to the case where the value of thresholdd is 1.56. On average, all subjects performed satisfactorily with high accuracy of 94.75 ± 3.92% and relatively high ITR of 108.63 ± 8.91 bits/min.