Brain-computer interface paradigm is a set of specific mental tasks or external stimuli carefully selected/designed by the BCI developer to represent the user’s intentions under specific brain imaging techniques. The purpose of the BCI paradigm is to “write” the user’s intentions into brain signals, which represent or code the user’s intentions. It is expected that the brain imaging technology used can detect the neural coding of the user’s intentions, laying the foundation for subsequent “reading” or decoding of the user’s intentions (Chavarriaga et al., 2017; Wolpaw et al., 2020). It is worth noting that it is difficult for BCI to decode the arbitrary or random mental activity of the user, as well as the arbitrary or random external stimulus received by the user.

Specific mental tasks are implicit mental activities, such as motor imagery (MI), visual imagery, speech imagery, mental arithmetic, and reasoning; specific external stimuli are explicit attentional tasks, such as visual, auditory, and tactile stimuli. Specific features of brain signals are induced by mental tasks or external stimuli, and identify specific mental tasks and specific stimuli, which provide the basis for subsequent BCI decoding. Mental tasks or external stimuli correspond to specific brain functions and brain activities, which are closely related to specific brain regions and brain networks/brain circuits. Figure 1 illustrates the relationship between the BCI paradigm, specific brain functions, and structures. Special attention should be paid to the fact that BCI paradigms are usually discussed in the context of specific brain imaging techniques, which means that BCI paradigms are closely related to specific brain imaging techniques.

There have been several BCI paradigms so far, such as MI, steady-state visual evoked potential (SSVEP), and P300 paradigms. These paradigms have their advantages and disadvantages, and many researchers are still improving these paradigms. The innovative design of BCI paradigms is one of the critical contents of BCI research. To translate the designed BCI paradigms into practical applications, the principles for designing the BCI paradigm are based on user-centered design (Kübler et al., 2014; Chavarriaga et al., 2017; Lyu et al., 2023) and human factors engineering of BCI (Allison et al., 2012; Kübler et al., 2013, 2020; Liberati et al., 2015; Martin et al., 2018; Kübler, 2020; Ramsey and Millán José del, 2020; Wolpaw et al., 2020; Branco et al., 2021) to evaluate it. The principles for designing the BCI paradigm are proposed in Table 1.

Brain-computer interface neural coding refers to the coding of different user intentions into central neural signals under a specific BCI paradigm, which is characterized by distinguishable brain signal features. Brain signals with encoded intentions can be detected by specific brain imaging techniques, at last, user intentions can be recognized by BCI neural decoding algorithms. The process of BCI neural coding is shown in Figure 2.

It is worth noting that BCI neural coding is closely related to the chosen level and parameter settings of brain imaging technology, such that the level of technology used to record the ECoG signals and the hardware settings (including the sampling frequency and hardware filters) affect the types of features that can be extracted and analyses that can be performed (Hill et al., 2012; Shirhatti et al., 2016).

Modeling BCI neural coding requires consideration of specific BCI paradigms, mechanisms of brain neural coding, neural signal features collected by different brain imaging techniques, and efficient decoding of the user’s intentions.

Brain-computer interface paradigm tasks are usually designed first, then the neural coding under the BCI paradigm is revealed, followed by the extraction of brain signal features from neural coding laws, and finally the neural decoding. The BCI paradigm with BCI neural coding is the basis or premise of BCI decoding. It should be emphasized that in BCI systems, there is no corresponding neural coding without the BCI paradigm, no neural decoding without BCI neural coding, or no high-performance neural decoding performance without good BCI paradigms and neural coding. Figure 3 illustrates the relationship between BCI paradigm, BCI neural coding, and BCI neural decoding.

Brain neural coding is the basis of BCI neural coding, which characterizes the hypothesized relationships between external stimuli / mental tasks and the responses of specific neurons or populations of neurons (Johnson, 2000; Brown et al., 2004). According to the theory that sensory and other information is represented in the brain by a network of neurons, it is believed that neurons can encode both digital and analog information (Thorpe, 1990). Computer information coding is the process of converting information from one form or format to another and can be used to represent the relationship of things, which can be represented by numbers, letters, special symbols, or combinations of them, to convert data into codes or coded characters that can be translated into the original data form. Figure 4 illustrates the relationship between BCI neural coding, brain neural coding, and computer information coding.

Inspired by the brain neural coding model in Figure 4, BCI neural coding models can be proposed, as shown in Table 2, and will be reviewed in sections 3.5–3.11. These BCI neural codes will eventually be transformed into computer information coding to be processed by computers.

Frequency/rate coding for intracortical BCI can be inspired by traditional models of neuronal firing rate coding. In most sensory systems, the firing rate increases, generally non-linearly, with increasing stimulus intensity (Kandel et al., 2000). Since the sequence of action potentials generated by a given stimulus varies from trial to trial, neuronal responses are typically treated statistically or probabilistically. The spike count rate for a single trial can be calculated from Equation (1) (Gerstner and Kistler, 2002).

The D_trial is the duration of a trial, with typical values of 100 ms or 500 ms (Gerstner and Kistler, 2002), and N_spikes is the number of spikes occurring within the D_trial.

The time-dependent rate of issuance in a time-dependent stimulus can be calculated from Equation (2) (Gerstner and Kistler, 2002).

where N_K is the number spike appearing on all repeated trials from time t and t+Δt, K is the number of retrials, t is the start time relative to the stimulus sequence, and Δt is the time interval, usually in the range of a millisecond or a few milliseconds. The FR_tdapplies to both resting and time-dependent stimuli, but it is unlikely to be the coding scheme used by neurons in the brain (Gerstner and Kistler, 2002).

Intracortical BCI time coding can be inspired by time coding models in neural coding (Butts et al., 2007; Gollisch and Meister, 2008). The time resolution of neural coding is on the millisecond time scale, suggesting that precise spike timing is an important element in neural coding (Theunissen and Miller, 1995). For example, the ability of many organisms to discriminate between stimuli (such as visual stimuli, auditory stimuli, tactile stimuli, gustatory stimuli, and olfactory stimuli) on a millisecond time scale suggests that time coding is also a model that functions in sensory systems (Gollisch and Meister, 2008).

The spiking activity features that can be extracted by time coding are time-to-first-spike after the stimulus onset (Victor, 2005; Carleton et al., 2010), phase-of-firing for background oscillations, features based on the second and higher statistical moments of the ISI probability distribution (Kostal et al., 2007), spike randomness, or precisely timed groups of spikes (Thorpe, 1990; Jolivet et al., 2006; Chen et al., 2009).

Information on the time structure of stimulus-evoked spike trains or dispensing rates is determined by the dynamics of the stimulus, the properties of the stimulus, and the nature of the neural coding process (Montemurro et al., 2008). In Equation (2) the code is rate coding if FR_tdchanges slowly with time and time coding if it changes rapidly.

Intracortical BCI phase coding can be inspired by phase-of-firing coding models in neural coding (Havenith et al., 2011). Phase-of-firing coding is a neural coding scheme that this type of code takes into account the spike count coding and a time label for each spike according to a time reference based on the phase of local ongoing oscillations at low (Montemurro et al., 2008) or high frequencies (Fries et al., 2007; Havenith et al., 2011). The feature of this code is that neurons adhere to a preferred order of spiking between a group of sensory neurons, resulting in a firing sequence (Havenith et al., 2011). For example, each neuron in the visual cortex has its own preferred/preferred relative firing time during the gamma oscillation cycle.

Intracortical BCI neuron population coding can be inspired by the joint activity of multiple neurons to characterize stimuli or mental activity. In BCI intracortical neuronal population coding, each neuron has a distribution of responses over some set of inputs, and the responses of many neurons may be combined to determine corresponding stimulus or mental activity about the inputs. Neuronal population coding captures the essential features of neural coding (Wu et al., 2002).

Experimental studies have revealed that this coding paradigm is widely used in the sensor and motor areas of the brain. for example, Neurons are modulated to the moving/motor direction in the visual area medial time (Maunsell and Van Essen, 1983). However, Individual neurons fire fastest or slower in one direction depending on the distance of the target from the neuron’s “preferred” direction, the vectors of all neurons, and the coding of motion signals.

Typical neuronal population coding involves neurons with Gaussian modulation curves whose mean value varies linearly with stimulus intensity. Positional coding in a population of neurons can be used to encode continuous variables such as joint position, eye position, color, or sound frequency. Rate coding of the entire population ensures higher fidelity and accuracy than rate coding of individual neurons (Mathis et al., 2012).

Intracortical BCI correlation coding can be inspired by the correlation coding model of neuronal firing (Panzeri et al., 1999). This model suggests that correlations between action potentials or “spikes” within a spike sequence may carry additional information beyond the simple timing of the spikes (Ahmadi et al., 2021; Zhang and Constandinou, 2023). Correlations may also carry information that is not present in the average firing rate of the two pairs of neurons (Decharms and Merzenich, 1996).

Intracortical biometric sparse coding can be inspired by the sparse coding model of neural coding, in which each activity of a stimulus or mental activity is encoded by the strong activation of a relatively small set of neurons, which do not use the full set of available neurons, but rather a subset of them. In the BCI decoding stage, algorithms for sparse signal representation and processing can be used.

The fact that only a few neurons in a population of neurons respond to a given stimulus and that each neuron responds to only a few stimuli out of all possible stimuli, may be a biological selective response. Theoretical studies of sparse distributed memory have shown that sparse coding increases the capacity of associative memory by reducing the overlap between representations (Kanerva, 1988). Experimentally, sparse representations of sensory information have been observed in several systems, including visual (Vision, 2000), auditory (Hromádka et al., 2008), tactile (Crochet et al., 2011), and olfactory (Ito et al., 2008).

Sparsity may be focused on time sparsity, for example, the features extracted from all frequency bands during motion imagery are not all well separable (Olshausen and Field, 1996); It may also focus on spatial sparsity, as in the sparsity of neuronal populations activated, or the sparsity of brain regions/brain networks activated, such that it is the sensory-motor-related brain regions or brain networks that are predominantly activated during MI.

Most sparse coding models are based on linear generative models (Rehn and Sommer, 2007), as shown in Equation (3) (Lee et al., 2006).

where ξ→⁢ϵ⁢ℝk is the set of k-dimensional real input vectors, bj→⁢ϵ⁢ℝk is the set of n k-dimensional basis vectors, and s→=(s1,s2,…,sj,…,sn)⁢ϵ⁢ℝn is the sparse n-dimensional vectors and s_j is the weight of each input vector b_j are combined linearly by Eq. (3) to approximate the inputs. Other models are based on match tracking and dictionary learning (Pati et al., 1993; Davis et al., 1994; Needell and Tropp, 2009).

To better or more fully characterize the relationship between the stimulus / mental activity and the neural response, a combination of several neural coding methods can be considered, which is a hybrid neural coding approach. To more accurately decode the stimulus or mental activity from the neural response, intracortical BCI hybrid coding schemes can combine two or more of the above models (Choi et al., 2017; Mussi and Adams, 2022). For example, global features such as pitch or formant transition profiles can be represented by both rate coding and place coding (Miller and Sachs, 1983).

The wrapping of intracortical electrodes has a significant effect on the collection of individual neuronal Spikes but not on LFP (Milekovic et al., 2018, 2019). LFP is expected to be used for long-term cortical control of an artificial device and is a low-frequency signal (<250 Hz) consisting of the sum of all the electrical activity in the region adjacent to the tip of the electrodes implanted in the cortex.

To date, most LFP-based BCI research has been carried out based on typical center-out arm movements (Rickert et al., 2005; Heldman et al., 2006; Wang, 2006), and some studies have employed point-to-point motor tasks (Ahmadi et al., 2021; Zhang and Constandinou, 2023). It has been shown that the M1 tune encodes movement direction (Georgopoulos et al., 1986) and velocity. There is a modest decrease in β-LFP amplitude beforehand movement and a large increase in HF-LFP spectral amplitude during hand movement. The time-domain characterization of LFP can be used to control computer cursors (Kennedy et al., 2004) and to encode motion parameters (Rickert et al., 2005; Ahmadi et al., 2021). Examples of the main existing LFP-BCI paradigms with neural coding studies are shown in Supplementary Table 1.

Electroencephalogram is the recording of the overall activity of a large population of neurons in a localized area by electrodes placed on the supernatural or subdural cortex, with time and spatial resolution of a few milliseconds and millimeters (ECoG is superior to MEG and EEG) (Katzner et al., 2009; Dubey and Ray, 2019), which is less affected by muscle activity and ocular artifacts (Ball et al., 2009), The ECoG has an excellent noise ratio. These advantages favor the coding of stimuli or mental tasks so that potential brain signal features are found to be well discriminated, Therefore, the brain imaging technique of ECoG is better suited for BCI.

The ECoG-BCI frequency coding model is the ECoG power spectrum associated with a specific event (stimulus or mental task), and studies have shown that the selection of appropriate electrodes and power can encode movement trajectories (Jang et al., 2022), and its temporal coding model is the peaks of the raw ECoG signals when time-locked to a specific event (a specific amount of time after stimulus presentation). It has been shown that ECoG high-frequency broadband (200–300 Hz) power variations carry a great deal of information about brain function and are coding information of robustness (Hermes et al., 2011, 2012; Siero et al., 2013, 2014; Vansteensel et al., 2016). In addition, visual, auditory, and tactile evoked ECoG potentials (Brunner et al., 2011; Hermes et al., 2015, 2017; Wittevrongel et al., 2018) and ECoG narrow-band (α, β, and γ) power variations (Crone et al., 1998; Pfurtscheller et al., 2003a; Miller et al., 2007; Kubánek et al., 2009; Brunner et al., 2011; Gunduz et al., 2012; Hermes et al., 2012, 2014; Burke et al., 2013, 2015; Brinkman et al., 2014, 2016) can characterize the function of specific brain regions and brain circuits, Their combination with high-frequency broad-band (Miller et al., 2009, 2016) power changes can sometimes improve decoding performance. Examples of the main existing ECoG-BCI paradigms with neural coding studies are shown in Supplementary Table 2.

Functional magnetic resonance imaging (Ogawa et al., 1990) has high spatial resolution (Logothetis et al., 2001), better robustness and user-friendliness, and individualized flexibility. The ability of fMRI to measure deep brain region structure and activity, map functional connectivity networks, and allow the use of amygdala and ventral striatum with BCI neurofeedback for user (Mehler et al., 2018) has become a core technique for mapping neuroplasticity (Seitz, 2010).

Spatial localization of brain functions using the fMRI-BCI space coding model to produce spatially distinct patterns of brain activation by engaging different combinations of brain regions during the time that subjects are receiving external stimuli or intentionally performing different mental activities (Yoo et al., 2004; Boly et al., 2007; Monti et al., 2010; Senden et al., 2019). Models of fMRI-BCI time coding reliably detect the onset, offset, and duration of single-trial fMRI responses evoked by various mental activities, which is the assignment of specific coding intervals for specific intentions (Monti et al., 2010). Models of fMRI-BCI amplitude coding using different fMRI signal levels within specific brain regions to encode different movement intentions (Yoo et al., 2001). The combination of fMRI-BCI hybrid coding models using the above signal features (spatial, temporal, and amplitude) can increase the degree of freedom to encode different units of information or increase the distinguishability of the evoked brain activation patterns, thus maximizing the decoding accuracy (Sorger et al., 2012). Examples of the main existing fMRI -BCI paradigms with neural coding studies are shown in Supplementary Table 3.

Functional near-infrared spectroscopy measures the hemodynamic response of brain tissue during resting state (Abdalmalak et al., 2021; Paulmurugan et al., 2021; Eastmond et al., 2022), external stimulus, and mental activity, including changes in oxy-hemoglobin (HbO) and deoxyhemoglobin (HbR) concentration, with the main advantage of good portability, tolerating a certain degree of head movement of the subject, and a favorable ecological effect. fNIRS-BCI can also be used to facilitate the rehabilitation of patients with motor dysfunction and/or cognitive dysfunction, such as those suffering from stroke and spinal cord injury.

The fNIRS-BCI time coding model extracts time-domain features of hemodynamic responses associated with a specific event (such as external stimulus or mental task), for example, the mean, peak, and variance of HbO and HbR (Holper and Wolf, 2011; Hwang et al., 2014; Naseer et al., 2014; Hong et al., 2015). Models of fNIRS -BCI amplitude coding using different fNIRS signal levels within specific brain regions to encode different movement intentions (Coyle et al., 2007; Kaiser et al., 2014). Examples of the main existing fNIRS -BCI paradigms with neural coding studies are shown in Supplementary Table 4.

Magnetoencephalography is a non-invasive neuroimaging technique for detecting weak magnetic field changes generated by the electrical activity of central neurons (Xu et al., 2022; Bu et al., 2023). The technique has high time (less than 1 ms) and spatial [2–5 μm (Cetin and Temurtas, 2021)] resolution and low sensitivity to artifacts generated by muscle activity, but its comfort, aesthetics, and ease of use leave much to be desired.

The MEG-BCI time coding model characterizes mental tasks or external stimuli by information such as the peaks of the time-domain waveforms of the MEG signals as well as the time points. The frequency coding model characterizes a specific event (mental activity or external stimulus) by the MEG power spectral features (Mellinger et al., 2007; Chen and Bai, 2009; Halme and Parkkonen, 2016). MEG signals are complex non-linear and non-stationary signals, which the single time or frequency coding model will lose some feature information, and a time-frequency coding model can be used to obtain the relationship between signal frequency over time (Chholak et al., 2019). MEG-BCI space coding models can be used to downsize the data using spatial filtering methods to differentiate between mental activities or external stimuli (Rathee et al., 2021). Examples of the main existing MEG-BCI paradigms with neural coding studies are shown in Supplementary Table 5.

A hybrid Brain-Computer Interface (hBCI) aims to improve the usability or efficacy of BCI systems hBCI consists of a mix of a BCI system (the main system) and an add-on system that assists the BCI (Li et al., 2019). which can be a non-external stimulus or non-psychological task-driven system, or an AI system (e.g., a deep learning-based machine vision or computer vision system) can be mixed to improve the accuracy of target recognition by the main system BCI and increase the number of brain-controlled/other commands. As shown in Figure 5.

As can be seen from Figure 5, BCI paradigms of the main system can be subsets of different external stimuli and mental tasks. For example, the P300 paradigm can be designed by visual, auditory, tactile, or olfactory stimuli according to the Oddball paradigm. The P300 paradigm, the SSVEP paradigm, and the kinesthetic imagery paradigm can be combined. BCI paradigms of the main system induce electromagnetic/metabolic activity signals related to brain activity, and multiple brain activity patterns in these brain signals can encode external stimuli and mental tasks. Non-external stimuli and mental tasks such as eye movement, limb movement, or heartbeat in the additional system can be characterized by other non-brain activity physiological signal patterns. The AI system in the add-on system can enhance the level of intelligence of the main BCI system.

The end user of BCI is the user, while the user itself is the source of the CNS signals that drive the BCI. The user is the most complex, active, and highly adaptive subsystem essential to the BCI system. Therefore, BCI systems are the most typical human-in-the-loop systems (the human brain is directly connected or coupled to the machine, a closed-loop system with direct brain-machine interaction), and the design and evaluation (usability and satisfaction) of BCI paradigms and coding need to be user-centered, taking into account BCI human factors engineering (Lyu et al., 2023).

Brain-computer interface paradigms and neural coding are closely related to the neural mechanisms of the user’s mental activities/tasks (Liberati et al., 2015). The performance of a BCI system (such as effectiveness and efficiency) is closely related to the user’s mental activity, such that the performance of a movement-imagery BCI system is largely dependent on the user’s effectiveness or ability to perform MI (Kübler et al., 2014; Martin et al., 2018).

It is worth noting that to evaluate the first principle of BCI paradigm design proposed in section 2.2, which states that CNS signals evoked by BCI paradigm specific tasks should have good separability, any innovatively designed BCI paradigm, and neural coding model typically requires offline data analysis and model establishment, and ultimately must be validated and evaluated by neural decoding in an online BCI system.

The BCI paradigm, from a perspective of communication principles and technology, is a coding protocol in which brain intentions are encoded into signals generated by neural activity through specific external stimuli or mental tasks.

So far, BCI has been developed for more than 50 years. However, current BCI paradigms are more limited, and the transformation faces great challenges, which need to significantly improve, we need to break through the traditional classical BCI paradigms (such as SSVEP-BCI, P300-BCI, and MI-BCI), and add new BCI paradigms that are more natural and more effective to interact with the user. In recent years, many important advances have been made in the innovation of BCI paradigms (Willett et al., 2021, 2023; Metzger et al., 2023).

The performances of the BCI paradigm and neural coding are directly related to the level of brain signal collection technology, which requires a breaking through of brain signal collection technology. How brain signals are acquired is crucial for the BCI paradigm and neural coding, which is related to the quality of the collected signals and the final BCI control effect. With the continuous development of micro-nano processing technology and electrode materials, electrodes for invasive BCI tend to be flexible, miniaturized, high-throughput, and integrated. Currently, the research and development of hydrogel EEG electrodes are more active (Xue et al., 2023), stereotactic EEG (sEEG) (Herff et al., 2020), and in-ear EEG electrodes (Wang et al., 2023) have also made positive progress. In addition, minimally invasive endovascular stent-electrode techniques (Oxley et al., 2016, 2021), minimally invasive local-skull electrophysiological modification methods (Sun et al., 2022), and other schemes have been proposed to innovate brain signal acquisition.

Currently, classical machine learning still has an advantage in BCI neural decoding, but deep learning also has the potential to enhance BCI decoding performance. Some studies have introduced suitable deep learning algorithms in decoding brain signals, and these studies show that BCI technology combined with advanced AI techniques is expected to significantly improve brain signal decoding performance (Willett et al., 2021, 2023; Metzger et al., 2023). Figure 6 illustrates the introduction of AI into BCI to improve the intelligence of BCI and facilitate the clinical translational application of BCI.