Recent advances in physiological signal measurement techniques enlighten the non-invasive closed-loop brain-machine interface (BMI) design. Such signals can be used to infer individuals' underlying cognitive brain states and regulate them via non-invasive interventions such as music (Fekri Azgomi et al., 2023). The human brain state regulation via musical stimuli would have profound applications in closed-loop BMI (CLBMI) (Ehrlich et al., 2019), neural rehabilitation (Ottonello et al., 2019; Salas et al., 2019), and cognitive impairment treatments (Ray and Mittelman, 2017). As an instance of such applications, the well-known Yerkes-Dodson law, a.k.a. the inverted-U law, explains that optimal cognitive performance can be achieved by setting the cognitive arousal within a moderate level (Yerkes, 1907; Yerkes and Dodson, 1908). This inspires us to design a pilot study involving a human-subject working memory experiment in the presence of arousing music stimuli. We record the multimodal physiological and behavioral signals to investigate the feasibility of regulating one's cognitive arousal and performance via background music with calming and exciting contents.

Several studies have explored how the introduction of music can impact cognitive functioning (Parshi et al., 2019; Khazaei et al., 2021). Multiple studies have used music to influence driving performance (Ünal et al., 2012, 2013). In Huang and Shih (2011) and Kuschpel et al. (2015), the positive effect of music on concentration and the effectiveness of using music to reduce cognitive stress in closed-loop systems have been shown. In our designed working memory experiment, two types of music are employed, namely, calming and exciting music. Notably, the calming and exciting music selected by participants such that the calming and exciting components are supposed to replicate the low and high arousing conditions, respectively. Here, we mainly focus on cognitive performance and arousal in the presence of calming and exciting music.

Cognitive performance is a hidden brain state that demonstrates the general performance of one's cognitive functions (Khazaei et al., 2024). The cognitive functions of humans can be divided into two groups, namely, basic cognitive functions and higher-level cognitive functions. Working memory is one of the basic cognitive functions that provides temporary storage and allows the manipulation of information (Baddeley, 1992). Here, the n-back task serves as a cognitive task of interest, which encourages working memory usage by inducing different cognitive loads (von Janczewski et al., 2021; Fekri Azgomi et al., 2023). Decoding the underlying cognitive performance in a continuous manner is one of the challenges in this paradigm, which can be addressed by employing informative data and applying decoding approaches. One of the most accessible and informative data in this context would be the behavioral data recorded during the cognitive task of interest. We consider the sequence of responses as well as the reaction time of participants as the available behavioral observation and quantify the cognitive performance using Bayesian filters within an expectation-maximization (EM) framework (Amin et al., 2021; Khazaei et al., 2021).

Another hidden brain state that may have impact on cognitive performance is the underlying arousal state. In particular, the amygdala plays a crucial role in connecting arousal to memory formation (McGaugh, 2004). Typically, the arousal state is linked to the degree of physiological alertness (Cudo et al., 2018), and variation in arousal is mainly due to the exogenous and endogenous stimulation, which can be accompanied by neural, hormonal, and other biochemical changes (Hobson and Lindsley, 1988). Specifically, previous research on the autonomic nervous system (ANS) inference presents the electrodermal activity (EDA) as an informative measurement of cognitive arousal (Wickramasuriya et al., 2018; Wickramasuriya and Faghih, 2020a). In particular, the skin conductance (SC) signal (a measure of EDA) reflects the sweat secretions process, which is firmly linked to the underlying ANS activity inside the brain (Amin and Faghih, 2022). The variation in skin conductivity can be used as a quantitative index of arousal during a cognitive task (Khazaei et al., 2021).

When it comes to designing a CLBMI architecture for human cognitive functions, it would be crucial to investigate how the environment and interactions affect the cognitive capacity of the human operator (Cain, 2007). To gain a better insight into BMI design procedures and analyze how different regions of the brain react to the stimuli, neuroimaging methods with relatively high spatial resolution can play a crucial role. These neuroimaging methods include but are not limited to magnetoencephalography (MEG), electroencephalography (EEG), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS) (Berka et al., 2007; Wendel et al., 2009; Baldwin and Penaranda, 2012; Fyshe et al., 2012). The fNIRS is a relatively new neuroimaging technique that can show significant spatiotemporal changes for memory tasks in the prefrontal cortex (PFC) (Hoshi et al., 2003). Also, the fNIRS has a higher spatial resolution than EEG, and it is practical for unobtrusive applications. In the designed experiment, we use the fNIRS and collect data from the prefrontal cortex (PFC) and occipital (OC) areas of the participant.

Multiple studies investigate the brain neural activity in the course of n-back tasks via fNIRS (Ayaz et al., 2007; Roy et al., 2013; Fishburn et al., 2014; Herff et al., 2014). In Ayaz et al. (2007) and Herff et al. (2014), the cognitive loads and hemodynamic responses association is considered, and the hemodynamic responses are classified accordingly. Another study considers the n-back tasks and presents the fNIRS sensitivity to cognitive load and transitioning from a resting state to a task (Fishburn et al., 2014). While the link between the cognitive load of n-back task and corresponding hemodynamic responses is studied vastly, the study of association between the continuous cognitive performance signal and fNIRS data in the presence of music is investigated relatively sparsely. Previous studies such as Meidenbauer et al. (2021) and Struckmann et al. (2021) have considered the distinctive measures of cognitive performance, like the number of correct/incorrect responses and reaction time, and investigated the hemodynamic response data association. While this set of measurements can be an informative index of performance, it merely presents a distinct measure of performance, and might not fully capture the underlying dynamic of the brain's cognitive state (Basu et al., 2023). In this research, we study the association between the oxygenated hemoglobin (HbO) concentration and the continuous performance state decoded from behavioral data. In particular, we employ a Bayesian-based performance decoder that accounts for the underlying dynamic of this brain state, and tracks its continuous trajectory using behavioral data. Also, we evaluate the energy of total hemoglobin (HbT) signal within each music session.

To summarize, the primary purpose of this pilot study is to provide a multimodal database and examine the viability of using music as an intervention. The database includes a diverse array of data encompassing multiple physiological measurements to shed light on the human-subject experiments using music intervention. The available signals include SC data, electrocardiogram (ECG), skin surface temperature, respiration, photoplethysmography (PPG), functional near-infrared spectroscopy (fNIRS), electromyogram (EMG), de-identified facial expression scores, sequence of correct/incorrect responses, and reaction time recorded during the n-back task. In this research, we focus on cognitive performance and arousal, and we present how the HbO concentration is correlated with cognitive performance. Also, the energy of the HbT signal is evaluated with respect to each music. We represent how the arousal and performance indices vary with respect to task difficulty and music type. Then, we discuss our findings and provide a summary of our approach, followed by future directions of this research.

Given the HAI boxplots in Figure 5, a higher variation in the arousal matrices with respect to the calming and exciting session is noted, while such variation can not be observed with respect to the task difficulty. The median values of HAI in 1-back and 3-back (1-back vs. 3-back) tasks do not diverge considerably. However, considering the music sessions (calming vs. exciting), the median values associated with the calming sessions do not fall within the range of exciting session boxes. According to the reported p-values and boxplots (Table 1), there is no significant difference between the HAI in calming session 1-back tasks and 3-back ones. In regards to HAI associated with exciting sessions, participants 3 and 4 are the only ones who depict a significant difference between the 1-back task and 3-back task data (Table 1). Considering the HAI with respect to the music, all the participants present significant differences when shifting from a calming to an exciting session (Table 1).

We evaluate the HPI in a similar manner, and as expected, we find that the median of HPI within the 1-back task trials is considerably higher than in 3-back task trials (Figure 6). Also, the HPI median values among all the participants are higher within the 3-back task trials during the exciting session compared to the calming session (Figure 6). The reported p−values in Table 2 depict the significant difference in HPI with respect to the task difficulty as well as music session in most of the cases.

Figure 7 presents the epochs of HbO concentration and performance across the task blocks for participant 2. As noted earlier, 16 specific brain regions from the left and right sides of the PFC and OC areas are considered. The correlation between the HbO epochs and performance epochs with respect to the types of the n-back task and the played music is illustrated. In particular, Figure 7A displays the correlation in the 1-back task blocks within the calming session; Figure 7B pays attention to the 3-back task blocks within the calming session; Figure 7C addresses the 1-back task blocks within the exciting session; Figure 7D considers the 3-back task blocks within the exciting session. The Pearson correlation coefficient (r) of performance and HbO concentration for each studied brain region is reported in a box next to each subplot. The highest HbO and performance positive correlation for participant 2 corresponds to the RF1 region (right side of the PFC) during the 3-back task blocks within the exciting session. Similar demonstrations for the other participants can be found in the Supplementary material. Also, the highest positive correlation can be observed within the 3-back task blocks for all participants. In particular, we can see that the highest correlation in participants 1–5 corresponds to LB1, RF1, LF4, LF3, and LF2, respectively. Four out of five participants presented the highest performance-HbO correlation with the acquired HbO signal from the left hemisphere. A similar trend can be noted concerning the PFC areas.

In Figure 8, we present the mean energy (black) and mean envelope (blue) of smoothed HbT signal collected from the PFC channels (i.e., channels 23–44). The Figure 8A presents the results within the calming session (green), and Figure 8B is related to the exciting session (red). The 1-back task blocks are indicated with lighter colors, while the 3-back ones are represented with more intense background colors. Aside from participant 3, it can be seen that the peak of the HbT energy is located within the exciting session.

We have performed a working memory experiment in the presence of musical stimuli to collect multimodal physiological signals along with brain hemodynamic response signals of participants and evaluate the possibility of using music as an intervention. Participants were asked to perform a working memory task with different difficulty levels and music types (i.e., calming and exciting). The difficulty levels were included to ensure different cognitive loads during the experiment. We have used fNIRS to record the brain hemodynamic response. By incorporating multiple physiological signals in the experimental data collection, we have derived a rich neuro-physiological dataset that could offer a more comprehensive picture of the body and brain's reaction to music and cognitive load. In this dataset, we have implemented the musical stimuli by conducting the cognitive task in two sessions–one with calming music content and the other with exciting music content. The presence of calming music was supposed to mimic the low arousal condition, while the exciting one was simulating the high arousal one. To incorporate the personalized version of CLBMI, we have used personalized music selected by the participants for calming and exciting music. While the applied music intervention offers us a personalized closed-loop architecture, it can induce the subject familiarity with the experiment environment. One possible approach to preserve the person-specific nature of the intervention and reduce the impact of the subject's familiarity is to employ newly generated music based on the subject's preference in future studies (Fekri Azgomi et al., 2023).

In the presented research, we have collected data using both research grade (e.g., Biopac) and wearable devices (e.g., Empatica E4). While wearable devices can be implemented in everyday life settings and they seem to be more aligned with the future closed-loop architecture, the employed signal processing algorithms and estimation framework are more compatible with the research grade devices, and they outperform in the lab settings. Hence, we perform our in-depth analysis based on the data collected using research-grade devices. Although the presented framework includes a multimodal measurement that can be used concurrently for brain state estimation and classification, here, we analyze the SC data, oxygenated hemoglobin, total hemoglobin, and behavioral signals. However, we deliver complete dataset and experimental settings to provide a framework for any researcher who is interested in performing research in this paradigm. Particularly, the collected data presents a unique opportunity for future investigations, and it holds the potential to unlock groundbreaking insights and guide future data collection in this context.

Recording multimodal physiological signals within a cognitive task can provide an opportunity to explore cutting-edge strategies that may extend human cognitive capacities (Mangaroska et al., 2021). However, multimodal physiological data collection comes with certain challenges (Cukurova et al., 2020). Specifically, a suboptimal sensor connection or the presence of motion artifact in the modalities can lead to data corruption (Mangaroska et al., 2021; Fekri Azgomi et al., 2023). Consequently, the participant removal rate in this form of research, as observed in this experiment, can be high. Given the fast-paced advances in biomedical sensor development, one potential approach to address this concern is to use a more advanced setup that employs a lower number of sensors with a more robust connection to collect the signal of interest. Employing the recently developed sensors may pave the way for future data collection in a more practical and efficient manner.

Based on our analysis on the SC signal, we can observe that the HAI level varies significantly within the music sessions, while the task-wise perspective does not reveal a significant variation. One may interpret that the difference in the induced cognitive load by 1-back and 3-back tasks is not significantly high enough to be reflected in the HAI signal. Another interpretation, though less probable, is that the impact of cognitive load on the arousal level is not as much as the effect of music type, and the cognitive load does not seem to have a mediating effect on arousal. Determining a general relationship that holds out of the scope of this dataset requires a comprehensive causality analysis, which is not the main focus of this study.

It should be noted that the HAI for participants 4 and 5 was reduced within the exciting session. This is the opposite of what the exciting content of music is supposed to elicit. This is a notable counterexample and demonstrates that the skin conductance response does not agree with the emotional content of music for participants 4 and 5. On the other hand, we can see that the HbT energy signal reaches its peak within the exciting session in both participant 4 and 5 cases. While the SC is a long-standing index of arousal (Greco et al., 2016), we can observe that decoding the arousal from a single SC measurement might lack the robustness needed to explain the observed behavior. To address this concern in future investigations and experiment designs, the graphene e-tattoos (GET) sensors can be applied (Jang et al., 2022), and the SC data can be collected and analyzed in a multichannel manner (Alam et al., 2023).

According to the presented HPI, except for participant 2, we can see that the performance indices are elevated within the second session, in which the exciting music was presented. While one may hypothesize that the improved performance is a result of establishing the arousal within a moderate range using the exciting music, other factors such as the learning impact and participant's familiarity with the environment can be influential, and we should avoid making any definite conclusion on the impact of music. Here, findings may suggest the potential for integrating music into the closed-loop system. Given the low sample size, the absence of any mental state score, and the possible confounding factors, further studies with a higher number of subjects, mental state annotation, inclusion of a control group, and shuffled task difficulty as well as music sessions are required for a decisive resolution.

In the context of BMI, users may experience various cognitive loads and emotional states during the interaction with a technical system (Herff et al., 2014). Both cognitive load and emotion status can trigger a particular brain response followed by variation in the cognitive performance (Fishburn et al., 2014; Bigliassi et al., 2015). Hence, it is crucial to understand the relation between the brain response under different cognitive loads and environmental stimuli to potentially optimize performance. The performed experiment provides an opportunity to study the hemodynamic response within different cognitive loads and music sessions. The highest HbO and performance positive correlation for participants 1–5 can be seen during the 3-back task blocks (Supplementary material). One may interpret that the HbO data has the potential to be applied as an informative biomarker of performance within high cognitive loads. Also, the observed high HbT energy over the PFC within exciting sessions may be interpreted as participants' higher brain activities within the exciting session compared to the calming session. This conforms with the findings in Zheng et al. (2020). Perhaps the participants had to concentrate more within higher arousal levels (Wickramasuriya et al., 2023). Overall, these findings may be an indication of a new avenue for decoder design research, leading to innovative fNIRS feature extraction to decode the hidden arousal and performance states. It is important to highlight that due to the small sample size, further studies with a higher number of participants and more interventions as well as cognitive loads would be beneficial in drawing a final conclusion.

It would be vital to note that the experiment serves as a pilot study to investigate the viability of using music as the brain state regulator. In general, our findings show variation in collected signals as well as decoded brain states between different music sessions. According to each person's unique physiology and the applied personalized intervention, we are interested in having a personalized perspective rather than a general view. Particularly, the brain structure of individuals and the network-level interaction between cognitive brain states of individuals can be studied independently regardless of the sample size (Hurlburt et al., 2015). The preliminary findings show that music may potentially have ramifications in the BMI system as a form of background stimulation, while more in-depth research is required to fully understand the role of music on arousal and performance. Specifically, in the CLBMI realm, it is crucial to have precise and robust measurements as well as feedback mechanisms. The presented pilot study should be improved to ensure the robustness of the CLBMI pathways. In the future, magnetoencephalography (MEG) can be employed in parallel with fNIRS technique to improve the spatial and temporal resolution of the brain recordings and implement the neurofeedback in our setup (Yucha and Montgomery, 2008). We intend this study to be a stepping stone for formulating such studies and developing more comprehensive experiments.

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

The studies involving humans were approved by Institutional Review Board at the University of Houston, TX, USA. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

SK: Data curation, Investigation, Methodology, Software, Visualization, Writing – original draft, Writing – review & editing, Formal analysis, Validation. SP: Investigation, Methodology, Visualization, Writing – original draft, Writing – review & editing. SA: Data curation, Writing – original draft, Writing – review & editing. MA: Data curation, Investigation, Methodology, Writing – review & editing. RF: Conceptualization, Funding acquisition, Investigation, Methodology, Resources, Supervision, Writing – review & editing.