Many applications of electroencephalography (EEG) and event-related potentials (ERP) (Luck, 2012; Nidal and Malik, 2014) require the use of triggers (or tagging) (Wang et al., 2016; Cattan et al., 2018) to indicate the exact onset of presented stimuli (mostly visual or auditory events) (Miyakoshi et al., 2021; Ignatious et al., 2023) so that the recorded physiological data can be synchronized. However, variability in various hardware and software typically lead to differences in latencies between the transmission and reception of triggers (Wang et al., 2016; Cattan et al., 2021; Iwama et al., 2022). It is of importance that these latencies in triggers should not be confused with another type of latency that is present in the neural markers, such as N170, P250, N400, etc. (Luck, 2012; Cattan et al., 2021; Miyakoshi et al., 2021; Abreu et al., 2023). Although there are several studies (Hoormann et al., 1998; Kiesel et al., 2008; Wu et al., 2013; Liesefeld, 2018; Ignatious et al., 2023) that showcase the computation of latencies in neural markers and their association with brain activities, the focus of the current work is on the computation of latencies in event-related potential TRIGGERS, which is a critical pre-processing step for any brain-computer interface (BCI) study. Existing literature (discussed in the next section) lacks a methodical approach with datasets to compute trigger latencies, especially for immersive virtual reality (VR) apparatus. The objective to overcome this knowledge gap formed the main motivation of this work.

While past studies (Cattan et al., 2018; Iwama et al., 2022) have highlighted shifting of ERP epochs based on computed average latencies, other studies illustrate details about setting up the triggers to overcome latency differences (Wang et al., 2016; Cattan et al., 2018). To the best of our knowledge, only a few studies exist which describe the importance and considerations for computing the latencies in triggers using immersive VR systems (Wang et al., 2016; Cattan et al., 2021; Iwama et al., 2022). In the literature (Wang et al., 2016; Lees et al., 2018), Rapid Serial Visual Paradigm (RSVP) (Wang et al., 2016; Lees et al., 2018) is one of the common, simple, yet effective approaches for sending triggers. Lab Streaming Layer (LSL) (Stenner et al., 2023) is another preferred choice in many studies (Wang et al., 2016; Iwama et al., 2022). The availability of open-source resources like Simulation and Neuroscience Application Platform (SNAP) (Kothe, 2023) developed in Python to ease stimuli presentation, also favored using LSL. However, this could mandate using a setup with bigger memory and displays with higher refresh rates compared to the RSVP approach (Wang et al., 2016). For efficient hardware-software synchronization with the LSL approach, an extra hardware setup like Light Diode Resistor Comparator Circuit (LDRCC) (Wang et al., 2016) has also been used. Hence, the RSVP approach with C# programming was followed in this work.

Most of the existing literature (Lopez-Calderon and Luck, 2014; Cattan et al., 2018, 2021; Lees et al., 2018; Williams et al., 2021; Huang et al., 2022; Iwama et al., 2022), which at least provide scattered details and some considerations for settings up triggers, are based on using only EEG sensors or at most a few auxiliary (AUX) sensors. Although synchronization with other modalities like motion-capture (MoCap) has been achieved (Miyakoshi et al., 2021), the knowledge about setting up its triggers and computation of latencies during VR experiments is lacking.

The work present work contributes to overcoming the existing knowledge gaps in the literature by setting up the following objectives: (1) to demonstrate the Computation of Latencies in Event-related potential Triggers (CLET) as a tool for measuring latencies in multi-model experiments, especially when VR apparatuses are used; and (2) to provide open access to novel datasets, codes, tables, and a tutorial video¹ to ensure transparency and reproducibility of results, as well as boost future improvements in the algorithms.

The work was performed to test the synchronization of triggers in a multi-model experiment that was designed to monitor biomechanics (specifically gait patterns) and physiological signals. This article is limited to the illustration of the CLET approach. The experimentation is explained in the next section, and the proposed method is detailed in the subsequent section. The computed latencies and their distribution are described in the results section, with the method’s advantages compared to the state-of-art being covered in the discussion section. Finally, the conclusions, limitations and future scope for improvement are described in the last section.

The experiment setup with the rapid serial visual paradigm (RSVP) is shown in Figure 1A. The apparatus included one desktop personal computer (PC1)–with Unity and Qualisys Track Manager (QTM) software installed and one laptop (PC2) with EEG Recorder (Brain Vision Recorder) software installed. The biomechanics monitoring setup included 3D MoCap cameras (nine Qualisys high-speed cameras and one Miqus camera), and the physiological monitoring setup included Brain Products LiveAmp with 64-channel wireless EEG with dual channels EMG and one PPG sensor. A photodiode was connected as an auxiliary (AUX) sensor with either a LED screen or an HMD at a time. The LED screen consisted of a Sony TV (KDL-75W855C) with dimensions of 167.7 × 96.9 × 7.9 cm and a refresh rate of 100 Hz. The HMD consisted of HTC VIVE Pro Eye with a field of view of 110° and a refresh rate of 90 Hz.

As shown in Figure 1A, PC1 was used to control the MoCap apparatus through a wired connection with the QTM software. The same computer was also used for stimulus presentation through the Unity software, which sent triggers to the EEG amplifier (amp.) unit via a USB connection to the wireless trigger box. Both displays were also connected to PC1. Data recorded using the amplifier was sent via Bluetooth to USB1 and USB2 dongles connected to PC2. The stimuli consisted of ∼100 images of black and white colors. The neutral image consisted of a gray color image with a red-colored cross at the center. The selection of these images represented ON (white screen) and OFF (black screen) input signals to the photodiode placed on the display. This procedure aligned with the protocol described in Cattan et al. (2021). The inter-stimulus interval (isi) was randomly varied between 1.0–1.5 s with a fixed stimulus duration of 0.3 s. The experimental paradigm was written in C# inside the Unity software. Here, the white and black stimuli were assigned to be displayed as “S1” and “S2” triggers, respectively, in EEG recordings. Similarly, the start and end of the recordings were assigned “S7” and “S8” triggers, respectively.

The photodiode was placed on the LED screen and covered with a black tape and then black cloth to prevent any disturbance from any external light source. The procedure to place and cover the photodiode for both displays is shown in Figure 1B. The entire experiment was performed in a dark room. After the calibration of Qualisys markers, the test involved recording QTM, then the Brain Vision Recorder (BV Rec.), followed by Unity. When the test was complete, a text notification was visible in the Unity console window. The operator was required to stop the software in the reverse order, i.e., Unity, then BV Rec., followed by QTM. The data was synchronized through triggers and time points noted in the log files. A similar process was repeated by the placement of the photodiode on the HMD and covering the sensor with black tape and black cloth. The procedure for placing the photodiode sensor is shown in Figure 1C. Each test lasted ∼5 min. The photodiode data recorded from each display is shown in Figures 2, 3. For better visualization of its shape, a section of 5 s is shown in Figures 2B, 3B. The methods developed to analyze the photodiode data recorded (see next section) have subtle variations for each of the displays due to the differences in their shapes.

In the non-immersive VR environment with the LED screen (Figure 4A), the CLET method efficiently computed the latencies for a set of white and black stimuli. For the white stimuli, the average latency was 121.98 ± 8.71 ms. Similarly, for the black stimuli, the average latency was 121.66 ± 8.80 ms. The distribution is maximum between 120–125 ms range (Figure 4A). Thus, the consistency in the latencies between the two sets of stimuli indicated the robustness of the CLET approach in this VR configuration.

In the immersive VR environment with the HMD (Figure 4B), the average latencies were 82.80 ± 7.63 ms, mainly distributed between 80–85 ms, and 69.82 ± 5.52 ms, mainly distributed between 67–77 ms for white and black stimuli sets, respectively. The lower latencies observed with the HMD setup than the LED screen setup suggested faster temporal dynamics for the immersive VR apparatus (Iwama et al., 2022). Although this observation is consistent with findings in the literature (Cattan et al., 2021), variations are particularly subject to protocols that rely on sending triggers for synchronization (Wang et al., 2016; Lees et al., 2018) and specifications of the hardware apparatuses (Rebenitsch and Owen, 2017; Cattan et al., 2018; Andreev et al., 2019; Wilson, 2023).

The results from both VR setups (Figure 4) also highlight the importance of considering latency distributions along with precision and accuracy (Williams et al., 2021) in the computation of latencies for aligning epochs, to avoid any timing discrepancies which would otherwise lead to misinterpretations of data (Cattan et al., 2021; Iwama et al., 2022).