Forty-five subjects aged 19 to 65 (M = 31.5 years, Mdn = 26 years, σ = 13.43 years) voluntarily participated in the study (without compensation). They were engineering students or colleagues from the TH Köln. They all reported no hearing complaints and no history of hearing loss or auditory processing disorders in a questionnaire completed prior to participation in the study. This was used as inclusion criteria. All subjects were native German speakers, and about 42% had previously participated in other listening experiments.

The study was designed following the principles of the Declaration of Helsinki (World Medical Association, 2013) and the guidelines of the local institutional review board of the Institute of Computer and Communication Technology at the TH Köln. All participants gave written informed consent for their voluntary participation in the study and the later publication of the results. All personal data and experimental results were collected, processed, and archived according to country-specific data protection regulations.

The experiment took place in the sound-insulated anechoic chamber of the acoustics laboratory of the TH Köln, which has dimensions of 4.5 × 11.7 × 2.3 m (W×D×H), a lower cut-off frequency of about 200 Hz, and a background noise level of about 20 dB(A) SPL. We used three Genelec 8020D loudspeakers for the baseline measurement, i.e., the loudspeaker-based environment, and an Oculus Rift with a pair of Sennheiser HD600 headphones for the virtual environments. An RME Fireface UFXII interface connected to a PC controlled the loudspeakers and headphones.

We used the German hearing in noise test with a male target speaker, which includes twelve phonemically- and difficulty-matched 20-sentence lists and a spectrally matched masker noise. All sentences are four-to six-word-long simple sentences incorporating common nouns and verbs used at the elementary school level (Joiko et al., 2021).

The HINT procedure includes four conditions where the target speech is always in front of the listener (0° azimuth). The noise (masker) is either at the same location, i.e., Noise Front (NF), shifted to the left (90° azimuth) or right (270° azimuth), i.e., Noise Left and Noise Right (NL and NR), or suppressed, i.e., Quiet (Q) (Joiko et al., 2021; Mönnich et al., 2023; Nilsson et al., 1994; Soli and Wong, 2008). The order in which the noise conditions were presented was randomized, changing each time a list of twenty sentences was completed.

Following the HINT procedure described by Soli and Wong (2008), speech and noise were initially presented at 65 dB(A) SPL measured in the free field at the listener’s position (see Presentation level calibration section for more details). The SNR was automatically adjusted based on the subject’s performance using a 50% intelligibility criterion. It decreased if the subject repeated at least half of the sentence correctly, e.g., at least two words in a four-word sentence. Otherwise, it increased. The level of the masker remained constant, and the SNR was adjusted by increasing or decreasing the level of the target sentences. The procedure uses step sizes of 4 dB for the first four sentences and 2 dB for the remaining 16 sentences (per list). SRTs are calculated by averaging the SNR over sentences 5-20 (including the SNR for a 21st sentence determined from the response to the 20th sentence) for each noise condition (NF, NR, NL, and Q), as described by Soli and Wong (2008).

Participants were randomly assigned to one of three groups. Each group underwent the HINT test in the loudspeaker-based environment and in one of the three different VR environments, as follows: Group A (n = 15, mean age of 33.4 years) was tested in the loudspeaker-based and VR1 environments, group B (n = 15, mean age of 29.5 years) was tested in the loudspeaker-based and VR2 environments, and group C (n = 15, mean age of 31.5 years) in the loudspeaker-based and VR3 environments.

The order in which subjects took the test in both environments (loudspeaker-based or virtual) was randomized across participants. Each participant completed four sentence lists (80 sentences) per test environment (160 sentences in total). No sentence or list was repeated per participant.

Before each test (loudspeaker-based or virtual), participants were informed that their task was to repeat aloud (or log into the system for VR3) what they heard. They were informed that they did not have to keep their head still (as in many laboratory-based listening experiments) and that dynamic binaural rendering was available in the headphone-based conditions. So they knew that they could move their head naturally if they wanted to, as in the loudspeaker-based condition. However, they were not given any verbal or written instructions about optimal head orientation strategies to preserve the undirected nature of the behavioral experiment concerning head movements.

After receiving instructions, participants had the opportunity to complete a practice run (5 sentences) in a random test condition (NL, NR, NF, or Q) to familiarize themselves with the test environment. This was followed by time to ask any questions they had before the test began. All subjects were given a 10–15 min break between tests.

Both tests took place in the same room (the anechoic chamber of the acoustics laboratory of the TH Köln). Participants sat in the same chair in the middle of the loudspeaker-based setup. The only difference between test conditions was whether the auditory presentation was loudspeaker-based or headphone-based and whether the subjects wore the HMD or not.

First, we adjusted the presentation level for the loudspeaker-based condition. We played the masker noise on each loudspeaker and adjusted their level independently until the free-field sound pressure level at the listener’s position was 65 dB(A). Then, for the headphone-based presentation, i.e., all VR conditions, we placed the dummy head (Neumann KU100) in the listener’s position, played the same stimuli on the central loudspeaker (0° azimuth), and measured the electrical level produced at the dummy head’s ears. Subsequently, the headphones were placed on the ears of the dummy head, and the same stimuli (noise signal from 0° azimuth) were played again via binaural rendering to adjust the headphone presentation level to the same electrical level.

Individuals may have different SRTs between the NL and NR test conditions. This difference may be due to (common) asymmetries between left and right hearing thresholds, cochlear function, neural processing, or head orientation strategies. Notably, in our study, these differences did not exceed 1  dB , as we tested only NH adults. Thus, to simplify the results’ interpretation, we averaged each participant’s SRTs from the NL and NR conditions. This resulted in one outcome measure for the spatially separated test conditions (NL and NR averaged) and one for the colocated noise condition (NF).

Following, we calculated the SRM as the difference between the SRTs in the spatially separated and colocated noise conditions (Equation 1). S R M = S R T N L + S R T N R 2 − S R T N F (1)

For statistical analysis, we performed a Bayesian repeated measures analysis of variance (ANOVA) with default priors (r scale fixed effects of . 5 , r scale random effects of 1 ) for SRM (which includes NL, NR, and NF noise conditions) and for the SRTs in quiet with the within-subjects factor environment[loudspeaker-based, virtual] and the between-subjects factor group [A, B, C](Keysers, et al., 2020; Rouder et al., 2012).

We performed posthoc testing through individual comparisons based on the default t-test with a Cauchy 0 , r = . 707 prior (Rouder et al., 2009; Wagenmakers, 2007) and corrected for multiple testing by fixing to . 5 the prior probability that the null hypothesis holds across all comparisons (Westfall et al., 1997). All statistical analyses were performed using the Bayesian Repeated Measures ANOVA module of the Jamovi software package (Jamovi, 2022).

Figure 3 shows the calculated SRTs as a function of the spatially separated and colocated noise conditions for all groups in both loudspeaker-based and virtual environments. The box plots show the individual SRTs per participant as points (with a horizontal offset for better readability).

The mean SRTs for the spatially separated noise conditions range from − 14.6 to − 14.0  dB SNR across groups and test environments, while the mean SRTs for the colocated noise conditions range from − 6.0 to − 5.7  dB SNR .

The variance in the results from the VR1 environment (Group A) is noticeably higher than the variance in the other groups. In particular, it is more pronounced towards lower (better) SRTs in the VR1 environment than in the loudspeaker-based baseline within the same group. This is the case for six of the fifteen participants in this group (40%) and holds for the same participants in both the spatially separated and the colocated noise conditions.

The effect may be due to the choice of visual feedback provided in the VR1 environment, as the auditory headphone-based presentation was the same in all other virtual environments, and this is not seen in the within-subjects comparison (against the loudspeaker-based baseline in the same group). One possible explanation is that the lack of visual feedback may have benefited some participants by allowing them to focus more on the auditory task, similar to closing their eyes and paying more attention to the auditory stimuli. This may have resulted in an advantage (compared to the loudspeaker-based measurement) where they could see the anechoic room, the loudspeaker array, and the experimenter, which may have distracted them and influenced their responses. However, as this effect appears to be homogeneous for both spatially separated and collocated noise conditions, it does not significantly affect our main outcome measure, the SRM values.

Supplementary Figure S1 shows the raw SRTs before averaging NL and NR conditions.

The data presented in Figure 4 shows the calculated SRM values for each subject per test environment and group. The trend lines at the bottom of the figure show individual performance trends across test environments. The line colors indicate an improvement (green) or deterioration (red) in SRM in the virtual environment compared to the baseline measurement in the loudspeaker-based environment.

The means of SRM across all groups and test environments range from 8.4  dB to 9.1  dB .

The Bayesian repeated measures ANOVA for SRM with default priors showed that the predictive performance P M d a t a of the null model was higher than the predictive performance of all the rival models with and without each factor and their interaction (Table 1).

There is positive (moderate) evidence that the null model is more likely than the models including the factor group BF 01 = 4.72 ,and strong evidence of the absence of an effect of both factors [environment + group] BF 01 = 13.73 and both factors and their interaction [environment + group + environment*group] BF 01 = 70.33 . However, the null model is only 2.84 times more likely than those including the factor environment, and even though the data tends to prove the absence of an effect of environment, it is still inconclusive 1 3 < B F 01 < 3 (Keysers, et al., 2020).

The analysis of effects across all models, however, reveals evidence of the absence of an effect of environment, group, and their interaction (all B F i n c l <   1 3 ) (Table 2), and a post hoc pairwise comparison confirms evidence for the absence of an effect of test environment in SRM BF 01 = 3.84 (Table 3).

Since the statistical analysis revealed no effect of group, we show the pooled data across groups in Figure 5 to provide a clearer visualization of the similarity between the SRM measures across test environments. The average SRM across subjects n = 45 when tested in a conventional loudspeaker-based system was 8.5  dB . In comparison, the average SRM increased slightly to 8.8  dB when subjects were tested in virtual environments.

Figure 6 displays the SRTs in quiet for each group and test environment in dB (A) SPL, i.e., free-field sound pressure level at the listener’s position. The means of SRTs in quiet across all groups and test environments range from 13.4  dB A to 16.2  dB A SPL.

The Bayesian repeated measures ANOVA for the SRTs in quiet revealed positive (moderate) evidence in favor of the absence of an effect of environment BF 01 = 3.665 and in favor of the absence of an effect of both factors [environment + group] BF 01 = 8.341 . However, for the models including the factor group and the full model (including both factors and their interaction), the evidence is too weak to be conclusive, i.e., there is an absence of evidence 1 3 < B F 01 < 3 (Table 4).

The analysis of effects across matched models provides further evidence that the null model is twelve times more likely than those including the interaction between test environment and group BF incl = 12.187 , confirms the absence of an effect of environment BF incl = 0.271 < 1 3 , but remains inconclusive regarding the factor group 1 3 < B F 01 < 3 (Table 5).

Post hoc pairwise comparisons confirm evidence for the absence of an effect of test environment BF 01 = 4.78 (Table 6) but remain inconclusive for all pairwise comparisons between groups 1 3 < BF 01 < 3 (Table 7).

The means of all speech-in-noise measures from our study, in both loudspeaker-based and virtual environments, are consistent with the norms reported in the literature for the German HINT (Figure 7) (Joiko et al., 2021; Mönnich et al., 2023).

Similarly, the average SRM from both loudspeaker-based and virtual environments align with the findings of previous studies using the same spatial configuration and masker type employed here (stationary noise at 90° azimuth) (Andersen et al., 2016; Beutelmann and Brand, 2006; Bronkhorst and Plomp, 1988; Cosentino et al., 2014; Jelfs et al., 2011; Müller, 1992; Ozimek et al., 2013; Peissig and Kollmeier, 1997; Platte and vom Hövel, 1980; Plomp and Mimpen, 1981).

The validity of our baseline measurement was demonstrated by results consistent with those of previous studies (Figure 8). Furthermore, the similarity of the results obtained in the virtual environments with those of the baseline measurement for both outcome measures (SRM and SRTs in quiet) suggests that it may be feasible to replicate speech-in-noise results from conventional loudspeaker-based setups using portable and inexpensive VR peripherals.

It should be noted, however, that our results are limited to the experimental conditions tested: anechoic environment, with a single target speech source and a single energetic masker in spatially separated (90°) and colocated conditions.

With this preliminary study, we aimed to provide initial evidence that state-of-the-art technology can be used to reproduce results from conventional laboratory and clinical settings. The positive results of our study encourage further testing in more complex, i.e., realistic, listening environments, which may lead to more ecologically valid results. In particular, evaluating the ability to reproduce results from “real” reverberant conditions in virtual audiovisual environments is of great interest.

In this context, a significant body of research has shown that with current methods and technologies, such as those used in this study, it is possible to create virtual auditory environments indistinguishable from reality, even in reverberant listening conditions. Plausible and authentic virtual acoustic presentations are possible in both anechoic (Arend et al., 2021a; Weber et al., 2024) and reverberant conditions (Lindau and Weinzierl, 2012; Brinkmann et al., 2017; Brinkmann et al., 2019; Arend et al., 2021b; Arend et al., 2024). However, as outlined in the introduction, exploiting multisensory integration is crucial to achieving this level of realism (Keidser et al., 2020). Many relevant technical aspects have to be considered, such as the use of real-time motion compensation, the use of matching visuals or matching the auditory environment perfectly to the visual real world, including appropriate descriptions of the source and receiver characteristics, e.g., source directivity and HRTFs, and correct headphone compensation filters, among others.

With this contribution, we include the Unity project (including all the virtual environments described here), which facilitates the reproducibility and extension of our experimental setup. Multiple sources can be easily added, and all stimuli (including audio and text) can be replaced using the regular file system. This means our setup can be easily transferred to different stimuli in different languages. The application also allows easy customization of various parameters, such as the number of noise conditions, lists, sentences, practice rounds, and adaptive step sizes. There is no need to modify the source code, as all customization can be done using the Unity Inspector interface.

To our knowledge, this study is the first to investigate speech-in-noise abilities using headphone-based dynamic binaural rendering with non-individual HRTFs. The role of spontaneous head movements in increasing the target signal level when speech intelligibility decreases, also known as the head orientation benefit (HOB), has been extensively studied. Kock’s work in 1950 was the first to demonstrate this phenomenon (Kock, 1950). Later, Grange and Culling (2016) investigated the benefits of head orientation away from the speech source in NH listeners. They analyzed spontaneous head orientations when listeners were presented with long speech clips of gradually decreasing SNR in an acoustically treated room. The speech was presented from a loudspeaker initially facing the listener, and the competing noise was presented from one of four other locations. In an undirected paradigm, they observed that listeners instinctively turned their heads away from the speech (between ±10° and ±65°) in 56% of trials in response to increased intelligibility difficulties. They then observed that when subjects were explicitly instructed to perform head movements, all turned away from speech at lower SNRs and immediately reached head orientations associated with lower SRTs.

Similarly, Brimijoin et al. (2012) investigated head orientation strategies in a speech comprehension task in the presence of spatially separated competing noise. They found a clear tendency to orient approximately 60° away from the target, regardless of the position of the distractor signal, in listeners with large (>16 dB) hearing threshold differences between their left and right ears.

We did not log the head-tracking data in this study because spontaneous head movements and the resulting HOB have already been studied and demonstrated for a long time. However, as expected, we observed that participants’ behavior regarding head orientations was consistent with the abovementioned findings.

The videos in the Supplementary Material, recorded from the listener’s perspective, exemplify the spontaneous use of head movements and illustrate the (very pronounced) level increase in one ear when head movements are exploited.

In addition to supporting auditory spatial perception by aiding externalization and distance estimation (Best et al., 2020), the use of visual cues in speech-in-noise testing paradigms may facilitate a better understanding of the mechanisms underlying auditory perception from a multisensory perspective and potentially lead to significant advances in hearing research. In clinical settings, using appropriate visual cues can improve current assessment procedures’ ecological validity and accuracy. For example, lip-reading has been shown to support speech intelligibility in noisy environments, and it is an aspect that is still overlooked in current speech-in-noise assessment methods (Helfer and Freyman, 2005; Williams et al., 2023; Yuan et al., 2021). In this study, we used a simple visual cue at the target speaker location without facial expressions or lip movements, and we found evidence of the absence of an effect of the visual feedback used in SRM.

Nevertheless, this may be different in more complex listening scenarios, for example, in cases where there is some uncertainty about the target speaker’s location or in multi-speaker or cocktail party scenarios. There may be interactions between vision and auditory perception that require further investigation, and VR can play an important role in supporting auditory research in this regard. Further work should focus on understanding these potential interactions and their impact on speech intelligibility and listening effort.

While currently available speech-in-noise tests are reliable, they require manual scoring by a clinician, which can be inconvenient in busy clinical settings. As a result, these tests are not widely used in routine hearing evaluations. The literature highlights the need for automated tests, which could allow testing while the patient is waiting in the clinic or remotely (Jakien et al., 2017). The use of closed-set tasks may facilitate unsupervised measurements.

To assess whether using a closed-set instead of the standard open-set procedure from the HINT would affect speech-in-noise measures, we incorporated the word selection feedback system into the VR3 test. We found evidence for the absence of an effect of group and environment on the SRM measure, suggesting that it is feasible to conduct speech-in-noise testing in VR, even with the unsupervised procedure introduced here. However, the evidence supporting our unsupervised procedure was too weak to be conclusive for SRTs in quiet.

Closed-set procedures may result in reduced (better) SRTs, mainly when the response set contains few phonetically dissimilar alternatives (Miller, Heise, and Lighten, 1951; Warzybok et al., 2015; Buss, Leibold, and Hall, 2016). Future research in remote or unsupervised speech-in-noise testing could explore alternative ways to design automated response systems. For example, Litovsky (2005) suggested adjusting task difficulty based on the listener’s age. Their results indicated that 4AFC tasks were easier for adults than children, resulting in lower SRTs. Johnstone and Litovsky (2006) subsequently found significant differences in adult SRTs using 4AFC and 25AFC tasks as response methods, but only when speech was used as a masker, not when modulated noise was used, suggesting that appropriate response methods could be both population and stimuli-dependent.

Another attractive alternative may be to incorporate automatic speech recognition (ASR) into the virtual tests, preserving the open-set nature of the task while allowing for unsupervised testing. Ooster et al. (2023) proposed using an ASR for automatic response recording based on a time-delay neural network. They estimate an SRT deviation below 1.38 dB for 95% of users with this method, suggesting that robust unsupervised testing may be possible with similar accuracy as with a human supervisor, even in noisy conditions and with altered or disordered speech from elderly severely hearing-impaired listeners and cochlear implant users.

While VR offers potentially groundbreaking opportunities, many issues must be carefully considered before virtual testing can be used for individualized screening in clinical settings. Some of them are:

Our preliminary results represent only a small sample of NH adults. These results cannot be generalized to other populations. Follow-up studies should include large standardized samples, including patients with hearing loss, auditory processing disorders, and NH controls of different age groups in both conventional loudspeaker-based and VR conditions.

Although VR and its potential to serve children has been extensively researched, particularly in educational and medical settings, such as a tool for pain distraction, assessment of Attention Deficit/Hyperactivity Disorder (ADHD) and Autism Spectrum Disorder (ASD), and psychotherapy, among others, many questions remain about the potential impact of VR on children’s development. It is still unclear whether we could use VR to assess speech-in-noise abilities in children. Further research should focus on creating controlled and safe environments that allow us to address these questions. This includes aspects such as appropriate exposure times, appropriate complexity of visual feedback, and considerations such as appropriate HRTF sets from a technical virtual acoustics perspective.

VR can cause motion sickness. In our study, participants did not report any adverse effects while using the HMD. However, this is a relevant aspect when using more complex visual feedback, as this may increase the likelihood of experiencing it.

Managing cognitive load is another major challenge in the design and use of VR. Excessive cognitive load can negatively affect the user experience, cause fatigue, and interfere with task performance. Therefore, future research should focus on understanding the relationship between increased realism in virtual environments, its associated cognitive load, and its potential impact on measures of auditory processing.

Recognizing and addressing the potential drawbacks of VR through research, innovation, and responsible use can help maximize its benefits while minimizing its risks. By promoting ethical and inclusive practices and fostering a balanced approach to VR adoption, we can harness this transformative technology in hearing research and audiological healthcare.