The study used a cross-sectional laboratory experimental design (Sabin et al., 2023). The sound performance ratings for 41 hearing devices were estimated based on KEMAR recordings of 12 realistic sound environments and multi-talker scenes. The hearing devices were fit to an age-related hearing loss audiogram with mild-to-moderate high frequency sloping pattern (see Table 1; Bisgaard et al., 2010) using an “Initial” and “Tuned” Fitting. The hearing devices included Rx HA (n = 10), OTC-SF (n = 10), OTC-PS (n = 8), and PSAPs (n = 13).

All hearing device testing was conducted in a custom-built near-anechoic acoustic lab where all surfaces were covered with significant sound-absorbing materials, other than the floor which was carpeted. The resulting test environment was measured to be sufficiently quiet and non-reverberant with an ambient sound pressure level of 34 dB LAeq (A weighted) and 4-frequency (0.5, 1, 2, and 4 kHz) reverberation time of 0.059s. A KEMAR (45BA model with RA0045 ear simulators and VA tapered ear canals) was positioned at the center of the room and used for binaural hearing device recordings (see Figure 1). Surrounding KEMAR was an 8-speaker horizontal array of Yamaha HS5 Powered Monitors (45° resolution). A speaker ring radius of 1-meter was used based on an estimated critical distance of 1.5-meter, ensuring recordings at the ring center were dominated by the direct speaker sound and not room reflections.

An Antelope Orion Studio external sound card, Pro Tools software, and custom Matlab program were used for the presentation of audio scenes through the speaker ring, calibration, and recording of signal from KEMAR's eardrum microphones. For a more detailed explanation see Sabin et al. (2023).

Hearing aids were configured to two fitting paradigms to replicate programming variations observed in the real-world. Specifically, the “Initial Fit” attempted to approximate what most people experience where real-ear measures are not performed and either basic instructions are used, or the default manufacturer recommendations are the basis for fitting (Mueller, 2014). Even a simple fitting process as this has many variables and therefore, a decision-tree flow chart was used for consistency across devices (Sabin et al., 2023). For many OTC products, this meant adjusting one primary parameter (e.g., volume control) to best match NAL-NL2 Experienced user targets (Keidser et al., 2011) for a 65 dB SPL presentation of the International Speech Test Signal (ISTS; Holube et al., 2010). For Rx HA, each respective device's fitting software was used to perform a “First-Fit” relying on manufacturer recommendations and their proprietary fitting algorithm. Device ear tip, or the acoustic coupling between device and KEMAR's artificial ear canals, was also considered and addressed in the flow chart. For DTC products this often resulted in using the default ear tip or, as was the case with Rx HA, using manufacturer recommendations based on N3 audiogram input or an on-ear hearing test.

Recordings were also made at a second “Tuned Fit” where audiologic best-practices were followed to optimize speech intelligibility benefits. In this fitting protocol, all parameters and available ear tips were adjusted to best match prescriptive targets for speech inputs at 55, 65, and 75 dB SPL. If a hearing device did not offer input specific gain adjustments, a 65 dB SPL presentation of ISTS was used for the fitting (see Figure 2). Across both fittings, the equivalent of real-ear measures (REM) was replicated on KEMAR using the output of the eardrum microphones to allow for real-time monitoring of device adjustments in custom Matlab programs [described in Sabin et al. (2023)].

Twelve realistic acoustic scenes from the Ambisonic Recordings of Typical Environments (ARTE) database were decoded and presented to our 8-channel 2-dimensional speaker ring (Weisser et al., 2019). A custom set of multi-talker scripts (1, 2, and 3 talkers) were also recorded with the help of voice actors (1 female and 2 males) in an acoustically treated recording studio. Each script was recorded twice with rotating actors and the associated background sounds were monitored over headphones throughout the recording process to elicit potential Lombard effects. To account for acoustic differences between environments, speech recordings were convolved using the multichannel impulse response from the ARTE database matching the reverberation levels for each corresponding acoustic scene. Individual voices were also positioned across speaker channels 1 (0 degrees), 2 (45 degrees), and 8 (−45 degrees) creating realistic spatial talker locations relative to KEMAR.

The final speech and background scenes were combined and presented following the observed environmental sound-pressure levels and signal-to-noise ratios outlined by Wu et al. (2018). All recordings were preceded by 15 seconds of isolated background scene to allow time for hearing device program switching. A total of 72 scenes were ultimately recorded for each hearing device across 12 background scenes, 3 talkers, and 2 actor rotations.

The audio quality of streamed music was also assessed by presenting five genres of royalty free music with a smartphone (iPhone 8 Plus iOS v16.7.5) to hearing devices positioned on KEMAR's ears. The music segments were 33.7 seconds in length on average and, like speech presentations, an extra 15 seconds of music was included in the beginning to allow time for program switching. Streamed music presentation levels were calibrated prior to recordings by way of a custom Matlab program and real-time spectral analysis of KEMAR's eardrum microphones. Specifically, the smartphone streamed audio levels were adjusted to be +/– 5dB of a music-based reference curve at 1 kHz (1/3 octave filter).

Finally, all audio files were diffuse field equalized to make them suitable for monitoring over headphones (i.e., removing acoustic effects of the manikin from the audio recordings). This was achieved by (1) recording uncorrelated white noise at the center of the speaker ring through KEMAR and a flat reference microphone and (2) computing a filter of the spectral differences between them. The resulting filter shape was largely in agreement with published values as illustrated in Figure 2 (Bentler and Pavlovic, 1992).

All hearing device audio recordings were analyzed across five dimensions of sound performance and scales to a 5-point scale. The first dimensions focused on predicted speech intelligibility benefits using HASPI v2 which models the impaired auditory system and has been found to predict intelligibility across a range of acoustic environments (Kates and Arehart, 2021). HASPI v2 scores were computed for each device, averaged across both ears, and separated into our (1) Speech Benefit in Quiet and Moderate and (2) Speech Benefit in Loud metrics. HASPI v2 values for each of these two categories were computed separately based on environmental scenes and whether the average sound pressure level was < or >70 dB SPL, respectively.

Own Voice Perception was our third sound performance metric which aimed to estimate subjective occlusion using Real Ear Occluded Insertion Gain (REOIG). This was previously obtained during calibration to verify ear tip occlusion on KEMAR and reflects spectral differences between open ear and occluded ear (with device off). Subjective occlusion for our Own Voice Perception metric was estimated by comparing our REOIG values to those from Cubick et al. (2022) and then mapping the relationship between our objective measurements and their subjective user ratings. For devices with active occlusion compensation (AOC), extra steps were taken to account for the influence of active cancellation on own-voice sound quality. This was estimated by measuring active occlusion with an Audioscan Axiom test box and similarly mapped our findings to REOIG values and the previously mentioned user ratings. These steps are described in detail in Sabin et al. (2023).

Feedback volatility, or the likelihood of a device to squeal during everyday use, was the focus of our fourth sound performance dimension (i.e. Feedback Handling) and is a common complaint among hearing aid users (Jenstad et al., 2003). We therefore tested the feedback canceller of each device by making KEMAR recordings in two challenging real-world conditions: (1) simulating a hair scratch motion by moving hands periodically by KEMAR's ears for 10 seconds and (2) repeatedly cupping KEMAR's ears for 10 seconds. These recordings were then subjectively rated during blind listening tests and mapped to a 5-point scale.

The final dimension was Streamed Music Audio Quality which again sought to estimate subjective sound quality preferences of hearing aid users. Appropriately, the Hearing Aid Audio Quality Index (HAAQI; Kates and Arehart, 2015) was used as it employs the same model of the impaired auditory system as HASPI v2 and was designed to match sound quality judgements of individuals with hearing loss. HAAQI scores were mapped to a 5-point scale after averaging scores across KEMAR's ears and the 5 audio compositions recorded.

To increase accessibility of our findings, we sought to simply the five dimensions of hearing device audio performance and two fitting paradigms to a single easy to understand number, i.e., “SoundScore.” This was achieved in three steps: (1) by combining the composite sound performance metrics using a user-centric weighted average, (2) combining the Initial and Tuned fitting paradigms in a manner that mirrors real-world user experiences, and (3) applying a normalization factor such that the top scoring hearing device received a 5.0. The specifics of this process are outlined in Sabin et al. (2023), and involved two separate forced-rank surveys of hearing aid consumers and professionals (seen in Table 2).

Descriptive statistics of the sound performance ratings (i.e., SoundScore) were examined. The difference in sound performance rating across hearing device categories based on technology level were assessed using one-way ANOVA if the data met the assumption of normality or using the Kruskal-Wallis test if the data failed the assumption of normality. Post-hoc multiple comparisons were performed using two sample t test or Wilcoxon rank-sum test with Benjamin-Hochberg corrections for multiple comparisons. In the presence of heteroscedasticity Welch ANOVA followed by the Games-Howell pairwise comparison tests were used. Similar analysis was conducted to assess the significant differences of sound performance across different fitting methods.

Difference in sound performance ratings in the first fit and tuned fit for Rx HA and OTC-SF were evaluated using paired sample t-tests. Finally, a simple linear regression model was utilized the impact of the form factor (i.e., BTE vs. ITE) on the SoundScore. Necessary regression model assumptions, including the normality were satisfied. All analyses were performed with R statistical software (Version: 4.2.2). All tests were two tailed and performed at a threshold of 5% level of significance.

Figure 3 depicts the variation in SoundScore across different hearing device categories. The highest median overall score can be seen with the Rx HA (median: 3.15, IQR: 0.35) while the lowest median overall score was seen with the PSAPs (median: 1.4, IQR: 1.4). PSAPs had the highest IQR (1.4) relative to all other categories.

Table 3 includes the mean (SD) values for different elements of audio performance ratings across different hearing device categories along with the results of the significance test among these categories. The Rx category displayed the highest mean SoundScore of 3.07 (SD: 0.3), while the PSAP category had the lowest at 1.48 (SD: 0.9). Differences in SoundScore between the hearing device categories was significant (p < 0.001) as illustrated in Table 3. SoundScore between Rx HA vs. OTC-PS (p = 0.018), Rx HA vs. PSAP (p < 0.001), and OTC-SF vs. PSAP (p = 0.006) were significantly different as shown in Supplementary Table 1. Significant differences were also observed for audio performance elements of speech benefit in quiet and moderate, speech benefit in loud, own voice sounds boomy, and overall scores for both first fit and tuned fit, but not for elements music streaming sounds good and does not feedback. Pairwise comparisons showed significant difference between Rx HA vs. OTC-PS, Rx HA vs. PSAP, OTC-SF vs. PSAP on several of these elements but no significant differences were observed among Rx HA vs. OTC-SF, OTC-SF vs. OTC-PS, and OTC-PS vs. PSAP (see Supplementary Table 1).

Figure 4 shows the SoundScore variation across hearing devices with different adjustment interface. The highest median overall score can be seen for devices with fitting software (median: 3.15, IQR: 0.35) while the lowest median overall score was seen with the hearing devices with preset programs (median: 0.85, IQR: 0.50).

Hearing devices with fitting software (Table 4) showed the highest mean SoundScore of 3.07 (SD: 0.3) while preset-based hearing devices had the lowest mean of 0.91 (SD: 0.5). A statistically significant difference in SoundScore between different fitting methods (p < 0.0001) was demonstrated (Table 4). Pairwise comparisons showed a significant difference in SoundScores for devices with fitting software vs. App-based (p < 0.01), fitting software vs. preset-based (p < 0.001), and App-based vs. preset-based (p < 0.001) as illustrated in Supplementary Table 2.

Significant differences were also observed for audio performance elements of speech benefit in quiet and moderate (in first fit and tuned fit), speech benefit in loud (in first fit and tuned fit), own voice sounds boomy (in tuned fit), music streaming sounds good (in first fit and tuned fit), and also for overall scores (in first fit and tuned fit) across hearing devices with different fitting methods as shown in Table 4. The pairwise comparisons are provided in Supplementary Table 2.

Figure 5 shows the spread of first fit (green dots) and tuned fit (red dots) scores for Rx HA and OTC-SF. For Rx HA, the overall first fit score variation (M = 2.99, SD = 0.4) is reduced substantially with audiologist tuned fit (M = 3.36, SD = 0.3). However, for OTC-SF, the tuned fit overall scores (M = 2.61; SD = 0.7) does not seem to change substantially when compared to initial fit overall scores (M = 2.36; SD = 0.8). These results suggest that the Rx HA category can be fine-tuned more than the OTC-SF. In addition, the Rx HA brand may have some difference in initial fit with manufacturers proprietary software, but the audiologist can achieve similar audio performance ratings following tuned fit irrespective of the device brand and price-point.

The initial fit and tuned fit scores for overall score and sound performance in loud (Table 5) were statistically significant for both Rx and OTC-SF (i.e., scored improved after tuned fit). Additionally, the initial fit and tuned fit scores for sound performance in quiet and moderate (scores improved after tuned fit) and does not feedback (scores decreased after tuned fit) scores were significantly different for Rx HA.

Figure 6 shows the SoundScore variation across hearing devices with different form factor. The mean and median SoundScore for BTE devices were higher (mean: 2.23, SD: 1.10, median: 2.65, IQR: 1.85) when compared to ITE devices (mean: 2.19, SD: 0.80, median: 2.10, IQR: 1.05). There was no significant difference (t = 0.13, p = 0.9) in SoundScores among hearing devices with different form factor (i.e., BTE vs. ITE). For this reason, no further analysis was performed related to the Form factor.

This study marks an initial step in developing a consumer-centric metric for assessing the audio performance of hearing devices. However, the novel approach introduced here, along with the resultant findings, must be carefully considered within the context of study's limitations. First, our methodology involved using a single audiometric profile typical of mild-to-moderate hearing loss and a limited array of acoustic environments. While this approach enabled a degree of standardization, it inherently does not encompass the wide variability in hearing loss patterns. This limitation suggests that our findings may not fully extend to the broader hearing-impaired population. Second, while the SoundScore metric introduced here offers an innovative means to quantify audio performance, it does not encapsulate other crucial factors that inform consumer decisions. Aspects such as comfort, usability, device features, and cost are also vital to the decision-making process but remain beyond the score of this metric. Hence, the SoundScore should be considered as one of multiple factors in comprehensive decision-making framework. Third, although the study aimed to represent a breadth of hearing devices, and classification was guided by the FDA categories, some devices within OTC-PS (FDA QEG category) demonstrated features more characteristics of OTC-SF devices. This overlap indicates a potential need to refine device classification or to consider feature-based, rather than category-based, differentiation in future research. Finally, it is most important to recognize that these group-level results represent a current snapshot in time. We expect that any category differences to change as the field continues to indicate. With this in mind, we intend to evaluate new devices as they are released.

Early validation of our method and metric has been promising, yet further research is essential to establish the relationship between the SoundScore and actual user benefit and satisfy action. Future studies should seek to determine the SoundScore thresholds that correlate with optimal hearing aid outcomes. Additionally, defining critical difference levels for the SoundScore would enhance its precision in differentiating between devices, particularly concerning their sound performance. Such research endeavors will not only fortify the validity of the SoundScore but will also expand its utility for consumers navigating the complex landscape of hearing devices.