The terms bright and dark, used commonly to describe colors, are also widely acknowledged as some of the most frequently used timbre descriptors, and the relationship between visual and timbral brightness has been investigated through numerous perceptual studies. For example, Wallmark et al. (2021) observed via a speeded classification task that incongruity between visual and timbral brightness increased error rate, though this did not affect response time; Saitis and Wallmark (2024) observed that timbral brightness modulated the perception of pitch and possibly visual brightness. More generally, past research has provided robust evidence for both pitch-brightness and timbre-brightness correspondences (e.g., Marks, 1982; Marks et al., 1987; Wallmark, 2019b; Wallmark and Allen, 2020). While the mechanisms behind such congruences are not fully understood, the audio-visual connection is readily apparent.

A relationship between pitch height and lightness is also well-documented: higher pitches are associated with lighter colors, while lower pitches are associated with darker colors (Marks, 1974; Marks, 1987; Marks, 1989; Melara and Marks, 1990; Martino and Marks, 1999; Mondloch and Maurer, 2004; Ward et al., 2006). Given that timbral variation can affect perception of pitch height (e.g., Warrier and Zatorre, 2002; Kuang et al., 2016), we included the terms high and low among our timbre semantic descriptors. We anticipated potential differences in semantic judgments on these terms even as pitch register is held constant (as in Experiment B)—and that these differences would influence lightness of color choice. Taken together, previous research on relationships among pitch, timbre, visual lightness, and visual brightness indicates that timbres perceived as brighter and higher will tend to be matched with lighter colors, whereas darker-and lower-sounding timbres will tend to be matched with darker colors.

The terms big, small, heavy, and light (in weight) are colloquially used in describing instrument timbre (Reymore and Huron, 2020). Previous research has not directly explored correspondences between timbral characteristics and size or weight; however, we can extrapolate predictions from past experiments that have demonstrated robust pitch-size associations, where lower sounds are linked to larger size (Walker and Smith, 1984; Mondloch and Maurer, 2004; Gallace and Spence, 2006; Evans and Treisman, 2010; Bien et al., 2012; Walker et al., 2012).

Given (1) the close connections described in the previous section among visual lightness/brightness, timbral brightness, and pitch height, and (2) the observation that contrasts of bright-dark, high-low, light–dark, and small-big are considered to be examples of polar dimensions that align with one another (Parise and Spence, 2013), we included the terms small, big, heavy, and light in weight among our semantic descriptors. Triangulating timbre-brightness, pitch-brightness, and pitch-size correspondences led us to hypothesize that timbres judged to be smaller and lighter in weight will be matched to lighter colors, whereas bigger and heavier timbres will be matched to darker colors.

Across work on audiovisual crossmodal correspondences, the role of saturation has been studied less frequently than lightness. Here, research on heaviness, saturation, and pitch guided our predictions. Alexander and Shansky (1976) observed that participants assigned heavier weights to darker and more saturated colors, while Walker et al. (2017) found that objects perceived as heavier were judged to be darker and to make lower-pitched sounds. From here, we used prothetic relationships and polar alignments to extrapolate to other terms in our set. For example, Walker and Smith (1984) found happy to align with high, little, and light (in weight) in an interference task; similarly, Eitan and Timmers (2010) observed that light in weight-heavy were consistently matched with high-low for both English-and Hebrew-speaking participants. Based on these findings, we hypothesized that timbres rated as heavier would not only be matched to darker colors, but to more saturated colors, guided principally by the findings of Alexander and Shansky (1976). In parallel, we anticipated big would also correspond to more saturated colors, whereas small and light (in weight) would map to less saturated colors.

Notably, an argument could also be made in favor of a relationship with saturation in the opposite direction, based on other previous studies. From a series of experiments using the implicit associations test, Anikin and Johansson (2019) found saturation to be positively associated with frequency and spectral centroid. Hamilton-Fletcher et al. (2017) asked participants to adjust hue and chroma to heard sounds while luminance was held constant. They similarly found that frequency, as well as a timbral manipulation caused by adjusting the spectral center of gravity, was positively correlated with saturation.

Open-ended interviews suggest that musicians readily use emotion-related words in timbral discourse (Reymore and Huron, 2020). Di Stefano (2023) presents an account of how timbre can be associated with emotional qualities via the concept of atmosphere, which has an essential affective component. He notes that the relationship between timbre and perceived emotion does not depend on high-level cognitive mediation but rather is motivated by acoustic features. Indeed, behavioral evidence implicates instrument timbre in ratings of valence, tension arousal, and energy arousal, with increases on these scales corresponding to changes in acoustical descriptors such as fundamental frequency and spectral centroid (e.g., McAdams et al., 2017; Eerola et al., 2012; Korsmit et al., 2023; see also Korsmit et al., 2024).

Spence and Di Stefano (2022) argue for an emotional mediation account of color-sound correspondences, positing that ultimately, emotion explains these correspondences, “no matter whether the stimuli are simple or complex” (p. 30). Palmer et al. (2013) found strong evidence consistent with an emotion mediation account in the context of matching excerpts of classical orchestral music to color. The emotion mediation hypothesis has been supported by findings from subsequent studies using single-line piano melodies (Palmer et al., 2016), excerpts from J.S. Bach’s Well-Tempered Clavier (Isbilen and Krumhansl, 2016), and musical excerpts from a variety of genres (Whiteford et al., 2018). Should emotion mapping be the primary mechanism at work in timbre-color matching when musical content is controlled, we expect to observe ratings on the emotion words happy and sad, as semantic descriptions of timbre, to explain as much as or more variance in color choice as the other words in our set.

From previous research in timbre semantics, two other descriptors stand out as ostensibly related to color associations: warm and cool. With respect to timbre description, warm is one of the most common descriptors in English and has been included in dimension labels across many timbre studies (see Saitis and Weinzierl for a review). Cool, or cold, as timbre descriptors, are far less common, but these terms do also emerge in timbre description (Reymore and Huron, 2020). In everyday parlance, these words are most commonly used to describe temperature, but warm and cool are also applied readily to colors, with warm suggesting colors like red, orange, and yellow, and cool suggesting colors like green and blue (likely related to temperature-associated imagery like fire and ice). This shared vocabulary across visual and auditory modalities suggests a potential association between warm and cool sounds and warm and cool colors; however, no previous studies have investigated this correspondence.

In sum, to generate hypotheses about relationships between timbre description and timbre-color matching, we drew on common crossmodal timbre terms, choosing specific terms related to previously established correspondences between pitch-brightness/lightness, timbre-brightness, pitch-size/weight, and music-emotion. Motivated by the literature reviewed above, we assembled a set of ten terms that were used to predict both lightness and saturation of colors matched to timbres. Our experiments also address whether the terms warm and cool, often used to describe both timbre and color, are related in a crossmodal context.

Consensus plots (Figure 2) demonstrate relative agreement across subjects in their color selections for each keyboard instrument and pitch register tested in Experiment A; these plots map onto the palette used in Experiment A, as shown in the figure. Each palette shows contour plots of consensus relative to the maximum consensus for that instrument. The maximum consensus, indicated in the upper-right corner of each panel, is expressed as the proportion of participants selecting the most frequently-chosen color square. Note that participants were able to select multiple color squares. In Experiment A, the median number of color samples chosen for a given stimulus was 4 (IQR = 6), with a minimum of 1 and a maximum of 44.

Stimuli in the lowest octave (F2) were concentrated toward greater saturation levels with selections moving toward less saturation as pitch register increases, at least from F2 to F3. Overall, there was a bias toward more saturated colors. All three keyboard instruments in their lowest octaves tended to map onto blues and purples, while the Lautenwerk and the Flemish harpsichord also include a second concentration of responses in highly saturated reds, oranges, and browns. In the upper two octaves, selections for the piano show a diverse range of hues, while selections for the Lautenwerk and Flemish harpsichord show a predilection for reds, oranges, yellows, and some greens. The highest octaves in both harpsichords show higher consensus, particularly on yellows. The piano does not appear to have a unique hue profile that remains consistent as register changes; both harpsichords demonstrate concentrations of warmer responses in all three octaves, though at F2, they also demonstrate a second concentration in blues and purples that is consistent across the three instruments.

To assess relationships between semantic descriptors and color choices (H1), we constructed three sets of regression models to predict color-related dependent variables (lightness, saturation, warmth-coolness) from semantic ratings of individual terms. The effects of musical instrument (H2) and pitch register (H3) on color choices were tested via three two-way ANOVAs with lightness, saturation, and warmth-coolness as dependent variables.

Experiment A varied both instrument timbre and pitch register; Experiment B was designed to experimentally control for pitch register while investigating timbre-color correspondences across a more diverse range of instrument timbres.

Consensus in participants’ color selections for the six orchestral instruments tested in Experiment B are shown in Figure 7. The format of each panel is the same as that for Figure 2, above, where contours indicate areas of relatively high consensus, and consensus is calculated irrespective of participants’ luminance settings. Maximum consensus values are displayed in the upper-right corner of each palette. In Experiment B, the median number of color samples chosen was 4 (IQR = 4) with a minimum of 1 and a maximum of 49.

While it is apparent that there is diversity in color selection, it is also clear that there is more agreement for some instruments than others. The flute and viola do not demonstrate obvious hue bias, although green and blue responses for the viola are less common compared to other hues. The oboe spans a number of hues, but color selections are more concentrated than for the flute or viola, avoiding teals and purples. Color selections for the trumpet show a hue bias toward reds, oranges, yellows. The violin demonstrates a tendency to be matched to reds and oranges. Clarinet selections are somewhat concentrated on blues.

Analysis for Experiment B replicates the procedures described for Experiment A.

H2 proposed a relationship between musical instrument type and color choice, operationalizing color choice, as in H1, in three ways (lightness, saturation, warmth-coolness). Results linking instrument type and lightness were mixed: the relationship was significant in Experiment A but not in Experiment B. Both experiments demonstrated significant differences among instruments with respect to saturation and the warm-cool index.

In Experiment A, pairwise tests carried out via the emmeans function in the emmeans package (Lenth, 2023) and adjusted using the Tukey method pointed to significant differences between the Flemish harpsichord and piano as well as between the Lautenwerk and piano, but not between the two harpsichords, for all three dependent variables. Among the orchestral instruments in Experiment B, only clarinet/trumpet and clarinet/violin contrasts were significantly different for saturation (see Figure 10 for a visualization of all instruments’ mean warm-cool indices in Experiment B). With respect to the differences in warm-cool index, the trumpet appears to be solely responsible for the significant ANOVA result (see Figure 9). Taken together, it seems that there may be certain instruments that are characterized by relatively high or low values for one or more dimensions of color, such as the trumpet, but that these dimensions are not equally relevant for all instruments and/or may only be evident with particular types of timbral contrasts. The extent to which relationships between instrument type and individual dimensions of color are driven by, or interact with, the motivation to match an instrument to a certain hue (such as the trumpet, to red), calls for further clarification.

H3, which proposed a relationship between pitch register and color, was tested in Experiment A only. We observed significant differences in lightness, saturation, and the warm-cool index as functions of pitch register (see Figures 4–6), and no interactions were found with instrument type. Replicating findings established in previous scholarship, increases in pitch register were associated with increasing lightness. Saturation decreased with increasing pitch register, consistent with our hypothesis but inconsistent with some previous findings in the literature (see 5.1.2 for a more in-depth discussion). The piano and Flemish harpsichord both showed a consistent increase of color warmth with pitch register, while the Lautenwerk’s matched colors were warmest in the middle register. Post hoc Tukey tests of the warm-cool index model show significant increases in warm-cool index between F2 and F3, and F2 and F4, but not between F3 and F4, likely due to the Lautenwerk’s divergent profile.

Although our Experiment A findings suggest a positive relationship between pitch register and warmth of color choice, trends in the semantic ratings across pitch register reveal further complexity: ratings of warm are highest in the middle register (F3) for each of the keyboard instruments, while ratings of cool tend to increase with pitch register! The latter observation is consistent with the results of work by Wang and Spence (2017), where participants matched the experience of drinking cold water with higher pitch, as compared to room temperature and hot water. However, it is notable that as timbre semantic ratings, warm and cool are not simple opposites when it comes to their relationships with pitch height.

Research on timbre semantics provides some insight into these relationships. Through interviews and an online survey with expert participants, Rosi et al. (2022) found that “a bright sound has most of the spectral energy in the high frequencies. It is often a high-pitched sound, with clarity, definition, and similarities with a metallic sound…A warm sound encloses substantial spectral energy in the low-mid frequencies. It is a rather low pitch sound…A warm sound is pleasant, enveloping, and rich” (480). That all three keyboards were rated as timbrally warmest in the middle register (rather than the lowest) suggests a kind of sweet spot for timbral warmth, rather than a simple linear relationship between pitch height and warmth. This middle register peak is not unexpected in light of findings from Reymore et al. (2023) and Reymore (2021) that other plausibly pleasant dimensions—smooth/singing and watery/fluid—show an inverted-U relationship with register. If warm timbres are associated with middle and lower registers, but color warmth is associated with higher pitch register, this could explain the null finding for H1c in Experiment A. Perhaps, participants’ gravitation toward warmer color choices in timbre-color matching is better explained by a combination of increased pitch and increased timbral brightness than by timbral warmth. In Experiment B, we noted that the instruments with higher warm-cool indices (oboe, violin, trumpet) seem to be brighter and more nasal than those with lower values (clarinet, flute, viola). Examination of average ratings on bright confirmed that this casual observation is consistent with participant ratings. Taken together, these post hoc observations suggest that timbral brightness may be more closely related to color-based warmth than is timbral warmth.

Thus, although we did find a significant relationship between ratings on warm and cool and warmth-coolness of color choice in Experiment B (but not Experiment A), results should be interpreted with caution. The significant effect may have been driven by other factors or may depend on the sample of instruments tested. Future work should assess judgments across a wider range of instruments to test generalizability and identify specificities that may interact with broader trends.

Palmer et al. (2013) posited that if correlations between color and musical excerpts were mediated by common emotional associations, they would find analogous results when asking participants to choose colors most/least consistent with music and with any other set of stimuli strongly associated with the same emotional dimensions. Results from both subsequent experiments designed to test this were consistent with the hypothesis. Previous research shows that participants can make judgments on perceived emotion of timbres based on short tones (e.g., McAdams et al., 2017; Korsmit et al., 2023), suggesting that it is possible for perceptions of emotion to drive timbre-color matching.

Should shared emotion, mood, or affect provide the best account for timbre-color matching, as posited in the emotion-mediation account, we could anticipate in our experiments that the terms happy and sad would provide equivalent or better explanatory power than other descriptors. In our experiments, ratings on the terms happy and sad were significantly related to lightness in both experiments, but sad was not significant for saturation in either experiment. For lightness, in Experiment A, happy and sad resulted in slightly lower marginal R² values among the significant terms; in Experiment B, R² values were equivalent to those for other terms. That is, our emotion terms seem to be less related to color choice than other terms in our set.

On one hand, it may be that happy and sad were not apt emotional descriptors for the available instrumental timbres, but that other emotional terms would have provided a better fit. However, this seems unlikely for the given musical context; Korsmit et al. (2023) found that two dimensions are sufficient for capturing variance in emotional assessment of single-note and chromatic scale stimuli. On the other hand, it seems likely that emotional qualities of isolated timbres are not always the most relevant consideration for a given task. Other, lower-level perceptual features might be more immediately relevant for timbre-color mapping. Indeed, Spence, (2020a) observes that emotion mediation tends to account for more variance for complex stimuli as compared to simpler and less emotionally valent stimuli. Similarly, from results of an auditory-conceptual association study, Di Stefano et al. (2024) note that their findings support the view that complex stimuli are more likely to generate emotional meaning than are simpler stimuli (e.g., isolated sounds). Future research could test the emotion mediation hypothesis using Palmer et al.’s (2013) method with isolated timbres to directly address the claim that this explanation is equally applicable to complex and simple stimuli.

Spence (2011) proposed three categories motivating crossmodal matching, including semantic, physiological, and statistical, where semantic mediation relates to linguistic or lexical correspondence (e.g., we use high to describe both elevation and pitch). Motoki et al. (2023) added the category of “affective” to this model to account for emotion mediation (see also Spence, 2020a). These categories are not mutually exclusive, and a given correspondence may be motivated by multiple categories. As we chose our terms for this study based on crossmodal terms observed in previous timbre semantics research, it seems reasonable that semantic mediation provides an overall better explanation for our results, at least with respect to lightness and saturation. Notably, the terms happy and sad are used less often to describe timbre as compared to the other terms in our set—from a semantic mediation perspective, this may explain why these terms explained relatively less variance.

Participant color choices for each instrument were diverse. Some instruments demonstrated no specific trends in hue but did show heavier concentrations of responses at particular lightness levels, such as the flute. Some instruments demonstrated trends in hue, such as the trumpet, violin, and clarinet. While specific instrument-hue associations from our participants only seem apparent for a few instruments, the origins of these associations provide an intriguing subject for further research, particularly in the case of the trumpet, which has shown robust hue correlations in both historical and empirical studies (Reuter et al., 2018).

In the case of the keyboard instruments, pitch register appears to have played an important role: the lowest octave showed a concentration on saturated blues and purples for all three instruments. The piano was associated with a dispersed range of hues in the top two octaves, but both harpsichords showed an increasing tendency toward yellows as pitch increased. For the harpsichords at F2 in Experiment A, we speculate that participants may have been responding to two different aspects of the sound—some may have chosen colors primarily on the basis of pitch register (blues and purples), whereas another group may have been influenced by the brighter timbres of the harpsichord to select oranges and reds, explaining the bimodal concentration of responses in Figure 2. In general, color choices for the two types of harpsichords are far more similar to each other in each octave than they are to the piano and are generally associated with warmer colors as compared to the piano. Similarities in color choices between the two types of harpsichords may reflect their relative perceived similarity in timbre as compared to the piano.

One limitation of our approach to analysis was our use of the mean in quantifying the saturation and warmth-coolness of participant color choices. Participants often selected multiple colors for a stimulus; for such observations, the saturation and warm-cool values used in modeling were approximated using the mean. This becomes potentially problematic when participants selected colors in different areas of the arrays. For example, if a participant selected a low saturation color and a high saturation color, the average saturation of their response is in the middle, which may not be representative of their response tactic. However, before settling on the use of the mean for analysis, we manually reviewed the data and determined that color selections were usually made in single clusters of contiguous color samples, for which the mean was a reasonable representation.

As previously mentioned, participants were not required to use the slider during their selections, and the data reveal that a subset of participants did not use the slider when choosing colors, defaulting to the lightest setting. It is unclear whether participants were intentional about this and felt that the lightest palette best exemplified the colors they wanted, misunderstood the directions, or opted not to use the sliders in order to get through the experiment more quickly. In experiments using a similar interface, we recommend enforcing slider use by requiring participants to acknowledge the slider via touch, even when they prefer to leave it in its initial location. We also acknowledge the possibility that for some subjects and conditions, even our most luminous palettes might not have been sufficiently luminous to match the subject’s subjective appraisal of the musical recording.

It should be noted while the use of the major scale for stimuli facilitated comparisons among instruments and allowed us to directly compare semantic ratings and color choices, using stimuli with different musical parameters (mode, articulation, rhythm, etc.) would likely result in somewhat different choices in both semantic descriptions and colors. While we hypothesize that the relationship between semantic descriptions and colors would hold in other simple musical contexts, future research might vary musical parameters to test the generalizability of our results.

Finally, though some portions of the variance in lightness and saturation were explained from the semantic ratings, the majority of variance in each case remains unexplained; a number of other factors likely influence timbre/color matching behavior. It should be noted that the purpose of the experiments reported here was not to thoroughly model the timbre-color matching process, but rather to test whether crossmodal language is related to timbre-color associations. Our results provide converging evidence in support of this theory but also suggest that the issue is multi-layered and complex, and that there are likely multiple principles guiding timbre-color matching behavior.

For example, interactions among the three color dimensions, particularly between hue and each of the other dimensions, may account for some of the variance in color choice. The trumpet provides a likely example of this. Color selections for the trumpet appear to be hue-focused: selections near red, orange, and yellow were most often selected. If a participant hears the trumpet and immediately thinks of a basic color category, such as that exemplified by a typical fire-engine red, the imagined hue necessitates a particular level of lightness—the lightness of the choice is in some way an artifact of the hue. Similarly, if a participant associates a sound with a typical yellow, the selection will be necessarily on the higher end of the lightness scale because as yellow darkens, it becomes brown. However, Hamilton-Fletcher et al. (2017) previously found crossmodal associations between non-musical sounds and color, including a relationship between yellow and high frequencies, even when controlling for the influence of lightness. Such observations raise the possibility that similar types of relationships may have been at play in the current study concerning instrumental timbre (note specifically the hue results among the keyboard instruments), though further research is needed to determine whether that is the case for the types of stimuli studied here and how such relationships may be connected to participants’ use of crossmodal language.

Although we did not observe that ratings on happy or sad were privileged among semantic terms in predicting color, timbre-color matching might be more generally a product of valence transfer (e.g., Weinreich and Gollwitzer, 2016). Emotion might also influence color choice via participant mood. For example, perhaps participants in a happier mood might be generally more prone to choosing lighter or yellower colors. Another potential mediating variable could be personal preference. Participants may be more likely to choose colors that they like, but preference might also play a more complex role: for example, participants might match colors they like to timbres they like and colors they dislike to timbres they dislike, where preference interacts with valence or qualia transfer.

Additionally, some color matching choices for recognizable instruments may be primarily driven by semantic rather than perceptual associations derived from cultural tropes. This might be the case for the trumpet’s association with red or the clarinet’s association with blue. Even if this is sometimes the case, however, it may still be possible that some of these cultural associations have perceptual origins, and so the two cannot be completely separated without further investigation. Future work could assess participants’ familiarity with instruments, which may help identify such connections.

We did not observe any particular patterns or differences among participants with sound-related synesthesia, though there were only a few such participants. The question of how synesthesia might interact with timbre-color associations could be addressed in a subsequent study with a larger population of synesthetes. In general, further systematic investigation of effects of other individual or cultural differences could also help contextualize variability of responses.

Finally, although the number of stimuli in the current experiment limited the generalizability of audio feature analysis, future research might include a larger and more varied stimulus set to assess the impact of particular features, such as spectral centroid. A stimulus set with increased timbral diversity would help clarify such potential relationships and might reveal more pronounced differences in semantic ratings and/or color choices. Future work might expand beyond instruments in the Western orchestral tradition to include instruments from other musical traditions in order to widen the array of available timbres.