There is growing interest in studying mixed emotions in which positive and negative emotions co-occur (Larsen and McGraw, 2011; Oh and Tong, 2022), and two general approaches have been used to measure mixed emotions. The first approach involves direct measures which directly instruct participants to report experiences of mixed emotions on self-report questionnaires (Barford and Smillie, 2016). The second approach involves indirect measures which do not directly ask participants about mixed emotions, but instead calculate them based on scores for positive and negative emotions; most commonly, the minimum index (MIN) has been recommended as a valid indirect measure which is computed by taking the smaller of two scores between positive and negative emotions (Larsen et al., 2017). There is substantial variability even within these approaches. For example, different items are often included in different direct measures (Barford and Smillie, 2016; Oh and Tong, 2021), and indirect measures can be calculated either from single scores for positivity and negativity (Berrios et al., 2018) or could be calculated from averaged scores for positivity and negativity (Oh, 2022). Nevertheless, evidence suggests that both direct measures and indirect measures computed using MIN validly assess mixed emotions when they occur (Berrios et al., 2015).

However, little has been done to address statistical issues related to their use in predictive models testing associations between mixed emotions and key outcomes. Specifically, the extent to which direct measures versus indirect measures predict key outcomes the same way, the extent to which these measures may be affected by statistical issues such as multicollinearity, and the extent to which these measures may be confounded by response biases remains unclear. We performed a pre-registered exploratory study addressing these issues.

Based on meta-analytic evidence assessing the measurement of mixed emotions following experimental manipulations, some potential limitations with indirect measures as computed using MIN have been suggested by Berrios et al. (2015). For example, as MIN works by taking the smaller score between positive and negative emotions, it may provide lower-boundary estimates of mixed emotions, which may lead to some degree of range restriction. Additionally, effect sizes for detecting mixed emotions were also found to be smaller when using indirect measures. Preliminary evidence also suggests that indirect measures may be more susceptible to multicollinearity when entered alongside positive and negative emotions (Oh, 2022)—in particular, as most people experience more positive affect than negative affect in daily life, the smaller of two scores between the two will commonly be the score for negative affect. Hence, scores for MIN are often very strongly correlated with negative affect, leading to multicollinearity. The typical consequence of multicollinearity is that errors may be inflated and statistical power may be reduced, making it more difficult to detect genuine effects and leading to Type II errors (Mason and Perreault, 1991). Moreover, some researchers have argued that multicollinearity could also be linked to Type I errors (Kalnins, 2022).

If so, the overall consequence of multicollinearity is that it may introduce interpretive difficulties. A common recommendation for dealing with multicollinearity is to remove multicollinear variables from regression equations—however, this is less feasible in correlational work aiming to establish a unique explanatory role of mixed emotions. In experimental designs, multicollinearity is less critical as such designs allow researchers to separate mixed emotions, positive emotions, and negative emotions into their respective conditions and analyses conducted generally do not require linear regressions (e.g., Rees et al., 2013; Oh and Tong, 2021). In correlational contexts, however, ruling out the explanatory role of positive and negative emotions virtually necessitates their statistical control to conclude that any associations are independently explainable by mixed emotions (Berrios et al., 2018; Dejonckheere et al., 2019; Oh and Tong, 2022). An empirical comparison of direct and indirect measures in predictive models is crucial in providing researchers with empirical guidance about the use of these measures.

An additional concern is that various response biases may limit the utility of either direct or indirect measures, or both. Firstly, compared to measures of single emotions (e.g., “happy”), direct measures of mixed emotions involve more complex items describing multiple emotions (e.g., “both happy and sad,” “a mixture of two opposing emotions”), and individuals high on trait cognitive complexity may be more likely to endorse these items. Secondly, as both negative emotions and mixed emotions are often uncomfortable states that are seen as undesirable (Rothman et al., 2017), social desirability may reduce the likelihood of reporting such states (Oh and Tong, 2021), leading to underestimated scores on both direct measures and MIN. Thirdly, high scores on self-reported measures could reflect acquiescence, the tendency to agree to self-report items haphazardly (Kam, 2016). To the extent that acquiescence inflates self-reported scores for positive emotions, negative emotions, as well as mixed emotions, both direct measures and MIN could be biased towards artificially inflated scores. It is thus crucial to provide empirical evidence that associations between mixed emotions and later outcomes are not attributable to such biases, which would lend weight to existing evidence about mixed emotions and support the use of these measures in future research.

We performed a pre-registered exploratory investigation of these methodological issues. We collected two separate samples which differ only in the emotion measures administered—as different researchers often use different scales, this allows replication of our conclusions across different measures of emotions. We assessed key outcomes based on existing evidence on mixed emotions. Given that mixed emotions fundamentally involve the co-occurrence of opposing affective states, several perspectives suggest that conflict is the central hallmark of mixed emotions (van Harreveld et al., 2009; Vaccaro et al., 2020), and we hence assessed psychological conflict as a key outcome variable. Some evidence also suggests that mixed emotions serve integrative functions, such that they signal the need to process a complex environment via enhancing receptivity to conflicting perspectives or new behaviors (Rees et al., 2013; Oh and Tong, 2021), and we hence also assessed psychological receptivity. Finally, diverse evidence has linked mixed emotions to well-being outcomes, most commonly conceptualized using eudaimonic measures. For example, Oh (2022) found that mixed emotions in daily life were linked to poorer eudaimonic well-being, while Berrios et al. (2018) and Oh and Tong (2021) instead found that mixed emotions in more specific contexts such as a graduation ceremony or the COVID-19 pandemic were linked positively to eudaimonic well-being. As such, we also assessed eudaimonic well-being using a four-dimension scale adapted from Costin and Vignoles (2020). We performed path analyses comparing the predictive validity of direct versus indirect measures of mixed emotions for these outcomes, with and without controls for positive and negative emotions. Additional analyses were performed to determine whether trait dialecticism, need for cognition, social desirability, and acquiescence could confound associations between measures of mixed emotions and these outcomes.

Correlation matrices for each sample are reported in Supplementary Tables S1, S2, while the full analytic models with all covariates are reported in Supplementary Tables S3–S6. Direct measures were strongly correlated with MIN (r_Sample-1 = 0.81; r_Sample-2 = 0.66; ps < 0.001), suggesting substantial convergence in the constructs they assess. Furthermore, as shown in Table 2, path analyses adjusting only for demographic variability indicate that direct measures and MIN were related to the outcome variables in overwhelmingly similar ways. Specifically, both measures predicted increased psychological conflict, reduced meaning, coherence, and purpose, but not receptivity. The sole difference is that MIN predicted reduced mattering in Sample 2, while direct measures did not.

We next examined path models in which positive emotions and negative emotions are controlled. In both samples, direct measures were correlated with positive (r_Sample-1 = 0.28; r_Sample-2 = 0.17; ps < 0.001) and negative emotions (r_Sample-1 = 0.71; r_Sample-2 = 0.60; ps < 0.001). In Sample 1, MIN was also correlated with positive (r_Sample-1 = 0.25, p < 0.001) and negative emotions (r_Sample-1 = 0.80; p < 0.001). However, in Sample 2, MIN was not correlated with positive emotions (r_Sample-2 = 0.03, p = 0.56) and was very strongly correlated with negative emotions (r_Sample-2 = 0.94, p < 0.001).

Multicollinearity is typically indexed using variance inflation factors (VIF) or tolerance statistics, which assess the degree to which variances of regression coefficients are inflated due to collinearity (O’brien, 2007). As VIFs and tolerance statistics are functionally and mathematically similar, we report only VIFs. Examining multicollinearity statistics when entering mixed emotions alongside positive and negative emotions, direct measures had lower multicollinearity scores (VIF_Sample-1 = 2.52; VIF_Sample-2 = 1.68) compared to MIN (VIF_Sample-1 = 3.53; VIF_Sample-2 = 9.46). MIN suffered from particularly severe multicollinearity in Sample 2, which made further analysis inadvisable—we nevertheless analyzed models using MIN to illustrate the impact of multicollinearity in predictive models where mixed emotions are entered alongside positive and negative emotions. As shown in Table 2, controlling for positive and negative emotions generally attenuates the predictive validity of mixed emotions. However, whereas direct measures remained predictive of conflict, coherence, and purpose in both samples, MIN remained consistently predictive only of conflict, and predicted purpose only in Sample 1 but not Sample 2. These findings are consistent with previous evidence suggesting that inflated multicollinearity reduces statistical power, making it harder to detect effects.

Finally, we examined models in which potential confounding factors were entered to determine whether the predictive validity of direct measures or MIN may be influenced by these factors. Both direct measures (r_Sample-1 = 0.33; r_Sample-2 = 0.32; ps < 0.001) and MIN (r_Sample-1 = 0.24; r_Sample-2 = 0.26; ps < 0.001) were correlated with trait dialecticism. Direct measures (r_Sample-1 = −0.02; r_Sample-2 = 0.07; ps > 0.15) and MIN (r_Sample-1 = −0.06; r_Sample-2 = −0.05; ps > 0.20) were not correlated with need for cognition. Both direct measures (r_Sample-1 = −0.28; r_Sample-2 = −0.25; ps < 0.001) and MIN (r_Sample-1 = −0.29; r_Sample-2 = −0.32; ps < 0.001) were negatively correlated with social desirability. Finally, in Sample 1, neither direct measures nor MIN were correlated with acquiescence (r_Sample-1-DM = 0.04; r_Sample-1-MIN = 0.07; ps > 0.15). However, in Sample 2, both direct measures and MIN were correlated with acquiescence (r_Sample-2-DM = 0.24; r_Sample-2-MIN = 0.18; ps < 0.001).

As shown in Table 2, associations between direct measures and MIN with the key outcomes remained relatively unchanged in models when these constructs were entered as covariates, suggesting that the key predictive validities of either direct measures and MIN are not attributable to response biases due to trait cognitive complexity, social desirability, or acquiescence. One exception is that in full models where all covariates are entered, the significant negative association between direct measures of mixed emotions with purpose fell to marginal significance in both samples. Additionally, controlling for trait dialecticism caused the non-significant association between direct measures and receptivity to turn significant in Sample 1, while controlling for need for cognition caused the non-significant association between direct measures and mattering to turn significant in Sample 2. In the full models with all covariates controlled, the non-significant association between direct measures and receptivity became significant in both samples. These are likely suppression effects resulting from statistical artifacts due to these covariates and should be interpreted cautiously.

Given that both direct measures and MIN significantly predicted psychological conflict, we further performed seemingly unrelated regressions¹ to compare the standardized coefficients of direct measures and MIN for psychological conflict in the full models with all covariates included. In sample 1, direct measures were significantly more predictive of psychological conflict than MIN, χ² (1) = 7.14, p = 0.008, though the difference was non-significant in Sample 2, χ² (1) = 1.30, p = 0.25. Taken together with the finding that direct measures but not MIN predicted coherence in both samples, these findings provide partial support for the idea that direct measures may have greater predictive validity than MIN in models controlling for positive and negative emotions.

In two samples, direct and indirect measures were strongly positively correlated and were related to outcome variables in similar ways, which suggests that both measures are likely to be valid indicators of mixed emotions (Berrios et al., 2015; Larsen et al., 2017) and also suggests some convergence between objective and subjective indicators of ambivalence (Ng et al., 2022). However, consistent with Oh (2022), we found that indirect measures were more vulnerable to multicollinearity when entered alongside positive and negative emotions, which complicates their use in predictive models seeking to conclude an additive role of mixed emotions over positive and negative emotions (Dejonckheere et al., 2019). Whereas direct measures remained predictive of increased conflict and reduced coherence in both samples after controlling for positive and negative emotions, indirect measures remained consistently predictive only of conflict. Finally, in models accounting for trait dialecticism, need for cognition, social desirability, and acquiescence, the predictive validities of direct and indirect measures largely remained consistent, suggesting that response biases from these constructs are unlikely to account for findings based on these measures.

Issues with multicollinearity are complicated. Rules of thumbs for determining multicollinearity (e.g., VIF < 5) provide simple heuristics for determining the severity of multicollinearity, but are not definitive (O’brien, 2007). For example, previous work suggests that a major consequence of multicollinearity is reduced statistical power, and at larger sample sizes which are statistically more powerful or with larger effect sizes that are easier to detect, even relatively higher multicollinearity statistics may be less problematic (Mason and Perreault, 1991; O’brien, 2007). Our findings are consistent with this to some extent. For example, indirect measures remained predictive of psychological conflict despite relatively severe multicollinearity, likely because large effect sizes for this variable alleviate losses in statistical power. In contrast, for outcome variables where effect sizes are smaller, indirect measures are less able to detect effects due to losses of statistical power. Generally, though higher multicollinearity does not guarantee statistical issues, such difficulties become more likely as multicollinearity increases.

We caution against simplistically interpreting our findings as implying that direct measures are not susceptible to multicollinearity, or that indirect measures are unusable in predictive models—neither of these conclusions are warranted. At small sample sizes or when trying to detect small effect sizes, even relatively lower multicollinearity statistics such as those found when using direct measures could introduce interpretive difficulties (Mason and Perreault, 1991; O’brien, 2007). Indirect measures remain valid and reliable in predictive models, but may be more appropriate in analytic situations where reduced statistical power is not a major problem, such as if large effect sizes are expected or if large sample sizes are available. A further implication is that it may be prudent to plan for larger sample sizes in anticipation of losses in statistical power that may occur due to multicollinearity for studies that seek to include indirect measures of mixed emotions along with positive and negative emotions in regression models.

Some limitations should be noted. Firstly, our findings apply to the measurement of mixed emotions in daily life and may not generalize to other contexts. For example, in adversity contexts (e.g., Oh and Tong, 2021) where negative emotions may be reported at higher levels than positive emotions, MIN is less likely to track negative emotion scores and we may likely expect less multicollinearity between MIN and negative affect. Additionally, experimental contexts of mixed emotions induced momentarily and examinations of more specific variants of mixed emotions (e.g., Oh and Tong, 2022) may potentially yield different results than global examinations of mixed-valenced mood. Future examinations should thus also compare the statistical properties of direct and indirect measures in more specific situational contexts of measurement.

Secondly, we have focused on comparing direct measures and MIN on certain statistical issues, but other important considerations should be noted. For example, apart from statistical considerations, methodological considerations are pertinent. In studies that assess positive and negative emotions over longer periods, such as over several days or weeks, MIN is not appropriate as an index of simultaneous co-occurrence, and direct measures may be more suitable (Oh and Tong, 2022). Additionally, even within direct and indirect measures, there is substantial variability in their statistical properties depending on factors like item choice. For example, there was greater multicollinearity in the direct measures of Sample 1 compared to Sample 2, and there was also greater multicollinearity in the indirect measures of Sample 2 compared to Sample 1. Moreover, as indirect measures are computed from two separate scores, a potential limitation of indirect measures is that estimates based on indirect measures may be especially imprecise due to them being influenced by the error variances of both positive emotion scores and negative emotion scores. As MIN provides a lower-boundary estimate of mixed emotions (Berrios et al., 2015), range restriction could speculatively also result in some statistical issues with the use of MIN. Finally, though we have focused on MIN given its prevalence in the literature, we note that there are other forms of indirect measures that may potentially yield different results from MIN.² These are important issues that remain to be addressed in future work.

Though our primary aim is methodological, we briefly discuss theoretical implications. Firstly, strong positive associations between mixed emotions and psychological conflict support existing perspectives that conflict is the central characteristic of mixed emotions (van Harreveld et al., 2009; Vaccaro et al., 2020). Secondly, our findings conceptually replicate evidence that naturalistic experiences of mixed emotions may interfere with eudaimonic indicators of well-being (Oh, 2022), though we found consistent associations primarily with coherence but not other components of well-being. This is also in line with the idea that mixed emotions fundamentally involve conflict, which suggests that any detrimental implications of mixed emotions could occur primarily on dimensions of well-being involving a sense of conflict or inconsistency, such as coherence. Some evidence has also linked mixed emotions negatively to hedonic measures of well-being such as life satisfaction (Sun et al., 2021), and further research is needed to identify other conditions under which mixed emotions may or may not interfere with well-being outcomes (e.g., Berrios et al., 2018). Thirdly, we did not replicate evidence linking mixed emotions to increased receptivity (Rees et al., 2013; Oh and Tong, 2021). One possibility is that mixed emotions in daily life may primarily be conflicting and may interfere with integrative effects like receptivity (Oh, 2022). Alternatively, receptivity effects could be limited to certain contexts. Oh and Tong (2021) examined mixed emotions arising from the COVID-19 pandemic and Rees et al. (2013) specifically examined the mixed emotion ‘happy-sad’, while the present study examines generalized mixed emotions from daily life. Future research should examine these possibilities.

Overall, we recommend transparently reporting both direct and indirect measures wherever appropriate to provide comprehensive evidence, especially in studies where the goal is in detecting the occurrence of mixed emotions (Berrios et al., 2015). In predictive models, our findings support these measures as being relatively unconfounded by biases due to trait cognitive complexity, social desirability, and acquiescence, though they also advise interpretive caution when performing regression analyses with indirect measures while controlling for positive and negative emotions.

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found in the article/Supplementary material.

The studies involving humans were approved by National University of Singapore Departmental Ethics Review Committee. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

The author confirms being the sole contributor of this work and has approved it for publication.

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.