Besides brain activity, emotional experiences also induce affective expressions (Sander et al., 2018; Scherer and Moors, 2019). Facial expression is the most investigated and predictive aspect of emotional expressions (Scherer and Ellgring, 2007; Plusquellec and Denault, 2018). In comparison to measures of brain activity, facial expressions responses are typically video-based, which requires no measurement preparation or application, can be obtained in ecologically valid environments, and is even applicable to online research. In order to capture emotionally relevant information from the whole face, researchers have heavily relied on observation techniques such as the Facial Action Coding System (FACS) to score intensity estimates of single facial movements called Action Units (AU) (Ekman et al., 2002; Mauss and Robinson, 2009). FACS is an extensive coding manual which allows for a very detailed description of facial responses through the combination of AU and shows good to excellent inter-rater reliabilities (Sayette et al., 2001). However, human FACS coding results in low scaling resolution of AU intensities and it is very time consuming (Schulte-Mecklenbeck et al., 2017).

Although there are several important theories that explain specific aspects of emotional facial expressions [for an overview, see Barrett et al. (2019)], there are currently only two relevant empirical approaches that map combinations of specific AUs to specific emotional states. On the one side, basic emotion theory predicts robust AU patterns that cohere with a limited amount of distinct emotion categories (i.e., joy, anger, disgust, sadness, fear, and surprise; e.g., Ekman et al., 1987). However, there is evidence for universal facial expressions beyond six basic emotions (Cordaro et al., 2018, 2020; Keltner et al., 2019; Cowen et al., 2021). Moreover, there are spontaneous emotional responses that are be much more variable and less universal for a distinct AU pattern (Durán and Fernández-Dols, 2021; Le Mau et al., 2021; Tcherkassof and Dupré, 2021). On the other side, componential process theory predicts several appraisal dimensions that elicit specific AU combinations like valence, novelty, and control (Sander et al., 2018; Scherer et al., 2018, 2021).

Although electromyography (EMG) research extensively measured corrugator and zygomaticus activity to approximate a valence dimension (e.g., Höfling et al., 2020), the investigation of other components in this theory is still preliminary. Hence, there is currently no consensus about the link between meaningful AU combinations regardless of the assumption of dimensional or categorial underlying emotional processes.

Recent advances in technology have enabled the measurement of facial expressions to obtain emotion-associated parameters through automatic facial coding based on machine-learning (AFC; Pantic and Rothkrantz, 2003; Cohn and Sayette, 2010). AFC parameters include separate facial movements, such as Action Units derived from the Facial Action Coding System on the one side and integrated “emotion-scores” such as joy or anger estimations, on the other side. There is evidence that AU parameters estimated by AFC correspond with estimates of human FACS raters between 71 and 93%, depending on the specific measurement system (Bartlett et al., 1999; Tian et al., 2001; Skiendziel et al., 2019; Seuss et al., 2021). Moreover, AFC classifies basic emotional facial expressions with impressive accuracy in prototypical facial expressions in pictures (Lewinski et al., 2014a; Lewinski, 2015; Beringer et al., 2019; Küntzler et al., 2021) as well as videos (Mavadati et al., 2013; Yitzhak et al., 2017; Calvo et al., 2018; Krumhuber et al., 2021).

While attempts to validate this innovative technology mainly focused on highly standardized and prototypical emotional facial expressions in the past, the number of validation studies that approximate more naturalistic or spontaneous facial expressions is still preliminary. AFC is less accurate in the detection of less intense and more naturalistic expressions (Büdenbender et al., 2023), which is also a commonly observed pattern in human emotion recognition (Krumhuber et al., 2021). Some studies find evidence that AFC can be transferred to the facial expressions of naïve participants that mimic emotional facial expressions in a typical laboratory setting (Stöckli et al., 2018; Sato et al., 2019; Höfling et al., 2022). To a limited extent, AFC detects highly unstandardized emotional facial expressions of professional actors depicted in movies (Küntzler et al., 2021). Furthermore, AFC can also track spontaneous emotional responses toward pleasant scenes, where AFC parameters correlate with emotional self-report and direct measures of muscle activity with EMG (Höfling et al., 2020). However, AFC is not sensitive to very subtle emotional responses, particularly if participants are motivated to suppress or control their facial responses (Höfling et al., 2021). Taken together, there is evidence that AFC validly captures spontaneous emotional states in a typical laboratory setting, especially for pleasant emotional responses.

According to the affect-transfer hypothesis (Brown and Stayman, 1992), there is evidence for a relationship between consumers’ emotional responses to advertisement stimuli and the subsequently elicited advertisement and brand effects. In this model, an advertisement elicits emotional responses which influence the attitude toward the advertisement (i.e., advertisement likeability). In the following step, a favorable ad likeability leads to changes in the attitude toward the brand (brand likeability), which increases the purchase intention of products and services of a particular brand in the final step of this process model.

AFC has been used successfully to predict the effects of video commercials’ on the subsequent processes of this advertisement and brand effect framework, such as the self-reported emotional response, advertisement likeability, as well as brand likeability and purchase intention. In the domain of political influencing, AFC measures have been shown to correspond with emotional self-report of pleasant (Mahieu et al., 2019) as well as unpleasant advertisements (Fridkin et al., 2021). They are also predictive to measure intended emotional responses in an a priori defined target audience (Otamendi and Sutil Martín, 2020). Accordingly, AFC of smiling intensity correlates with advertisement likeability (Lewinski et al., 2014b; McDuff et al., 2014, 2015), brand likeability (Lewinski et al., 2014b), and the purchase intentions of advertised brands (Teixeira et al., 2014; McDuff et al., 2015). Furthermore, increased smiling was also found to reduce zapping behavior (Yang et al., 2014; Chen et al., 2016), decrease attention dispersion (Teixeira et al., 2012), and predict long-term attitude changes (Hamelin et al., 2017). Taken together, there is evidence that AFC of smiling predicts several steps of the advertisement and brand effects proposed by the affect-transfer hypothesis. In contrast to AFC, approaches to measure advertisement effects with human FACS were less successful (e.g., Derbaix, 1995).

The scientific knowledge regarding a direct link between emotional facial expression on the one side and advertisement or brand effects on the other side is very limited due to the following reasons: First, it is unclear whether facial expressions significantly predict advertisement and brand effects beyond relevant precursor self-report dimensions. Such quantification is only possible if statistical models of facial expression parameters are compared with models that control for relevant self-report, which has not been investigated in previous research. Second, relevant research mainly used integrated parameters for joy or entertainment, relying almost exclusively on measurements of smiling (AU12). Reporting such integrated scores serves a lower level of scientific transparency in comparison to a description with AU terminology because it is largely unknown how AFC classifiers are trained and, correspondingly, how integrated parameters are estimated. As an additional consequence, it is unclear how different AUs that are relevant for emotional facial expressions ensembles to predict advertisement and brand effects of video commercials’ effectiveness beyond smiling. Third, there is no consensus on whether the degree of amusement or entertainment in video commercials shows a linear or non-linear relationship on branding effects: While Lewinski et al. (2014b) report a weak linear relationship between AFC of smiling and brand effects, Teixeira et al. (2014) report an inverted u-shaped relationship between AFC of smiling and brand effects. Finally, there is no study available that investigates the relationship between emotional facial expressions and all relevant steps of advertisement and brand effects according to the affect-transfer hypothesis in a within-subject study design.

In order to close these existing research gaps, we investigated the predictive value of facial expressions while viewing video commercials to forecast all subsequent components of the affect-transfer model of advertisement and brand effects: emotion ratings, ad likeability, changes between pre- and post-measurements of brand likeability, and purchase intention. To this end, we broaden the spectrum of analyzed movements in the face to 20 AU that can be measured with a state-of-the-art AFC algorithm to predict relevant outcome criteria (see Figure 1 for an overview of measured AUs and self-report ratings). In addition, we aim to determine non-linearities between facial expressions and self-report with semi-parametric additive mixed models that can account for non-linear effects (Wood, 2006; Wood et al., 2013; Wood and Scheipl, 2020). We expect strong relationships between self-reported emotion ratings and advertisement likeability and changes in brand likeability as well as purchase intention. In order to collect data from a variety of industries and emotional content, we established different stimulus groups which only differed in terms of advertisement videos and corresponding brand stimuli. This is the first study that investigates the relationship between emotional facial expressions measured by artificial intelligence and all relevant components of advertisement and brand effectiveness proposed by the affect-transfer framework.

A total of 257 volunteers were randomly assigned to one of eight stimulus groups. General exclusion criteria were age under 18, acute psychoactive medication use, acute mental disorder episode, severe somatic disease, and wearing glasses. Participants with corrected-to-normal vision were asked to wear contact lenses during the experiment. After visual inspection of the analyzed videos, data from 38 participants were excluded because of face detection problems of the AFC software (e.g., partially covered face with the hand, scarfs, and other accessories or large body movements that lead to false face detection). Hence, we only used data from participants with good recording and analysis quality, resulting in an overall sample of 219 (115 females) participants of mainly Caucasian descent. Age ranged from 19 to 58 years (M = 23.79, SD = 5.22). All participants received compensation of either 8€ or student course credit, and they signed informed consent before the data collection. University Research Ethics Committee approved the experiment.

Participants filled in various questionnaires to compare relevant states and traits related to emotional responsiveness and expressivity to ensure comparability between experimental groups. After a socio-demographic questionnaire (e.g., gender, age, and educational level), the Social Interaction Anxiety Scale (SIAS; Stangier et al., 1999), the State-Trait Anxiety Inventory (STAI State and STAI Trait; Laux et al., 1981), the Positive-and-Negative-Affect-Schedule (PANAS PA and PANAS NA; Krohne et al., 1996), the Self-Rating Depression Scale (SDS; Zung, 1965), the Berkley Expressivity Questionnaire (BEQ; Mohiyeddini et al., 2008), and the Behavioral Inhibition and Activation Scale (BIS and BAS; Strobel et al., 2001) were administered before starting the main experiment.

Following informed consent and completion of the questionnaires, participants were seated in front of a computer screen. In order to collect data from a variety of industries and emotional content, we established eight different stimulus groups to which participants were randomly assigned. These groups only differed in terms of advertisement videos and corresponding brand stimuli; all other aspects remained constant between groups. Before the main experiment started, participants were instructed to maintain a neutral facial expression for 10 s, which was later utilized to calibrate the AFC analysis individually. The main experiment comprised three experimental blocks: pre-evaluation of brands, viewing video advertisements and evaluation, and post-evaluation of brands.

The pre- and post-evaluations of brands were set up identically, and hence, effects on brands elicited by the video advertisement can be traced by pre-post rating changes. The logo of the eight advertised brands was also presented in both brand evaluation blocks, each started with a 1 s fixation cross. The presentation duration of the brand logos lasted until participants decided when to proceed with several ratings of a specific brand by pressing the space bar. All scales were presented as a nine-point semantic differential. Participants rated their familiarity with the brand (1 = familiar and 9 = unfamiliar), brand likeability (1 = like and 9 = dislike; 1 = good and 9 = bad), and brand purchase intention (1 = probable purchase and 9 = unprobeable purchase; 1 = purchasing definitely and 9 = purchasing definitely not) after the brand presentation. All advertisements were indicated by a 1 s fixation cross in the advertisement block. After watching a particular video, each video was rated on nine-point semantic differentials containing advertisement familiarity (1 = familiar and 9 = unfamiliar) and advertisement likeability (1 = like and 9 = dislike; 1 = good and 9 = bad). In addition, participants rated how they felt during each video presentation on several one-item emotion scales (i.e., joy, sadness, anger, disgust, fear, surprise; 1 = strong emotion and 9 = no emotion) immediately after the video presentation. All scales were inverted to improve the readability of the results. If two items were used to measure a construct (i.e., advertisement likeability, brand likeability, brand purchase intention), the average of both items would have been calculated. Internal consistencies were excellent for these three scales (Cronbach’s α > 0.90).

Each group watched eight different commercials and corresponding brand logos in randomized order. The video material was selected from the list of award-winning commercials at the Cannes Lions International Festival of Creativity between 2016 and 2017. The videos were counterbalanced between groups in terms of video duration, familiarity with the brand, familiarity with the video, and emotional content (see also Section “Results”). Appendix 1 contains all commercial video advertisements sorted by groups. Two other non-commercial video advertisements per group were presented but not included in the analysis because the brand ratings did not apply to the non-commercial section.

High-precision software (E-Prime; Version 2.0.10; Psychology Software Tools, Pittsburgh, PA, USA) was used to run the experiment. Stimuli were shown centrally on a 19-inch monitor with a resolution of 1,024 × 768, approximately 70 cm away from the participant. Optimal illumination with diffused frontal light was maintained throughout. Videos of participants’ faces were recorded with a Logitech HD C615 web camera placed above the computer screen. Videos were processed off-line with FaceReader Software (FR; Version 7.0, Noldus Information Technology) and further analyzed with Observer XT (Version 12.5, Noldus Information Technology). Furthermore, Observer XT synchronized stimulus onset trigger from E-Prime and the video recordings. FR is a visual pattern classifier based on deep learning approaches and extracts visual features from videos frame by frame. In accordance with neuro-computational models of human face processing (Dailey et al., 2002; Calder and Young, 2005), FR detects facial configurations in the following steps (Van Kuilenburg et al., 2005, 2008): (1) The Cascade classifier algorithm finds the position of the face (Viola and Jones, 2004). (2) Face textures are normalized and the active appearance model synthesizes a digital face model representing facial structure with over 500 location points. (3) Compressed distance information is then transmitted to an artificial neural network. (4) Finally, the artificial neural network connects these scores with relevant emotional labels through supervised training with over 10,000 samples (pictures) of emotional faces to classify the relative intensity of a given facial configuration. As a result, FR estimates activity of 20 AU, which includes AU01 Inner Brow Raiser, AU02 Outer Brow Raiser, AU04 Brow Lowerer, AU05 Upper Lid Raiser, AU06 Cheek Raiser, AU07 Lid Tightener, AU09 Nose Wrinkler, AU10 Upper Lid Raiser, AU12 Lip Corner Pull, AU14 Dimpler, AU15 Lip Corner Depressor, AU17 Chin Raiser, AU18 Lip Puckerer, AU20 Lip Stretcher, AU23 Lip Tightener, AU24 Lip Pressor, AU25 Lips Part, AU26 Jaw Drop, AU27 Mouth Stretch, and AU43 Eyes Closed. The estimated parameter of each AU ranges from 0 to 1. FR measures were calibrated per participant based on the baseline measurement at the beginning of the experiment. “East-Asian” or “elderly” face models were presented instead of the general face model as recommended by the user manual. For the duration of each video, the mean and peak activity for all AUs were exported and analyzed.

Across all participants (N = 219) and commercial video stimuli (n = 64; eight per participant), 1,752 data points were collected on advertisements and corresponding brand ratings. Our data analysis required several steps: analysis of participant characteristics, aggregation of video- and brand-wise means for analysis of stimulus characteristics, and also to report correlations between facial expression parameters and relevant self-report ratings, as well as semi-parametric mixed additive regression models to predict emotion, advertisement and brand effects based on facial expression parameters.

First, we calculated ANOVA with the factor stimulus group (eight groups) to determine differences in participant characteristics separately for STAI State, STAI Trait, SIAS, BIS, BAS, PANAS PA, PANAS NA, BEQ, and SDS. In addition, we report differences in gender distribution in stimulus groups with the Chi-Squared test.

Second, we calculated pre-post difference scores for brand likeability and brand purchase intention and averaged relevant self-report ratings and AU parameters separately for each video and the corresponding brand. On the one side, we determined differences in stimulus characteristics based on stimulus-wise averages for the stimulus groups with ANOVA for video duration, brand familiarity (pre-rating), video familiarity, and emotion ratings (i.e., joy, sadness, anger, fear, disgust, and surprise). We applied a Greenhouse–Geisser correction for ANOVA when appropriate. Eta-squared (η²) is reported as an effect size for F-tests (η² ≥ 0.01 small; η² ≥ 0.06 medium; η² ≥ 0.14 large; Pierce et al., 2004).

Third, we identified AU that showed zero or near-zero variance to exclude AU that will not contribute to the prediction of relevant outcomes. This exclusion criterion applied for eight AU, in particular, AU01 Inner Brow Raiser, AU02 Outer Brow Raiser, AU09 Nose Wrinkler, AU10 Upper Lid Raiser, AU18 Lip Puckerer, AU20 Lip Stretcher, AU26 Jaw Drop, and AU27 Mouth Stretch, which were excluded from further analysis. Further, we report spearman’s rho correlations between relevant AU and self-report based on averages for each video and the corresponding brand. Effect sizes were interpreted following Cohen (1988): r ≥ 0.1 small, d ≥ 0.3 medium, and d ≥ 0.5 large.

Fourth, all self-report ratings were z-transformed on the group level for better interpretability and comparability of the effects. To resolve the limitation of the one-item scale of emotion ratings for joy, z-scores are calculated based on individual participant ratings. In contrast to self-report ratings, AU variables were not transformed in any way because of their scale properties (e.g., exact zero-point) and correspondence to the intensity measurement of the Facial Action Coding System: Values from >0.00–0.16 are classified as trace (E), 0.16–0.26 as slight (D), 0.26–0.58 as pronounced (C), 0.58–0.90 as severe (B), and 0.90–1 as max intensity (A).

Fifth, as predictive models, we carried out semi-parametric additive mixed modeling with the R-package “gamm4” (Wood and Scheipl, 2020). In the first step, we developed a basic model controlling gender and stimuli as fixed factors and the participants and stimulus group as random factors. Next, we calculated main effect models for all AU separately for peak and mean activity to predict joy ratings, advertisement likeability, brand likeability change, and brand purchase intention change. Furthermore, we estimated the combined effect of AU and relevant rating scales to determine the relative predictiveness of AFC versus self-report resulting in 17 independent models. Visualization of the most substantial effect patterns is presented as smoothed effect plots with 95% confidence intervals. We report R²_adj, Akaike-Information-Criterion (AIC), and Bayesian-Information-Criterion (BIC) as a goodness of fit indices (Burnham and Anderson, 2004). According to Cohen (1988), we interpreted the adjusted R² ≥ 0.01 as small, R² ≥ 0.09 as moderate, and R² ≥ 0.25 as a large proportion of explained variance of each predictive model. A large change in model fit was interpreted by an absolute change of 10 for AIC and BIC. The significance level was always set to α = 0.05.

Spearman correlations based on unstandardized average values per stimulus between emotion, advertisement, brand ratings, and AU intensity for mean and peak activity over the video duration are reported in Table 1. There were strong and positive correlations between facial expression measures of AU6, AU12, and AU25 for ratings of joy as well as surprise for both mean and peak AU activity. Although feelings of surprise can be elicited by pleasant to unpleasant emotional events, the advertisements presented in this study elicited the same correlational patterns of facial activity for higher surprise and higher joy ratings. Therefore, these two emotion ratings might be confounded in the present study design, probably because the videos mainly triggered pleasant emotions. Hence, we focused on joy ratings as a self-report measure of emotion elicited by video commercials in the main analysis (i.e., semiparametric models of AU activity).

We fitted separate semiparametric additive mixed models to account for non-linear relationships while controlling for gender and stimuli as fixed factors and the participants and stimulus group as random factors. In the first step, we calculated regression models exclusively based on self-report ratings to test for specific relationships proposed by the affect-transfer hypothesis (see Table 2). It is evident that advertisement likeability is strongly predicted by joy ratings, brand likeability change is only significantly predicted by advertisement likeability, and purchase intention change is strongly predicted by brand likeability. This pattern strongly supports a hierarchical influence of advertisement and brand effects as postulated by the affect-transfer hypothesis.

Next, we fitted separate models to predict joy ratings (Table 3), advertisement likeability (Table 4), brand likeability change (Table 5), and purchase intention change (Table 6) based on mean and peak AU activity. In addition, we estimated the combined effect of AU and rating scales to determine the relative predictiveness of AFC versus self-report for the models that contain advertisement and brand effect self-report ratings.

One aspect of the present study is the exclusive use of self-report ratings as the criteria in a cross-sectional design. The usage of ad hoc self-report scales has two significant limitations: First, it is unclear how stable reported advertisement effects on relevant brand dimensions are over time. Long-term effects of advertisement might be explored through longitudinal study design in future research, for example, by inviting participants again weeks or months after the main experiment to probe the stability of brand likeability and purchase intention changes. Second, it is unclear whether psychologically assessed intentions to purchase products or services of a particular brand elicit an actual purchase behavior. Future research should focus on predictions of actual behavior or even population-wide effects like it is approached with other methods in the consumer neuroscience literature (Berkman and Falk, 2013). Hence, out-of-sample criteria in the consumer research area that facial expression parameters might predict advertisement effects on a market-level response, such as monetary advertising elasticity estimates (Venkatraman et al., 2015) or video view frequencies on media platforms (Tong et al., 2020). Furthermore, facial responses toward music and movie trailers could predict actual sales figures in the music and movie industry, as already demonstrated with measures of neural response (Berns and Moore, 2012; Boksem and Smidts, 2015). Hence, future research needs to explore the predictive capability of facial expression recognition technology beyond within-subject measured self-report.

Automatic facial coding has also some advantages in comparison to emotional self-report because it enables a passive and non-contact assessment of emotional responses on a continuous basis. In contrast, emotional self-report is typically rated after stimulus presentation, and hence, reflect a more global and possibly biased evaluation of the recipients (e.g., Müller et al., 2019). Furthermore, AFC provides a rich data stream of emotion-relevant facial movements, whereas self-report is typically assessed on a limited number of emotion scales. AFC technology enables a moment-to-moment analysis of elicited emotional responses, which allows for the assessment of emotional responses toward dynamic emotional content as in video commercials. For example, stories can have very different emotional dynamics such as an unpleasant beginning and a pleasant end and vice versa (Reagan et al., 2016). Hence, future research should investigate the differential impact of emotional dynamics of advertisement commercials and whether differences in the emotional dynamics affect relevant advertisement and brand effects.