The base ratings for static emojis referred to the LEED database (Rodrigues et al., 2018), as they included emojis for the largest number of operating systems (iOS, Android) and took into account the matching of emojis across operating systems with canonical systematic ratings, which could be used in our experiments to control for stimulus features. Taking into account possible differences between cultures and subjects, in our previous pre-experiment, we investigated user’s ratings for emojis with similar ratings in the database, which were often used and are not offensive. Finally, we chose as emoji material four iOS facial emojis distributed in the four quadrants of the emotional space in the LEED database. Their valence and arousal scores were shown in Figure 1 (ranging from 1 to 7). The representation and abbreviations of the four quadrants referred to the study of Koelstra et al. (2012), High-arousal and high-valence (HAHV), e.g., Tears of Joy (A: 6.11, V: 5.87). Low-arousal and low-valence (LALV), e.g., Expressionless (A: 3.00, V: 3.15). High-arousal and low-valence (HALV), e.g., Angry (A: 6.25, V: 2.05). Low-arousal and high-valence (LAHV), e.g., Sleeping (A: 2.13, V: 5.08).

Combining the motion effects of existing dynamic emoticons, eight types of motion that do not produce deformation in kinetic animations were selected for exploration (Harrison et al., 2011), where animations with similar motion paths were not repeatedly selected. The motion effects we selected include Hanging Sign (HS), Whole Icon Wave (WIW), Tick-Tock (TT), Heart Beat (HB), Steam Engine (SE), Spin (S), Bounce (B), and Shake No (SN), as shown in Figure 2. Hanging Sign and Whole Icon Wave have a 30-degree left/right oscillation angle. Tick-Tock has a 15-degree in situ oscillation angle. Heart Beat has a 1.2× magnification. Steam Engine and Spin have a counterclockwise rotation. Bounce and Shake No have a 1.5× stroke. The duration of the motion was chosen to be 2 s. We set four rhythms, which are the slowest rhythm (R1), the slower rhythm (R2), the faster rhythm (R4), and the fastest rhythm (R8), as shown in Figure 3. Finally we used After Effect to create dynamic effects for each of the four emojis, resulting in our final set of 128 dynamic emojis.

A total of 20 participants (10 males, 10 females) were recruited for the experiment. Ages of the participants ranged from 21 to 30. Every participant regularly used a smartphone and had no documented sensorimotor disabilities. After the trial, they were compensated appropriately.

The experiment used a smartphone (an iPhone 12 with a 6.1-in touchscreen) with a resolution of 1,170px × 2,532px, and dynamic emojis were displayed sequentially in the center of the phone screen, where participants could tap to play to see dynamic emojis.

Every dynamic emoji was played in the video format with a frame rate of 60 fps and an emoji size of 80px × 80px, as shown in Figure 4. The experiments were conducted using a mixture of within-subject and between-subject design experimental methods, including four emojis, and the stimuli were divided into eight groups (HS, WIW, TT, SE, S, B, SN, and HB) according to the motion effects, and each group included four different animation rhythms (R1, R2, R4, and R8). For the four emojis, this was an 8 × 4 mixed design experiment with a total of 128 stimuli.

The entire experiment was divided into two parts: a subjective questionnaire and a semi-structured interview. Participants were seated in chairs in an empty room, and the experimenter explained the concepts of valence and arousal. Participants were informed that the main purpose of the experiment was to rate the valence and arousal elicited by each dynamic emoji on a Likert scale (1–7). Participants were free to express their opinions at any time during the experiment, and a semi-structured interview was held at the end of the experiment. Before the formal experiment, we randomly selected 12 dynamic effects from four emojis as a training group to help users familiarize themselves with the testing process and establish their own rating criteria.

The experimenter asked the participants to hold the experimental cell phone in the customary position, as shown in Figure 5, and then click to play the dynamic emoji. Participants told the experimenter their valence and arousal ratings of the current dynamic emojis and could make free comments, then the experimenter recorded the scores and ratings before evaluating the next dynamic emojis. During the experiment, a 1-min break was required for each completed group to avoid perceptual bias caused by fatigue. For each participant, the motion effect order and the rhythm order within the group are random, and the order in which the emojis appear is “Tears of Joy,” “Expressionless,” “Angry,” and “Sleeping.” After rating, participants also conducted a structured interview. The average duration of the experiment for each participant was 50 min.

The data analysis was conducted using IBM SPSS Statistics software. A repeated measures ANOVA was performed to examine the effects on valence and arousal. The factors included motor effect (HS, WIW, TT, SE, S, B, and SN) and rhythm (R1, R2, R4, and R8). The significance of the main effects of motor effects and rhythms was tested, and the significance of the interaction was tested. The Mauchly W sphericity test was carried out, and the results of the one-way ANOVA were used for the data satisfying the spherical hypothesis of p > 0.05, and the results of the multivariate test or the correction results of the one-way ANOVA were used for the data of p < 0.05 not satisfying the spherical hypothesis. F-statistics were validated using Greenhouse–Geisser corrected degrees of freedom, and paired Bonferroni corrected t-tests were used for post hoc comparisons.

The main effect tests and interaction effect tests affecting the valence and arousal of the four emojis are shown in Table 1. The mean and standard deviation of the valence of the four emojis with different rhythms under different motion effects are shown in Figure 6, and the mean and standard deviation of arousal are shown in Figure 7.

During the experiment, we collected unstructured feedback from the participants about how they rated the favorability of the emojis and how they felt while rating dynamic emojis. This section will process and analyze these qualitative results.

Participants selected their favorite dynamic emoji for each motion effect. Among HS, S, SE, and SN, four participants dropped their choices, resulting in 152 votes. The votes are shown in Table 5.

HAHV emojis received the most favorable ratings. “Tears of Joy” (76) > “Sleeping” (35) > “Angry” (31) > “Expressionless” (11). In terms of rhythm type, R4(29) > R2(25) > R8(12) > R1(11) for “Tears of Joy,” R2(18) > R4(9) > R1(3) > R8(1) for “Angry,” R1(28) > R2(6) > R4(1) > R8(0) for “Sleeping,” and R2(5) > R4(5) > R8(1) > R1(0) for “Expressionless.”

Participants also provided feedback on their perception of the dynamic effects. In WIW, half of the participants thought that “Tears of Joy” was joyful and relaxing. Three participants chose “Sleeping” with R1, and P7 (Participant Seven) mentioned that it looked like an old person taking a nap. Among motion effects, HB-R1 was more consistent with the sense of motion of sleeping, and participants thought that the motion effect of “Sleeping” should be slow. P5 thought that “Sleeping” with R1 was very relaxing, much like the breathing of a sleeping baby, and more vivid and immersive than the static emojis. In TT, P3, who selected “Expressionless” with TT-R4, thought that the left-to-right flick looked like an oddity. Those who chose “Sleeping” said it looked like a naughty person swaying with his eyes closed. In HS, four participants mentioned that they felt some oppression, especially when matched with “Angry” and “Sleeping.” They thought that there was a feeling of being hung. In S, most participants reported that they felt some dizziness and could not see the emojis when the rhythm was accelerated. Some other participants thought that this group of motion effects had no clear meaning. In terms of favorability, no one chose high-valence emojis with S either. In SN, nine participants mentioned that they thought the swaying “Angry” could be used to express their true emotions in social situations and that they thought “Angry” with SN-R2 could be a moderate explanation for their “Angry” emotion. Five of the participants expressed their dislike for “Tears of Joy” with SN, mentioning that the left-to-right panning of “Tears of Joy” seemed contradictory and mocking to them. This explained the reason for the low rating of high-valence emojis with SN: the fast-moving, negative-meaning motion effect affected the evaluation of high-valence emojis, resulting in low valence ratings. In B, participants who chose “Tears of Joy” thought the up-and-down bobbing felt like laughing backwards and forwards. In SE, four participants mentioned that SE looked a bit exaggerated. The only two participants who chose “Expressionless,” P6, said that moving “Expressionless” relieved the feeling of silence and made it more interesting, like craning the neck.

The effects of rhythms on arousal for all dynamic emojis were significant and positively correlated. As the rhythm accelerated, the arousal ratings consistently and significantly increased. Rhythm is an expression of the time and speed of motion, and the perception of rhythm could be changed by adjusting the interval of motion in animations. Our experiments adjusted rhythm by varying the repetition interval time and speed of different motion effects, and the results showed that emojis with the fastest rhythm (R8) had higher arousal and emojis with the slowest rhythm (R1) had lower arousal. This finding is consistent with the study of online animated advertisements by Sundar and Kalyanaraman (2004), which mentioned that animations with a faster rhythm had higher arousal. We discuss more rhythm types to further confirm the effect of rhythms on arousal.

The effects of rhythms on the valence of all emojis were equally significant, and we also observed different trends in the valence ratings of rhythms for emojis with different emotional types. In the experiment, the valence ratings of LAHV emojis increased as rhythms increased, being most pleasant at the faster rhythm (R4) and starting to decrease at the fastest rhythm (R8). In the study by Kim et al. (2020), the emotion of “vigor” was stronger in short intervals, and “vigor” can be classified as a high-arousal, high-valence form of stimulation. This is similar to our “Tears of Joy.” When short-interval fast rhythm dynamic effects match HAHV emotions, similar to the mechanism of fast rhythm short videos, users secreted more dopamine in a short period of time and felt more pleasure (Meng and Leung, 2021). But the valence rating of the fastest rhythm (R8) became lower, which we believe was related to the speed of the rhythm being too fast. It suggests that the use of interval time should also be controlled within a reasonable range when designing dynamic emojis. In addition, HAHV emojis need to be matched with the faster rhythm’s dynamic effects.

The valence ratings of LAHV emojis decreased as rhythms increased; the valence ratings of the slowest rhythm (R1) were the highest, and the valence ratings of the faster rhythm (R4) were the lowest. In films or cartoons, the rhythm of motion is more rapid for positive emotions such as happiness and victory and generally slower for emotions such as quietness and frustration (Luhta and Roy, 2013). A sense of ambivalence arises when the emotional quadrant in which the emoji itself resides does not align with the emotion conveyed by the rhythm. Aldunate et al. (2018) argued that negative emotions were perceived more strongly for high-valence emojis with inconsistent content and emoji emotions. This is also reflected in the emotional congruence of dynamic effects with emojis. When the accelerated rhythm produced higher arousal, the user felt ambivalent about the low-arousal emojis, which led to unpleasant feelings. Therefore, when matching dynamic effects for LAHV emojis, the slowest rhythm’s dynamic effects should be considered.

Low-arousal and low-valence emojis significantly showed the lowest valence ratings only at the fastest rhythm (R8). The appearance of this result is related to the paradoxical feeling of the fastest rhythm (R8) and low-arousal mentioned above, where the user produces a more unpleasant feeling. “Expressionless” belongs to a neutral category among emojis, but they often have negative meanings (Kralj et al., 2015), and the perceived emotional intensity of neutral emojis was not evident in many experiments (Fischer and Herbert, 2021). Subjects’ emotional perceptions of the LALV emoji varied the most and were closer to the neutral region. We think the first reason is the difference in emotion perception due to cultural differences, and the second is the limitation of the emoji itself, which carries a certain level of arousal because it seems to be a more interesting expression of emojis than just words or images, so it is difficult for the arousal of LALV to fall significantly below a certain level, as reflected in the U-curve (Kutsuzawa et al., 2022). But our results provided at least one point to avoid: Avoid too fast a rhythm for neutral emojis or LALV emojis.

High-arousal and low-valence emojis had the highest valence ratings at the slower rhythm (R2), and the valence ratings decreased as the rhythm became faster (R4 and R8). The faster the rhythm, the higher the arousal becomes. The faster rhythm will amplify the displeasure of low-valence emojis, so users tend to like a slower rhythm’s dynamic effect. However, when combined with the contradictory emotions elicited by the slow rhythm and high-arousal emojis, as well as the user’s evaluation consideration, emojis that are excessively slow may lead to more ambiguity for the user and can also intensitfy the unpleasant feeling. Our results suggest that a slightly slower rhythm’s dynamic effect be designed for HALV emojis.

For all dynamic emojis, the arousal of TT was significantly lower than that of S. TT had the smallest angle of motion (15 to −15°), while S had a larger angle of motion. Rimé et al. (1985) mentioned that in emotional perception, it was not the shape of the object presented that was important, but the kinetic structure of the motion. This perception is caused by a joint change of speed and direction, so S with a greater change of motion and the fastest rhythm (R8) has a greater sense of visual stimulation for the user and so produces a higher arousal.

In terms of valence, only some of the motion effects had a significant effect on the valence of high-valence emojis but not on the valence of low-valence emojis. Although we did not find a significant association between the motion effect and the low-valence emojis, we determined the valence tendency of the different motion effects in the subsequent user evaluations.

For HAHV emojis, the valence of WIW was the highest in the first three rhythms (R1, R2, and R4). Compared to static emojis, the valence of WIW-the faster rhythm (R4) had the most significant increase in HAHV emojis, which was the more preferred effect in subsequent user evaluations. The valence of S-the fastest rhythm (R8) was significantly lower than that of the first three rhythms for HAHV emojis. Combined with the users’ evaluations, the reason is that after the rhythm is accelerated, fast speed attracts attention (Sundar and Kalyanaraman, 2004), users pay more attention to the way the emojis move, and too fast rhythm makes users unable to recognize the emotional meaning of the emojis themselves, so the decrease in valence is more obvious. In addition, only the valence of TT-the fastest rhythm (R8) was significantly higher. The change in visual stimulation was less pronounced for the smaller motion effect after accelerating the rhythm compared to the other motion effects. We speculate that the reason for this is that the fast rhythm compensates for the visual stimulation caused by the small amplitude of motion, resulting in higher valence ratings. For LAHV emojis, HB-the slowest rhythm (R1) significantly increased the valence of LAHV emojis, and users agree that the effect of it can produce the feeling of sleep and light breathing. WIW-R1 and TT-R1 also produce relatively more pleasurable feelings for the user. In post hoc comparisons of the slowest rhythm (R1) and the slower rhythm (R2), the valence of HB, WIW, and TT were higher than that of SN and S. TT also exhibited higher valence than HS in the faster rhythm (R4) and the fastest rhythm (R8). We found that WIW, TT, and HB moved less compared to SN and S. SN carried a negative connotation, and S rotated at too large an angle to give the user a metaphorical feeling that was not positive, so this difference grew as the rhythm accelerated. Users also mentioned that the HS had a sense of suspension, and although the HS moved with the same magnitude as the WIW, the difference in the center point of motion led to a negative feeling for the user.

We found that rhythms and motion effects could modulate high-valence emojis to produce a greater range of valence variation. As shown in Figures 6A,D has a greater range of valence variation, while Figures 6B,C has a smaller range of variation. For low-valence emojis, the rhythms affect their valence ratings at some level but produce a smaller range of valence change.

In general, motion effects can modulate arousal levels to some extent, and motion effects modulate the valence of high-valence emojis significantly but not for low-valence emojis. When matching motion effects to high-valence emojis, it is necessary to consider a combination of motion amplitude and emotions originating from the center of motion.

Users’ evaluation of favorability is more based on the valence perception of emojis, and HAHV emojis received the most favorability. In addition, users prefer emotionally connected and metaphorical emojis, while emojis that are unrecognizable or do not conform to dynamics can be confusing to users. Heider and Simmel (1944) found that users attribute different intentions, attitudes, and emotions to the anthropomorphism of moving objects, and that this anthropomorphism can assign human-like emotional and behavioral properties to moving objects. The type of emotion elicited varies according to the motion effects (Rimé et al., 1985). There were also differences in the feelings produced by the eight motion types in our study. Every user who chose HAHV emojis (e.g., “Sleeping”) with HB-R1 mentioned that it looked like a small baby’s breathing and was very quiet. The anthropomorphism generated by emojis was enhanced by the fact that users assigned human-like emotional and behavioral properties to the emojis, and that they were positive emotional metaphors. Similar motion effects also include WIW and TT. S is the motion effect that users associate with the least description because S does not match the realistic way of face motion. It is difficult to generate specific anthropomorphic meanings and situational metaphors, so it is confusing and unpleasant for the user. Additionally, HS has relatively lower likeability, as evidenced by the presence of more negative comments. Some users mentioned that they thought there was a feeling of being hung and that it created a sense of oppression. Furthermore, SN received more negative comments, while B and SE received more neutral comments, indicating a stronger anthropomorphic effect.

Some users also expressed their preference for combining interesting motion effects with low-valence emojis (e.g., “Angry,” “Expressionless”) for some novelty and excitement. For example, in SN and TT, some users also selected low-valence emojis because they perceived “Angry” with SN as a moderate expression of their feelings, and found the side-to-side swaying of the “Expressionless” emoji with TT to be playful. This phenomenon may shed light on the popularity of emojis, many of which exhibit a certain sense of contrast. This sense of contrast enhances the emotional richness for users (Zhou et al., 2017). However, it is worth noting that this point may contradict the earlier discussion on the inconsistency between emojis and dynamic effects. Exploring whether the ambiguity resulting from this contradiction is positive or negative could be a potential direction for future research.

In Figure 7, we can observe the black dotted line which mean the arousal rating of this static emoji. It shows that after adding the dynamic effect, the static emojis with high-arousal had lower arousal, while the emojis with low-arousal had higher arousal. Also, after adding the dynamic effect, the static emojis with high-valence have lower valence, the static emojis with HALV had higher valence, and the static emojis with LALV have lower valence. The addition of dynamic effects seems to have an impact on the valence of high-valence emojis. It is found that dynamics decreased the valence of high-valence static emojis instead of becoming more positive, which is different from the results reported in a previous study (Detenber et al., 1998). The reason is that we were exploring rhythms and motion effects that better matched emojis with different emotion types, so only some of the better matched emojis showed this trend, such as the HAHV emojis of WIW.

We proposed the following design principles for dynamic emojis:

Consider rhythms: emojis with high-valence and high-arousal (HAHV) are better suited to fast rhythm; emojis with high-valence and low-arousal (LAHV) are better suited to slowest rhythm; and emojis with low-valence and high-arousal (HALV) are better suited to slower rhythm; for emojis with low-valence and low-arousal (LALV), avoid using fast rhythm.