In the art agency theory (Gell, 1998), both artistic and ethnographic artifacts are characterized as semiotic indexes (Esquenazi, 1997; Coupaye, 2017). In this framework, the index, by indicating the cause of its existence, conveys a capacity for action, or, more precisely, a potential “agency.” Here, agency denotes a relationship of connection with a social agent. According to Gell (1998), agency is ascribed to individuals or objects perceived as instigating “events to happen” in their surroundings through acts of mind, will, or intention.⁷ In this theoretical perspective, artifacts possess attributes akin to social agents. Although it might seem unconventional to attribute agency to artifacts, this concept aligns with the notion of secondary agency advanced by Gell (1998) and embraced by other anthropologists, such as Descola (2006). The core of Gell’s theory places social agency at the forefront of the relationship with artifacts. This relationship is built on the foundation of a cognitive approach that leads to the inference of intentions implied by the artifact through a delegation of intentionality. The artifact expresses an intentionality not inherently its own but rather one delegated by a social agent. Siri Hustvedt encapsulates this idea succinctly: “Visual art exists only to be seen. It is the silent encounter between the viewer, ‘I’, and the object, ‘it’. That ‘it’, however, is the material trace of another human consciousness. The artist, who is missing from the scene, has nevertheless left us a work, an act of pure will […] The painting carries within it the residue of an ‘I’ or a ‘you’. In art, the meeting between viewer and thing implies intersubjectivity. […] The intersubjectivity inherent in looking at art means that it is a personal, not impersonal act” (Hustvedt, 2006, xix; cited in Gallese, 2017).

In the theory of art agency, the connection engendered by the index is referred to as “abduction of agency.” This denotes that, from the artifact’s indexical function, the perceiver imputes a social agency to it and formulates inferences about the intentions conveyed by the artifact. As a physical interface, the artifact functions as a conveyer of intentions, reflecting desires to act. These intentions may be actual (stemming from the human creator of the artifact) or presumed (associated with entities believed to be the source of the artifact, or those represented by the artifact).

In Gell’s approach (Gell, 1998), the theory of mind hypothesis appears to be employed. The theory of mind involves the capacity to attribute mental states to oneself and others, also known as mentalization (Goldman, 2012). According to this hypothesis, all healthy human beings possess this faculty, primarily enabling them to understand and engage with others. Gell proposes that art utilizes this theory of mind for secondary, non-utilitarian purposes, such as when individuals project human mental states onto characters, whether animals or machines, in a narrative (Bloch, 1999). This cognitive perspective provides a unifying lens applicable across different cultures and eras. For instance, it allows for the exploration of relationships with robots, Flemish portraits, or Hopi kachinas alike. This is achievable because individuals in industrialized societies, 17th-century Northern Europe, or Hopi Native Americans share common cognitive mechanisms that enable them to recognize a particular agency in their artifacts. Gell’s arguments, as suggested by Bloch (1999), may draw on the work of thinkers like Dennett, leading us to consider that the theory of mind approach referenced by Gell aligns with the rationality-teleology theory (Dennett, 1987). The rationality-teleology theory, fundamentally philosophical, relies on thought experiments and extends the concept of theory of mind beyond attributing mental states solely to humans, encompassing various creatures, including animals and machines. This theory posits that a target is viewed as a rational agent that acts in accordance with its propositional attitudes. Based on its beliefs and desires, the target exhibits behavior that could be deemed rational. In this context, attributing rational behavior to a non-human target does not involve recognizing beliefs and desires in animals or machines as one does with humans. Instead, it involves attributing these mental states to the target based on the principles of the theory of mind.

Recently, Norman & Wheeler conducted an experimental study on the perception of several masks from various cultures (Norman and Wheeler, 2020). They demonstrated that the masks evoke strong perceptions of emotion with considerable variations. Participants evaluated the masks based on the six basic emotions: happiness, sadness, anger, fear, surprise, and disgust (Ekman, 1999; Ekman and Cordaro, 2011; Izard, 2011; Levenson, 2011; Panksepp and Watt, 2011). Not only were they able to discern emotions, but they could also perceive different intensities of these emotions. Similarly, clay figures from early Japanese cultures elicited emotional perceptions in the participants (Kawabata et al., 2021). Moreover, the more the figures were perceived as happy, the more they were rated as approachable, and conversely, the more they were perceived as fearful, the less they were rated as approachable. Norman & Wheeler assert that “the ability of the masks to produce effective perceptions of emotion was due to the artists’ inclusion of facial features that reliably signal emotions in everyday life” (Norman and Wheeler, 2020, p. 1). This observation could apply equally well to clay figures from early Japanese cultures.

The signals of emotions are produced by humans across cultures and societies. In the late ‘80s, a study already explored the universality of signals,¹¹ such as threat, through facial icons (Aronoff et al., 1988). This study showed that the emotional expression of threat in facial icons shares universal characteristics. American students were instructed to imagine themselves as bounty hunters from the New Guinea people, preparing for an expedition. They were then asked to draw the mask they would wear to intimidate their prey. The drawings from American students were compared with various masks from 16 different cultures, categorized as either threat-related or non-threat-related. Facial features depicting the expression of threat were identified in both the student drawings and the multicultural mask group. A more recent study further confirmed the universality of certain facial expressions (pain, anger, sadness, determination/stain, and elation) through an investigation into the perception of ancient American facial icons (Cowen and Keltner, 2020). The authors concluded (p. 4) in this study that “ancient American artists shared some of present-day Westerners’ associations between facial muscle configurations and social contexts in which they might occur.” These findings on the universality of facial expressions and emotions, spanning from contemporary Westerners to ancient and diverse cultures, support the notion that the perception of facial icons can be approached through emotion processes beyond ethnographic and cultural perspectives. It allows for the generalization of the emotion index property to a broad range of cross-cultural facial icons.

In the realm of Western art, studies have explored emotional reactions to various genres of face-representation artworks (Siri et al., 2020; Gallese, 2022), as well as reactions to face representations under different presentation conditions (Rychlowska et al., 2012). Research in the first category discovered that self-portraits elicited emotional experiences, with self-portraits being perceived as more intense in emotional expression and socially engaging compared to portraits (Siri et al., 2020; Gallese, 2022). Studies falling into the second category examined emotional responses to portraits under varying conditions of eye contact (Rychlowska et al., 2012). Artistic portraits with and without direct eye contact were presented to participants in two conditions: in the first, the represented subject’s eyes were obscured by a mask, and in the second, the represented subject’s eyes were visible. For each trial, participants answered the question “How emotional is the impact of the painting?.” The emotional impact was found to be greater when the eyes were visible compared to when the eyes were masked, particularly when eye contact was established (Rychlowska et al., 2012). This outcome suggests an embodied emotional reaction to portraits where the eyes play a crucial role.

We posit that facial icons, by more faithfully representing the attributes of a human being among artifacts, can be regarded as amplifiers of cognitive processes related to the theory of mind. This amplification does not rely on symbolic analogy but would be facilitated through the visual aspect of the depicted facial expressions, perceived and recognized by unconscious (cognitive) processes. The theory of mind we are invoking here is the simulation theory, characterized by “the capacity to represent and reason about others’ mental states” (Barlassina and Gordon, 2017). It involves attributing mental states by projecting oneself into the other, essentially occupying their perspective. Goldman encapsulates this concept by describing it as an attempt to generate mental states in oneself similar to those of the target (Goldman, 2012). In our application of the simulation theory, akin to Dennett’s utilization of the theory of mind (Dennett, 1987), we extend the human target “others” to the artifact target “facial icons.”

In a prior study (Achour-Benallegue et al., 2016), we observed correlations between participants’ felt emotions and their assessments of the intensity of expression in cross-cultural facial icons. The “felt emotion” variable in this study design represents a projection of the depicted emotion in facial icons, indicating the extent to which participants felt an emotion in themselves that aligns with the emotion portrayed by the facial icon. Given that the intensity of expression also reflects the intensity of the expressed emotion, this correlation suggests the presence of a simulation process toward facial icons, potentially indicating emotional contagion. Emotional contagion entails experiencing the subjective feeling state or somatosensory and motor experience of an emotion perceived in someone else (Hatfield et al., 1994; Lundqvist and Dimberg, 1995; Dimberg and Thunberg, 2012; Sato et al., 2013; Hess and Fischer, 2014; Hess, 2021). Furthermore, in another study (Achour-Benallegue et al., 2021), we explored the effect of the expression intensity of facial icons on arousal responses through subjective reports using the Self-Assessment Manikin scale. Our results demonstrated a higher self-arousal assessment among participants when exposed to intense images as opposed to neutral ones. This outcome might signify an emotional contagion suggesting a simulation of the perceived arousal affect. Emotional contagion is regarded as “basic building block of human interaction—assisting in “mindreading” (allowing people to understand and share the thoughts and feelings of others”; Hatfield et al., 2014, p. 159). When the “others” are facial icons, emotional contagion may indicate an ascription of psychological interiority to these artifacts.

Upon conducting additional analyses¹² on our earlier study (Achour-Benallegue et al., 2016), we observed that the felt emotion (which may refer to simulation) explains 49% of the attention devoted to the facial icon, whereas the esthetic value of the facial icon explains only 4% of the attention to the image. This suggests a significant involvement of simulated emotion in the artistic relation with facial icons, where their status as indexes of emotion appears to play a prominent role.

In human interaction, emotion simulation is not merely suggested by unembodied perception of emotion in facial expressions but is also triggered by the motor accomplishment of these expressions (see Figure 3; refer to Box 3 in Supplementary materials). Within the simulation theory, simulated mental states stem from sensorimotor processes¹³ that play a crucial role in facial expression processing. Motor simulation of facial expressions, that is allowed by the premotor cortex (PMC), the inferior parietal lobe (IPL), and the frontal operculum (FO), enables the recognition of others’ facial expressive movement (Gallese, 2014; Paracampo et al., 2017; Gallese, 2022). One of the essential processes that may be encompassed within sensorimotor processes is facial mimicry (Wood et al., 2016). Facial mimicry is the tendency of individuals to imitate others’ facial expressions (Hess and Fischer, 2014; Borgomaneri et al., 2020). It occurs unconsciously and spontaneously and is difficult to suppress (Dimberg et al., 2000; Bailey and Henry, 2009; Palagi et al., 2020). However, some studies showed that it may be moderated by contextual information (Murata et al., 2016; Arnold and Winkielman, 2020). Although contextual information plays a moderating role in facial mimicry, it remains a crucial factor in supporting the perception of emotional expressions. Disrupting or altering feedback from facial muscles and neural processes involved in facial mimicry impairs the processing of others’ expressions of emotion by reducing both speed and accuracy (Niedenthal et al., 2017; Borgomaneri et al., 2020). Facial mimicry plays a crucial role in shaping social and emotional interactions, immersing individuals in the simulation of another person (Beall et al., 2008; Hess and Fischer, 2013; Prochazkova and Kret, 2017). Additionally, it has the potential to result in emotional contagion (Prochazkova and Kret, 2017). For instance, when observing a smile on someone’s face, the observer may undergo the corresponding emotional experience involving the motor activation of the zygomaticus. This process involves the reconstruction of affective states associated with the sensorimotor states linked to the perception of the smile.

A recent study by Wessler and Hansen (2021) recorded facial EMG activity in response to face stimuli expressing various emotions (happy, sad, and angry), including stick figures and photographs of real humans. The study demonstrated congruent reactions of facial muscles to emotional expressions in both stick figures and photographs, emphasizing the role of mimicry even in situations with limited affiliative cues and reduced socially meaningful contexts. Stick figures not only triggered facial mimicry as much as did the photographs of real faces, but they also provided better material for recognizing emotions than photographs (Wessler and Hansen, 2021). In a previous study (Achour-Benallegue et al., 2021), we observed mimicry reactions, particularly of mouth expressions, in response to cross-cultural facial icons, suggesting that these icons might enhance a simulation process through sensorimotor reactions. In this study, mimicry reactions to the upper part of facial icons were challenging to observe, but findings on the effect of valence on muscle reactions suggested a possible tendency of mimicry reactions in the corrugator (activation of the corrugator muscle was observed in response to negative valence, while relaxation of the corrugator occurred in response to positive valence, accompanied by increased zygomaticus activity in positive valence conditions; however, in positive valence condition, the presentation of a strong-corrugator expression in facial icons impeded participants’ zygomaticus activation).

It has been suggested that in cases where the perceived facial expression is ambiguous, sensorimotor simulation might play a more significant role in emotion processing (Wood et al., 2016). The observed facial mimicry toward facial icons may indicate the involvement of sensorimotor simulation in emotion processing when dealing with facial expressions that are inherently complex and ambiguous due to their nature as freely crafted artifacts. In light of the embodied emotion theory (refer to Box 3 in Supplementary materials), the observed facial muscle reactions toward cross-cultural facial icons (Achour-Benallegue et al., 2021) and stick figures (Wessler and Hansen, 2021) provide substantial support to the simulated emotion we identified in our earlier study (Achour-Benallegue et al., 2016). These findings further endorse our hypothesis regarding facial icons serving as indexes of emotions within the framework of the simulation theory.

The neurobiological correlates of facial mimicry have been linked to the inferior frontal gyrus, a component of the classical Mirror Neuron System (MNS; Rymarczyk et al., 2018). Earlier research has posited the integration of motor and intentional aspects of action, suggesting that mirror neurons play a role in discerning an actor’s underlying motivation and intention (Rizzolatti and Sinigaglia, 2007; Rizzolatti and Fogassi, 2014, although see Ruggiero and Catmur, 2018). Moreover, the neurobiological correlates of facial mimicry extend beyond motor areas to include the neural network of mentalization, which contributes to understanding the mental states of others (Schilbach, 2016). Understanding other mental states extends beyond emotions alone, as mimicry not only relates to perceived emotions but also facilitates accurate and efficient recognition of the intention behind facial expressions (Blakemore and Decety, 2001; Wood et al., 2016). These findings align with evolutionary research, indicating that the evolution of higher cognitive abilities, such as mentalizing has been associated with the integration of perceptual and motor systems involved in face processing (Tramacere and Ferrari, 2016).

To our knowledge, experimental studies addressing the perception of intentions in facial icons are very scarce, with most of these studies originating from the human-robot interaction domain. For instance, Mutlu et al. (2009) investigated whether humanlike robots’ gaze cues and intentions can be interpreted by humans, and whether the physical design of the robot affects these inferences. Their findings revealed that gaze cues led to attributions of intentions to the robot, even when these cues were not explicitly reported, suggesting an unconscious and automatic interpretation and response. However, this effect was observed primarily in very humanlike robots and among pet owner participants. Despite the limited experimental evidence currently available for understanding the underlying intentions in facial icons, we propose that, similar to the perception of human facial expressions, the perception of facial icons could benefit from the neurobiological correlates of facial mimicry in grasping the underlying intention behind the depicted facial expressions. Drawing on studies like Blakemore and Decety (2001), motor simulation appears to be the crucial mechanism for elucidating how intentions are inferred from the shapes’ movement (Heider and Simmel, 1944; Tremoulet and Feldman, 2006; Pantelis et al., 2011). We suggest that this form of simulation, such as mimicry, also enables us to automatically infer intentions in facial icon expressions. Nevertheless, additional studies are needed to investigate and provide insights into this aspect.

In anthropology and archeology materials such as artifacts remain a means to explore culture (Kumar and Ahmed, 2022). Analyzing cultures that incorporate facial icons could benefit from our hypothesis, particularly in a context where methodological recommendations are still evolving. In archeology, for example, sensorimotor simulation processing enhanced by facial icons could be included in the comparative analysis of anthropomorphic artifacts that has been conducted recently by Matsumoto (2021). Facial icons as indexes of emotions and intentions contribute to the multi-layered and complex meanings of anthropomorphic artifacts, which are regarded as a rich source of information about social structures (Rice, 2019; Matsumoto, 2021). Matsumoto (2021, p. 65) suggests that “the features of anthropomorphic figurines can provide us with clues for understanding how humans recognize other humans.” Continuing along this line of thought, we may question the role of embodied simulation of facial icons in unveiling such understandings: could the difference in sensorimotor simulation to different corpuses of facial icons reflect the differences in how the respective populations -who made the facial icons and interacted with them- perceived their peers? Previous research conducted by Matsumoto and Kawabata (2010), has already suggested that anthropomorphic artifacts (specifically Jomon clay figurines) serve as a reference point for discussions on universal human cognition. Their preliminary experimental analysis, focusing on the perception of figurine faces presented in line drawings (rather than photographs) of Japanese ancient figurines, revealed both similarities and differences between native Japanese speakers and non-native Japanese speakers. The findings suggested that our innate inclination to assess gender based on facial features is relatively robust compared to more subtle emotion reading. The authors concluded that anthropomorphic artifacts, such as facial icons including figurines, have a unique characteristic as being closely related to both social and technical cognition, encapsulating rich and complex information for their original makers and for archeologists alike. Such research, as well as the one analyzing the impressions and emotions portrayed in Japanese clay figures (Kawabata et al., 2021) represent one of the crucial steps toward integrating cognitive processes of face perception into archeological research.

In anthropology, the artifacts have traditionally been considered less significant in analysis compared to social interactions, to the extent that it has been referred to as “missing nonhumans” (Ingold, 2012). However, anthropologists such as Ingold (2012) and Woodward (2007) have emphasized the pivotal role of artifacts in construction and transmission of human culture. This has led to the emergence of approaches that consider what is commonly referred to as ‘material culture’ (Woodward, 2007; Berger, 2016). Moreover, there has been a growing recognition of the agency of material objects (Gell, 1998; Mitchell, 2005), which leads to an expansion of methods and tools for uncovering cultural and social information. In this context, our hypothesis regarding facial icons may offer an additional tool for analyzing material culture in anthropology. This can involve “setting up appropriate experimental tasks and procedures” such as suggested by Saito (2021, p. 201), that might be close to her use of children’s drawings to understand the cognitive characteristics and production processes of ancient artworks.

Human-robot interaction is a pivotal research domain within social robotics (Sheridan, 2020), encompassing numerous studies focused on facial features in robots and their significance in social interaction such as in Mutlu et al. (2009), Carlson et al. (2019), and Stroessner and Benitez (2019), to name just a few. The field of human-robot interaction might stand to gain from the application of our hypothesis in various ways, particularly as social roboticists advocate for the integration of social psychological research into the development of social robots (Sheridan, 2020). Drawing parallels with studies like that of Dubal et al. (2011), which explores emotion perception in humans vs. low human-likeness robots, our hypothesis could shed light on psychological phenomena related to interactions with face representations commonly used in social robotics. It could be instrumental in investigations aimed at understanding reactions to robots from the perspectives of emotion and intention relying on simulation processes. Moreover, our hypothesis, along with upcoming research findings including information from morphometric features (Achour-Benallegue et al., 2024), could contribute to the design of various devices incorporating facial icons for social interactions. Robots designed for social interactions, as previously mentioned, present one potential application. For example, robot-assisted therapy seems to be a promising area in therapeutic approaches for children with autism spectrum disorders (Puglisi et al., 2022). Integrating our hypothesis into this domain could contribute to the development of next-generation devices and might potentially count among the foundational elements in designs aimed at improving the complex triadic interaction involving teachers, children, and robots.

Moreover, our hypothesis could aid in the further interpretation of findings related to emoticon perception, as seen in recent studies by Kutsuzawa et al. (2022a,b). Additionally, it could contribute to the development of new models of communication through emoticons. In summary, our hypothesis has the potential for integration across diverse domains that use facial representations to convey, communicate, inform, and enrich social meanings.

In light of these conclusions, several compelling questions emerge, inviting further exploration and investigation into the subject, succinctly summarized as follow: