In this paper, intelligent agents (IAs) refers to an autonomous and goal-directed artificial agent that utilizes machine learning (ML) methods such as deep reinforcement learning (DRL) to achieve certain objectives. With the recent development of ML, IAs have achieved good performance in complex tasks (Mnih et al., 2015; Silver et al., 2017), and an increasing number of studies are focusing on the application of real-world robots (Kahn et al., 2018; Kalashnikov et al., 2018). Introducing ML methods can be a promising approach to realize effective goal-directed human-agent collaboration.

However, many challenges remain that hinder collaboration between people and IAs. One of the major difficulties is the lack of explainability of an agent’s future behavior. Decision-making modules that utilize modern ML tend to be a black box (Fukuchi et al., 2017; Hayes and Shah, 2017). In particular, the DRL model embeds the control logic in high-dimensional parameter space and usually does not provide human-comprehensible expressions of the agent’s plans, goals, or intentions. Therefore, most people cannot understand what an IA is aiming to do. The ability to understand and predict a human coworker’s behavior can help robots to effectively collaborate with people (Lasota and Shah, 2015; Huang and Mutlu, 2016). Similarly, human agents also should be able to better understand their coworker agents’ future behavior.

Previous studies have proposed methods to explain diverse aspects of an IA’s decision making. Saliency maps are commonly used to explain the reason a deep learning module made a specific decision based on its input modality, and they are also applied to an IA’s policy model (Tamagnini et al., 2017; Iyer et al., 2018; Mott et al., 2019). While many approaches focus on explanations for AI practitioners, Cruz et al. proposed a method for end users (Cruz et al., 2021). Their method enables an IA to explain its next action with the probability of success. Hayes et al. proposed a natural language question answering system that can handle some templates of questions about an IA’s policy such as “when do you {action}?” and “what will you do when {state}?” (Hayes and Shah, 2017).

Humans sometimes consider an artificial agent to have a mind and thus to be a valid target for theory-of-mind inference. This attitude is called the “intentional stance” (Dennett, 1987). Many studies have investigated which characteristics of an artificial agent lead people to adopt the intentional stance. Human-likeness is a factor that leads humans to adopt an intentional stance (Wimmer and Perner, 1983; Perez-Osorio and Wykowska, 2020), but even geometrical figures can be targets of the intentional stance (Gergely et al., 1995) when they appear to be goal-directed (Premack and Premack, 1997), rational (Gergely et al., 1995), self-propelled (Luo and Baillargeon, 2005), or in violation of Newtonian laws (Scholl and Tremoulet, 2000).

IAs have many of these characteristics, which suggests that people can adopt an intentional stance toward them. For example, a rational agent is expected to take actions that maximize its own utility (Jara-Ettinger et al., 2015), and the RL framework is designed to address exactly the problem of utility maximization. The utility function for a general RL agent is designed to drive that agent to achieve certain goals in an efficient (or rational) manner; positive rewards encourage the RL agent to achieve certain tasks, while negative costs urge the agent to choose more efficient actions.

A person can infer another person’s beliefs, goals, or intentions even without explicit communication. In effect, people can infer the minds of others simply by observing their behavior (Baker et al., 2017). However, there is also a risk of misunderstanding or of forming false beliefs about other people based on such observations. When a person adopts the intentional stance with regard to an IA and attempts to infer the agent’s mind, the same problem can arise.

In this context, objective self-awareness, or the ability of a person to recognize themselves as an object of attention (Duval and Wicklund, 1972), becomes an important component for an IA to help humans correctly understand its behavior. Objective self-awareness is considered to have two aspects: private self-awareness and public self-awareness (Feningstein, 1975; Falewicz and Bak, 2015). A privately self-aware person is self-reflective and attentive to their own thoughts. A publicly self-aware person, on the other hand, focuses on the self as a social object and is attentive to how he or she appears to others. If an IA has a private self-awareness model, it can reveal ι*, the intention that is going to be achieved by the agent. A public self-awareness model will also enable the agent to infer the intention ι that a human observer will attribute to it based on its behavior. On this basis, the agent can select an action a that will lead the observer to infer the agent’s true intention a = argmax a P ( ι = ι * | a ) . Similarly, an agent can also select actions that will mislead the observer to infer a particular false intention.

Multiagent systems (MASs) represent one research field that aims to introduce the concept of theory of mind to artificial agents. Theory of mind ability enables agents to choose their actions based on what another agent is going to do, resulting in better performance in both cooperative and competitive situations (Zettlemoyer et al., 2009; Raileanu et al., 2018).

One of the major challenges of inferring an other’s mind is its multiply nested structure. Here, we formalize the nested inference structure using belief-desire-intention logic (Cohen and Levesque, 1990). Suppose that there are two agents, an actor agent that performs actions, and an observer agent that attempts to infer the actor’s intention based on its actions. Let us call the latter a first-order inference and denote the inferred intention ι ¹: BEL o b s e r v e r I N T E N D a c t o r ι 1 , where (BEL i X) means agent i believes X, and (INTEND i ι) means agent i intends to achieve ι. Actors can also have second-order beliefs, that is, beliefs about the observer’s belief. For example, BEL a c t o r BEL o b s e r v e r I N T E N D a c t o r ι 2

means that the actor believes that “the observer believes that the actor intends to achieve ι ².” In this paper, a superscript k means that the corresponding variable represents a k-th order belief. We can consider an arbitrary order of inference by repeating this manipulation.

Zettlemoyer et al. proposed sparse distributions over sequences (SDS) filtering (Zettlemoyer et al., 2009), an algorithm to compute the nested belief. SDS filtering can efficiently solve the problem of sequential inference about nested beliefs by utilizing a sequence distribution, which represents the probability distribution of an agent’s belief about the environment and the other agents’ beliefs given a set of possible sequences of states.

The computational theory-of-mind model has also been studied in the field of cognitive science. Baker et al. proposed the Bayesian theory of mind (BToM) model (Baker et al., 2017). The BToM describes an observer agent’s first-order inference of an actor agent’s mental state, such as a belief, desire, or intention, while observing the actor agent’s behavior. Equation 1 describes the inference model: P ( b t + 1 1 , d 1 , ι t + 1 1 | o : t + 1 ) ∝ ∑ o t + 1 1 , s t + 1 , s t , a t , b t 1 , ι t 1 P ( ι t + 1 1 | b t + 1 1 , d 1 , ι t 1 ) ⋅ P ( b t + 1 1 | b t 1 , o t + 1 1 ) ⋅ P ( o t + 1 1 | s t + 1 ) ⋅ P ( o t + 1 | s t + 1 ) ⋅ P ( s t + 1 | s t , a t ) ⋅ P ( a t | b t 1 , d 1 , ι t 1 ) ⋅ P ( b t 1 , d 1 , ι t 1 | o : t ) , (1) where a is an action performed by the actor, and belief b _t is a probability distribution representing the probability that the environmental state is s _t given past observations o _:t , i.e., b _t (s _t ) = P(s _t |o _:t ). The observer attributes mental states to the actor at time t, such as observation o t 1 , belief b t 1 , desire d ¹, and intention ι t 1 based on the observer’s observation history o _:t from times t = 0, 1, …, t. Experiments demonstrated that BToM accurately captures human mental state judgments.

The PublicSelf model extended BToM to an actor’s second-order belief inference (Fukuchi et al., 2018). We can consider PublicSelf as a computational model of an actor’s public self-awareness. PublicSelf can be represented in the form of a Bayesian network (Figure 3). From the actor’s observations o _:t , the probabilities with which belief b t 2 , desire d ², and intention ι t 2 will be attributed to the actor can be calculated: P ( b t + 1 2 , d 2 , ι t + 1 2 | o : t + 1 ) ∝ ∑ o t + 1 2 , o t + 1 1 , b t + 1 1 , s t + 1 , s t , a t , b t 2 , ι t 2 , b t 1 P ( ι t + 1 2 | b t + 1 2 , d 2 , ι t 2 ) ⋅ P ( b t + 1 2 | b t 2 , o t + 1 2 ) ⋅ P ( o t + 1 2 | b t + 1 1 ) ⋅ P ( b t + 1 1 | b t 1 , o t + 1 1 ) ⋅ P ( o t + 1 1 | s t + 1 ) ⋅ P ( o t + 1 | s t + 1 ) ⋅ P ( s t + 1 | s t , a t ) ⋅ P ( a t | b t 2 , d 2 , ι t 2 ) ⋅ P ( b t 2 , d 2 , ι t 2 | o : t ) . (2)

In PublicSelf, first, a belief about the environment is constructed based on the actor’s own visual observations. That is, an observation of an object o _t increases the likelihood of the actor’s belief of possible environment states s in which the object exists at the observed position. Then, the observer’s belief about the environment, denoted by b ¹, is considered, i.e., b t 1 ( s t ) = P ( s t | o : t 1 ) , where o ¹ is an inference concerning the observer’s observations. From b ¹, one can then estimate o ² and b ², the observation and belief attributed to the actor by the observer, where b t 2 ( s t ) = P ( s t | o t 2 ) .

The important point of PublicSelf is that it distinguishes mental states that are based on an actor agent’s actual observations, first-level beliefs of an observer’s mental state, and second-level beliefs attributed to the actor by the observer. This ability allows PublicSelf to infer the intention the observer attributes to the actor while considering information asymmetry. A user study demonstrated that PublicSelf enables an IA to accurately infer the mental state attributed to it by a human observer in situations where partial observability causes information asymmetry between an actor and its observer. However, previous work on PublicSelf focused only on the accuracy with which it infers the mental states attributed to an IA by a human observer, and PublicSelf has not previously been applied to generating an agent’s actions based on the inference.

Dragan et al. proposed a method for generating an artificial agent’s behavior with awareness of a human observer’s theory of mind, specifically, behaviors that communicate the agent’s intention to a human observer. In particular, legible motion aims to allow an observer to quickly and correctly infer an agent’s intention (Dragan et al., 2015a). By means of legible motion, an agent attempts to increase the probability that the intention an observer attributes to it will match its true intention.

In previous studies on the generation of legible motion, it was assumed that the environment is limited and that both the human observer and the artificial agent share complete information about the environment, such as what exists and where it exists. However, most actual collaboration scenarios are subject to uncertainty, and information asymmetry typically exists between human and artificial agents, meaning that one agent may possess information that the other does not. Different observations result in different beliefs, which is important to consider when modeling human theory of mind. Information asymmetry is deliberately employed in daily social acts including deception, and many related psychological experiments, such as false-belief tasks, have been performed (Wimmer and Perner, 1983). Therefore, this paper claims that an artificial agent needs to handle information asymmetry between the agent and a human observer when generating publicly self-aware behavior.

To validate our claim, we compare PublicSelf legible motion that considers information asymmetry with legible motion that does not account for information asymmetry, which we call FalseProjective.

With FalseProjective, an actor does not distinguish its own belief regarding the environment from an observer’s belief and falsely identifies its own belief with the observer’s one.

The work by Nikolaidis et al. is the most closely related to this paper’s concept (Nikolaidis et al., 2016). They proposed a method for generating legible motion considering the effect of a human observer’s viewpoint. However, their focus was on depth uncertainty and occlusion of a robot arm and did not target the differences in beliefs about the world state such as what is where in the environment.

We developed the FetchFruit task in a simulated environment (Figures 1, 2) and implemented our method of generating legible motions for an IA in this simulated environment. The environment is a square room containing an apple, a pear, an actor, and a human observer. The initial positions of the actor and the observer are fixed.

The actor’s actions are driven by a policy for the retrieval of an apple or a pear. Our method is independent of the implementation of the policy model as long as the probability of the actor taking each action can be calculated. In this paper, we use the asynchronous advantage actor-critic (A3C) algorithm (Mnih et al., 2016), which is one of the most representative algorithms for DRL. Every 0.5 s, the actor selects an action from the action space A, which is composed of three discrete actions: accelerate forward, turn clockwise, and turn counterclockwise.

The environmental state s _t is composed of the locations of the apple and pear, the area that is within the observer’s sight, and the actor’s state, which includes the actor’s location, velocity, and direction as well as the area within the actor’s sight. In this paper, only the actor’s state changes over time.

The human observer does not move and observes the environment from a fixed viewpoint. The observer can acquire only information that is in his/her field of view. That is, the observer can know where a fruit is only if s/he can see it. Similarly, the actor’s location, velocity, and direction are provided when the observer can see the actor, but none of the above is provided when the actor is out of the observer’s field of view.

Algorithm 2 shows how Eq. 2 is calculated in our implementation. There are infinite possible environmental states because s _t consists of continuous values and is subject to uncertainty due to partial observability. We solve the problem by sampling possible states to simplify the state space, which is analogous to the sequential Monte Carlo method (Doucet et al., 2001). We first randomly sample n states s _0,0, s _0,1, …, s _0,n−1 from among the possible states. The implementation of PublicSelf is based on the SDS filtering concept (see Section 2.4). PublicSelf includes four filters for inferring the beliefs and desires that the observer will attribute to the actor: Φ t 0 ( s : t ) ∝ b 0 ( s : t ) , Φ t 1 ( s : t ′ , s : t ) ∝ ∑ b t 1 b t 1 ( s : t ′ ) ⋅ δ ( b t 1 , B o b s ( s : t ) ) , Φ t 2 ( s : t ′ , s : t ) ∝ ∑ b t 2 b t 2 ( s : t ′ ) ⋅ δ ( b t 2 , B a c t ( s : t ) ) , Ψ t 2 ( d 2 , s : t ) ∝ P ( d 2 | s : t ) .

The delta function δ(α, β) returns a value of 1 when α = β and a value of 0 otherwise. B ^obs (s _:t ) and B ^act (s _:t ) return the observer’s and actor’s beliefs, respectively, given the history of the environmental states. Here, for simplicity, we will suppose that the transitioning of the environmental state is a Markov process and s _:t can be denoted as s _t . Φ t 0 ( s t ) denotes the actor’s belief regarding the environment. Φ t 1 ( s t ′ , s t ) is the probability that the observer will believe that the environmental state is s t ′ when the actual environmental state is s _t . Φ t 2 ( s t ′ , s t ) is the probability that the observer will infer that the actor believes that the environmental state is s t ′ when the actual state is s _t . Ψ t 2 ( d 2 , s t ) represents the probability with which desire d ² will be attributed to the actor by the observer given state s _t . These filters are initialized as uniform distributions.

Based on the actor’s observation o _t , Φ t 0 ( s t ) is updated by multiplying it by the observation probability P(o _t |s _t ) because P(s _t |o _t ) ∝ P(s _t |o _t−1) ⋅ P(o _t |s _t ). We can similarly update Φ t 1 ( s t ′ , s t ) and Φ t 2 ( s t ′ , s t ) by estimating the actor’s and observer’s observations o ¹ and o ² under each s _t and multiplying them by the observation probabilities P ( o t 1 | s t ′ ) and P ( o t 2 | s t ′ ) : Φ t 0 ( s t ) ← Φ t 0 ( s t ) ⋅ P ( o t | s t ) , Φ t 1 ( s t ′ , s t ) ← Φ t 1 ( s t ′ , s t ) ⋅ ∑ o t 1 P ( o t 1 | s t ′ ) ⋅ δ ( o t 1 , O o b s ( s t ) ) , Φ t 2 ( s t ′ , s t ) ← Φ t 2 ( s t ′ , s t ) ⋅ ∑ o t 2 P ( o t 2 | s t ′ ) ⋅ δ ( o t 2 , O a c t ( s t ) ) , where O ^obs (s _t ) and O ^act (s _t ) return the observer’s and actor’s observations, respectively, under s _t .

During the update process, an s ∈ S _t may appear such that ∑ b 2 b 2 ( s ) = 0 , which means that the observer no longer thinks that the actor holds any belief about s at all. We resample S _t by removing such states s with zero probability and making branches of samples with high probability.

The actor’s state branches depending on the actor’s choice of action. Let S t + 1 a t be a set of predicted environmental states to which the actor’s action a _t will lead from s ∈ S _t , and let S _t+1 be the union of the states predicted under each action, ⋃ a S t + 1 a . The function Pred : S _t × A → S _t+1 returns the states at time t + 1 to which the environment transitions with each action from S _t . In this study, we trained a model for Pred by means of supervised learning. The new values of the filters Φ _t+1 are inherited from the previous values: Φ t + 1 0 ( P r e d ( s t , a ) ) ← Φ t 0 ( s t ) , Φ t + 1 1 ( P r e d ( s t ′ , a ′ ) , P r e d ( s t , a ) ) ← Φ t 1 ( s t ′ , s t ) ⋅ δ ( a ′ , a ) , Φ t + 1 2 ( P r e d ( s t ′ , a ′ ) , P r e d ( s t , a ) ) ← Φ t 2 ( s t ′ , s t ) ⋅ δ ( a ′ , a ) .

We assume that the actor can have only two intentions, namely, ι _a and ι _p , which are the intention to retrieve an apple and that to retrieve a pear, respectively. We also consider that the desire to retrieve a fruit directly generates the intention to retrieve it, that is, P ( ι 2 = ι a | d 2 = d a ) = P ( ι 2 = ι p | d 2 = d p 2 ) = 1 , where d _a and d _p are the utility functions when the actor’s target is an apple and a pear, respectively. Then, P ( a | s 2 , d a 2 , ι a 2 ) simplifies to P ( a | s 2 , d a 2 ) , which can be estimated with a model-free RL algorithm. Ψ t + 1 2 ( d 2 , P r e d ( s t , a ) ) ← Ψ t 2 ( s t ) ⋅ P ( a | s t , d 2 )

To generate legible motion, we need to determine P(ι ²|a), the probability that the observer will attribute an intention ι ² to the actor given action a. In our implementation, we equate ι ² with d ² and calculate P(d ²|a) instead of P(ι ²|a). P(d ²|a) can be obtained from the four filters as follows: P ( d 2 | a ) = ∑ s ∈ S t + 1 Ψ 2 ( s , d 2 ) ∑ s ′ ∈ S t + 1 Φ 2 ( s ′ , s ) ∑ s ″ ∈ S t + 1 a Φ 1 ( s ″ , s ′ ) ⋅ Φ 0 ( s ″ ) .

Since the PublicSelf model infers the actor’s own belief b ⁰, PublicSelf can also be used to generate FalseProjective legible motion. A filter Ψ t 0 ( s , d 0 ) , which represents the probability of an actor’s own desire d ⁰ estimated independently of the observer, enables us to generate FalseProjective legible motion in a manner similar to the generation of PublicSelf legible motion.

Here, we present the motion generated in the example FetchFruit scenarios. We consider five example scenarios: Center, Side-Visible, Side-Invisible, Blind-Inside, and Blind-Outside. Table ?? summarizes the settings for these scenarios, which cover all 2 × 2 possibilities with regard to whether the observer can see the actor’s target and/or the nontarget objects. In every scenario, the target and nontarget objects are adjacent to each other. The actor is assumed to observe the positions of both fruits from its initial position; consequently, the actor does not need to explore the environment but simply moves to its target. We generated three types of motion in each of the example scenarios: the original motion, the FalseProjective legible motion, and the PublicSelf legible motion.

In the Center scenario (Figure 4), the apple and pear are immediately in front of the observer. It is difficult to quickly infer the actor’s intention from the original motion because the actor first moves straight to the point between the apple and the pear (Figure 4A), whereas the FalseProjective motion enables the observer to infer the actor’s intention more quickly because of the agent’s curved movement toward its actual target (Figure 4B). The PublicSelf legible motion follows the same route as the FalseProjective motion.

In the Side-Visible scenario (Figure 5), the human observer can observe only the apple. With the original motion trajectory, the actor moves directly to the apple. Because the observer cannot see the pear next to the apple, the original motion presents much less ambiguity than in the Center scenario. Here, PublicSelf generates the same trajectory as that for the original motion. The FalseProjective motion, on the other hand, follows a different trajectory; the actor first moves forward toward the observer and then follows a curved trajectory toward the apple. This motion would avoid ambiguity if the observer knew the location of the pear; however, due to the observer’s limited view, this motion may instead lead the observer to believe that the actor’s target is behind him or her; thus, the actor’s intention is less clear in this case than it would be if the actor moved directly to the apple.

Figure 6 shows the results in the Side-invisible scenario. With the original motion trajectory, the actor first begins to move to the space between the apple and pear and then shows a gently curved motion. Early motion could mislead the observer into thinking that the actor intends to retrieve the pear, whereas through the PublicSelf legible motion, the actor can convey that the pear is less likely to be its target by making a detour to avoid causing the observer to misunderstand the actor’s intention. Here, the PublicSelf legible motion is the same as the FalseProjective motion.

In the two Blind scenarios (Figures 7, 8), the actor’s motion does not convey any information to the observer, who does not know the location of either the apple or the pear; therefore, the FalseProjective motion merely introduces a detour and increases the time required to retrieve the apple. By contrast, the PublicSelf legible motion does not change the actor’s actions because PublicSelf can infer that changing the motion would have no effect on the observer’s inference.