We propose that the traditional approaches followed by previous studies in the Affective Computing field are far from enough to achieve real artificial affective empathy. Emotion should be modeled as the bottom-up internal state of the agent so that it can process environmental information more adaptively and carry out affective empathy more reasonably. This subsection discusses the modeling of a particular emotion–pain.

As mentioned in Subsection 2.1, the first step of the Artificial Pain Model is to quantify the actual damage of intelligent agents. From the perspective of Free Energy, body damage is an unexpected state, called the entropy increasing state (Friston, 2010). However, the entropy of organisms cannot be directly quantified; the Free Energy is the upper bound of entropy, and minimizing Free Energy approximates minimizing the entropy (Ramstead et al., 2018). Thus, FEP is a way to quantify the damage.

Bogacz demonstrated the relationship between Free Energy and entropy (Bogacz, 2017), providing the definition and expression of Free Energy in detail: the brain could not directly know the actual state of the external world ϕ; it can only constantly estimate the state of the external world ϕ^; the sensory information s received by the sense organs is used to verify the estimation. The Free Energy is finally simplified and expressed as the negative logarithm of the joint probability distribution of the state estimation ϕ^ and the sensory information s. According to the definition, the larger the joint probability distribution, the smaller the Free Energy, and the more accurate the estimation. Expanded by the Bayesian formula p(s,ϕ^)=p(s|ϕ^)p(ϕ^):

We replace the probability distribution with a Gaussian distribution f(s; g(.), σ) with mean g(.) and variance σ, supposing σ = 1:

The first item in Equation 2 is the sensory dynamic (McGregor et al., 2015). g_s() is the sensory generating function, mapping the relationship between the world state and the sensory information learned by experience in advance. The brain actively estimates the world state ϕ^, then predicts sensory information through the generating function gs(ϕ^). The actual sensory information s received through the sense organs is used to calculate prediction errors. The second term in Equation 2 is the environmental dynamic (McGregor et al., 2015). The estimation of the brain depends on prior knowledge μ_ϕ of the environment and this item is similar to the prior error of Bayes theorem. Environmental dynamic and sensory dynamic jointly determine the value of Free Energy.

For robotic systems, the world state mentioned above is the body state of the robot and the sensory information for robots are vision s_v and proprioception s_p (Lanillos and Cheng, 2018), as shown in Equation 3. The robot continuously estimates its body state ϕ^, and predicts the vision gsv(ϕ^) and proprioception gsp(ϕ^), respectively. The generating functions g_sv() and g_sp() can be learned by motor babbling in advance (Lanillos and Cheng, 2018), which is the mapping relationship between the body state and the vision information or proprioception information. Therefore, the sensory dynamic of the robotic system includes visual prediction error and proprioception prediction error. For the environmental dynamic, the body state ϕ of the robot at the current moment is determined by the state ϕ and the action a′ at the last moment. g_ϕ() represents the mapping relationship. When the robot's body structure is deformed, or the sensor is damaged, the sensory dynamic will be affected. When the robot motor structure is damaged, the environmental dynamic will be affected. In these cases, the value of Free Energy will change. So the value of Free Energy is taken to evaluate the robot's body damage.

In living organisms, physical damage can result in a pain state, and physical damage can be quantified by the value of Free Energy. Thus, in a robotic or virtual agent system, Artificial Pain can be considered relative to the value of Free Energy that represents their body damage. When the Free Energy is greater than 0, the body is in an injured state and then causes a pain state; When the Free Energy equals 0, the body is in a normal state.

In some potential dangerous scenarios, substantial injury has not occurred, but the organisms can also feel pain. This is because the brain can associate the experience of pain with injury-related cues so that it can quickly avoid similar cues. In the actual application of the robot, after the robot suffered substantial body damage, the robot's vision system should capture the corresponding dangerous scenarios. The robot will then associate it with the internal pain state and avoid potential damage. The significance of pain is that it is a warning signal for the actual and potential body damage of the living organisms to protect themselves (Melzack, 2001). The Artificial Pain created with the proposed model can not only generate a pain warning signal when the body damage actually occurs, but also avoid potential damage in the future, achieving the same significance as the living organisms' pain.

When different internal emotional states are generated through the Artificial Pain Model, the AE-SNN simulating mirror mechanism will be trained. This subsection describes the concrete design and implementation of the AE-SNN.

1. LIF neuron model and STDP learning rule. Compared to conventional artificial neural networks, spiking neural network (SNN) captures more essential characteristics of the biological brain (Ghosh-Dastidar and Adeli, 2009; Khalil et al., 2017, 2018a; Zhao et al., 2018). In recent years, SNN has been successfully applied in many aspects of cognitive function modeling, such as decision making and creative processes (Héricé et al., 2016; Khalil and Moustafa, 2022). The Leaky Integrate-and-fire (LIF) neuron is a well accepted computational model for SNN neurons (Tal and Schwartz, 1997), and we use it as the basic unit of our model. The LIF neuron dynamics are described by the following Equation 4 and Equation 5:

u_t is the membrane potential of the neuron at time t, u_rest is the membrane potential at steady-state, R is the resistance, I(t) is the input current, and τ_m is the time constant. When the membrane potential u_t exceeds a certain threshold u_th, the neuron fires and t_f is the firing time. Once the neuron has fired, the membrane potential returns to its reset state u_reset (Fang et al., 2021). In this article, the parameters of the LIF neuron are: u_rest = u_reset = 0mV, τ_m = 30ms, u_th = 60mV.

We use Spike Timing Dependent Plasticity (STDP) as a synapse learning rule to update synaptic weights. STDP is the most basic learning method in the brain, which relies on the time difference between the firing of pre-synaptic neurons and post-synaptic neurons to train the synaptic weight (Caporale and Dan, 2008). The weight update can be written as Equation 6:

Here, Δt = t_i−t_j, t_i and t_j are the time of the firing of the pre-synaptic neuron and the post-synaptic neuron, respectively (Zhao et al., 2018). A⁺ and A⁻ are the learning rates. τ₊ and τ₋ are the time constants. According to this rule, the connections will be strengthened when pre-synaptic neurons fire before post-synaptic neurons and will be weakened when pre-synaptic neurons fire after post-synaptic neurons. Here, τ₊ = τ₋ = 10ms, A⁺ = 0.25, A⁻ = 0.01.

2. The training process. We describe the training process of AE-SNN from two aspects: First, set the value of fixed weights in the three modules. In the Motor Module, the excitatory synaptic weights between SMA neurons and M1 neurons are set so that once the SMA neurons are activated, the connected M1 neurons are activated immediately. In addition, The Emotion Module connects all neurons in the Motor Module except M1 neurons. We set these fixed synapse weights because the different emotion overt actions are determined by hereditary genetic factors (Darwin, 2015). Different emotion neurons' firing will lead to different motor neurons' firing. Since the connection between the two modules is bidirectional, these synapse weights can also be used in reverse when known the activation of the Motor Module to infer the activation of the Emotion Module.

Second, train the synapse weights between the Perception Module and the Motor Module. For the emergence of mirror neurons, we simulated the process of action execution and action re-afference and used the STDP learning rule to adjust the synapse weights. It takes 100 ms from the firing of the motor neurons to action execution, 100ms from the hearing/watching of that action to trigger the firing of the perception neurons, and the total time delay is 200 ms (Keysers and Gazzola, 2014). In the training process, the emotion neurons fire at first, the corresponding motor neurons will fire and execute the emotional overt action, and after 200 ms, the corresponding perception neurons will fire. Because action execution is a continuous process, the firing of the motor neurons is still going on, as shown in Figure 3. Due to this temporal correlation, the excitatory connections between the motor neurons executing the action and the perception neurons representing the same action will be strengthened by STDP, forming the mirror neurons in the Motor Module. In the emergence process of anti-mirror neurons, due to the connections between the SMA neurons and the perception neurons being inhibitory, the inhibitory connections between the SMA neurons and the perception neurons representing the same action will be strengthened by STDP. However, M1 neurons are only activated by SMA neurons, so M1 neurons are deactivated after the SMA neurons are inhibited, and the connection between the M1 neurons and the perception neurons will not be established. The M1 neurons are only activated during action execution, reproducing the property of anti-mirror neurons.

In the training stage, the connections among the three modules are established, and the mirror neurons and anti-mirror neurons emerge. When perceiving the emotional overt action of others, the perception information of overt action will be encoded in the Perception Module as input. Then it will activate the mirror neurons in the Motor Module, and further activate the corresponding emotion neurons to empathize with others.

We place dangerous objects, a switch, walls, and an agentA in the grid world. The agentA explores the environment and uses the Artificial Pain Model to detect its own internal state, completing the training of the AE-SNN.

We set the damage rule of agent: when an agent collides with a dangerous object, its ability to act will be impaired. For example, if a command is given to the right, the agent actually moves a unit to the left. For the Artificial Pain Model, the 'environmental dynamic' of the Free Energy will result in errors, as the third item in Equation 3. We first set the mapping relationship gϕ(ϕ′,a′) of the Artificial Pain Model, which is the moving rule of the agent: When the agent performs an action, its horizontal and vertical coordinates will change accordingly. For example, if an upward action is performed, its vertical-coordinate's value reduces, and if a rightward action is performed, its horizontal-coordinate's value adds. The agent knows its moving rule in advance and uses this rule to continuously and actively predict its next position. When the prediction is wrong, the Free Energy is greater than 0, indicating that damage occurs and the agent will be in the pain state.

Figure 6A shows that agentA is colliding with the dangerous object. Figure 6B shows that the next motion command of agentA is toward the right, the prediction is the position of the green dotted line. But Figure 6C shows that the actual action of the agentA is toward the left. At the same time, the motor damage will be detected by the Artificial Pain Model, and the agentA will be in the pain state. After the pain state generates, the first 20 neurons in the Emotion Module that represent pain will fire and activate the neurons in the Motor Module to produce painful overt action. In this experiment, the color of the agentA turns red, as shown in Figure 6D. After that, the agentA's Perception Module will perceive its own color information. The first 20 neurons of the Perception Module that represent perceiving red will be activated, completing a training epoch of AE-SNN in the pain state by STDP. After 100 training epochs, the AE-SNN finishes training, and the agent has pain empathy ability. Figures 6E,F show that agentA touches the switch during exploration and establishes the passage from the danger zone to the safety zone. Then it enters the safety zone and recovers to its normal state. These two figures aim to illustrate the role of the switch.

Figure 7 represents the value of Free Energy during exploration. The X-axis represents the number of steps. The Y-axis represents the value of Free Energy. The value of Free Energy is the sum of the square of the difference between the predicted agentA's coordinates and the actual agentA's coordinates. We designed the agent as a pentagon, each vertex has a 2D coordinate (X, Y), so the agent has five sets of 2D coordinates at each position, i.e., (X₁, Y₁), (X₂, Y₂)...(X₅, Y₅). The Free Energy is equal to the sum of the squares of the differences in the corresponding coordinates, as described in Equation 7. The unit grid length is 25, resulting in the Free Energy range of [3125,12500]. Since our model is only related to whether the Free Energy is greater than 0, we put all values that are greater than 3125 equal to 3125. PointA indicates that the agentA collides with a dangerous object at 19 steps, and the damage generates. The Free Energy is greater than 0, causing the agentA to be in the pain state. PointB indicates that the agentA enters the safe zone at 82 steps, and recovers its normal state. The Free Energy is equal to 0, causing the agentA's pain state to be alleviated.

Figures 8A–C is the spike diagrams of the three modules of AE-SNN in the pain state. The X-axis represents the firing time, and each unit represents 100 ms. The Y-axis represents the index of the neurons. Figure 8A represents the firing of neurons that represent the pain state in the Emotion Module. Figure 8B shows the firing of neurons that represent the painful overt action in the Motor Module as the agentA's color turns red. Figure 8C shows the firing of neurons in the Perception Module, which corresponds to agentA perceiving its color information. The firing of the Emotion Module will then trigger the firing of the Motor Module by setting weights in advance, and then the Perception Module will fire after a 200ms delay, indicating the time interval for action execution and action re-afference. With the time correlation, the connections between the Perception Module and the Motor Module can be established by STDP.

Figure 9A shows the synaptic weights training process from the Perception Module to the Motor Module. The X-axis represents the training epochs. The Y-axis represents the value of weights. The purple line indicates the change of inhibitory weights between the Perception neurons and the SMA neurons. The green line indicates a change of weights between the Perception neurons and the M1 neurons, and due to the gate mechanism of the SMA neurons, the weight is maintained near 0, which is equivalent to not establishing synaptic connections. The yellow line indicates the change of excitatory weights between the Perception neurons and other neurons in the Motor Module, and the value of the weight gradually increases, indicating that these neurons are mirror neurons. Figure 9B shows the trained synaptic weights between the Perception Module and the Motor Module. The X-axis represents the index of Perception neurons. The Y-axis represents the index of Motor neurons. The color represents the value of the weight. The purple areas represent inhibitory synaptic weights between the Perception neurons and the SMA neurons. The yellow areas represent excitatory synaptic weights between the Perception neurons and the mirror neurons that represent the same emotional overt action. The green areas represent unestablished synaptic connections, which contain the connections between M1 neurons and Perception neurons.

Using the AE-SNN trained in Section 3.2, we designed an altruistic rescue task. Figure 10A shows that two agents, a dangerous object, a switch, and walls are placed in the grid world. In the previous task, the agentA with affective empathy ability is placed in the right safety zone, and agentB (green pentagon with black border) is placed in the danger zone for exploration. We place the yellow switch on the side of the safety zone, so agentB has no self-rescue ability and could only be rescued through agentA.

When agentB explores the environment, it will inevitably encounter a dangerous object, resulting in pain and turning into red color, as shown in Figure 10B. AgentA will perceive the color information of agentB, then generate empathy pain through AE-SNN. The intrinsic altruistic motivation will drive agentA to take action to rescue agentB and find the optimal strategy by the reinforcement learning method. Eventually, agentA learned to go to the switch as soon as possible and establish the passage from the danger zone to the safety zone, as shown in Figure 10C. After the switch is turned on, agentB will enter the safety zone and relief the pain state, then agentA's empathy pain will also be relieved through AE-SNN as shown in Figure 10D, completing the rescue of the companion.

Figures 11A–C shows the spike diagrams of the three modules of the agentA during the altruistic rescue process. Figure 11A shows the firing of the neurons in the Perception Module, representing agentB's color information. Figure 11B shows the firing of the mirror neurons in the Motor Module, which realizes the understanding of the pain state at the motor level. Figure 11C shows the firing of the neurons in the Emotion Module, which realizes the understanding of the pain state at the emotional level. Moreover, in this process, compared with Figure 8B mentioned in Section 3.2.2, not all neurons in the motor area are activated (in fact only mirror neurons are activated), so the conditions for executing painful overt actions do not arrive. Therefore, although the agentA feels pain, it does not turn red.