To learn a mapping between the robot joint configurations and facial expressions, we train a generator network to convert from joint angles to a simplified facial image X image = h robot ( θ ) (1) where X _image denotes the image of the facial features generated by the network, h _robot the generator network, and θ the joint configuration of the robot. To reduce computational effort, we obtain simplified images by applying the dlib facial feature detector (King, 2009) on the full images of the robot and constructing a smaller and less detailed black and white image from this. An example of such a simplified image can be seen in Figure 3.

For the generator network we use a CNN with five layers, which generates images of size 48 × 48 × 1. The network structure is given in Table 1. We created a dataset of the images of facial feature extraction at various values of joint configurations from the robot. Next, we trained the generator network h _robot to generate images of facial feature extraction from robot joint configuration. The generator network that we use is similar to the generator network in standard Generative Adversarial Networks (GANs), except we do not use a discriminator to train the images being generated. As the output consists of only simplified images of facial features, we found it is sufficient to train the generator network by reducing the mean squared error between the pixels of the actual output and the predicted output.

Most studies that perform facial expression recognition directly train on facial images (Barros et al., 2015; Ahmed et al., 2019). In contrast, we train the Convolutional Neural Network (CNN) on simplified versions of the training images generated from landmarks detected via dlib (King, 2009). As training images, we used the KDEF dataset (Lundqvist et al., 1998) and additional hand-labeled training images for our robots Alfie and Elenoide. We used data augmentation to slightly translate the training images both vertically and horizontally so that the faces are visible completely. We also evaluate the models trained with different combinations of data, i.e., only human facial expressions, only robot facial expressions, and both combined. The use of the extracted features, i.e. landmark points, makes the classifier hereby less sensitive to environmental changes such as lighting conditions and generalizes better between different robot types and human images. From the set of labeled training images, the classifier should learn the mapping from the simplified facial images to discrete predefined expressions y expression = f classifier ( X image ) ∈ { a n g r y , h a p p y , n e u t r a l , s a d , s u r p r i s e } (2) with the expression classifier f _classifier and the simplified image of the extracted facial features X _image of size 48 × 48 × 1. For the expressions classifier, we used CNN with four convolutional layers and one fully-connected layer. The network structure is given in Table 2.

Here, we explain in detail how we automatically calculate and obtain the values of joint configurations θ by performing grid search and gradient descent. To automatically find a set of joint configurations that generate a desired facial expression y _j on the robot, we minimize the cross-entropy loss L between the predicted expression of the classifier y j ̂ = f classifier h robot θ and the desired expression L = − 1 N ∑ j = 1 N ( y j ⁡ log y j ̂ ) + ( 1 − y j ) log ( 1 − y j ̂ ) (3) with N predefined facial expressions. To find stable joint configurations, we determined and averaged the loss for five generator networks and five classifier networks.

Specifically, we first find the global minima for θ values in the whole range. To this end, we consider θ values at discrete steps and find the combination of joint angles for which the loss is minimal for each expression. Then, we find the global minima. We assume a range for values of θ instead of just a single joint configuration per expression by considering θ ∼ N ( μ , σ 2 ) where μ is the mean and σ 2 is the standard deviation. We used reparameterization to sample from μ and σ by considering θ = μ + ϵ ʘ σ and correspondingly propagate the gradients to minimize the loss via gradient descent min θ L f classifier h robot θ (4) using element-wise multiplication ʘ and Gaussian noise ϵ ∼ N ( 0 , 1 ) .

The mean and the variance are updated by gradient descent μ k + 1 = μ k − α ∂ L ∂ μ , σ k + 1 = σ k − α ∂ L ∂ σ (5) using a learning rate α and computing the gradients by ∂ L / ∂ μ = ∂ L / ∂ θ and ∂ L / ∂ σ = ∂ L / ∂ θ ʘ ϵ .

Algorithm 1 summarizes the overall approach. Here, f is the expression-classifier and h is the image generated for Alfie’s and Elenoide’s face. The loss L and gradients ∂ L / ∂ θ ⃗ and ∂ L / ∂ θ ⃗ ʘ ϵ ⃗ are averaged over minibatch.

Algorithm 1

Generating multiple configurations for various expressions.

We conducted experiments with the approach described in Section 3 on our robots Alfie and Elenoide. The code is written in Python using TensorFlow. Adam was used as an optimizer. For the grid search, we considered discrete intervals of 0.1 for each of the three joint angles for Elenoide and 0.2 for each of the six joint angles for Alfie. For fine-tuning the θ ⃗ values and obtaining a range for each expression, we initialized the mean to 0 and standard deviation to 0.25 for all joint angles ( θ ⃗ ) for both Elenoide and Alfie. We considered the following joints for Alfie: BLINK_LEFT, BLINK_RIGHT, BROW_UP_LEFT, BROW_UP_RIGHT, BROW_DOWN_LEFT, BROW_DOWN_RIGHT, SMILE_OPEN, PHONE_CH_J_SH, and PHONE_BIGAAH. For Elenoide, we considered the joints in the eyes, eyebrows, and mouth (mouth opening and mouth corner pull). Alfie is able to generate angry, happy, neutral, sad, and surprise expressions (see Figure 4). Elenoide is able to generate only happy, neutral, and surprise expressions (see Figure 5). Owing to restrictions in the degrees of freedom in the mouth and eyebrows, Elenoide is not able to generate negative facial expressions like sad and angry.

To have consistent values for the joint configurations, we averaged the loss in Equation 4 for five generator and five classifier models with different random initializations. The graphs showing the loss and accuracy for the five classifier models can be seen in Figure 6. The epoch number where the loss is minimum is selected for the five classifier models that are then used in the ExGenNet for obtaining the values of the joint angles for various expressions.

After generating the expressions for the two robots, we verified the facial expressions using the previously trained classifiers. For both Elenoide and Alfie, we tested the five classifiers to recognize the expressions for images obtained at μ − 2 σ , μ − σ , μ , μ + σ , μ + 2 σ . We also sample 10 images from μ and σ and average the classification results. On Elenoide, the average accuracy for recognizing the expressions correctly was 81% for the five classifiers trained on human and robot dataset (see Figure 7). For classifiers trained on only robot and only human datasets, the average accuracies were 69 and 52% respectively. In the case of Alfie, we trained the ExGenNet and obtained the mean and standard deviation for various expressions on the simulator. Figure 4 shows the final results on the real robots. For the classification results on the images of the real robot, the average accuracy was 70% for the five classifiers trained on the human and robot dataset. The classifiers trained on only human and only robot datasets had average accuracies of 43 and 30%, respectively. We also tested the classification results on Alfie’s simulator. The average accuracy for the five classifiers trained on human and robot data combined was 75%, followed by the classifiers trained on only robot data 51%. The accuracy for the classifiers trained on only human data was 39%. For Elenoide, Alfie, and Alfie’s simulator, the classifier that is trained on the combined dataset of humans and robots gives the best results. While the classifier trained on only robot data has a higher expression recognition rate for Elenoide and Alfie’s simulator, the classifier trained on only human data has better results in the case of Alfie. This is because the classifier trained on only robot data consists of simplified face images of only Elenoide and Alfie’s simulator. As Alfie and Alfie’s simulator has a different rendering and the classifier trained on only robot data overfits to the data of Elenoide and Alfie’s simulator, the classifier trained on only robot data performs worse than the classifier trained on human data for Alfie.

To validate the results further, we also conduct a pilot study to investigate how images of autogenerated expressions are perceived by humans. Here, we conducted three surveys, one for humans to be able to recognize the expressions generated on Elenoide (happy, neutral, and surprise) and another two for humans to be able to recognize the positive expressions (happy, neutral, and surprise) and the negative expressions (angry, neutral, and sad) on Alfie. In each survey, we randomly showed three images per expression and asked two questions. The first question is how positive or negative the expressions look and the second question is to recognize the expression in the image out of five categories: angry, happy, neutral, sad, and surprise. There is also an additional optional category for “other” expressions in case it seems that the expression in the image does not belong to any of the five categories. The first question is rated between 1 and 7, where 1 indicates that the expression looks negative and 7 indicates that the expression looks positive. The second question is rated on a five-point Likert scale, where 1 indicates strongly disagree and 5 indicates strongly agree. In between the survey, there was a screening question to check the participants’ attention. The screening question asked the participants to choose “strongly agree” on the same five-point Likert scale.

We obtained responses from 30 participants for each survey. We did not consider the responses of the participants who answered the screening question incorrectly and ended up evaluating the results for 27 participants in the case of Elenoide, 29 participants to recognize the positive expressions for Alfie, and 27 participants to recognize the negative expressions for Alfie. Out of all the participants, 65% were male and 35% were female with an average age of 35 years. We performed the Pearson’s correlation test using the Scipy library in Python. First, the mean values of all the expressions for all participants on a five-point Likert scale were calculated. Next, the intended labels were assigned. For example, if the intended class is happy, the intended labels would be [1,5,1,1,1,1]. The Pearson’s correlation test was performed to compare the classes chosen by humans with the intended classes. The Scipy library in Python returns both the Pearson’s correlation coefficient and a two-tailed p-value for the Pearson’s function. The Pearson’s correlation coefficient and the two-tailed p-value are given in Table 3 for various expressions.

We found that in the case of Alfie, all the positive expressions are positively correlated, happy being the most positively correlated, followed by surprise and thereafter neutral. In the case of negative expressions, sad was found to be negatively correlated. Alfie does not have a joint configuration in its mouth that can make the corners of its mouth turn downward in case of a sad expression, always forming a slight smile. Therefore, Alfie cannot express sadness the way a human does. Neutral and angry were found to be positively correlated. The p-value for all expressions except sad was less than 0.05. Therefore, the correlation coefficient for all expressions except sad is significant. In the case of Elenoide, all three expressions were positively correlated, “surprise” is the most positively correlated, followed by happy and neutral. For neutral, the p-value was greater than 0.05, implying that the result is not significant. While observing the data obtained from the participants, it was found that the neutral images of Elenoide were reported as both happy and neutral. This is probably because the corners of the mouth for Elenoide are always pulled upward even for neutral expressions, forming a smile.

While our proposed approach successfully generated multiple joint configurations for facial expressions of the two robots, which were correctly recognized by human subjects in the majority of cases, we also noticed some limitations of the current method.

The robots used in our experiments seemed to show hardware limitations, which made it harder to generate negative expressions than positive ones. The mouth corners of both the robots cannot directly move down, making it hard for the two robots to express sadness the way humans did in the training images. Also in the case of the neutral expression on Elenoide, human subjects would sometimes mistake it for happy, due to the constantly slightly upward pull of the mouth corners. One question here is whether we require robots to show negative expressions and if yes, whether they should show these in the same way humans would or not. Besides considering full ranges for expression generation possibilities in hardware design, it might also be interesting to investigate if there would be ways to express sad on our robots, which would deviate from the human training images but could still be classified correctly by human subjects.

Another limitation of our method in the current form is that even though we are able to generate multiple joint configurations per expression, they do not directly map to a level of intensity of the expression. However, humans usually are able to generate and recognize different intensities in expressions and we, therefore, consider this an interesting direction for extending our method in future work.