Our proposed conversational agent consists of two platforms. First, a template-based platform that contains conversations with pre-defined options and exercises that assist participants in improving their mental distress based on CBT principles (CBT-Bot). Second, a generative dialogue platform that allows conversations regarding various emotional issues in an open-ended manner (i.e., without requiring the users to choose predefined conversational options) and provides emotional support (ES-Bot).

Prior to the study, we used G*Power 3.1 to calculate the required number of participants. We set the large effect size f to be 0.40 while setting the power (1−β error probability) and α error probability to be 0.90 and 0.05, respectively. Thus, the required number of participants was calculated as 102. An online poster was made to recruit participants. We asked colleagues and friends to help release the recruitment information on their social media platforms, such as WeChat and Weibo. Participants who were interested in the study could contact our team based on the provided information in the advert. The following criteria were used to recruit participants through online posters: participants were required to be at least 18 years old, able to use a smartphone, not currently in therapy as it would interfere with our study, and not suffering from physical issues such as physical illness or not taking medicine as they might influence their psychological state.

A total of 412 participants registered for the intervention, and 301 met all the above criteria. A research assistant, who was blinded to the purpose of the study, assigned a code based on the order that the participants contacted them. Accordingly, the participants were randomly assigned to three groups: Emohaa (CBT-Bot), Emohaa (Full), and the control group. Considering the relatively long waiting time and the potential number loss in the control group, we randomly allocated 30 more participants to the control group, adopting an approximate 3:3:4 allocation ratio for the 3 groups. The current study used a blank control group in which participants were asked to wait for a month before they would receive mindfulness intervention.

After signing the consent form, participants were instructed to take the pre-test (T1) questionnaires, including PHQ-9, GAD-7, PANAS, and ISI (Section 2.2), and their demographic information. Sixteen participants were excluded from the study and referred to relative authorities for professional help as they were at risk of suicide according to their scores on an item from PHQ-9 (i.e., “how often have you been bothered by the thoughts that you would be better off dead or thoughts of hurting yourself in some way?”), and 38 participants were excluded because they did not complete the pre-test survey. Overall, 72 participants in the Emohaa (CBT-Bot) and 70 participants in the Emohaa (Full) completed the pre-test questionnaires. 105 participants in the control group completed pre-test questionnaires.

The entire intervention lasted for three consecutive weeks (i.e., 21 days). Then, one day after the end of the intervention, all the participants were asked to fill in the post-test (T2) questionnaire, which included the same items as T1. Additionally, one month after the end of the experiment, participants were invited to fill in a follow-up questionnaire (T3) intervention to track the lasting effect of the intervention. From the perspective of health ethics and practical reasons, other forms of intervention could have been provided to the control group after the intervention. Hence, there were no valid data at T3 for the control group. The above recruitment process is illustrated in Figure 4.

Of the randomized participants, 54.2% (134/247) went on to provide partial or complete data at T2. Independent t-tests analyses did not detect evidence of significant differences at T1 between those who dropped out of the study versus those who did not on age (t=1.51; p=0.132); gender (t=0.37; p=0.709); working tenure (t=0.92; p=0.357); PHQ-9 (t=0.95; p=0.342); GAD-7 (t=0.59; p=0.558); PANAS of positive (t=1.02; p=0.145) and negative (t=1.21; p=0.227) affect scores; or on insomnia (t=1.17; p=0.244). Additionally, we employed MCAR analyses (54) to test whether the data are missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR). The results showed that Chi-Square = 42.98 (p<0.001), demonstrating the MCAR pattern.

Regarding the conversations with Emohaa, participants were instructed not to share any personal information (e.g., name, address, and date of birth) that could be used to identify them. The data collected during the experiment are anonymized, stored securely, and will be available for research purposes through a request to the corresponding author. The studies involving human participants were reviewed and approved by Beijing Normal University’s Institutional Review Board (IRB Number: 202209150101). Written informed consent to participate in this study was provided by the participants.

Demographic information of our studied sample (n=247) is provided in Table 1. Overall, the majority of participants were female (107/134, 79.85%). The average age of the studied sample was 30.90 years old (SD=7.92). Participants had worked for an average of 7.87 years (SD=8.45) prior to the experiment. All of the participants were from Mainland China. As the baseline for participants’ mental distress, on average, the samples showed moderate ranges of depression (Mean=16.43, SD=5.01), moderate anxiety (Mean=16.23, SD=4.37), and moderate insomnia (Mean=16.45, SD=5.38). In regards to PANAS, participants, on average, demonstrated moderate levels of positive (Mean=24.76, SD=7.20) and negative affect (Mean=22.34, SD=6.35).

We employed ANOVA and chi-squared test to see whether there were significant differences in baseline variables (age, gender, education, PHQ-9, GAD-7, PA, NA, Insomnia) among the three groups. The results showed that the three groups were not different in terms of baseline demographics of age (F=2.17, p=0.117) and gender (χ2=3.56, p=0.173). Additionally, the baseline variables of PHQ-9 (F=2.45, p=0.088), GAD-7 (F=0.93, p=0.396), PA (F=0.83, p=0.438), NA (F=2.82, p=0.061), and Insomnia (F=2.76, p=0.065) showed no significant differences among the three groups.

We used the last observation forward (LOCF) method to conduct ITT analyses. Previous systematic reviews have shown that LOCF is one of the most commonly used and relatively conservative strategies in ITT analysis (56, 57). To investigate whether the effects of interventions were different from each other and from that of the control group, we conducted a one-way repeated measures MANOVA with time (two levels: pre-test and post-test) and group type (three levels: Emohaa (CBT-Bot) vs. Emohaa (Full) vs. Control) as the independent variables and the five mental health indicators as the dependent variables. First, as presented in Table 2, there were significant Group × Time interaction effects on depression, F[2,244]=6.26, p=0.002, η2=0.050, indicating a significant difference in participants’ depression changes among the three groups, and such difference had a relatively small effect size that is smaller than 0.06 (58). Specifically, as Figure 5A shows, the intervention effects on depression stemmed from decreases in both Emohaa (CBT-Bot) (t=−2.25, p=0.027) and Emohaa (Full) group (t=−2.09, p=0.040) from pre-test to post-test, but there was an increase of depression in the control group (t=2.04, p=0.044) from pre-test to post-test. Moreover, we did not find significant differences between the two types of interventions (F[1,140]=0.76, p=0.386).

Similarly, we conducted the same MANOVA analyses to test whether participants’ anxiety changed differently over the intervention period. The results revealed a main effect of time on anxiety, F[1,244]=27.66, p<0.001, η2=0.102, indicating participants’ anxiety decreased over time. As Table 2 shows, the interaction effect of Group × Time was not significant on anxiety F[2,224]=0.60, p=0.556, η2=0.006. No significant differences were found in the effects of the two types of interventions (F[1,140]=0.38, p=0.538).

Additionally, the results showed that there was no main effect of time on positive affect (F[1,244]=2.39, p=0.123, η2=0.010) nor on negative affect (F=2.88, p=0.091, η2=0.012), meaning that participants’ positive affect increased and negative affect did not change over time. We did not find a significant interaction effect of Group × Time on positive affect (F[2,244]=1.58, p=1.208, η2=0.013). The interaction effect of Group × Time was significant on negative affect (F[2,244]=6.09, p=0.003, η2=0.048). Specifically, as shown in Figure 5B, participants’ negative affect decreased significantly in both Emohaa (CBT-Bot) (t=−2.20, p=0.031) and Emohaa (Full) (t=−2.04, p=0.045) groups from pre-test to post-test, but their negative affect significantly increased in the control group (t=2.11, p=0.037). The post-hoc results showed that there were no significantly different effects of PA (F[1,140]=0.01, p=0.943) and NA (F[1,140]=0.01, p=0.955) between types of interventions.

Finally, the MANOVA results demonstrated a main effect of time on participants’ insomnia (F[1,244]=4.49, p=0.035, η2=0.018), indicating that participants’ insomnia decreased during the period of intervention. Besides, the results showed a significant interaction effect of Group × Time on insomnia (F[2,244]=3.69, p=0.026, η2=0.024), but the differences had a relatively small effect size that is slightly larger than the small effect size of.01 (58). Specifically, as Figure 5C reveals, the effects stemmed from a significant insomnia decrease in the Emohaa (CBT-Bot) group (t=−2.28, p=0.026), no significant change of insomnia in Emohaa (Full) group (t=−2.03, p=0.055), and no difference in the control group between the pre-test and post-test (t=0.88, p=0.379). We did not find significantly different effects of the two types of interventions (F[1,140]=0.02, p=0.936).

During the experiment, participants had 7 conversation sessions with Emohaa on average (SD=6.62, Max=18, Min=4). These sessions had a mean of 17 conversational turns (SD=10.68, Max=87, Min=5). N-gram analysis was used to investigate the characteristics of participants’ conversations with Emohaa. The most discussed keywords were found to be 感觉 (feeling; 33%), 工作 (work; 20%), 心情 (mood; 11%), 学习 (pressure; 10.5%), 朋友 (friends; 9.2%), and 孩子 (children; 7.7%). Percentages indicate the proportion of conversations that included the keyword. The main problems that participants wanted to talk about were 工作环境 (Work environment), 工作压力 (Work pressure), 浪费时间 (Wasting Time), 集中注意力 (Keeping focus), 牺牲休息时 (Sacrificing leisure time), and 转移注意力 (Diverted attention). In general, participants were mainly interested in 正念冥想 (Mindful meditation), 早点休息 (Resting early), 身体健康 (Being healthy), and 提供情绪价值 (Providing emotional value).

After the end of the intervention, participants who had used Emohaa during the experiment were instructed to complete an additional survey to evaluate the agent’s performance. Most participants (60/69, 86.9%) reported that they had never received psychological counseling before the experiment, and only two had taken psychotropic medication.

Initially, participants were asked to rate the CBT-Bot platform’s ease of use, provided content, and interface quality on a 10-point Likert scale. Most participants reported moderate to high levels of satisfaction (scores ranging from 7 to 10) with the platform’s functionality (56/69, 81.16%) and the designed exercises (47/69, 68.12%). In addition, more than half of the participants (43/69, 62.32%) were satisfied with the interface design. Overall, the majority (49/69, 71%) reported that they would recommend this platform to others.

Similarly, participants who had used Emohaa’s ES-Bot platform were instructed to rate its performance. This platform was considered by most of the participants as an appropriate chatting partner (24/31, 77.42%) and channel for emotional venting (20/31, 64.5%), and more than half of the participants (18/31, 58.1%) reported that chatting with this platform made them feel heard. When asked about their expectations of the platform, the majority believed it to be a suitable companion for emotional companionship and support that can accurately interpret their emotions and provide emotional counseling (21/31, 67.74%). In addition, more than half of the participants were satisfied with the interface (20/31, 64.5%) and reported that they would recommend it to others (19/31, 61.3%). Independent t-tests showed that there was no significant difference between participants’ satisfaction with the Emohaa CBT-Bot platform and ES-Bot platform (t=1.16; p=0.250).

Lastly, participants were asked to provide feedback on their experience with Emohaa. Table 3 summarizes the most common themes in the collected responses. The most frequently raised concerns were technical issues, unclear instructions, and limited content and choices. The reported technical issues were mainly regarding the user interface (e.g., “Cannot click the next page” and “Accidentally closing the app removed all my progress”). Many participants were overwhelmed with the number of categories in the guided writing exercises and felt some topics were illogical and not applicable to real life. Participants also felt that the stories in the automatic thinking exercises were excessive while there were not enough options to describe their mood and emotions after completing the exercise. Moreover, it was suggested that the dialogue options in the platform’s template were inadequate.

Issues regarding the performance of the ES-Bot platform were also raised. Several participants reported that the conversations were rigid, and the system needed user guidance to continue the conversation. In some instances, the generated responses were reported as irrelevant or incoherent to the conversation. Participants also highlighted the platform’s occasional inability to remember what had been said in the early stages of the conversation, initiate conversation topics, and understand various input types (i.e., audio, video, and image).

In addition, participants were also asked to provide suggestions on how to improve Emohaa. As concerns regarding lack of content and options were mentioned, it was suggested that additional scenarios, stories, instructions, and options be included in the CBT-Bot platform. The importance of regularly updating the platform and promptly fixing technical issues was also highlighted. Regarding the ES-Bot platform, nearly half of the participants (14/31, 45.1%) believed that improvements for making the generated responses less rigid were necessary. It was also suggested that support for different input types be added to create a more interactive and engaging experience. Many also believed that recommending mental health-related content during conversations and taking the initiative in conversations would benefit this platform.

The obtained results demonstrated Emohaa’s efficacy as a short-term intervention for depression, negative affect, and insomnia. Based on the survey results, users experienced reduced levels of mental distress in the measured categories after using Emohaa. Compared to the control group, there was a significant decrease in depression among the participants who used Emohaa, as measured by the PHQ-9 questionnaire. Similarly, as measured by the PANAS and the ISI questionnaires, their negative affect and insomnia were also considerably reduced. Therefore, as shown by the experimental results, Emohaa can be seen as an effective tool for mental health support.

Regarding the difference in outcomes between the two groups that used Emohaa, no significant differences were found in the short term. Both interventions effectively relieved individuals’ mental health symptoms. However, as provided by the supplemental analyses (Section 3.2.1), participants who used the ES-Bot platform showed comparatively fewer indicators of insomnia. This finding highlights a potential benefit of emotional venting in improving problems regarding sleep in the long term.

Based on the obtained feedback, most participants were satisfied with this agent and considered recommending it to others. In line with previous research (16, 17), the results of the conversation analysis indicated that participants were willing to self-disclose their emotional problems, as shown by their most discussed keywords and topics. Moreover, most participants considered Emohaa’s ES-Bot platform a chatting partner that can effectively listen to their problems and provide a channel for them to vent their emotions. Notably, the majority felt that this platform could understand their emotions, an essential feature of conversational agents for support and a crucial trait for establishing a therapeutic connection (26). Therefore, our findings suggest that Emohaa can also be seen as an acceptable and feasible tool for support.

In addition to highlighting Emohaa’s effectiveness in mental health support, this study demonstrated the potential of generative conversational agents and combining emotional and cognitive support to reduce mental distress symptoms. Our findings suggest that allowing users to converse about their desired topics with the agent freely has a complementary effect when added to more common forms of machine-based support (i.e., template-based conversations and exercises for cognitive support through CBT).

This study had several limitations regarding its design and methodology. The study duration was limited; thus, only two assessments of participants’ mental distress were made. Although a follow-up screening for participants that had used Emohaa during the experiment was conducted, no data regarding the control group’s participants were gathered as they might have received other interventions after the initial two-week screening. Furthermore, the number of remaining participants in the follow-up survey is inadequate to draw a conclusion that the conversational agent (ES-Bot) is better than the CBT-Bot. Future studies would benefit from collecting more data from the three groups in the follow-up surveys to support the complementary effects of generative dialogue platforms for emotional support.

It is believed that the number of participants was sufficient to demonstrate the preliminary effects of employing conversational agents for mental health support in theory. However, the sample size and the experiment duration are inadequate for generalizing the obtained results of this study to the public. Future experiments will include a larger sample size and longer study duration to further ensure the generalizability of Emohaa’s effectiveness in reducing mental distress. In addition, our adopted method of advertisement for this study could have introduced a bias in our recruitment process, in which individuals who were in some way connected to our helping colleagues and friends were more likely to participate in the study. This could have also affected the male-to-female ratio among the participants, leading to the over-representation of female participants in our sample.

As mentioned, Emohaa’s several technical issues could substantially impact the users’ perceived level of empathy and support (14), so they should be resolved promptly. A management system for addressing similar issues on time should also be implemented in future work. Moreover, several participants raised issues regarding Emohaa’s limited content (i.e., exercises and options) and unclear instructions. Similar to Liu et al. (26), a wider variety of psychological resources will be consulted in future work to expand the provided content in the CBT-Bot platform and revise the instructions to avoid user misinterpretations or confusion. Lastly, although our requirements regarding the daily usage of this platform could be applicable in a trial, such constraints are not practical in real-life applications. Hence, future work could further improve user engagement within machine-based interventions.

Regarding the ES-Bot platform, several reported instances suggested that Emohaa forgets the information in previous turns and that the generated responses are irrelevant to the context, which could impair user engagement and rapport. This is a well-known issue in current language models (59), and the main reason could be the limited number of words in the model’s input (128 words for Emohaa). A feasible approach to address this issue is to add a module that could summarize the essential information of the previous turns in the conversation (60). In addition, previous work has demonstrated the benefits of adding persona (61–63) and commonsense knowledge (64, 65) for improving user experience with generative conversational agents. Future work could explore these additions to study their efficacy and corresponding improvements in mental health support.

The present study introduced Emohaa, a Chinese conversational agent for mental health support. Emohaa employs CBT principles to provide cognitive support through template-based guided conversations for expressive writing and automatic thinking exercises. In addition, it includes a platform for providing emotional support in which users can discuss their desired emotional problems. This study examined the effectiveness of Emohaa in reducing mental distress and investigated its feasibility and acceptability as a tool for mental health support in China. Our findings demonstrated that participants experienced fewer symptoms of mental distress after using Emohaa for the duration of the study. Hence, we believe this agent could serve as a valuable tool for reducing users’ mental distress, namely depression, negative affect, and insomnia. In addition, we found that there might be a complementary effect on long-term insomnia when implementing the generative dialogue platform for emotional support. This finding highlights the potential of generative conversational agents for the future of mental health support. In the future, we hope our work can inspire other studies to expand upon our research, leverage generative models for providing support, and investigate their comparative efficacy.