VP systems can provide explicit medical skills training recommended for health professionals’ education to reduce the impact of future diagnostic errors and potential patient harm Cleland et al., 2009; Balogh et al., 2015. Several research projects are investigating the use of VP systems in the education of medical students Campillos-Llanos et al., 2020; Pantziaras et al., 2015; Mavrogiorgou et al., 2022; Graf et al., 2023b. Thereby, the art of VPs can vary from chatbots Cameron et al. (2019) to embodied conversation virtual agents Campillos-Llanos et al., 2020; Pantziaras et al., 2015. They can be accessible via different devices like computers Pantziaras et al., 2015; Campillos-Llanos et al., 2020 or VR headsets Mavrogiorgou et al., 2022; Graf et al., 2023b. Several review articles have investigated the effectiveness of VP systems over the past years Cook et al., 2010; Milne-Ives et al., 2020; McGaghie et al., 2010; Plackett et al., 2022; Kocaballi et al., 2019; Isaza-Restrepo et al., 2018. A systematic review by Cook et al. (2010) evaluated computerized VPs, especially in educating health professionals on the learning outcome. They also focused on the design features of the respective virtual patients. Their review included 48 articles, including VPs for medicine students, nurses, and other health professionals. Their results show that VPs show positive learning effects on clinical reasoning and knowledge acquisition compared to no intervention but relatively small effects compared to non-computerized interventions. Regarding the design features, they could show that repetition, extended feedback from the VP system, and explicitly contrasting cases can improve learning outcomes. Furthermore, features essential for the students were natural case progression (including collecting data, offering more and less restricted options, and adapting to the actions of learners), case realism, realistic dialogue flow, and working together in a group of students. Another later review by Plackett et al. (2022) also investigated the effectiveness of VPs, especially regarding clinical reasoning skills. They included 19 research articles covering VP systems from a range of disciplines. Only 58% of the reviewed studies reported significant positive effects of the VP systems on clinical reasoning skills, while 21% indicated mixed effects and 21% no effects. However, compared to other teaching methods (i.e., tutorials), 75% of the students showed no effects. Thus, VP systems seem to outperform having no intervention but not other teaching interventions regarding improved clinical reasoning skills. Their review also identified two main intervention features in VP systems. Most of the VP systems (68%) use feedback on the learners’ performance and thus align with recommendations from studies about simulation-based learning Schubach et al., 2017; Isaza-Restrepo et al., 2018. 50% implement a high level of interactivity, requiring the learners to gather information from the VP. Another review by Milne-Ives et al. (2020), focused on evaluating conversational agents in healthcare that are supported by artificial intelligence. Again, the review indicates positive or mixed effectiveness (76.7%) of the VP systems. Additionally, the majority of the reviewed VP systems seems to have good usability (90%) and user satisfaction (83.9%). Further, qualitative user feedback revealed that the most common complaint with conversational agents was poor comprehension due to a lack of vocabulary, inaccurate voice recognition, or improper word input error management. Users disliked the repetitive conversations, and the conversational agents frequently had to ask questions more than once to process the response. Furthermore, negative aspects were the difficulty of empathizing with the VP and the lack of representation of the situation’s complexity by the agent. They liked that VPs provided a risk-free learning environment, as they were not actual patients.

There are just as many disciplines for VP systems as there are in the education of medical students, not only in practicing ambulatory medicine Buysse et al. (2002), medical ethics Fleetwood et al. (2000), but also for mental health assessment skills Washburn et al. (2016) or diagnostics skills Mavrogiorgou et al., 2022; Pantziaras et al., 2015 developed an interactive desktop application where medical assistants conducted a psychiatric interview. The VP responded with pre-recorded video sequences. They can also physically examine the patient and order laboratory and imaging examinations. The learners then draw up a differential diagnosis and a treatment plan. In addition, they receive feedback from the patient regarding the consultation and from a virtual consultant who refers to the clinical performance. Their results show that the acquisition of basic knowledge in the field of psychiatry was improved. Mavrogiorgou et al. (2022) also developed a VP system for adult psychiatry using VR and embodied agents that allows students to interview an embodied VP using natural language input and output. However, this system still needs to be evaluated.

To design embodied virtual agents in a learning context, Doering et al. (2008) developed a framework that implies an agent should be attentive and responsive during the interaction and ready to respond. It should be able to reply to queries and obtain feedback. The messages it communicates should be adapted to the user’s experience and needs and contain congruent verbal and non-verbal elements. Finally, the agent should awaken believability and trust. The believability of virtual characters describes the acceptance that someone or something in a virtual world is perceived as real Allbeck and Badler (2001). Aspects that play an essential role in increasing the believability of virtual characters can be their appearance, body language, and voice Lim and Aylett 2007; Demeure et al., 2011 or interactivity Knoppel 2009; De Rosis et al., 2003; Baylor and Kim 2009 showed in their study that a visible and physically present agent positively influenced users’ motivation compared to a voice or a text box. Thereby, the appearance of virtual characters affects a player’s perception. For example, while Baylor and Kim (2009) showed that realistically designed agents were more beneficial, as cartoon-style agents reduced motivation in users, Graf et al. (2023a) showed that a comic-like and even animal-like virtual character could influence the emotional experience, as well as the motivation and performance of the players. Again, Zibrek et al. (2018) showed that participants were more concerned with a realistically rendered character than with characters rendered in less realistic styles. Considering the uncanny valley effect is crucial when choosing a degree of realism. It describes the sudden change of a user’s evaluation of an artificial human from positive to negative if it approaches photorealism but still has subtle characteristics that limit its realism Mori et al. (2012). Besides the appearance, an appropriate display of emotions is crucial for the believability of virtual agents. A study by Lim and Aylett (2007) showed that virtual agents showing appropriate emotions are more believable than those showing no emotions. Studies showed that learners liked VPs showing empathy and when having a personality Cameron et al., 2019; Dimeff et al., 2020 or disliked it when it was missing it Ly et al., 2017; Borja-Hart et al., 2019; Cook et al., 2010 define interactivity as the “degree to which the course design encouraged learners to engage cognitively.” Former research results are inconclusive about the effect of interactivity on learning outcomes Homer and Plass (2014). According to studies, increased interactivity can encourage more engaged users and deeper learning, but it can also increase cognitive load, which can impede learning Kalet et al., 2012; Homer and Plass 2014. In the VP context, studies showed that learners liked the interactivity of the VPs Hudlicka 2013; Ly et al., 2017 or wished for more interactivity Cameron et al., 2019; Håvik et al., 2019.

We evaluated our application with medical students. Each student had a conversation with a 14-year-old female VP suffering from depression using the Meta Quest Pro. Before entering the virtual world, the catalog was shown and explained to the participants. They interacted with the VP in the virtual world for 25.13–54, 56 min (M = 33.7, SD = 11.7) while they had to ask all 58 questions from the catalog. Therefore, the time they spent in VR was at least the time they needed to ask all the given questions. The total number of questions for each participant could differ as participants could also try to ask questions not included in the catalog. It could take longer depending on how long it took the participants to ask the questions. After that, they filled out question items regarding the believability of the dialog between them and the VP and its appearance. In the end, we conducted a 10-min interview with each participant to identify the advantages and pitfalls of their conversations with the virtual patient and how believable they perceived the situation. Furthermore, we tracked the progress of the questions, the answers given by the VP, and whether and how many hints the participants had asked for while using the application. As shown in Figure 1, for example, in the category habits and consumption behavior, the first hint shows the keyword alcohol, which means that the user should ask a question regarding the VP’s alcohol consumption. The second hint then shows the specific question Do you drink alcohol?

We recruited five medical students (3 female, 2 male) aged 23–26 (M = 24.2, SD = 1.3) in their 7th to 12th clinical semester via advertisement on a digital bulletin board; previous experience in child and adolescent psychiatry was not mandatory. Two students had prior experience in psychiatry diagnostics and relatively little VR (M = 3.6, SD = 0.89, Mdn = 4.00), gaming (M = 3.8, SD = 2.17, Mdn = 3.00), or experiences with virtual agents (M = 2.0, SD = 1.23, Mdn = 2.00) measured on a scale from 1 (= no experience at all) to 7 (= a lot experience). The participants filled out the questionnaire and were interviewed in German.

We designed a VR application for the teaching of conducting clinical interviews and diagnosing mental disorders in child and adolescent psychiatry using embodied virtual agents as patients. The application is structured by following a catalog of questions we created based on the AMDP system together with a Child and adolescent psychiatrist (see the catalog in the Supplementary Material). The AMDP system ¹ is an international standard for the methodical documentation of psychiatric diagnoses, developed by a German association for methodology and documentation in psychiatry. The catalog covers 17 categories relating to different symptoms or characteristics (e.g., habits and consumption behavior or affective disorders). Each category then contains one to six subcategories (e.g., alcohol, drug, and media consumption, or aggressiveness and mood swing) the students need to address in their interview. A pre-defined sequence of the question catalog guides the students through the conversation. However, asking about any symptoms and repeating individual questions is possible. The application uses natural language understanding (NLU) as input using Wit.ai ² by Meta. Wit.ai converts the user’s question into text and then assigns it to an existing intention, which outputs a corresponding voice output from a selection of prerecorded audios (see Figure 2). Based on the virtual patient’s answers, the students eventually have to decide on a psychiatric diagnosis. The VP is an embodied virtual model of a teenager with stylized aesthetics. We chose a stylized yet realistic approach compared to photorealism due to the risk of the uncanny valley effect Mori et al. (2012). Furthermore, studies show that photorealism is unnecessary to achieve behavioral realism Blascovich et al. (2002) or believability Graf et al. (2023b). Further, we decided against an even more stylized design, as in previous discussions, medical students indicated the preference for a realistic VP Graf et al. (2023b). Accordingly, we made the facial features, skin, hair, and clothes childlike. We also mapped the prerecorded facial expressions of a real actress onto the agent’s face and created a corresponding body language in the form of animations. We used Elevenlabs ³ , a generative voice AI, to synthesize a natural voice for the VP’s versatile responses retaining the emotionality of a human voice. We also created a user interface integrated into a virtual tablet the users have in the virtual world (see Figure 1). The tablet gives the students an overview of the catalog of questions. It displays the current category and allows students to ask for two levels of hints. The first hint shows the keyword of the subcategory of questions, and the second hint shows a sample question if students do not know how to ask for the respective symptom. We chose this two-level hint system so that students can use the application regardless of their previous knowledge of child and adolescent psychiatry.

Table 1 shows the descriptive values of the quantitative results regarding the believability of the dialog between participants and the VP and of the VP’s appearance. The believability of the dialog was rated on average on M = 4.2(SD = 1.92). But there was strong agreement that many hints were taken during the conversation. Satisfaction with the accuracy of the VP’s answers was rated particularly high (M = 5.2, SD = 1.92), and the lowest score was given to the question of how real the conversation with the VP felt (M = 3.6, SD = 1.52). Regarding the believability of the VP’s appearance, it received higher values. Here, the highest agreement was that the VP was appropriate for the situation (M = 6.8, SD = 0.45), and the lowest agreement was assigned to the VP’s facial expression (M = 4.4, SD = 1.52).

We analyzed the interaction logs (Table 2) and counted 19.8 (28.62%) errors in total by a total amount of 68.8 asked questions on average. All participants, besides participant (p01), asked all given questions and more. Participant (p01) had to stop early because the application crashed due to an internet connection error. The natural language understanding achieved 71.38% correct allocations on average. Overall, we found more errors defined as concept errors (20.3%) than system errors (8.32%). It can also be observed that a high number of hints are used for the correctly allocated turns (no errors). On average, both hints were used 13 times for the correct allocation, while both hints were used one time on average for concept errors and one time for system errors. With an average value of 30 times, participants used the level 2 hint (sample question) most frequently for the correct allocation. On average, up to three hints were used when errors appeared.

We transcribed all interviews and formed categories using the MAXQDA ⁴ . After one researcher categorized the answers into categories, another researcher independently checked whether they would assign the aspects to the same categories. We calculated no inter-rater agreement. In total, we derived five categories.

To our knowledge, only a few research articles focused on believability of simulation patients (human actors) Baylor et al. (2017) or virtual patients Rizzo et al. (2011), though without investigating individual design aspects of the VR system. One significant finding was that the predefined questions of the catalog limited the participants in the flow of their conversation with the VP, which resulted in a student-VP interaction that felt less natural. Our design aimed to create a natural conversation using natural language input and to guide the students through the clinical interview by presenting the question catalog. The fact that more system errors occurred shows that we needed to consider more questions on a conceptual level in advance. It resulted in participants asking questions to which the VP had no answer. Even after revising the catalog of questions and implementing a more accessible design, such errors could still occur. It is difficult to predict and thus prepare prefabricated answers to all possible questions. One enhancement could be the generation of missing answers to unforeseen questions using AI. Studies show that dynamic response behaviors of virtual agents were rated more positively compared to predefined ones Toader et al., 2019; Hsu and Lin 2023. However, the accuracy of the statements generated by the AI needs to be better and apparent beforehand Wang et al. (2023). This limitation poses a particular problem in the education context and, especially in a psychiatric context, statements invented by the AI could be over- or misinterpreted. Generative AI, such as ChatGPT, also usually follows certain restrictions, such as not making statements about suicidal behavior to protect users. Therefore, future research should validate the adjustment of the prompts precisely to monitor response behavior in the best possible way regarding the teaching content.

We want to have a critical look at our evaluation. To measure the believability of the design elements, we did not use validated questionnaires but custom items applicable to our use case. Other researchers may consider using a scale like the one of Guo et al. (2023). This and the small number of medical students who tested our application must be considered when inferring conclusions from our results for other projects. Further, we have yet to assert the effectiveness of our system. In the future, we want to enhance, in particular, the dialog system and evaluate further elements like a feedback system (e.g., the VP giving verbal feedback during the dialog) or playful elements (e.g., receiving points for correct diagnoses) and their impact on believability and eventually students’ learning outcomes.