Fourth-year medical students complete an obligatory 1-week practical course at our psychiatric clinic. In the winter semester of 2019/20, when the course was held on-site, we examined 156 students in an objective structured clinical examination (OSCE) of which 150 (return rate: 96% female: 87, male: 54, unknown: 9) participated in the questionnaire. For the online teaching summer semester of 2020, 175 students were examined and 156 (return rate: 89% female: 93, male: 65, unknown: 2) participated in the questionnaire.

Additionally, we questioned all 16 (return rate: 75%) lecturers teaching in the summer semester about their experiences. The lecturers were the same in the on-site and online semesters, with the exception of four lecturers due to clinical rotations.

The participating medical students and lecturers voluntarily agreed to participate in the survey. The students were explicitly informed that participation would not impact their exam grades and that the questionnaire answers and grades would be processed anonymously. This study was submitted to the ethical committee of the Friedrich-Alexander University Erlangen-Nuremberg and received a designation of exempt.

The practical education consists of 5 course days and the following OSCE. Each day includes a 45-min theoretical seminar about common psychiatric syndromes presented by one of our senior physicians. In previous semesters, two to three real and/or simulated patients were then interviewed by the students in groups of seven on the wards. During the online semester, we used conference calls, in which only simulated patients were available due to privacy protection concerns. The actors, lecturers, and students gave feedback based on their perception of the interview. We expanded the course in the digital setting by offering a 20-min group exercise every day. Additionally, we offered a quiz for self-testing and attached resources based on our online lecture content such as podcasts and reading material (5). After 1 week, one of four standardized patient cases was given to the students as an OSCE. In the summer semester of 2020, this OSCE was also carried out via video conference.

The pool of simulated patients includes 6 professional actors and 10 amateurs, with a mean of seven semesters of experience as a simulated patient (5 males, 11 females) and around 16 assignments in each semester. Roles played include dementia, delirium, schizophrenia, major depression, bipolar disorder, adaptive disorder, obsessive–compulsive disorder, anxiety disorder, sexual arousal disorder, and sexual dysfunction. They are regularly trained and supervised in their roles, as well as in giving feedback to the students.

Simulated patients will perform one out of four clinical cases (dementia, schizophrenia, bipolar disorder (current depressive episode), obsessive–compulsive disorder). The examiners are not the same as the lecturers. To provide a fair exam, we use a standardized questionnaire, rating performance of communication models, medical thoroughness, and the following case presentation (Supplementary Figure 1). The time frame of the interview was 15 min plus 90-s patient presentation. We taught and examined two communication models: the WRMS model (Wait-Repeat-Mirror-Summarize), used to support active listening (10), and the more advanced NURSE model (Naming-Understanding-Respecting-Supporting-Exploring Emotions), aiding the use of emotions in communication (11). One of the four cases (major depression/bipolar disorder) included taking the sexual history of patients. The scores we compared can be seen in Table 1. The same OSCE has been performed in the last four semesters, showing no improvement of the overall results over time. This indicates that previous knowledge of the exam structure is not advantageous to the students' performance; rather, their actual subject knowledge is examined.

The students answered the questionnaire following completion of the course and prior to the exam. We assessed 15 items, including interest, motivation, self-assessment, in-course feedback and overall enjoyment. The rating was based on German school grades [1 (totally agree) to 6 (totally disagree)]. The students' gender and individual study duration were collected as demographic data (Supplementary Figure 2).

Furthermore, students were asked to compare the online course with their previous on-site learning experience (better–same–worse). Lecturers were asked to compare the online course to the previous semesters (better–same–worse).

We used Zoom (Zoom Version 5.0.2., Zoom Video Communications Inc., California, USA) as a platform for conducting video conferences.

Ilias StudOn (ILIAS Version 5.4.17, ILIAS open-source e-Learning e.V., Cologne, Germany) is the online e-learning system of the University of Erlangen-Nuremberg.

Statistical analysis was performed using SPSS (SPSS 21, IBM, Armonk USA). We used hierarchic linear modeling (HLM 7, Scientific Software International, Inc., Skokie, USA). Level-1 variables describe data collected from a single student (gender, duration of study), while level-2 variables consider data collected in the same week (lecturer, simulated patients, examiner, OSCE case, online/on-site teaching) (12). Results of the hierarchic linear modeling are reported as the difference between slopes (b) of the regression curves and its significance (p).

We compared the total number of points achieved in the OSCE as well as various sub-scores in the last on-site semester (winter semester 2019/20, n = 150) with those achieved in the e-learning semester (summer semester 2020, n = 156). The result of the power calculation indicated that with a total n of 306, small-to-medium effect sizes could be detected (ρ = 0.185). The intercept-only model revealed an ICC of 0.385. Thus, 38.5% of the variance in total exam score is between groups (including variables such as the semester, the lecturer, and the examiner) and 61.5% of the variance in the total exam score is between students (including variables such as age and duration of study) within a given group.

There was no significant difference in the total score (b = −1.47, p = 0.127), the communication scores (WRMS: b = −0.12, p = 0.425; NURSE: b < −0.01, p = 0.991), the psychopathological symptoms score (b = −0.28, p = 0.561), or the clinical presentation (b = −0.39, p = 0.306). However, there was a significant difference in the taking of the patient's sexual history (b = −0.91, p = 0.011), meaning that the students of the on-site semester performed better at this task than the students of the e-learning semester (Figure 1).

The students' gender significantly affected the exam results when including data from both semesters (b = 1.57, p = 0.003). In the subsequent analysis of the individual semesters, we discovered no significant relationship between gender and total score in the on-site semester (b = 0.73, p = 0.290), while there was a significant relationship between gender and total score in the online semester (b = 2.43, p = 0.002). Male students scored significantly fewer points in the online semester. We analyzed the relationship between gender and total score with consideration of the teaching method (online vs. on-site) as a potential moderator/mediator. In the analysis, we observed a difference between the estimation of fixed effects and the estimation of fixed effects with robust standard errors. Therefore, we decided to use the more conservative approach and report the results of the fixed effects with robust standard errors. There was a significant relationship between the method (online vs. on-site) and the total score (b = −4.42, p = 0.023), as well as a significant cross-level interaction between method and gender (b = 1.90, p = 0.042), whereas the relationship between gender and total score was no longer significant (b = 0.49, p = 0.495). This suggests that the method (online vs. on-site) is a mediator in the relationship between gender and total score: there is no stand-alone effect of gender on the total score; the effect is completely mediated by the fact that male students had worse total scores only in the online course environment.

The return rate of the student questionnaire was 98% for the on-site winter semester (n = 153) and 89% for the online summer semester (n = 156).

During the on-site semester, the students rated the following items better: the subjective support they received from their lecturer (b = 0.37, p = 0.017, Figure 2D), how much their interest in psychiatry was increased by the lecturer (b = 0.39, p = 0.030, Figure 2B), how much they enjoyed the course (b = 0.28, p = 0.031, Figure 2C), as well as how they would grade the course overall (b = 0.20, p = 0.034, Figure 2A). However, as previously reported, these differences did not impact the exam grade.

Contrarily, students' self-assessment in the online semester improved for three communication heavy skills: the use of the WRMS strategy (b = −0.43, p = 0.002, Figure 2F), the use of the NURSE strategy (b = −0.61, p < 0.001, Figure 2G), and their ability to take the sexual history of a patient (b = −0.40, p = 0.035, Figure 2H).

The questionnaire item “I think the structure of the course is suited to teaching psychiatric expertise” was rated better for the on-site semester (b = 0.32, p = 0.003, Figure 2E).

When asked to compare the online course with their previous on-site learning experience, 50.0% of the students of the summer semester preferred the on-site semester, 24.2% preferred online teaching, and 25.8% rated them equally (n = 120).

Lecturer feedback had a return rate of 75%. Seventy-five percent of the lecturers preferred the on-site semester, 8% preferred the online semester, and 17% rated them equally (n = 12).

The existing digital structures were flexible enough in executing our plans to combine video conferences with online group work, as well as providing online materials for self-study. Zoom proved viable in imitating a telemedical setting. The online learning tool of the university, based on ILIAS, provided options to implement resources and assignments in a variety of media formats (weblinks, pictures, videos, podcasts, and texts), allowing for a more diverse learning experience. Students, lecturers, and actors had no issues using the platforms after an introduction. Technical problems did not occur on a relevant scale. Thus, the realization of the project did not meet many obstacles. As a result, the exam and questionnaire results were interesting to observe.

The students' overall performance and the sub-scores were not relevantly impaired in the exclusive online teaching environment.

The only score that significantly changed was the taking of the sexual history. It is worth noting that this score is tested only in one-fourth of the exams (major depression/bipolar disorder). Wide fluctuations of this score have therefore also been observed in previous semesters. However, since sexual history often is a difficult topic to address (13), one could argue that the training lacked in thoroughness, the lecturers expressed more personal reservations in the online setting, or the online setting was indeed unsuitable in forming a dependable student–teacher relationship. Regarding the latter, Palmer et al. (14) created an online course to train students to interview patients about adverse sexual effects of SSRIs and observed positive communicative developments. This could indicate that the online setting itself is not at fault, rather, the degree to which our lecturers adapted the teaching of the sexual history taking to an online environment. We hypothesize that reservations (e.g., shame) regarding talking about sexuality may be harder to overcome in an online environment, and this impacted the quality of the teaching.

Overall, the results did not indicate a difference in the general student performance, thus, suggesting that the transfer of our course to an online setting did not have a considerable impact on the learning effectiveness.

Several authors studying the quality and practicability of online medical teaching reported similar findings. There is a wide range of methods described by the term “web-based learning,” complicating the navigation of literature (15). Our study included multiple such approaches, which were mostly tested as a stand-alone intervention. There are several findings that video lectures and live lectures are equally effective in teaching new content (1, 16, 17). Providing digital content, in addition to lectures and simulated patient interviews, is a useful tool to promote deep learning (18, 19). Topic-focused interventions can improve specific skills, such as taking the patient's sexual history (20, 21). The core of our course is the training of communication skills. Kaltman et al. (22) found that students trained with interactive online videos demonstrated more of the taught patient-centered interview skills in a face-to-face. The usefulness of simulations in medical education has been long since established (23). Parisi et al. used an approach using Zoom in small groups with simulated patients with diabetes, similarly to our study. They found it to be effective as well, but emphasized the lack of non-verbal communication training, which is especially important in intercultural contexts (24). Mohos et al. found the transfer of their course to an online environment to be a success, with consistent outcomes compared with previous semesters. Nevertheless, they raised concerns considering the missing personal contact (25), which we came across as well.

The students evaluated the on-site semester better in student–teacher-related items (subjective support from the teacher, increased interest in psychiatry, and overall enjoyment), as well as the course as a whole. This may indicate that the establishment of face-to-face contact with the lecturer plays a significant role in the perception of the experience as pleasant and helpful (26). As the participants' satisfaction influences the success of a didactic method (on-site/online), it may be considered that while the exam results stayed the same, the course did not entirely meet its standards (27). Thus, it should be aimed to include elements in online courses in which space for such interaction is provided (28). For example, an introductory round at the start of the week, as well as daily flash feedback rounds could be included to promote group synergy. More time for discussion of the topic could further improve this. With respect to the structure of the course, we repeatedly received verbal feedback emphasizing that it is preferable if the lecturer does not switch during the week.

In the online semester, the students self-assessed their communication skills (WRMS, NURSE, and SEX) better. It is notable that only the communication strategies but no other learning content, such as the examination of symptoms or the clinical presentation, were rated differently. This may have various reasons. The simulated patients may have been primed to react to these communication models in a more obvious way. It is also possible that without proper non-verbal communication, the students had to rely on the learned strategies or could better focus on the patient's reaction. Perhaps the lowered social anxiety allowed for bolder self-assessment. Special consideration should be given to the self-assessment of summer students, who rated their sexual history taking skills higher than the previous semester but achieved lower scores in their exams. This may indicate that the self-assessment of students, at least in the form tested in this study, is not reliable when compared with the results of an OSCE (29).

The method (online vs. on-site) is a mediator in the relationship between gender and total score. We found that female students reached a significantly higher average total score in the online semester. We cannot conclude whether this is because female students performed better during the online semester or if male students struggled more, neither whether it was directly caused by our e-learning method or surrounding circumstances. González-Gómez et al. (30) found that female students were more accepting of e-learning, which might be the reason for their better performance in the OSCE during the online semester. Thus, gender differences should be considered when investigating e-learning in educational research, as findings similar to ours have been reported before (31, 32).

We also observed that the majority of students (50%) and lecturers (75%) of the summer semester preferred on-site teaching. Students also thought that the structure of the on-site course is more suited to teach psychiatric expertise.

Mullins et al. (33) also found that students preferred on-site psychiatric lectures, even though they liked the flexibility and availability of online lectures. While there is still a preference for traditional teaching methods, the acceptance of online teaching methods is increasing, and participants adapt well when confronted with them (34, 35). Nevertheless, the lecturers' willingness and trust in the teaching method has a relevant role in its proper implementation (9).

The power analysis showed that we had a sufficient number of participants, but it should be considered that the summer semester of 2020, at the time of our intervention, was also severely influenced by the COVID-19 pandemic, which may have had a number of unforeseen effects on our results (36–39). Also, our groups where not randomized or controlled (40).

The OSCE testing we performed has been validated throughout the previous four semesters and has proved reliable. However, it cannot be excluded that conducting the exam online has an effect on the examiner that mingles with the influences of our intervention.

A general problem is that the variance of the lecturer, student, and examiner has a significant impact on the outcome of learning and exam results. The influence of the teaching method we investigated may be overshadowed by these variables.