Documentation burden poses a particularly significant challenge across various domains of medicine, exerting high pressure on healthcare providers and impacting the delivery of quality patient care (1). In clinical settings, precise documentation serves as a cornerstone for effective communication, continuity of care, and legal compliance (2). However, the process of capturing and recording patient information is often plagued by its labor-intensive and time-demanding nature. This burden manifests in diverse ways, from the need to manually write detailed clinical encounters to the exhausting task of navigating complex electronic health record systems. As such, physicians devote an estimated one-third to two-thirds of their workday on electronic health records, having to spend more time on health records than on direct patient contact (3). For example, a study regarding the documentation burden for physicians in the USA showed a mean of 1.77 (95% CI, 1.67–1.87) hours daily spent on documentation outside of office hours (1). Another US-based study reported 59% of the on average 1.5 hours after work spent on electronic healthcare record (EHR) (3). During office hours, 49.2% of physicians’ total time was spent on EHR and desk work. That, in turn, can be directed to clinical burnout (4–6) and lack of documentation quality, which can even lead to clinical errors (7).

While documentation burden can be pervasive throughout all fields of medicine, its repercussions may be particularly pronounced within the field of psychiatry. Unlike many other medical specialties, psychiatry revolves around the nuanced exploration of patients’ mental health and emotional well-being, relying heavily on interpersonal communication and rapport-building during interviews (8). Additionally, legal requirements for documentation play a growing role in psychiatry (9). As such, the documentation process in psychiatry is not merely a logistical hurdle but a crucial component of holistic patient care, underscoring the need for innovative solutions to streamline and enhance documentation practices in psychiatry and psychotherapy.

Artificial intelligence (AI) presents promising solutions for addressing the documentation burden pervasive in psychiatric practice and medicine at large. With the power of machine learning algorithms and natural language processing techniques, AI technologies offer the potential to optimize clinical documentation workflows. Through automated transcription, summarization, and analysis of clinical encounters, AI systems may alleviate the manual burden of documentation, freeing up clinicians’ time for more meaningful patient interactions and therapeutic interventions. Moreover, AI-driven documentation solutions have the capacity to enhance the accuracy, completeness, and consistency of patient records, mitigating the risks associated with human error and variability in documentation practices (10).

Lin et al. (2018) presented a theoretical prototype for an AI recording patient–physician encounters via speech recognition and summarizing, sorting, and assembling clinical information. The authors went even further, suggesting the AI’s ability to make clinical recommendations, predict clinical risks, calculate scores, and conclude ICD codes (11). They concluded that many technologies necessary for such an autoscribe already exist. However, they presumed that it remains unclear how autoscribes can be exactly developed regarding the high amount of data necessary. In the meantime, some AI tools have been evaluated in other fields of medicine. For example, a proof-of-concept study proved the ability of a software tool to automatically create surgical reports (12). In the field of radiology, several products have been discussed to improve clinical documentation via AI, for example, through voice-recording examination rooms or smart watches that transcribe conversations and create clinical notes (10). However, there have been no AI solutions proposed for the psychiatric field.

In this study, we aimed to evaluate the functionality and potential applicability of an AI software solution that is currently in development and has been utilized across various other medical specialties (13). While AI technologies have demonstrated promising results in automating clinical documentation tasks (14, 15), their utilization and effectiveness in psychiatry remain relatively unexplored. Recognizing the unique challenges and nuances of psychiatric practice, we aimed to assess the feasibility and performance of this AI software within the context of psychiatric interviews and their documentation. By conducting a proof-of-concept evaluation, we aimed to elaborate on the strengths and limitations of implementing AI-driven documentation solutions in psychiatric settings.

We aimed to examine how effective the AI software would be in voice-to-text transcription and summarization by calculating its error rate, and especially in how accurately content would be assigned to its respective category within the psychiatric report, by comparing AI reports to human-written reports. Our goal was to explore not only overall quality and accuracy but also differences regarding different categories of psychiatric reports, to, for example, individually test the capability of recognizing and interpreting psychopathology as a key aspect of psychiatric evaluation.

We utilized an AI software called QUIXXS, which is currently under development by Amnexis Digital Solutions GmbH, an Ireland-based tech company specializing in digital health solutions utilizing AI (13). The software is already in use in several clinical fields, but has not yet been evaluated in psychiatric environments and has not been tested in clinical studies. The AI software runs on smartphones as an Android or IOS application and utilizes the recording function of the smartphone.

During a doctor–patient encounter, the smartphone is positioned in such a way that both the patient and the doctor can be heard well. The application can also be used for dictations without patients. The recorded audio data can be converted into a transcript using speech-to-text functions. To meet high data protection requirements, transcripts are initially pseudonymized. Afterward, transcription is performed using the Web API Whisper, a neural net automatic speech recognition (ASR) system by OpenAI, which has been proven to show human-like accuracy and robustness (16). Employing GPT-4, a large language model (17), a medical information filter compresses the pseudonymized transcript by extracting medically relevant information. A stream-based approach was employed, in which the transcript is divided into segments of approximately 5 minutes each. Every segment is compressed using a domain-specific summarization prompt. The resulting partial summaries are then sequentially linked together into an overall synthesis, which serves as the essential basis for form filling and further AI functions (e.g., cross-sectional analyses). In the final step, a previously selected form template is completed with the obtained information. Templates are used to specify the external form, structure, and desired content of a report. A data/form matcher fills the individual fields of the template with the medical information, thus completing the creation process. Figure 1 depicts the processing pipeline of QUIXXS from audio input to clinical report generation.

For this proof-of-concept study, we simulated psychiatric interviews, as they would occur during the admission of patients for inpatient treatment at the clinic for psychiatry and psychotherapy of the University Hospital Charité Berlin Mitte. We conducted six interviews, one of which was held by a final-year medical student who was just completing their psychiatric rotation and conducting psychiatric interviews on the acute ward regularly, and two by a psychiatric resident. Interview partners, simulating a patient, were psychiatric nurses, residents, and volunteers who were not of a medical profession whatsoever. Those who were simulating to be a patient were instructed to present themselves as patients looking for psychiatric help because of some, not further specified, psychiatric symptoms. Those who were conducting the interview did not know beforehand which diagnosis the patients would introduce themselves with.

The interviews were held in the same manner as during a real, first physician–patient encounter on our acute psychiatric ward. Interviews were held in German and in accordance with standardized internal guidelines and protocols of our clinic, which were designed for psychiatric assessments. Patients were asked about their main symptoms, their psychiatric and medical history, their sociobiography, their family’s psychiatric history, and their current and recent medication. Psychopathology was assessed using a semi-structured interview according to the AMDP System: Manual for Assessment and Documentation of Psychopathology in Psychiatry (18).

During the interview, audio was recorded and then processed using QUIXXS. QUIXXS then interpreted from the recorded and transcribed audio files the role of the physician and the patient. Based on the transcribed and processed documents, QUIXXS assigned information to predetermined categories, such as patient information (including their name, age, and gender), current complaints or reason for presentation, psychiatric history, medical history, allergies, medication, social history, family history, vegetative symptoms, psychopathology, and preliminary diagnosis. The reports created automatically by QUIXXS were then saved as html files. After the interview, the physician wrote a report with the same, above-mentioned categories, as for a real inpatient treatment. The human reports were saved as Word files for future comparison. All reports were written in German, but excerpts for this paper were always translated into English for better understandability.

To test the quality of the AI reports, we tested error-proneness for two different steps of the report creation: transcription and reporting.

To test the accuracy of the voice-to-text transcription, the word error rate (WER) was used, which is the ratio of errors in a transcript to the total words spoken, the most common way to report the quality of voice-to-text transcription (19). Errors in this case would be substitutions, deletions, or insertions. For comparison, a transcript was created by a human as a ground-truth transcript. Both the human and AI transcripts were normalized: punctuation was deleted, and uppercase letters were changed to lowercase letters. Additionally, following changes were made: abbreviations were written out, slang words (such as “nah” instead of “no”) and contractions (“can’t” instead “cannot”, and “gonna” instead of “going to”) were written out, filler words (such as “hmm” or “uhm”) were deleted, and non-lexical conversational sounds were replaced with their equivalent (e.g., “no” instead of “uh-uh”). Grammatical errors of non-native speakers were corrected. Word error rate was then calculated using python v3.12.3 (20) and speechmatics python client v1.14.8 (21). In addition to WER, the Levenshtein score was calculated, which measures the similarity between the human and AI transcripts based on the minimum number of single-word edits (insertions, deletions, or substitutions) required to transform one into the other (22). This score provides a normalized similarity metric between 0 and 1, with 1 indicating perfect similarity. The calculation was performed using the python-Levenshtein library (23).

To assess the accuracy and quality of the reports, a reference standard was defined using the original audio recordings of the clinical interviews. A structured codebook was developed to operationalize systematic comparisons between this gold standard, the manually written reports, and the AI-generated reports. The codebook was grounded in standard domains of psychiatric assessment and comprised the following sections: personal information, current complaints, psychiatric history, somatic history, allergies, medication, substance use, social history, family history, vegetative symptoms, psychopathology, and preliminary diagnoses. Several sections of the codebook used multiple-choice response formats that combined fixed answer options with an additional free-text field. Fixed options covered the most frequently occurring categories, for example, commonly prescribed medications such as sertraline or mirtazapine, while free-text entries allowed entries that were not represented among the predefined options. Other sections relied exclusively on fixed single-choice options. An example is the assessment of visual hallucinations, with predefined responses of present and not present. If a report did not address visual hallucinations at all, no option was selected, and the corresponding item remained blank. Each response option within the codebook was treated as an independent binary item. For instance, within the substance use domain, nicotine use, alcohol use, and cocaine use were coded as separate variables. This design enabled a fine-grained, item-level analysis and deliberately avoided reliance on global free-text similarity measures. The exported codebook data were processed programmatically, and all responses were converted into string-based representations. Multi-select fields were automatically decomposed into individual binary indicators, resulting in a one-hot encoded item matrix in which each response option corresponded to exactly one analyzable variable. For each case and each item, concordance was assessed between the gold standard and the manual report, as well as between the gold standard and the AI-generated report. Items documented in a report but not supported by the audio recording were classified as false positives, whereas items present in the audio recording but absent from a report were classified as false negatives.

Structured codebook development and independent double coding, as employed here, follow recognized frameworks for quantitative content analysis in clinical research, providing a transparent and reproducible method for annotating and comparing key clinical concepts across sources.

To assess the inter-rater reliability of the proposed evaluation framework, a second independent rater re-rated all manual and AI-generated reports using the same codebook and rating procedure as the primary rater. Both raters were psychiatrically trained and blinded to each other’s ratings. Ratings were based on the structured item matrix derived from the original codebook, in which each answer option was represented as an individual binary item. Before statistical comparison, ratings were manually normalized to ensure semantic equivalence across raters, particularly for items derived from free-text inputs. This included harmonizing different phrasings referring to the same concept, for example, treating “leg fracture” and “broken leg” as equivalent. For each item, agreement between raters was computed on a per-case basis. Inter-rater reliability was primarily quantified using Cohen’s κ, which estimates agreement beyond chance for binary ratings (24). Because κ is sensitive to prevalence and marginal distributions, particularly in sparse or highly imbalanced items, two additional complementary measures were calculated. First, percent agreement was reported to provide an intuitive measure of absolute concordance (25). Second, Gwet’s AC1 was computed as a more robust chance-corrected agreement coefficient that is less affected by prevalence effects and is therefore recommended for reliability analyses involving binary clinical data with skewed distributions (26).

Performance was evaluated at both the global level and the section level. Accuracy was defined as the proportion of correctly classified items among all evaluated items. To account for class imbalance and to distinguish between overreporting and underreporting of clinical information, precision, recall, and the F1 score were additionally computed, with the F1 score defined as the harmonic mean of precision and recall. All performance metrics were calculated separately for manual reports relative to the gold standard and for AI-generated reports relative to the gold standard. In addition, absolute counts of false positive and false negative items were reported to characterize systematic error patterns. This evaluation framework, while novel in the psychiatric domain, follows established approaches for the objective assessment of medical AI outputs, in which performance is quantified using item-based accuracy, precision, recall, and F1 scores derived from false-positive and false-negative counts (27–29).

For performance comparison between manual and AI-generated reports, independent two-tailed t-tests were conducted. These analyses were used to test for differences in mean accuracy, precision, recall, and F1 scores between report types. All analyses and visualizations were performed using Python version 3.12.3 (30), and figures were generated using Matplotlib (31).

WER and Levenshtein scores are reported in Table 1. The mean WER was 9.44% (SD = 2.13%). Interviews contained an average of 2,684.33 words (SD = 800.68). Substitutions (M = 3.34%; SD = 1.60%) and deletions (M = 3.31%; SD = 1.21%) occurred more often than insertions (M = 2.77%; SD = 0.64%). The mean Levenshtein score was 0.966 (SD = 0.006).

However, some relevant errors occurred, which can be seen in Table 2. For example, when a patient was asked to spell the word “radio” backward to test for impairment of concentration, he was able to do so correctly. The AI transcript, however, only detected three of five letters correctly, therefore leading to a transcript that would suggest impairment of concentration, where there is none. Another example of significant mistakes was the misunderstanding of medical diseases. For instance, “hypertonia” was once mistaken for “hypothermia”, which could lead to a faulty medical history. Also, in many cases, names of medication were transcribed incorrectly, for example, “metazapine” instead of “mirtazapine” or “Zuprexa” instead of “Zyprexa”.

Inter-rater reliability was high overall. Across all 209 items and both report types combined, the mean Cohen’s κ was 0.80 (SD = 0.33), the mean percent agreement was 0.96 (SD = 0.07), and the mean Gwet’s AC1 was 0.93 (SD = 0.12), indicating robust agreement beyond chance despite heterogeneous item prevalence (mean prevalence = 0.24, SD = 0.21). For AI reports, the mean Cohen’s κ was κ = 0.75 (SD = 0.38), accompanied by a high mean percent agreement of 0.96 (SD = 0.09) and a mean Gwet’s AC1 of 0.93 (SD = 0.14), indicating robust agreement. For manually written reports, inter-rater reliability was similarly high, with a mean Cohen’s κ of 0.81 (SD = 0.36), a mean percent agreement of 0.96 (SD = 0.09), and a mean Gwet’s AC1 of 0.94 (SD = 0.15), reflecting slightly higher overall consistency between raters compared with AI-generated content.

Figure 2 shows accuracy, F1 scores, false negatives, and false positives for human and AI reports.

Across all categories and items, human-generated reports showed substantially higher agreement with the gold standard than AI-generated reports. Human reports achieved a mean accuracy of 0.94 (SD = 0.01), whereas AI reports reached a mean accuracy of 0.78 (SD = 0.08). This difference was statistically significant, t(5) = 6.33, p = .003. Similarly, mean F1 scores were significantly higher for human reports (M = 0.89, SD = 0.02) compared to AI reports (M = 0.55, SD = 0.13), t(5) = 7.38, p = .001. Error analysis revealed that AI reports produced more false negatives (M = 33.50, SD = 10.31) than human reports (M = 6.33, SD = 2.07), a difference that was statistically significant, t(5) = −8.39, p <.001. False positives did not differ significantly between human (M = 7.00, SD = 1.90) and AI reports (M = 13.17, SD = 7.03), t(5) = −2.02, p = .092.

Figure 3 shows the performance of AI reports throughout the different categories and Figure 4 shows F1 scores. AI performance was the closest to human documentation in categories with limited semantic ambiguity and clearly defined factual content. For family history, AI achieved perfect agreement with the gold standard, with a mean accuracy of 1.00 (SD = 0.00) and an F1 score of 1.00 (SD = 0.00), matching human reports exactly (accuracy = 1.00, SD = 0.00; F1 = 1.00, SD = 0.00). Similarly, in medical history, AI reports demonstrated high accuracy (M = 0.92, SD = 0.13) and F1 scores (M = 0.90, SD = 0.15), approaching the performance of human reports (accuracy: M = 0.98, SD = 0.04; F1: M = 0.98, SD = 0.05). Comparable patterns were observed in medication history, where AI achieved an accuracy of 0.88 (SD = 0.08) and an F1 score of 0.73 (SD = 0.17), while human reports remained superior but closer in magnitude (accuracy: M = 0.95, SD = 0.07; F1: M = 0.90, SD = 0.12). For describing current complaints, AI performance declined more noticeably. AI reports reached a mean accuracy of 0.60 (SD = 0.36) and an F1 score of 0.64 (SD = 0.31), substantially below those of human reports, which achieved perfect performance in this category (both accuracy and F1: M = 1.00, SD = 0.00). This discrepancy reflects AI difficulties in consistently capturing symptom onset, course, and emphasis as expressed in free-form patient narratives. More pronounced performance gaps emerged for psychiatric history, where AI reports reached a mean accuracy of 0.79 (SD = 0.17) and an F1 score of 0.58 (SD = 0.32), compared to substantially higher values in human reports (accuracy: M = 0.97, SD = 0.03; F1: M = 0.96, SD = 0.06). Similarly, for psychopathology, AI performance was markedly lower, with a mean accuracy of 0.74 (SD = 0.07) and an F1 score of 0.50 (SD = 0.11), whereas human reports achieved near-ceiling performance (accuracy: M = 0.96, SD = 0.03; F1: M = 0.94, SD = 0.04). For social history, AI reports demonstrated moderate performance (accuracy: M = 0.70, SD = 0.21; F1: M = 0.62, SD = 0.34), again falling short of human documentation (accuracy: M = 0.84, SD = 0.18; F1: M = 0.74, SD = 0.25). Notably, both AI and human reports showed lower performance in vegetative symptoms, often omitting several items. AI achieved an accuracy of 0.74 (SD = 0.12) and an F1 score of 0.45 (SD = 0.19), which was similar in accuracy but lower in F1 compared to human reports (accuracy: M = 0.73, SD = 0.13; F1: M = 0.56, SD = 0.28). Finally, preliminary diagnoses revealed a distinctive pattern. While AI accuracy was low (M = 0.42, SD = 0.49), its F1 score reached 1.00 (SD = 0.00), mirroring human reports (accuracy: M = 0.83, SD = 0.41; F1 = 1.00, SD = 0.00). This reflects the binary structure of the diagnosis items and the fact that, when diagnoses were named, AI tended to reproduce them consistently, despite frequent omissions reflected in reduced accuracy.

Table 3 shows examples of both human and AI reports, compared to the transcript, to visualize how information from the recorded conversation was converted to the report and to contextualize the quantitative performance differences across categories.

An example of when AI performance was comparatively close to the human report and mirrored the interview content correctly is the current complaints section of interview no. 1. The patient reports a symptom duration of 2 to 3 years with marked worsening in recent weeks and describes spending most of the day in bed or on the couch watching television. Both the human report (“The symptoms have been present for 2–3 years but have significantly worsened in recent weeks…”) and the AI report (“He has been feeling extremely exhausted and isolated for the past 2–3 years…”) correctly captured symptom chronicity, functional impairment, and behavioral withdrawal.

In contrast, regarding their psychiatric history, in interview no. 1, the patient reports a switch from sertraline to mirtazapine due to gastrointestinal side effects, slight weight gain, improved sleep, absence of psychotherapy, and no prior inpatient treatment. While the human report explicitly integrated all these elements into a coherent longitudinal summary, the AI report selectively emphasized medication changes and side effects but omitted or fragmented contextual information, such as the lack of psychotherapy or inpatient stays. These omissions led to false negatives at the item level, reducing recall and F1 despite largely correct information.

In the medical history and medication categories, the excerpts show both the strengths and risks of AI enrichment. Compared with the human report, the AI sometimes appended additional medication context, such as an assumed indication or duration of treatment, even when the human report listed only the medication names. In interview no. 1, for example, the human report listed hypertension and appendectomy, penicillin and hay fever, and medication with mirtazapine and ramipril. The AI report, in contrast, included additional specifications, such as stating that mirtazapine had been prescribed for 1.5 years and assigning an indication (“against sleep problems”), and it also provided an indication for ramipril (“against hypertonia”). While indication and duration fields can be clinically useful, this excerpt also demonstrates the central failure mode: the assigned indication for mirtazapine was incorrect because the transcript indicates that mirtazapine was prescribed after sertraline for depressive symptoms, with sleep improvement reported as an effect rather than the primary reason for prescribing. This kind of plausible-sounding but incorrect attribute assignment would be expected to reduce precision and therefore depress F1, particularly in categories where correct attribution of diagnoses, indications, and historical treatments is central.

By contrast, social history errors more frequently involved abstraction and loss of specificity. In interview no. 2, the patient provides detailed information about studies, employment interruption, living situation, social isolation, geographic distance from friends, and family contact. The human report integrated these details into a structured narrative, whereas the AI report condensed the information into broader statements (“Often feels lonely” and “family contact per telephone”) and omitted temporal qualifiers such as the duration of unemployment or partnership status. These compressions are clinically plausible but failed to meet item-level criteria, leading to lower F1 scores despite superficially adequate summaries.

Sometimes, errors occurred due to faulty transcription. For example, once, the AI wrote that the patient had not been in a relationship for 5 to 6 years, although the patient stated that her relationship ended 5 to 6 months ago, missing a direct temporal link to the beginning of the current depressive episode, only because of the false transcription of years instead of months. In other cases, errors occurred, although the transcription was correct.

In some cases, the content of certain categories was misplaced altogether. Table 4 shows a few examples where a category consisted of almost no adequate information. We could not identify what caused this error.

At last, we show examples of AI reports for psychopathology, the most challenging category. As can be seen in Table 5, in some cases, AI described certain aspects of psychopathology correctly, especially mood and drive, but sometimes also suicidal thoughts, compulsions, phobias, and panic attacks. Impairments in concentration or memory were correctly identified some of the time, but sometimes incorrectly as well. Both formal and content thought disorders were incorrect most of the time. Interestingly, one time, the human report was wrong, and the AI report was correct: in interview no. 2, the human incorrectly stated that the patient was able to recall three out of three words after 10 minutes, documenting no impairment of memory retention, although the ground-truth transcript proved that the patient did not memorize all three words, which was correctly stated by the AI and apparently incorrectly memorized or documented by the physician.