Outpatient reports in German language, generated by eight different neurologists, were extracted from the clinical information system filtering for reports from our MS clinic of visits between 01-Jan-2023 to 31-Dec-2023 of an academic hospital with neuroimmunological focus. Reports were only included if the visit and report date matched. Reports of 30 patients were randomly selected, manually anonymized and used for primary LLM prompt analysis and reiteratively used for refinement of the prompt.

Data of interest were defined in a tabular form for human report evaluation. A first-generation prompt in plain language was generated to query the LLMs.

Medical data were analyzed in one run by trained medical personnel (TMP) and the LLMs (Figure 1a, orange path). The results of the LLMs were compared with those of the TMP by neurology specialists (NSPs) (Figure 1a, blue path). The latter extracted their structured results based on the reports, the prior human answers and LLM answers from the first prompt. These data were analyzed and a structured human reference was developed in agreement with the working group consisting of two specialists and two assistant physicians from the field of neurology (Figure 1a, green path). During the creation of the reference data, a unanimous decision was made for each variable. This reference was then used to iteratively improve the prompt (Figure 1b).

The plain language prompt was converted into a JavaScript Object Notation (JSON)-format for the structured output functions of the different LLMs, which allows responses to also be returned in JSON format.

We used a mix of different variable types with mixed complexity. To obtain an assessment of complexity, the working group in consensus gave a subjective assessment of the complexity of a variable, ranging from low-complex, medium-complex to high-complex. The variables analyzed were all related to the disease MS but were not specifically selected for the application of an LLM-supported evaluation. The selection of variables represents a classic evaluation of a retrospective data set in MS. The following 19 variables were examined: diagnosis (low), disease course (low), date of first manifestation (medium), date of first diagnosis (medium), current EDSS (low), oligoclonal bands (OCBs) status (low), Aquaporin-4(AQP4)-antibody (AB) status (low), myelin-oligodendocyte-glycoprotein(MOG)-AB status (low), last cranial MRI (cMRI) date (medium), cMRI activity (medium), own cMRI report interpretation (high), current immunotherapy as active substance name (low), start date of current immunotherapy (medium), previous immunotherapies as active substance name (medium), other diagnoses (medium), comedication (low), walking distance (high) and walking aid (medium).

In the evaluation of the creation of structured outputs, we also considered the following additional variables: functional scores (FS) of the EDSS, results of the Multiple Sclerosis Functional Composite (MSFC). In total, all of these latter 13 variables were only mentioned in a very small proportion of the medical records. Therefore, we excluded them from the content analysis. We also excluded the question “current neurological symptoms” from the content analysis as a reliable and distinct evaluation was not possible due to the widely differing wording of the structured results.

We decided to use commercial LLMs because they allow access to models with a higher number of parameters without requiring extensive infrastructure. From our perspective, the use of commercial LLMs reflects the simplest approach to generating structured output with limited financial resources and without significant technical effort. In this study, we used the following commercially available versions of LLMs to analyze the data: claude-3-haiku-20240307 (Haiku; Anthropic), claude-3-opus-20240229 (Opus; Anthropic), claude-3-7-sonnet-20250219 (Sonnet-3.7; Anthropic), gemini-1.5-flash-002 (Gemini-1.5-flash; Google), gemini-1.5-pro-002 (Gemini-1.5-pro; Google), gemini-2.0-pro-exp-02-05 (Gemini-2-pro; Google), gpt-4o-2024-08-06 (4o; OpenAI), gpt-4o-mini-2024-07-18 (4o-mini; OpenAI) and o3-mini-2025-01-31 (o3-mini; OpenAI). The LLMs were accessed via the manufacturers’ application programming interfaces (APIs) when not stated otherwise. All requests were made with a temperature value of 0.7 and a top p value of 1 if applicable. Wherever possible, the data were analyzed as a batch analysis. The order of the models within this paper is based on alphabetical order and does not represent a qualitative ranking. Models are always listed in the same order as they are listed in the methods. We used the same prompt for the queries to the different LLMs. All requests for the final analysis were made between 22nd of March 2025 to 28th of March 2025.

For the consensus LLM output, we compared the output of the Sonnet-3.7, Gemini-2-pro and o3-mini models (Figure 1c). At the time of writing, these models were considered “flagship” models, offering the highest range of functionality and the best performance in the company’s own benchmarks. The idea was to use LLMs from different providers, as these have different software architectures and different training data sets. To be included, the output of two or more models had to match. If no consensus could be reached, the output was excluded from the analysis and not counted as an error.

Errors of structure: Within the generation of a structured output, any deviation from the desired data format was considered an error. The accuracy of the content of the responses was not a factor in the evaluation of the correctness of the format.

Errors of content: Within the content evaluation, all deviations from the human consensus reference were evaluated as errors. All variables of the consensus output and the NSPs were individually compared with regard to the quality/severity of the error.

Statistical differences between the various evaluations were tested using Fisher’s exact test.

For clarity, in addition to the number of errors, we have also specified an accuracy, which is calculated as 1-(number of errors/number of variables analyzed). Based on the variables collected, it was not possible to categorize the responses of the LLMs into true positives, false positives, true negatives, and false negatives. Therefore, a conscious decision was made not to calculate precision, recall, and F1 scores, and instead to use accuracy as a measure.

The following definition was used to assess whether an error was a true-error:

“Enumerations”: In the case of defined values from a set of possibilities, all deviations from the required answer were considered errors.

Retrospective chart analysis within our monocentric neuroimmunological registry has been approved by the ethics committee Westfalen-Lippe, Germany (registration number 2024-590-f-S). Strict data anonymization has been performed prior to any usage of the data, in particular prior to data entry into either of the LLMs.

A first hurdle in the use of LLMs for the evaluation of routine clinical data is the reliable generation of a structured output. There are two different ways to generate a structured output: The first way is via a plain text prompt and the second way is via a built-in function of the LLM. This can be, for example, a function calling function or a structured output function, which is supported by most of the LLM providers.

As a proof-of-concept, we first tested the generation of structured outputs using the web interface of 4o-mini. We used a text-only prompt to generate an excel spreadsheet. Our first observation was that due to the context window, we were limited in the number of medical records we could provide to the LLM. If we used a prompt at the beginning and pasted the medical records afterwards, the LLM would lose track of its task and the output would not comply with the task. We could observe that handing over the prompt with each medical record made the output much more reliable. In 2 out of 30 cases, the LLM was unable to produce an output which could be easily fixed by re-handing the task to the LLM.

As a first step, we wanted to generate a constant output of the correct columns. The columns of the spreadsheet mainly contained 3 different error types deviating from the desired format: A deviation from the column naming, a deviation from the number of columns (omitting or adding columns different from the prompt) and a deviation from the requested order. To reduce these errors, the first step was to adapt the prompt with an explicit reference to compliance with the structure. Contrary to our expectations, this increased the number of errors in 2 out of 3 error types. Overall, we were unable to generate a pure text prompt that would allow a reliable, uniformly structured output of the data (Table 1).

As a second step, we converted the plain text prompt to a JSON-format and used the build-in functions of different providers’ API to achieve a structured output. Many providers of LLMs offer a corresponding function to obtain a spreadsheet-like JSON output. With the help of structured output functions, we were able to drastically reduce the rate of faulty column outputs so that no more errors occurred in the structure (Table 1). From there on, we tested our prompt with the 6 most commonly used LLMs at that timepoint: 4o, 4o-mini, Gemini-1.5-pro, Gemini-1.5-flash, Opus and Haiku. All further evaluations were carried out using the structured output functions.

When generating a structured output through the methods described above we observed an important problem regarding possible further downstream analysis of the data: Data types often did not match the desired format (e.g., date formats, number formats etc.). In the next step, we therefore concentrated on generating consistent data formats.

In the first step, we started by detecting faulty data types and correcting them as best we could. The first option available for this within the structured output functions is to specify an expected data type. By setting the type of the data, we were able to reduce incorrect datatypes drastically. Although many LLMs allow to use a common schema object, not all providers support the same functions. As the LLMs “Haiku” and “Opus” do not have a specific structured output function (but can be forced to create JSON files by forcing the use of tools) and therefore were not modified by specifying a return data type, we excluded them from further analysis.

Another way to improve the creation of a structured data set is to use enumerations. With the help of these, the LLM can be given certain answer options from which to choose. In our example, we searched for diseases in the context of multiple sclerosis. By limiting the possible answers to a known set value, we were able to further reduce the number of data type errors.

As a last step, we adjusted the description of the task within the prompt and the system prompt and further emphasized the importance of sticking to the structured output.

By using these methods, we were able to reduce the rate of incorrect data formats by up to 88 percent (Table 2). Nevertheless, even after several iterations of improvement, we were not able to generate completely error-free outputs for the initially three tested models with regard to the data format.

As a final step, we tested our JSON-format prompt with the newer LLMs 3o-mini, Gemini-2.0-pro and Sonnet-3.7. Using these models, we could not detect any errors with regards to the structure.

With the help of the first two steps, we were able to create a prompt that was able to deliver mostly consistent results in terms of output. As we could already see in the first steps that both GPT-4o-mini and Gemini-1.5-flash performed the same or worse in terms of output structure than the larger LLMs, we did not carry out the further analysis with these two. However, as they performed better in terms of structured output, we also performed our evaluations with the newer LLMs o3-mini, Gemini-2.0-experimental and Sonnet-3.7.

To evaluate content quality of the outputs, we compared the different models with a human reference created for the data set. Overall, we found that errors in the TMP control and in the NSP control were lower than in the LLM evaluations. Yet, the error types within the different variables were very similar: LLMs tended to make similar errors to the human control. In particular, the variables “interpretation of cMRI activity” (incorrect interpretation), “current immunotherapy” and “previous immunotherapy” (indication of drugs instead of drug names) and “walking aid” (distinction between missing and no walking aid) caused diverging results. Interestingly, the LLM-based output highlighted certain inaccuracies in the human reference evaluation by TMPs which was revealed by the evaluation of the results by the NSPs, particularly in complex variables such as MRI interpretation and treatment timelines (Figure 2; Table 3).

We have further improved the prompt based on this feedback from the NSPs. The main changes include: The explicit specification of criteria for cMRI interpretation, the explicit naming of active substances and their trade names with the enforced request to name only active substances and the explanation to interpret the walking distance and walking aid also in the context of the remaining examination findings and the EDSS. Thereby, we were able to reduce the number of errors by more than 25% from 15 to 11.1% in average. As this was the maximum we could achieve under several iterations of improvement, we compared an “LLM consensus” response with our reference as described in the methods part. Overall, no consensus could be found for 3 values out of a total of 540. Using this method, we were able to decrease the error rate by around another 30% compared to the best performing LLM. This method enabled us to overcome our ceiling effect and decrease the total percentage of errors to 6.7%. Nearly all variables profited from the consensus approach in terms of a reduction of the error rate (Figure 3; Table 3).

In a final step, we checked the content of all deviating answers from the LLM consensus and the NSPs individually. We were able to determine that of the 36 deviations categorized as errors, only 8 answers were actually real content-related errors. The remaining 28 answers rated as incorrect related to previously mentioned edge cases, differed only to a very small extent and could often be justified by a diverging interpretation of the unstructured data (e.g., a difference in the initial diagnosis of a few days or a few months), resulting in a true-error-rate of 1.48%. In comparison to that, out of the 22 errors by the NSPs, 11 answers were content-related errors (which were mostly caused by non-adherence to the requirements within the prompt), resulting in a true-error-rate of around 2%. The difference between the LLM consensus evaluation and the NSPs was not significant (p = 0.6447; odds ratio = 1.38; 95% CI 0.57–3.31).