Over the last 2 decades and since “To Err is Human” was published, several concepts (e.g., clinical pharmacy services, electronic prescribing, and clinical decision support systems) have been developed to optimize medication and patient safety. (Institute of Medicine US Committee on Quality of Health Care in America, 2000).

Clinical pharmacy services include various areas of activity, such as medication reconciliations and reviews, ward rounds, antibiotic stewardship programs, and patient training (Fernandes et al., 2015; Dreischulte et al., 2022). Multiple studies showed that medication reconciliations based on the best possible medication history in combination with medication reviews reduced medication errors (ME) (Mekonnen et al., 2016; Manias et al., 2020; Nguyen et al., 2020; Mengato et al., 2023). Furthermore, performing medication reviews and participating in ward rounds can contribute to the reduction of ME and, therefore, improve medication safety (Kaboli et al., 2006; Mansur, 2016; Blassmann et al., 2018; Kiesel et al., 2022). Clinical pharmacists play an important role as members of the interprofessional team for patient care in several countries worldwide. In Germany, clinical pharmacy services are expanding but have not been established nationwide (Schulz et al., 2021).

Electronic medical records (EMR) can include computerized physician order entry (CPOE) and clinical decision support systems (CDSSs). Both CPOE and CDSSs have the potential to reduce ME and subsequently optimize medication safety (Bates et al., 1999; Garg et al., 2005; Vélez-Díaz-Pallarés et al., 2018). However, the implementation of CPOE and CDSSs is associated with risks such as overalerting and nonacceptance among the end users (Ammenwerth et al., 2008; Wolfstadt et al., 2008; van der Sijs et al., 2009; Nuckols et al., 2014; Ranji et al., 2014; Jia et al., 2016; Nanji et al., 2018; Abell et al., 2023). Thus, the implementation needs to be well planned and the performance monitored in clinical practice (Ammenwerth et al., 2014; The Office of the National Coordinator for Health Information Technology, 2016; Wasylewicz and Scheepers-Hoeks, 2019). As data on the pre-implementation evaluation of CDSSs were scarce, we previously developed an algorithm for the interprofessional validation of a medication CDSS and performed the evaluation for Meona, a commercial CPOE and CDSS (Bauer et al., 2024). We focused on evaluating the general functionalities of the CDSS as well as on performing a technical and content-related validation of single elements of the CDSS (e.g., drug–drug interactions). As a result of this prior evaluation, we decided interprofessionally which elements of CDSS should be used and how they should be configured. Thus, we implemented a thoroughly customized medication CDSS into clinical practice at Erlangen University Hospital (Bauer et al., 2024).

Previous studies on medication safety investigated the effect of clinical ward pharmacists in combination with CPOE/CDSS. These studies showed that clinical ward pharmacists can identify drug-related problems in addition to the implemented CPOE/CDSS (Bedouch et al., 2009; Bedouch et al., 2012; Zaal et al., 2013; Cornu et al., 2014; Berger et al., 2022; Seiberth et al., 2022). However, only a few studies performed a pharmaceutical validation of the displayed CDSS alerts in real-world clinical settings (Zaal et al., 2013; Cornu et al., 2014).

The present investigation is the first to describe in detail the synergy between a medication CDSS and clinical ward pharmacist activities. For this reason, and because Meona is one of the most widely used CPOE/CDSS in Germany but lacks evaluation data, we conducted this study within a large German university hospital. We aimed to examine the performance of our customized CDSS after implementation in clinical practice by analyzing the frequency, type, and severity of the displayed CDSS alerts and to evaluate the content appropriateness and patient relevance of the displayed CDSS alerts. A further objective of this investigation was to characterize both the interventions performed by the clinical ward pharmacists supported by the CDSS and CDSS-independent interventions according to their reason, level of severity, and acceptance rate.

To the best of our knowledge, we performed the first investigation to determine the performance of the Meona-CDSS in clinical practice. We did not set many inclusion criteria to obtain clinical data of a broad and real-world representing patient collective. Furthermore, no prospective studies in Germany assessed the performance of a commercial, integrated CDSS in clinical practice by using a pharmaceutical validation of the CDSS alerts (see Figure 1). We evaluated our customized medication CDSS (see Supplementary Figure S1) within four different clinical departments at Erlangen University Hospital by analyzing the number, type, and severity level of shown CDSS alerts. Subsequently, the content appropriateness and patient relevance of the displayed CDSS alerts were determined by clinical pharmacists using a four-eyes principle. The clinical ward pharmacists performed interventions for all patient-relevant CDSS alerts, and the CDSS-independent identified drug-related problems during regular ward rounds and documented these in ADKA-DokuPIK. Thus, reasons, severity using NCC MERP, and acceptance rate of pharmaceutical interventions were evaluated (see Figure 1).

In general, we registered 1,799 CDSS alerts in total (in median 9.0/patient) and 609 severe CDSS alerts (in median 3.0/patient) (see Table 2). The best comparable study by Zaal et al. found 2.7 CDSS alerts/patient using a CDSS system based on the Dutch national drug database with interruptive alerts (overdose, allergies, duplicate therapy) and on-demand alerts (drug–drug interactions) (Zaal et al., 2013). Mc Coy et al. observed 1.6 CDSS alerts on acute kidney injury per patient displayed by a locally developed CDSS (McCoy et al., 2012). Russmann et al. reported 15.5 CDSS alerts per patient using two different CDSSs [PharmaVista (drug–drug interactions, therapeutic duplications, maximum dose, potential inadequate medication in the elderly) and MediQ (focus on drug–drug interactions)] (Russmann et al., 2023). However, a direct comparison is difficult as the study time, the methods used, and the CDSS systems are different.

The highest numbers of CDSS alerts were recorded for patients from Medicine 4 and Vascular Surgery (see Figure 2). This can be explained by the fact that most prescription lines were recorded for these clinical departments (see Supplementary Table S2), predominantly treating patients with a variety of comorbidities (e.g., diabetes, renal impairment) who required polymedication. Previous studies showed a positive correlation between the number of drugs and drug-related problems (Saedder et al., 2016; Høj et al., 2021; Jung-Poppe et al., 2022; Berger et al., 2023).

The most frequently observed types of CDSS alerts were “drug-drug interaction” and “information about renal impairment” (see Figure 3). Other studies reported similar results for the most reported type of CDSS alerts (van der Sijs et al., 2009; Zaal et al., 2013). In our investigations, it is possible that only a few “duplicate prescription” and “allergy” CDSS alerts were still displayed using the check button because these two Medication-Safety-Validators are used with the PUSH-(&PULL)-modus (see Supplementary Figure S1B). Thereby, these CDSS alerts are displayed as pop-up alerts during the medication prescription process. Physicians probably registered these CDSS alerts when prescribing a medication and adjusted their medication prescriptions accordingly.

We performed the pharmaceutical validation for all severe (609) and “duplicate prescription” (38) CDSS alerts with a two step-approach (see Figure 1). First, we determined a content appropriateness of 82.7% (535/647) among the CDSS alerts meeting our inclusion criteria (see Figure 4A). Second, we found overalerting, as only 19.6% (105/535) of the content-appropriate CDSS alerts were classified as patient relevant (see Figure 4A).

Interestingly, many of the content-inappropriate CDSS alerts in this study are based on a few recurring scenarios, such as severe “drug–drug interaction” CDSS alerts for drugs with only possible risk for QT-prolongation according to AZCERT criteria or “information about renal impairment” CDSS alerts for acetylsalicylic acid low dose. These scenarios were discussed with the manufacturer to adapt the rule-based CDSS accordingly (e.g., regarding their severity). In general, better performance of the medication CDSS can be achieved by initiating the improvements through clinical pharmacists (Cuvelier et al., 2021; Skalafouris et al., 2023).

The fact that a content-appropriate CDSS alert was often rated not patient relevant (430/535, 80.4%) was usually due to the insufficient specificity of the CDSS alerts (Fritz et al., 2012; de Wit et al., 2015; Cuvelier et al., 2021; Russmann et al., 2023; Bauer et al., 2024). In the case of “information about renal impairment” CDSS alerts, for example, the prescribed dose is not checked. A CDSS alert is triggered if the renal function is below a certain threshold regardless of the actual dose prescribed (e.g., eGFR 35 mL/min and Sitagliptin 50 mg 1-0-0-0 caused an CDSS alert, although the prescribed dose was already correct). As another example, drug–drug interactions are also displayed, even though a required action, such as dose reduction (e.g., amlodipine and max. 20 mg simvastatin) or specific administration interval (e.g., calcium and levothyroxine) had already been considered to preclude the interaction. Only a few PIM (potential inadequate medication in the elderly) CDSS alerts were rated patient relevant, due to the lack of better alternatives for a given indication in clinical practice. Our results showed that the medication CDSS should be improved to enable more efficient clinical decision support and to reduce overalerting. Improvements of the CDSS should include considering inter alia the actual doses prescribed, the time of administration (e.g., morning or evening), the diagnoses, as well as more laboratory data and clinical parameters such as blood pressure or pain scores.

Other studies reported that overridden rates are higher among CDSS alerts regarding drug–drug interactions, geriatric recommendations, and renal dose adjustments as opposed to alerts affecting duplicate prescription, allergy, and overdose (van der Sijs et al., 2009; Poly et al., 2020). These findings are in line with the results of our pharmaceutical validation as “duplicate prescriptions” alerts were most frequently rated as patient relevant, followed by “information about renal impairment” alerts (see Figure 4B).

Overalerting is associated with nonacceptance as well as reduced satisfaction and reliability of the CDSS in clinical practice and is frequently reported by other investigations (Zaal et al., 2013; Nanji et al., 2018; Abell et al., 2023). Most of the alerts displayed by our customized CDSS are rated as content appropriate, but the CDSS still reported many not-patient-relevant CDSS alerts with respect to the patient situation in clinical practice, leading to overalerting while using the check button (see Figure 4A). This may disturb the physicians’ efficient use of the check button. Nevertheless, with our special configuration (“only-PULL-modus”) for several Medication-Safety-Validators (see Supplementary Figure 1B), CDSS alerts were at least not displayed as popup alerts during medication prescription, which might be an effective strategy to reduce overalerting (Knight et al., 2019; Bauer et al., 2024).

In general, experience with the CDSS or other databases is required to assess the CDSS alerts in an appropriate way. This might be challenging for healthcare professionals in clinical practice. Thereby, training on handling the CDSS for all healthcare professionals, especially physicians and clinical pharmacists, should be provided (The Office of the National Coordinator for Health Information Technology, 2016; Van de Velde et al., 2018; Abell et al., 2023). However, an integrated CDSS can support healthcare professionals, especially clinical pharmacists, to improve medication safety, as more review time can be provided due to integrated clinical decision support (Urbina et al., 2021).

Our investigation showed that the clinical ward pharmacists recorded a significantly higher number of CDSS-independent interventions (1.1/patient) during their regular rounds than pharmaceutical interventions triggered by the CDSS (0.6/patient) (see Figure 5). One reason for the high number of CDSS-independent interventions by the clinical ward pharmacists could be the high level of clinical expertise, as each clinical ward pharmacist had been assigned to a specific clinical department as part of the interprofessional team for a long time. Several published studies emphasized that clinical pharmacists can additionally identify pharmaceutical interventions beyond the CDSS by performing medication reviews in different settings (e.g., a renal pharmacist consultant service or closed-loop medication management) (Bedouch et al., 2009; Bedouch et al., 2012; Zaal et al., 2013; Berger et al., 2022; Seiberth et al., 2022). One reason might be that clinical pharmacists can consider patient individual aspects (e.g., laboratory parameters, diagnoses) in their medication review, which is limited for most ruled-based medication CDSS (e.g., identifying missing drug treatment despite existing indications). Therefore, in our analysis, the causes for pharmaceutical interventions (CDSS-triggered and CDSS-independent interventions) differed between the two intervention types (see Supplementary Figure S4; Supplementary Table S6). Clinical ward pharmacists intervened much more frequently on the indication or appropriateness of a drug. These are also the most frequently cited reasons for pharmaceutical interventions in other studies (Bedouch et al., 2012; Zaal et al., 2013; Berger et al., 2022; Langebrake et al., 2022).

The severity level (see Figure 6A) of our pharmaceutical interventions according to NCC MERP is, in general, comparable to other studies (Langebrake et al., 2015; Berger et al., 2022; Langebrake et al., 2022). In detail, some studies reported severity levels A and B (no error or error did not reach the patient) more frequently, which may be due to the daily activity of the clinical pharmacists in these studies compared to our once-weekly rounds (Berger et al., 2022). In our investigation, the NCC MERP severity levels A and B occurred significantly more often for the CDSS-independent interventions than for the CDSS-triggered interventions, thus indicating that clinical ward pharmacists perform many preventive interventions (e.g., constipation prophylaxis for opioid-treated patients). On the other hand, the CDSS alerts are only displayed by using the check button when the error has usually already reached the patient (i.e., after medication prescription).

Furthermore, the acceptance rate of our performed interventions was very high (92.2%, 225/244) in total and did not differ significantly between the two intervention types (see Figure 6B). Other studies reported acceptance rates of pharmaceutical interventions between 55.0% and 91.9% (Bedouch et al., 2012; Zaal et al., 2013; Zaal et al., 2020; Berger et al., 2022; Langebrake et al., 2022). A reason for this might be that all interventions were directly communicated by clinical ward pharmacists, which seemed to be more effective than the digital communication within the EMR used in prior studies (Bedouch et al., 2012; Zaal et al., 2020).

In contrast to Bittmann et al., who analyzed the performance of their CDSS (AiDKlinik (Dosing GmbH, 2023)) by analyzing the overridden rates (Bittmann et al., 2021), we used a prospective pharmaceutical validation with a four-eyes principle. In the published literature, we could only identify one similar multi-level approach for the pharmaceutical validation of CDSS alerts (Cuvelier et al., 2021). Other studies evaluated the appropriateness of overridden popup-CDSS alerts retrospectively by clinical pharmacists or interprofessional teams (McCoy et al., 2012; Nanji et al., 2018; Van De Sijpe et al., 2022). In our investigation, the interaction between an integrated medication CDSS and a clinical ward pharmacist service was described and examined in detail for the first time in order to record and evaluate the synergies between both approaches to improve medication safety. Our investigation confirmed that customizing the CDSS by using the “only-PULL-modus” is a useful configuration option to prevent interruptive overalerting during medication prescription, as most of the displayed CDSS alerts were rated not patient relevant (Bauer et al., 2024).

As a constraint of our investigation, the findings were only collected in four clinical departments at one site and with one site-specific, customized medication CDSS. Therefore, the results cannot be transferred directly to other sites and settings. Other studies showed that clinical pharmacists could identify additional and more drug-related problems than a medication CDSS, but the number and type of CDSS alerts might be different within other CDSS and sites (Bedouch et al., 2009; Bedouch et al., 2012; Zaal et al., 2013; Seiberth et al., 2022). Data interpretation might further be limited by systemic biases resulting from local resources and might limit scalability (e.g., time burden of pharmaceutical medication reviews and clinical expertise of clinical pharmacists). Our approach to determine the performance of our customized medication CDSS has some further limitations: Due to the point-prevalence design, we assessed the CDSS alerts and pharmaceutical interventions only at one single time point and not for the whole stay of the patients. The pharmaceutical validation of the CDSS alerts was examined with the four-eyes principle, but we did not perform a Delphi approach. Furthermore, the inter-rater reliability during the independent pharmaceutical validation was not determined in a structured way. We only validated severe CDSS alerts and all duplicate prescription CDSS alerts, potentially excluding important CDSS alerts with lower severity. The validation of the CDSS alerts was solely carried out by clinical pharmacists without involving an interprofessional team during the validation. Due to the high acceptance rate of the CDSS-triggered interventions by clinical pharmacists, we can rule out overalerting of the physicians with non-patient-relevant interventions. Other studies directly involving physicians in validating the CDSS alerts also found that most of the CDSS alerts were correct in terms of form, technology, and content but were frequently not classified as patient relevant (Pfistermeister et al., 2016; Nanji et al., 2018). We did not investigate the physicians’ responses to the displayed medication CDSS alerts in this analysis, as, for example, McCoy et al. (2012) reported. Thus, it was not possible to determine whether the check button was invoked or which CDSS alerts were read or accepted.

This investigation showed that a locally customized medication CDSS indicated a high number of CDSS alerts per patient, a third of which were classified as severe. Although content-inappropriate CDSS alerts were still identified, the majority of the clinical pharmacist-validated CDSS alerts were rated as content appropriate. Only a minority of the content-appropriate CDSS alerts were classified as patient relevant and consequently required an intervention by clinical ward pharmacists. These results can be considered as overalerting in clinical practice and showed the need for improvements in the medication CDSS (e.g., including diagnoses and laboratory data) to enable more efficient clinical decision support. In addition, the clinical ward pharmacists carried out significantly more CDSS-independent than CDSS-triggered interventions. To conclude, a customized medication CDSS can support healthcare professionals in optimizing medication safety. However, our results underline the continuing need for clinical pharmacists to interpret the CDSS alerts in the individual patient context and to perform a comprehensive medication review to identify all drug-related problems.