This formative evaluation study was conducted at four health institutions across three European countries. These were the North Estonia Medical Centre (NEMC) in Estonia, Maastro and the Maastricht University Medical Centre (MUMC+) in Netherlands, and the Medical University of Graz (MUG) in Austria. The patients tested G1 for at least 2–4 weeks, with support from the research team. The study flow for the patients is visualized in the Supplementary Materials. In addition, a list of the AI and non-AI tools deployed in G1 can be found in the Supplementary Materials.

The inclusion goal was set at 90 adult patients for adequate evaluation, equally dividing 30 patients between NEMC, Maastro/MUMC+, and MUG, each of which included 15 patients with BC and 15 patients with CVD, specifically with type 1 myocardial infarction. The recruitment period for G1 lasted from May 2024 to December 2024. The inclusion and exclusion criteria for G1 prototype testing are listed in Table 1. The development of G1 took place with the support of patient consultants selected by the European Patient Cancer Coalition (ECPC) (20) and European Heart Network (EHN) (21). They did not contribute their PHD but acted as patient representatives in the co-development of G1.

The start of the study was superseded by a “dry-run workshop”, conducted with the patient consultants in May 2024, as preparation for the evaluation of G1. A comprehensive training plan was developed, which included role specifications for study nurses, research associates, expert data curators, patients, and data users (clinicians) (Table 2). The patient consultants provided many valuable insights and feedback to implement before G1 testing that may have reduced the chance of errors during actual deployment.

The dry run had three objectives, namely to (1) gather feedback on the AIDAVA prototype (pre-G1) at that stage and the health data intermediary (HDI) integration, (2) align the evaluation process across sites, and (3) consolidate feedback on the Research Electronic Data Capture (REDCap) tool. The patient consultants were asked to comment on the user journey, and the final draft is available in the Supplementary Materials. After the dry run, the formative evaluation started, and eligible patients were invited to an information session to explain the purpose of the study and go through the informed consent. Patients who signed informed consent forms were guided by the study nurse or research associate in a 1-h training session. In this session, the study nurse or research associate (1) reminded the patient of the objective of AIDAVA, (2) created an account with the patient for G1 and the HDI, and (3) explained the data flow and steps, as presented in Supplementary Figure S1.

The patients worked through four steps (data ingestion, data curation, data publishing, and data use) after the deployment of G1 at the four health institutions acting as the test sites in this study (Supplementary Figure S1). The first three steps required active patient involvement, which include HITL mechanisms to improve the quality of the final IPS that the patient receives. In step 1, data ingestion, the patients consented to have their PHD identified and extracted from the hospital EHR, as well as from the HDI. In this context, HDI refers to an entity or platform that facilitates the collection, integration, and controlled sharing of PHD across different healthcare systems and data users. The patient actively connects their HDI account to the AIDAVA account. Then, these PHD were transferred to the AIDAVA data store, which was exclusively available within the hospital testing environment of the local site or a secure national cloud service. An overview of the complete data flow is illustrated in Figure 1.

In step 2, data curation, the patients activated their AIDAVA account to transform, integrate, and complete their ingested PHD into a standardized representation, the PHKG. As G1 is in the prototyping phase, the patients were warned that any output from the virtual assistant may be incorrect. G1 generated questions for the patients when complete automation of data curation failed, which was generally expected to occur due to missing data. This was the main role and implementation of the HITL mechanism, i.e., to catch any errors prior to data publication. If the patients were unsure of the answer or the question generated by G1 did not make sense to them, they were able to forward the question to the expert data curator appointed by the local health institution.

In step 3, data publishing, the data to be extracted from the PHKG were specified. For this formative evaluation study, three data publishing outputs were defined: (1) extraction and visualization of the patient's IPS, one of the six critical data categories proposed in the EHDS and compliant with the international Health Level Seven Fast Healthcare Interoperability Resources (HL7 FHIR) guideline (18); (2) extraction of data elements to create local BC registries at each site (BC use case); and (3) extraction of key variables supporting the automatic calculation of the patient's SMART risk score (17) (CVD use case). These data elements and key variables can be found in the Supplementary Materials. The curated PHD, which are the source of these three different outputs, only need to be curated once as the respective data extraction processes are executed multiple times in the PHKG.

In step 4, data use, these three outputs were exploited by different data users (Figure 1). The patients were able to visualize their IPS through a specific visualization tool (MIDATA IPS Viewer). The BC specialist could access the metrics calculated from their local BC registry and from interoperable cross-border registries. The prototype executes federated queries for the virtual cross-border BC registry to avoid any data transfer across hospitals. The CVD specialist could visualize the automatically calculated SMART risk score and the details of its calculation for each patient. Due to the possibility of inaccurate data curation at this prototype stage, the SMART risk scores were solely used to test the prototype's accuracy, not for communicating actual risk to the patients.

The patients began testing G1 under the guidance of their local study nurse and/or research associate. Each patient tested and evaluated G1 for a duration of 2–4 weeks, starting with an on-site visit in which the study nurse supported the patient with the activation of their AIDAVA account. In addition, the patients created a personal HDI account, filled out the Medical and Digital Literacy questionnaires, and created an account for their blood pressure device (only for patients with CVD).

As shown in Figure 1, there were two paths for uploading data into the patient's AIDAVA account. First, after the patient had signed the informed consent form, the study nurse/research associate extracted the PHD of the patient from the hospital EHR and imported these data into the patient's personal AIDAVA account. Second, the patients uploaded PHD to their HDI account (including data from their blood pressure device), which were also sent to the patient's personal account within the hospital’s secure environment setup for AIDAVA, respecting the security requirements of each institution. In the remaining weeks, the patients tested the system with general support from the study nurse/research associate, and technical support from the expert data curator, who used a shared ticketing system to quickly solve any issues.

Once the data from different sources, typically in a heterogeneous format, were ingested, the patient could then request the AIDAVA system to curate them into a PHKG. This structured representation of PHD is compliant with the AIDAVA ontology based on the Swiss Personal Health Network (SPHN), including the SNOMED, LOINC, and FHIR profiles and is easily mappable to multiple standards, supporting semantic interoperability. AIDAVA uses an AI-powered semantic transformation infrastructure to orchestrate curation workflows; it leverages predefined data source descriptions (i.e., dictionaries) to process and transform data based on its meaning, rather than just its format. Each attribute/column in (semi-)structured data sources is first semi-automatically mapped either to the AIDAVA ontology with classical extract, transform, and load (ETL) transformation tools, or to dedicated curation tools such as entity linking (mapping to clinical terminologies) and entity alignment (linking terms from two different terminologies), supporting transformation of the data source into an SHKG compliant with the AIDAVA ontology. Unstructured data were directly processed by the natural language processing (NLP) tools and transformed into another SHKG. When data from each SHKG are integrated into a single patient's PHKG, the entity deduplication tool removes duplicate records referring to the same real-world entity and validation is performed using individual data quality checks. Quality scores are computed across multiple dimensions (e.g., completeness and consistency) and categories (e.g., valid code and temporal order consistency) and each detected quality issue is formulated as a question to the user to support data improvement. The entire curation and quality enhancement process is recorded within an audit trail to meet regulatory compliance.

The main task for the patients was to test data ingestion and data curation of their PHD, while checking the accuracy of the published PHD in their own IPS (Figure 1). More precisely, this meant that patients ingested files and documents that were uploaded to their personal AIDAVA account, so that these files and documents would then be available to them for automatic curation. To preserve data ownership, the patients could always choose whether to start automatic data curation for all or only some of the ingested files. G1 addressed questions to the patient whenever issues during the automatic curation arose (i.e., missing data, incompatible formats, unrecognizable documents, etc.). The patients answered these questions in a format related to the question (i.e., when asked for a date, they were presented with a “mm/dd/yyyy” format input field). The patients always had the possibility to either answer the question if they knew the answer or skip the question by clicking the “I don't know the answer” option. When patients selected the “I don't know the answer” option, the question was automatically forwarded to the expert data curator. To finalize the question, the patients either provided feedback on the question or pushed the “skip feedback” button. Furthermore, the patients were always able to check their IPS. At the closing evaluation session, the patients provided feedback on each step in the evaluation study, and the patients’ comments were documented in REDCap forms (22, 23) by the study nurse and/or research associate.

Evaluation data were collected via REDCap forms (questionnaires and narrative feedback) by the study nurse and/or research associate. The patients' answers to the Medical and Digital Literacy questionnaire were collected during the first on-site visit. The patients' answers to the G1 evaluation questionnaires were collected after testing the prototype for 2–4 weeks after the first on-site visit. These questionnaires included the system usability scale (SUS) (24) and study-specific questionnaires on the explainability/causability of G1 and the patient's interest in AIDAVA-like technology.

The patients were asked to comment on the questionnaires, the HDI, the blood pressure device, and each of the four deployment steps of G1 (data ingestion, data curation, data publishing, and data use). These comments were collected in the REDCap forms. The patients were given a time sheet to track how much time they spent on testing G1, and how much time they spent on study-related activities other than G1 testing (which include blood pressure measurements and other study-related activities).

All the quantitative data from the questionnaires are presented as mean ± standard deviation, median, and range. The study-specific Medical and Digital Literacy questionnaires contained six questions in each domain. The answers ranked from 0 (i.e., no knowledge) to 5 (i.e., expert knowledge). The purpose of the Medical and Digital Literacy questionnaires was user profiling, so that in an ideal situation, the virtual assistant could adjust communication to the patient accordingly. The evaluation questionnaires comprised the SUS questionnaire, which contains 10 questions, six study-specific questions on explainability/causability, and six questions related to the patient's interest in AIDAVA-like technology. The answers to these questions ranged from 1 (strong disagreement) to 5 (strong agreement) on a 5-point Likert scale. The item responses for each of the three sets of SUS questionnaires were analyzed using cumulative link mixed models, including a random intercept per patient. The Cronbach's alpha values were used to assess the reliability of the SUS questionnaires. The original 10 SUS questions were used to calculate the SUS score (0–100), which was then correlated to a level of acceptability and net promoter score (25). Data on time spent on data ingestion and data curation are presented as mean ± standard deviation, median, and range. Comparisons between the two use cases (BC vs. CVD) and questionnaire items were analyzed using independent-samples tests (t-test or Mann–Whitney U test, as appropriate). A P-value of 0.05 or less was considered statistically significant in all the analyses.

The study was approved by each local ethics committee at the participating test sites. An ethical advisory board, which includes external advisors, oversaw the G1 development process and the dry run workshop to ensure the study met all ethical and procedural standards.

A total of 423 patients were screened for inclusion in the study, with 182 for the CVD use case and 246 for the BC use case (Table 3). A total of 83 patients signed informed consent forms for G1 testing. However, 13 patients withdrew before or during the testing of G1. The reasons for withdrawals were lack of motivation, difficulty using digital devices, personal reasons, the perceived effort required for testing G1, or a combination of these. Ultimately, 70 patients successfully tested G1.

The patients reported significantly higher mean digital literacy scores (18.2 ± 6.7) than medical literacy scores (15.6 ± 5.9) (P = 0.01) (Table 4). Moreover, the patient-reported medical literacy scores were significantly higher for the patients in the BC use case (17.3 ± 5.9) compared to those in the CVD use case (13.3 ± 5.0) (P < 0.01). There was no significant difference in the patient-reported digital literacy scores between the BC use case (18.1 ± 6.3) and the CVD use case (18.3 ± 7.1) (P = 0.92).

The results suggest that the usability of G1 was marginally acceptable to the patients, as the mean score of the total was 59.1 ± 19.7 (Table 4) (Cronbach's α = 0.85). In total, 18 patients scored G1 higher than 68, which was considered the threshold of complete acceptability. In contrast, 18 patients scored G1 lower than 52, which means one should consider them detractors. The statement that scored the highest was “I am ready to use this system frequently”, with an average score between neutral and agreement (3.5 out of 5) (Figure 2). The lowest scores were for the statements “I would need the support of a technical person to be able to use this system”, “I found the system very awkward to use”, and “I needed to learn a lot of things before I could get going with the system”, with average scores between disagreement and neutral (2.3–2.6 out of 5). The comments by patients ranged from acceptance to rejection, as evidenced by the following two quotes:

The patients had moderately positive scores for the explainability/causability questions, scoring between 3.3 and 3.8 out of 5 (Figure 3) (Cronbach's α = 0.69). The results indicate a significant difference between the scoring for the question “For me it is important to know where the different curated health data are coming from” and the scoring of the question related to its explainability in AIDAVA, “In my opinion, information regarding this aspect is sufficient in AIDAVA” (3.8 vs. 3.2; P = 0.01). The patients in both the BC and CVD use cases agreed in their scoring of explainability/causability. A specific comment by a patient in the CVD use case emphasized the lack of explainability as to where the PHD came from:

The patients were moderately positive to positive in their interest in AIDAVA-like technology, scoring between 3.4 and 4.4 out of 5 (Figure 4) (Cronbach's α = 0.78). The patients in both the BC and CVD use cases were in agreement in their scoring, except for the question “I am ready to spend the needed time to ensure proper data curation of my data”, as the patients in the CVD use case were more willing to spend time to complete data curation than the those in the BC use case (4.3 vs. 3.7; P = 0.012).

The time spent on G1 consisted of “data ingestion and data curation” and “other activities,” such as the training, visit(s), or visualization of PHD. On average, the patients spent 124 ± 132 min on data ingestion and data curation, with a median of 75 min and a range of 0–635 min (Table 5). The patients in the BC use case spent significantly less time on average on data ingestion and data curation than those in the CVD use case (78 vs. 176 min, P < 0.01). The patients spent 78 ± 37 min on average on other activities related to G1 testing, with no significant differences between the BC and CVD use cases (78 ± 35 vs. 78 ± 38, P = 0.99).