Front. Health Serv.Frontiers in Health ServicesFront. Health Serv.2813-0146Frontiers Media S.A.10.3389/frhs.2025.1629242Health ServicesOriginal ResearchMeasuring the level of implementation of advance care planning – a fidelity-based cross-sectional studyWestbyeSiri Færden1*RomørenMaria1AhmedMarc2HermansenKarin Berg3BrøderudLinn1WahlAstrid Klopstad4SævareidTrygve Johannes Lereim1ThoresenLisbeth4PedersenReidar1Centre for Medical Ethics, Institute for Health and Society, Faculty of Medicine, University of Oslo, Oslo, NorwayDepartment of Geriatric Medicine, Oslo University Hospital, Oslo, NorwayDepartment of Health Sciences, Norwegian University of Science and Technology, Ålesund, NorwayDepartment of Public Health and Interdisciplinary Health Sciences, Institute of Health and Society, Faculty of Medicine, University of Oslo, Oslo, Norway
Edited by:Marios Kyriazis, National Gerontology Centre, Cyprus
Reviewed by:Preshit Ambade, Central Michigan University, United States
Heryyanoor Heryyanoor, Sekolah Tinggi Ilmu Kesehatan Intan Martapura, Indonesia
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
Introduction
Advance care planning (ACP) is supported by evidence, legal and ethical principles and ACP guidelines. However, this complex intervention is often poorly implemented. This article presents a novel fidelity scale to measure the implementation of ACP and reports the preliminary testing of the scale at baseline in a larger trial aiming at implementing ACP in hospitals in Norway.
Method
The research design conducted was a cross-sectional measurement of fidelity to guidelines, conducted in 12 geriatric hospital units in Norway recruited using purposive sampling. The sample size for the larger trial was determined, based on prior research, to be at least four clusters in each arm. Due to the possibility of unit drop-out during the project period and to secure sufficient power, we included six units in the intervention arm and six in the control arm. For the baseline measurement we developed an ACP fidelity scale including three subscales: implementation, quality and penetration rate, each with 4–11 items. We ensured face and content validity through input from relevant theories and research, experts and users. Items were scored from 1 to 5, with 1 indicating no implementation and 5 indicating full implementation. Data was analyzed using descriptive statistics, Cronbach's alpha and calculation of interrater reliability for the scales. Further psychometric testing could not be conducted at this stage due to the lack of variance in the items.
Results
The mean score for the implementation subscale was 1.213, with a median of 1, a standard deviation (SD) of 0.08, a standard error (SE) of 0.01, a variance of 0.01, and a range of 0.28 (minimum 1.14 and maximum 1.42). The scores in the subscale showed that none of the units had recommended implementation measures. Only a few professionals reported they had heard of ACP, but not the whole staff. Cronbach's alpha could not be estimated due to the lack of variation in the scores for this subscale. On the quality subscale, which assesses whether ACP is practiced in accordance with practice guidelines, the mean score was 1.11, the median was 1, the SD was 0.48, the SE was 0.06, the variance was 0.13, and the range was 1.27 (minimum 1 and maximum 2.27). The scores in this subscale showed that ACP was practiced sporadically by the palliative care team in only one unit, while the other staff did not engage in this practice at all. Cronbach's alpha for the subscale on quality was 0.887 (11 items) showing an acceptable internal consistency. For the penetration rate subscale, which measures how widespread the practice is, the mean score was 1.08, the median was 1, the SD was 0.28, the SE was 0.05, the variance was 0.08, and the range was from a minimum of 1 to a maximum of 2. Among the total number of admitted geriatric patients, only 10% had received ACP in only one of the 12 units. Also, for this subscale, the model for Cronbach's alpha could not be applied. There was little variation in the low measurements, thus the interrater reliability was high, reflected in the Intraclass Correlation Coefficient (ICC). The ICC was 0,916 [−0,721,0.976] for the implementation subscale, 1.00 for the quality subscale, and 1.00 for the penetration rate subscale.
Conclusion
Our findings indicate very low implementation of ACP in acute geriatric hospital units in Norway. The newly developed ACP fidelity scale has the potential to serve as an important tool for improving the quality of healthcare services for older patients. However, more data is needed to validate the psychometric properties of the scale. Our study should be considered as a preliminary study, and the scale should be used with caution as long as its properties are not well validated.
Clinical Trial Registration
ClinicalTrials.gov, NCT05681585.
advance care planningimplementation sciencepatient centeredolder patientshealth systems302880Norwegian Research Councilsection-at-acceptanceImplementation ScienceIntroduction
ACP is defined as a process that supports patients in understanding and communicating their personal values, life goals, and preferences regarding present and future medical care to ensure that people receive medical care consistent with these preferences (1). Advance care planning (ACP) is increasingly recommended in clinical practice guidelines beyond palliation (2, 3), also in Norway (4, 5). However the implementation of ACP is patchy and numerous barriers to implementation have been reported (6–9). This is despite the recent evidence synthesis showing that ACP has the potential to support the involvement of older patients and their next of kin and improve communication (10, 11) and is today a global problem (8).
To improve the implementation of ACP, a first step is to measure the degree of implementation of ACP. Within implementation science various “fidelity”-scales to measure implementation of evidence-based practices (EBP) have been developed (12). Such scales can be used to measure the degree of implementation of an EBP and monitor the effects of implementation programs (12).
“Fidelity” can be narrowly defined as adherence to evidence-based guidelines focusing on how to do the clinical intervention, to ensure adequate quality when the intervention is applied. However, “fidelity” can also be defined more broadly, to include also an assessment of implementation interventions, i.e., any measures or strategies used to ensure that the clinical intervention is used in line with EBP-guidelines, and to what extent the clinical intervention is used (often called “penetration”) (13). Most fidelity studies focus on the narrower definition of fidelity (14). This also applies to the relatively few ACP implementation studies assessing fidelity that we have been able to identify (15, 16).
Both ACP as a clinical intervention, and ACP-implementation measures and strategies can be defined as “complex interventions”, e.g., interventions that require a range of behaviors required of those delivering or receiving the intervention at different levels (17). Broader approaches in implementation studies and fidelity measurements are particularly important when it comes to complex interventions, because an intervention or treatment will not be effective if it is not implemented well, e.g., perfect ACP-quality is of limited value if ACP is very rarely done (18).
However, we were not able to identify any measurements developed to comprehensively assess the implementation of ACP. Thus, we decided to develop a fidelity scale based on the broad definition of fidelity.
This was done as part of a larger study aiming at improving the implementation of ACP in the routine care of acutely admitted home-dwelling geriatric patients in Norway (19), where the scale was used to assess the implementation of ACP at baseline in the participating units.
The aim of this article is both to present the development of the new ACP fidelity scale, and to report the results of the baseline assessment as/including a preliminary validation of the scale.
Material and methods
This article corresponds to the 'Strengthening the reporting of observational studies in epidemiology (STROBE) statement' (20) (Please insert Supplementary File 1 here).
Study design, setting and participating units
This study examines the current level of ACP at baseline, prior to conducting a trial aimed at implementing ACP in the routine care of acutely admitted patients in acute geriatric units (19). We have chosen a more general geriatric hospital setting because ACP has become increasingly relevant beyond palliation and oncology (2) and because this context has been little studied (21). The literature on ACP mostly covers the context of end-of-life care and/ or nursing homes. But the expert consensus definition for ACP recommends it across the life continuum (1) and that it should ideally start early in adulthood (1). A preceding study in Norwegian nursing homes found ACP may profit from commencing before nursing home admission due to cognitive impairment (22). In this project we therefore included home dwelling patients, that were acutely admitted to hospital. Previous research shows that the hospital setting can increase the relevance of ACP (23). Furthermore, we included patients living with frailty because they are a particularly “in need” group, given their increased risk of morbidity and mortality (21). The reason for selecting geriatric units was also that we found a high level of competence in communication with older patients in these units, as well as a motivation and interest for ACP.
The study is based on a cluster-randomized design and included 12 hospital units (19). A cluster was defined as an acute geriatric unit or an internal medicine unit with a geriatrician among staff. In Norway, geriatrics is a “branch” of internal medicine. This implies that one has to specialize in general internal medicine before one can become a specialist in geriatric medicine, that is a specialist in diseases in old age. We thus included acute geriatric units, and internal medicine units with a geriatrician among staff when this was the only offer to geriatric patients. 12 out of 15 eligible units in the Southeast Norway Regional Health Authority agreed to participate, covering a catchment area of over two million inhabitants. The main reason for the non-participation of the remaining units was the lack of time and personnel.
After randomization, the intervention arm received 18 months of training and support in implementing ACP guidelines and national recommendations. To increase implementation, the control arm received a shorter version of the same program after the 18 months period. Fidelity was measured in the units at baseline and two follow ups: at 7–10 months and 18 months. For the study as a whole, the sample size was calculated to be at least four clusters in each arm. Due to the possibility of unit drop-out during the project period and to secure sufficient power, we included six units in the intervention arm and six in the control arm (19). Table 1 summarizes the characteristics of all clinical units, and the variation in the organization. The lack of time and resources, including adequate staffing, is a significant barrier highlighted in the literature (6, 24, 25). In our sample the number of patients per Full-Time Equivalent (FTE) staff varied considerably with a range of 3.18 (minimum 1.77 and maximum 4.95).
Description of the 12 clinical sites included in the cluster randomized trial.
Site
Type of unita
Catchment population
The maximum of patients in the units
Reported percentage of the maximum number of patients who have participated in ACP in the last six months
Mean length of stay for acute geriatric patients (days)
Full-time equivalent staff (FTE)
Patients per FTE
1
IMU
30 267
10
10%
3,48
17,7
1,77
2
AGU
50 000
6
0%
4,55
23
3,83
3
AGU
60 000
5
0%
3,3
30,7
1,33
4
AGU
120 000
9
0%
4
40
4,44
5
AGU
150 000
16
0%
4,1
42,7
2,67
6
AGU
155 000
4
0%
2,5
19,8
4,95
7
AGU
168 000
8
0%
4
39,4
4,9
8
AGU
180 000
8
0%
3,6
33,7
4,21
9
AGU
200 000
20
0%
4,9
36
1,8
10
AGU
230 000
9
0%
4
16,5
1,83
11
AGU
323 453
18
0%
4
47,2
2,6
12
AGU
594 000
23
0%
6
53,5
2,32
AGU: Acute geriatric unit, IMU: Internal medical unit with geriatrician among staff.
Instruments
In the present survey, we developed three subscales with 4–11 items (22 items in the total scale). The scale was written in Norwegian, and the study was conducted in Norwegian, but we have translated the scale into English to share this instrument with clinicians and researchers (please see Supplementary File 2).
In each scale, items are scored on a scale from 1 to 5, with 1 indicating no implementation and 5 indicating full implementation. The scores of the items are added together and divided by the total number of items in the respective subscales, and the overall scale to determine the average scores. The purpose of the scales was to operationalize the guidelines and recommendations for ACP practice and implementation support in healthcare. The order of items is chronological as a model for action, and to ensure a logical and easily understandable order for the researchers and the units being evaluated, as well as for other users of the scale (26). A schematic overview of the scale can be seen in Figure 1.
Schematic overview of fidelity scale for advance care planning (implementation measure items are marked in blue, quality of practice items in green, and penetration rate items in purple).
Fidelity scale for advance care planning with 22 steps, highlighting key actions like kickoff meetings with staff, writing action plans, appointing coordinators, providing training, systematic screening, evaluating patient consent, involving patients, adapting information, quality documentation, and ongoing evaluations.
The study protocol describes how the project group developed the ACP interventions by selecting recommendations from the guidelines and synthesizing them into key elements (19).
In parallel to this process, we developed the items that would cover these key elements of the ACP interventions. A description of the ACP interventions can be found in the study protocol (19). An overview of the interventions can be seen in Figure 2.
ACP interventions in the larger implementation project. (We have changed the terminology in the project after the publication of the protocol article, so that we now define implementation intervention as both implementation measures and implementation strategies. Together with the clinical intervention, this constitutes the entire ACP intervention package in the project. The new terminology is to be presented in a forthcoming article).
Flowchart illustrating three sections: Clinical Intervention, Implementation Intervention, and Implementation Strategies. Clinical Intervention includes advance care planning, routine identification, ACP discussions, and documentation. Implementation Intervention covers the team, ACP coordinator, training, toolkit, structured fidelity measurements, and evaluations. Implementation Strategies highlight leadership commitment, responsive evaluation, whole ward approach, train-the-trainer model, and project sustainability. Each section is visually differentiated by color.
In developing the scales, we sought to include items that measure both practice adherence to the guidelines and the implementation measures needed to sustain the practice, as well as the penetration rate. By penetration rate, we mean the percentage of eligible patients and/or next of kin who receive a particular invitation, treatment, service, or practice (27). Thus, the scale emphasizes the importance of reaching a large number of patients while also practicing ACP accurately. Thus, the three subscales measure 1) the quality of the intervention, 2) an assessment of the implementation measures and 3) the penetration rate in the participating units.
An overview of the evidence base for the items can be found in Supplementary File 4.
Methodology
The project group followed the standardized procedure for scale development described by Bond and colleagues (12). For the subscale measuring the quality of ACP practiced, we constructed the items based on the Delphi panel recommendations (1–3), which are also recommended as the most rigorous source in the development of new scales (12). The second most important base was Up to Date's summary of the practical steps recommended in an ACP conversation (28), which complements the Delphi panel's consensus definitions (1, 2). We have included that the patient's decision-making capacity should be assessed as one of the steps in ACP (28). This is recommended in the national guidelines because the role of next of kin becomes more important in a situation where the patient's capacity to consent is weakened - either as a support for the patient or as a representative of the patient when the patient's competence has ceased (5). This is in line with the most recent consensus definition of patients with dementia (3). Under Norwegian law, next of kin do not have the authority to make medical decisions on behalf of the patient; however, healthcare providers are required to consult them regarding the patient's wishes. Serious interventions may only be performed if they align with the patient's best interests and if there is reason to believe that the patient would have consented had they been able to do so (29).
The third source we used to develop the items, as recommended by Bond, was systematic reviews on ACP (12). An overview of the literature search for the reviews can be found in Supplementary File 3. Randomized controlled trials that were included in these reviews were trials that provided a description of the implementation programs or a description of the intervention or the training program for healthcare professionals. A challenge was that ACP interventions vary widely in the literature (8). The interventions used were also poorly described and were often licensed programs that are not openly accessible. However, some recurring steps would be the identification of patients approaching the end-of-life, assessment or exploration of the patient's needs and personal values, conversations about different care options and choices, as well as final decisions and ensuring a coordinated documentation of ACP which we made sure were reflected in the items on quality (21, 30, 31).
To adapt the quality subscale to the Norwegian context, the most useful source was an interview guide for ACP that was tested in a cluster randomized trial in nursing homes (32). The trial used a whole-ward approach, which is also a strategy used in this study. The strategy aims at involving the entire staff in the ACP process, facilitating both formal conversations where the patient and their next of kin are invited to engage, as well as informal discussions where patients can initiate conversations with staff about what matters to them, such as during care or other daily activities. We also ensured that the scale was in line with the recent Norwegian recommendations on ACP (5). Finally, we looked at a procedure for ACP developed at Oslo University Hospital (33). The procedure is based on the Gold Standard Framework, which is one of several ACP models (34).
The items in the subscales measuring implementation support and penetration rate were generated based on previous scales validated and tested for reliability in the project to improve family engagement (13), as no established measures of ACP implementation or quality exist. We have also included a review article presenting the Gold Standard Framework implementation model (35). In addition, we have included the known barriers and facilitators to ACP implementation to systematize the areas that would be relevant in the implementation process in the implementation support subscale. An overview of the literature search for barriers and facilitators can be found in Supplementary File 3.
The next step involved content validation, which included gathering feedback from experts and users to inform revisions of the items in group meetings. Content validity refers to the extent to which the items in a test are reasonably representative of the entire domain that the test attempts to measure (36). In response to feedback, in some cases tasks were tailored to local circumstances, which is recommended when implementing complex interventions (17). An important example is how ACP should be documented and communicated in a context where documentation systems differ from hospital to hospital, and where the primary healthcare system lacks a unified system that is shared with specialist health services. With input from experts and clinicians, we concluded that the most important themes discussed during the conversation, along with any specific preferences regarding treatment and care, should be included in the discharge summary. This document is read by the general practitioner or nursing home physician. Consequently, we included this as an item in the penetration rate, to measure not only how many patients received ACP, but also how many of them had ACP documented appropriately to be effective beyond that hospital stay. We also considered content validity during piloting which consisted in testing the version of the scale we had agreed on during internal validation with staff groups that would be interviewed, e.g., a geriatrician, a head of department, and a nurse. Here we also sought to ensure face validity, i.e., the appropriateness, meaningfulness, or relevance of the test as it appears to the people answering the test (37). User acceptability, as recommended by Bond and colleagues (12), was further assessed during baseline measurements. An exception to the Bond and colleagues' methodology is that the scale could only be tested to a limited extent in this study due to time constraints. Some items were therefore modified after the baseline data was collected. The scale was adjusted both at the 7–10-month follow-up and 18-month follow up, which resulted in some minor changes to the scale. We present the changes made and the rationale behind them in the amended version of the scale in Supplementary File 2 (please see Supplementary File 2). In this article we present the amended version of the scale that was used in the larger study implementing ACP (please see Supplementary File 2).
We chose to use the scale in the larger study before it had been more thoroughly validated because we had limited time and resources available, because we had had good experience with a similar scale in the previous study investigating family involvement in people with psychotic disorders (38), and because we had conducted a relatively thorough development process in advance. However, we also consider that the results need to be interpreted with caution until further validation is available, and for the time being should be regarded as the results of a preliminary validation. At the same time, our experience and reliability tests gave us no reason to make any major changes or not to continue with this scale in the larger study.
Data collection
The clinical units were recruited from fall 2021 to February 2022. Baseline measurements took place between May and July 2022. Fidelity as a method can use different data sources, such as observations, interviews with staff, or a review of medical records and program documents (12). In this fidelity study trained fidelity assessors visited each unit and measured fidelity by conducting structured interviews with the heads of the department, clinicians and any local resource persons such as the palliative care team. Some of the measurements were conducted digitally for feasibility reasons. We considered this acceptable because none of the units practiced ACP in a systematic way.
We also asked whether the units had any written or online material on ACP. Only one unit had a written procedure for ACP. The head of the department (department or section head) was interviewed individually, while clinicians and resource persons (if applicable) were interviewed in separate groups of 1–5 people. In total, 2–4 interviews lasting 1–1.5 h were conducted at each site. We also collected organizational data (Table 1).
At each site, the two fidelity assessors first rated all items independently and then arrived at a common rating for each item. The assessors and the composition of assessors varied from site to site. The raters were selected from a pool of eight researchers who also worked in healthcare, but none of them worked at the clinical units.
The assessors prepared a detailed report for each site, which was attached to the score results. The results and reports were sent to the units in the intervention arm after randomization so that they could correct any misunderstandings and adjust the results if necessary. However, none of the units provided feedback that led to an adjustment of the scores. The units in the control arm did not receive their assessments or reports so as not to influence them during the intervention phase.
Data analysis
We examined item distributions for all three subscales, including means, median, standard deviation (SD), standard error (SE), ranges, minimum and maximum values and the number of units that achieved low, adequate, and full implementation of the various items. All data analyzes were conducted using SPSS version 28.
Based on the organizational data from the clinical units and the scores, we also calculated the percentage of patients admitted to the ward during the last six months who had received ACP.
Results
The distribution of items and interrater reliability are presented in Table 2.
Fidelity baseline results, distributions, interrater reliability and percentage agreement for the fidelity scale, after testing in the 12 hospital units.
Item number
Item description
Mean
Median
SD
SE
Variance
Range
Minimum
Maximum
Numbers of sites achieving score
Agreement (%)
ICC (confidence intervals)
Implementation subscale
1
2–3
4–5
1
Meeting
1
1
0
0
0
0
1
1
12
0
0
100
1.000
2
Action plan
1
1
0
0
0
0
1
1
12
0
0
100
1.000
3
Program philosophy
2.25
2
0.11
0.11
0.42
3
1
4
1
11
0
66.67
0.784 [0.284,0.937]
4
Coordinator
1.33
1
0.48
0.08
0.23
1
1
2
8
4
0
100
1.000
5
Training and supervision
1
1
0
0
0
0
1
1
12
0
0
100
1.000
6
Systematic identification of eligible patients
1
1
0
0
0
0
1
1
12
0
0
100
1.000
22
Continuous implementation measures
1
1
0
0
0
0
1
1
12
0
0
100
1.000
Subtotal for implementation measures
1.23
1
0.08
0.01
0.01
0.28
1.14
1.42
12
0
0
—
0.916 [−0,721,0.976]
Cronbach's alpha for implementation subscale
N/A
Quality subscale
9
Assessing decision-making capacity
1
1
0
0
0
0
1
1
12
0
0
100
1.000
10
Verbal and written invitation.
1
1
0
0
0
0
1
1
12
0
0
100
1.000
11
Level of involvement
1.08
1
0.28
0.47
0.79
1
1
2
11
1
0
100
1.000
12
Provide information
1
1
0
0
0
0
1
1
12
0
0
100
1.000
13
What is important to the patient
1.08
1
0.28
0.47
0.79
1
1
2
11
1
0
100
1.000
14
Elicit preferences regarding future and current treatment and care
1.08
1
0.28
0.47
0.79
1
1
2
11
1
0
100
1.000
15
Ask the patient if they have written down any previous wishes regarding healthcare
1 (0)
1
0
0
0
0
1
1
12
0
0
100
1.000
16
Permission to pass on information to other health personnel
1.08
1
0.28
0.47
0.79
1
1
2
11
1
0
100
1.000
17
Summing up and evaluating the ACP-discussion
1.25
1
0,84
0.14
0.71
3
1
4
11
0
1
100
1.000
19
Quality of documentation
1.25
1
0,48
0.14
0.71
3
1
4
11
0
1
100
1.000
20
Dissemination of ACP in discharge summary
1.33
1
1,12
0.19
1.26
4
1
5
11
0
1
100
1.000
Subtotal for quality
1.11
1
0,36
0.06
0.13
1.27
1
2.27
11
1
0
—
1.000
Cronbach's alpha for quality subscale
0.887
Penetration rate subscale
7
Proportion invited
1 (0)
1
0
0
0
0
1
1
12
0
0
100
1.000
8
Proportion given ACP (penetration rate in the unit)
1 (0)
1
0
0
0
0
1
1
12
0
0
100
1.000
18
Checking in the patient's record if ACP has been done previously
1,33
1
1,12
0,19
1,26
4
1
5
11
0
1
100
1.000
21
ACP in discharge summary
1 (0)
1
0
0
0
0
1
1
12
0
0
100
1.000
Subtotal for penetration rate
1.08
1
0,28
0.05
0.08
1
1
2
12
0
0
—
1.000
Cronbach's alpha for penentration rate subscale
N/A
Scale total
1.13
1
1.16
0.03
0.26
0.60
1.05
1.64
12
0
0
—
0.983 [0,945;0,995]
For the implementation subscale, the mean score was 1.213, with a median of 1, a SD of 0.08, a SE of 0.01, a variance of 0.01, and a range of 0.28 (minimum 1.14 and maximum 1.42). This reflected that in four of the units, an ACP coordinator had been appointed prior to their involvement in the project, but no kick-offs had been held with important staff, and there were no action plans for implementing ACP in the units or leadership commitment concerning ACP. As a result, none of the units had successfully integrated ACP into their organizational practices.
Regarding practical training, no units offered this to staff, apart from one digital theoretical course that had been held for physicians. In terms of supervision, two units had routines in place to discuss ethical issues for the nurses, but not for physicians or other staff. A minority of staff in five of the units were familiar with the principles of ACP, primarily through English-language literature or a personal interest in palliative care. However, the key principles of ACP according to guidelines was not fully understood by staff. Furthermore, none of the units had systems in place to identify patients that could be eligible for ACP. Cronbach's alpha could not be estimated because the items lacked sufficient variance.
For the quality subscale, the mean score was 1.11, the median was 1, the SD was 0.48, the SE was 0.06, the variance was 0.13, and the range was 1.27 (minimum 1 and maximum 2.27). The assessment of decision-making capacity specifically for ACP was not practiced in any of the units. Involvement of patients and their next of kin was also patchy due to the lack of time, and when they did take place, conversations were almost only about the limitation of life-prolonging treatment and not ACP. In one of the units ACP was practiced sporadically by the palliative care team. They asked about the level of involvement, what was important to the patient, and identified preferences and asked for the permission to share them, as well as also documented ACP in the discharge summary, covering a considerable amount of principles. However, in none of the units ACP was fully practiced as recommended or a systematic practice part of routine care. Cronbach's alpha for the subscale on quality was 0.887 (11 items) showing an acceptable internal consistency.
In the subscale measuring the penetration rate of ACP, the mean score was 1.08, the median was 1, the SD was 0.28, the SE was 0.05, the variance was 0.08, and the range was from a minimum of 1 to a maximum of 2. This shows that in the previously mentioned unit, the palliative care team checked if the patient had participated in ACP previously in the patient record. Notably, this was also the only unit reporting that they had offered ACP to patients during the last six months; however, only 10% of their admitted patients had received this offer (please see Table 1). In contrast, the majority of staff across the other 11 units did not engage in ACP. Cronbach's alpha could not be calculated due to insufficient variance among the items.
We also calculated the percentage of exact agreement for each item and examined the interrater reliability reflected in the Intraclass Correlation Coefficient (ICC). The ICC was 0,916 [−0,721,0.976] for the implementation subscale, 1.00 for the quality subscale, and 1.00 for the penetration rate subscale, and 0.983 [0,945;0,995] for the total fidelity scale.
Discussion
This study presents the development and preliminary testing of the first fidelity scale to measure the structure, content, penetration rate and implementation of basic ACP. The scale is intended for use in geriatric hospital units, but with some modifications may also be suitable for health services in general, as its content is basic. This is of value since ACP is increasingly acknowledged as relevant to all individuals, and that it should be integrated into routine care, normalizing these discussions (39). The results should be interpreted with caution as there was little variation in measurements and validation of the scale should be further developed to include larger samples and samples over time. However, the results are stark and important and confirm that ACP requires a radical shift in its implementation to remain relevant (8, 9, 39).
The main findings of this study indicate that crucial structures necessary for implementing ACP in the units are absent, even in geriatric units that possess knowledge, tradition and expertise in communicating with older patients and next of kin. These include designated personnel responsible for coordinating and maintaining ACP practices, and commitment from leadership, training and supervision for staff, and having systems for identifying eligible patients, as well as documenting ACP to support quality improvement.
Regarding the quality of ACP, although some staff members were familiar with the principles, the majority of personnel lacked adequate knowledge about them. When ACP was practiced, several important components of the guidelines were often overlooked. These included assessing the patient's decision-making capacity, providing information about their health status tailored to the patient's preferences for information, and asking whether they had documented their healthcare wishes, such as through an advance directive or a living will which is more commonly used in Norway (1–3, 5). The observation that only the palliative care team in one of the included hospitals practiced ACP occasionally, also indicates a need for interdisciplinary collaboration and communication, and that ACP should not be limited to one team but integrated across all care teams involved in geriatric patient care, which is known from previous literature (9).
The results of this study also align with findings from other baseline data regarding the confidence levels and the extent of communication training on ACP among healthcare personnel in this context. A survey on self-assessed confidence in communication about future deterioration and medical care and the patient's own wishes towards end-of-life showed that this varies, and is limited among many geriatric health personnel in this setting (40). As part of the trial, we also aimed to investigate barriers and facilitators from within the Norwegian health care context more broadly (41). Known barriers that were echoed were the lack of knowledge and inadequate training that pose as a lack of communication skills and preparedness, as well as clinicians' discomfort and reluctance to talk about existential issues and dying and death (9, 24, 25, 42). This study highlights the need to build confidence and competence among healthcare personnel regarding ACP through knowledge, training, and supervision, as well as explicit prioritization from managers and policymakers, to embed ACP into the organizational culture.
Concerning the penetration rate, very few patients received an invitation to ACP, or were offered an ACP discussion that was also documented in the discharge note. This also suggests that policies and practices need to be revised to facilitate better integration of ACP documentation. Two previous reviews exploring the uptake of ACP for older patients including home dwelling elders in communities, describe that inconsistencies and variation in documentation act as barriers (43), and that healthcare professionals are often wary of documentation because they fear it may go against the patient's wishes (44). This study shows the absence of systems to document and communicate ACP across different levels, which is essential for ensuring continuity and supporting ACP as an ongoing process (1–3, 5, 9).
Subsequently, within the larger project, we used the knowledge gained from the baseline measurements to identify specific areas needing improvement and tailored our implementation strategy accordingly (45). For example, we offered practical training to units, facilitated by a communication expert, to help alleviate uncertainty around engaging in ACP. Additionally, we trained local staff and involved all personnel through a train-the-trainer model. We also developed and provided documentation templates for sharing among the units and are currently working towards agreeing on a common template to document ACP nationally.
Additionally, in the reports accompanying the fidelity measurements, the project team noted existing routines that could be used to improve the uptake, for instance, that nurses could be encouraged to identify eligible patients for physicians. Previous findings also suggests that a nurse-led pathway has been effective in increasing the frequency of ACP discussions in primary care settings (46).
Strengths and weaknesses
The main strength of this study is the development of a new instrument that measures the implementation level of ACP in a standardized and structured manner. This allows for the assessment of all units using a single sample (45), and that areas that need improvement can be targeted, and monitored to track progress during an implementation program of ACP (45). It is also a strength that we have a better measure of the implementation level of ACP than in any previous study. Many of the items are also generic, which is a strength in terms of transferability to other parts of the health services.
A factor analysis cannot be performed at this stage, since the items have limited variation and thus undermine the possibility of using the models. The validation that was possible given the skewed data, a cross-sectional measurement, and that there are no other established measures of ACP practices, consisted of face and content validity, Cronbach's alpha for the subscales where this was possible, and a preliminary test of interrater reliability. In a follow-up study presenting fidelity assessment results at 7–10 months and 18 months, we will re-examine interrater agreement, conduct a CFA (Confirmatory Factor Analysis) and determine Cronbach's alpha for all subscales when we expect the level of implementation to be more varied across the units. We also plan to assess sensitivity to change, i.e., the ability of the scale to detect meaningful changes in ACP implementation after support (12) by comparing with other data in the larger project at the patient/next of kin level, clinical level and service level.
The gold standard in fidelity measurement is a site visit conducted by independent and experienced fidelity reviewers who observe the intervention, attend team meetings, interview staff and clients, and review medical records and program documents. A guiding principle for enhancing validity of ratings is triangulation, that is, seeking information from multiple sources to confirm ratings (12). However, this was not possible in this project due to the lack of time and resources, and the measurements are based solely on interviews with staff. Such an approach can make it feasible for units to adopt the fidelity scale to evaluate their own practice. To enhance the reliability and validity of ratings, future studies might consider including patient record reviews or direct observations, which would add depth and credibility to the assessments. Record review is time-consuming, especially when the item at issue is not consistently documented in a specific place such as ACP. In this study an important step was to set up a document type of its own so that ACP was consistently documented in the same type of document. A random selection of records that would include this type of document could save time in future iterations of this study, and a random selection would also guard against obvious sources of bias. Concerning the subscale on implementation, attending implementation team meetings could probably also be added as a source of data to confirm the accuracy of responses because ACP is a team-based EBP (involving the cooperation of different staff groups) (12), and the teams represent key staff involved in the implementation process. Reviewing program documents such as local procedures, electronic handbooks or manuals on ACP, information brochures and leaflets for patients could also increase the validity and reliability of the measurements (12), particularly on quality of practice. Including patients and next of kin as a data source could also increase the measurements on quality to confirm whether practice is in adherence with guidelines. Alongside the intervention, we will also conduct separate qualitative interviews with patients and next of kin, and collect questionnaires from staff, patients, and next of kin to assess service outcomes and patient outcomes and experiences. These data sources could also be used in further applications of the scale to triangulate findings on quality.
In terms of the representativeness of the sample, we only included acute geriatric units that have expertise in the care of older patients. This could pose as a problem of selection bias, and the scale should be tested in other contexts. In terms of external validity, these results do not necessarily reflect the situation in other wards that receive acutely ill home-dwelling older patients. Nevertheless, our preliminary measurements of ACP practices show that this is not well implemented even in wards that specialize in the care for older patients. Therefore, we have little reason to believe that ACP practices will be more systematic or of higher quality in other wards with older patient groups.
This scale is part of an ongoing national effort to improve ACP implementation and is intended for national or more local benchmarking after further validation, to establish standards for delivering ACP as a service to older patients. Evaluating whether improved fidelity gives better outcomes of the ACP intervention at the patient and service levels is the primary objective of the larger study (19). Future iterations of this scale may serve as a strategy to influence policy decisions by demonstrating the relationship between improved fidelity and program effectiveness. This is a way of providing evidence to support the allocation of resources to ACP programs that achieve high fidelity rates, particularly for more vulnerable patient groups. The scale assesses the structures necessary for integrating ACP into routine care, adherence to practice guidelines, as well as the penetration rate because dissemination of ACP is not simply about creating small exclusive programs for selected patients, but to make ACP easily accessible across the public health system. The scale can be used for internal quality improvement when using staff interviews, because it makes it feasible for units to use the scale to evaluate their own practices. We will also use the scale to do a research evaluation, including evaluating how closely the implementation of the intervention follows the original research protocol, checking whether all components were delivered as planned, and in which areas the research team should intervene to improve fidelity. Furthermore, we are continually refining and recalibrating the items based on our experiences with the scale, as well as evolving research and practices. This ensures that the scale remains relevant and reflective of best practices while also considering real-world conditions in clinical units.
Conclusion
This study addresses a significant gap in the literature: the lack of a measurement tool to comprehensively assess the implementation of ACP to tackle the global challenge of insufficient implementation. This cross-sectional assessment conducted as a preliminary test of the scale i indicates that ACP is not routinely implemented in participating geriatric units, and that staff training, leadership involvement, and systematic follow-up are all lacking.
The scale is in its preliminary stages and further validation is necessary. Additional data are required to further establish its psychometric properties. We advise exercising caution in the use of the scale and in the interpretation of the results in this preliminary study.
The fidelity scale is intended for national or more local benchmarking to promote the uptake of ACP and improve the quality of health services for older patients. There is a gap between evidence and practice, and this scale can be used to identify and measure that gap, ultimately improving health services for older patients and their families.
Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics statement
Informed consent was obtained from all subjects. This research complies with international and national standards and has been performed in accordance with the Declaration of Helsinki. It has been approved by the appropriate national ethics committees. SIKT- the Norwegian Agency for Shared Services in Education and Research, approved the study on February 25, 2020. The legal basis for their approval is the Personal Data Protection Regulation Act art. 6 no. 1 letter 1, cf. art. 9 no. 2 letter a, cf. Personal Data Act § 10. SIKTs reference number is 805491.
An application for prior approval of the research project was also processed by the Regional Committee for Ethics in Medical Research and Health Care (REK). The assessment concluded that the project will not gain new knowledge about health or disease as healthcare research. The project therefore does not fall within the remit of the REK according to the Norwegian Health Research Act § 10, cf. Their evaluation was completed on February 2, 2020. Their reference number is 52825.
Author contributions
SW: Writing – original draft, Writing – review & editing. MR: Writing – original draft, Writing – review & editing. MA: Writing – original draft, Writing – review & editing. KH: Writing – original draft, Writing – review & editing. LB: Writing – original draft, Writing – review & editing. AW: Writing – original draft, Writing – review & editing. TS: Writing – original draft, Writing – review & editing. LT: Writing – original draft, Writing – review & editing. RP: Writing – original draft, Writing – review & editing.
Funding
The author(s) declare financial support was received for the research and/or publication of this article. The Norwegian Research Council funded the study. The grant number is 302880. The funder had no influence on the design of this study and played no role in the conduct, analyses, interpretation of the data, or decision to publish the results.
Acknowledgments
We would like to thank Associate Professor Kristin Heiervang from the Department of Health, Social and Welfare Studies at the College of Southeast Norway for excellent help in developing the fidelity scale and MD and Associate Professor Lars Hestmark at the Centre for Medical Ethics at the University of Oslo for excellent help in calculating the ICC, as well as Professor in statistics Magne Thoresen at the Institute of Basic Medical Sciences at the University of Oslo for excellent help regarding the statistical analysis and psychometrics. We would like to express our gratitude to postdoc Lennart Kiel at the Department of Psychology and Behavioral Sciences at the University of Aarhus and postdoc Patrick Cairns at the Department of Behavioral Medicine at the University of Oslo. Their invaluable advice on psychometric testing, particularly in contexts where the methods have not yet been implemented, has greatly contributed to our work. We would also like to thank Senior Consultant Geriatrician Pål Friis and Professor and Senior Consultant Geriatrician Siri Rostoft at the Department of Geriatric Medicine, Oslo University Hospital for excellent comments during the face validation of the scale. Finally, we would like to thank postdocs Shereen Cox and Tore Hofstad at the Centre for Medical Ethics at the University of Oslo for very valuable comments in the early writing process.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Generative AI statement
The author(s) declare that Generative AI was used in the creation of this manuscript. InstaText was used for accuracy and spelling of words and formulations in the preparation of this manuscript, but not for writing the manuscript as a whole. The authors verify and take full responsibility of the use of this generative AI.
Any alternative text (alt text) provided alongside figures in this article has been generated by Frontiers with the support of artificial intelligence and reasonable efforts have been made to ensure accuracy, including review by the authors wherever possible. If you identify any issues, please contact us.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frhs.2025.1629242/full#supplementary-material
Abbreviations
ACP, advance care planning; EBP, evidence-based practice; SD, standard deviation; SE, standard error.
ReferencesSudoreRLLumHDYouJJHansonLCMeierDEPantilatSZDefining advance care planning for adults: a consensus definition from a multidisciplinary Delphi panel. J Pain Symptom Manage. (2017) 53(5):821–32.e1. 10.1016/j.jpainsymman.2016.12.33128062339RietjensJACSudoreRLConnollyMvan DeldenJJDrickamerMADrogerMDefinition and recommendations for advance care planning: an international consensus supported by the European association for palliative care. Lancet Oncol. (2017) 18(9):e543–e51. 10.1016/S1470-2045(17)30582-X28884703van der SteenJTNakanishiMVan den BlockLDi GiulioPGonellaSIn der SchmittenJConsensus definition of advance care planning in dementia: a 33-country Delphi study. Alzheimers Dementia. (2024) 20(2):1309–20. 10.1002/alz.13526Decision-making processes in the limitation of lifeprolonging treatment (Norwegian Directorate of Health) (2013). Available online at:https://www.helsedirektoratet.no/veiledere/beslutningsprosesser-ved-begrensning-av-livsforlengende-behandling/Decision-making%20processes%20in%20the%20limitation%20of%20life-prolonging%20treatment.pdf/_/attachment/inline/fcaec913-6115-485a-831d-10ab57c1b7ef:31ae370fb2b675b6d56f042ed36dc9ca54569632/Decision-making%20processes%20in%20the%20limitation%20of%20life-prolonging%20treatment.pdf(Accessed August 15, 2024).National recommendations for advance care planning (Norwegian Directorate of Health) (2023). Available online at:https://www.helsedirektoratet.no/faglige-rad/Forhandssamtaler-og-planlegging-ved-begrenset-forventet-levetidSharpTMoranEKuhnIBarclayS. Do the elderly have a voice? Advance care planning discussions with frail and older individuals: a systematic literature review and narrative synthesis. Br J Gen Pract. (2013) 63(615):e657–68. 10.3399/bjgp13X67366724152480LundSRichardsonAMayC. Barriers to advance care planning at the end of life: an explanatory systematic review of implementation studies. PLoS One. (2015) 10(2):e0116629. 10.1371/journal.pone.011662925679395JimenezGTanWSVirkAKLowCKCarJHoAHY. Overview of systematic reviews of advance care planning: summary of evidence and global lessons. J Pain Symptom Manage. (2018) 56(3):436–59.e25. 10.1016/j.jpainsymman.2018.05.01629807158CombesSNicholsonCJGillettKNortonC. Implementing advance care planning with community-dwelling frail elders requires a system-wide approach: an integrative review applying a behaviour change model. Palliat Med. (2019) 33(7):743–56. 10.1177/026921631984580431057042McMahanRDTellezISudoreRL. Deconstructing the complexities of advance care planning outcomes: what do we know and where do we go? A scoping review. J Am Geriatr Soc. (2021) 69(1):234–44. 10.1111/jgs.1680132894787MalhotraCShafiqMBatcagan-AbuegAPM. What is the evidence for efficacy of advance care planning in improving patient outcomes? A systematic review of randomised controlled trials. BMJ Open. (2022) 12(7):e060201. 10.1136/bmjopen-2021-060201BondGRDrakeRE. Assessing the fidelity of evidence-based practices: history and current status of a standardized measurement methodology. Adm Policy Ment Health. (2020) 47(6):874–84. 10.1007/s10488-019-00991-631691055HestmarkLHeiervangKSPedersenRHanssonKMRuudTRomørenM. Family involvement practices for persons with psychotic disorders in community mental health centres - a cross-sectional fidelity-based study. BMC Psychiatry. (2021) 21(1):285. 10.1186/s12888-021-03300-434078306CarrollCPattersonMWoodSBoothARickJBalainS. A conceptual framework for implementation fidelity. Implement Sci. (2007) 2(1):40. 10.1186/1748-5908-2-4018053122VolandesAEZupancSNPaasche-OrlowMKLakinJRChangYBurnsEAAssociation of an advance care planning video and communication intervention with documentation of advance care planning among older adults: a nonrandomized controlled trial. JAMA Netw Open. (2022) 5(2):e220354. 10.1001/jamanetworkopen.2022.035435201306VaccaroLButowPNLeeDJohnsonSBBellMClaytonJFidelity is fundamental: intervention predictors in advance care plans in terminal cancer. BMJ Support Palliat Care. (2019) 9(4):397–403. 10.1136/bmjspcare-2019-00191731537578CraigPDieppePMacintyreSMichieSNazarethIPetticrewM. Developing and evaluating complex interventions: the new medical research council guidance. Br Med J. (2008) 337:a1655. 10.1136/bmj.a1655ProctorESilmereHRaghavanRHovmandPAaronsGBungerAOutcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. (2011) 38(2):65–76. 10.1007/s10488-010-0319-720957426RomørenMHermansenKBSævareidTJLBrøderudLWestbyeSFWahlAKImplementation of advance care planning in the routine care for acutely admitted patients in geriatric units: protocol for a cluster randomized controlled trial. BMC Health Serv Res. (2024) 24(1):220. 10.1186/s12913-024-10666-0ElmEAltmanDGEggerMPocockSJGøtzschePCVandenbrouckeJP. Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Br Med J. (2007) 335(7624):806–8. 10.1136/bmj.39335.541782.ADHopkinsSABentleyAPhillipsVBarclayS. Advance care plans and hospitalized frail older adults: a systematic review. BMJ Support Palliat Care. (2020) 10(2):164–74. 10.1136/bmjspcare-2019-00209332241957Sævareid. Advance Care Planning in Nursing Homes: A Mixed Method Study of a complex Intervention Using a Whole-ward approach. Oslo: University of Oslo (2019).PeckVValianiSTanuseputroPMulpuruSKyeremantengKFitzgibbonEAdvance care planning after hospital discharge: qualitative analysis of facilitators and barriers from patient interviews. BMC Palliat Care. (2018) 17(1):127. 10.1186/s12904-018-0379-030518345FienSPlunkettEFienCGreenawaySHeylandDKClarkJChallenges and facilitators in delivering optimal care at the End of life for older patients: a scoping review on the clinicians’ perspective. Aging Clin Exp Res. (2021) 33(10):2643–56. 10.1007/s40520-021-01816-z33713331CarterCLeanzaFMohammedSUpshurREGKontosP. A rapid scoping review of end-of-life conversations with frail older adults in Canada. Can Fam Physician. (2021) 67(11):e298–305. 10.46747/cfp.6711e29834772723NilsenP. Making sense of implementation theories, models and frameworks. Implement Sci. (2015) 10:53. 10.1186/s13012-015-0242-025895742Substance Abuse and Mental Health Services Administration. Integrated treatment for co-occurring disorders: evaluating your program. DHHS Pub. No. SMA-08-4366. Rockville, MD: Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, U.S. Department of Health and Human Services (2009). Available from:https://library.samhsa.gov/sites/default/files/ebp-kit-evaluating-your-program-10112019.pdf(Accessed October 16, 2025).DeteringKMSilveiraMJ. Facilitating advance care planning discussion (UpToDate). Available online at:https://www.uptodate.com/contents/advance-care-planning-and-advance-directives?search=advance%20care%20planning&source=search_result&selectedTitle=1∼150&usage_type=default&display_rank=1#H2094995 (Accessed 2020 1.2.2020).Norwegian Law, Patients’ Rights Act. 1999.GleesonANobleSMannM. Advance care planning for home health staff: a systematic review. BMJ Support Palliat Care. (2021) 11(2):209–16. 10.1136/bmjspcare-2018-00168030808626JainACorriveauSQuinnKGardhouseAVegasDBYouJJ. Video decision aids to assist with advance care planning: a systematic review and meta-analysis. BMJ Open. (2015) 5(6):e007491. 10.1136/bmjopen-2014-00749126109115ThoresenLLLSævareidTJLGjerbergEFørdeRPedersenR. Guide. Advance care planning (ACP) - planning for future health care and end-of-life care together University of Oslo (2015). Available online at:https://www.med.uio.no/helsam/english/research/projects/end-of-life-communication-in-nursing-homes/acp-guide070617.pdf(Accessed 2022 03.08.2022).AhmedM. Procedure for Advance Care Planning. Oslo: Oslo University Hospital (2020).Gold Standards Framework, Advance care planning (2019). Available online at:http://www.goldstandardsframework.org.uk/advance-care-planning(Accessed August 15, 2024).ReynoldsJCroftS. How to implement the gold standards framework to ensure continuity of care. Nurs Times. (2010) 106(32):10–3. PMID: 20879638SalkindN. Encyclopedia of Research Design. Thousand Oaks, California: SAGE Publications, Inc. (2010).HoldenRB. Face Validity in the Corsini Encyclopedia of Psychology. 4th ed. Hoboken, New Jersey: Wiley (2010).HestmarkLRomørenMHeiervangKSHanssonKMRuudTŠaltytė BenthJImplementation of guidelines on family involvement for persons with psychotic disorders (IFIP): a cluster randomised controlled trial. Adm Policy Ment Health. (2023) 50(3):520–33. 10.1007/s10488-023-01255-036797515MalhotraC. Reimagining and rebranding advance care planning. Age Ageing. (2024) 53(10):1–3. 10.1093/ageing/afae233Ihle-HansenHIhle-HansenHRostoftSMunkhaugenJWestbyeSFRomørenMAdvance care planning towards the end of life. Tidsskr Nor Laegeforen. (2024) 144(13):1040–1. (English, Norwegian). 10.4045/tidsskr.24.0105WestbyeSFRostoftSRomørenMThoresenLWahlAKPedersenR. Barriers and facilitators to implementing advance care planning in naïve contexts - where to look when plowing new terrain?BMC Geriatr. (2023) 23(1):387. 10.1186/s12877-023-04060-437353744KeLSHuangXO'ConnorMLeeS. Nurses’ views regarding implementing advance care planning for older people: a systematic review and synthesis of qualitative studies. J Clin Nurs. (2015) 24(15–16):2057–73. 10.1111/jocn.1285325940451FrechmanEDietrichMSWaldenRLMaxwellCA. Exploring the uptake of advance care planning in older adults. An integrative review. J Pain Symptom Manage. (2020) 60(6):1208–22.e59. 10.1016/j.jpainsymman.2020.06.04332645455AwDHayhoeBSmajdorABowkerLKConroySPMyintPK. Advance care planning and the older patient. Qjm. (2012) 105(3):225–30. 10.1093/qjmed/hcr20922075012BondGRDrakeREMcHugoGJRappCAWhitleyR. Strategies for improving fidelity in the national evidence-based practices project. Res Soc Work Pract. (2009) 19(5):569–81. 10.1177/1049731509335531GabbardJPajewskiNMCallahanKEDharodAFoleyKLFerrisKEffectiveness of a nurse-led multidisciplinary intervention vs usual care on advance care planning for vulnerable older adults in an accountable care organization: a randomized clinical trial. JAMA Intern Med. (2021) 181(3):361–9. 10.1001/jamainternmed.2020.595033427851