The main objective of the POPP study is to evaluate the outcome and process of psychotherapies amidst the complexity and fluidity of naturalistic conditions over time at varying lengths. A distinct emphasis is placed on examining the therapeutic relationship, focusing on the collaborative and trustful bond between a therapist and a patient (the alliance), and gender dynamics. Importantly, the study also includes data from patients who end their therapy prematurely for various reasons, offering an inclusive and thorough perspective on the psychotherapeutic process. In particular, the POPP study will answer the following research questions:

It is crucial to clarify that the POPP study does not aim to juxtapose the efficacy of the four theoretical orientations or their numerous therapeutic modalities. Due to the inherent design of a naturalistic outcome study, there is no control group, and participants are not assigned randomly to different treatment modalities. Consequently, there might be intrinsic variations amongst individuals who opt for different therapeutic modalities. These variations impede direct comparisons (44).

The POPP study, designed to reflect the prevalent psychotherapy conditions in Austria, offers an authentic insight into the field. Austria, an early adopter of psychotherapy legislation, enacted a law in 1991 (BGBL 361/1990) defining psychotherapy as a scientifically-based treatment for behavioural disorders and psychological suffering on a psychosocial or psychosomatic basis (45). The study operates within Austria’s mental health system, featuring 23 accredited psychotherapeutic modalities categorised into four broader theoretical orientations: humanistic, psychodynamic, systemic, and behavioural. As of 2021 (46), the distribution of psychotherapists according to their theoretical orientation was as follows: 38% practised Humanistic therapy, 26% were Psychodynamic therapists, 24% practised Systemic therapy, and 12% were Behavioural therapists.

The Austrian model exemplifies an uncommonly diverse blend on a global scale and sets Austria apart from regions dominated by only a few mainstream modalities, such as psychoanalysis and behavioural therapy. Notably, this model provides an array of entry pathways and multiple training opportunities in psychotherapy (47, 48). The POPP study allows us a closer look into, and a deeper understanding of, the particular characteristics of Austria’s unique psychotherapy model.

Launched in 2020, the POPP project utilises a sophisticated mixed-methods research design, synergising quantitative and qualitative methodologies. It involves a wide-ranging participant pool, encompassing patients with various and combined symptoms and undergoing treatment durations ranging from as brief as three sessions to as extended as four years. Furthermore, the study aims to incorporate these patients’ corresponding psychotherapists to align and compare their data.

The quantitative part of the study involves pre- and post-therapy online surveys taken by patients and their therapists. Both groups are also requested to assess the therapeutic alliance after each session, enabling real-time process monitoring. Therapists document their cases using a specific case report format.

The qualitative part of the study encompasses interviews with patients and their respective psychotherapists. These interviews occur randomly at several therapy stages, either at the initial, the mid-point, or the concluding phases, whereby individuals are only asked once during the process. These interviews, guided by a semi-structured interview guide, probe deeply into individual experiences.

During the analysis process, quantitative and qualitative data forms corroborate each other. The qualitative data contribute valuable insights into the psychosocial subtleties and dynamics of therapy that might potentially elude the quantitative data. The quantitative analysis will primarily scrutinise the relationship between the therapeutic alliance and therapy outcomes across different gender pairings, considering the perspectives of both patients and therapists. Furthermore, the qualitative data, in an independent capacity, will shed light on the subjective experiences of patients and therapists alike.

This research project represents a collaborative endeavour involving the University of Continuing Education Krems (UWK) and the Karl Landsteiner Private University for Health Sciences in Krems. Research approval for the study was obtained from the Data Protection Office of the University of Continuing Education Krems (UWK) as well as from the Ethics Committee of the University of Continuing Education Krems (UWK), Austria (EK GZ 28/2018-2021). The timeline of the study can be accessed in Figure 1 .

The success of the POPP study hinges on psychotherapists actively participating and being ready to include two to three of their patients. Psychotherapists eligible for involvement must be officially licensed and listed in the Austrian Federal Ministry for Social Affairs, Health Care, and Consumer Protection’s directory, demonstrating their qualification to provide individual outpatient psychotherapy to adult clients in their psychotherapy practice; group therapies are not incorporated in this study.

To ensure a broad and representative sample of patients, psychotherapists are instructed to consecutively invite all their patients in the nascent therapy stages between their first and third sessions to partake in the study. They should invite on a consecutive basis without bias towards the severity, type of symptoms, or personal preferences. Psychotherapists must record the reasons if an invited patient either declines participation or is not extended an invitation, thereby aiding in detecting recruitment biases.

The study aimed to register a minimum of 100 therapists and 300 patients to secure representation across all four theoretical orientations and the 23 therapeutic modalities recognised in Austria. In the spirit of collaboration and inclusive research, all professional therapeutic associations in Austria have been invited by our research team to contribute a minimum of ten psychotherapists each, to facilitate individual data evaluation of their respective therapeutic modalities.

Numerous professional psychotherapy associations, including the Austrian Federal Association for Psychotherapy (ÖBVP), assist with recruitment by disseminating information via emails, newsletters, and social media. The research team has also briefed therapists at various scientific meetings and congresses.

A central aspect of our recruitment strategy involves arranging multiple online information workshops throughout the year. These workshops aim to demystify the research design, clarify the role of the psychotherapists, and provide information about incentives. Psychotherapists who choose to participate in the study will receive a certification of five training hours per patient per year, which will count towards the obligatory 150 hours of further training over a five-year period, as mandated by Austrian law.

This study presents a unique opportunity for psychotherapists to contribute to the research community, examine and potentially improve their therapeutic practice by reflecting on their therapeutic alliance after each session, and reinforce the effectiveness of their therapeutic modalities, especially those currently lacking substantial empirical support.

Psychotherapists interested in participating in the study are encouraged to contact our research team for further information and workshop dates. Basic information about the study is also available online, hosted by the Karl Landsteiner Private University ¹ . The dates and the link to the online info workshops are also provided via a Facebook site ² .

Upon contacting the research team via the designated email address to request study materials, psychotherapists receive comprehensive informational resources in digital format or as printed materials. These resources contain thorough details about the study, handouts intended for the patients, and case report forms. Patients are then invited by their therapists to participate in the study and provided with these informational materials.

The psychotherapist’s and the patient’s materials include secure hyperlinks to the Research Electronic Data Capture (REDcap) platform (49), where they can complete quantitative surveys after granting their electronically informed consent. Patients interested in contributing to the study’s qualitative interviews can sign up through a separate online contact form.

Despite our extensive efforts and incentives to motivate psychotherapists to participate in the study, the recruitment process has emerged as the most significant challenge, as can be seen in Figure 2 . By now, we have acquired data from 93 therapists and 173 patients, and we are far from realising the goal of securing a minimum of 10 psychotherapists for each therapeutic modality. The study’s conclusion, initially scheduled for 2023, has been deferred to December 2024 due to the slow pace of data collection. This means we included new psychotherapists in the study until mid-2023 and collect data until the end of 2024. The therapy, of course, goes on even after the study is finished. Most psychotherapists were admitted in the first year, and the average psychotherapy time in private practice of 2-4 years was considered in our planning.

Contrary to the difficulties experienced in the broader recruitment for the study, soliciting qualitative interviews has been noteworthy successful. Over the course of two waves carried out in the summers of 2021 and 2022, we have successfully completed 43 patient interviews and 41 psychotherapist interviews. During the initial wave, 6 students conducted interviews with 21 patients and 17 therapists. This was followed by a second wave, where an additional 8 students interviewed 22 patients and another 17 therapists. Notably, a subset of therapists was interviewed twice, with each occasion focusing on a distinct patient.

The comprehensive dataset thus compiled is particularly impressive, given that these interactions occurred amidst the constraints posed by the COVID-19 pandemic.

While we have collected over 3,452 process surveys from therapists until today (January 2024), patient submissions amount to 1,990. Missing process survey data from patients underscores the importance of having a simple and user-friendly process for data collection. Furthermore, we assume that a stronger commitment to the study on the part of the therapists would also have led them to regularly remind their patients to enter the data for the process survey. This happened too little.

One frequent difficulty encountered is patients’ inconsistent usage of codes when they login to complete the questionnaires compared to when the interviewers contact them via phone. This inconsistency hampers the data merging process. We partly ascribe these issues to the pathology patients grapple with. This pathology also introduced challenges for the interviewers, as voiced by one of them during a focus group:

However, there have been instances where psychotherapists failed to use consistent codes as well, indicating that mental pathology cannot be the sole reason for such confusion.

Working with student researchers presented great advantages and only minor challenges due to their diverse experiences with scientific research and qualitative methodologies. While many have social or health sciences backgrounds, others have never worked in a scientific field, impacting the interview process and data quality. However, due to their psychotherapy training, these student researchers are adept at conversational techniques. They have learned to form a strong rapport with their counterparts and reflect on the information they receive. All of them were already working with patients under supervision, which positively influences the openness of interviewees and the depth of conversation. However, during focus groups, it became apparent that students often struggled to differentiate between their roles as therapeutic interviewers and researchers. They found it challenging to disengage from a therapeutic conversation and follow the interview guide.

During a focus group, one researcher expressed,

A recurrent challenge faced by our student researchers pertains to the expansive geographical scope of participant recruitment, encompassing all regions of Austria. As a result, students often needed to travel significant distances to interview patients and psychotherapists. Despite our endeavours to align the students’ places of residence with the locations of our study participants, such coordination was not always feasible.

Even though Austrian law requires a scientific approach to psychotherapy, a 2018 study found Austrian therapists to be more resistant to process and outcome monitoring than their counterparts in Belgium, the UK, and Australia. These therapists expressed apprehension about complex procedures, the perceived minimal treatment benefits, and potentially detrimental effects on the therapist-patient relationship (68). As we already mentioned in the introduction, similar problems were encountered in previous psychotherapy research efforts (15).

Despite our comprehensive and strategic efforts, which encompass refining our research approach, dedicating significant human resources, and actively involving stakeholders (7) in the context of the POPP study, we persistently encounter challenges that impede our recruitment process. Throughout the series of 22 informational workshops from October 2020 to June 2023, we have identified several concerns expressed by potential participants that could discourage their involvement:

Concerns about the study have been notably focused on data protection and potential disruption of the therapeutic process. Therapists worry about guaranteeing their patients’ confidentiality and the possible negative effects of ongoing data collection, particularly post-session questionnaires, on their therapeutic relationships. They also expressed concerns about interpreting individual data and gauging their therapeutic efficacy on a personal level.

There were also concerns about the strain filling out questionnaires and process monitoring puts on patients.

The complexity of the study information, as necessitated by data protection regulations, emerged as a further barrier to participation. Therapists expressed doubts about their patients’ capacity to understand such intricate details. Furthermore, the study’s complex design was occasionally misconstrued as a randomised controlled trial (RCT), evoking apprehensions about reducing psychotherapeutic processes to symptom-oriented constructs. Particularly, therapists from humanistic modalities emphasised their focus on less tangible aspects like personality development and enhancement of self-perception, contending that these elements could not be accurately captured through standardised questionnaires. In this vein, previous research on the attitudes of Austrian psychotherapists on process and outcome monitoring has found that cognitive-behavioural therapists generally held favourable views; therapists employing humanistic-existential modalities, on the other hand, expressed negative attitudes (68). This attitude is anchored in broader, epistemological objections to psychology’s characterisation as a quantitative science and its aspiration to produce “objective” explanations for psychological phenomena (69). Despite these findings, however, most participating psychotherapists belong to humanistic psychotherapy modalities.

Despite our consistent efforts during the workshops to clarify that our study did not aim to compare the effectiveness of various therapeutic modalities, we probably could not alleviate these concerns sufficiently. Given Austria’s endorsement of 23 psychotherapy modalities and the upcoming revision of the Psychotherapists Act, there is a prevalent sense of unease about potential regulatory changes. Particularly, smaller professional associations expressed fears that their modalities may not be deemed sufficient, thus jeopardising their future inclusion. This, coupled with a perceived sense of inferiority against larger, mainstream approaches such as behavioural therapy, likely dissuades therapists from participating in research more broadly.

We think the efforts to explain our study design during workshops were largely unsuccessful due to a widespread unfamiliarity with the concept of scientific research. As pointed out by Castonguay et al. (2014), many clinicians had not had the opportunity to work with a supervisor who was both a practising psychotherapist and a researcher. This gap in early career training means they lacked exposure to a role model who could identify the most appropriate research methods to answer the most relevant research questions (10).

The intensive documentation needed for participation was another deterrent, in line with findings from Laireiter and Kopetzky (2014) (15).

Some therapists expressed challenges in persuading new patients to join the study. They felt it was inappropriate to ask patients to contribute additional effort for the sake of psychotherapy, especially since the patients were already paying for the therapists’ services. However, therapists who firmly believed in the study’s value reported facing no issues enlisting patients. This indicates that the therapists’ personal hesitations might serve as a barrier to motivating patient involvement in the study.

Qualitative data collection proceeded more smoothly than quantitative data, indicating less therapist resistance. This inference comes from the noticeable increase in qualitative interview registration following email communication urging therapists to encourage their patients to participate in qualitative interviews.

Finally, unlike many therapists, patients probably did not share the same apprehensions concerning data protection, the complexities of the study’s framework, or the affiliated duties. Additionally, they value the distinctive introspective possibility offered by these personal interviews. Almost unanimously, every patient approached by our student interviewers for a deeper explanation regarding qualitative interview participation (after registration for the qualitative survey via RedCap) agreed to participate, with only a few exceptions.

The issue with patients and therapists using different codes when logging in and during interviews indicates a need for clearer communication and an easier procedure for creating a patient code.

The challenges interviewers face, as reflected in the focus group sessions, highlight the need for support systems for the interview team. Handling challenging situations, such as deciphering whether a patient is speaking in metaphors or experiencing psychosis, can be draining. Since our student interviewers are being trained as future psychotherapists, they did handle such situations well. Moreover, since we provided regular sessions for reflection and support via email and telephone, they could address pressing issues immediately.

From the outset, the project prioritised data quality, aiming to build a robust database to facilitate a wide array of future research inquiries. Several quality assurance strategies have been implemented to ensure this high-quality data, including:

Standardisation of the management of qualitative data proved to be advantageous. Given the complexity of a longitudinal mixed-methods dataset, understanding the data structure, familiarising oneself with documentation routines, and tracking documents for deposition and their scheduled dates can be challenging. We thus recommend having at least one person oversee data management, processing, and archiving (70).

Certain limitations should be acknowledged. Despite comprehensive efforts to enhance the feasibility of the POPP study, certain limitations impacting the recruitment process persist. The resistance among Austrian psychotherapists, influenced by concerns about data protection, potential disruption of the therapeutic process, and the strain on patients, poses a significant feasibility challenge. The complexity of study information, particularly related to data protection regulations, further hampers participant understanding and engagement. Efforts to clarify the study design during workshops faced difficulties due to widespread unfamiliarity with scientific research concepts among therapists. The study’s sample was comprised only of therapists interested in the study, thus limiting its generalizability. Although therapists were advised to inform all consecutive patients about the study, adherence to this protocol couldn’t be verified. Hence, the possibility of therapists inviting only a specific subset of patients, such as those with less severe illnesses, cannot be discounted, which could potentially limit patient-side generalizability. Further limitations include missing data, especially among patients. While the baseline assessments experienced the least amount of missing data, there was a noticeable increase in missing data in the session-based questionnaires and those conducted post-therapy. Although we periodically send emails to therapists, requesting them to remind their patients about completing the questionnaires during the course of therapy, a significant amount of data will be missing, which needs to be handled appropriately based on the final percentage of missing data as well as the distribution of missing data. Possible inaccuracies due to the use of erroneous codes by patients or therapists represent an additional limitation, which could affect the interpretation of study results.

Another limitation regards the collection of qualitative interviews. Our objective is to gather qualitative interviews from all stages of therapy (beginning, middle, and end). However, realising this goal presents certain challenges. A key limitation is the unpredictability of each patient’s therapy duration. This uncertainty is evident when conducting interviews, as we cannot foresee the total length of the therapy process for each patient. Additionally, the process of conducting interviews is time-intensive, and given our limited personnel resources, it becomes impractical to accurately manage the distribution of interviews across different therapy phases. Therefore, these factors combined hinder our ability to evenly distribute the collection of interviews throughout the various stages of therapy. Consequently, we observe a predominance of interviews originating from the middle phases of therapy. This imbalance restricts our ability to compare experiential patterns characteristic of early therapy stages, late stages and processes with premature termination, limiting the scope of our analysis.

A notable limitation arises in the collection of consistent outcome measurements during the practice study, primarily stemming from the challenges posed by varying start times and the decentralised locations of patients. The nature of real-world practice settings introduces a level of unpredictability, making it difficult to ensure uniformity in the timing of outcome measurements across different treatment trajectories.