Medical professionalism (MP) exemplifies a set of values, beliefs, and attitudes that underpin the public’s trust in the profession of medical professionals in society (1, 2). MP is fundamental to modern medical education as it encompasses conduct, ethics, and professional identity formation, ensuring professional integrity and competence (3). Teaching MP using student-centered pedagogical approaches, such as team-based learning with real-world examples, helps bridge the gap between theory and practice (1). Professional behaviors (PBs) are the manifestations of these traits in the form of actions (4). These actions span various facets ranging from confidentiality maintenance to cultural sensitivity, ability to raise concerns, and practicing self-care, among others (1).

The aim of modern medical education is not only to impart knowledge but also to inspire and assess meaningful changes in PBs (4, 5). Published literature critiques that PB excellence is not at par with medical knowledge proficiency, highlighting areas such as MP, where improvements are needed (4). Hence, measuring changes in PBs after teaching MP is essential to assess the effectiveness of educational efforts (6). To objectively measure the impact of these educational efforts and to determine whether healthcare professionals (HCPs) can exhibit the desired changes in PB, a rigorous assessment of such MP interventions is vital. It is recommended that these interventions have a theoretical underpinning. Theory in research can help to guide intervention development, predict behaviors, and explain what works and does not work in an intervention (7). This integral process encompasses the application of underlying behavioral theories (e.g., sociological and psychological theories) and educational learning theories (e.g., constructivism theories), a careful and reflective curriculum design, and the utilization of questionnaires to gage the extent of these changes (8, 9).

Designing an effective questionnaire requires a deep understanding of the behavior change model and its constructs. Data collection questionnaires are frequently used to probe participants’ cognitive and social processes within social, behavioral, and psychological sciences (10). The theory of planned behavior (TPB) underpins the pedagogy of MP by emphasizing the interplay between TPB domains (i.e., attitudes, subjective norms, and perceived behavioral control), making them a model to understand, predict, and induce changes in PB (11). TPB is a social psychological theory, widely used to predict human intentions which are precursors of behaviors (12, 13). Despite the fact that there are two detailed guides for developing TPB questionnaires (14, 15), the published literature is critical of ambiguity and confusion in the statements of the TPB-based questionnaire, which undermine the validity and reliability of data collected (16, 17).

Essential strategies to develop a robust questionnaire involve the use of clear questions, unbiased language, and relevant content to ensure accurate responses. To do that, it must meet rigorous scientific criteria through psychometric evaluations to assure reliability and validity, answering potential objections of questionnaire-based research as “soft science” (18). This can be further achieved with piloting to intensify the robustness of results. Constructing and validating a questionnaire using theoretical underpinning is a meticulous process that ensures the instrument aligns with the chosen behavioral theories. Prior to implementing questionnaire-based interventions in predictive studies, it is imperative to ensure that the questionnaire is rigorously developed and validated.

To overcome the challenges in the validity and reliability of the TPB questionnaire, researchers must tailor its statements based on aspects of behaviors. A recent study (19) coined the Target, Action, Context, and Time (TACT) principle with the potential to mitigate challenges in questionnaire construction. While TPB and TACT principles provide guidance, the role of curriculum development and curated resource selection cannot be ignored. Thus, we aimed to validate a bespoke questionnaire, modified from Medisauskaite et al. (20), to measure the change in PB; the TPB was adopted to examine a change in intentions by looking into attitudes, subjective norms, and perceived behavioral controls of medical students (21).

Ten experts, comprising of medical educators and clinicians, were involved in the content validity and reliability assessment. The number of experts was determined according to Polit’s research to avoid the coincidence of arguments (22). The validity and reliability were performed during the months of November and December 2022 after obtaining informed consent from the experts.

This paper sought to present the methodological process of modifying and validating a TPB questionnaire (through quantitative and qualitative approaches) to allow valid assessment of specific MP domains covered in an educational intervention, in our case the PROPER study. The PROPER intervention, comprising a series of expert-designed online workshops for pre-clinical undergraduate medical students, plans to focus on four key domains of importance to medical students and the hidden curriculum, namely, confidentiality, raising concerns, self-care/wellbeing, and cultural sensitivity. Importantly, the methodology used in this study could be adapted by educators to assess the impact of educational interventions in a number of other domains. Clarity and relevance are closely intertwined with these concerns and their analysis can help ensure that our items are explicit, contextually accurate, and intricately aligned with the construct of interest (23). The I-CVI scores for relevance ranged from 0.9 to 1, signifying that all items were considered highly relevant by the panelists (25). The I-CVI calculated for each item within the questionnaire was pertinent to the construct being measured (23). While clarity items ranged between 0.7 and 1, indicating areas for development (25). Three items within the attitude construct indicated a fair clarity (I-CVI of 0.7) (26). However, Brod et al. (30) recommend that a minimum score of I-CVI should be ≥0.78 if the number of panelists is six or more. Despite the S-CVI/Ave for clarity was 0.9, signaling that most items were rated as clear, there was room for improvement for items in the attitudes construct. This suggests that, in general, the items effectively measure the intended constructs and are well accepted by expert panelists in terms of relevance and clarity (25, 27, 31).

Clarity is vital for ensuring the quality and validity of research findings and for accurately measuring the factors that influence behavioral intentions (32). To further assess the degree of agreement among panelists regarding clarity, the kappa statistics for these items indicated “fair” agreement (29). This highlighted the concerns related to clarity pointing to ambiguity and confusion in the constructed questionnaire for the PROPER study. Such a finding is expected in questionnaire development, as some constructs may inherently involve complex or multifaceted concepts that are difficult to express clearly and explicitly (32).

A thorough qualitative feedback provided a deeper understanding of the questionnaire items and identified potential areas for improvement in terms of clarity. Regarding the construct of subjective norms, experts rightly pointed out how diverse social and cultural settings, context, the rising importance of social media, and the concept of personal and professional identity influence individuals’ subjectivity and decision-making (33). Recognizing this issue, we addressed the clarity of these items to ensure that potential PROPER research participants could accurately and uniformly interpret the statements. The “choice of phrase” and “reference to the context” highlight the need for precise and context-specific wording in questionnaire items. In addition, the suggestion to create a “distinct separation of personal and professional self” resonates with the complexities of TPB constructs, particularly in healthcare and medical contexts. This highlights the nuanced role of subjective norms in personal and professional identity formation (34). This differentiation is fundamental in healthcare, especially when novice HCPs on the outskirts of professional communities of practices splint and patch their fledgling professional identities within the context of their personal roles (35). Moreover, the proposal to include “social media” as an additional source of subjective norms recognizes the evolving social landscape and the impact of digital communication on individuals’ behaviors and attitudes (34). This suggests considering the dynamic change of social influence and how they can be incorporated into our TPB-based questionnaires, which can lead to the specification of social media in the subjective norms construct. Such ambiguous statements were clearly separated to avoid indefinite boundaries.

To enhance the predictive power of TPB-based questionnaires, questions should be behavior-specific and situated within a particular context. This aspect was well grounded by the panelists. Using the TACT principle (19), it is imperative to construct questionnaire items to ensure situational context, which could enable participants to relate to the significance of PB. By acknowledging a diverse range of contextual expertise of the panelists, we contextualized the domain of maintaining confidentiality in the “daily placement of a medical student.” Underpinning the TACT principle (15), we improved our TPB questionnaire which aims to target undergraduate medical students to measure their change in PB by applying confidentiality principles in their daily context of medical placements in a pre–post and delayed post time frame.

A wealth of published literature on TPB has discussed that the subjective norm construct is a less effective predictor compared to others such as attitude and perceived behavioral control for PB in the healthcare context (16, 17). This may be particularly due to the problem participants encountered with subjective norm items. These items assess the degree to which an individual is inclined to align his actions with the perceived expectations. To address this, we respected the normative beliefs of our panelists and brought a sharp distinction between personal and professional self. This supported the advantages of capturing the normative beliefs of our PROPER study participants if they hold a personal or professional self-significance in their professional practice.

The applied methodology that is tailored to the PROPER study is theory based with a robust validation process consisting of diverse panelists. However, a diverse sample of experts with a varied understanding of TPB may have led to individual interpretations and inherent subjectivity while validating the items. Second, the absence of a pilot test with a representative sample could have refined the clarity and relevance of statements and enhanced the predictive validity of the questionnaire of the PROPER study (22, 23). However, a robust quantitative and qualitative process of validation underlines the strength of our modified questionnaire.

This article details the methodological process conducted to validate a modified TPB-based questionnaire with the goal of ensuring its efficacy in accurately measuring TPB constructs. This process may be able to measure changes in PBs across a range of MP-based educational interventions in diverse medical institutions. By integrating qualitative insights and highlighting its clarity and relevance, this validated questionnaire may enable accurate measurement of TPB constructs in the PROPER study. The utilization of such practices may benefit medical students by identifying the facilitators and barriers to PBs and may also improve patient care and experiences through targeted educational interventions.

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

The studies involving humans were approved by Board Royal College of Surgeons Ireland Dublin (REC202205007) and Queens’ University Belfast (MHLS 22_184). The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

SSG: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Visualization, Writing – original draft, Writing – review & editing. EC: Data curation, Investigation, Project administration, Resources, Visualization, Writing – review & editing. AS: Methodology, Resources, Validation, Visualization, Writing – review & editing. MS: Formal analysis, Project administration, Resources, Visualization, Writing – review & editing. SH: Formal analysis, Project administration, Resources, Visualization, Writing – review & editing. FD: Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing. GK: Investigation, Methodology, Project administration, Resources, Validation, Visualization, Writing – review & editing. MH: Investigation, Methodology, Project administration, Resources, Visualization, Writing – review & editing. AR: Investigation, Project administration, Resources, Visualization, Writing – review & editing. FB: Investigation, Methodology, Project administration, Resources, Visualization, Writing – review & editing. AB: Investigation, Project administration, Resources, Validation, Visualization, Writing – review & editing. SYG: Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing. DH: Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing.