The assessment of clinical, communication, and practical skills is an important component of health professions education for both feedback provision and informing progress decisions. Learners are placed and observed in a variety of settings—teaching facilities, simulation centers, and “real world” practice—to ensure that learning outcomes are achieved. Expectations of learners evolve through programs, with the focus changing from individual components (e.g., taking a history, measuring blood pressure, examining a body system) to more integrated comprehensive tasks that require focus on presentations, diagnostic reasoning, and management plans. Marking sheets may include checklists, rating scales for individual items or competencies, global scores and free-text comments. Assessors are a diverse group, including people in various combinations of academic and clinical roles, in different clinical specialties, and with different levels of experience as clinicians and assessors. Their assessments often rely on relatively brief observations of performance to inform judgements, drawing on both stated learning outcomes and their own experiences and expectations of good clinical performance (1).

Significant variability in clinical assessor judgements has been reported (2–5). However, variations often persist despite assessor training and standardized station design (6–8), raising concerns about cognitive bias in assessor judgments. Cognitive biases, also known as “heuristics,” are cognitive short cuts used to aid our decision-making (9). Studies have shown that clinical decision-makers are at risk of error due to bias but often lack insight into their own biases (10). There are various causes of bias, and these include learned or innate biases, social and cultural biases, and environmental stimuli (11). This highlights a need for greater understanding of the cognitive processes of clinical assessors to inform strategies that enhance fair and robust judgements. Our previous research showed that judging candidate performance is complex, cognitively challenging and mentally demanding, particularly when borderline performance is observed in an “exit” Objective Structured Clinical Examination (OSCE) (12). In this “grey” zone of candidate performance, assessors used academic institutional marking criteria as a “safety blanket” to guide judgement, but also used additional criteria that were not necessarily explicit in the marking sheet, based on professional expectations (candidate demeanor and patient safety). The emergence of the concept of a “prototypical” intern (in Australia and New Zealand, this is a first postgraduate year medical graduate working under supervision in teaching hospitals) suggested that calibration was guided by a rapid, internal cognitive process based on a mental construct of assessors' expectations of a new medical graduate working in their clinical team.

In educational psychology this construct is known as a heuristic cognitive process, conscious or unconscious, whereby “rule of thumb” judgement decisions are made, possibly neglecting some presented information (13–15). This mental “shortcut” is faster and reduces cognitive complexity when working memory is overloaded by time pressure or increased complexity and is also more likely in experienced assessors who recognize patterns more quickly (14, 16, 17). Applying this concept to our previous study, assessors appeared to use a representativeness heuristic to consider “how much does the observed clinical performance of a senior medical student compare with what I expect of a ‘prototypical’ intern” (12)? Such representational heuristics may be influenced by assessors' roles and experiences, contrast effects, use of inference, working memory effects, different interpretations of behaviors, predisposition to consider a particular perspective (e.g., of the learner or patient), different pre-existing frames of reference, exposure to different learner cohorts, and the examiners' own clinical skills and perceptions of task difficulty (18–21).

So far, exploratory studies on assessor cognition have focused mainly on workplace-based assessments (WBAs), usually involving learners performing authentic tasks in clinical settings (5, 22–25). These studies consistently find that assessor judgements are complex (5, 23, 25–27). Examiner decision-making in OSCEs has been less researched, yet OSCEs remain a popular clinical assessment format despite the trend toward WBA and the challenges imposed by the recent COVID-19 pandemic. Whereas, WBA requires assessors to interpret the clinical case and set specific expectations within a marking framework, OSCE assessor judgements are guided by prescribed expectations and scoring criteria provided via the mark sheet by educators for more time-limited, standardized and pre-scripted encounters. This may require assessors to adapt their expectations of learner performance according to situational constraints. This paper reports a further exploration of the cognitive processes of exit level OSCE assessors with a primary aim of exploring how the prototypical intern concept influences clinical assessors' judgements in senior medical student OSCE. The secondary aim was to explore the complexity of making OSCE assessment decisions amidst potentially contradicting pressures from academic and clinical perspectives. Two specific research questions have been developed to address these aims—(1) How does the prototypical intern concept influence clinical assessors' judgements in senior medical student OSCE? (2) how do OSCE assessors balance academic (focussing on achieving graduate outcomes) and professional (being suitable for work in a clinical team) expectations when assessing senior medical students?

This study uses the CHAT as the theoretical framework to contribute new knowledge to understanding the decision-making processes of assessors during clinical assessment of candidates at a major professional transition point—graduating from a primary medical degree course to entering the workforce. Clinical assessors make judgements that combine, to a varying extent, two interacting sets of roles and experiences that have different origins. The academic construct is the achievement of agreed, expected, graduate learning outcomes; in contrast, the clinical workplace construct is the ability to successfully work within professional settings. The former is reliant on knowledge, skills and behaviors represented in marking sheets in several scheduled, controlled assessment events, one of which is the OSCE. Academic progress decisions are informed by combining assessment data from many assessment events, converting a series of “snapshots” to a “low frame-rate” moving image. The clinical workplace construct is reliant on respect, organization, reliability, teamwork, and trustworthiness exhibited over time. Clinical assessors form impressions of what constitutes a “safe” junior doctor, a “prototypical intern,” through personal clinical experience of working with graduates. These impressions may be shared by close colleagues but different to those in other fields of health care, producing a form of cognitive bias that is likely based on a combination of “signal” and “noise.” This may provide an implicit set of “rules” that reflects “noise” and necessitates adaptation of the academic marksheet. When candidate performance is “borderline,” time is pressured or the assessor is more experienced, this characteristic heuristic, “how would this person fit into my clinical team?” is likely to be relied upon as a benchmark, either consciously or unconsciously, merging rules and influencing resulting judgements. The application of CHAT explained the intricacies and contradictions that may occur within OSCE assessor decision-making.

This study explains how assessors in certain circumstances fall back on the clinical workplace experience to influence judgements about candidates. The extent of the influence may depend on the balance of academic and clinical experiences of individual assessors. Predominantly, clinical assessors may respect the academic rules, but over-ride them or merge them with clinically informed rules. This may explain why some assessors add components to checklists, add data points to rating scales and provide negative feedback to faculty on marksheets content. Another position may be that the more academic and less clinically relevant the marksheet, the more assessors may follow clinical logic in making decisions on clinical performance—overriding the marksheet criteria. The dominance of one or other activity system may also explain why examiner training appears not to be effective and why “doves” and “hawks” are difficult to change. “Debiasing” training is becoming popular to improve clinical reasoning, but success may be elusive (61). It may also explain the popularity among clinicians of the concept of Entrustable Professional Activities (EPAs), which adopts a more holistic view of performance (62). Of particular interest is that assessors strongly held the notion that clinical performance was more than just the sum of its parts, and that professional behaviors and identity (e.g., trustworthiness, safety, and reliability) were the most highly valued attributes in candidates—over and above just the taught content. Further, assessors believed that these attributes could be judged in an OSCE station. While our participants framed their “prototypical intern” characteristics around some relatively objective professional traits such as safety, other traits e.g., trustworthiness introduce the possibility of an underlying source of cognitive bias. Halo effects derived from superficial perceptions of the similarity of candidates to assessors may be able to falsely inflate feelings of trustworthiness and thus increase the inter-rater variability when using the construct. Additionally, the assessors were seeking and interpreting performance evidence based on their existing beliefs and expectations within the heuristic of the “prototypical intern” during the OSCE, implying confirmation bias (63). Future research exploring the finer details of the individual components of the prototypical intern construct e.g., what constitutes “trustworthiness,” would provide greater insights into these potential sources of bias.

The relevance of this theory to other forms of clinical assessment, or clinical assessment at course levels other than the graduating cohort, cannot be determined from this study. The analogy may be stronger for exit assessments for other health professionals, where clinicians are assessors in an OSCE-like event. The model may be less relevant to WBA, where the clinical workplace activity system is likely to dominate, even though relatively low failure rates and the “failure to fail” phenomenon have been reported (64). Although the prototypical intern concept may be less relevant to OSCE than WBA because of standardization of encounters and marksheets, acknowledging, and discussing prototypical intern qualities in standard setting may help drive fairer assessment. We believe that the contradictions have not been resolved, but rather clarified, potentially assisting further research. There may not be a “final” theory, but a clear message emerges: assessor judgements balance potentially conflicting perspectives that should be acknowledged and discussed in assessor training, standard setting, and calibration. Future research should investigate the identified contradictions.

There are implications for both marksheet design and assessor training. Marking rubrics may reflect what learners are “taught” but not necessarily the expectations of their imminent clinical service roles, suggesting a potential disconnect between achieving program learning outcomes and working in the clinical environment. The OSCE may currently present patient care as a set of individual tasks, whereas healthcare is being conceptualized increasingly as a team activity (65). Would marksheets that are more aligned with clinician constructs improve utility and compliance amongst predominantly clinical assessors? Should assessor training always include group discussion of how academic and clinical workplace constructs align as part of a more explicit “de-biasing” exercise? Should improving the fidelity of OSCEs to better reflect interprofessional healthcare teams not be possible, OSCEs may be better used as assessment “hurdles” that complement an increased emphasis on workplace-based assessment methods. These findings support calls for a review of the role of the OSCE as a clinical assessment tool (52).

Less experienced assessors were less represented because they were more difficult to recruit to interview sessions due to less flexible workload in their clinical service and specialty training requirements and a preference by medical schools for utilizing experienced assessors where available. Therefore, the perspectives of less experienced assessors may be under-represented. “Volunteer bias” is also possible, where assessors who follow medical school examiner training guidelines volunteered to be a part of this study—and “rogue examiners” did not volunteer. Hence, the statement that assessors usually followed institutional rules and guidelines for the OSCE may be overrepresented in this work. Additionally, the authors are insider researchers, and this could serve as potential bias, however, trustworthiness and credibility of the findings were enhanced through member checking and analytical group confirmations.

Clinical assessment judgements in exit-level medical school OSCEs are complex, with individual assessors balancing perspectives from two different but interacting constructs that overlap and compete. The first is that of the academic system which is more task-oriented, emphasizing knowledge, skills and behaviors, based on achieving agreed graduate outcomes. The second is that of the clinical workplace, where graduates will soon have defined roles and responsibilities within a clinical team. The balance of the influence of these activity systems on judgements varies for individual assessors. Less experienced assessors tend to follow the academic rules listed in marksheets. More clinically engaged and more experienced assessors tend to fall back on a heuristic, mental construct of a “prototypical intern,” to calibrate judgements, particularly, in difficult situations. This heuristic is based on personal clinical experience and discussions with workplace peers, emphasizing professional attributes and trust, and may lead to a form of confirmation bias, that dominates thinking when candidates are “borderline;” time is pressured; or assessors are more experienced. Further research is needed to explore whether designing assessment marksheets and assessor training to more closely align the two systems decreases the cognitive load and increases the validity of assessor decision making. Designing marking frameworks should consider the possible introduction or amplification of unconscious biases. Further, assessor training may benefit from explicit “de-biasing” by aiming to increase awareness of a heuristic that is shared by assessors and caution against over-reliance on this strategy, thereby reducing unconscious bias.

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

The studies involving human participants were reviewed and approved by the James Cook University Human Research Ethics Committee under Permit H6833. In addition, all participating schools obtained ethics approval from their local Ethics Committee. The participants provided their written informed consent to participate in this study.

BM-A, RH, and KD'S conceived the study and conducted the focus groups. BM-A, KJ, RH, KD'S, and SS advised on data analysis and interpretation. BM-A, RH, KD'S, SS, AC, and RT contributed to writing the original draft of the manuscript. All authors facilitated collection of data, reviewed, edited, and accepted the final version of the manuscript.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.