The inclusion of anesthesia and intensive care medicine (ICM) in undergraduate medical student placements is a relatively new development (1). Recent publications have sought to define suitable curricula in these disciplines (2, 3). With expanding placement opportunities come an ever-increasing obligation to ensure that student learning is effective and efficient, that student time is “well-spent,” and that we “maximize assessment for learning while at the same time arriving at robust decisions about learner's progress” (4).

The ICU and the anesthetic room can be challenging areas for student learning. Opportunities for history-taking and clinical examination are variable (1, 5). Patients undergoing anesthesia require a focused history and examination tailored to the upcoming anesthetic and surgical procedure (5). ICM patients are commonly sedated and/or confused, impeding history-taking. Clinical examination in the intensive care unit (ICU) is more challenging in the context of an immobile, unresponsive patient on extracorporeal devices (dialysis, mechanical ventilation). Furthermore, in both disciplines, procedural learning is often limited by the complex, high-stakes, time-sensitive aspects of common tasks (1, 5).

Conversely, anesthesia and ICM share learning opportunities not readily available during other placements. They are ideal environments for the vertical integration of primary and clinical sciences (6). Many learning topics are unique to these disciplines (e.g., acute respiratory distress syndrome, clinical brainstem death evaluation, inhalation anesthesia, pharmacological neuromuscular blockade). Other key elements of their curricula (e.g., the management of acute respiratory failure, shock, acute airway emergencies, sedation administration) are generic, high-stakes, transferrable clinical skills that could be viewed as important competencies for all doctors.

Current evidence suggests that ICM and anesthesia placements can achieve effective student learning outcomes (7). Nonetheless, the unique nature of their curricula may require a bespoke approach to learner assessment. Furthermore, valid and reliable tools are central to assessment decisions regarding high stakes competencies such as effective acute and perioperative patient care. Despite this, three recent papers on curriculum and effective teaching in the ICU and anesthetic room make no recommendations about student assessment (2, 3, 5). The first objective of our review therefore was to evaluate the nature and robustness of published assessment strategies in these high-stakes clinical specialties, incorporating an analysis of the theoretical bases for these publications.

The expansion of undergraduate anesthesia and ICM placements has occurred contemporaneously with an evolution in medical education assessment. Accordingly, practice is moving away from evaluating low-level cognitive learning objectives such as knowledge and understanding (using MCQs, written examinations) toward knowledge application (using extended matching questions, OSCEs) and most recently to clinical performance, either in a simulated or workplace environment (8–11). Furthermore, longitudinal methods such as programmatic assessment have in recent years gained in popularity (12). The extent to which this evolution has translated to assessment in undergraduate anesthesia and ICM education was the second objective of our review.

A scoping review methodology was used for two reasons. First, the authors had prior knowledge of the research topic and recognized that the range of published literature was unlikely to yield research of sufficient quality to enable a systematic review or a meta-analysis. Second, in light of recent advances in assessment practice, we anticipated a knowledge and/or research gap in the areas of anesthesia and ICM assessment. Our study methodology therefore needed to be tailored to identifying these gaps were they to exist (13, 14).

We used the 6-step adaptation of Arksey and O'Malley's (15) scoping review framework as proposed by Levac et al. (13). These steps are (1) identifying research questions, (2) identifying relevant articles, (3) study selection, (4) charting the data, (5) collating, summarizing, and reporting the results and (6) consulting with stakeholders. In addition, we applied a scoping review quality checklist to enhance the rigor of our findings (16). The overarching purposes of our scoping review were (a) to describe the nature of existing research about undergraduate assessment after anesthesia/ICM placements and (b) to identify research gaps in this area.

The following six steps were applied.

A total of 2,435 results were returned from the initial search between the 5 databases, of which 17, published between 1983 and 2020, were selected for full-text review (6, 20–34, 36). The second search performed in August 2021 returned 2 further studies published in 2021 (19, 35). The findings of these 19 studies are shown in Table 1.

Of the 19 studies, 9 (47.4%) involved anesthesia (21, 23–25, 28–32), 8 (42.1%) intensive care medicine (6, 19, 20, 26, 27, 33–35) and 2 (10.5%) a combination of both disciplines (22, 36). The primary research focus was on the assessment instrument and on student learning in 9 (22–26, 28–30, 36) and 7 (19–21, 27, 32, 33, 35) studies respectively. The remaining 3 studies had equal research focus on learning and the assessment tool.

All were single-center studies, 14 of which (73.7%) were conducted in Canada, USA and Hong Kong. The average sample size across all studies was 119 students (range 5–466). Clinical placements lasted 2–12 weeks, with a median duration of 4 weeks. Fourteen studies (73.7%) had 2 or 4 week placements.

Twelve of 19 studies (63.2%) had a non-randomized design and collected assessment data at one timepoint only (20–25, 28–32, 36). Conversely, the remaining 7 studies (36.8%) were either RCTs and/or had a pre-/post-test study design (6, 19, 26, 27, 33–35). Despite all studies reporting solely or mainly quantitative data, none conducted a power analysis to evaluate the required sample size. Ten studies (52.6%) considered the issues of the reliability and/or validity of their assessment tools (6, 21–26, 28, 30, 31). Two additional studies used standardized assessment questions from the Society of Critical Care Medicine and the American College of Physicians (20, 35). The average MERQSI score was 12.3 (range 5–15) out of a maximum of 18.

Our scoping review illustrates the heterogeneity of literature around assessment in undergraduate anesthesia and intensive care medicine. Published studies used numerous assessment instruments targeting learning outcomes that were either knowledge-based (using selected-response MCQs and constructed-response written tests) or performance-based (using OSCEs or a simulated clinical environment). The findings of the review also attest to the meaningful undergraduate learning that can occur in these clinical settings.

Though heterogenous, a majority of the included studies used evidence-based assessment strategies insofar as either the chosen tools have strong evidence supporting their use in UGME (MCQs, written exams, simulation, OSCEs) or more than one method was used to inform assessment decisions (8, 9, 37). Furthermore, all except one study (33) used an assessment strategy appropriate to the learning outcomes mapped to Miller's pyramid.

A key objective of undergraduate medical education is to equip students with the competencies to deliver effective and safe patient care in their first year of medical practice and beyond. The assessment of knowledge, or the theoretical application of that knowledge alone may not be sufficient to judge whether students are equipped with those skills (38). Accordingly, 9 studies in our review adopted a competency-based approach and used OSCEs or simulation to evaluate student performance. Only 3 of these studies however used complementary tools to evaluate learning in the domains of knowledge and understanding as well as competence (22, 23, 26). These are the most informative studies in our review for educators making instructional design decisions about student assessment after anesthesia and ICM placements.

The most frequent tool used to evaluate student performance in our review was simulation, whereby learners were assessed in the “show how” learning domain. Performance in a simulated environment however correlated poorly with written assessments, suggesting a role for both tools in reaching a more complete assessment decision about a student's learning outcomes. There was scant evidence in the studies however of observed performance assessment in the workplace–the “does” domain of Miller's pyramid–which likely reflects a lack of active student work in ICUs and anesthetic rooms; students in these environments are more likely to learn by observing than by doing.

Nonetheless, recent trends in undergraduate assessment have led to greater emphasis on workplace performance and to the use of entrustable professional activities (EPAs) (39–41). EPAs entail assessors observing students performing “units of work” (42) (p2) thereby judging the level of supervision each student needs with that activity–the entrustment decision (11). They are mapped to learning curricula and are commonly informed by assessment in the workplace (11). To date, most EPA studies in UGME centre around internal medicine and general surgery. While we identified no studies using EPAs solely for the purposes of undergraduate anesthesia/ICM assessment, some include anesthesia or ICM placements as part of a broad learning curriculum (43–45). Furthermore, of the 13 undergraduate EPAs published by the Association of American Medical Colleges, 3 (e.g., recognize a patient requiring urgent or emergent care and initiate evaluation and management) have direct relevance to ICM and anesthesia (46). The use of EPAs also helps address long-standing concerns about graduating student's readiness to commence internship (47). A strong case can therefore be made for applying EPAs to anesthesia and ICM.

We did not identify any studies using a programmatic approach to assessment, though our literature search may not have found studies which included anesthesia or ICM as part of a broader programme-wide assessment strategy. Moreover, programmatic assessment challenges the “module-specific” nature of traditional UGME, viewing a clinical placement within a broader context of an overall curriculum (12). Therefore, it does not readily apply to our review, which ab initio focused on the assessment of a specific placement in anesthesia or ICM.

A common criticism of education research is that theoretical considerations are not brought to the fore. This also applies to the majority of articles in our review. Notwithstanding this, most of the included studies were designed in such a way that the use of a theoretical paradigm could be implied. Moreover, some of the studies used instructional design methodology. The primary use of technology to enhance learning in 3 studies (27, 29, 32) draws from eLearning theory (48). Adopting problem-based learning in 3 further studies acknowledged the importance of constructivism in effective education (6, 26, 34). Aspects of workplace learning theory were evident in the numerous studies that promoted learning within the operating theater and/or the intensive care unit (6, 20, 22, 23, 26, 29, 30, 33, 34, 36). The 6 studies using simulation for assessment (21, 23, 25, 26, 30, 31) likely reflected the importance of experiential learning theory and reflection (49).

A recent development that is difficult to ignore is the impact of the COVID-19 pandemic on undergraduate education. Accounts of formal assessment in the context of students working or learning during the pandemic appear to be rare (19). This likely reflects the time constraints and staff redeployment during the pandemic when clinical activities took precedence over faculty pursuits (50). Paradoxically, when viewed through the lens of Miller's pyramid, at a time when students may have been more actively involved in clinical care–in “doing” work–they were least likely to be formally assessed.

Our study therefore highlights a gap between published research in anesthesia/ICM assessment and recent advances in undergraduate assessment. However, for practice to change in these disciplines, the approach to student education in anesthetic rooms and ICUs must first evolve to allow more active student participation in daily clinical activities. Observed workplace assessment can only occur in environments where learners play a legitimate role in patient care. This is the main research gap identified in our review and is an important area for future research into the undergraduate study of anesthesia and intensive care medicine.

Our study results may be limited by the omission of some relevant articles. Each author however individually performed the search and results were then combined. Studies were individually evaluated for inclusion and though Cohen's coefficient showed a suboptimal correlation, author discussion at each step of study selection addressed any differences in opinion. We included any study where discussion did not resolve perceived differences. To improve the rigor of our study, we used the guidelines published by Maggio et al. in all stages of our review (16). We consulted with key external stakeholders but this step yielded no additional information.

In conclusion, our findings yield three useful insights. First, they act as a practice guide for educators directly involved in the design, delivery, and assessment of undergraduate learning in anesthesia and intensive care medicine. Second, they are informative for university educators tasked with the general organization and design of undergraduate medical education, helping them position anesthesia and intensive care medicine in strategies around programmatic assessment and workplace-based entrustable decision-making. Finally, they identify a large research gap for future studies to focus upon.

EO'C and ED were equally involved in all aspects of this study including design, literature searches, data collection, and manuscript writing. Both authors contributed to the article and approved the submitted version.

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.