This work employed a qualitative, exploratory design to capture expert perspectives on the use of artificial intelligence (AI) for OSCE station generation. The approach focused on gathering experiential insights from educators who had directly implemented AI-assisted scenario development within high-stakes clinical examinations.

Six medical education experts were recruited through purposive sampling, selected intentionally to ensure representation across undergraduate and postgraduate programs, clinical specialties, and assessment leadership roles. All participants had prior experience in OSCE design, blueprinting, and faculty development, and had also been involved in pilot projects that used AI tools for scenario generation.

Two members of the research team led an organized, in-person focus group meeting to gather data. In order to ensure that all participants responded to the same prompts and that fundamental topics were covered methodically, the session adhered to a set of pre-planned questions.

Due to the lack of audio recording, both facilitators took thorough notes in real time, recording participant reactions, areas of agreement or disagreement, and providing clarification on examples discussed. The facilitators checked and combined their notes right away to make sure they were accurate and comprehensive. Using standardized file-management practices, all data was safely kept on an encrypted institutional server.

Four predetermined domains were covered by the structured question set:

The discussions were supplemented by the authors' experiential evidence from direct implementation of AI-assisted case generation during national OSCE clinical examinations.

The study was conducted within the context of the National Institute of Health Specialties (NIHS), United Arab Emirates. NIHS is responsible for administering final written and clinical examinations for residency programs across multiple specialties. Since its establishment, NIHS has conducted more than 45 clinical examinations, engaging over 200 examiners from diverse disciplines and training backgrounds. The NIHS experience provided a robust, real-world perspective on integrating AI into high-stakes assessment design at scale.

Three distinct sources of information contributed to this work, each serving a different purpose and treated separately throughout the analysis:

Braun and Clarke's reflexive thematic analysis framework, following the stages of familiarization, coding, theme development, and iterative refinement, was applied to synthesize findings from focus group discussions. The structured focus group notes were coded using an inductive method to make sure that themes were based on participant viewpoints rather than presumptions. Initial coding recommendations were produced by AI-assisted tools, but the research team manually reviewed, validated, and improved each code to guarantee accuracy, contextual sensitivity, and compliance with reflexive thematic analysis principles. Any disagreements were reviewed cooperatively and settled by team consensus. Emergent themes were organized into 10 concrete, evidence-informed recommendations. These recommendations were iteratively refined through author consensus meetings, incorporating illustrative examples from real examination scenarios to maximize relevance and applicability for educators.

Findings from the focus group discussions, together with experiential evidence, highlighted key opportunities and challenges in the use of AI for OSCE development. These insights were complemented by a focused literature review, which revealed a rapidly expanding body of evidence on AI in medical education and assessment. Together, these sources formed the foundation of a broader developmental workflow in which evidence was synthesized and iteratively refined to generate the 10 practical tips presented in the Results, as illustrated in Figure 1.

Six key themes emerged:

Large language models (LLMs) have shown the ability to generate realistic and diverse OSCE stations that mimic authentic patient presentations (16). Studies demonstrate that AI can produce structured checklists, candidate instructions, standardized patient (SP) scripts, and examiner probes with acceptable face and content validity (7, 10). However, faculty review is indispensable to ensure accuracy and curricular alignment (8).

AI-assisted case generation significantly reduces the time and effort required to produce examination materials (7). This efficiency enables broader coverage of clinical conditions, enhances sampling of competencies, and facilitates creation of large exam banks across multiple specialties (17).

Although promising, AI-generated outputs may contain factual inaccuracies, inconsistencies, or oversimplifications. Clinical, demographic, or cultural bias remains a concern. For these reasons, human oversight is essential to safeguard validity, reliability, and fairness (11).

Embedding AI-generated scenarios within programmatic assessment frameworks supports alignment with competency-based curricula and blueprint requirements (18). Emerging applications also include augmenting problem-based learning (PBL), adaptive testing, and blueprint design (19, 20).

Key issues include data confidentiality, intellectual property rights, transparency, and risk of over-reliance on AI (11). Institutional policies, faculty training, and continuous QA processes are emphasized as critical enablers of safe use (15).

Implementing AI-assisted OSCE development presents several practical challenges. Institutional policies regarding data security, examination confidentiality, and approved digital tools may limit access to generative AI platforms, particularly for high-stakes assessments. Faculty preparedness varies widely, requiring targeted training in AI literacy, prompt design, and validation processes. Resource constraints, including limited access to secure enterprise-level AI systems and protected assessment management platforms, may further restrict implementation. During NIHS examinations, challenges included initial faculty skepticism, variability in prompt quality, and the need for additional review time to address AI-generated inconsistencies. Addressing these barriers requires institutional governance, phased implementation, and investment in faculty development and secure infrastructure.

The development of a robust, competency-based blueprint is paramount to the quality and effectiveness of any Objective Structured Clinical Examination (OSCE).

AI is a powerful tool, but its outputs are only as effective as the inputs it receives. One of the most common pitfalls in using AI for OSCE case generation is providing vague prompts without a structured assessment framework. In the absence of clear guidance, AI may generate cases that are verbose, unfocused, or misaligned with curricular goals (20).

To mitigate this risk, faculty should define key assessment design parameters before engaging AI. Establishing these parameters provides the structural clarity necessary for AI systems to generate focused and relevant OSCE cases.

At a minimum, assessment designers should specify the intended station format, such as history taking, physical examination, data interpretation, counseling, or procedural skills. They should also identify the primary competency focus, for example communication, professionalism, or clinical reasoning, and clearly articulate the tasks expected of the candidate.

Timing constraints and scoring structure should be defined in advance. Typical timing ranges from 7 to 10 min for undergraduate OSCEs and 12–15 min for postgraduate assessments. Scoring approaches may include checklist based methods, global rating scales, or hybrid models. Together, these design decisions ensure that AI generated OSCE cases are aligned with assessment objectives and ready for faculty validation.

The quality of AI generated OSCE cases depends strongly on the specificity and richness of the contextual information provided in the prompt. When prompts lack sufficient clinical, institutional, or cultural context, AI systems tend to produce generic or poorly contextualized scenarios that may not align with local practice or learner needs (8).

Enriching prompts with references to clinical guidelines, institutional assessment policies, and local epidemiological patterns strengthens content validity and improves alignment with authentic clinical practice.

This approach is particularly important for assessment because AI can be deliberately guided using established evidence based frameworks such as Entrustable Professional Activities, CanMEDS roles, or EmiratesMEDs competencies. Anchoring prompts to these frameworks helps ensure that generated cases target clearly defined learning outcomes and assessment purposes (18, 22). Embedding these frameworks into prompts improves consistency across specialties and supports competency-based medical education (CBME).

In practical terms, prompt enrichment may include explicit reference to relevant clinical guidelines such as those issued by the ADA, NICE, or WHO, incorporation of demographic and cultural characteristics that reflect local patient populations, and alignment with institutional or national competency frameworks such as EmiratesMEDs, CanMEDS, or ACGME. Specifying disease prevalence or priority conditions based on local health needs further enhances relevance and educational defensibility.

The contrast between a weak and an enriched prompt illustrates this effect clearly. A prompt such as “Write an OSCE case on hypertension” is likely to yield a broad and unspecific scenario. In contrast, a prompt that asks the AI to “Generate a 10 min OSCE case for a Year 4 medical student in the UAE, focusing on counseling a patient recently diagnosed with type 2 diabetes, aligned with ADA 2025 guidelines, and reflecting cultural considerations in diet and lifestyle” produces a more realistic, contextually grounded, and assessment ready case.

The effectiveness of AI in OSCE case generation depends largely on the quality of the prompt provided. Poorly constructed prompts often yield vague, overly simplistic, or irrelevant scenarios (8). Conversely, structured, precise, and layered prompts can generate highly relevant cases, complete with candidate tasks, standardized patient (SP) roles, and scoring checklists (7).

Prompt engineering is, therefore, a critical skill for educators using AI in assessment design. This involves breaking down requests into clear instructions, embedding contextual details, and specifying the desired output format. For example, prompts should identify the clinical scenario, learner level, competencies targeted, case duration, and assessment format. Adding output constraints (e.g., number of checklist items) supports standardization and enhances the defensibility of OSCE scores (20). Importantly, educators can also ask the AI itself to refine or recreate a better prompt, allowing for iterative improvement and more tailored case outputs.

Well engineered prompts play a critical role in ensuring that AI generated assessment materials accurately reflect the intended competencies and assessment constructs. By explicitly defining the task, target learner level, clinical focus, and competency framework, structured prompting reduces the risk of construct irrelevant variance and minimizes the introduction of unintended difficulty or bias. This is particularly important in OSCE design, where small variations in case framing can substantially influence performance.

In addition, structured prompting supports reproducibility and consistency across examination cycles. When prompts are standardized and systematically documented, AI generated cases can be regenerated, reviewed, and refined in a controlled manner, supporting fairness, comparability, and quality assurance within the assessment process.

A weak prompt such as “Write an OSCE case about asthma” provides minimal guidance to the AI and is therefore likely to generate a broad, generic scenario with limited educational or assessment value. In contrast, a well structured prompt that specifies the learner level, clinical context, task duration, and required outputs leads to a far more targeted and usable case. For example, asking the AI to “Generate a 10 min OSCE case for Year 4 undergraduate students on the acute management of an adult with asthma exacerbation in the emergency department, including candidate instructions, a standardized patient script with key phrases, examiner prompts, and a 12 item checklist mapped to communication, history taking, and clinical reasoning domains” results in a coherent, assessment ready OSCE station that is clearly aligned with intended competencies and scoring criteria.

The second prompt provides specificity (clinical scenario, learner level, setting, task) and constraints (duration, checklist length, competency mapping), resulting in a structured and usable output that supports standardization and inter-rater reliability.

Effective prompt engineering benefits from a structured and sequential approach. Using stepwise instructions, such as requesting the AI to first generate the case stem, followed by the standardized patient role and then the scoring checklist, helps control output flow and improves internal consistency.

Explicitly specifying competency alignment further strengthens prompt quality. Referencing established frameworks such as CanMEDS roles or EmiratesMEDs competencies ensures that AI-generated cases are clearly linked to intended learning outcomes and assessment standards.

Providing clear output constraints is also essential. Defining parameters such as the number of checklist items, station duration, and expected level of detail helps standardize AI outputs and reduces unnecessary variation across cases.

Prompt development should be treated as an iterative process. Refining prompts based on initial AI outputs, using structured feedback loops, allows educators to progressively improve relevance, clarity, and alignment with assessment goals.

Finally, employing meta-prompts, such as asking the AI to assess a prompt for clarity, completeness, or alignment with objectives, can further enhance design quality. This reflective layer supports more deliberate and defensible use of AI in OSCE case development.

Educators should also recognize that prompt refinement is iterative: initial outputs often require follow-up instructions such as “revise checklist to reduce redundancy” or “adjust SP role to reflect cultural communication styles.” This iterative dialogue ensures accuracy and contextual fit (19). Iteration also reduces construct drift and helps maintain fidelity to assessment specifications.

AI can enhance OSCE case development, but its outputs are not self-validating. Faculty play a central role in ensuring accuracy, fairness, and alignment with curricular outcomes (18). Across the literature, recurring limitations; such as factual inaccuracies, cultural or demographic bias, and inconsistency in output detail, were identified, reinforcing the need for purposeful faculty preparation (8). Therefore, capacity building and faculty development are essential for safe and effective integration of AI into assessment design. Strengthening faculty capability also supports defensible assessment decisions and reduces variability in how AI tools are used across departments.

Even when prompts are carefully structured, AI generated OSCE cases may still contain factual inaccuracies, cultural mismatches, or internal inconsistencies. If these issues are not systematically identified and corrected, they can undermine exam validity, fairness, and reliability (23). For this reason, AI generated cases should not be adopted directly for formative or summative assessment without a clearly defined validation and quality assurance process.

Validation should begin with expert review, in which subject matter experts examine AI outputs for clinical accuracy, alignment with current guidelines, and relevance to the local clinical and educational context (21). This step is essential for identifying subtle clinical errors or inappropriate assumptions that may not be immediately apparent in automated outputs.

Following expert review, cases should be cross checked against the assessment blueprint to confirm that they address the intended competencies and are appropriately positioned within the overall examination structure (3). A subsequent peer review by an independent faculty member can further strengthen quality by identifying gaps, redundancies, or inconsistencies in task design, scoring criteria, or case flow.

Pilot testing provides an additional safeguard by exposing the case to a small group of students or residents prior to formal use. This step helps identify ambiguities, timing problems, and imbalances within checklists or examiner prompts, which are particularly common in early AI generated drafts that vary in depth, pacing, and clarity (24). After implementation, ongoing psychometric monitoring is required to evaluate case performance using indicators such as difficulty indices, discrimination, and inter rater reliability, thereby confirming that the case functions as intended in practice (9).

Although this process closely resembles traditional quality assurance procedures for OSCE development, it must be deliberately adapted to address AI specific risks, including hallucinated content and embedded biases. Systematic validation therefore remains a critical safeguard when integrating AI generated cases into assessment programs (Table 5).

This streamlined workflow illustrates how AI output integrates into existing QA systems while requiring adaptations for AI-specific risks.

AI systems learn from large datasets that often embed historical, cultural, or clinical biases. Left unchecked, these biases can appear in OSCE cases, perpetuating stereotypes or underrepresenting important patient groups (23). For example, AI may default to male patients for cardiac cases or neglect culturally sensitive expressions of symptoms (25). Such biases undermine fairness and may disadvantage certain candidate groups.

Additionally, AI introduces new exam security risks. Cases generated or stored on unsecured platforms could be accessed by candidates, raising concerns about content leakage and academic integrity (26). Institutions must therefore combine bias detection with robust security protocols.

Promoting fairness and safeguarding exam security are critical when integrating AI into OSCE development. One essential step is systematic bias screening, which helps counteract AI's tendency to reproduce default demographic patterns or generate unrepresentative cases. Faculty should ensure diversity across patient demographics, including age, gender, ethnicity, and socioeconomic status, avoid stereotypical portrayals such as linking specific conditions or behaviors to particular groups, and include clinical conditions that reflect local epidemiology.

Fairness should also be evaluated through structured audits of AI-generated cases. Mapping cases to blueprint domains helps ensure balanced representation across competencies and content areas, while review by subject matter expert panels allows identification of unintended differences in case difficulty across learner subgroups.

Cultural adaptation is another key consideration. Standardized patient scripts should be reviewed and adapted to reflect locally appropriate communication styles and sociocultural norms. Involving regional educators in reviewing language, cues, and contextual details helps ensure authenticity and reduces the risk of cultural mismatch.

Robust security measures are necessary to protect exam integrity. Institutions should use secure, institutional, or encrypted AI platforms and avoid public tools for generating or storing final examination content. AI-generated cases should be housed within secure assessment management systems that maintain access logs, and case variants should be rotated across exam cycles to reduce predictability and content leakage.

Finally, fairness and security require ongoing monitoring after exam administration. Performance data should be analyzed across variables such as gender, language, or educational background to detect differential item functioning. Stations that demonstrate evidence of unfairness or bias should be revised or retired to maintain the integrity and defensibility of the assessment.

Reusing identical OSCE stations across exam cycles increases the risk of content leakage and predictability. This undermines fairness by rewarding candidates who have prior access to cases, rather than those demonstrating competence (3). Traditionally, creating multiple case versions is resource-intensive, limiting the number of variations available for high-stakes exams.

AI can generate multiple permutations of a core case by varying demographics, context, or complicating factors while preserving the intended learning outcomes (27). This not only strengthens exam security but also improves authenticity by reflecting the variability seen in real-world clinical practice (9).

Variation generation also helps mitigate recurring AI limitations; such as overly generic outputs, by prompting more contextualized and diverse scenarios from a single validated core case.

The generation of multiple case variants using AI should begin with clearly defining the core elements that must remain constant across all versions. These include the primary diagnosis, the intended competencies to be assessed, and the essential checklist items that underpin scoring and decision making. Fixing these elements helps ensure construct consistency and protects against unintended shifts in difficulty.

Once the core elements are established, selected variables can be deliberately modified to create meaningful variation. These may include patient demographics such as age, gender, or ethnicity, as well as differences in clinical context such as emergency, outpatient, or rural settings. Additional variation can be introduced through comorbidities, including the presence or absence of relevant risk factors, and psychosocial factors such as caregiver involvement or language barriers, provided these do not alter the underlying construct being assessed.

Structured prompts should then be used to instruct the AI to generate a limited number of variants, typically three to five, while explicitly maintaining comparable difficulty, scope, and station timing. Clear constraints within the prompt are essential to prevent drift in case complexity or assessment focus across versions.

Each generated version must undergo independent validation to confirm clinical accuracy, fairness, and alignment with the assessment blueprint, following the same quality assurance principles applied to single AI generated cases. Finally, validated case variants can be rotated across examination circuits or assessment sessions to reduce recall bias and enhance exam security, while preserving comparability across cohorts (Table 6).

This approach ensures that all versions test the same learning outcomes (history, risk factor identification, urgent management) but in diverse and realistic contexts.

Developing an OSCE case requires much more than writing a clinical vignette. Each station needs candidate instructions, standardized patient (SP) scripts, examiner checklists, global rating scales, and sometimes supplementary resources such as lab results, imaging, or patient education materials. Traditionally, creating these materials is highly time-consuming and requires coordination among multiple faculty (27).

AI can streamline this process by generating structured supporting materials from a validated case stem. This not only reduces faculty workload but also enhances consistency across exam circuits, improving fairness and reproducibility (7).

Introducing AI into OSCE case generation is not a one-time innovation but a continuous process. Even well-designed, validated stations require ongoing evaluation to confirm that they function as intended in real testing environments. Psychometric feedback and iterative refinement ensure that AI-generated cases remain valid, reliable, and fair over time (3).

This ongoing evaluation is particularly important because several limitations identified across the literature; such as inconsistency in output structure, variable difficulty, and occasional factual errors, can only be detected through post-administration monitoring.

Tips 1–3 highlight the importance of blueprinting, structured case design, and contextual enrichment. The broader implication is that AI enhances competency-based assessment only when its use is anchored in explicit curricular expectations and contextualized with local clinical realities. AI does not inherently ensure competency alignment; this alignment is created through deliberate educational design choices embedded within prompt structure and case specifications (21).

Tips 4–6 focus on prompt engineering, faculty training, and validation processes, which are critical to mitigating the risks of flawed or biased outputs. The validation workflow, which combines subject matter expert (SME) review, blueprint cross-checks, peer feedback, and psychometric monitoring, parallels best practices in traditional OSCE design while adapting them to the AI era (3). This iterative, multi-step validation ensures stations are not only clinically accurate but also psychometrically robust.

This suggests a hybrid quality model in which human expertise and psychometric evidence jointly regulate the reliability of AI-assisted assessments.

Tips 7 and 8 emphasize equity and exam security, highlighting the need for bias audits, cultural adaptation, and the use of diverse case permutations. AI can unintentionally reproduce stereotypes or exclude minority patient profiles (25); deliberate diversity in AI prompts, combined with fairness audits and post-exam analysis for differential item functioning, helps safeguard equity. Generating case variations also mitigates the risks of item leakage, strengthening the integrity of high-stakes exams (27).

When these findings are interpreted widely, fairness and security appear not as discrete tasks, but as systemic safeguards required for justifiable AI usage in high-stakes assessments. AI prompts must actively counteract demographic defaults, and case modification methodologies achieve two goals: increasing authenticity while decreasing predictability and item exposure.

Tips 9 and 10 illustrate how AI can streamline the generation of supporting materials (SP scripts, examiner checklists, multimedia resources) and facilitate continuous improvement through psychometric feedback. These innovations reduce administrative burden, improve consistency across circuits, and foster authenticity by mirroring real-world variability in clinical encounters (24). Beyond efficiency, an important implication is that AI generated materials, when validated, can enrich authenticity by offering dynamic variations and multimodal resources that more closely mirror real clinical environments. Moreover, integrating post exam psychometric data into AI assisted revision cycles reflects a shift toward continuous quality enhancement rather than one time case design.

Despite early support for the feasibility of AI assisted OSCE generation (9, 18), findings from focus group discussions revealed notable skepticism and highlighted several critical areas requiring further investigation. Participants expressed particular concern about the reliability and psychometric defensibility of AI generated materials, reflecting uncertainty about whether such outputs can consistently meet established assessment standards.

One key priority identified was the need for rigorous validation of psychometric properties, including reliability and generalizability, across diverse educational and clinical contexts. Addressing these concerns is essential to demonstrate consistency of output and to strengthen evidence for content validity. Participants also emphasized the importance of examining whether AI generated case variations are effective in maintaining exam security, as predictability and item leakage were perceived as ongoing challenges despite the use of multiple case versions.

Further inquiry was recommended into the applicability of AI generated assessments beyond undergraduate medical education. Participants questioned whether similar approaches could be extended to interprofessional and competency based assessments across other health professions, raising issues of generalizability and contextual transferability. In addition, the long term impact of faculty development initiatives emerged as an important area for research, particularly in light of the wide variation in confidence, readiness, and risk awareness observed among faculty users.

Collectively, addressing these research gaps will help establish a stronger empirical foundation for institutional policy development and accreditation guidance, ensuring that the adoption of AI in assessment is aligned with principles of quality, fairness, and equity.

While the developmental workflow efficiently combined experiential evidence, focus group insights, and expert consensus, these sources inevitably had limits. Experiential perspectives are frequently linked to local practices and institutional cultures, thereby limiting the applicability of findings to other educational settings. Furthermore, expert agreement is inevitably influenced by the opinions and assumptions of participating educators, notably their expectations about AI readiness or established evaluation standards. Variations in participants' prior experience with AI technologies may possibly have altered the prioritizing of perceived difficulties or opportunities. These limits do not diminish the validity of the recommendations, but rather highlight the need for ongoing empirical validation across a wide range of programs and situations.