Artificial Intelligence (AI) is transforming medical education by enabling automated item generation, difficulty estimation, and individualized feedback that was previously impractical (Pohn et al., 2025; Shaw et al., 2025; Gordon et al., 2024; Mir et al., 2023). When integrated with adaptive testing and cognitive diagnostic models, AI allows feedback to evolve from static score reporting into a continuous, formative process that guides learning (Sunmboye et al., 2025). This approach is particularly valuable in surgical education, where cognitive, technical, and non-technical skills intersect and are required, allowing educators to identify specific strengths and weaknesses and promote deliberate practice toward surgical competence (Gomez et al., 2025; Rosendal et al., 2023; Ounounou et al., 2019; Dedy et al., 2013).

Although feedback is widely recognized as a key driver of learning, its application in the cognitive domain, particularly during undergraduate surgical education, remains underexplored (Garner et al., 2014; El Boghdady and Alijani, 2017). Traditional mechanisms often fail to provide timely, specific, and actionable insights into students’ cognitive performance, especially in complex domains such as reasoning and decision-making (Burgess et al., 2020; Shaughness et al., 2017). As medical curricula increasingly adopt competency-based models, adopting assessment strategies that deliver targeted, data-informed feedback has become essential to enhance learning and self-regulation (Ross et al., 2022).

Cognitive competence involves integrating basic scientific knowledge with clinical information to interpret findings and make informed decisions under uncertainty, skills that are critical in surgical practice (Crebbin et al., 2013; Madani et al., 2017). However, traditional multiple-choice exams often emphasize factual recall, provide delayed feedback, and fail to capture deeper levels of reasoning (Butler and Roediger, 2008).

Providing feedback in summative assessments remains a challenge in medical education. Concerns over item security, fairness, and resource limitations often lead institutions to restrict item-level feedback (Appelhaus et al., 2023; Harrison, 2017). Consequently, summative assessments frequently become “black boxes,” offering scores without meaningful guidance and reinforcing a culture of performance rather than development.

When integrated into assessment systems, AI can support the creation of high-quality questions, estimate item difficulty, and generate individualized diagnostic feedback with minimal faculty effort. These capabilities complement Computerized Adaptive Testing (CAT), a psychometric method proposed by Lord (1971), Owen (1975), and Chang and Ying (1996) that dynamically adjusts item difficulty based on student responses (Chang and Ying, 1996). By tailoring the test to the learner’s ability level, CAT increases the efficiency and precision of assessment with fewer items, reducing test fatigue while maximizing information. It can also be useful in determining the true score and competence of an examinee (Collares and Cecilio-Fernandes, 2019; Burr et al., 2016). In the context of medical education, CAT has been successfully implemented in progress testing, licensing exams, and residency selection processes (Burr et al., 2016; Xu et al., 2023; Van Wijk et al., n.d.; Seo et al., 2024). Its potential as a formative tool for learning, however, remains underutilized.

CAT can also be coupled with Cognitive Diagnostic Modeling (CDM), a psychometric framework that analyzes student responses to infer the proficiency of specific cognitive attributes. Unlike classical test theory, which provides a single overall score, CDM enables a multidimensional understanding of performance by categorizing items and responses according to specific cognitive processes. For instance, an item might assess recall of factual knowledge, interpretation of clinical signs, or the application of pathophysiological reasoning. By classifying and analyzing items in this way, CDM supports the generation of individualized feedback, offering students a roadmap for targeted improvement (Ma et al., 2023; Williamson, n.d.; Leighton and Gierl, 2007; Barthakur et al., 2022). This granular diagnostic capability is particularly relevant in complex curricular areas like surgery, where different cognitive skills are needed to approach diverse clinical scenarios. In this study, items were categorized into three cognitive diagnostic models (memory, analysis, and decision) reflecting the cognitive tasks proposed by the National Board of Medical Examiners (NBME) for assessing medical knowledge application (Billings et al., n.d.). These domains were selected because they align with the NBME’s framework for evaluating progressively complex levels of cognitive processing, ranging from factual recall to clinical reasoning and decision-making, which are particularly relevant in the context of surgical education.

Despite their theoretical advantages, CAT and CDM have rarely been explored as learning tools in undergraduate surgical education. By evaluating an AI-supported CAT–CDM intervention in a real educational setting, this study aims to advance data-informed feedback practices, improve alignment between formative and summative assessment, and support the development of self-regulated surgical learners.

The final item bank used in the CAT exam was composed of 150 MCQ items, 84 extracted from the previously calibrated database (OAIPT) and 66 created by faculty members, experienced in item writing. From that item bank, the software developed an individual customized exam with 48 questions, tailored to the student’s level of competence, with a matching blueprint of the same topics as the progress test and the previous surgical curricular unit. The 48 questions were equally divided between the three different diagnostic models: decision, memory, and analysis.

A total of 147 students were included in the analysis, of whom 116 completed the formative CAT. The participant inclusion process and attrition are illustrated in Figure 2. A post-hoc power analysis for the ANCOVA comparing CAT participants (n = 116) and non-participants (n = 31), with α = 0.05, f = 0.173 (derived from η² = 0.029) and two covariates, indicated approximately 0.49 power to detect this effect size. This suggests limited sensitivity for small effects, consistent with the exploratory nature of the study.

Normality was evaluated using the Shapiro–Wilk test. Most variables, including Progress Test, Exam 1, and Exam 2, and CDM-specific measures, deviated from normality in at least one group (p < 0.05). Therefore, non-parametric tests were applied when assumptions were not met. As expected, grades from the CAT exam followed a normal distribution. Table 1 summarizes the descriptive statistics for all key variables, including the overall CAT score, cognitive sub-scores, and summative exam results.

Spearman’s rank correlations were computed to examine the relationships between the final CAT score, cognitive diagnostic sub-scores (Skill–Memory, Skill–Analysis, Skill–Decision), and academic outcomes (Exam 1, Exam 2, and Progress Test). Results are detailed in Table 2 and visually represented in Figure 3.

The final CAT score showed strong correlations with Skill–Memory (ρ = 0.73, p < 0.001) and Skill–Analysis (ρ = 0.63, p < 0.001), and a moderate correlation with Skill–Decision (ρ = 0.42, p < 0.001).

Regarding academic outcomes, the Progress Test was moderately correlated with Skill–Memory (ρ = 0.38, p < 0.001) and showed weaker associations with Skill–Analysis (ρ = 0.26, p = 0.006) and Skill–Decision (ρ = 0.28, p = 0.003). Both Exam 1 and Exam 2 correlated more strongly with Skill–Memory than with the other sub-scores. Additionally, Exam 1, Exam 2, and the Progress Test were highly intercorrelated (ρ ≈ 0.8, p < 0.001), indicating consistent performance across summative assessments.

A Mann–Whitney test revealed that CAT participants scored significantly higher on the Progress Test than non-participants (U = 1,311, p = 0.021, rank-biserial correlation = −0.271). The differences in Exam 1 (U = 1392.5, p = 0.054) and Exam 2 did not differ significantly between groups (U = 1,466, p = 0.115). Figure 4 illustrates the group comparison of Progress Test scores between students who completed the formative CAT and those who did not.

An ANCOVA was conducted to evaluate the impact of participation in the formative CAT exam on the summative score. After adjusting for prior performance (Exam 1 and Exam 2), participation in the formative CAT exam had a statistically significant effect on the Progress Test, F(1, 143) = 4.239, p = 0.041, η² = 0.029. Both Exam 1 (F = 35.835, p < 0.001) and Exam 2 (F = 48.388, p < 0.001) were also significant predictors of performance, indicating that earlier academic performance was strongly associated with the outcome. These findings suggest that the formative exam were associated with improved performance beyond what could be explained by prior academic achievement alone. Figure 4 illustrates the regression relationships between CAT performance metrics and summative Progress Test results.

A series of linear regressions were conducted to evaluate the predictive value of the final CAT score and each cognitive diagnostic sub-score (Skill–Memory, Skill–Analysis, and Skill–Decision) on Progress Test performance.

The model using the final CAT score as a predictor was statistically significant, F(1, 114) = 25.364, p < 0.001, explaining 18.2% of the variance in Progress Test scores (R² = 0.182). The final CAT score emerged as a significant positive predictor (β = 0.427, p < 0.001), indicating that higher adaptive test performance was associated with improved summative outcomes.

When each cognitive skill domain was analyzed separately, Skill–Memory was the strongest predictor of Progress Test performance, F(1, 110) = 24.196, p < 0.001, accounting for 18.0% of the variance (R² = 0.180; β = 0.425, p < 0.001), indicating a moderate positive association between memory skills and Progress Test performance. Skill–Analysis also significantly predicted Progress Test scores, F(1, 110) = 9.574, p = 0.003, though with a smaller effect size (R² = 0.080; β = 0.283, p = 0.003). In contrast, Skill–Decision explained only 2.9% of the variance and was not a significant predictor (R² = 0.029; β = 0.170, p = 0.072). However, when all domains were entered simultaneously into a multiple regression model, only Skill–Memory remained a significant independent predictor (β = 0.41, p = 0.029), confirming its dominant role after controlling for intercorrelations among cognitive domains (Table 1). Figure 5 illustrates the regression relationships between CAT performance metrics and summative outcomes.

This study explored the use of CAT combined with CDM as a formative assessment tool in undergraduate surgical education. A distinctive feature of the intervention was the integration of AI into both item development and feedback generation, allowing efficient test calibration, individualized difficulty estimation, and automated reporting with minimal faculty workload. These findings align with the growing evidence that AI-driven assessment systems can enhance precision, scalability, and personalization in medical education (Pohn et al., 2025; Shaw et al., 2025; Gordon et al., 2024; Mir et al., 2023; Sunmboye et al., 2025; Gomez et al., 2025).

The association between participation in the formative CAT–CDM exam and higher performance in the subsequent summative Progress Test suggests that adaptive, feedback-oriented assessments may promote more effective study strategies. The multiple regression analysis, including all cognitive domains and the total CAT score, confirmed that only Skill–Memory remained a significant independent predictor of summative performance, whereas Skill–Analysis and Skill–Decision did not contribute additional explanatory value once intercorrelations were controlled for. This finding indicates that the predictive relationship between analytical and decision-making domains and summative performance is largely shared with memory-based competence. It also reinforces the interpretation that current multiple-choice examination formats primarily reward factual recall rather than complex reasoning or integrative decision-making. These results highlight the need for assessment strategies capable of isolating higher-order cognitive processes from underlying knowledge recall. One possible explanation is that items categorized as decision may not have captured the full complexity of real-world clinical reasoning. As highlighted in previous work, problem-solving in surgery often depends on context, uncertainty, and prioritization rather than discrete knowledge application (Ross et al., 2022; Crebbin et al., 2013) Multiple-choice questions, even when well constructed, are limited in their ability to elicit such integrative reasoning. In contrast, memory items, more closely aligned with the structure of summative exams, may show stronger statistical relationships simply because both assessments rely on similar cognitive processes. This alignment may reflect systemic bias toward factual recall in traditional testing, emphasizing the need for new formats that authentically measure complex reasoning, such as rich clinical vignettes, branching scenarios, or virtual patients (Gomez et al., 2025; Billings et al., n.d.; Rice et al., 2022) Beyond item design limitations, several alternative explanations merit consideration. First, decision-making competence may not yet be sufficiently developed in fifth-year medical students to exhibit measurable variance, given that authentic clinical decision-making typically matures during postgraduate training. Second, the near-zero correlation between analysis and decision domains (ρ = 0.08) suggests potential overlap or misalignment within the cognitive classification framework itself, indicating that the current three-domain model may have limited discriminant validity. Third, the summative Progress Test used as the external criterion primarily measures factual and analytical reasoning, which may not adequately fully capture the situational judgment or uncertainty management, considered core features of surgical decision-making. Forth, decision-making items may demand context integration and abstraction beyond what can be effectively assessed in a brief, text-based MCQ, causing students to rely on pattern recognition rather than deliberate reasoning. Fifth, the weak association might also reflect a misalignment between formative and summative constructs: whereas the CAT–CDM aimed to assess applied reasoning, the Progress Test could be predominantly capturing factual recall, thereby reducing shared variance by design. These combined factors likely explain the absence of significant associations and underscore that current multiple-choice formats are inherently constrained in representing complex cognitive processes such as risk–benefit reasoning, ethical trade-offs, and context-specific prioritization. As such, the null finding highlights an important boundary in the construct validity of decision-making assessment and identifies an area for future instrument development and validation.

From an educational perspective, these results have practical implications for curriculum design. Although statistically reliable, the effect sizes were modest, suggesting that the practical impact of short formative interventions may be incremental rather than transformative. If recall remains the main predictor of summative success, surgical educators risk over-emphasizing rote knowledge at the expense of reasoning and decision-making competence. Integrating adaptive cognitive diagnostics into teaching could help identify students who rely predominantly on memorization and guide them toward deliberate practice in interpretation and judgment. Educators should consider coupling CAT-CDM data with simulation or case-based discussions, transforming feedback into structured remediation plans. In parallel, AI-supported feedback dashboards could provide students with a dynamic map of their evolving cognitive profile, encouraging self-regulated learning and early correction of deficiencies. By integrating these tools, surgical curricula could progressively shift from knowledge reproduction to diagnostic reasoning and decision-making mastery.

Beyond its psychometric contribution, the present study provides several educational insights. The CAT–CDM framework aligns closely with principles of adaptive learning, in which instructional content and assessment dynamically adjust to each learner’s ability level, which we believe to be key to learning. By identifying specific cognitive domains requiring reinforcement, adaptive testing provides personalized diagnostic feedback that can guide self-regulated learning. Students can use the domain-specific results to direct their study strategies toward weaker areas, engage in deliberate practice, and monitor progress over time. From an instructional standpoint, such diagnostic information enables educators to allocate remediation resources more efficiently and to tailor teaching toward common cognitive gaps.

Furthermore, the integration of AI-assisted item generation and feedback represents a scalable model for formative assessment with minimal faculty workload, consistent with the growing emphasis on feedback as a continuous, learner-driven process rather than an episodic event. By converting performance data into interpretable cognitive profiles, this approach helps close the feedback loop and fosters reflection, autonomy, and iterative improvement.

Several limitations must be acknowledged. First, participation in the formative exam was voluntary, introducing potential selection bias, as more motivated students may have been more likely to participate. Although all students from the same curricular unit were invited to participate, and overall participation was high (79%), voluntary participation may have led to self-selection bias, with more motivated or academically stronger students being overrepresented among CAT participants. This possibility should be considered when interpreting the association between formative and summative performance since it restricts the ability to establish causal inferences; therefore, the observed differences should be interpreted as associations rather than direct effects of the intervention. This short interval of 3–5 days before the summative Progress Test mirrors real-world preparation patterns in which students consolidate learning shortly before assessment. The aim was not to measure long-term knowledge retention but to determine whether adaptive feedback could inform final revision strategies. Future studies will extend the interval between formative and summative assessments to investigate sustained learning effects and behavioral change over time. Second, this was a single-center study, potentially limiting applicability to other educational contexts. Finally, the study focused on short-term outcomes; future research should examine whether the observed benefits persist in long-term knowledge retention and clinical performance. Although initial inter-rater reliability for cognitive classification was substantial (κ = 0.76), and retrospective double-blind validation of a sample of items demonstrated similarly substantial agreement (κ = 0.68), some residual subjectivity in distinguishing analysis from decision items cannot be fully excluded. This issue is particularly relevant given the weaker associations observed in the Skill–Decision domain. Future research should include independent expert classification of the entire item bank to further strengthen the construct validity of cognitive domain distinctions.

Future research should employ randomized or crossover designs to confirm efficacy, explore the durability of learning effects, and examine the longitudinal impact of adaptive feedback on self-regulated learning. Establishing causal links between adaptive testing, feedback quality, and performance improvement would provide stronger evidence for scaling this approach. Integrating the CAT–CDM model with simulation-based or virtual-patient environments could enhance authenticity and allow assessment of higher-order decision-making under uncertainty—an essential but often under-evaluated component of surgical competence. Expanding item banks with scenario-driven and branching questions may further strengthen construct validity for complex reasoning. Ultimately, such investigations could contribute to a more holistic and adaptive assessment ecosystem encompassing the cognitive, technical, and non-technical dimensions of surgical education.

This study found that the use of Computerized Adaptive Testing (CAT) combined with Cognitive Diagnostic Modeling (CDM), supported by AI-based item generation and automated feedback, was associated with higher performance in a subsequent summative assessment. Although decision-making skills were underrepresented in predictive models, the results highlight the need for curricular strategies that better promote higher-order cognitive processing in surgical education. Within the acknowledged methodological limitations, AI-enhanced CAT–CDM emerges as a promising approach for delivering meaningful cognitive feedback and fostering data-informed learning in medical training.