Consistent monitoring of embryo development, whether conducted at discrete time-points or with the aid of time-lapse technology (TLT), represents a routine and critical task within assisted reproductive technology (ART). Considering the established relationship between embryo morphology and reproductive success (1–3), a visual assessment of embryo morphology is paramount towards the birth of a healthy child. To this end, TLT systems have enhanced the descriptive resolution of embryo monitoring, leading to its widespread clinical adoption (4–6). However, despite these advancements, morphological assessments remain time-consuming (7) and prone to variability across clinics (8), protocols (9, 10) and operators (11). As a means of addressing these limitations, several commercially available artificial intelligence (AI) systems have experienced a sharp increase in utilization (12–14), with demonstrated improvements in cost- and time-efficiency within clinical settings (15). Indeed, the application of various deep learning (DL) models, particularly convolutional neural networks (CNN) have been successfully implemented in classification tasks (16–20), embryo ranking strategies (21, 22), prediction of morphokinetics (23) and reproductive viability (24–26). However, as numerous studies suggest the equal (27–29), if not improved (25, 30) predictive performance of DL models compared to embryologists, reaching a clinical consensus on model evaluation remains an ongoing effort.

Model performance is often evaluated based on accuracy, which provides a measure of how closely a model prediction aligns with a known outcome (31). While reflective of some clinical tasks, such as embryo stage classification, several publications acknowledge the misleading properties of reporting model accuracy alone, particularly within the imbalanced datasets available in medical science (31–33). Consequently, as AI-powered tools gain traction within the field of ART, models are being evaluated with a greater range of performance indicators (5, 31, 34), including a lens of agreement (21, 29, 35–37). Providing a measure of consistency to clinical gold-standards, benchmarking DL model performance against embryologist performance has become a common method in rationalizing the use of AI systems within in-vitro fertilization (IVF) laboratories (13). Practically, however, a major barrier to adoption has been attributed to difficulties in interpreting how a model has reached a particular decision. Thus, the lack of transparency within “black box” DL architectures, where decisions can carry significant ethical, emotional and medical implications, leads to mistrust, misinterpretation and poor clinical integration (38, 39). Moreover, an incomplete understanding of model predictions raises concerns about whether predictions are based on biologically meaningful features, or spurious correlations (40). Remarkably, very few studies have examined whether embryologists and DL models reach the same decisions based on shared interpretive features (36), thereby representing a notable gap in the effective evaluation of clinical AI (41).

Building upon these observations, explainable artificial intelligence (XAI) is emerging to bridge the gap between accuracy and agreement (5, 38, 42–44), a distinction that is critical when model classifications must be justified or explained. XAI methods aim to increase interpretability by visualizing regions or features that contribute strongly to resultant predictions (43, 45). One such method, Gradient-weighted Class Activation Mapping (Grad-CAM), generates local, gradient-based saliency maps, known as heatmaps, that visualize model attention upon classification tasks (46–48). Another occlusion-based approach perturbs the input image to create Local Interpretable Model-agnostic Explanations (LIME), which outline salient input regions that contribute towards changes in the class of a given model (19, 49). Despite the growing interest in XAI, applications of Grad-CAM (37, 47, 48, 50–52), LIME (53, 54), or a combination of XAI methods (19, 49, 51) still lack comprehensive evaluation by domain experts. In fact, most existing work focuses on predictive outcomes such as blastocyst quality or implantation potential, without deeper evaluation of spatial attention patterns (13, 15). Consequently, reports that demonstrate alignment between model accuracy, agreement and domain expert interpretability remain scarce.

This study aims to address these gaps by evaluating the accuracy and agreement of clinical embryologists and two DL models, ResNet-34 and VGG16, in the classification of single-point images representing embryonic developmental stages. The choice of DL models in this study is based on our previous publication (49) and represents some of the most frequently used architectures for embryo development and assessment tasks (13). Crucially, beyond model performance alone, we apply Grad-CAM to each model, thereby creating “explanations” that are evaluated for biological relevance and interpretability by embryologists. Additionally, we apply LIME explanations to the same models in order to quantitatively evaluate the degree of spatial overlap between independent XAI techniques. The selection of local, post-hoc XAI methods such as Grad-CAM and LIME are intended to cater to the equally local or individual differences observed within embryo cohorts, thereby allowing insight into specific predictions, rather than global markers. By investigating classification patterns, domain expert assessments and explainability-based evaluations, this study aims to provide deeper insight into the reliability of AI-driven embryo stage classification, thereby contributing to the responsible development and implementation of AI tools in ART.

This study investigated differences in accuracy and inter-rater agreement between human operators and DL models during embryo stage classification. Embryologists, individually and collectively (89.9%), achieved greater accuracy in embryo stage classification compared to ResNet-34 (78.8%, p < 0.001) and VGG16 (74.3%, p < 0.001). Despite these observed differences in accuracy, all raters demonstrated excellent overall agreement with the reference standard, and with other operators. Contrastingly, further analyses conducted at the developmental stage level revealed lower agreement between DL models, thereby suggesting variations in underlying classification performance that may not be represented by accuracy assessments alone. Further divergence between models was detected upon assessing the biological relevance of Grad-CAM explanations. Although our findings indicate that Grad-CAM explanations generated by ResNet-34 were more frequently deemed biologically meaningful (89%) compared to their VGG16 Grad-CAM counterparts (59%, p < 0.001), trends of spatial overlap with LIME explanations suggest that the spatial focus of models may vary by embryo stage, and by the XAI method utilized. Taken together, these findings demonstrate that accuracy, agreement and interpretability are metrics that may not be consistently aligned. Instead, these performance indicators may represent distinct facets of model behavior that may be used to improve trustworthiness and further development of AI-driven assessment tools.

Although accuracy remains the most frequently reported metric within embryo classification models, clinical reliability is often established by measuring agreement between operators (56). Therefore, we evaluated inter-operator agreement using quadratic-weighted Cohen's kappa (κ) as an additional model performance indicator (57). Intriguingly, despite reduced classification accuracy by both DL models, their agreement to the reference standard remained high (ResNet-34: κ = 0.932, p < 0.001; VGG16: κ = 0.940, p < 0.001). Similarly, pairwise comparisons indicated excellent agreement between both DL models, and each individual embryologist. Based on this finding, we speculate the increased weighted κ may not be representative of the high proportions of adjacent-cell stage errors by DL models. Although the use of a quadratic-weighted Cohen's κ mirrors the ordinal nature of embryo development, this method of assessing agreement penalizes adjacent cell-stage errors less than misclassifications at more distant stages. This notion was reinforced upon investigating per-class agreement between operators, where DL models displayed moderate to fair agreement (κ range: 0.385–0.681) across embryo stages, compared to consistently high embryologist agreement (κ range: 0.778–0.952). Notably, the calculated κ value was unstable at the 2-cell and 3-cell stages, indicating this analysis may be underpowered and requires cautious interpretation. Nonetheless, this divergence revealed that while ResNet-34 and VGG16 achieve comparable classification accuracy, DL models may perform with different intrinsic reasoning that contributes towards unstable agreement (35). In fact, our findings align with another study reporting low inter-AI agreement (21), which remains, to the best of our knowledge, the only investigation reporting agreement metrics between AI-systems. Thus, unlike many studies that focus primarily on performance (12, 13, 58, 59), our results join recent efforts in highlighting the importance of understanding how models reach their decisions.

Recognizing the reduced agreement between ResNet-34 and VGG16 classifications, we further examined embryologists' assessment of Grad-CAM heatmaps to ascertain differences in model spatial attention. Overall, ResNet-34 Grad-CAM outputs were rated as biologically relevant more frequently (89%) compared to VGG16 generated heatmaps (59%, p < 0.001). Furthermore, consistently “good” assessments of ResNet-34 Grad-CAMs at the embryo stage-level implies that the model may prioritize features more congruent with embryologist decision-making than VGG16. However, this interpretive advantage was not always aligned with accuracy. At the 5-cell stage, 100% of ResNet-34 Grad-CAMs were rated as “good”, yet only 43% of classifications were correct, implying that well-rated explanations of a qualitative nature may not reliably reflect the features driving model predictions (51, 60, 61). Crucially, these findings raise the concern of interpretability without accuracy, and empirically validate several concerns raised by van Royen et al. (39) regarding the deployment of XAI in clinical settings. In fact, our results exemplify the risk of “illusory understanding” that arises when embryologists are presented with visually convincing post-hoc explanations that appear plausible, but may obscure underlying discrepancies in model behavior (62). In such cases, explanatory visualizations may actually impede clinical skepticism by overinterpretations of causality and an increased risk of confirmation bias (39). Conversely, VGG16 achieved high accuracy at the morula stage (94%) despite 70% of its Grad-CAM outputs being rated “poor,” thereby reflecting accuracy without interpretability. Despite a dearth of literature surrounding these discrepancies within embryo-related tasks, correct predictions that arise from spurious or non-biological features are known to undermine trust and implementation within a range of healthcare settings (60, 61, 63, 64). Importantly, the observed divergence in interpretability between ResNet-34 and VGG16 Grad-CAM outputs occurred despite similar overall accuracy, thus demonstrating that accuracy and interpretability may represent distinct dimensions of model behavior rather than interchangeable indicators of reliability. Taken together, our findings underscore the need for caution when integrating saliency-based XAI tools into embryo assessment workflows, where explanation quality metrics should be evaluated independently from model reliability, instead of a proxy for model performance. This highlights a meaningful gap between model performance and interpretive agreement that warrants further research in embryo-related tasks.

Further, we expanded our investigation to determine whether independent XAI methods, Grad-CAM and LIME, converged on salient features using IoU-based spatial overlap. Overall, we observed that ResNet-34 exhibited a greater degree of median overlap between Grad-CAM and LIME explanations (26.6%) compared to VGG16 (25.2%, p = 0.006). Upon stage-wise assessment, however, significant variations in median overlap were only observed at the morula (ResNet-34: 22.5% vs. VGG16: 0%, p < 0.001) and blastocyst stage (ResNet-34: 21.0% vs. VGG16: 17.0%, p = 0.006). Given that both models achieved high accuracy at these stages, the limited spatial overlap between explanations is unexpected and prompts reconsideration of whether saliency-based spatial attention serves as a reliable proxy for classification accuracy. Indeed, based on embryologists’ qualitative assessment of Grad-CAM outputs, the majority of VGG16 explanations at the morula stage were “poor”. Yet, the complete absence of overlap with LIME at this stage suggests that while Grad-CAM visualizations appeared uninformative to embryologists, LIME may have captured alternative, potentially relevant regions of interest. Even so, perfect classification accuracy at the blastocyst stage encapsulating mostly “good” Grad-CAM explanations, resulted in the lowest IoU for both models. Importantly, this highlights that XAI tools may capture different, potentially arbitrary, features of the same input (64, 65), rather than converging on causal factors driving classification decisions. Furthermore, this raises concerns about whether XAI methodologies offer genuine mechanistic insight that is grounded in model reasoning, or simply provide a veneer of interpretability that satisfies human intuitions (63). Further, we speculate that that within the range of developmental stages explored here, the morula and blastocyst present with the most distinct embryo morphologies, at relatively low numbers. This raises concerns regarding overfitting, which may have also contributed to their increased accuracy (66). It is also important to acknowledge that LIME is inherently stochastic, such that super pixel boundaries may vary across runs, thereby limiting the stability of IoU-based comparisons (67, 68). Collectively, these findings illustrate that agreement between post-hoc XAI methods is not necessarily informed by accuracy but may prove useful in the enhancement of human-perceived interpretability.

Collectively, this work provides a systematic evaluation of accuracy, agreement and interpretability within embryo-stage classification, demonstrating that each component represents an independent dimension of model performance that may diverge within practical tasks. Notably, we highlight that accuracy may arise without biologically grounded interpretive focus, but also that observed interpretive focus lacks application-based and functionality-grounded evaluation (69). Granting clinical reliability remains largely based on inter-operator agreement, our findings highlight how agreement metrics can quantitatively and qualitatively vary between equally performing models (35). To our knowledge, this is the first investigation to evaluate embryo-level XAI outputs through direct embryologist review, thereby establishing domain expert focus within interpretability assessments, rather than model performance metrics alone. Notwithstanding, this study harbours several limitations. First, only single-frame images were extracted from retrospectively annotated TLT recordings for the purpose of classification, which may not fully capture the complexity and variability of dynamic embryo development. Additionally, the inclusion of embryos with the most complete morphokinetic annotations upon data collection, in comparison to arrested embryos for example, may have introduced selection bias. Moreover, subjective evaluation of Grad-CAM explanations by embryologists may have introduced confirmation bias towards the biological relevance of generated heatmaps. Similarly, although morphological classification is strongly associated with reproductive potential, it is not equivalent to the primary end-goal of live birth, which limits its clinical generalizability. As such, further research should prioritize the use of large, multicenter datasets to quantitatively and qualitatively interpret model behaviors that are tailored towards the intended domain. For example, as studies strive to create objective measures of explainability, the quantitative use of saliency map-based segmentation shows promise within user interpretability (35, 60, 70). However, our findings caution the calibration of trust within AI predictions based on the perceived quality of model explanations alone, as the observed dissociation between XAI interpretability and accuracy may, paradoxically, lead to further compromised decision-making than “black box” models. Overall, establishing robust standards and alignment between accuracy, agreement and interpretability will be essential for the responsible implementation of AI-tools within IVF laboratories.