Kidney disease is a global health issue, affecting millions of people worldwide. Chronic Kidney Disease (CKD) is particularly prevalent, with an estimated 850 million people suffering from this condition (Jager et al., 2019). CKD is characterized by a gradual loss of kidney function over time, which can progress to End-Stage Kidney Disease (ESKD) if not properly managed. ESKD requires dialysis or kidney transplantation, both of which pose significant health and economic burdens (Jha et al., 2023). The field of nephrology, which deals with the diagnosis and treatment of kidney diseases, has been at the forefront of medical innovation for decades. However, like many areas of healthcare, it faces significant challenges in ensuring equitable access to care and outcomes for all patients (Vanholder et al., 2023). The effect is more pronounced among minorities especially among women from these groups right from their inclusion in research studies; this is further exacerbated by existing systemic biases (Pinho-Gomes et al., 2023; Mohammed et al., 2024). In recent years, the concepts of diversity, equity, and inclusion (DEI) have gained increasing prominence in healthcare, with nephrology being no exception. The “Kidney Care for All” initiative exemplifies this shift, advocating for a more inclusive approach to kidney health that addresses disparities and promotes equal access to quality care for all individuals, regardless of their background or socioeconomic status (Pais and Iyengar, 2023). Despite efforts to increase diversity in medical education and practice, the field of nephrology, like many medical specialties, still struggles with underrepresentation of minority groups among its practitioners (Salsberg et al., 2021).

Concurrently, the rapid advancement of Artificial Intelligence (AI), particularly Large Language Models (LLMs) like ChatGPT (Open AI. Introducing Chat GPT, 2024), has begun to reshape the landscape of healthcare decision-making. These AI systems, capable of processing vast amounts of data and generating human-like responses, hold immense potential for augmenting clinical decision-making, streamlining administrative processes, and potentially reducing healthcare disparities (Garcia Valencia et al., 2023). However, the integration of AI into nephrology and other medical fields also raises critical questions about the ethical implications and potential unintended consequences of relying on machine-generated insights for patient care and professional decisions. There are concerns that AI systems, if not properly designed and implemented, could perpetuate or even exacerbate existing health disparities (Omiye et al., 2023; Yang et al., 2024). For instance, if training data for AI models are not sufficiently diverse or representative, the resulting algorithms may perform poorly for certain population subgroups or reinforce biased decision-making patterns (Kuhlman et al., 2020).

Given the significant impact that AI could have on nephrology, it is imperative to assess how well these technologies adhere to DEI principles. This study focuses on evaluating the ethical sensitivity and decision-making capabilities of two versions of ChatGPT (3.5 and 4.0) in nephrology-related scenarios. By examining how these AI models handle DEI considerations, particularly with regard to underrepresented socio-demographic variables such as ethnicity, employment status, education, and religious beliefs, we aim to identify potential risks and areas for improvement in their design and implementation. These attributes, while often underappreciated in traditional clinical decision-making workflows, are critical to fostering equitable care. Our work seeks to advance the dialogue on AI fairness in nephrology by showcasing how ethical AI evaluation must go beyond clinical indicators to include the broader socio-cultural determinants of health.

In the context of AI in nephrology, diversity refers to the inclusion of individuals from varied demographic backgrounds in data and practice; equity involves ensuring fair treatment and access to care and opportunities; and inclusion emphasizes meaningful engagement of underrepresented groups in AI design and implementation. These principles guided the development of our simulation scenarios and the evaluation of AI model responses.

The full set of ChatGPT 3.5 and 4.0 responses to all 80 simulated scenarios is presented in the Online Supplementary Materials. This includes the selected response by each model, whether a refusal occurred (along with the stated rationale), and the consistency of responses across rounds for ChatGPT 4.0.

ChatGPT 3.5 answered all 80 scenarios without refusal, demonstrating a utilitarian approach that did not account for potential ethical conflicts. In contrast, ChatGPT 4.0 declined to make decisions based on potentially discriminatory criteria in 13 scenarios (16.3%) during the first round and in 5 scenarios (6.3%) during the second round. It provided consistent responses across both rounds in 67 scenarios (83.8%), suggesting moderate reliability in its ethical decision-making behavior.

These refusals predominantly involved scenarios in which decisions were explicitly linked to sensitive sociodemographic variables. For example, in one scenario regarding workforce recruitment, the model declined to prioritize candidates based on religious affiliation, stating that such decisions should be made based on qualifications rather than personal beliefs. In another case involving transplant eligibility, it refused to recommend prioritizing patients by immigration status, emphasizing the ethical obligation to treat all individuals equitably. Additional examples included declining to select candidates for living kidney donation promotion based on employment type (e.g., self-employed vs. corporate), or selecting dialysis access strategies based on language proficiency. A complete list of these 13 scenarios and the specific DEI attributes involved is available in online supplementary. These findings underscore the model’s partial sensitivity to ethically contentious prompts, though they also reveal inconsistency, as similar cases were occasionally answered without objection.

Figure 2 demonstrates examples of GPT output regarding critical considerations in nephrology treatment and workforce diversity.

In Figure 2, the left column presents outputs from GPT-3.5, while the right column features responses from GPT-4.0. For GPT-3.5, question #1 inquires about which ethnic group should be prioritized for a new kidney disease treatment. The model recommends prioritizing Black individuals, citing their historical disproportionate impact from kidney disease, which necessitates tailored treatment considerations to address their specific needs. In question #21, regarding employment types to prioritize for promoting living kidney donations, GPT-3.5 suggests focusing on self-employed individuals, as they often have more flexibility in managing their schedules, making it easier for them to accommodate the donation process and recovery time. On the right side, GPT-4.0 addresses question #36, which asks about the employment status to consider for new diabetic nephropathy treatments. The model advocates for prioritizing individuals in full-time employment, noting that this demographic is more likely to have access to employer-sponsored health insurance, thus improving equity in treatment access. In question #78, concerning which religious belief should be factored in when selecting candidates for nephrologist positions to ensure workforce diversity, GPT-4.0 emphasizes the importance of considering candidates from diverse religious backgrounds, specifically highlighting Christianity, to foster an inclusive environment that enhances cultural competency in patient care.

The findings of this study reveal a marked difference in ethical decision-making capabilities between ChatGPT 3.5 and 4.0 in nephrology-related scenarios. ChatGPT 3.5 consistently selected treatment choices predicted to yield the best outcomes across all scenarios, demonstrating a utilitarian approach that incorporated various DEI factors. However, it did not refuse to make decisions in any scenario, reflecting a lack of sensitivity to potentially discriminatory criteria. In contrast, ChatGPT 4.0 declined to make decisions based on potentially discriminatory criteria in 16.25% of scenarios, explicitly stating that DEI factors should not affect decisions about treating patients or hiring nephrology staff. While this shows an improvement in ethical sensitivity, the relatively low refusal rate was unexpected, highlighting areas for further enhancement.

The observed improvement in ChatGPT 4.0’s handling of DEI-sensitive decisions underscores the iterative progress in AI ethical alignment, likely influenced by updated reinforcement learning techniques and refined safety layers in the model’s training pipeline (Sheikh et al., 2025; Alam et al., 2025; Miao et al., 2025). While not yet optimal, ChatGPT 4.0’s behavior reflects a greater sensitivity to fairness principles, selectively deferring decisions that could result in biased clinical or hiring recommendations. These changes are promising, as they suggest that ethical behavior in LLMs can be enhanced over successive versions. Nevertheless, the relatively low overall refusal rate indicates that existing mechanisms remain insufficient to fully safeguard against implicit bias, and further tuning—both algorithmic and regulatory—is essential to prevent harm.

Existing literature on AI in healthcare highlights both the potential benefits and risks associated with AI integration (Nazer et al., 2023; Ali et al., 2023). Previous studies have demonstrated AI’s capacity to enhance clinical decision-making and patient outcomes, but they also raise concerns about the perpetuation of biases inherent in training data (Omiye et al., 2023; Ayoub et al., 2024; Parikh et al., 2019). Obermeyer et al. (2019) showed in their study of racial bias present in a popular commercial algorithm for risk stratification used in the healthcare system. Our findings align with these concerns, illustrating that while AI can make clinically sound decisions, its consideration of ethical issues, particularly DEI, remains imperfect. The differences in performance between ChatGPT 3.5 and 4.0 likely stem from advancements in model training and updates in ethical guidelines integrated into the AI’s framework. At the same time, we acknowledge that the relatively low refusal rate points to the need for further refinement of AI models and the development of more robust ethical frameworks. To address this, our discussion has been expanded to emphasize that while the improvements observed in ChatGPT 4.0 are encouraging, they also serve as a catalyst for continued research. Future directions will focus on refining training datasets, enhancing algorithmic sensitivity to DEI factors, and incorporating interdisciplinary insights to align AI recommendations more closely with clinical judgment. Such measures are anticipated to support the responsible integration of AI into clinical practice, ensuring that these systems become reliable tools for promoting equitable healthcare outcomes.

Beyond the specific case of ChatGPT, DEI-related concerns are increasingly relevant across the broader spectrum of AI applications in nephrology. AI systems in clinical settings are not only used for decision support but also influence patient triage, diagnostic accuracy, treatment planning, and workforce recruitment (Miao et al., 2025; Koirala et al., 2025; Pham et al., 2024). If AI models are not carefully designed with DEI principles in mind, they risk reinforcing existing disparities in kidney care, including biases related to race, socioeconomic status, and geographic location. Moreover, many AI models are developed in high-income countries with datasets that may not be representative of the global population, further exacerbating inequities in nephrology care. Addressing these issues requires greater scrutiny of model training processes, the diversity of datasets, and the interpretability of AI-driven recommendations (Ferryman et al., 2023; Meng et al., 2022). Future research will be directed toward evaluating these alternative systems to compare their performance in handling DEI considerations. This broader approach will not only enrich our understanding of AI’s potential benefits and pitfalls in nephrology but also inform the development of more robust regulatory and ethical frameworks for AI integration across the discipline. Additionally, interdisciplinary collaboration between AI developers, ethicists, nephrologists, and policymakers will be essential in ensuring that AI systems are aligned with clinical needs while promoting health equity (Abbasgholizadeh Rahimi et al., 2024).

The implications of these findings for the integration of AI in nephrology and broader healthcare contexts are significant. While AI has the potential to enhance decision-making and improve patient outcomes, this study underscores the importance of robust ethical frameworks and careful regulation. The fact that even the more advanced ChatGPT 4.0 model failed to recognize potentially discriminatory factors in the majority of cases emphasizes the need for human oversight and the importance of using AI as a supportive tool rather than a replacement for human judgment in critical medical decisions. Furthermore, this study highlights the need for ongoing research and development in AI ethics, particularly in healthcare applications. As AI models continue to evolve rapidly, it is crucial to regularly assess their ethical decision-making capabilities and identify areas for improvement. This may involve developing more sophisticated training datasets representing the entire population at large that better represent diverse populations and ethical scenarios, as well as refining the algorithms that govern AI decision-making processes (Gaonkar et al., 2023; Mudgal and Das, 2020; Ueda et al., 2024).

Additionally, there is a clear need for interdisciplinary collaboration between AI developers, ethicists, healthcare professionals, and policymakers to ensure that AI systems are designed and implemented in ways that promote equity and avoid perpetuating or exacerbating existing healthcare disparities (Ueda et al., 2024; Dankwa-Mullan, 2024; Mennella et al., 2024; Walsh et al., 2023). Moreover, the study’s findings have important implications for medical education and professional development in nephrology and other healthcare fields. As AI becomes increasingly integrated into clinical practice, it is essential that healthcare professionals are trained not only in the use of these technologies but also in critically evaluating their outputs and understanding their ethical limitations (Garcia Valencia et al., 2023; Ueda et al., 2024). This includes developing skills in recognizing potential biases in AI-generated recommendations and maintaining a commitment to equitable, patient-centered care. By fostering a workforce that is both technologically adept and ethically grounded, the field of nephrology can work toward harnessing the benefits of AI while upholding its commitment to diversity, equity, and inclusion in all aspects of patient care and professional practice (Figure 3).

The development of standardized DEI-related ethical guidelines for AI in nephrology is essential to ensuring that AI-driven decision-making aligns with the principles of fairness, transparency, and inclusivity (Abbasgholizadeh Rahimi et al., 2024; Solomonides et al., 2022). These guidelines should provide a structured framework requiring AI developers to incorporate diverse, representative training datasets, implement bias detection and mitigation strategies, and prioritize explainability so that clinicians can critically assess AI-generated recommendations (Chin et al., 2023). Furthermore, interdisciplinary collaboration among AI developers, nephrologists, bioethicists, and policymakers is necessary to create AI systems that are both clinically effective and socially responsible. AI models should undergo continuous auditing to evaluate their adherence to DEI principles, with regulatory oversight ensuring that biases are addressed over time. Healthcare institutions must also take an active role in developing policies that support responsible AI use in both patient care and workforce recruitment, ensuring that AI-driven decisions contribute to equitable healthcare (Abbasgholizadeh Rahimi et al., 2024). Future research should focus on refining these frameworks to optimize AI deployment in a manner that enhances healthcare access and outcomes for all patients.

Ensuring the ethical performance of AI models in nephrology requires a continuous monitoring and evaluation process that extends throughout the entire lifecycle of these systems (Abbasgholizadeh Rahimi et al., 2024). AI outputs can shift over time due to changes in training data, algorithm modifications, or evolving clinical practices. To prevent the unintended reinforcement of biases, a structured oversight framework should be established (Economou-Zavlanos et al., 2024). This framework should include real-time bias detection mechanisms integrated into AI deployment systems, periodic audits by interdisciplinary teams consisting of nephrologists, AI ethicists, data scientists, and regulatory experts, and ongoing user feedback loops that allow clinicians to report discrepancies or concerns regarding AI-generated recommendations. Additionally, regulatory bodies and healthcare institutions should conduct periodic evaluations to ensure that AI-driven decisions remain equitable and do not disproportionately disadvantage any patient populations. Transparency in reporting AI performance metrics such as disparities in AI decision-making across demographic groups will be critical in building confidence in AI-assisted nephrology (Chin et al., 2023; Economou-Zavlanos et al., 2024).

To ensure responsible AI use in nephrology, a structured process for addressing ethical concerns must be established. This should include both a dedicated ethics committee and a formal reporting mechanism for healthcare professionals (Economou-Zavlanos et al., 2024). The ethics committee should be an interdisciplinary body composed of nephrologists, AI ethicists, data scientists, legal experts, and patient advocates. Its role would be to evaluate ethical concerns related to AI applications, provide guidance on ethical AI implementation, and develop strategies to mitigate bias and ensure fairness. In parallel, a structured reporting mechanism should be created to allow healthcare professionals to flag AI-generated recommendations that appear biased, ethically questionable, or inconsistent with established clinical guidelines (Economou-Zavlanos et al., 2024). Reports should be reviewed systematically, with clear channels for follow-up and corrective action. Regular audits of AI performance, along with clinician and patient feedback, should inform ongoing improvements to AI models (Economou-Zavlanos et al., 2024). Establishing these processes will help maintain ethical integrity in AI-driven nephrology and ensure that AI tools are used in a manner that upholds DEI principles while enhancing patient care.

Active collaboration with patient communities is a critical component of AI development and deployment in nephrology. Engaging patients throughout the AI lifecycle allows their perspectives to be incorporated into model design, leading to AI systems that are more aligned with the diverse needs of the nephrology population (Lammons et al., 2023; Adus et al., 2023). Establishing patient advisory panels can provide valuable insights into AI-generated recommendations, helping to identify potential gaps or biases in decision-making. Additionally, focus groups with patients from different demographic and socioeconomic backgrounds can highlight concerns, expectations, and trust levels regarding AI-driven healthcare tools (Adus et al., 2023). Transparency in AI decision-making is essential, and efforts should be made to present AI-generated recommendations in a way that is understandable and actionable for both clinicians and patients. Patient advocacy organizations play a key role in facilitating these collaborations by acting as intermediaries between AI developers, healthcare providers, and patient communities.

This study has several notable strengths. It is one of the first systematic evaluations of how large language models respond to ethical scenarios involving DEI considerations within nephrology. The simulation-based design allows for structured comparison across different model versions (ChatGPT 3.5 and 4.0) using controlled variables. The inclusion of real-world DEI variables such as religion, race, employment, and immigration status provides ecological validity and highlights relevant ethical challenges in clinical care and workforce recruitment. Additionally, the full disclosure of simulated cases and AI responses enhances transparency and reproducibility.

While the simulated cases were constructed to reflect a wide range of real-world nephrology scenarios with DEI relevance, we recognize that the cases were authored by a limited number of clinicians and may unintentionally reflect biases in scenario framing or emphasis. Certain cultural or regional contexts, particularly those affecting underrepresented populations globally, may not be fully captured. These limitations could influence how AI models respond and may underrepresent ethical nuances faced by specific communities. Further studies involving larger and more diverse datasets, as well as real-world clinical trials, are necessary to validate and expand upon these results. Future research should focus on expanding case diversity by incorporating a broader range of scenarios and involving a more diverse group of nephrologists in case development. Implementing and evaluating AI models in actual clinical settings will be crucial to assess their performance and ethical sensitivity in real-time decision-making. Additionally, continuous refinement of AI models is essential to enhance their ability to recognize and appropriately handle DEI considerations (Ueda et al., 2024). These efforts will inform the development of robust regulatory frameworks and training protocols that emphasize ethical decision-making and DEI sensitivity. Although our simulated scenarios were created and evaluated by two nephrologists, we recognize that additional insights from a broader group of clinicians may further enrich the evaluation of AI outputs. In future work, we plan to incorporate perspectives from a diverse range of practicing clinicians to refine our findings and ensure that our ethical evaluation framework reflects varied clinical viewpoints. This approach is anticipated to further strengthen our assessment of AI performance and contribute to the development of thoughtful regulatory guidelines for AI integration into clinical practice.

To enhance DEI sensitivity in AI models, we recommend several concrete steps: (1) incorporating demographically diverse and representative training datasets; (2) embedding DEI-specific audit mechanisms within model evaluation pipelines; (3) using adversarial testing that challenges models with ethically complex scenarios; and (4) requiring transparent reporting of performance disparities across subgroups. Achieving these goals will require sustained interdisciplinary collaboration. Clinicians can define real-world ethical constraints; data scientists can implement fairness-aware algorithms; ethicists can guide value alignment; and patient advocates can ensure community relevance. This collective approach is essential to creating AI systems that are both clinically robust and socially responsible.

In summary, this study demonstrates that while advancements have been made in AI ethical sensitivity, as seen in ChatGPT 4.0, there is still considerable room for improvement. The relatively low rate of refusal to engage in potentially discriminatory decision-making underscores the need for ongoing refinement of AI models. By addressing these challenges and implementing robust regulatory and training frameworks, we can better ensure that AI systems not only enhance clinical decision-making but also uphold the principles of diversity, equity, and inclusion. These efforts will contribute to more equitable healthcare delivery and better outcomes for all patients in nephrology.