This retrospective observational study assessed the concordance between AI-generated recommendations from Scholar GPT and clinical decisions made by board-certified thoracic surgeons in a high-volume tertiary university hospital. All outpatient thoracic surgery consultations conducted over one week were included, without case selection or exclusion criteria, ensuring a representative sample of routine clinical practice. Visits were performed by certified thoracic surgeons; Six surgeons participated during the study week.

Data collected included patient demographics (age, sex), consultation type (first visit or follow-up), consultation modality (in-person or telemedicine), and the final clinical diagnosis established by the surgeon. The full distribution of first vs. follow-up and in-person vs. telemedicine is reported in Results.

Scholar GPT, (OpenAI, version accessed January 2025) a large language model based on GPT-4 architecture, was selected due to its ability to access and synthesize information from biomedical databases, including PubMed and evidence-based clinical guidelines. The model was accessed via a dedicated medical consultation interface during January 2025, with output traceability. The system had no connectivity to the hospital EHR and no local fine-tuning with institutional data.

For each consultation, the attending surgeon documented the primary reason for the visit, relevant clinical history, imaging results, and the diagnostic and therapeutic plan. The same clinical scenario was entered into Scholar GPT, prompting the AI to provide a diagnostic assessment and treatment recommendation. AI responses were recorded verbatim without modifications. The AI received a structured text summary (history, comorbidities/medication, and key findings transcribed from imaging reports); no image files were uploaded, and no external oncology notes were provided. Physical examination findings, when available, were included as text.

Additionally, the number of clarification questions generated by the AI before formulating a recommendation was documented to evaluate the model's reasoning process. Across the 81 consultations, the mean number of clarification turns was 1.02. Two independent thoracic surgeons reviewed the AI output and assigned a concordance score according to the predefined scale.

A specific 6-point ordinal concordance scale (0–5) was developed to assess the level of agreement between AI-generated recommendations and the surgeons' decisions, focusing on both diagnosis and treatment alignment:

0: No diagnosis or treatment proposed.

Descriptive statistics were used to summarize the distribution of concordance scores between Scholar GPT's recommendations and the surgeons' decisions. Normality of continuous variables was assessed using the Shapiro–Wilk test. As the concordance scores did not follow a normal distribution (p < 0.001), non-parametric tests were employed. Comparisons between groups were performed using the Mann–Whitney U-test for two-group comparisons, while categorical variables were assessed with chi-square tests. Additionally, subgroup analyses were performed to compare concordance scores based on diagnosis type, consultation modality (in-person vs. telemedicine), and visit type (first-time vs. follow-up). Statistical significance was set at p < 0.05. Given the exploratory scope and one-week sampling, no a priori sample-size/power calculation was undertaken. Formal normality testing and inter-rater reliability metrics (e.g., ICC) were not computed; these are acknowledged as limitations and targets for future work.

This study was retrospective and based on anonymized clinical data. All patients had previously signed a general institutional consent authorizing the use of their anonymized data for research purposes. According to institutional policy, no additional ethical approval was required. No patient-identifiable information was entered into the AI system.

A critical aspect that requires thorough discussion, as highlighted by the reviewers, is the clinical safety implications of AI decisions, particularly the potential risks in low-concordance cases. While our safety-relevant review did not identify AI recommendations that would be clearly harmful if executed verbatim, the potential for suboptimal management in low-concordance scenarios represents a significant concern that extends beyond our immediate findings.

Recent comprehensive evaluations have demonstrated that current LLMs are not ready for autonomous clinical decision-making (18). Hager et al. found that LLMs tend to make hasty decisions without consistently following diagnostic guidelines, often omitting essential physical examinations and misinterpreting basic laboratory results—fundamental errors that pose serious risks to patient safety without extensive clinician supervision (18). This finding is particularly relevant to our study, as it suggests that even in cases where we observed moderate concordance, the underlying reasoning process may be fundamentally flawed.

The “black box” nature of LLMs further complicates their integration into surgical decision-making contexts. Unlike traditional clinical decision support tools where the reasoning pathway can be traced and validated, LLMs operate through complex neural networks that make their decision-making process opaque and uninterpretable (19). This lack of transparency poses significant challenges for surgical practice, where understanding the rationale behind a recommendation is crucial for patient safety and medicolegal accountability. As Xu et al. argue, the unexplainability feature of medical AI systems may cause harm that is currently underestimated, particularly when these systems make incomprehensible mistakes that are difficult to detect (19).

The implications of this opacity are particularly concerning in thoracic surgery, where decisions often involve high-stakes interventions with significant morbidity and mortality risks. When an AI system recommends a specific surgical approach or suggests delaying intervention, surgeons need to understand the underlying reasoning to make informed decisions about patient care. The inability to interrogate the AI's decision-making process undermines the fundamental principle of evidence-based medicine and may lead to either inappropriate reliance on AI recommendations or complete rejection of potentially valuable insights.

This study offers a real-life evaluation of LLM-assisted consultation in a high-volume outpatient service, contributing to a growing body of work emphasizing AI's role in enhancing healthcare efficiency (20). However, important limitations must be noted, and their impact on result interpretation requires careful analysis.

First, LLMs operate without direct access to physical examination findings or dynamic patient interaction, factors essential for nuanced decision-making in surgical practice. This limitation is particularly significant in thoracic surgery, where physical examination findings such as respiratory mechanics, chest wall deformities, and lymph node palpation often provide crucial diagnostic information that cannot be captured in text-based summaries.

Second, the lack of interpretability in LLM outputs—commonly referred to as the “black box” problem—poses a barrier to clinician trust and accountability (21). The absence of baseline intra-group concordance assessment (AI reproducibility and inter-surgeon agreement) as control measures, as noted by the reviewers, represents a significant methodological limitation that prevents us from establishing the reliability of our concordance measurements.

Third, our study lacks formal normality testing before applying t-tests and the absence of inter-rater correlation coefficient analysis. These methodological deficiencies impact the interpretation of our results and should be addressed in future studies.

Furthermore, AI outputs may vary based on prompt phrasing, regional practices, and training data, as demonstrated in comparative analyses between generalist LLMs and domain-specific models (22, 23). Although Scholar GPT achieved high scores in our study, its performance cannot be generalized without further multi-centre validation. The small sample size of 81 cases is insufficient for multiple subgroup analyses, resulting in inadequate statistical power for many of our diagnostic-specific comparisons.

A major limitation of this study is the absence of inter-rater reliability assessment. While two independent thoracic surgeons assigned concordance scores, only the final consensus scores were recorded, preventing calculation of inter-rater agreement statistics such as Cohen's kappa or ICC. This represents a significant methodological deficiency that limits confidence in the reliability of our primary outcome measure.

Despite these limitations, the potential applications of AI in thoracic surgery continue to expand. Recent comprehensive reviews have highlighted AI's growing role in enhancing diagnostic accuracy, surgical precision, and postoperative care in thoracic surgery (24). AI applications now span the entire perioperative period, from preoperative risk assessment and surgical planning to intraoperative guidance and postoperative monitoring (5).

In the preoperative phase, AI has demonstrated promise in imaging analysis, tumor classification, and surgical candidacy assessment. During surgery, AI-assisted augmented reality systems have shown feasibility in robotic lung surgery, achieving high accuracy in gesture recognition and potentially improving surgical precision (25). Postoperatively, AI assists in pathology assessment, complication prediction, and long-term outcome modelling.

However, as noted in recent scoping reviews, significant challenges remain, including the need for larger, more diverse datasets, standardization of AI applications across institutions, and development of robust validation frameworks specific to thoracic surgery (26).

LLMs may serve as cognitive aids, particularly in settings with limited subspecialty coverage, such as remote clinics or after-hours consultations. Yet, AI should complement—not replace—clinical judgment, especially in patient-specific scenarios involving comorbidities or psychosocial factors (27).

Future efforts should focus on fine-tuning LLMs using thoracic surgery-specific datasets, integrating multimodal inputs (e.g., imaging, labs), and enabling real-time feedback loops from clinical users (11). Ethical frameworks and regulatory guidelines will also be essential to ensure responsible AI deployment in patient care (28). Most importantly, future studies must address the methodological limitations identified in our work, including larger sample sizes, multi-centre designs, formal inter-rater reliability assessments, and comprehensive safety evaluations.

The development of explainable AI systems that can provide transparent reasoning for their recommendations will be crucial for gaining clinician trust and ensuring safe integration into surgical practice. Until these challenges are addressed, LLMs should be used with extreme caution in clinical decision-making, with mandatory human oversight and validation of all AI-generated recommendations.