This review was prospectively registered in PROSPERO (CRD420251090315) on 09/07/2025 and conducted in accordance with the registered protocol. No deviations from the protocol occurred.

We defined eligibility criteria a priori in line with PRISMA 2020 recommendations.

Outcomes were predefined and categorized into two distinct domains. Primary outcomes included objective diagnostic performance and reliability measures, such as accuracy, sensitivity, specificity, positive and negative predictive values, area under the receiver operating characteristic curve (AUC), Top-k accuracy, and agreement coefficients (e.g., Cohen's κ and Gwet's AC1), reflecting diagnostic validity relative to the applied reference standard.

Secondary outcomes comprised subjective and perception-based measures of model output quality and usability, including explanation quality, plausibility ratings, interpretability, narrative usefulness, and perceived clinical utility. These outcomes represent qualitative assessments of reasoning support and user experience and do not constitute measures of diagnostic correctness.

Primary and secondary outcome domains were analyzed and reported separately to avoid conceptual overlap and overinterpretation of clinical relevance.

A comprehensive search was performed in PubMed, CINAHL, Embase, Web of Science, and ScienceDirect covering the period from January 2022 to 20 July 2025, using MeSH terms and free-text keywords related to large language models and oral lesions. No date limits were applied. English-language restriction was applied due to the limited comparability of cross-language LLM performance. Full search strategies are provided in Supplementary Appendix 1 (PRISMA-S).

Records were deduplicated in EndNote and screened independently by two reviewers in Rayyan. Discrepancies were resolved by a third reviewer. Inter-reviewer agreement was calculated using Cohen's κ for both title/abstract and full-text stages. The single included study with overlapping authorship was evaluated exclusively by reviewers without overlapping authorship, with discrepancies resolved by an independent third reviewer.

Data extraction was conducted independently using a standardized, piloted form and verified by a third reviewer. Extracted items included study design, sample characteristics, model version, input modality, comparator, lesion type, and outcomes (accuracy, sensitivity, specificity, AUC, agreement coefficients, explanation quality).

Risk of bias was appraised using an adapted QUADAS-2 tailored for LLM-based diagnostic evaluations, assessing case selection, index test, reference standard, and flow/timing. Judgments were categorized as low, high, or some concerns. Summary and traffic-light visualizations were generated (Supplementary Appendix 2).

Due to anticipated clinical and methodological heterogeneity across large language model families, input modalities (text-only, image-only, and multimodal), lesion domains (oral mucosal and jaw lesions), comparator reference standards, and outcome definitions, meta-analysis was not undertaken. Instead, a structured narrative synthesis was performed. Findings were organized according to: (1) LLM family, (2) input modality, (3) comparator reference standard, and (4) lesion category. Descriptive statistics were used to summarize, as distinct outcome domains, study-level diagnostic performance metrics, agreement/reliability measures, and utility-related or subjective outcomes as reported in the original studies. No pooled estimates (e.g., means or summary effect sizes) were calculated; results were presented descriptively using individual study values and ranges, as appropriate, to avoid inappropriate aggregation across heterogeneous study designs.

The search identified 1,178 records (PubMed 476; CINAHL 172; EMBASE 50; Web of Science 380; Google Scholar 100). After removing 291 duplicates, 887 records were screened; 867 were excluded. 20 full texts were sought; 1 was not retrieved and 2 were excluded (wrong setting/population). 17 studies were included for qualitative synthesis (Figure 1).

Inter-reviewer agreement was substantial, with a Cohen's κ of 0.82 for title and abstract screening and 0.88 for full-text eligibility assessment.

The studies (published 2024–2025) evaluated a range of large language models (LLMs), including ChatGPT-3.5/4/4o/4 V, o1-preview, DeepSeek-V3/R1, Gemini 1.5 Pro/Flash, Claude 3/3.5, LLaMA 3.2, and Copilot (Table 2).

The included studies (2024–2025) evaluated a range of large language models, including ChatGPT-3.5/4/4o/4V, o1-preview, DeepSeek-V3/R1, Gemini 1.5 Pro/Flash, Claude 3/3.5, LLaMA 3.2, and Microsoft Copilot. Input modalities varied, with text-only approaches in 11 studies, image-only in 6 studies, and multimodal configurations in 6 studies, with some evaluating more than one modality. Lesion focus ranged from single-entity cohorts, such as OLP, OPMLs, leukoplakia, and syndromic lesions, to broader mixed oral mucosal case sets. Reference standards included histopathology/biopsy in four studies, expert consensus in seven studies, published clinical solutions in four studies, and hybrid standards in two studies. Sample sizes ranged from 16 to 1,142 cases (Table 2).

Comparative performance statements reported in this section are restricted to within-study evaluations; cross-study differences are presented descriptively and should not be interpreted as direct model rankings. Table 3 organizes outcomes into two distinct domains: diagnostic performance metrics (objective measures of accuracy and agreement) and subjective perception-based outcomes (plausibility and reasoning quality assessments). This separation prevents conceptual overlap and reduces the risk of overinterpretation of clinical relevance. Objective diagnostic performance varied substantially according to model family, input modality, lesion complexity, and study design (Table 3).

Diagnostic accuracy varied substantially across models, input modalities, and prompting strategies (Table 3). Early-generation models such as ChatGPT-3.5 demonstrated modest Top-1 performance (25%–65%), whereas newer models showed variable accuracy ranging from 36%–89% across different studies and lesion types, with ChatGPT-4 and ChatGPT-4o achieving peak performance of 78%–89% in some clinical datasets, compared with 87%–100% for expert reference standards (16–20).

Multimodal input was associated with the largest performance gains. GPT-4 multimodal achieved Top-1 and Top-2 accuracies of 81% and 96%, while multimodal implementations of ChatGPT-4 and ChatGPT-4 V reached peak accuracies of 86%–93%, exceeding performance observed under text-only (20%–81%) and image-only (27%–87%) conditions (21–24).

Top-k analysis further demonstrated improved diagnostic coverage. In heterogeneous oral lesion cohorts, ChatGPT-4o achieved Top-1, Top-3, and Top-5 accuracies of 38%, 76%, and 84%, with comparable performance reported for DeepSeek-V3 (25). Differential diagnosis tasks showed ChatGPT-4 outperforming Microsoft Copilot (FDx/DDx: 70%/74% vs. 46%/60%), while remaining below expert-level final diagnosis performance (26).

Prompt-guided and trained configurations improved accuracy across multiple models, with GPT-4o achieving 67%–69% accuracy and Gemini Flash up to 80.5% under example-guided prompting, and trained GPT-4o achieving 59%–77% accuracy compared with lower performance for Claude (15%–50%) and Chat-Diagrams (64%–68%) (27, 28). Progressive performance gains across model generations were also observed, with correct first diagnosis rates increasing from 40% (ChatGPT-3.5) to 62% (ChatGPT-4) and 80% (o1-preview) (29). Additional classification tasks reported approximately 60% accuracy for ChatGPT-4 (30).

Only a limited number of studies formally evaluated diagnostic output consistency. Tomo et al. reported moderate repeatability for primary diagnostic hypotheses (Cohen's κ ≈ 0.53) and fair consistency for alternative diagnostic classifications (18). In a separate multimodal evaluation, Suárez et al. demonstrated substantial diagnostic repeatability for ChatGPT-4o using Gwet's AC (AC = 0.834) across repeated runs (22). Collectively, these findings indicate that structured clinical context improves output stability; however, variability in final diagnostic classification remains evident.

Subjective and perception-based evaluations primarily assessed output plausibility, reasoning coherence, and perceived usability rather than objective diagnostic performance. Single study incorporating expert Likert-scale scoring reported higher perceived plausibility and contextual grounding of diagnostic reasoning when multimodal inputs were used compared with image-only conditions (31). Importantly, these outcomes reflect expert perception of output quality and should not be interpreted as measures of diagnostic accuracy or clinical validity.

Clinical utility was primarily described in terms of decision-support functionality rather than autonomous diagnostic use. Multimodal studies reported that LLM outputs assisted with structuring differential diagnoses, summarizing key diagnostic features, and organizing clinical information to support preliminary clinical reasoning and educational workflows (22, 24). Output stability across repeated runs was considered favorable for training and triage-oriented applications (22). Expert plausibility scoring indicated moderate-to-high perceived usefulness of diagnostic reasoning outputs, with higher ratings observed for more contextually grounded responses (31). However, limitations related to prompt sensitivity, reduced performance in complex cases, and lower interpretability under image-only conditions were consistently highlighted (22, 24). Overall, LLMs were positioned as adjunctive tools that may support clinical reasoning but should not replace expert clinical judgment.

Based on the QUADAS-2 assessment, two studies were rated as high risk of bias, primarily due to non-representative or synthetic data sources. Six studies were rated as low risk, with clear case definitions, appropriate reference standards, and consistent test procedures. The remaining nine studies were judged to have some concerns, most commonly due to convenience sampling, non-uniform reference standards, or incomplete reporting. These findings reflect reasonable but variable methodological quality across the included literature (Figures 2–3).

Despite encouraging trends, the current evidence base is constrained by substantial methodological heterogeneity and threats to validity. Many studies relied on curated benchmark datasets, synthetic case vignettes, or highly selected clinical examples, introducing spectrum bias and case enrichment effects that may inflate apparent diagnostic performance. Reference standards varied widely across studies, ranging from histopathology-confirmed diagnoses to expert consensus and published case solutions, introducing verification bias and subjective variability. Additional heterogeneity in prompting strategies, case presentation formats, lesion categorization schemes, and reported outcome definitions further limited cross-study comparability and precluded quantitative meta-analysis. These findings highlight the urgent need for standardized evaluation frameworks guided by emerging reporting standards such as STARD-AI and QUADAS-AI.

Translation of experimental performance into real-world clinical deployment remains uncertain. Practical implementation barriers include workflow integration challenges, variable image quality, incomplete clinical histories, regulatory considerations, and the risk of model drift over time. Furthermore, routine oral diagnostic practice involves diagnostic uncertainty, multimorbidity, and atypical lesion presentations that are underrepresented in current evaluation datasets. As such, high benchmark accuracy should not be interpreted as evidence of chairside diagnostic reliability. Prospective, multicenter clinical validation studies will be essential to establish safety, generalizability, and clinical impact.

From a clinical and educational perspective, LLMs show promise as adjunctive decision-support tools, particularly in primary care and resource-limited settings where specialist access is constrained. They may assist clinicians in generating differential diagnoses, identifying red flags, and prioritizing further investigations. In dental education, LLMs may support diagnostic reasoning training and case-based learning. However, safeguards are required to mitigate automation bias, hallucination risk, and overreliance on model-generated output.

Future research should prioritize prospective multicenter validation, blinded reader comparison studies, standardized benchmarking protocols, and the development of native multimodal architectures capable of jointly processing clinical text, photographs, radiographs, and histopathology. Additional emphasis should be placed on explainability, bias mitigation, and continuous external validation frameworks to support safe clinical translation.