An American Medical Association (AMA) survey reported that 93% of physicians believe prior authorization (PA) has a negative impact on patient care, and 95% reported an association with professional burnout (1). Twenty four percent of survey respondents reported serious consequences of PA, including permanent impairment, hospitalization, or death (1). PA is a utilization management process used by health insurers to determine whether a prescribed medication, procedure, or service meets predefined criteria for coverage before reimbursement is approved (2, 3). In practice, PA typically requires clinicians to submit detailed documentation outlining the clinical indication, accurate diagnostic coding, prior treatment history, justification of medical necessity, and supporting evidence aligned with clinical guidelines or regulatory labeling (4–6). Physicians spend an estimated 12–13 h per week submitting approximately 39 PA requests (1, 7).

Physicians report that PA delays or insurer hurdles lead patients to discontinue care, with up to 78% noting that they have seen treatment abandonment in their own practice (8). Delays created by PA often leave physicians with less time for direct patient care and add to the administrative load required to keep treatment plans moving. These interruptions can affect patients' health and also add pressure on clinicians, contributing to frustration, reduced efficiency, and higher operating costs, as noted in recent reports from the AMA and the Healthcare Business Management Association (HBMA) (7, 8). Although detailed data for individual specialties remain limited, emerging evidence shows that PA requirements create similar challenges across a broad range of clinical conditions. A recent systematic review conducted by Johns Hopkins University that examined 25 primary studies from the United States found consistent reports of treatment delays, disease exacerbations, avoidable hospitalizations, prolonged inpatient stays, and worse survival outcomes, particularly in cancer care (9). These effects were seen across oncology, cardiology, behavioral health, pediatrics, rheumatology, and infectious diseases, suggesting that the burden of PA is not confined to any single area of practice (9).

Conceptually, PA and AI-assisted documentation can be understood through three complementary theoretical perspectives (2, 10–12). Health services research frames PA as a utilization management mechanism intended to promote evidence-based and cost-conscious care, while simultaneously introducing administrative burden and potential barriers to timely treatment when documentation quality is suboptimal (12). Theories of human-AI collaboration view large language models as assistive tools designed to augment clinician workflows rather than replace clinical judgment, particularly for documentation-intensive tasks. In parallel, trust and reliability frameworks for clinical decision support systems emphasize that adoption depends on consistent performance across core domains such as factual accuracy, transparency of reasoning, and appropriate use of supporting evidence (13–15). Together, these perspectives underscore the importance of systematically evaluating AI-generated documentation in high-stakes administrative contexts.

Given the complexity and high stakes of PA submissions, the quality of documentation, including the accuracy of clinical reasoning, diagnostic coding, and supporting references, is critical to successful approval (2–6). In recent years, there has been increasing interest in the use of artificial intelligence (AI), particularly large language models (LLMs), to support documentation-intensive tasks in medicine (16–19). These tools are already being applied to generate a range of clinical communications, including faxed submissions to health plans, letters to patients, referral notes, and inter-provider correspondence among them (20). For example, roughly one-quarter of pediatricians report using such systems to help prepare letters, request PAs, or support patient and family education (21).

Despite growing interest in clinical applications, the performance of LLMs in real medical settings remains inconsistent. A recent systematic review found that ChatGPT answered medical questions correctly only about half the time, with an overall accuracy of 56% (95% CI, 51%–60%) (22). In nephrology, LLMs accuracy varied widely, with dietary potassium and phosphorus classification ranging from 66%–100% across models (23). In a separate citation study, correct references accounted for only 3%–38% of outputs across different LLMs (24). These inconsistencies underscore the need for caution when applying LLM-generated text to high-stakes administrative tasks such as PA submissions, where factual errors, incorrect coding, or inappropriate citations may directly affect patient access to care.

ChatGPT, developed by OpenAI, is a large language model designed to generate human like text and assist with information retrieval and writing tasks (25). It is now commonly used for summarizing articles (26), drafting academic writing (27), and organizing complex information (28, 29). Several published studies, including prior work by our group, have evaluated LLM performance in medicine and nephrology, focusing on general medical question answering, specialty-specific knowledge, educational use cases, and the reliability of generated citations (17, 30–37). However, these studies have largely emphasized feasibility, general accuracy, or informational tasks rather than structured evaluation of payer-facing administrative documents.

Given the substantial administrative burden associated with PA and the rapid adoption of AI-based drafting tools, it is timely to examine whether LLMs can meaningfully reduce workload without compromising accuracy, clinical reasoning, or safety. The differentiating factor of the present study is its focus on a high-stakes, payer-facing administrative use case rather than general medical question answering or narrative documentation. The objective of this study was to systematically evaluate the quality of prior authorization letters generated by ChatGPT-5 for commonly encountered nephrology scenarios, with specific assessment of factual accuracy, ICD-10 coding correctness, citation validity, and the strength of clinical reasoning using a standardized, task-specific framework.

We developed 29 standardized nephrology clinical scenarios involving medications commonly requiring PA. Each scenario included a diagnosis coded using the International Classification of Diseases, 10th Revision (ICD-10), with medication indications supported by FDA-approved labeling (38), KDIGO guideline (39), and major randomized trials. PA letters were generated using ChatGPT-5 (OpenAI) accessed via the web-based interface, using default model settings (no user-specified temperature, top-p, or token limits) (Supplementary Material 1). No system-level messages, custom instructions, retrieval tools, plugins, or external reference materials were provided beyond the standardized prompt. All outputs were captured verbatim and were not edited or post-processed prior to evaluation.

Among the 29 ChatGPT-5-generated PA letters reviewed, most met the basic expectations for accuracy, diagnostic coding, reference use, and clinical justification (Figure 2). Only 1 letter (3.5%) contained a false statement. ICD-10 coding was correct in 23 letters (79.3%), and 27 letters (93.1%) used valid citations. Clinical reasoning was the strongest domain, with 26 letters (89.7%) rated as strong and the remaining 3 letters (10.3%) still rated as adequate. Errors were generally narrow in scope and concentrated in predictable areas such as chronic kidney disease (CKD) staging, citation accuracy, and omission of key safety considerations. A more detailed examination of each domain is described below.

This study provides an early look at how ChatGPT-5 performs when asked to generate PA letters for commonly used nephrology medications. Overall, the model produced PA letters that were mostly accurate, appropriately structured, and supported by strong clinical reasoning. Most submissions were free of factual errors, the majority used correct ICD-10 coding, and almost all cited reasonable sources. These findings suggest that, at baseline, the tool can create letters that resemble what clinicians routinely prepare in practice.

The pattern of errors is instructive. The single false statement identified in the sample was not subtle. Citing the INNO2VATE vadadustat trial (41) in support of a phosphate binder illustrates how confidently the model may pull in unrelated information. Although infrequent, these errors underscore the importance of verifying sources to ensure accuracy and consistency in clinical documentation. A similar issue was seen in the nedosiran scenario in which data from the lumasiran program were used instead of the PHYOX trials (44, 45) that form the evidence base for nedosiran. These errors may seem small but can weaken the credibility of a PA submission, especially when reviewers scrutinize supporting literature.

ICD-10 coding represented another area where lapses were more common (48). The most frequent mistake involved staging CKD (42). Several scenarios with eGFR values around 45 to 50 mL/min per 1.73 m² were labeled as stage 3b rather than the correct stage 3a based on the provided eGFR values. In the rituximab scenario, both a code for GPA with renal involvement and a code without renal involvement were listed together, creating a confusing and internally inconsistent diagnostic picture. These errors did not alter the clinical intent of the letters but did reduce the overall precision of diagnostic coding. For payers who rely on correct coding to determine benefit coverage, this level of inconsistency can introduce unnecessary friction.

Despite these shortcomings, clinical reasoning was the model's strongest domain. Nearly ninety percent of letters offered a well-constructed explanation grounded in patient-specific details and aligned with guideline or FDA-approved labeling criteria. The remaining letters fell short not because the indications were incorrect but because important considerations were omitted. These included gastrointestinal tolerability for tenapanor (46), discussion of safety monitoring for nedosiran, and meningococcal vaccination or prophylaxis for eculizumab (47). These are elements a human author would typically include automatically because they are tied to risk-mitigation strategies or boxed warnings. Their absence is a reminder that LLM-generated text may overlook details that clinicians regard as routine.

Several established quantitative metrics have been proposed to evaluate LLM performance, including accuracy scores, factual consistency measures, and text similarity benchmarks (35, 49, 50). However, most of these metrics were developed for general natural language processing tasks or medical question answering and do not adequately capture the task-specific requirements of PA letters (35, 49–51). In the PA context, clinically meaningful errors often relate to diagnostic coding precision, appropriateness of cited evidence, or completeness of payer-facing clinical justification, domains that are not well reflected by generic LLM performance metrics (35, 49, 50). Accordingly, we intentionally adopted a domain-specific, clinician-centered evaluation framework that prioritizes attributes directly relevant to PA review and approval. This approach is intended to complement, rather than replace, existing LLM benchmarking strategies by emphasizing practical reliability in a high-stakes administrative setting (15, 35, 49, 50, 52).

Our findings highlight a concern but encouraging picture. ChatGPT-5 reliably captures the overall framework of PA justification and articulates it clearly in many cases. At the same time, it can overlook coding nuances, substitute incorrect trial data, or omit safety considerations that are critical for payer review. These limitations are manageable if the tool is used to support rather than replace clinician judgment. With thoughtful clinician supervision, AI-generated drafts may reduce the administrative time required to prepare PA letters, but they cannot yet be relied on without thorough review. As health systems consider adopting such tools, attention to validation, error-checking workflows, and clinician sign-off will be essential to ensure safe and accurate use.

Several study limitations should be emphasized. The evaluation was based on a fixed set of 29 standardized and relatively straightforward nephrology scenarios, which do not capture the full complexity, ambiguity, or longitudinal context of real-world PA requests. All letters were generated using a single model version at a single time point, and performance may vary across model updates or alternative architectures. In addition, the study focused on document-level quality metrics and did not assess payer-facing outcomes such as approval rates, turnaround times, or the need for appeals.

Future research should extend this work to more complex and less structured clinical scenarios, evaluate performance across multiple LLMs and model versions, and examine real-world payer responses to AI-assisted PA submissions. Integrating electronic health record data, along with automated checks for diagnostic coding accuracy, drug–evidence alignment, and safety requirements, may further improve reliability (53, 54). Prospective studies measuring administrative efficiency, clinician workload, and downstream payer outcomes will be critical to defining the appropriate role of LLMs in supporting PA workflows.

ChatGPT-5 generated PA letter drafts that were generally accurate and well structured, with acceptable clinical reasoning in most scenarios. The errors mainly involved coding, reference selection, and incomplete safety discussions. Our findings highlight the need for careful review before use. With appropriate oversight, LLM-generated drafts may help reduce administrative burden, but they are not yet reliable enough to be used without clinician verification.