This study focuses on Chinese-language (Mandarin) skin cancer educational materials intended for lay readers and patients, reflecting a common real-world scenario in which the public uses consumer-facing LLMs to obtain health information. We selected Doubao, DeepSeek, Wenxin Yiyan, and Tongyi Qianwen as representative publicly accessible LLM products widely available to Chinese users, and included GPT-5 as a widely used international reference model to enable cross-ecosystem comparison under identical prompts. Given the Chinese-language outputs, we employed the Chinese version of the Patient Education Materials Assessment Tool for Printable Materials (c-PEMAT-P) to support linguistically appropriate quality evaluation.

Generative Large Language Models (LLMs) offer unprecedented automation for creating health education content at scale (5). While their text-generation capacity is evident, their responsible deployment in healthcare necessitates rigorous, multidimensional validation (5). A pivotal question is whether LLMs consistently achieve the requisite integration of substantive quality and public readability (6). Existing research typically examines these metrics in isolation, often assuming correlation or treating them as a unitary construct (7, 8). This approach leaves a critical gap: the lack of systematic investigation into the explicit relationship between the quality and readability of LLM-generated health information. Without this understanding, the promise of LLMs in public health communication remains uncertain, creating a potential hazard since deficiencies in either dimension undermine essential educational objectives.

The pressing demand for reliable, accessible skin cancer information in dermatology drives interest in LLMs as a scalable source for generating consistent educational content (6–8). Initial LLM benchmarking utilizes standardized instruments—such as the c-PEMAT-P (for understandability/actionability) and the GQS (Global Quality Scale, for overall quality)—alongside readability formulas (e.g., Flesch–Kincaid, SMOG) (1, 9). However, the prevailing trend is to report these metrics separately, yielding a fragmented view of performance (10). The fundamental and largely overlooked problem concerns the inherent link between these evaluative axes: Does superior quality inherently correlate with improved readability, or are they distinct characteristics influenced independently by model choice and content subdomain? Resolving this empirical gap is critical for developing optimized deployment protocols. If coupled, quality benchmarking suffices; if decoupled (the central hypothesis of this study), the strategy requires nuance, as high factual quality may yield overly complex text, and vice versa. Thus, a focused analysis is essential to disentangle the effects of model architecture, content topic, assessed quality, and measured readability within AI-generated skin cancer education. While large language models (LLMs) show promise in generating high-quality content for public health, it is essential to address the risks of hallucinations and biases inherent in these models. Such issues could result in the spread of incorrect or biased information, which may compromise the reliability of the generated content.

In this study, content quality refers to the clarity, usefulness/actionability, and perceived reliability of patient-oriented information (assessed using validated instruments such as c-PEMAT-P and GQS), whereas readability refers to cognitive accessibility approximated by surface-text complexity captured by readability formulas. In patient education practice, materials are commonly expected to use plain language, minimize jargon, and present information with clear structure and actionable steps. Because widely cited grade-level readability targets are primarily defined for English texts and there is no universally accepted grade-level threshold for Chinese patient materials, the readability indices in this study are interpreted mainly as comparative benchmarks across models and topics, and are complemented by c-PEMAT-P to reflect patient-oriented understandability/actionability.

Accordingly, we formulated three operational research questions (RQs). RQ1 (model effect): Under identical prompts, do different LLMs generate patient-oriented skin cancer education materials with significantly different quality and readability? RQ2 (topic effect): Do quality and readability vary across content categories (etiology/risk, clinical manifestations, diagnosis/screening, treatment/prognosis, and prevention/patient education)? RQ3 (quality–readability relationship): Across all generated outputs, is content quality associated with readability? We hypothesized that (H1) both quality and readability would differ by model, (H2) readability (and potentially quality) would differ by content category, and (H3; central hypothesis) quality and readability would show weak or non-significant correlations, indicating partial decoupling.

Addressing this need, our study compared five prominent LLMs (Doubao, DeepSeek, Wenxin Yiyan, Tongyi Qianwen, and GPT-5), evaluating their skin cancer educational outputs using validated quality scales (C-PEMAT, GQS) and seven readability indices. Results show that while models yielded materials of comparable understandability/actionability (C-PEMAT), overall perceived quality (GQS) differed substantially; GPT-5 achieved the highest score. Crucially, readability fluctuated significantly by LLM and content category (e.g., Prevention/Treatment vs. Clinical Manifestations), revealing no single model excelled across all metrics. Furthermore, correlation analysis established that quality and readability are largely independent dimensions. This disparity—that high quality does not necessitate high readability—indicates that optimizing AI-generated health education materials requires tailored, multifaceted strategies, necessitating joint consideration of the selected LLM and the specific content topic for intended public health applications. While expert evaluations offer valuable insights, it is equally important to validate AI-generated content with end-users (patients) to ensure its clarity and relevance in real-world healthcare contexts (11).

Data utilized in this investigation were exclusively synthetic, derived from Large Language Models (LLMs). This methodology involved no human or animal experimentation, biological samples, or personal identifying information. Consequently, ethical review is waived under established academic standards.

To evaluate the performance of Large Language Models (LLMs) in dermatological knowledge dissemination, two clinical experts compiled 20 frequently asked questions (FAQs) concerning skin cancer on November 14, 2025. These questions were systematically grouped into five categories: (1) Etiology and Risk Factors, (2) Clinical Manifestations and Classification, (3) Diagnosis and Screening, (4) Treatment and Prognosis, and (5) Prevention and Patient Education. The complete set is detailed in Table 1. Researchers subsequently queried five publicly accessible LLMs—Doubao, DeepSeek, Wenxin Yiyan, Tongyi Qianwen, and GPT-5—with these FAQs. The resulting responses were analyzed across three critical metrics: readability, reliability, and quality. It must be acknowledged that using a fixed set of 20 FAQs may not fully capture the diversity of prompts and real-world contexts, thereby limiting the generalizability of the results. Furthermore, as the study was conducted in Chinese, its findings may not be entirely applicable to multilingual or cross-cultural settings. An overview of the study workflow is provided in Supplementary Figure S1.

To strengthen reproducibility of the benchmarking procedure, we standardized the querying workflow across all five LLMs. For each of the 20 FAQs (Table 1), the same prompt template was applied to every model, and each FAQ was submitted in an independent session (i.e., a new chat/conversation was initiated for every FAQ) to avoid contextual carryover. The prompt template specified the intended audience (lay public/patients), the requested output language (Mandarin Chinese), and basic formatting requirements (e.g., plain language, structured bullet points/headings, and avoidance of unnecessary jargon) (full prompt text provided in Appendix/Supplementary material).

Where the platform allowed user control of generation parameters (e.g., temperature, top-p, maximum output length/tokens), these parameters were set identically across models; when such parameters were not exposed in the public interface, default platform settings were used and explicitly recorded. We also recorded whether web browsing/search augmentation was enabled (if applicable and user-configurable) and ensured that the same setting was applied consistently within each model. The access date (and interface/channel) for each LLM was documented to account for possible model updates over time.

The unit of analysis was one model response per FAQ (20 FAQs × 5 models = 100 outputs). Each output was exported as plain text for subsequent quality evaluation (c-PEMAT-P and GQS) and readability analysis. Complete model-specific access information and generation settings are summarized in Supplementary Table S1, and the full prompt set is provided in the Supplementary appendix to enable replication.

Readability analysis of large language model (LLM) responses utilized multiple formulas derived from the Text Readability Assessment Tool.¹ Acknowledging the current absence of an authoritative gold standard or consensus regarding the most reliable metric, this study consequently employs the mainstream system documented extensively in prior literature (9, 12, 13).

This study employed seven established readability metrics—the Coleman-Liau Index (CL), Linsear Write (LW), Automated Readability Index (ARI), Simple Measure of Gobbledygook (SMOG), Gunning Fog Index (GFOG), Flesch Reading Ease Score (FRES), and Flesch–Kincaid Grade Level (FKGL)—to quantify the comprehensibility and linguistic proximity of large language model (LLM)-generated output to natural, everyday speech. All outputs were generated in Mandarin Chinese and analyzed without translation. As these readability formulas were originally developed for English, we use them primarily for relative comparison of text complexity across models/topics rather than for absolute grade-level interpretation in Chinese; this is complemented by c-PEMAT-P for linguistically appropriate assessment.

The study did not conduct external factual validation by comparing the generated content with current clinical guidelines or other authoritative sources, which could have strengthened the robustness of the findings.

The assessment of literature quality and accessibility utilized two instruments: the c-PEMAT-P (Chinese version of the Patient Education Material Readability Assessment Tool) and the GQS (Global Quality Score) (1, 14–16). The c-PEMAT-P comprises 24 indicators across two dimensions: Comprehensibility (16 items, focusing on logical structure and terminology accessibility) and Practicality (8 items, evaluating specific action guidance and audience suitability). Scoring is binary (0/1), resulting in a 0–24 total score, where higher values signify greater user accessibility. For analysis, c-PEMAT-P was calculated per output as the summed 0–24 total score and then summarized as mean (±SD) across outputs within each model/category, yielding non-integer means. The GQS employs a 1–5 point scale, ranging from ‘Poor quality’ (Score 1: illogical content, no practical value) to ‘Excellent quality’ (Score 5: rigorous logic, significant practical value). Two independent clinical experts, each with over 3 years of relevant clinical experience, conducted the evaluations. Inter-rater reliability was quantified using Cohen’s Kappa (κ), interpreted as follows: κ > 0.75 indicated excellent agreement, 0.40 ≤ κ ≤ 0.75 acceptable agreement, and κ < 0.40 poor agreement. Discrepancies between the initial two raters were resolved through consensus discussions, with a third expert adjudicating the final rating when necessary. Verification confirmed that both the c-PEMAT-P and GQS scales achieved κ > 0.75, confirming excellent inter-rater reliability (17, 18). While the study did not include real-world comprehension tests or patient validation, it does not directly assess health literacy from a patient’s perspective.

All statistical analyses were performed using IBM SPSS Statistics 25.0 (IBM Corp., Armonk, NY, USA). Continuous outcomes were summarized as mean ± standard deviation when appropriate, or as median with interquartile range for non-normally distributed variables. Normality was assessed using the Shapiro–Wilk test, and homogeneity of variance for ANOVA was assessed using Levene’s test. For outcomes meeting parametric assumptions, one-way analysis of variance (ANOVA) was used to compare groups, followed by Tukey’s honestly significant difference (HSD) test for post-hoc pairwise comparisons. For outcomes violating normality and/or homoscedasticity assumptions, the Kruskal–Wallis H test was used for multi-group comparisons, followed by Dunn’s post-hoc pairwise comparisons. Multiple comparisons were adjusted using the Bonferroni correction. Correlations between quality metrics (GQS and c-PEMAT-P) and readability indices were assessed using Pearson’s correlation coefficient (r). All tests were two-tailed, and p < 0.05 was considered statistically significant.

To compare the performance of five mainstream large language models (Doubek, DeepSeek, Wenxin Yiyan, Tongyi Qianwen, and GPT-5) when generating skin cancer popularization materials, we initially assessed model-level distinctions by utilizing the following metrics extracted from Table 2: quality scores (C-PEMAT and GQS) and readability indices (ARI, FRES, GFOG, FKGL, CL, SMOG, and LW).

At the model level, GQS scores exhibited significant variation (F = 40.50, p < 0.001). GPT-5 recorded the highest mean GQS score (4.40), underscoring its superior overall content quality compared to lower scores registered by Doubao (3.45), DeepSeek (3.30), Tongyi Qianwen (2.20), and Wenxin Yiyan (~2.10). Conversely, C-PEMAT scores did not differ significantly across models ($p = 0.065$), as evidenced by mean values clustered within a restricted range (8.70–9.40), suggesting broad equivalence in understandability and actionability across the five models.

Readability metrics demonstrated greater heterogeneity across models. Six indices showed significant inter-model variation: ARI (GPT-5 median: 18.22 vs. DeepSeek: 16.14; χ² = 17.82, p = 0.001), FRES (χ² = 16.52, p = 0.002), CL (χ² = 28.32, p < 0.001), SMOG (χ² = 13.02, p = 0.011), GFOG (χ² = 11.89, p = 0.018), and FKGL (χ² = 11.09, p = 0.026). The LW index, however, failed to reach statistical significance (p = 0.102). Collectively, these findings confirm that the readability of skin cancer educational content is model-dependent, though no single model demonstrated consistent superiority across all indices. Post-hoc pairwise comparisons were conducted using Dunn’s test with Bonferroni correction; detailed pairwise Z statistics and adjusted p values are provided in Supplementary Table S2.

Analysis of quality and readability by content category (Table 3) revealed a significant divergence in C-PEMAT scores (F = 3.86, p = 0.006). The Prevention and Treatment category recorded the highest mean score (9.50), considerably exceeding the lower mean (8.65) of the Clinical Manifestation category. This disparity suggests model-generated prevention-and treatment-related content was inherently more understandable and actionable. Conversely, GQS scores exhibited no significant difference across content categories (p = 0.336).

Readability indices, however, varied markedly across content categories, exhibiting significant differences across all tested metrics: ARI (χ² = 18.19, p = 0.001), FRES (χ² = 32.88, p < 0.001), GFOG (χ² = 42.51, p < 0.001), FKGL (χ² = 25.88, p < 0.001), CL (χ² = 21.92, p < 0.001), SMOG (χ² = 31.84, p < 0.001), and LW (χ² = 27.19, p < 0.001). The Prevention and Treatment category, overall, demonstrated the most favorable performance across several of these metrics.

These findings demonstrate that GPT-5 achieves the highest overall Content Quality Score (GQS) in skin cancer science popularization. Conversely, readability depends significantly on both model choice and content category. Consequently, optimizing educational materials for specific applications necessitates the joint consideration of model selection and content tailoring.

Because these readability formulas were originally developed for English and there is no universally accepted grade-level threshold for Chinese patient materials, we interpret the indices primarily for relative comparisons across models and content categories. For transparency and comparability with prior English-language health communication literature, we additionally summarized the proportion of outputs with FKGL ≤ 8 (and ≤ 6) as an exploratory analysis; detailed proportions are reported in Supplementary Table S4, S5.

The C-PEMAT score distributions across the five models were visualized (Figure 1) to assess differences in the educational content’s understandability and actionability. While visual inspection suggested variation in distribution shape and density—GPT-5 scores, for instance, appeared more concentrated at higher values—all pairwise comparisons proved statistically non-significant (“ns”). This outcome indicates that the C-PEMAT quality of educational materials generated by these large language models (LLMs) was broadly similar, demonstrating no statistically demonstrable advantage for any single model.

Figure 2 illustrates the distribution of Global Quality Scale (GQS) scores, which reveal statistically significant disparities across the five models, a stark contrast to C-PEMAT results. Specifically, GPT-5 registered the highest median GQS score, correlating with a demonstrable upward shift in its distribution. Subsequent pairwise analyses confirmed GPT-5’s consistent superior ranking over all tested competitors—Doubao, DeepSeek, Wenxin Yiyan, and Tongyi Qianwen (all statistically significant, labeled “****”)—whereas Wenxin Yiyan consistently exhibited comparatively lower scores. These findings collectively establish GPT-5’s superior overall perceived quality (GQS) for skin cancer educational content.

Correlation analyses were conducted using Pearson’s correlation coefficient (r) across all outputs (n = 100). The full correlation matrix (r) with two-tailed p values is provided in Supplementary Table S3, and the heatmap is shown in Figure 3. A correlation heatmap was employed to analyze the correlation structure between quality assessments and textual readability, specifically comparing the two quality metrics (GQS and C-PEMAT) against seven readability indices (ARI, FRES, GFOG, FKGL, CL, SMOG, and LW) across all skin cancer educational texts.

The analysis revealed a strong positive correlation between GQS and C-PEMAT, demonstrating consistency in ranking overall content quality. Substantial shared variance in capturing textual complexity was similarly evident across several readability indices, where strong positive correlations emerged between SMOG and CL, GFOG and FKGL, and ARI and FRES.

By contrast, GQS and C-PEMAT demonstrated only weak, often negative, correlations with most readability indices; the associations were weakest with LW. The weak correlation between content quality (GQS/C-PEMAT) and readability indices suggests that these two factors function independently. While high-quality content can be comprehensive and accurate, it may not always meet simplified readability requirements, indicating that improving one aspect does not necessarily lead to improvement in the other. Consequently, optimizing AI-generated skin cancer educational materials requires complementary strategies that target these elements separately, as high quality does not correlate with inherent readability.