All interactions were conducted in Macau, China, between November 1 and November 10, 2025. To approximate typical end user usage, all models were evaluated via their official web interfaces rather than via APIs. The evaluated products included ChatGPT using GPT-5, released on August 7, 2025; Gemini 2.5 Pro, released on June 17, 2025; DeepSeek (V3.1), released on August 21, 2025; Grok 4, released on July 9, 2025; and Microsoft Copilot. All products were commercially deployed closed source systems, so training data and model parameters were not accessible and generation settings such as temperature and top p remained at platform defaults. Each question was asked in a single turn without follow up prompts and submitted in a new chat session to minimize carry over effects. Outputs were captured verbatim via scripted logging with manual cross checking. We did not actively enable any explicit browsing or citation modes. However, in closed source web products the backend retrieval and routing behavior cannot be fully verified; Microsoft documentation indicates that Copilot Chat runs on GPT-5 by default and automatically routes prompts to the best performing models for each task, so the exact model and routing path used for any given response cannot be reliably identified or controlled, which may limit strict reproducibility.

This study employed a double-blind design to ensure objective evaluation and minimize bias. The data from each model’s output were anonymized by an independent third party to eliminate any identifying information. The scoring process was carried out by two trained clinical evaluators who were blinded to the model type, prompt variations, search augmentation, generation parameters, and any metadata. Both evaluators, each with more than 10 years of clinical experience and based at tertiary (Grade A) hospitals in China, independently assessed the outputs. They had extensive experience in obstetrics and some research experience in perinatal maternal mental health. Prior to scoring, the evaluators received standardized training on the use of the assessment instruments. Any discrepancies in ratings were adjudicated by a third senior physician. Reliability was assessed using five validated tools; detailed scoring criteria are provided in Table 2.

DISCERN is used to evaluate consumer health publications, particularly the quality of information on treatment choices (32). The instrument comprises 16 items grouped into three domains: reliability of the publication, quality of information on treatment options, and an overall rating. Each item is scored from 1 (lowest) to 5 (highest), yielding a total score ranging from 16 to 80.

The EQIP (Ensuring Quality Information for Patients) tool assesses the quality of online patient information. Developed by Moult et al. (33) in 2004, EQIP includes 20 items designed to evaluate the completeness, presentation, and usability of written patient information, and it has been widely used in health information quality assessment studies (34, 35). Each item is rated using a binary yes/no criterion (“yes” = 5 points; “no” = 0 points). The overall score is calculated as the proportion of “yes” responses multiplied by 100, resulting in a total score ranging from 0 to 100.

The JAMA benchmarks, proposed by the Journal of the American Medical Association in 1997, provide a simple framework for assessing the quality of online health information sources (36). They evaluate four transparency components: authorship, attribution, disclosure, and currency. Each criterion is scored as 1 if present and 0 if absent, yielding a total score from 0 to 4, with higher scores indicating better information quality.

GQS (The Global Quality Score), developed by Bernard and colleagues in 2007, is a subjective overall rating tool commonly used to assess the quality of online health information, particularly in internet-based health promotion research (37). It provides a rapid global judgment of medical accuracy, completeness, and educational usefulness. GQS is typically rated on a 5-point Likert scale, where 1 indicates very poor/misleading information and 5 indicates excellent/highly useful information.

HONCODE is maintained by the Health on the Net Foundation (founded in 1995) and is one of the oldest and most widely recognized certification standards for online medical and health information. It assesses compliance with eight ethical and transparency principles. Each principle is scored as 0, 1, or 2 (0 = not compliant; 1 = partially compliant; 2 = fully compliant), producing a total score ranging from 0 to 16, with higher scores indicating greater reliability and transparency (38).

Readability was evaluated using six indices, each reflecting a different aspect of textual complexity. The Automated Readability Index (ARI) estimates the grade level required to comprehend a text based on sentence length and word length. The Gunning Fog Index (GFI) reflects reading difficulty using average sentence length and the proportion of complex words. The Coleman–Liau Index (CLI) estimates grade level based on characters per word and sentences per text. The Original Linsear Write Formula (OLWF) captures the balance between word length and sentence length on a 0–100 scale. The Linsear Write Grade Level Formula (LWGLF) reflects difficulty based on the proportion of simple versus complex words. The FORCAST Readability Formula (FRF) evaluates readability based on word complexity and sentence structure.

The National Institutes of Health (NIH) recommends that public health information be written at approximately a sixth to seventh grade reading level (39).therefore, we adopted the sixth-grade level as our benchmark. For grade-based indices (ARI, GFI, CLI, LWGLF, and FRF), the output approximates U.S. School grade levels, with higher scores indicating more difficult text. Accordingly, a target value of 6 was used, and scores closer to 6 indicate better alignment with recommended public readability levels. For OLWF, higher scores indicate easier reading; based on the published correspondence between OLWF values and sixth-grade materials, we used OLWF = 94 as the sixth-grade benchmark. Using these benchmarks, we compared model outputs against the reference values in the Results: grade-based indices substantially above 6 indicate higher reading burden, whereas OLWF values substantially below 94 indicate insufficient readability.

Means and standard deviations were used as descriptive statistics to summarize reliability and readability scores. Given the repeated measures design, we used the Friedman test to assess overall differences among models, followed by paired Wilcoxon signed rank tests with Bonferroni adjustment for post hoc pairwise comparisons. Effect sizes (r) were calculated to quantify the magnitude of pairwise differences. Inter rater agreement between the two physicians for reliability ratings was assessed using the interclass correlation coefficient (ICC). Data analyses were performed in R (version 4.5.2).

Reliability was assessed using five instruments: DISCERN, EQIP, JAMA, GQS, and HONCODE. Inter rater agreement between the two obstetricians was high across instruments, with ICC values of 0.787 for DISCERN, 0.847 for GQS, 0.797 for JAMA, 0.729 for EQIP, and 0.777 for HONCODE. Figure 1 summarizes model scores, and Table 3 reports overall between model differences using Friedman tests. Significant differences were observed for DISCERN, EQIP, and HONCODE (all p < 0.001), whereas overall differences were not significant for JAMA and GQS.

For DISCERN, which reflects the completeness and quality of information about treatment options, models differed significantly in performance. Grok4 and Copilot achieved the highest scores, suggesting that their responses were more clearly structured and comprehensive, particularly in describing treatment options as well as potential risks and benefits. ChatGPT-5 scored the lowest on this dimension; overall scores fell in the good to fair range, implying room for improvement in content depth and information organization.

For EQIP, DeepSeek performed best in clarity of expression, formatting standardization, and logical flow, followed by Copilot and Grok4. Gemini-2.5 scored the lowest, reflecting limitations in readability, information structure, and explanatory detail. Overall, except for Gemini-2.5 and ChatGPT-5, most models were within the good range, indicating generally acceptable clarity but with opportunities for further optimization.

For JAMA, which emphasize authority and transparency, Copilot and Gemini-2.5 performed relatively better, particularly with respect to authorship identification, source attribution, and disclosure statements, aligning more closely with standards commonly expected in medical publishing. Grok4 scored the lowest on this dimension, which may reflect more limited or conservative practices in providing explicit attribution and disclosures within its responses.

For GQS, Grok4 and Copilot scored slightly higher than other models in overall perceived information quality, including usefulness and user perceived value. ChatGPT-5 was relatively weaker, suggesting that it may require improvement in conveying key clinical details effectively. Most models were rated in the fair range, and mean differences were modest, indicating that the generated health information was generally adequate but not consistently high quality.

For HONCODE, which evaluates ethical and transparency principles of health information, Copilot and Grok4 performed best, scoring significantly higher than other models, suggesting better alignment with expectations related to transparency, accountability, and information governance. Gemini-2.5 and ChatGPT-5 were slightly above average, whereas DeepSeek scored the lowest, indicating that ethical and transparency aspects may warrant further strengthening for some models.

Based on the Friedman results shown in Table 3 and the effect size analysis in Supplementary document, significant between model differences were observed in overall quality assessments, including DISCERN (χ² = 70.79, p < 0.001), EQIP (χ² = 19.08, p < 0.001), and HONCODE (χ² = 60.38, p < 0.001). These findings indicate that the evaluated models differed in their performance on quality constructs relevant to complex medical information.

Post hoc pairwise comparisons further showed that the magnitude of these differences varied across instruments. For DISCERN and HONCODE, several comparisons yielded large effect sizes (r > 0.6), particularly for model pairs involving Grok4 and Copilot, suggesting substantial differences in reliability related performance. For EQIP, effect sizes spanned large, moderate, and small ranges, indicating that this instrument provided meaningful discrimination among models. In contrast, no statistically significant overall differences were detected for the simpler, more subjective scales such as GQS (p = 0.396) and JAMA (p = 0.385). Consistent with this, post hoc analyses generally showed small or negligible effect sizes, suggesting that model performance was broadly comparable when evaluated using these basic quality benchmarks.

Table 4 presents Bonferroni corrected post hoc pairwise comparisons of reliability scores among the five large language models. Significant pairwise differences were primarily observed for DISCERN, EQIP, and HONCODE, whereas few significant differences were detected for JAMA and GQS. Notably, DeepSeek and ChatGPT-5 showed no significant differences across most reliability metrics, while Grok4 differed significantly from several other models on multiple measures.

Across the five AI models and six readability indices, detailed scores and Friedman test results are presented in Table 5 and Figure 2. None of the models achieved the recommended sixth-grade benchmark on ARI, GFI, CLI, OLWF, LWGLF, or FRF. Unlike the grade level indices, higher OLWF scores indicate easier readability; therefore, OLWF values substantially below the sixth-grade benchmark reflect increased reading difficulty. Overall, this indicates that text generated by all models exceeded the complexity expected for sixth grade reading level.

Grok4 and Gemini-2.5 obtained the highest scores on multiple indices, suggesting that they produced the most complex text and thus imposed the highest reading burden. Copilot and DeepSeek had slightly lower scores, but remained well above recommended levels, indicating marginally easier text that may nonetheless be unsuitable for individuals with lower health literacy. ChatGPT-5 was closer to the overall average on some indices, yet its outputs still tended toward higher complexity, particularly on CLI and ARI. Collectively, these findings suggest that model generated responses were generally complex and associated with a relatively high readability threshold.

The Friedman test results indicated statistically significant between model differences for ARI, CLI, OLWF, LWGLF, and FRF (all p < 0.01), whereas GFI did not reach statistical significance (p = 0.064). The non-significant GFI result likely reflects a shared characteristic across models: when responding to perinatal depression–related queries, all models inevitably used a substantial number of multisyllabic medical terms (e.g., “postpartum,” “antidepressant”), resulting in broadly similar levels of complex vocabulary. In contrast, indices more directly related to reading difficulty, such as ARI, CLI, and FRF, showed distinguishable differences across models, suggesting variability in linguistic complexity and reading burden.

By comparison, differences were smaller for indices that emphasize sentence structure and word combination features, such as GFI and LWGLF, indicating that models were broadly similar in average sentence length and in their overall control of complex word proportions. Effect size analyses further suggested that the magnitude of these differences was generally small to moderate, implying limited practical separation in readability across models. Post hoc pairwise comparisons showed that differences varied by readability index and did not yield a consistent pattern of superiority across all measures.

There were substantial differences across large language models in how they handled safety critical perinatal mental health scenarios, particularly with respect to clinical appropriateness, risk communication, and referral to professional care. Although all models included general disclaimers that they cannot replace clinicians, they varied markedly in the depth of their clinical framing and in how closely they aligned with established care pathways.

Copilot and Grok4 provided the most comprehensive responses and were most consistent with clinical practice. Both models repeatedly emphasized that treatment decisions during pregnancy require individualized risk–benefit assessment under joint supervision by obstetric and psychiatric clinicians. Their responses explicitly acknowledged the risks of untreated perinatal depression for both the mother and the fetus, which is consistent with consensus guidance, while appropriately avoiding directive or prescriptive medical advice. ChatGPT consistently emphasized that it cannot replace a clinician and, in the context of postpartum depression, demonstrated strong safety awareness by encouraging timely professional evaluation, highlighting warning signs of self-harm, and directing users to crisis resources when indicated.

Grok4 performed particularly well in escalation behaviors, proactively providing region specific crisis hotlines and explicitly encouraging urgent care when severe symptoms were mentioned. It also attempted to identify nearby counseling or medical services by providing local addresses and phone numbers. However, Grok4’s responses were more variable in structure and sometimes prioritized extensive resource lists over a systematic clinical explanation, which may pose comprehension challenges for users with lower health literacy.

Gemini-2.5 generally recognized perinatal depression as a medical condition and encouraged help seeking; however, its responses were less consistent across similar prompts. Variation in emphasis across related questions suggests weaker internal consistency, and recommendations for crisis resources were less consistently embedded within an explicit clinical risk framework. Such inconsistency may limit the reliability of model outputs in high-risk mental health contexts, where stable and consistent safety signaling is essential.

DeepSeek’s responses were generally supportive and non-harmful, but they often remained at a broad educational level. Compared with other models, it less frequently provided explicit guidance on when urgent or emergency care is warranted, and its referral pathways were less specific. While this level of response may be acceptable for general psych education, it offers more limited support for risk recognition, stratification, and action planning when risk is elevated, or safety signals are ambiguous.

Based on our results, Grok4 achieved the highest score under the DISCERN framework. As Grok4 is a relatively new model, there is currently limited published evidence available for direct comparison of DISCERN scores. The scoring profile of Grok4 suggests that its strengths are not primarily reflected in presentation style or formal citation conventions, but rather in its safety-oriented responses in high-risk scenarios. This characteristic may carry potential clinical value in perinatal mental health contexts, while also indicating room for improvement in the standardized presentation expected for public facing health education materials. In this sense, its outputs appear more aligned with a “clinical communication” style than an “educational material” style.

Among the remaining models, Copilot achieved the highest DISCERN score, outperforming Gemini-2.5 and DeepSeek, which is consistent with findings from two prior studies (40, 41). Copilot frequently emphasizes source attribution and often references academic journals and official guidelines; its strong performance on the JAMA further suggests that it strikes a balance between quality, breadth, and prioritization of key information. Although Gemini-2.5 obtained a relatively high JAMA score, it performed less well on DISCERN. This discrepancy may reflect the fact that DISCERN evaluates not only accuracy but also the structured, user oriented, and comprehensive presentation of information, including discussion of risks and alternative options. Thus, while Gemini-2.5 may perform well on transparency related criteria captured by JAMA, limitations in organization, user centered framing, or comprehensiveness may contribute to a lower DISCERN score. DeepSeek performed particularly well on EQIP, a pattern also reported in prior work. Its outputs resemble a “health education/health literacy” mode rather than a “clinical decision support” mode, which may be well suited for public health education.

Joint analysis of DISCERN and HONCODE showed a positive association between the two scores, consistent with previous findings (42). High performing models such as Grok4 and Copilot ranked higher on both instruments, whereas lower-performing models such as ChatGPT-5 tended to score lower on both. This suggests some overlap between information quality and adherence to established standards in consumer health contexts: higher-quality information is often more likely to align with transparency and accountability principles. A high DISCERN score indicates that the model’s content is reliable, well organized, and comprehensive, while a high HONCODE score indicates stronger alignment with ethical and transparency standards, such as attribution and authority. Together, these properties may increase public confidence and reduce exposure to misleading or incorrect information. If AI systems can further improve transparency and authority signaling, they may help bridge the gap between public understanding and professional medical knowledge—an especially important goal in an environment of information overload, uneven online content quality, lower health literacy in parts of the population, and persistent information asymmetry.

Readability is critical to determining whether health information can be effectively communicated. The NIH explicitly recommends that public health materials be written at approximately a sixth to seventh grade reading level to ensure that the public, particularly individuals with lower health literacy, can understand medical content (43). In our study, readability scores for all AI generated responses exceeded the sixth-grade level, with most approaching a high school reading level. This pattern is inconsistent with the NIH recommended benchmark for public facing health information. When the readability of health information exceeds the public’s reading ability, several problems may arise. First, comprehension barriers may occur information may be available but not truly accessible, and readers may struggle with vocabulary, sentence structure, or logical organization. Even well-educated readers may misinterpret specialized medical terminology, treatment recommendations, or risk explanations, leading to misunderstanding or disregard of key information. Second, trust may be undermined: overly technical content can appear less trustworthy, and highly complex text may create psychological distance or trigger skepticism, especially when source attribution, update dates, or author credentials are unclear, thereby reducing acceptability and communication effectiveness (44). Third, health behaviors may be impeded: information may be encountered but not translated into action. Poor readability reduces usability, making it difficult for lay readers to extract actionable points, which can weaken adherence and limit behavior change. This concern is particularly salient for perinatal women, who may experience substantial psychological stress; when health information is obscure or difficult to understand, uncertainty and anxiety may increase. In contrast, clear and plain language information can be reassuring. Low readability may also lead pregnant or postpartum individuals to misinterpret advice regarding medications, diet, or psychological interventions, potentially resulting in poor adherence or avoidable risk. By improving readability, AI based tools may help reduce such misunderstandings. Moreover, high quality, readable information may enable healthcare professionals to guide women toward reliable resources more efficiently, potentially reducing clinical workload. Therefore, to maximize the public health value of AI generated content, future efforts should prioritize improving readability without compromising information quality, ensuring that medical information is both accurate and understandable.

This study examined the reliability and readability of AI generated responses to perinatal depression related questions. Although numerous information quality assessments have been conducted in other medical domains, evidence in perinatal mental health remains limited. We evaluated five widely used, up to date AI models using well established instruments for reliability and readability, providing practical evidence to inform public facing consultation and the use of AI generated health information in this context.

This study has several limitations. First, we interacted with commercially deployed models through their official web interfaces; as a result, generation parameters such as temperature and top p were governed by platform defaults, and the generation process could not be fully standardized. Second, closed-source web products such as Microsoft Copilot may employ platform-level model selection and dynamic routing, and specific underlying model versions and routing behaviors are not fully disclosed. Therefore, in a web-based setting, researchers cannot reliably identify or control the exact model and routing pathway used for each response, which may affect strict comparability and reproducibility across models. In addition, LLM outputs may vary over time due to silent platform updates, policy changes, or routing adjustments; thus, our findings reflect performance within a specific time window. Third, this evaluation was conducted only in English and used a limited FAQ set derived from Google Trends and ACOG resources, which may not capture different cultural contexts, varying levels of health literacy, or higher risk clinical scenarios. Finally, while we focused on reliability and readability, we did not systematically quantify safety critical elements in perinatal mental health contexts, such as crisis resource guidance, risk communication, or referral recommendations. Future studies should extend this work across multiple languages and time points, under more controllable generation settings, and incorporate explicit safety metrics to further validate and generalize our findings.