To achieve comprehensive retrieval coverage, this study determines the standardised medical term for “Thunderstorm Asthma” through the Medical Subject Headings (MeSH) database. A theme is a collection of terminology that shares the same conceptual framework across languages and may encompass the most relevant information (14). Ensure the comprehensiveness and accuracy of the research. The following MeSH entry terms have been identified: “Thunderstorm Asthma”, “Asthma Thunderstorm”, “Asthma”, “Tornadoes”, “Asthma storm”, “Thunderstorm and Health”, “Thunderstorm Warning”.

Google Trends serves as a dynamic indicator of search behaviour and public interest over time (15). In this study, relevant standardised terms were input into the Google Trends system to capture, to the greatest extent possible, the global search trends related to “Asthma Thunderstorm,” thereby enabling the identification of the most relevant search queries associated with the “Asthma Thunderstorm” topic. To conduct a five-year longitudinal analysis (2020–2025) of global search patterns and public attention (parameters: Worldwide; All categories; Popular searches; time frame: September 2020 to September 2025). To make up for the limitations of search engine data, respiratory critical care experts excluded duplicate and irrelevant information. After a series of screenings, 24 representative results related to “Thunderstorm Asthma” were finally determined to be used for the model performance benchmark information assessment of the LLM (see Table 1).

To improve the systematic assessment, the final search results are summarised. This study selected four mainstream LLM, including ChatGPT-4-4o, Deepseek-V3.2, Perplexity Pro, and Microsoft Copilot (version: 1.240.110.0), to conduct a benchmark performance analysis on the quality of information responses provided by LLM. Large language model have a certain degree of exclusivity. Their internal weight information, training language databases, etc. are not made public. It is impossible to determine whether the training data sets of LLM contain relevant public health information on “thunderstorm asthma.” To avoid data contamination that may result from the opacity of data structures and prevent any potential bias risks, all browser-related data are cleared before each prompt word is input to prevent artificial intelligence from relying on cached data or internal algorithms from the previous interaction.

The Wilcoxon signed-rank test was used to evaluate the information responses of LLM robots against the reading comprehension level of sixth graders. The readability scores generated by artificial intelligence were compared with the established reference values for sixth graders to analyse the statistical differences.

All tests were two sided statistical tests and the predetermined level of significance was p < 0.05. The data processing, statistical analysis, and visualisation were done using R (4.5.1).

This study utilised the DISCERN, EQIP, JAMA, and GQS assessment tools to evaluate the reliability of LLM generated responses. The reliability index of each LLM programme is measured by descriptive statistics (median [Q1, Q3]) (see Table 2). The inter-rater intraclass correlation coefficient amongst all evaluators was 0.813 (0.767–0.864). In the model stability assessment, to ensure consistency in the generated informational responses and mitigate response randomness, all LLM bots were operated under deterministic parameters (temperature = 0.0) and executed three times independently. The results were entirely consistent, yielding a stability coefficient of 100%. The Kruskal-Wallis test results indicate statistically significant differences in DISCERN, EQIP, and JAMA scores across the different LLM systems. In contrast, no statistically significant difference was observed for GQS scores (p = 0.843).

The DISCERN analysis reveals significant differences in the benchmark performance of information responses amongst various language models. The Perplexity Pro [45 (42.75, 47)] demonstrated a moderate level of performance, whereas ChatGPT-4-4o [40 (36, 47)] exhibited comparatively inferior information quality. This suggests significant variability in response reliability amongst LLM based conversational agents during information retrieval and analytical tasks. In contrast, the information responses generated by ChatGPT-4-4o exhibit a baseline performance of low quality, attributable to issues with source reliability, support capabilities, and decision-making processes. Although Microsoft Copilot [48.5 (46.5, 50)] and Deepseek-V3.2 [47 (46, 50)] received relatively higher scores and delivered moderately structured responses, their baseline performance is still rated as average overall (see Figure 1).

The EQIP results demonstrate that all evaluated LLM based agents received favourable quality assessments. Microsoft Copilot achieved the highest score [70 (65, 71.25)], while Deepseek-V3.2 received the lowest [60 (60, 60)]. All LLM based systems demonstrated satisfactory performance in terms of overall structure and clarity, with their comprehensive performance ratings evaluated as “good.” However, deficiencies were noted in the provision of detailed source information, textual readability, and balanced multi-perspective analysis (see Figure 1).

The GQS scoring results indicated no statistically significant differences. The performance of artificial intelligence in providing basic health information was moderately satisfactory, with an overall favourable impression, and the assessed LLM systems were rated as good (median = 4.0) (see Figure 1).

The JAMA rating scale revealed a statistically significant difference in outcomes, with Deepseek-V3.2 (median score: 0 [0, 0]) and Microsoft Copilot (median score: 2 [0, 2]) demonstrating distinct performance levels. The performance of LLM in adhering to evidence-based medicine standards reveals significant deficiencies in providing reliable author-related information and disclosures, particularly regarding authorship attribution and conflict-of-interest statements (see Figure 1).

To delineate the baseline performance disparities in information response capabilities amongst LLM, a Dunn’s test was employed for pairwise comparative analysis, thereby elucidating the inter-model performance variations (see Table 3).

On the DISCERN instrument for assessing information reliability and quality, ChatGPT-4-4o demonstrated significantly superior performance compared to Microsoft Copilot (p < 0.001) and Deepseek-V3.2 (p < 0.001). Meanwhile, Perplexity Pro demonstrated significantly superior performance compared to Microsoft Copilot (p = 0.004) and Deepseek-V3.2 (p = 0.023). However, no statistically significant differences were observed between ChatGPT-4-4o and Perplexity Pro (p = 0.201), nor between Microsoft Copilot and Deepseek-V3.2 (p = 0.522).

On the Evaluation of the Quality of Information Provision (EQIP) scale, the differences between the vast majority of model pairs did not reach statistical significance (p > 0.05).

In the evaluation using the Global Quality Score (GQS) scale, pairwise comparisons amongst all models did not reveal statistically significant differences (all p-values >0.999), indicating that the outputs of all models were perceived to be of comparable quality in terms of overall perceptual assessment.

In assessments conducted using the JAMA scale to evaluate information transparency, the performance of Deepseek-V3.2 was significantly inferior to all other models. By contrast, no statistically significant differences were observed in pairwise comparisons amongst ChatGPT-4-4o, Microsoft Copilot, and Perplexity Pro (all p > 0.05).

This study reveals that model performance is critically dependent on the selected evaluation framework. It further confirms the pervasive challenge faced by current LLM based systems in simultaneously ensuring the quality of structured information provision and adhering to conventional medical information transparency standards.

This study employs six widely recognised readability assessment tools: the ARI, FRES, Gunning-Fog, FKGL, Coleman-Liau, and SMOG indices. The median readability scores, presented as median (Q1, Q3) (see Table 4).

The results indicate that all analysed metrics for evaluating the readability of responses generated by major LLM —namely ARI, GFI, FKGL, CL, and SMOG—significantly surpassed the recommended benchmark for sixth-grade reading comprehension. The FRES score falls significantly below the benchmark range of 80–90 corresponding to the 6th-grade reading level. Even the top-performing Microsoft Copilot achieves a median score of merely 45.25, remaining within the difficult readability range. The results from ARI, GFI, FKGL, CL, and SMOG assessments collectively indicate that the text’s reading difficulty aligns with that of upper middle-grade levels. The standard information responses of the LLM robot evaluated in this study exhibit substantial readability challenges, with a comprehension level exceeding that of the general public (sixth-grade equivalent), thereby imposing certain limitations on vulnerable populations with low digital health literacy (see Figure 2).

This study employs the DISCERN instrument, EQIP, GQS, and JAMA benchmarks to evaluate the reliability and quality of health information responses regarding “thunderstorm asthma” generated by LLM. By leveraging the respective strengths and relevance of these four distinct assessment tools, a comprehensive, multi-faceted evaluation of response reliability and information quality is conducted (25). DISCERN and EQIP are widely used to assess the reliability and quality of written health materials. The former emphasises explicitly optimal evaluation criteria for studies on treatment efficacy, focusing on health information relevant to treatment decisions—including therapeutic benefits, risks, and quality of life outcomes (26). The latter focuses on the characteristics of information content in clinical application contexts. The GQS evaluation tool provides a subjective assessment of overall quality based on users’ individual needs and perceptions, as well as the universality of the information. JAMA evaluates four essential characteristics based on the standards for internet healthcare information published in the Journal of the American Medical Association, including authorship and attribution (e.g., author names and affiliations, cited references, copyright information), currency (e.g., date of posting, date of update), and disclosure (e.g., conflicts of interest, sponsorship, advertising, and commercial funding) (27).

The evaluation conducted through DISCERN, EQIP, GQS, and JAMA benchmark analyses indicates that Microsoft Copilot exhibits superior performance in information response compared to other LLM. The transparency and precision of the information sources utilised by Microsoft Copilot contribute significantly to its favourable reliability assessment. Meanwhile, ChatGPT-4’s reliability is compromised by the lack of source attribution and insufficient disclosure of information origins, whereas Deepseek-V3.2 demonstrates significantly lower scores on reliability and source transparency metrics—particularly under the DISCERN and JAMA evaluation frameworks—compared to other LLM systems. When seeking health information through LLM, internet users should prioritise the reliability of response sources, as the scientific rigour and evidence-based support of these sources have become increasingly critical indicators for assessing the credibility of health-related information, Incomplete dissemination of information and the provision of unreliable content may lead to harmful consequences, such as delays in seeking medical assistance or the adoption of erroneous preventive measures during similar public health emergencies. The transparency of information sources determines perceived reliability in health communication, whereas the dissemination of incomplete or misleading information undermines this credibility (19). The use of social media as a source of health information carries the potential risk of disseminating incomplete or misleading content (28).

This study employs the ARI, CL, FKGL, FRES, GFI, and SMOG indices to evaluate the complexity of health information responses concerning “Thunderstorm Asthma” generated by LLM, thereby assessing the readability of these informational replies. Relevant research indicates that educational disparities are more likely to result in internet resources serving as a primary contributor to non-adherence to medical advice. Vulnerable populations, including old adults individuals and those with lower educational attainment, frequently encounter online health information that is highly readable yet unreliable. The development of health education materials tailored to the public’s reading level ensures that internet users can fully comprehend preventive measures for thunderstorm asthma and the safety of pharmacological treatments, thereby facilitating informed decision-making. High readability and reliable information responses are essential to prevent the dissemination of misinformation and enable the public to make informed preventive decisions regarding public health safety issues, such as thunderstorm asthma. Although studies indicate that both SMOG and GFI are more suitable as optimal indices for evaluating the readability of online health-related materials (29). However, for complex public health emergencies such as thunderstorm asthma, this study aims to assess the readability and complexity of LLM generated health information using multidimensional indices to yield comprehensive evaluation outcomes (30–32). The findings of this study indicate that the readability of all artificial intelligence programme-generated responses exceeds the recommended sixth-grade reading level, posing significant challenges in conveying health information to individuals with limited eHealth literacy. This study employs a multi-index evaluation framework to identify deficiencies in the content, structure, and other aspects of health information consultations provided through LLM question-answering systems. For instance, the Perplexity model’s responses regarding “thunderstorm weather forecasting” included statements such as “require carrying emergency medications” and descriptions about “symptoms and treatments of thunderstorm asthma”. The text includes the specialised terms “inhaled corticosteroid medications” and “airway spasm”; these ambiguous expressions and domain-specific terminology pose significant challenges for non-expert users in comprehension and information navigation. Science communication should employ accessible language that aligns with public needs, such as describing the mechanism as “alleviating airway hyperresponsiveness through inhaled aerosolised medications, exemplified by corticosteroids such as Budesonide,” to convey information clearly and concisely.

Currently, LLM can provide us with a vast amount of convenient information and have shown good performance in numerous studies. However, in complex interdisciplinary fields, especially in the health domain, the difficulty of responding to information may be too high for users with low e-health literacy, making it hard for them to understand and equally access key information. This is a problem that needs to be addressed at present. At the same time, when it comes to obtaining external information, if the retrieved information is from non-medical authoritative sources, the quality of such information cannot be guaranteed.

This study is subject to certain limitations. First, the exclusive reliance on Google Trends as a data source limits access to search query topics from alternative search engines, including regions affected by internet censorship or limited connectivity (33). Our global-scale analysis seeks to account for regional variations in thematic inquiries, linguistic diversity, and cultural differences, all of which may influence research outcomes. Currently, Google Trends remains the primary tool for analysing global trends in online health information searches. Furthermore, artificial intelligence-based LLM are constrained by limited sample sizes and are applicable only for interpreting probabilistic emergent events, such as thunderstorm asthma.

This study is based solely on a limited set of query data, which may not fully capture the broad spectrum of user needs concerning thunderstorm asthma health information. The scope of the training data, potential computational biases, and other inherent limitations may affect the accuracy, completeness, and clarity of the generated responses.

The readability and reliability metrics used in this study may not fully capture the nuanced distinctions inherent in LLM generated content. While evaluation results for various indices indirectly reflect information quality, they do not account for subjective factors, such as users’ contextual needs.

In the future, LLM systems will undergo continuous iteration and refinement. The evaluation of LLM generated responses in healthcare information must be regularly updated accordingly. Furthermore, it is essential to explore real-time user interaction data to accurately assess the reliability, readability, and practical utility of health information provided by LLM applications.