Chronic non-bacterial osteitis (CNO) is an autoinflammatory bone disease that commonly affects children and adolescents (Zhao et al., 2021). CNO can lead to severe complications, including bone pain and bone damage. Its pathophysiological characteristics are marked by increased inflammasome assembly and imbalanced cytokine expression. Treatment medications include nonsteroidal anti-inflammatory drugs (NSAIDs), corticosteroids, etc. (Hedrich et al., 2023). Though reliable epidemiological data are lacking, CNO is likely to be one of the most common autoinflammatory syndromes (Schnabel et al., 2016). In Germany, incidence rates were estimated at 0.4 per 100.000 children/year (Jansson et al., 2011). However, despite recent advances in our understanding of CNO, the number of detected cases has increased in recent years (Lenert and Ferguson, 2020). Nevertheless, there are considerable practice variations in the marking, diagnosis, and treatment of CNO currently (Winter et al., 2025). Therefore, there is an urgent need for health education efforts to enhance awareness of CNO.

Artificial intelligence (AI) is a system’s capacity to analyze data. It uses computers and machines to boost humans’ decision - making, problem - solving, and tech innovation capabilities (Tang et al., 2022). Currently, AI is becoming increasingly popular in China and even worldwide, and it is being applied in various fields (Wei et al., 2022). With the aging of the population, the global demand for high-quality healthcare has increased, while artificial intelligence (AI) has been widely applied in modern medicine—for instance, in diagnosis and treatment (Higuchi et al., 2024; Chadebecq et al., 2023). These AI tools can assist clinicians in different fields to make more informed decisions (Kulkarni and Singh, 2023). In the field of oncology, AI can be used for cancer screening and diagnosis (Abbasi, 2020). In cardiovascular medicine, it can integrate various forms of patient data to aid doctors in treatment (Lüscher et al., 2024). Even in surgical procedures, AI can improve multiple aspects such as preoperative, intraoperative, and postoperative care (Varghese et al., 2024). The rapid development of AI has provided convenience for the entire medical industry. In daily life, it offers both patients and doctors channels to access disease-related knowledge. However, people are easily misled due to inaccurate and outdated information.

In clinical practice, it is common to encounter patients using these artificial intelligence language models to inquire about disease-related knowledge. Previous studies have evaluated the accuracy and reproducibility of ChatGPT in answering questions related to Helicobacter pylori (Lai et al., 2024). The results showed that ChatGPT could provide correct answers to most queries related to Helicobacter pylori, demonstrating good accuracy and reproducibility.

Deepseek V3, Doubao, and Kimi1.5 are the three most popular and freely accessible artificial intelligence language models in China. They have been widely applied in the country and are used by a large number of people, so it is necessary to assess the accuracy and reproducibility of these three language models in answering disease-related questions.

In the field of patient care, AI can serve as a “highly efficient auxiliary tool” for clinicians: On one hand, when doctors in primary hospitals treat patients with suspected CNO, AI can quickly generate an initial diagnosis differential checklist and examination recommendations based on the patients’ symptoms (Apornvirat et al., 2024). This helps shorten the time for diagnostic decision-making and reduce missed diagnoses. On the other hand, when patients report side effects during follow-ups, AI can immediately provide “directions for preliminary treatment plan adjustments” to assist doctors in responding quickly (Hartman-Kenzler et al., 2024). This avoids risks caused by long intervals between follow-up visits and indirectly improves the efficiency and safety of patient care. In the field of patient education, AI can provide “personalized and accessible health guidance”: Traditional patient education mostly relies on general manuals, which are difficult to adapt to individual conditions (Apornvirat et al., 2024). In contrast, AI can integrate a patient’s medical history (e.g., a history of diabetes) (Davis et al., 2023) and treatment plan (e.g., hormone medication) (Natchagande et al., 2024), while also issuing warnings about “signals requiring emergency medical attention” (e.g., sudden vision loss, worsening eye pain). This helps patients understand the logic of diagnosis and treatment, improve treatment adherence, and reduce treatment interruptions or neglect of risks caused by cognitive biases.

In summary, in this study, among Deepseek V3, Doubao, and Kimi1.5, Deepseek V3 took the longest time to think when answering questions, while Doubao responded the fastest. Based on the recommendations outlined in clinical expert guidelines, both reviewers confirmed that an answer to each question could be found in the expert consensus. Since each expert conducted independent evaluations, the majority of the scores were consistent. However, individual cognitive characteristics of experts—such as personal clinical experience and attention allocation—may also lead to discrepancies in scoring. This is an inherent attribute of human subjective judgment and constitutes a bias that is controllable but cannot be completely eliminated. Additionally, among all the answers, Doubao provided the most words in its responses, whereas Kimi1.5 provided the fewest. There were no significant differences in the scores given by the two reviewers for the answers provided by these three AIs, and the scores all fell between 3 and 4, indicating the accuracy of the AI’s responses is highly satisfactory. More importantly, the high scores were maintained across all three rounds, demonstrating good repeatability in the AI’s responses. However, for the fourth question (“Which parts of the body are most frequently affected by chronic non-bacterial osteitis in adults?”), none of the three AIs received a score of 4 and Doubao provided an incorrect answer to the fourth question in the third round. Regarding this issue, the latest guidelines suggest that in adults, CNO predominantly manifests in the anterior chest wall, including the clavicle, upper ribs, and sternum, which are the most commonly affected sites (Winter et al., 2025).

Currently, there have been numerous studies on the application of ChatGPT in medicine, as well as comparative analyses of its research performance in the medical field against different AI models (Bradshaw, 2023). Ozan Yazıcı et al. compared ChatGPT and Perplexity in terms of treatment response and reliability assessment for rectal cancer, finding that Perplexity had higher accuracy than ChatGPT (Yazici et al., 2023). Zhou et al. (2025) evaluated the performance of ChatGPT and DeepSeek in generating educational materials for patients undergoing spinal surgery, discovering that DeepSeek-R1 produced the most credible answers, although the AI models generally received moderate DISCERN scores. Even within the same AI model, differences can exist between versions. Hu et al. examined two versions of ChatGPT in relation to questions about Helicobacter pylori, finding that both versions had high accuracy, with ChatGPT 3.5 performing well in the areas of indications, treatment, and gut microbiota, and ChatGPT 4 excelling in diagnosis, gastric cancer, and prevention (Hu et al., 2024). Additionally, researchers have found that the fusion of emotion-aware embeddings in large language models (including Flan-T5, Llama 2, DeepSeek-R1, and ChatGPT 4) is applied to intelligent response generation (Rasool et al., 2025). In this study, when answering relevant questions, we can observe that each AI model has its own characteristics. Deepseek V3 reminds users after each answer that the response is for reference only. Doubao expands on other related questions after answering the initial query. Kimi1.5 not only expands on related questions but also displays the answer sources on the right-hand side.

Although these three AIs scored relatively high, they all have limitations. (1) their sources are not based on clinical evidence but rather on internet sources, many of which are not professional literature guidelines. (2) When new clinical guidelines are updated, AIs do not provide answers according to the latest clinical guidelines or expert consensus, which can easily lead to outdated information and incorrect answers. Furthermore, when AI models are updated, their accuracy also changes. In future analyses, we will also include re-testing after major model updates. (3) The conclusions of this study are only applicable to Chinese Q&A scenarios. Due to the variations in treatment guidelines across different specialties and regions worldwide, giving priority to the guidelines of those regions poses a significant challenge for AI. (4) This study only based its inquiries on 16 questions. Though these questions cover such aspects as disease definition, diagnosis, and treatment, the number is relatively limited. Moreover, the grading system is based on a 4-point scale. Though it has been used in relevant literature, the non-refined scale has the drawback of missing minor errors. (5) Semantic similarity-based evaluation indicators such as BERTScore have not been applied in this study, yet these indicators are an important factor for conducting objective comparisons between models and driving model performance improvement.

Furthermore, patient inquiries in the real world may differ from expert-developed questions in terms of complexity and wording. In future studies, the number of questions should be further increased, and the wording used by patients when they ask questions via AI should be collected. Nowadays, the use frequency of AI in clinical work is increasing. Future research can shift from static Q&A to dynamic clinical reasoning by designing case-based scenarios—requiring models to interpret symptoms, order tests, and propose treatment plans, thereby moving from factual retrieval to diagnostic reasoning (Rasool et al., 2020). Therefore, we suggest that in the future, AI-generated medical answers should undergo appropriate certification and regular review. This approach can provide patients with more accurate disease information, enhance their understanding of the disease, and improve their management of the condition.

Overall, through this study, we found that these three AI models demonstrate relatively high accuracy and reproducibility when answering CNO-related static questions based on their training data. With no significant differences observed. We believe that with the continuous development of AI prediction models, Deepseek V3, Doubao, and Kimi1.5 have the potential to serve as supplementary tools in addition to expert consensus and guidelines.