Using Python web crawling technology, we scraped short videos related to seven Chinese keywords on the Chinese versions of TikTok and Kuaishou: “cerebrovascular disease,” “cerebral apoplexy,” “stroke,” “cerebral infarction,” “transient ischemic attack”, “intracerebral hemorrhage”, and “subarachnoid hemorrhage.” The crawler employed the DrissionPage library for browser automation and network request monitoring, the time library to set random time for simulating human browsing behavior, and a custom DataRecorder module for data logging. The crawling frequency was configured such that a random wait time of 2–4 s was imposed after each page scroll, with 3–5 random scrolls performed per crawling cycle. The formulation of keywords was based on the standard terminology specified in the Guidelines for the Prevention and Treatment of Cerebrovascular Diseases (2024 Edition) issued by the National Health Commission of the People’s Republic of China (4), as well as commonly-used public search terms, covering the main subtypes of cerebrovascular diseases. We included all videos related to the keywords published between December 10th, 2023, and December 10th, 2024. A total of 2,205 videos from TikTok and 1880 videos from Kuaishou were retrieved using 7 keywords. After merging and removing duplicates of videos related to different diseases, 1,467 TikTok videos and 1,505 Kuaishou videos remained. Following manual screening to exclude videos unrelated to health science popularization, duplicate content, pure picture videos, and nonexistent videos, a final total of 541 TikTok videos and 374 Kuaishou videos were included in this study (Figure 1).

Basic information of the videos, including video title, type of account, video duration, number of likes, number of favorites, number of comments, and number of shares, was extracted. For data missing from the web crawling process, manual video retrieval was performed to supplement the dataset. In this study, the quality of the video content was evaluated, while the quality of the video images and music was not assessed in the study. According to the authentication information displayed on the platforms (such as official certification badges, profile descriptions, and verified qualifications), account types were categorized into the following categories with specific definitions to facilitate subsequent processing: (1) Medical professional accounts: Accounts authenticated with credentials of medical practitioners, with profile information clearly indicating their professional identity (for example, “neurologist at XX Hospital”); (2) Hospital accounts: Official accounts registered by medical institutions with the full name and affiliation of the hospital in their profiles, such as hospitals, clinics, or medical centers; (3) Government accounts: Accounts operated by government agencies related to public health, such as health commissions and centers for disease control and prevention; (4) Media accounts: News entities with legal news qualifications, such as Yangshipin (China Media Group’s online platform) and XX City Radio and Television Station; (5) Company accounts: Accounts registered by enterprises or commercial entities, such as pharmaceutical companies and health product brands; (6) Non-professional individual accounts: Personal self-media accounts without official certification of medical, institutional, or media qualifications.

The quality of the included short videos was scored using the Global Quality Score (GQS), modified DISCERN (mDISCERN) and the Patient Education Materials Assessment Tool (PEMAT) to perform a comprehensive assessment of the included videos. The GQS was employed to assess the overall quality of included short videos. The 5-point GQS was developed by Bernard et al. in 2007 (24). Singh et al. first applied it to video assessment in 2012 (25), and it has now been widely adopted for this purpose (26–28). The total score ranges from 1 to 5, with lower scores indicating poorer video quality and higher scores representing superior quality. The specific scoring details for GQS are shown in Supplementary Table 1. The mDISCERN was used for reliability evaluation. The DISCERN criteria are a validated scoring system developed by a research team at the University of Oxford to assess the information quality and reliability of health content related to consumer treatment options. The Cronbach’s α coefficient was 0.78 (29). The mDISCERN tool, modified by Singh et al. (25), is more suitable for evaluating videos (30). It consists of five questions, each question was scored 1 for “yes” and 0 for “no,” and high scores indicated that the video was reliable. The specific scoring details for mDISCERN are shown in Supplementary Table 2. The PEMAT was adopted for assessing the comprehensibility and operability of the included short videos. The PEMAT-A/V (31), designed specifically for audiovisual materials, consists of 17 questions, with 13 questions that evaluate the understandability of health information received by patients and 4 questions evaluating the actionability of recommendations by videos. The Cronbach’s α coefficient was 0.71 (31). Each question is scored as “agree = 1, disagree = 0, not applicable = N/A”, and the score of the understandability or actionability section is calculated as “Total Points/Total Possible Points×100%”, with higher scores indicating better performance in terms of understandability and/or actionability of the video. The specific scoring details for PEMAT-A/V are shown in Supplementary Table 3. Two clinical experts independently used GQS, mDISCERN and PEMAT-A/V to evaluate the quality of health science popularization short videos about cerebrovascular diseases on short video platforms. Before accessing the short videos, the assessment criteria of each scoring tool were reviewed by the two experts, and a detailed introduction about the three evaluation tools was given to the two experts to reduce the errors caused by cognitive biases. The scoring process was independently conducted by two clinical experts, and the final quality score of each video was determined by taking the average of their respective ratings.

All data were double-checked. SPSS 27.0 was used for statistical analysis, and Origin 2024 was used for graphing. The consistency evaluation of the scoring results of the two raters was carried out by using the Intraclass Correlation Coefficient (ICC). Since the data were nonparametrically distributed, the median (IQR) was used for the descriptive statistics. The Kruskal-Wallis test was used to assess the differences between groups. And Spearman correlation analysis was used to evaluate the relationship between the video features and quality score of videos. The significance of the statistical analysis was set as 0.05. The Random Forest regression model was used to examine the degree of importance of video features on video quality. The initial parameters were set with the number of decision trees at 100, all other parameters were kept at their default values, and 5-fold cross-validation was adopted to optimize the model parameters (Figure 2A).

We retained 915 health science popularization short videos related to cerebrovascular diseases in this study, including 541 from TikTok and 374 from Kuaishou. The videos were published by accounts with different entities. On TikTok, medical professionals released the highest number of videos, accounting for 77.3% (418/541), followed by non-professional individuals (43/541, 7.9%), media organizations (36/541, 6.7%), hospitals (36/541, 6.7%) and government agencies (8/541, 1.4%) (Figure 2A). Similarly, on Kuaishou, medical professionals released the highest number of videos, accounting for 79.9% (299/374), followed by non-professional individuals (61/374, 16.3%), media organizations (11/374, 2.9%), hospitals (2/374, 0.6%) and companies (1/374, 0.3%) (Figure 2B). The number and proportion of different keywords retrieved from the two platforms were detailed in Supplementary Table 4. The results showed that videos related to “cerebrovascular disease” accounted for the largest proportion (241/915, 26.3%), followed by those related to “cerebral apoplexy” (201/915, 22.0%), and videos related to “subarachnoid hemorrhage” constituted the smallest proportion (76/915, 8.3%).

Furthermore, we conducted a comparative analysis of TikTok and Kuaishou short videos across five key features: video duration, likes, favorites, comments, and shares (detailed in Table 1). The results showed that the TikTok videos got more likes, favorites, comments, shares than those on Kuaishou (p < 0.001). The median video duration on Tiktok was 89 (58–140) seconds, was significantly longer than those on Kuaishou, which was 68 (42–101) seconds (p < 0.001). A detailed comparative analysis of these video features was presented in Figure 3.

The consistency between the two raters was satisfactory, with ICC ≥ 0.7, details were shown in Table 2. The certain differences in the video quality scores across different keywords were detailed in Supplementary Table 4. The median Understandability score for “cerebrovascular disease videos” was the highest, at 65.38% (IQR: 57.69–80.77%), and the median Actionability score for “transient ischemic attack” and “subarachnoid hemorrhage” videos was the lowest, both at 25% (IQR: 25–50%). The mDISCERN and GQS scores of videos associated with different keywords were essentially consistent. Regarding videos on TikTok, the median mDISCERN score was 3 (IQR 2.5–3), the GQS median score was 3 (IQR 3–3), the median Understandability score was 65.38% (IQR 61.54–76.92%) and the median Actionability score was 50% (IQR 37.5–50%), indicating that the videos on TikTok were of fair quality. Regarding Kuaishou videos, the median mDISCERN score was 2.5 (IQR 2.5–3), the GQS median score was 3.25 (IQR 2.5–3.5), the median Understandability score was 61.54% (IQR 53.85–70.19%) and the median Actionability score was 37.5% (IQR 25–50%). The comparison of scores between TikTok and Kuaishou videos across different score ranges evaluated by various tools was shown in Figure 4. The results demonstrated that TikTok videos achieved significantly higher scores than Kuaishou across all video quality metrics, including the mDISCERN scores, 5-level GQS, Understandability, and Actionability (Figure 5).

Concurrently, we compared video quality across different account types. Results demonstrated that there were statistically significant differences in GQS scores, Understandability scores, and Actionability scores among videos posted by distinct creator accounts (p < 0.05; Figure 6). Figure 6A showed that for the GQS scores of different accounts, the score distribution of hospitals was relatively concentrated and high at 3 (IQR 3–3.5), compared with that of media organizations at 3 (IQR 2–3), and the difference between them was statistically significant (p = 0.006, p < 0.05). For the mDISCERN scores of different accounts, the scores were relatively dispersed, and there were no particularly significant differences between groups (Figure 6B). Figure 6C showed that in terms of Understandability scores, there was a difference between the score of hospitals at 69.23% (IQR 61.53–76.92%) and that of non-professional individuals at 61.53% (IQR 57.69%-69.23) (p = 0.004, p < 0.05). Figure 6D showed that in terms of Actionability scores, the score distribution of hospitals was relatively concentrated and high at 50% (IQR 50–62.5%), which was significantly different from that of medical professionals at 50% (IQR 25–50%) (p = 0.001, p < 0.05).

The correlations between video features and quality assessment scores (mDISCERN, GQS, Understandability, and Actionability) were presented in Table 3. There were relatively strong correlations between video duration and mDISCERN score (r = 0.219, p < 0.001), GQS (r = 0.495, p < 0.001), Understandability score (r = 0.282, p < 0.001), and Actionability score (r = 0.361, p < 0.001). Additionally, there were moderate correlations between the Understandability score and likes (r = 0.153, p < 0.001), favorites (r = 0.169, p < 0.001), and shares (r = 0.163, p < 0.001). Similarly, likes (r = 0.221, p < 0.001), favorites (r = 0.217, p < 0.001), shares (r = 0.228, p < 0.001), and comments (r = 0.171, p < 0.001) showed a moderate correlation with the Actionability score. Furthermore, a positive correlation was observed between the publisher and the Actionability score (r = 0.123, p < 0.001).

The results of the random forest regression model regarding the importance of video features to video quality were presented in Table 4. The four models demonstrated favorable fitting performance, with R² values ranging from 0.862 to 0.903 and small error metrics such as MAE, MSE, and RMSE. Video duration emerged as the most critical feature contributing to all four models, with a weight proportion of 28.10% in the mDISCERN model (Figure 7A), 53.92% in the QRS model (Figure 7B), 32.11% in the Understandability model (Figure 7C), and 34.94% in the Actionability model (Figure 7D). The remaining features (likes, favorites, comments, and shares) had relatively lower weights, with slight variations in proportions across different models.

In this study, a comprehensive evaluation was conducted on the quality of health science popularization short videos related to cerebrovascular diseases on two major Chinese short video platforms, TikTok and Kuaishou. Our results showed that there were more videos on TikTok, and these videos were more popular compared to videos on Kuaishou. We found that 88.17% of the videos on TikTok have a GQS score ≤ 3, 87.43% of the videos have an mDISCERN score ≤ 3, while 97.59% of the videos on Kuaishou have a GQS score ≤ 3, and 93.85% of the videos have an mDISCERN score ≤ 3. These results indicated that the overall quality of health science popularization short videos related to cerebrovascular diseases on TikTok and Kuaishou was moderate to low, which was consistent with the findings of Ge et al. (32) in their study on the quality evaluation of stroke-related videos. The results may be related to the entertainment-oriented nature of short-video platforms and the fact that health science popularization via new media is still in its initial stage. As commercial entertainment platforms, short video platforms are primarily profit-driven (33) and tend to prioritize the promotion of lighthearted, engaging content over scientifically rigorous materials. With the gradual surge in health-related videos on these platforms, some creators have resorted to oversimplifying or exaggerating scientific content to pursue traffic. This phenomenon is mainly due to the low appeal of complex and accurate scientific information to the general public, whereas false or misleading health information spreads more easily (15). Furthermore, there is no unified evaluation standard for the quality of health science popularization short videos until now, which makes it impossible for government agencies and new media platforms to exercise comprehensive control and supervision over the quality of health science popularization short videos.

This study found that health science popularization short videos corresponding to different keywords show differences in both quantity and quality. In terms of quantity, videos related to “cerebrovascular diseases” and “cerebral apoplexy” accounted for 48.2% in total, while those associated with “subarachnoid hemorrhage” only made up 8.3%. This discrepancy may be related to the varying levels of public attention toward different diseases and creators’ perceptions of the prevalence of these conditions. In terms of quality, videos about “transient ischemic attack” and “subarachnoid hemorrhage” obtained the lowest operability scores, indicating that the science popularization content for these two disease types is insufficient in guiding the public to take specific actions. Given that transient ischemic attack is an important warning sign of stroke (34), and subarachnoid hemorrhage is characterized by acute onset and high mortality (35), it is imperative to enhance the practical guidance of science popularization content for such diseases, so as to help the public identify key symptoms and take correct and timely responses.

We found that the majority of the health science popularization videos related to cerebrovascular diseases published on both platforms were created by medical professionals. This might be associated with national policies such as the “Healthy China 2030” Planning Outline. Encouraged by these policies, an increasing number of medical personnel are using short video platforms to disseminate health knowledge (23). Previous studies have shown that viewers preferred to viewing health science popularization videos from hospital accounts (11). This could be explained by the fact that the public hospitals gained more trust from the public. However, our findings indicated that the quality of health science popularization videos released by medical professionals was far from satisfactory. This may be related to the fact that they have limited time for content creation, lack video production skills, and have access to scarce training resources (36). Meanwhile, TikTok has prohibited medical professionals from engaging in live-streaming e-commerce, and its fan group function was also discontinued in 2023 (37). Such regulations may have diminished medical professionals’ motivation to produce high-quality health education videos. Our findings were inconsistent with the conclusions of Wang et al. (38). In their study evaluating the quality of stroke-related health science popularization short videos on TikTok, they found that videos created by medical professionals were significantly superior to those produced by other creators. This discrepancy might stem from the fact that they only selected the top 100 videos with the highest number of likes, whereas our study included a much larger sample of videos.

We found that the number of likes, favorites, comments and shares of health science popularization short videos on TikTok platform were higher than those on Kuaishou platform, indicating that users on TikTok platform had a higher level of participation and more interactions. This finding was consistent to previous research which found TikTok exhibited the highest engagement levels (39). Our study revealed that TikTok videos related to cerebrovascular diseases consistently outperformed those on Kuaishou across all evaluated metrics, including mDISCERN, GQS, Understandability, and Actionability scores. These findings aligned with previous studies which noted platform-specific variations in the quality of health information, such as the higher reliability of TikTok videos compared to other platforms in contexts like Liver cancer and acute pancreatitis (40, 41). This discrepancy may be associated with TikTok’s more sophisticated content dissemination algorithm (42), which takes interactive metrics as its core recommendation criteria. While prioritizing interactivity, TikTok’s algorithm also incorporates content quality into its evaluation, thus creating a positive feedback loop where high-quality health education videos generate greater user engagement and, in turn, achieve wider exposure. In addition, compared with Kuaishou, TikTok has implemented a more stringent content review mechanism. It has launched a Knowledge Creation Support Program and provides targeted traffic incentives for professional and valuable content.

However, it is worth noting that the general public tends to prefer videos with highly entertaining, sensational, or fragmented content (11, 43, 44). Such content usually generates higher real-time engagement, and the platform recommendation algorithms amplify their reach, thereby overshadowing the visibility of high-quality yet low-engagement health education videos. This phenomenon reflects the problem of short-video platforms’ over-reliance on interactive metrics, indicating that platforms could further optimize their recommendation mechanisms in the future to strike a balance between interactivity and content value. In addition, platforms could establish a dedicated review team for health-related content and implement a mandatory quality labeling system. They could offer traffic incentives for high-quality videos and facilitate collaborations between medical professionals and individual creators to co-produce content, thereby enabling professional and engaging high-quality videos to gain greater exposure through algorithmic recommendations.

The strong correlations between video duration and the four quality assessment metrics (mDISCERN, GQS, Understandability, and Actionability) observed in this study, coupled with its dominant weight in the Random Forest regression models (28.10 to 53.92%), highlighted video duration as a pivotal influencing feature of the quality of cerebrovascular disease health science popularization videos on TikTok and Kuaishou. Our finding was consistent to the previous studies which also found that video duration was a key predictor of the quality of the health science popularization short videos (45–47). This may be associated with the fact that shorter videos fail to convey sufficient information, whereas longer ones contain more comprehensive and complete health content. Therefore, publishers may appropriately extend the duration of health science popularization short videos.

Additionally, metrics such as likes, comments, favorites, and shares showed moderate correlations with Understandability and Actionability scores, and their weights in the Random Forest regression models were non-negligible. These features, to a certain extent, reflected the popularity of the videos (48). Our findings were consistent with Yeung’s research which reported that attention-deficit/hyperactivity disorder (ADHD) videos featuring personal experiences scored high in Understandability and were the most popular (49). This might be related to the fact that content with high actionability or easy understandability is likely to trigger higher engagement, and videos with higher engagement will be further disseminated and diffused under the recommendation of algorithms. Meanwhile, we found no correlations between video engagement metrics (likes, favorites, comments, and shares) and the mDISCERN and GQS scores of the videos. This research result was inconsistent with that of Gong et al. (50). This may be related to the bias resulting from the researchers’ manual search on the platform and the inclusion of short videos with relatively high user engagement.

The advantages of our research were as follows. First, we selected TikTok and Kuaishou, the two most representative short video platforms in China. Compared with other studies, this approach overcomes the limitations of single-platform research. These platforms, with the largest user bases in China and high user loyalty, have become an indispensable part of the daily lives of internet users. Second, we employed web crawler technology to collect videos. Unlike other studies that only select the top 100 videos based on the platform’s comprehensive rankings, the data collected through crawling technology is more comprehensive, making the research objects more representative. Third, we utilized multiple evaluation tools (GQS, mDISCERN, and PEMAT-A/V) to assess the quality of cerebrovascular health science popularization videos, enabling a comprehensive and multi-dimensional measurement that reflected distinct aspects of content quality. This multi-tool approach avoided the limitations of single-instrument assessments, and thus enhanced the robustness and validity of our evaluations. Furthermore, we conducted a further analysis of the correlations between video quality and video features including account type, video duration, likes, favorites, comments, and shares. The aim was to identify the key factors influencing video quality, thereby providing video creators with better creative ideas.

However, our research also had some limitations. First, the selection of video keywords in this study may not cover all relevant expressions of cerebrovascular diseases, resulting in a certain degree of selection bias. Future research could expand the scope of keywords by incorporating natural language processing (NLP) techniques. The algorithms of short video platforms are dynamic, leading to temporal variations in video exposure and visibility. This study could not fully eliminate the confounding effects of algorithmic fluctuations on sample selection, which imposes certain constraints on the representativeness and stability of the study sample. Second, despite the fact that both raters received relevant training prior to scoring, the research tools employed were relatively subjective. Coupled with visual fatigue caused by raters viewing a large number of videos, this might result in relatively subjective scoring outcomes. Future research could adopt more objective evaluation methods to assess the quality of health education videos. Third, our study is limited to the data of health science popularization videos, and does not incorporate the behavioral data of video audiences. Future research could integrate the behavioral data of video audiences to further explore the relationship between health science popularization videos and audience behaviors.