“Tree Hole” AI robots were used to crawl users' messages in the “Zoufan” tree hole from November 6, 2018 to May 5, 2020. According to the certainty of suicide manner and the urgency of the time, a suicide risk rating was established by using Knowledge Graph technology (12). The Knowledge Graph is a graph-based knowledge representation and organization method, which can represent systematic, structured, and integrated domain-specific knowledge based on semantic technology (12, 15). The suicide risk classification standards are as follows (12): level 10 (suicide may be in progress), level 9 (suicide method has been determined and may occur soon), level 8 (suicide has been planned, and the suicide date is generally determined), level 7 (suicide method has been determined, and the suicide date is unknown), level 6 (suicide has been planned, and the suicide date is unknown), level 5 (expression of strong desire to commit suicide, and the suicide method is unknown), level 4 (suicidal desire has been expressed, and the specific method and plan are unknown), level 3 (intense survival pain, and no suicidal wishes expressed), level 2 (survival pain has been clearly expressed, and no suicidal wishes expressed), level 1 (survival pain is partially expressed, and no suicidal wishes expressed), and level 0 (no expression of survival pain noted). Messages were graded automatically by AI robots, and levels from 6 to 10 were marked as high suicide risk messages. The accuracy of the identification of suicide risk has reached 82% (12).

The age, gender and region of Sina Weibo users were described by frequency and percentage. The users' level of activity was analyzed by calculating the number of messages per hour.

Words in the messages were segmented by a Jieba tool. The keywords and a keywords co-occurrence matrix were extracted using a TF-IDF algorithm. Jieba tool is a method suitable for Chinese word segmentation, which can divide continuous word sequences into word sequences. TF-IDF (Term Frequency-Inverse Document Frequency) algorithm is a statistical method used to evaluate the importance of words in the text, and a weighted technology in information retrieval and data mining (16). Researchers divided high-frequency keywords into four classifications (suicide-related words, emotion expression words, role-relevant words and time and place relevant words) manually according to the contents. Keywords co-occurrence network analysis was constructed by Gephi 0.9.2 software. Each node represented a keyword; “degree” represented the frequency of the paired keywords that appeared together in each message; “edge” meant the connection between the paired keywords. The weight of the edge referred to the closeness of the paired keywords. The greater the weight of the edge, the closer the relationship between the two nodes.

To explore the suicide characteristics, researchers reviewed each message, manually annotated the suicide reasons, and identified if the users sent suicide invitations.

This study received approval from the Ethics Committee of Wuhan University School of Medicine (code: 2020YF0075).

In this study, there were totally 5,760 high suicide risk messages: 1,190 (20.66%) were level 6, 4,222 (73.30%) were level 7, 48 (0.83%) were level 8, and 300 (5.21%) were level 9.

According to users' registration information, 2,105 (64.79%) were female users, 762 (23.45%) were male users and 382 (11.76%) had no gender identified. Many users (1,818) reported a geographic location, including outside of China (249, 13.70%), Guangdong (236, 12.98%), Beijing (142, 7.81%), Jiangsu (115, 6.33%), and Sichuan (102, 5.61%), etc. (see Figure 1).

About 19.08% of messages were posted between 23:00 and 01:00, while the fewest messages (264, 4.58%) were posted between 05:00 and 07:00 (see Figure 2). The horizontal axis represents the time of the message posting and the vertical axis represents message volume per hour.

The keywords extracted by the TF-IDF algorithm were sorted by weight. The weight is a statistical measure calculated by TF-IDF to evaluate the importance of a word in a text or corpus (17). The top 20 keywords with great weight are shown in Table 1. The top 50 keywords are shown in the Appendix.

The high-frequency keywords were divided into four classifications, including suicide-related words, emotion expression words, role-relevant words and time and place-relevant words (see Table 2). Suicide-related words and emotion expression words were the most mentioned classifications. Jumping off buildings, wrist slashing, burning charcoal, hanging and sleeping pills were the most mentioned suicide methods.

The keywords co-occurrence network analysis showed that “world-leave,” “jump off buildings-really,” “jump off buildings-slash wrist,” “jump off buildings-hang,” and “suicide-burn charcoal” were the top five co-occurrence relationships. The top five related keywords with “pain” were “jump off buildings,” “burn charcoal,” “slash wrist,” “hang,” and “really.” The top five related keywords with “happy” were “jump off buildings,” “really,” “jump down,” “alive,” and “burn charcoal” (see Table 3).

Both of the keywords and the keywords co-occurrence network analysis showed that high suicide risk users had a strong willingness to end his or her life, and they were concerned about suicide methods and expressed great negative emotions. It is consistent with many original posts. For example, “I searched many methods, jumping into a river, burning charcoal, taking medicine, jumping off the building. The highest success rate should be to jump off the building……”

Among all high suicide-risk messages, 1,011 (17.55%) included suicide invitations and 327 (5.68%) were automatically rated as high suicide risk by AI roots but didn't express definite suicide ideation, such as “No, friends. The world may not be worth living, but there are also some things in the world that you will cherish.”

There were different reasons for sending suicide invitations to others. Some users want to suicide with others because they have no courage or they want to have a companion: “It's lonely to die alone, is there anyone die with me?” “I thought that if two people died together, maybe I'll have more courage.”

Negative cognition and emotions such as sadness, desperation and meaninglessness, mental disorders and specific problems in life such as family issues were common suicide reasons in high suicide risk messages. Some users were not willing to seek help, for example, “I know I have been ill for a long time, but I am reluctant to spend money to see a doctor…” Some users expressed suicidal ideation but did not implement it because they were worried about their family: “I want to jump off the building, but I worried my parents will be sad, I have no courage.”

First, previous studies mainly analyzed the content characteristics of posts on users' home pages, but this study included the comments with high suicide risk that were publicly available under the last post of a girl who killed herself named “Zoufan.” Because the “Zoufan” tree hole is one of the biggest “Tree Holes” in Sina Weibo, these comments can help to provide a better understanding regarding the inner world of users with high suicide risk. Second, our previous study about the general comments under the last post of “Zoufan” showed that 52% of comments expressed negative emotion. By comparing the characteristics of general comments with high suicide risk comments, relationship-related keywords (e.g., “I love you,” “boyfriend,” “break up”) and sleep-related keywords (e.g., “can't fall asleep,” “insomnia”) were frequently mentioned in general comments (13), but these were not high-frequency keywords in high suicide risk comments. In contrast, time and place relevant words (e.g., “school,” “hospital,” “roof”) were high-frequency keywords in high suicide risk comments but not in general comments (13). Suicide-related words such as jump off buildings, slash wrist, burn charcoal were high-frequency keywords in both but were mentioned more frequently in high suicide risk comments than general comments (13). These linguistic features help to form the portrait of high suicide risk users on social media, which is crucial for suicide monitoring and intervention.

The findings of this study provide evidence to improve the performance of the “Tree Hole Intelligent Agent,” which is developed based on knowledge graph technology (12). Ontology can be regarded as a specific type of knowledge graph (12). Now the “Tree Hole Ontology” has four parts: suicide ontology, time ontology, space ontology and desire ontology. This study showed that besides suicide-related words, time and place related words, emotion expression words and role-related words were also high-frequency words mentioned by high suicide risk users. Therefore, emotion ontology and relationship ontology can be added to improve the performance of “Tree Hole Intelligent Agent.” The results of content analysis in this study also contribute to the complement of “suicide dictionary” which can be used in suicide tendency analysis. The co-occurrence words help to enrich and expand the logical rules of knowledge-based methods (18), as domain knowledge to build deep learning based sentiment analysis algorithms (19).

The suicide crisis interventions require close collaboration among the government, society, social media platforms, healthcare professionals and the family. Government and the society should publicize mental health education and life education, help people to enhance positive coping skills, destigmatize mental illness and emphasize the importance of seeking help when necessary. Since expressing emotions is a protective factor of suicide (20), it is important to create a friendly and supportive environment for people to express their emotions. People with suicidal thoughts are more willing to seek help from mental health hotlines and the internet due to the convenience and anonymity (21), so the government, social media platforms and healthcare institutions could work together to set up mental health hotlines and online forums to provide professional consultations and interventions.

When building mental health service systems, the government and healthcare institutions should pay more attention to high-risk regions, actively publicize ways to cope with work and life pressure and focus on improving social support networks. Because high risk suicide users are more active at night, when healthcare providers deliver mental health education through social media, sending posts at night may be more effective (22). Due to the reduced availability of healthcare workers at night, it is necessary to develop automatic identification technology to enhance monitoring during the nighttime. Artificial intelligence technology could monitor the messages continuously and identify high suicide risk messages automatically. Since time is vital for suicide intervention, it is necessary to develop crisis intervention guidelines for suicide risk users on social media. Social media platforms can also introduce relevant policies like forbidding users to post messages including suicide invitations.

Family support is a crucial protective factor of suicide, and good family relationships can be a protective factor in preventing suicide (23). Therefore, the family needs to focus on the emotional status of their family members, notice early signs of suicide and seek professional interventions when necessary (24). Healthcare professionals should also incorporate family support and education into crisis interventions.

Because Sina Weibo has a 140-character limit for each comment, future studies could analyze users' comments in “Tree Holes” and posts on their home page together to get more comprehensive user portraits. The cross-cultural research of characteristics of suicide messages may also be implemented. Apart from the “Zoufan” tree hole, there are many other “Tree Holes” on Sina Weibo, so the automatic suicide monitoring model and intervention system developed for the “Zoufan” tree hole can be extended to other “Tree Holes.”

Firstly, messages in “Tree Holes” are fragmented and can't completely reflect the user's overall state. Secondly, because of the anonymity of online social media, the authenticity of messages and users' information can't be completely assured. Thirdly, using a software program to do the text analysis based on an identified algorithm may not accurately depict the characteristics of users' messages. For instance, sometimes software wrongly identifies the message advising others not to jump off buildings as a high suicide risk message. Future technology development can make up for this limitation, and software is expected to automatically identify and classify the cause of suicide to help researchers work more efficiently. In addition, classifications of keywords were based on keywords rather than deep semantics analysis of the complete text. More mature text semantic analysis technology can solve this problem in the future.