This study was approved by the institutional review board (IRB) of the participating institution (COEP Ciencias Médicas). All procedures followed the ethical standards for human subjects research, and a waiver of informed consent was granted due to the retrospective design, in accordance with national regulations. The original data follows the rules from the medical records data storage, which must be retrieved only for patient benefit and must be safely stored for 20 years. Data was anonymized by removing names, addresses, document IDs, and other sensitive information. We report only aggregated patient statistics to avoid any risk of reidentification.

We conducted a retrospective observational study with children and adolescents receiving emergency psychiatric care. The dataset comprises 2,365 health records of patients admitted to CEPAI-FHEMIG, some with multiple admissions, resulting in 1,720 unique patients (Lobato Machado et al., 2022). The data was collected from June 2017 to May 2018, and the subjects' ages range from 1 to 18 years. The dataset includes 27 sociodemographic features, such as race, gender, place of birth, household geographical location, school situation, living situation, and 123 clinical features, encompassing details such as reasons for seeking psychiatric assistance, family history of mental disorders, substance abuse information, psychiatric diagnoses received after treatment at the facility, neuro-psychomotor development delays, and previous traumatic events, among other relevant aspects.

We excluded 75 admissions where patients: (i) had no information at all (5); (ii) had only personal information (37); (iii) had no information about the motivations for looking for help or diagnoses (31); (iv) had not completed the screening stages at the center (1); or (v) had missing age information (1). Regarding the features, 12 of 110 contained textual information (e.g., a written reason for the patient leaving school). On average, more than 87% of the data in these 12 features were missing, and therefore these features were discarded. The original dataset also contained 21 features related to suicidal behaviors, which were, unfortunately, mostly missing (more than 80% of instances were missing) and were therefore excluded due to the high risk of data imputation in this context.

With the assistance of psychiatry professionals, out of the remaining 77 features, 57 were considered more informative for the tasks at hand. From the selected features, 33 were categorical. These categorical features were binarized, yielding 21 new binary features (e.g., whether the patient had depression, schizophrenia, or learning difficulties). The final dataset had 2,289 admissions for 1,687 unique patients (1,071 male and 616 female), with each admission described by 154 features.

Three target values—self-harm, suicidal ideation, and suicidal attempt—were extracted from the patient's motivation for seeking help, which was the prediction target. In this scenario, we had 337 out of 2,291 cases of self-harm (14.71%), 309 cases of suicide ideation (13.48%) and 323 positive suicide attempts (14.09%).

Self-harm was operationally defined as any non-suicidal manifestation of self-poisoning or self-injury lacking the explicit intent to induce mortality. Suicide ideation was identified in instances where individuals self-reported contemplation or strategic planning associated with suicidal tendencies. Concurrently, a suicide attempt was characterized as a deliberate act of self-poisoning or self-injury with the explicit aim of inducing mortality.

To build prediction models for all outcomes, the classic methodology for handling data was followed. The first step was data preparation, as reported in the previous section. Next, the prepared datasets were used to train ML models, and oversampling was used to address class imbalance. As one of our main objectives is to interpret the contribution of each predictor to the model's decisions, we also employed SHapley Additive exPlanations (SHAP) (Lundberg and Lee, 2017) to understand predictions better. Each of these steps is detailed next.

The three ML models used in our experiments were Logistic Regression (LR), Random Forest (RF), and eXtreme Gradient Boosting (XGB) (Zaki and Meira, 2014). LR was chosen for its interpretability, whereas RF and XGB, as tree-based models, excel at capturing non-linear relationships and may facilitate the emergence of interpretable models. The experiments were conducted using the Scikit-learn (Pedregosa et al., 2011) and XGBoost libraries (Chen and Guestrin, 2016). A detailed description of these algorithms can be found in (Bishop 2006).

Given the imbalanced nature of the data, instances with suicidal behaviors were oversampled to compose 30% of the training set. Oversampling is a well-known technique that alters the data distribution in the training set to improve predictive performance by increasing the number of instances of the rarer class.

The models were trained on two distinct feature sets. The first set incorporates all available features and is denoted as “All Features” in the results table. The second set, referred to as “Feature Selection,” was generated using pre-selected features based on expert knowledge of the specialist who curated the dataset. It comprises features such as motivations for seeking help, diagnoses during medical hospitalization, whether it is the first-time admission to CEPAI-FHEMIG, and personal data, including gender, age, and the number of people in the household. We are aware that this type of selection can bias the model, and, for that reason, experiments with the full set of available features were also performed, and their results were compared with those obtained using only the features selected by the specialist.

During model training, the target variables were excluded from the set of predictors. Considering the relationship between suicide ideation and suicide attempt, ideation was used as a predictor for suicide attempt. Still, the suicide attempt was removed from the data used for suicide ideation prediction. For the self-harm prediction task, both suicidal ideation and suicide attempts were used as predictors, in line with previous research (Mars et al., 2019b).

Each model generated provided feature importance scores for the predictors, enabling us to compare the most significant features across classification tasks. Additionally, SHAP values were generated (Lundberg and Lee, 2017). SHAP is an approach rooted in game theory and frequently used to explain the output of ML models. It reveals the extent to which each feature contributes to the target feature. The interpretation of SHAP is akin to feature importance, but SHAP goes further by indicating whether each feature has a positive or negative relationship with the predicted value. It allows a richer understanding of feature impact, since it allows us to identify the most influential predictors associated with increased risk of suicide-related behaviors.

To assess model performance, we used a stratified 10-fold cross-validation strategy, following the methodology detailed in (Bishop 2006). To prevent data leakage, we employed patient-level grouping: all admissions associated with a single individual were restricted to a single fold, ensuring the model was never tested on data from a patient it had already seen during training. For each iteration, the dataset was partitioned into training (eight folds), validation (one fold), and independent test sets (one fold). The validation set was used exclusively for hyperparameter optimization, whereas the test set was used only to assess model generalization. It is important to note that, in experiments using oversampling, the test set retained the original data distribution.

The performance of the models was assessed using four metrics:

Area under the receiver operating characteristic curve (AUC): assesses the model's performance across all classification thresholds of the true positive (TP) rate and false positive (FP) rate, providing a comprehensive view of the model's ability to discriminate between classes.

These metrics collectively provide a comprehensive evaluation of the model's performance on classification tasks. However, in the context of this paper, the sensitivity is considered more important than the other metrics, because the impact of not identifying a positive case of suicide attempt (or ideation, or self-harm) is considered more serious than misclassifying a negative case. This metric will be used to evaluate the models' performance relative to one another.

Regarding hyperparameter tuning, for XGB, a grid search was performed to identify a near-optimal parameter combination. This search encompassed four parameters: (1) the number of boosting iterations, (2) the ratio of features used to train a tree, (3) the maximum tree depth, and (4) the ratio of training instances in a subsample.

For RF, five parameters were assessed: (1) number of trees in the forest, (2) minimum number of samples required to split an internal node, (3) maximum depth of the tree, (4) minimum number of samples required to be at a leaf node and (5) whether bootstrap samples are used when building trees. Due to the large number of possible parameter combinations, an initial random search was conducted over the hyperparameter space. Subsequently, the best values identified in this search were evaluated using a grid search to determine the best parameter configuration. Table 1 presents the best parameter values found for Random Forest in the three scenarios investigated.

Once the parameters for both RF and XGB models were determined, further tuning involved adjusting their classification thresholds, which determine whether an example is classified as positive or negative (default value is 0.5), using the validation set. This experiment was also performed for LR. In this scenario, we balanced sensitivity and specificity using the F1 score, which is the harmonic mean of PPV and sensitivity. After that, the models were evaluated on the test sets.

The three models underwent testing with both the complete set of attributes and the reduced set of features (“Feature Selection”). Both experiments were conducted with and without oversampling.

Figure 1 shows the basic statistics of the dataset. As observed in Figure 1a, most patients (~62%) were between 13 and 18 years old. Figure 1b shows the number of admissions to CEPAI-FHEMIG. Although most of the patients were admitted only once, 21% of the individuals were admitted at least twice in the studied period, revealing an incidence of repeated suicide behaviors, in line with the literature. Figure 2 illustrates the distribution of diagnoses given by the psychiatric professional (after assessment). Depression and general mood disorders were diagnosed in 21,69% of all admissions, being the most common diagnosis.

Table 2 presents descriptive statistics for a subset of features, including motives that led the family to seek help and other dependent features considered relevant to the model, using suicide attempt as the dependent variable. The most common reasons were related to agitation, aggressiveness, irritability and learning difficulties, which are not traditionally associated with suicidal behaviors. Factors commonly related to suicidal behavior include depression, self-harm, anxiety, and self-aggressiveness, which were present in 19,07%, 14,7%, 14,36% and 5,28% of the admissions registered, respectively, and follow the distribution in the table.

Suicide-related behaviors were among the motivations given by patients or their guardians to seek psychiatric care. As previously reported, the motivations were used to define the model's outcome variables. The dataset had 337 out of 2,291 cases of self-harm (14.71%), 309 cases of suicide ideation (13.48%) and 323 positive suicide attempts (14.09%).

Tables 3–5 present the results of the prediction models predicting suicide attempts, suicidal ideation, and self-harm, respectively. In the tables, the acronym OV denotes Oversampling, referring to models in which the data distribution was modified. The ratio of the outcome we want to predict corresponded to 30% of the training dataset. The RF model achieved the best overall performance across the four metrics for all three tasks. Oversampling does not affect the current sample's results, as the method's results with and without oversampling differ only in confidence intervals, which overlap. For suicide attempt and ideation prediction, the best-performing model used only the “Feature Selection” subset of features, achieving a sensitivity of 0.7118, specificity of 0.7592 and PPV of 0.3350 for suicide attempt prediction. For suicide ideation prediction, the method presented a sensitivity of 0.7804, specificity of 0.7763 and PPV of 0.3602. In the case of self-harm, the best model was trained using all available features, resulting in a sensitivity of 0.6937, specificity of 0.8177 and PPV of 0.4086.

In summary, a combination of sensitivity above 0.69 and specificity above 0.75 indicates accurate predictions, correctly identifying more than 69 out of 100 positive cases and more than 75 out of 100 negative cases. However, further studies are still needed to improve PPV. PPV indicates the likelihood that an individual with a positive suicide prediction is truly going to commit suicide. Although in this context false positives are more desirable than false negatives, more accurate models have better economic and social advantages and are more feasible. Improving the current model would require additional data, and we believe longitudinal data are key. We are pursuing efforts to obtain a more comprehensive dataset from CEPAI-FHEMIG. Given that we initially intend to use the system for triage, the burden of low PPV is smaller than that of a fully predictive system in operation. The worst-case scenario would prioritize an individual with low risk over one with higher risk.

Finally, it is important to say that we have analyzed the differences in the metrics regarding the training and test sets, and the model does not present any indication of overfitting. A cross-validation procedure was performed to obtain statistically meaningful results. However, we cannot guarantee the method will generalize to other contexts without further adaptation, and testing it in new scenarios is part of our future work.

As we are interested in identifying the factors that most influence the prediction of suicidal behaviors, the impact of each feature on the model's output was assessed using SHAP values. Figures 3–5 depict these values for the three tasks. In these plots, the horizontal axis indicates whether a particular value is associated with a higher or lower prediction value. The colors denote whether that attribute presents a high (red) or low (blue) value. For this analysis, the gender feature is coded as 0 for female and 1 for male.

For example, higher values of “Agitation,” “Gender” (which means the patients are male), and “Learning difficulties” tend to produce lower predicted values for a suicide attempt (axis x), suggesting that male patients reporting agitation and learning difficulties are less likely to make a suicide attempt than the patients characterized by the opposite values of these binary variables (i.e., female patients, with no signs of agitation or learning difficulties). On the other hand, patients with depression, self-harm and suicidal ideation as motives for looking for help, who were hospitalized in CEPAI-FHEMIG and diagnosed with depression, are more likely to have a suicide attempt.

For suicide ideation (Figure 4), intellectual disability is linked to decreased risk, while isolation and hallucinations are associated with increased risk. As for the self-harm model (Figure 5), the relationships of gender and age with its prediction seem to be more regular, with older female patients being associated with a higher risk, while a patient with no trauma history using only CEPAI-FHEMIG's social services is linked to decreased risk, highlighting the distinct factors contributing to each outcome.