Depression can be seriously harmful to one's physical and mental health (1, 2). Early detection of depression is crucial for timely intervention and for improving patient prognosis. Currently, the commonly used methods for early detection of depression are health questionnaires (3), sleep pattern analysis (4), and heart rate variability (HRV) analysis (5). Health questionnaires, such as the Patient Health Questionnaire (PHQ-9) and General Anxiety Disorder Scale (GAD-7), are widely used due to their simplicity and convenience (6). However, self-reported data are often subjective and susceptible to biases, such as patients' preconceptions and memory recall. To reduce the subjectivity inherent in health questionnaires, sleep pattern analysis has been proposed as an alternative method for assessing mental health status (7). Nevertheless, sleep issues may arise from multiple causes, such as physical illness, lifestyle, and even the sleep environment. Additionally, the accuracy of sleep tracking by consumer wearable devices remains limited, so the reliability of sleep pattern analysis needs further improvement (8).

In recent years, HRV has gained increasing attention as an objective and quantifiable method for early depression detection (9). Compared with health questionnaires and sleep pattern analysis, HRV has the advantages of objectivity and easy quantification, so it has been widely applied in depression detection. HRV refers to the variability of the time interval between consecutive heartbeats and is an important indicator of autonomic nervous system (ANS) function (10). It reflects the balance between the sympathetic nervous system (SNS) and the parasympathetic nervous system (PNS). Lower HRV is usually associated with increased SNS activity and decreased PNS activity (9). In 2019, Hartmann et al. (11) studied the HRV between 62 depressive individuals without antidepressant medication and 65 healthy controls. The results revealed differences in HRV parameters between the depressive patients and the healthy controls at baseline, and the changes in HRV parameter values are correlated with changes in symptom severity of depression. In 2018, Koch et al. (12) conducted a comprehensive analysis of 21 studies on patients with major depression, including 2,250 depressive patients and 1,982 healthy controls. The results showed that all the measured HRV parameters of patients with major depression were lower than those of healthy controls. In 2023, Wu et al. (13) conducted a meta-analysis of 43 papers on depressive patients. The comprehensive analysis results showed that the HRV measurement parameters of depressed individuals were lower than those of healthy controls, except for LF/HF, suggesting that depressed people may be at a higher risk of cardiovascular diseases than healthy people.

Although HRV is closely related to depressive symptoms and can be used as an effective biological indicator for early detection of depression, imbalanced data usually exist when collecting HRV data from participants, especially when sample sizes are small (14). Imbalanced data indicates that the sample size of the majority class is much larger than the sample size of the minority class, and it is quite common in medical diagnosis (15). When the value of minority class samples overwhelms that of majority class samples, the effect of imbalanced data becomes particularly significant, such as in the detection of malignant tumors (16). However, traditional classification algorithms usually assume that the training dataset is balanced, or assume that the misclassification loss of different classes is the same (17, 18), which is not applicable in imbalanced diagnosis data. Consequently, how to train machine learning algorithms that can effectively classify imbalanced HRV data remains an urgent problem (19).

To address this issue, data resampling techniques have been widely adopted to adjust the class distribution. Among them, the Synthetic Minority Oversampling Technique (SMOTE) and its variants have proven particularly effective in medical signal analysis. For instance, Hussain et al. (20) demonstrated that applying SMOTE to multimodal HRV features significantly improved the sensitivity and overall accuracy of congestive heart failure detection, verifying that synthetic oversampling can preserve the discriminative power of physiological signals. Beyond standard oversampling, hybrid strategies have also been proposed to further refine classification boundaries. Xu et al. (15) introduced a method combining SMOTE with Edited Nearest Neighbor (ENN), which not only balances the class distribution but also eliminates noisy samples from the majority class. Their results across various datasets suggest that such hybrid resampling is superior in handling the noise and overlap often found in complex medical data. These studies collectively indicate that data resampling is a viable strategy for improving the classification performance of imbalanced physiological data. Currently, studies on imbalanced data based on SMOTE typically use original datasets with large sample sizes (20–22), generally ranging from several thousand to millions. However, its effectiveness in the specific context of depression detection using limited and sparse HRV samples remains to be fully explored.

In this work, a refined SMOTE-ENN hybrid optimization method based on machine learning algorithms is proposed to address the classification challenges of limited and imbalanced sample distribution. The refined Synthetic Minority Over-sampling Technique (r-SMOTE) generates new minority class samples by in-line and off-line linear interpolation. Afterwards, the refined Edited Nearest Neighbor (r-ENN) under-sampling method is employed to remove most of the noisy data and retain selected boundary data to reduce the overfitting risk. The performance of the refined SMOTE-ENN method is demonstrated by optimizing the imbalanced HRV data from 321 participants, with each record containing nine features. These data are divided into seven ANS states, with minority class samples comprising less than 5% of the total data.

Four machine learning algorithms are employed to evaluate the classification performance after data optimization, including Support Vector Machine (SVM), Random Forest (RF), Neural Network (NN), and K-Nearest Neighbor (KNN). The classification results show that following refined SMOTE-ENN optimization, the overall classification accuracy of the four machine learning algorithms exceeds 91%, with the AUC (area under the receiver operating characteristic curve) values surpassing 0.92. In comparison to the classification results with classical SMOTE optimization of the four machine learning algorithms, the mean accuracy, precision, recall, and F1 score improved by 0.12, 0.12, 0.10 and 0.11, respectively. The refined SMOTE-ENN method improves the detection performance of machine learning algorithms in classifying limited and imbalanced data, providing valuable technical support for the early detection of depression.

Currently, most public datasets focus on binary classification, e.g., Stress vs. Rest in WESAD, or Arrhythmia vs. Normal in MIT-BIH. To the best of our knowledge, there is no public dataset that provides the fine-grained, 7-category taxonomy of autonomic nervous system states used in this study. This scarcity actually highlights the novelty and clinical value of the proposed categorization system. On the other hand, classical SMOTE positions newly generated samples on the lines connecting the original samples in the sample space. The refined SMOTE-ENN introduces off-line interpolation, which expands the data space and increases the diversity of generated data through nonlinear transformations introduced by trigonometric functions, overcoming the limitations of linear interpolation in the traditional SMOTE method. The proposed method demonstrates good optimization capabilities for imbalanced datasets, especially when the dataset size is small.

The overall performance of the four classification models is presented in Table 5. Accuracy is defined as the ratio of the number of samples correctly predicted to the total number of samples, reflecting the overall performance of the model. Precision is defined as True Positive/(True Positive + False Positive), indicating the reliability of the model's prediction of positive samples. Recall is defined as True Positive/(True Positive + False Negative), reflecting the model's ability to capture positive samples. The F1 score is defined as the harmonic mean of the precision and recall, reflecting the balance between these two parameters. The classification accuracy for all models exceeds 0.91, and the F1 score exceeds 0.9. Specifically, KNN demonstrates the best performance, with the overall classification accuracy of 0.97 and F1 score of 0.96, indicating that all four machine learning models have achieved strong classification results on the optimized dataset.

The confusion matrices of the four machine learning models are shown in Figure 4. The classification results for labels 2, 5, 6, and 7 are particularly strong, with label 2 showing slight overfitting in the SVM model. This may be due to the limited amount of original data for label 2, resulting in the synthetic data being overly concentrated in the feature space. To solve this issue, the k-value can be appropriately increased in the r-ENN to introduce a small amount of noisy data, thereby enhancing the model's generalization ability. The classification accuracy for label 1 is the lowest across the four models, which may be attributed to the generation of excessive noise data for this label by r-SMOTE. To address this issue, a smaller value of λ and θ could be used in r-SMOTE to ensure that the synthetic data is closer to the original data in the feature space.

The classification results on the data using the traditional SMOTE method of the four models are summarized in Table 6. For the SMOTE optimization, the mean accuracy, precision, recall, and F1 score are 0.82, 0.82, 0.83, 0.82, respectively. After applying the refined SMOTE-ENN optimization as shown in Table 5, the mean accuracy, precision, recall, and F1 score are 0.94, 0.94, 0.93, 0.93, respectively, improving by 0.12, 0.12, 0.10, and 0.11, respectively. These improvements demonstrate that the refined SMOTE-ENN method is more suitable for optimization on datasets with small sample sizes and imbalanced data distributions compared with the traditional SMOTE, which effectively enhances the classification performance of the four models.

The Receiver Operating Characteristic (ROC) curves of the four machine learning models are calculated to evaluate the classification performance of the model, as shown in Figure 5. True Positive Rate (TPR) is taken as the vertical axis, which is defined as TPR = True Positive/(True Positive + False Negative), indicating the rate of samples that are correctly predicted as positive and are truly positive. False Positive Rate (FPR) is taken as the horizontal axis, which is defined as FPR = False Positive/(False Positive + True Negative), indicating the rate of samples that are incorrectly predicted as positive and are truly negative. The ROC curves of all models reach a TPR of more than 80% when the FPR is 10%, which is notably higher than the diagonal line, suggesting that the four models exhibit strong discriminative ability. All curves are positioned close to the upper left corner, and the AUC values exceed 0.92 for all models, demonstrating their capability to effectively distinguish between positive and negative samples. Two similar previous studies are compared with this work, and the results are shown in Table 7. Compared to previous classification results using the SMOTE method on large samples, the refined SMOTE-ENN has certain advantages in optimizing and classifying small datasets.

In some cases, such as the SVM model for label 2, the AUC value reaches 1.0, suggesting potential overfitting. The main reasons are as follows: refined SMOTE-ENN generates new samples by interpolating between two real minority class samples. The synthetic samples are highly similar to the original samples, which might lead to data leakage and overfitting. Furthermore, the experimental dataset has a small number of samples, with some classes having fewer than 10 samples. Refined SMOTE-ENN generates a large amount of data based on these few points, causing the features of that class to be extremely densely distributed in certain narrow regions and resulting in overfitting. To address these issues, samples can be generated only at the classification boundaries, instead of across all minority class samples. Additionally, the data processing flow can be further adjusted to prevent data leakage.

The feature importance distribution of the four models is evaluated to examine the clinical interpretability of HRV classification, as shown in Figure 6. The feature importance rankings vary across models. For instance, in the SVM model, the three most influential features are HR, SDNN, and rmSSD, while in the RF model, the top three features are SDNN, HF, and LF. Table 8 presents the ranking of the nine features across all four machine learning models. The results indicate that SDNN has the highest mean rank of 1.5. SDNN represents the standard deviation of all NN intervals, which is a key indicator of overall heart rate variability and exerts the most significant influence on HRV classification. A high SDNN value typically signifies good ANS performance and robust heart adaptation to external stressors. Conversely, changes in the standard deviation of NN intervals are often associated with significant shifts in HRV. The feature with the lowest ranking is TSP, suggesting that TSP has the least impact on HRV classification. This may be due to the fact that, although ECG signals from healthy individuals and depressed patients exhibit varying power levels across different frequency bands, their total power sum is similar, leading to minimal distinction in this feature.

A limitation of this study is the lack of external validation on public datasets. This is primarily due to the unique nature of the 7-category autonomic nervous system labeling protocol, which does not directly map to the binary or ternary labels (e.g., stress/non-stress) found in currently available public databases. Therefore, the generalization performance of the proposed model on populations with different demographic distributions remains to be verified in future studies. Furthermore, this work is positioned as a pilot study demonstrating the feasibility of identifying complex autonomic states using HRV features. One of the primary goals is to establish the correlation between the proposed 9 features and the 7 functional states within a controlled cohort, and the cross-dataset generalization will be the critical next step of this work.

Currently, studies on imbalanced data based on SMOTE typically use original datasets with large sample sizes, generally ranging from several thousand to millions. However, its effectiveness in the specific context of depression detection using limited and sparse HRV samples remains to be fully explored. In this work, a refined SMOTE-ENN hybrid optimization method is proposed to address the classification challenges of limited and imbalanced sample distribution. The model utilizes the r-SMOTE algorithm to generate synthetic minority class samples by in-line and off-line linear interpolation, combining it with the r-ENN under-sampling technique to remove noisy data while retaining selected boundary data to reduce the overfitting risk. The experimental results show that after refined SMOTE-ENN optimization, the overall classification accuracy of the four machine learning models exceeds 91%, with the AUC values exceeding 0.92. In comparison to the classification results with classical SMOTE optimization of the four machine learning algorithms, the mean accuracy, precision, recall, and F1 score improved by 0.12, 0.12, 0.10 and 0.11, respectively. The classical SMOTE positions newly generated samples on the lines connecting the original samples in the sample space. The proposed method introduces off-line interpolation, which expands the data space and increases the diversity of generated data through nonlinear transformations introduced by trigonometric functions, overcoming the limitations of linear interpolation in the traditional SMOTE method. The refined SMOTE-ENN optimization method improves the detection performance of machine learning models and provides reliable technical support for the early detection of depression.

Despite the promising results, this study has limitations that point to directions for future research. First, the proposed method effectively addressed the data imbalance, and future studies should focus on collecting larger-scale datasets from multi-center clinical trials to further validate the generalization ability of the model. Second, relying solely on HRV features may limit the feature space; therefore, future work could explore multimodal fusion strategies by integrating HRV with other physiological indicators, such as electroencephalogram signals or voice acoustics. In terms of real-world applications, the proposed method demonstrates certain potential due to its computational efficiency and non-invasive nature, making it suitable for deployment on edge devices, such as smartwatches and health monitoring bands.