In this study, our dataset is sourced from the public dataset of the 2018 Tianchi Precision Medicine Competition¹, which is curated to explore the correlation between various common indicators and the progression of diabetes mellitus. Specifically, it facilitates the prediction of the progression of DM in a population (using blood glucose concentration as the key indicator), by analyzing clinical data and physical examination indicators from diabetic patients. Intelligent prediction of blood glucose levels aids in diagnostic support, while an interpretable predictive model, by analyzing the relationships between different indicators and blood glucose levels, can help expand medical and research perspectives. The dataset comprises 6,642 rows and 42 columns, representing 42 attributes for 6,642 patients, including ID, gender, age, date of physical examination, various indicators, and so forth. The last column is filled with blood glucose concentration, which is our target for prediction. BG is measured after an overnight fast and is often used to gauge the effectiveness of diabetes management on a somewhat longer scale.

In Table 1 , we present a detailed list of the attributes from the dataset used for our prediction task. These attributes are typical of tabular data and can be broadly divided into two categories: categorical features, such as gender, and numerical features, including age and various medical test indicators. We have excluded attributes like user ID and physical examination date from the model building process, as they do not have a direct correlation with health conditions. Apart from basic demographic information like age and gender, the dataset includes 37 medical test indicators. These indicators are further categorized into six groups (B-G) based on the specific bodily functions they assess, such as liver function tests and lipid profiles. Table 1 also shows the extent of missing data for each feature. We observe that, besides basic demographic data, all medical indicators have some level of missing data, with the number of patients missing these data ranging from 21 (e.g., WBC) to 5110 (e.g., HBsAg). The presence of missing data in this dataset highlights a key challenge in our research. That is, how to develop effective algorithms to address widespread data gaps, thereby enabling the creation of a universally applicable diabetes progression prediction model.

In the realm of tabular data processing for predictive tasks (regression or classification), traditional approaches have predominantly relied on tree-based ensemble learning algorithms, such as Random Forest Svetnik et al. (25), Gradient Boosting Machines (GBM) Friedman (26), XGBoost Chen and Guestrin (27), and LightGBM Ke et al. (28). These algorithms excel with structured data due to their effective handling of non-linear relationships between features Nguyen and Byeon (29). Additionally, tree-based models are generally more interpretable, a quality of significant importance in various business and decision-making contexts. In contrast, deep learning algorithms are more adept at managing large-scale unstructured data Shwartz-Ziv and Armon (30), such as images Shen et al. (31), text Chatterjee et al. (32), and audio Purwins et al. (33). Deep learning models are capable of autonomously extracting complex, hierarchical features, which is particularly crucial when dealing with intricate data types like pixel data or natural language.

In recent years, with the advent of the TabNet model Arik and Pfister (22), the application of deep learning in processing tabular data has seen a notable increase Li et al. (19) Yan et al. (34) Chen et al. (35). TabNet, a deep learning model specifically designed for tabular data, stands out for its integration of the Attentive Transformer and significant interpretability. The Attentive Transformer enables TabNet to dynamically select and focus on the most crucial input features, thereby enhancing predictive performance and the ability to handle complex datasets. Furthermore, TabNet’s design emphasizes model interpretability, primarily due to its dynamic feature selection and attention mask mechanism. This allows for effective identification and utilization of the most predictive features, improving both the model’s performance and interpretability. Such capabilities are particularly vital for applications requiring high levels of transparency and interpretability.

However, the direct application of the existing standard TabNet model to predict blood glucose concentration encounters several practical issues: (i) How can we effectively tackle the issue of missing data in tabular datasets, as exemplified by the data missingness detailed in Table 1 ? (ii) How can the robustness of the TabNet model’s predictive accuracy be enhanced to ensure reliable predictions, even in the face of fluctuations in individual indicators? (iii) How can the TabNet model be optimized to better mine and utilize the inherent associations within data features? As Table 1 demonstrates, common test indicators can be naturally grouped into different feature sets based on the bodily functions they assess. Applying heuristic information from these sets not only has the potential to enhance the model’s predictive power but also provides a basis for set-level analysis and interpretation of the significance of various features. This aspect is crucial for a deeper understanding of the model’s decision-making process and for improving its applicability in clinical settings.

In the following section, we will delve into a detailed exposition of how we build upon the standard TabNet to construct our Group-informed Attentive Framework for Diabetes Mellitus Progression Prediction (GADMP) model. This discussion will encompass the methodologies employed to adapt the TabNet architecture to our specific research context.

In related work, two primary methods have been employed to address the issue of missing features: one approach involves filling in missing values with the mean or mode of the corresponding columns in the training dataset Jerez et al. (36) Neves et al. (37), while the other leverages machine learning algorithms, such as Random Forest or k-Nearest Neighbors, to predict these missing values Pantanowitz and Marwala (38) Tang and Ishwaran (39). However, given the severity of missing features in medical examination data, as evidenced in Table 1 , relying on simplistic mechanisms or algorithms for data imputation can be ineffective under the constraint of limited known information, potentially impacting the accuracy of target outcome predictions.

In this work, we adopt a group information-inspired multi-stage strategy to achieve as accurate data imputation as possible. Our approach is as follows: (i) We use feature columns with less significant missing phenomena in tabular data (i.e., the 19 features in Table 1 : A ₁, A ₂, F ₁ − F ₁₂, and G1 − G5) as clustering features and apply the DBSCAN algorithm Ester et al. (40) to divide all patients into different groups; (ii) For each patient, we calculate the mean values of all other features (excluding clustering features) within their group to create m additional cluster-level features (averaged indicators of patients in the corresponding cluster); (iii) Using these augmented features and the existing non-missing feature values, we progressively predict missing values using the LightGBM algorithm Ke et al. (28). Specifically, we first predict the missing values of the B ₁ feature, then use the predicted B ₁ values as known information to predict the next set of missing values, and continue this process until all missing feature values have been predicted.

We answer two key questions to elucidate the motivation behind adopting a group information-inspired multi-stage strategy for missing data imputation: (i) Why is clustering performed initially? In medical examination cohorts, individuals within the same group often exhibit similarities in various indicators Sakib et al. (41) Wahlqvist et al. (42). This implies that the missing feature values of a patient in a group might be similar to the corresponding indicators of other patients in the same group. Therefore, we utilize the feature values of corresponding feature columns within the group as additional information, providing a more comprehensive basis for predicting missing values. (ii) Why predict missing values in a stepwise manner? As Table 1 shows, there is a tendency for missing data to occur across all tests within the same category, likely because patients tend to choose tests based on the bodily functions that these tests are intended to assess, leading to entire categories (like Lipid Profile, feature group D) being skipped. Directly using a unified model to predict multiple missing indicators for a patient might not yield optimal results. Hence, we adopt a stepwise filling approach, where each prediction cycle uses the values predicted in the previous cycle as known information to build a new model for predicting the next missing indicator.

Our strategy for missing data imputation employs the DBSCAN clustering algorithm and the LightGBM prediction algorithm. This choice is driven by: (i) DBSCAN’s ability to cluster based on data point density, automatically determine the number of clusters, robustness to noise, and suitability for tabular data; (ii) LightGBM’s efficiency in handling complex relationships and large-scale data, particularly apt for datasets with numerous features and in scenarios with missing data. The results of our experiments further validate the effectiveness of our proposed group information-inspired multi-stage data imputation strategy.

To enhance the robustness of predictions for tabular data, existing work has primarily focused on the design of machine learning Shah and Pradhan (43) Kudari (44) or deep learning algorithms Rajkomar et al. (45) Somepalli et al. (46), with a lack of solutions targeting underlying factors at the data level. Considering the unique nature of medical examination indicators, where patients’ metrics can fluctuate within a certain range, fluctuations in individual indicators at the data level can fundamentally affect the predictions of a trained model, thereby impacting its robustness. In this work, we propose a dual-layer data augmentation approach, encompassing both sample augmentation and feature augmentation, to further enhance the robustness of the TabNet model in predicting blood glucose levels.

Sample augmentation involves adding random perturbation values Δ f to specific features f of a patient, where −ϵf < Δf < ϵf. This method allows us to create augmented samples, expanding the training environment and enhancing the model’s robustness against fluctuations in indicators. Feature augmentation involves adding group-level features to each sample. Assuming that samples within the same group share similar test indicator values to some extent, incorporating group-level features can help mitigate the impact of minor fluctuations in individual indicators within the group on the model. During the augmentation process, all features are used as clustering features, followed by clustering using the DBSCAN algorithm. We then select specific feature sets within each group and use their averaged indicator values to augment the feature columns of every patient in the group. This two-pronged augmentation strategy is designed to bolster the model’s resilience to variations in data, thereby improving its overall predictive robustness.

In practical implementation, for sample augmentation, we randomly select n features (perturbed features) from the feature columns (excluding gender and age) and add perturbation to them. The corresponding ϵf is set to one-twentieth (1/20) of the average value of each selected feature column. For each patient, we generate two augmented samples. The experimental results indicate that our proposed augmentation method not only bolsters the robustness of the TabNet model but also enhances its predictive accuracy. This strategy effectively counters the inherent variability in medical data, ensuring that the model remains reliable and precise, even in the presence of individual feature value fluctuations. By introducing controlled variability through augmentation, the model is trained to be more resilient to the variations commonly encountered in real-world data, thereby improving its applicability and effectiveness in clinical scenarios.

Current aggregation methods in diabetes management, such as those proposed by Sampath et al. Sampath et al. (47), primarily focus on combining glycemic control indices like the Low Blood Glucose Index (LBGI) to predict outcomes such as nocturnal hypoglycemia. These methods typically utilize linear combinations of individual glycemic indices derived from daily glucose measurements to enhance prediction accuracy. In contrast, our approach significantly advances these methodologies by integrating a more diverse array of clinical data across various categories. This broader spectrum of data allows our model to capture complex interactions between different health indicators, enhancing its applicability to various diabetes progression scenarios. Our innovative use of a deep attentive transformer to integrate features at an earlier stage and dynamically weigh their importance marks a substantial improvement over the traditional linear aggregation methods. Furthermore, we proactively address the challenge of missing data with sophisticated imputation strategies, ensuring robust performance across datasets with incomplete information. This comprehensive and advanced approach not only aligns with but also significantly advances current methodologies, offering a more effective tool for diabetes management in clinical settings.

In the standard TabNet model, individual-level feature masking is typically employed during the modeling process to learn and interpret the importance of each independent feature for model predictions. This individual-level masking mechanism has achieved commendable results in various prediction and classification tasks Arik and Pfister (22). However, for tasks like predicting diabetes progression based on physical examination and medical test indicators, this approach might actually impede model effectiveness. This is because medical test indicators, corresponding to different bodily functions, are naturally divided into distinct sets. Indicators within the same set, indicating the same bodily function, tend to have similar effects on diabetes progression prediction. Individual-level feature masking could lead to inconsistencies in the importance assigned to indicators within the same set, potentially causing conflicts in model predictions.

Therefore, in this paper, we explore the uniform impact of the same group of indicators on the model, that is, applying the same mask to features within the same set (group), guiding the model’s learning process. This group masking strategy prevents the model from overly relying on any single feature during training, thereby enhancing its generalization capability. Additionally, it allows the model to perceive heuristic information between feature groups, further leveraging the natural and tight correlations among features to improve the model’s predictive performance. This approach not only addresses the limitations of individual-level masking in the context of diabetes progression prediction but also harnesses the collective strength of feature groups to achieve more accurate and reliable predictions.

During our experimental process, we selectively applied group masking to different sets of medical test indicators. This approach allowed us to further analyze and summarize the impact of various bodily function indicators on the progression of diabetes. By implementing group-level masking, we aimed to facilitate researchers in understanding the influence of different bodily functions on diabetes at a group level. This method not only enhances the interpretability of the model but also provides valuable insights into how different sets of indicators, each representing specific bodily functions, contribute to the overall prediction of diabetes progression. Such an approach is instrumental in advancing the understanding of diabetes from a more holistic and function-oriented perspective.

In evaluating the performance of all compared methods, we employ two standard metrics: Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE). MAE is defined as the average of the absolute differences between the predicted and actual values, mathematically represented as MAE  = 1 n ∑ i = 1 n | y i − y i ^ | , where y_i and y_i denote the actual and predicted values, respectively, and n is the number of observations. This metric is particularly useful for its interpretability and robustness to outliers. On the other hand, MSE, defined as M A E   = 1 n ∑ i = 1 n ( y i − y i ^ ) 2 , provides a measure that penalizes larger errors more severely by squaring the differences between predicted and actual values. RMSE, given as R M S E   = 1 n ∑ i = 1 n ( y i − y i ^ ) 2 , is essentially the square root of MSE and provides a more interpretable scale of error magnitude. The employment of these metrics facilitates a thorough assessment of model accuracy, considering both the magnitude of errors and the significance of larger deviations in predictions.

The aforementioned metrics (i.e., MAE, MSE, and RMSE) are selected for their ability to quantify the precision of blood glucose predictions at specific time points. While our approach focuses on the accuracy of predictions rather than tracking diabetes progression over time, these metrics are critical for assessing the current state of diabetes in patients. High accuracy in these predictions can indicate stability, improvement, or deterioration in the patient’s diabetic condition, which is crucial for effective disease management. By reliably determining the state of diabetes at each measured point, these metrics facilitate informed clinical decisions, helping to tailor interventions and adjust treatment strategies.

In this section, we conduct a comprehensive analysis and comparison on the method we proposed from multiple perspectives.