All electronic health record (EHR) data came from the Affiliated Hospital of Southwest Medical University. A total of 16,004 EHRs and 43 usable physical examination indicators were screened out, of which half explicitly contained information about a confirmed T2DM diagnosis ( Figures 1 , 2 ; Table 1 ). In order to capture characteristics of early-stage T2DM, the EHRs of T2DM patients were limited to their first record in the hospital system. The physical examination indicators could be divided into three categories: routine urine indicators (9 indicators), blood cell analysis indicators (24 indicators), and biochemical indicators (10 indicators) ( Figures 1 , 2 ), the name and the abbreviation of the indicators were shown in Supplementary Table S1 .

For these datasets divided according to the physical examination items, to facilitate their description here, we chose some standardized abbreviations: Whole physical examination indicators dataset (PEI dataset), Blood cell analysis dataset (BCA dataset), Urinalysis dataset (Uri dataset), Biochemical dataset (BioChem dataset).

On the collected data, three steps were performed before modelling:

The missing rate of each feature is shown in Supplementary Table S3 . When using the MVT-WDD-DI and MVT-WDD-BF algorithms for modelling, we kept the missing value of each dimension feature of positive and negative samples consistent to eliminate bias. (Details in the Supplementary Note 4 : Biased distribution misleading model using features).

The WDD algorithm, initially proposed by Maron for multi-instance learning problems (27), utilizes radial basis distance metrics to measure classification probabilities. We developed three new algorithms based on WDD in this work. Its non-linear nature and transparent framework make it particularly suitable for the medical field, where high interpretability in models is essential. Building upon the WDD framework, we have made enhancements to adapt it for medical classification problems, ensuring it can effectively handle missing values. Additionally, another reason for selecting the WDD method is its capability to provide a risk score for each sample, similar to the diagnostic approach of a clinical physician. This is achieved through its internal distance function, rather than merely outputting a categorical label. By decomposing the distance function at the feature level, we are further able to extract the feature-based criteria on which the model relies. This significantly enhances the transparency of the model. Unlike traditional models that offer limited insight into their decision-making process, WDD allows for a deeper understanding of how and why certain diagnostic conclusions are reached. This alignment with clinical practices not only aids in the interpretability of the model but also fosters greater trust and reliability in its application in medical settings. The ability to dissect the model’s reasoning at a feature level offers invaluable insights into the diagnostic criteria, bridging the gap between machine learning outputs and clinical decision-making.

The major principle of the WDD algorithm ( Figure 1C ) is to find an optimal point x = { x 1 , x 2 , … , x f } in the data space that maximizes the probability density that positive samples ( b i + = { b i 1 + , b i 2 + , … , b i f + } ) are near this point and minimizes the probability density that negative samples ( b i − = { b i 1 − , b i 2 − , … , b i f − } ) are near it, where i is the index of the sample, f is the number of features. The modified WDD algorithm can be represented as:

t is the target point, Pr ( x = t   |   b i + ) is the probability density that positive samples are near this point, and Pr ( x = t   |   b i − ) is the probability density of negative samples. The implicit functions could be given as:

where distance function (Dist) for both positive and negative samples is defined as the sum of weighted squares:

where k is the index of the feature.

The formulas yield a function to measure an optimal point based on both positive and negative samples, and the position of the point can be optimized during a deep learning process.

In this work, based on the idea of Maron’s work, we modified the distance function to achieve our predictive goals. One of the algorithms first imputes the data with missing values by k-nearest neighbour (KNN) and then uses WDD for prediction. The other two algorithms, named MVT-WDD-DI and MVT-WDD-BF, do not need to impute data but use penalty mechanisms for missing value tolerance.

As mentioned above, we modified the WDD algorithm proposed in Maron’s work to improve the algorithms’ ability to handle data containing missing values. The modifications are introduced in the following subsections:

Since the goal of WDD is to find a point to maximize the diversity density, we used a deep learning framework for implementation. We used the Adam optimizer to solve the optimal problem, and the loss function of the three algorithms is shown in Equation (14):

After training, we obtained the optimal position x t a r g e t from the dataset, which optimized Equation (1). Thus, all the samples could be classified by calculating the maximum diversity density from its instances:

Using Equation (15) calculated diversity density, a sample could be assigned a label by setting a threshold. In this work, we used the method of analysing the ROC curve from the training set. First, we calculated all the diversity density values of the samples in the training dataset. Then, we calculated the false positive rate (FPR) and true positive rate (TPR), also called recall, under several different cut-off points and selected the best cut-off as the threshold when TPR-FPR reached its maximum The details are given in Equations (16) - (18):

Similar to many other works, we employed the area under the receiver operating characteristic curve (AUC), accuracy (ACC), precision, recall, and F1 score as the metrics, calculated as Equations (19) - (22):

where TP, TN, FP and FN represent the number of true positives, true negatives, false positives and false negatives, respectively.

Additional details about the method, such as parameter tuning and training process, are provided in Supplemental information.

In this study, we optimized the WDD model using gradient descent to identify an optimal point that is close to the distribution center of data from T2DM patients and far from the distribution center of data from normal individuals ( Figure 1C ). The D i s t function is engineered to directly reflect the T2DM risk score, enabling the model to predict an input sample as T2DM if its risk score exceeds the risk threshold which is learned by the model. Through the analysis of learnable parameters within the D i s t function, we have identified key features. Utilizing this characteristic, we have been able to uncover significant features across different age and gender groups, enhancing our understanding of T2DM risk factors.

To the above four datasets, we applied three weighted diversity density (WDD)-based algorithms to construct diagnostic prediction models. WDD-KNN refers to an algorithm using k-nearest neighbor (KNN) for imputing missing values. The two MVT-WDD algorithms denote missing value tolerant (MVT) algorithms with penalty terms, where ‘DI’ and ‘BF’ represent two different methods of penalization. A total of 12 models were obtained. The models were evaluated by 10-fold cross-validation and independent test with 1000 repetitions ( Figure 2 , cross-validation set: independent test set = 8:2, the independent test dataset was consistent for each repetition, details in Supplementary Note 1 ). The results are shown in Table 2 , the best AUC achieved 0.9185 ( ± 0.0035) on the whole PEI dataset, which proves the accuracy of the model.

From the Table 2 , we could see that the three algorithms had their own advantages on different sub-datasets. However, it was notable that the WDD-KNN algorithm had a KNN imputation step that the other 2 missing value adaptation algorithms did not have. Generally, imputation is limited by the template dataset. Large template datasets are often owned by a few large institutions and are difficult to share for reasons such as ethical review. To their advantage, the 2 missing value adaptation algorithms can skip this step when preprocessing dataset, and the built model does not need a template dataset for prediction, which will be beneficial in practical situations.

After ensuring the reliability of our modeling results through model scores, we delved deeper into the model’s internal attributes and parameters in the following sections. This deeper analysis allowed us to extract valuable information pertinent to T2DM prediction, further validating the utility and interpretability of our algorithms.

The first aspect of our model’s transparency is reflected in how the distance function illustrates the model and feature contributions to predict T2DM. In physical examinations, clinicians often use blood glucose, sometimes with urine glucose as a reference, to initially assess whether a person may have T2DM. For WDD, every sample was given a risk score (Dist for each algorithm, see Method details) by the models and classified according to a risk threshold. We compared the risk stratification through blood glucose and the risk scores (Dist) from our models ( Figures 3A–C ).

With the PEI dataset ( Figures 3A–F ), we saw that in the models WDD-KNN, MVT-WDD-DI, and MVT-WDD-BF gave scores of confirmed T2DM patients that clustered in the ranges of 0.5-0.75, 0.4-0.82, and 0.4-3, respectively, while the physical examination population was clustered in the score ranges of 0.3-0.53, 0.1-0.5, and 0-1, respectively. All three models performed well in distinguishing the two populations, with WDD-KNN working best. However, it was difficult to completely distinguish the two groups if they were separated only by the level of blood glucose ( Figures 3A–C ), and many people with T2DM still had the same blood glucose levels as normal people. Also, we analysed the risk scores on the three sub-datasets in Supplementary Note 6 .

To provide more information on the importance of the EHR features to every patient, we also calculated the Dist k score (see Method details) for each feature between the patients and normal people ( Figures 3G–I ; Supplementary Figure S3 ). As we can see, the selected important features mostly had different scores by each model. For example, in the model built on the PEI dataset using the MVT-WDD-DI algorithm, the Dist k scores of selected important features ( Figures 3H , 4 ) such as Bact, BLD, Baso, Baso-R, and MCV were much higher than those of other features. T2DM patients and normal people could be well distinguished by the Dist k score of these important features. The result indicates that our model assigns higher significance to features with more pronounced differences, identifying them as important and thus selecting them as effective indicators for T2DM screening.

The results show that our model effectively differentiated T2DM patients from healthy individuals in the PEI dataset, as shown in the scatter density heat map. Visualized model scores underscored performance differences, revealing potential for early T2DM detection. Notably, normal blood glucose levels don’t rule out T2DM, highlighting our model’s diagnostic value. Additionally, by analyzing the model’s distance function, we gained a deeper understanding of the mechanism behind the model’s selection of important features, enhancing our comprehension of the model.

After understanding the scoring mechanism of the model and the mechanism for selecting important features, in this section and Supplementary Notes 3 and 7 , we identified crucial diagnostic indicators for T2DM and analyzed the significance of the selected indicators for T2DM combining clinical knowledge.

The feature’s significance is determined by its weight within the models, identifying the indicators most associated with T2DM. Important features were defined as those ranking in the top 25% by weight across the 12 models. We visually represented this distribution of relative feature weights with a histogram and the specific details of these crucial features are detailed in Figure 4 . We employed the Mann-Whitney U test to evaluate the level of feature differences between the T2DM and normal groups, finding that the selected important features exhibited significant differences (P value< 0.0001) ( Supplementary Table S6 ). In addition, we compared the important features selected by using the internal weights of WDD with least absolute shrinkage and selection operator (LASSO) regression and SHAP framework. The important features showed certain consistency ( Supplementary Note 9 , Supplementary Information Figures S14, S15 ).

When the three algorithms were applied each dataset, the selected important features intersected. For example, on the PEI dataset, the indicators judged as important features by all three models were BASO, BASO-R, and HCT, and the features given high weights by two of the three models were HDL-C, MUCUS, BACT, BLD, LYM-R, MCH, and MCV. In this dataset, the AUCs of all three models were higher than 0.88, so these indicators were selected as having great significance for T2DM prediction ( Figure 4 ). On the BCA dataset, the important features selected by the three algorithms are the same, which further proves the potential diagnostic value of these indicators. Moreover, we analysed the same and different important features extracted by the three algorithms, details are shown in Supplementary Note 7 .

We conducted a literature review to integrate our clinical expertise with published research findings and investigate the clinical correlations between these important features and T2DM. Our analysis revealed that most of these important features are shown to have direct or indirect associations with T2DM. For example, urinary tract infections are known to be correlated with diabetes (28) and some indicators associated with urinary tract infections, such as haematuria and bacteria in urine, have been selected as important biomarkers. The details are collated in the Supplementary Note 3 .

Leveraging our model’s transparency and feature extraction capabilities, we conducted group modeling for populations with varying demographic characteristics to unearth the diagnostic value of indicators across different groups. To mitigate the potential model bias introduced by data imbalance, we ensured basic balance in the sample volume of each age and sex category for both T2DM patients and normal individuals, as illustrated in Supplementary Figure S5 and Supplementary Table S2 .

In most cases, age and sex will be correlated with T2DM incidence, which indicates that T2DM in different age and sex groups might have different characteristics. To further explore the importance of each feature for T2DM prediction in different age and sex groups, we tried three additional ways to divide the PEI Dataset: i. by age; ii. by sex; and iii. by age and sex. The number and proportion of people in different age and sex groups are shown in Supplementary Figure S6 and Supplementary Table S2 . After the division of the datasets, 26 sub-datasets (8 ages + 2 sexes + 8 ages × 2 sexes) were generated, and 78 additional models (26 datasets × 3 models) were built. The performance of each model is shown in Figures 5A, B . The WDD-KNN and MVT-WDD-DI algorithms performed well on each group of datasets, with AUC values mostly above 0.9, while the performance of MVT-WDD-BF was not as good. Therefore, in the subsequent analysis, only the weights of the first two algorithms were taken into consideration.

The results showed that the importance of the clinical indicators varied in different age and sex groups ( Figure 5C ; Supplementary Figure S6 ). We not only integrated the results of these models using the WDD-KNN and MVT-WDD-DI algorithms but also analysed the distribution of their measured values ( Supplementary Figures S7-13 ) to explain the various importance of these indicators for T2DM diagnosis in the different groups.

The significance of glomerular filtration rate (GFR) in T2DM diagnosis diminishes with age, showing greater importance in the 5-49 age group ( Figures 5C , 6B ). Elevated GFR in T2DM patients aged 5-49 distinguishes them from normal groups, particularly in the 5-39 ( Figure 6 ). This aligns with studies linking diabetes and GFR, where early diabetic kidney disease (DKD) phases show increased GFR due to various changes in ultrastructural, vascular, and tubular factors (29). As renal health declines, GFR decreases (29, 30). Our findings suggest that GFR’s diagnostic value for T2DM varies across ages, particularly useful for early screening in younger populations, extending beyond its role in DKD.

Triglycerides (TG) showed greater significance in the 5-39 age group compared to others ( Figure 5C ; Supplementary Figure S6 ), with notable differences in TG distribution between T2DM patients and normal individuals in this age range ( Supplementary Figure S7 ). This variation is attributed to age-related dietary and metabolic differences and a genetic link identified by Saxena, R. et al. (31) High TG levels in T2DM patients are associated with increased cardiovascular risks (32) and metabolic changes (33). Our model emphasizes TG’s importance in T2DM, especially in younger age groups, aligning with current research trends.

Haemoglobin (HGB) was more important in the 55- to 95-year-old group ( Supplementary Figures S5C, S6A ). In T2DM patients, as age increased, their lower HGB compared to that in normal people became more pronounced ( Supplementary Figure S12 ). Based on our knowledge and experience, anaemia is diagnosed by HGB decline, so the association between anaemia and T2DM might be the use of metformin. There are reports supporting that long-term metformin use in T2DM patients can cause anaemia (34, 35), and our EHR included patients who used metformin since metformin has been a commonly prescribed drug for T2DM patients for decades. Similarly, in diabetic patients with chronic kidney disease (CKD), some factors cause iron-deficiency anaemia, such as low intestinal absorption and gastrointestinal bleeding (36). In addition, erythropoietin deficiency and hyporesponsiveness can lead to anaemia in diabetic patients with CKD (36–38). Nephrotic syndrome, characterized by oedema, hypoalbuminaemia, dyslipidaemia, and increased transferrin catabolism, contributes to anaemia due to iron and erythropoietin deficiency (36, 39, 40). Long-term administration of angiotensin-converting enzyme (ACE) inhibitors and angiotensin receptor antagonists in diabetic patients also leads to a reversible decrease in HGB through a direct blockade of the proerythropoietic effects of angiotensin II on red cell precursors, degradation of physiological inhibitors of haematopoiesis, and suppression of IGF-I (36, 41). Thus, based on many studies and reports, taking HGB as an important feature will be a useful indicator for older patients with longer duration of diabetes, so HGB was selected after modelling the EHR data. In other words, HGB decline might be a marker of T2DM or T2DM-correlated disease, but it might be interfered with by some confounding factors, so its use for early diagnosis might be limited. This limitation is caused by the lack of medication information in our EHR data. Despite our meticulous selection of the patient’s first record within the hospital system, we cannot guarantee that they have not undergone therapeutic interventions at other institutions. To address this shortcoming, cohort studies with long-term follow-up are needed.

The Neutrophils (NEU), neutrophil rate (NEU-R), lymphocyte rate (LYM), and lymphocyte rate (LYM-R) have also been observed to correlate with age or sex, as discussed in Supplementary Note 8 .

The result of this section demonstrates that through group modeling and the model’s feature extraction capability, we identified several age and sex-related biomarkers for T2DM prediction. Integrating insights from the results, it’s evident that our model can glean valuable auxiliary diagnostic information from internal parameters like feature weights and distance functions. This effectively showcases the algorithm’s transparency throughout the modeling process, highlighting its capacity to provide interpretable insights crucial for clinical application.