In order to assess the predictive ability of the fingerprint-based machine learning models, multiple data set were analysed. A set of 79 pharmacologically important biological targets were initially used as a means to benchmark performance, proceeding which model performance was assessed on more challenging targets that are described briefly in the following sections.

Molecular fingerprints have a long history of having been used in similarity searching (Muegge and Hu, 2022). Their popularity can be largely attributed to their ability to evaluate vast libraries of compounds using just a fraction of the resources and time (Venkatraman et al., 2022) that would otherwise be used with more compute intensive approaches. The fingerprint representations used in this study can be grouped into:

1. Those based on pre-defined generic substructures/keys (Bender et al., 2009) such as PUBCHEM (NCBI, 2009), Klekota-Roth (Klekota and Roth, 2008) and MACCS (Durant et al., 2002)

Table 2 provides a summary of the fingerprints used for predictive modelling. Machine learning models for a total of 12 different fingerprints adapted from a set of fingerprints studied earlier by Riniker and Landrum (2013) were evaluated. These fingerprints have been widely used as molecular representations with applications in similarity searching and modelling structure-activity relationships (Zagidullin et al., 2021; Muegge and Hu, 2022; Orosz et al., 2022). The fingerprints were generated using available routines in open source cheminformatics software such as RDKit (Landrum, 2022) and the Chemistry Development Kit (Willighagen et al., 2017).

Prior to modelling, a data cleaning step was followed wherein the SMILES were standardized and cleaned using MayaChemTools (Sud, 2016). Subsequently, for each data set, the available data was randomly split into calibration (80%) and test sets (20%). Model training was performed using random forests (Breiman, 2001) (RF) where the number of trees was set to 500. A 5-fold cross-validation on the training set was carried out to tune the parameter “mtry” (number of input features that will be randomly sampled at each split when creating the tree models). Prediction performances were subsequently assessed on the test set. The train/test splitting (80:20 ratio) was repeated 3 times to assess variability of the prediction performance and to rule out any significant impact on performance owing to selection. The RF models were built using the caret (Kuhn, 2022) and ranger (Wright and Ziegler, 2017) packages in R (R Core Team, 2022). The classification models were evaluated using the balanced accuracy score (Kelleher et al., 2015) which accounts for the skewness of the class distributions B A C C = S e n s i t i v i t y + S p e c i f i c i t y 2 (1) Here, the sensitivity ( T P T P + F N ) and specificity ( T N T N + F P ) are defined in terms of the counts of true positive (TP), true negative (TN), false positive (FP) and false negative (FN). For comparison, other metrics such as the area under the curve (AUC) are also reported.

In order to address the issue of applicability domain of the models, outlier detection using isolation forest (Liu et al., 2008) has been employed. Here, a test compound is assessed for its tendency to separate from the majority of samples using an isolation forest constructed from binary trees. Isolation forests make use of decision tree (are an unsupervised version of random forests) and work on the assumption that for non-outlier points, it takes a large number of splits to separate them into individual buckets (i.e., number of partitions that it takes to isolate a point). By contrast, anomalous points are likely to take much shorter paths for isolation. In this study, the isofor package in R was used to identify potential outliers.

The performance of the fingerprint models was first assessed on the 79 targets (data summary in Supplementary Table S1 in SI) earlier studied by Allen et al. (2020). The heatmap of the balanced accuracies in Supplementary Figure S1 in the SI shows that with the exception of some selected targets such as MAPK1, PTPN11 and hERG, the fingerprint models perform quite well with average accuracies (average of the BACC values across all targets) close to 0.90 for most targets (see Supplementary Figure S2 in the SI). The prediction results for the fingerprint models compare favourably with the metrics reported for deep learning neural networks (Allen et al., 2020) and can be attributed to the fact that the data sets are relatively balanced (positive data percentage of ≈50%). The fingerprint models were also evaluated against six types of cardiac toxicity outcomes: arrhythmia, cardiac failure, heart block, hERG toxicity, hypertension, and myocardial infarction (see Supplementary Table S1 in the SI). These data sets were previously studied by Iftkhar et al. (2022) who used a combination of graph-based signatures and fingerprints to identify models capable of identifying molecules likely to be toxic. Figure 1 summarizes the performance of the fingerprint models which as can be seen, achieve relatively better predictive performance in terms of the AUC.

As further validation of the fingerprint models, predictive performance on a series of structurally diverse datasets consisting of 33,757 active and 21,152 inactive compounds for different breast cancer cell lines was also evaluated. The data sets were earlier studied by He et al. (2021), where a number of descriptor-based machine learning models such as naïve Bayes (NB), support vector machine (SVM), k-nearest Neighbors (KNN), extreme gradient boosting (XGB) as well as deep learning methods were tested. Comparison of the metrics obtained for fingerprint models with those reported by He et al. (2021) shows that the former achieve higher predictive accuracies with BACC > 0.70 (see Figure 2).

Overall, the performance on multiple data sets clearly shows that fingerprints have good predictive power. The majority of the data however, has minimal skew, i.e., near equal numbers of actives and inactives with some even displaying greater bias towards active compounds. Most machine learning approaches are likely to yield strong performances for such balanced data distributions. Data sets drawn from PubChem have typically strongly imbalance and the question is whether fingerprints can yield robust structure–activity relationship models for such data.

Encouraged by the performance of the fingerprints on the different targets, model performance was further assessed on 24 different bioactivity data sets. Table 3 lists the balanced accuracies for the calibration/test sets (average of 3 independent trials) obtained for the targets. Although the performance varies, it is generally seen that the fingerprint models yield reasonable results even for cases with severe imbalance. The heatmap in Figure 3 shows that in a number of cases such as potassium channel inhibitors, KDM4E, LMNA and TARDBP, the selected fingerprints show only a marginal difference in performance with balanced accuracies ≈0.55. Among the fingerprints evaluated in this study, best results were frequently seen to perform well include AVALON, ECFP2/FCFP4/FCFP6 and RDK5.

The fingerprint performance was compared with that of a graph isomorphism network (Xu et al., 2019; Wu et al., 2021b) (GIN) which is a powerful graph neural network (GNN) for graph classification (Kim and Ye, 2020). Using the torchdrug (Zhu et al., 2022) machine learning framework, the GIN was built with 4 hidden layers (number of hidden units set to 256), using an Adam optimizer and binary cross entropy loss function with batch normalization applied to every hidden layer. The model was subsequently trained for 100 epochs with the splits for train/valid/test sets set to 60%, 20% and 20% respectively. The barplots in Figure 4 show the comparison of the test set AUCs (mean of 3 independent runs) achieved by the RF and GNN models. As can be seen from the plots, for the majority of the data sets, RF models achieve relatively better metrics while for others the performances are comparable.

For all data sets, isolation forest (built using 500 trees) based outlier scores were calculated. Here, values closer to 1 indicate potential outliers while those around 0.50 typically suggest average outlierness. Values closer to 0 are more difficult to categorize. Supplementary Figure S5 in the SI shows the histograms of the distributions of the calculated scores. Examination of the plots show that for most of the data sets studied here, a cutoff of 0.5 (for some a lower value is recommended) may be used as a decision threshold to identify potential outliers (see Supplementary Figure S6 in the SI). Compared with other distance based approaches [such as the local outlier factor (Breunig et al., 2000) and one-class support vector machines (Chen et al., 2013)] where the algorithms typically try to fit the regions where the training data is the heavily concentrated, isolation forests do not use any distance metrics and instead rely on the concept that an ensemble of random trees are likely to produce shorter path lengths for outliers.

The model performance although encouraging for some does need significant improvement especially for data sets where the availability of actives is quite low. While a case for balanced data sets can be made, the skewed ratio between active and inactive compounds is a realistic representation of the high-throughput screening hit rates that are typically < 1% (Dreiman et al., 2021). For some data sets, improved performances were seen with substructure fingerprints such as AVALON that are based on pre-defined generic substructure patterns. For others, fingerprints such as ECFP/FCFP that take into account the neighborhood of each atom yielded slightly better classification models. Nonetheless, for many of the data sets (see Figure 3), the model metrics showed only marginal differences. In an earlier study, Riniker and Landrum (2013) observed strong correlations between the fingerprints which may explain the similarities in the obtained metrics. Overall, the choice of which fingerprint to use for modelling is far from trivial and is to a large extent dependent on the target. In this study, Avalon and FCFP4 fingerprints are generally seen standout as useful descriptors and may serve as useful starting points for future benchmarking studies. A potential avenue for improvement in prediction performance could be to combine 2D fingerprints with structure-based graph representations (Choo et al., 2023). Alternatively, one may look towards language representations which have recently been shown to yield good results on several classification and regression benchmarks (Ross et al., 2022).

Given the relative success of the fingerprint-based RF classification models, an immediate question is whether the performances can be replicated for regression tasks. To this end, RF regression models were computed for a number of previously analysed data sets that used graph based signatures and other auxiliary attributes to identify potential candidates against mycobacterium tuberculosis (Pires and Ascher, 2020), cancer (Al-Jarf et al., 2021), and G protein-coupled receptors (Velloso et al., 2021) (GPCRs). A total of 1904 fingerprint-RF models were computed, spanning 36 different GPCRs, 8 organism-specific Mycobacteria species (M. avium, M. caseum, M. kansasii, M. phlei, M. tuberculosis, M. bovis, M. fortuitum, M. smegmatis and M. intracellulare) and 74 distinct cancer cell lines corresponding to 9 tumor types (renal, breast, CNS, colon, leukemia melanoma, non small cell lung, ovarian, prostate, and small cell lung). Supplementary Figures S7–S9 in the SI summarize the regression performances of the different fingerprints. When compared with the graph signature based approaches, although marginal improvements were seen for some cases, the overall performance measured in terms of the squared Pearson correlation (R ²) was largely found to be comparable, with only models for tuberculosis yielding slightly lower R ² values (see Figure 5). The fingerprint performance observed for these data sets mirrors the trends seen for a number of ADMET-related responses that were studied in a previous article [see (Venkatraman, 2021)] and suggest that purely fingerprint-based models may have low predictive utility for regression.