A schematic diagram of the overall procedure for predicting shear transformed atoms is shown in Figure 1. First, a C u 64 Z r 36   metallic glass specimen composed of 10,000 atoms was prepared using the EAM potential (Sheng et al., 2011) for the interatomic interaction via molecular dynamics simulation package LAMMPS (Plimpton, 1995). A periodic boundary condition was applied to all three directions of the supercell. During the equilibration period of the liquid-state sample for 10 ns through a NPT ensemble at 2,000 K and 0 atm, we randomly choose 75 out of 1,000 microstates (snapshots taken every 10 ps). These were quenched to 300 K at a rate of 10 10 K / s to produce 75 different solid specimens. Given that shear transformation occurs at very distinct locations depending on the applied shear direction, an athermal quasi-static shear simulation was performed on 75 specimens in the six directions of xy, yz, xz, -xy, -yz, and -xz, until the shear strain reaches 4 × 10 − 2 with a shear strain increment of 10 − 4   . The non-affine displacement of atom i is defined as D min , i 2 = ∑ j ∈ N i | d j i ( t − Δ t ) J i ∗ − d j i ( t ) | 2 where N i is the number of atoms neighboring atom i, d j i is a displacement vector between atom j and atom i, and J i ∗ is the affine transformation tensor that minimizes ∑ j ∈ N i | d j i ( t − Δ t ) J i − d j i ( t ) | 2 (Falk and Langer, 1998). As the applied shear strain was increased, the non-affine displacement ( D min 2 ) values of all atoms were calculated for every shear step. When a group of atoms with D min 2 greater than D t h r e s 2 initially occurred, those atoms were labeled as the initially shear transformed atoms (label 1), with the others labeled as normal atoms (label 0).

Next, for the initial state before applying shear deformation, the atomic properties of each atom were calculated and assigned as a feature vector, f i = [ f i 1 ,   f i 2 ,   … ,   f i 19 ]   ( f i j   : j t h   p r o p e r t i e s   o f   a t o m   i )   . First, with one-hot encoding, the type of atom was assigned as a feature: [1,0] ( f 1 = 1 ,   f 2 = 0 )   for Cu and [0,1] ( f 1 = 0 ,   f 2 = 1 ) for Zr. Also, because the Cu-centered icosahedron is known to be a locally stable structure, the feature is set to [1 0] ( f 3 = 1 ,   f 4 = 0 ) for the atoms whose Voronoi index is < 0,0,12,0>, and [0 1] ( f 3 = 0 ,   f 4 = 1 ) otherwise (Lee et al., 2011; Ding, 2014). In addition, certain scalar atomic properties, in this case the Voronoi volume, potential energy, and participation fraction for vibrational normal mode, were calculated and assigned as features ( f 5 ,   f 6 ,   f 7 ) , and the tensor composites of the atomic-level stiffness tensor ( f 8 ∼ f 13 : C 14 ,   C 24 , C 34 , C 44 , C 54 , C 64 ) and stress tensor (   f 14 ∼ f 19 : σ 11 ,   σ 22 ,   σ 33 ,   σ 12 ,   σ 23 ,   σ 31 ) were assigned as features after properly accounting for relative orientations compared to the applied shear. Then, we made a graph structure in which each atom is a node and the nearest neighbor atoms obtained from Voronoi tessellation are connected by edges, with the f i feature vector then assigned to each node. Finally, 450 graphs, each of which consisting of 10,000 nodes with 19-dim node features, were generated by considering 75 specimens and six different shear directions. These graphs were divided into training, validation, and test sets at a ratio of 3:1:1. Specifically, although it is true that STZ activation depends on the applied shear direction, there may be some correlation between STZs activated from different shear directions. Therefore, 75 specimens were divided into 3:1:1 and the data for six shear directions of a specimen were made to belong to the same set.

As a machine learning model for predicting the initial STZ activation, a graph attention network (GAT) was selected to account for the features of surrounding atoms efficiently. In each layer of GNN, the feature vectors of neighbor nodes are aggregated and multiplied by a trainable weight matrix. Therefore, as the number of layers increases, the features of farther atoms can be aggregated. Through this, medium-range features, which are known to be important in predicting STZ, can be applied. Also, Unlike a graph convolutional network (GCN), which aggregates feature vectors of neighboring nodes equally, the GAT shows better performance because it performs a weighted sum using trainable attention vectors (Veličković et al., 2017). In terms of the architecture, like most graph neural network structures, fully connected layers were combined after several GAT layers, and Mish was used as an activation function (Misra, 2019). In addition, because the performance of a neural network changes greatly according to the structure of the network or the hyperparameters, the number of GAT layers (L: [2,4,6,8,10,12]), the output dimension of each layer (M: [10, 15, 20,25, 30, 35,40]), and the number of attention vectors (N: [6, 9, 12, 15]) are optimized by means of grid search and four-fold cross-validation. In addition, in order to prevent the model from being biased toward the major class during training due to the class imbalance, the weighted cross-entropy is used as a loss function.

Finally, unlike previous studies, the predictive power of the trained machine learning model is measured with the six evaluation metrics of recall, precision, the F1 score, the Matthews correlation coefficient (MCC), AUC-ROC and the average precision (AP). The test dataset was evaluated by the trained model and the results were classified as true positives (TP, true: label 1 and predict: label 1), false positives (FP, true: label 0 and predict: label 1), false negatives (FN, true: label 1 and predict: label 0), and true negatives (TN, true: label 0, predict: label 0). With these values, recall ( = T P T P + F N ) , precision ( = T P T P + F P ) , the F1 score ( = 2 T P 2 T P + F P + F N ) , and MCC ( = T P × F N − F P × T N ( T P + F N ) ( T P + F P ) ( T N + F P ) ( T N + F N ) ) are calculated. The receiver operating characteristic (ROC) curve is drawn and the AUC value is obtained as the area below the curve. The AP is calculated by averaging the precision of the precision-recall (PR) curve.

The performance of the model was evaluated with the six metrics of recall, precision, the f1-score, the Matthews correlation coefficient (MCC), AUC-ROC, and the average precision of the precision-recall curve (AP). The reference value, D t h r e s 2 , the threshold value with which to classify the atoms as initially shear transformed atoms (label 1) or normal atoms (label 0), was set to 0.1Ų. Table 1 shows the hyperparameter sets of the GAT models optimized for the six different metrics and their performance outcomes. Depending on the model, the evaluated metrics vary by approximately 10–20%, but no dramatic changes were observed after hyperparameter optimization. For the model optimized for AUC (L = 4, M = 15, N = 9), the AUC of the test dataset was found to be 0.8508 (Figure 2A), which is higher than that in a previous study (Wang and Jain, 2019; Wang et al., 2020). However, the other metrics were nearly equal to zero, except for recall (Recall = 0.7002, Precision = 0.0086, f1-score = 0.0169, MCC = 0.0662, and AP = 0.0160), meaning that the predictive power of the trained model is extremely low.

To better understand the prediction results, we plotted the ROC curve of the test dataset and visualized the true and predicted STZ atoms in a supercell, as shown in Figure 2. Figure 2B visualizes all atoms predicted as STZ atoms, while the true STZ atoms among them are shown in red. In Figure 2C, all true STZ atoms are shown, while the correctly predicted STZ atoms among them are shown in red. Our results indicate that a large proportion of true STZ atoms are correctly predicted as STZs, whereas only a small portion of the predicted STZ atoms are true STZ atoms. More quantitatively, for the classification task of our test dataset atoms, true positives (TP) numbered 1,134, false positives (FP) amounted to 126,180, false negatives (FN) numbered 486, and true negatives (TN) amounted to 772,200. Because the number of false positives (FP), referring to the number of erroneous predictions of normal atoms as STZ atoms, was very high, the precision score, which is the ratio of the true STZ atoms among the predicted STZ atoms, was significantly low. Therefore, even if the AUC is relatively high at 0.85, it may have been overestimated due to the class imbalance, and various metrics such as precision must be considered together to assess the predictive power of a ML model fairly.

As shown in Figure 3A, the prediction results were also evaluated with a probability plot while varying the non-affine displacement value. In many existing studies, the validity of machine learning models was evaluated with this plot, although only the recall of atoms with a specific non-affine displacement value can be assessed. As the non-affine displacement increases, the accuracy of the trained model (evaluated only with the recall metric) converges to a value over 70%, but approximately 20% of the atoms with low non-affine displacement are incorrectly predicted as STZs. However, as shown in Figure 3B, with an increase in the non-affine displacement value, the number of atoms undergoing that amount of non-affine displacement exponentially decreases. Hence, the probability plot indicates that very few of predicted STZ atoms are true STZ.

However, the fact that the model more correctly classifies atoms with large non-affine displacement has certain implications. Figure 3C shows a true STZ cluster and its cross-section view, and the non-affine displacement value of each atom is expressed in color. It can be seen that in a STZ cluster, atoms located at the center tend to have a larger non-affine displacement value. Also, Figure 3D shows atoms classified as STZs by our ML model for the same specimen shown in Figure 3C; the figure on the right is a cross-section at the same height (i.e., an identical z value) as that shown in Figure 3C. In Figure 3D, the confidence (probability that the model classifies the atom as STZ) is shown in color. We note that the probability in Figure 3D differs from that in Figure 3A. Probability (ratio) in Figure 3A refers to the proportion of atoms classified as STZ atoms by the model among atoms with a specific non-affine displacement value. On the other hand, the confidence in Figure 3C is the ‘probability that an atom is STZ’ obtained by the softmax function in the last layer of the ML model, and if the confidence is 0.5 or more, the model determines that the atom is STZ.

As mentioned above, the model misclassifies too many normal atoms as STZ atoms. However, in clusters predicted as STZ, it can be seen that the atom located at the center has a higher probability value, which means that the model classified central atoms with greater confidence. Not only for true STZ atoms but also the incorrectly classified STZ clusters, the confidence level is higher for atoms located closer to the center of each cluster. Hence, we can conclude that the ML model learned some characteristics of the STZ cluster, i.e., that the non-affine displacement is larger towards the center. It is plausible that incorrectly classified STZ clusters may undergo shear transformation under greater shear strain. However, even if this is the case, because we only learn and evaluate the ML model with the initially activated STZs, those must be counted as false prediction cases. Additionally, most atomic features used in the present study have scalar values, and it can be inherently difficult to predict STZs that occur in different patterns under different shear strain directions for an identical specimen.

Additionally, we trained the GAT model while varying the threshold value of the non-affine displacement and evaluated its predictive power with six different metrics. The non-affine displacement of an atom is a value indicating how inhomogeneously the atom is displaced relative to its neighbor, and its distribution is a monotonically decaying function with a long tail. Because there is no abrupt change in the histogram, existing studies chose a certain threshold value above which the recall of the ML model converges (Cubuk et al., 2015) or an arbitrary threshold value (Wang and Jain, 2019). However, as discussed earlier, the performance of the classification model for class imbalance problem cannot be assessed by a single metric; therefore we ran the evaluation with all six metrics, as depicted in Figure 4.

Figure 4A shows the class ratio, the ratio of STZ atoms to normal atoms as a function of the threshold value. Because the non-affine displacement value increases as an atom is located closer to the core of a STZ cluster, the class ratio decreases quickly with an increase of the threshold. Hence, all metrics change dramatically with a change of the threshold value (Figures 4B–G). As the threshold increases, recall and AUC increase, whereas precision, F1, and AP decrease. We note that although AUC is often used as a metric in many existing studies (Wang and Jain, 2019; Wang et al., 2020), it is questionable if AUC can serve as a reasonable metric, as it changes from 0.7 to 0.85 (note that this is a dramatic change, considering its bound of [0.5, 1]) with an adjustment of the threshold. Interestingly, MCC shows a maximum at an intermediate value of the threshold, which appears properly to reflect the increase in the recall and the decrease in the precision with a change of the threshold. Unlike AUC, which overestimates the model’s performance with a decrease of the class ratio, MCC reflects the limited predictive power of our GAT model and can be considered as a more comprehensive and proper metric. Our results imply that the performance of a ML model must be analyzed with various metrics to account for the class imbalance.