The requirement of the black-box model explanation led to development of many explanation methods. A large part of methods follows from the original LIME method (Ribeiro et al., 2016). These methods include ALIME (Shankaranarayana and Runje, 2019), Anchor LIME (Ribeiro et al., 2018), LIME-Aleph (Rabold et al., 2020), SurvLIME (Kovalev et al., 2020), LIME for tabular data (Garreau and von Luxburg, 2020a,b), GraphLIME (Huang et al., 2022), etc.

To generalize the simple linear explanation surrogate model, several neural network models, including NAM (Agarwal et al., 2021), GAMI-Net (Yang et al., 2021), and AxNNs (Chen et al., 2020), were proposed. These models are based on applying the GAM (Hastie and Tibshirani, 1990). Similar explanation models, including Explainable Boosting Machine (Nori et al., 2019) and EGBM (Konstantinov and Utkin, 2021), were developed using the gradient boosting machine.

Another large part of explanation methods is based on the original SHAP method (Strumbelj and Kononenko, 2010) which uses Shapley values (Lundberg and Lee, 2017) as measures of the feature contribution into the black-box model prediction. This part includes FastSHAP (Jethani et al., 2022), Kernel SHAP (Lundberg and Lee, 2017), Neighborhood SHAP (Ghalebikesabi et al., 2021), SHAFF (Benard et al., 2022), TimeSHAP (Bento et al., 2021), X-SHAP (Bouneder et al., 2020), ShapNets (Wang et al., 2021), etc.

Many explanation methods, including LIME and its modifications, are based on perturbation techniques (Fong and Vedaldi, 2019, 2017; Petsiuk et al., 2018; Vu et al., 2019), which stem from the well-known property that contribution of a feature can be determined by measuring how a prediction changes when the feature is altered (Du et al., 2019). The main difficulty of using the perturbation technique is its computational complexity when samples are of the high dimensionality.

Another interesting group of explanation methods, called the example-based explanation methods (Molnar, 2019), is based on selecting influential instances from a training set having the largest impact on the predictions and its comparison with the explainable instance. Several approaches to the example-based method implementation were considered in Adhikari et al. (2019), Cai et al. (2019), Chong et al. (2022), Crabbe et al. (2021), and Teso et al. (2021).

In addition to the aforementioned methods, there are a huge number of other approaches to solving the explanation problem, for example, Integrated Gradients (Sundararajan et al., 2017), and Contrastive Examples (Dhurandhar et al., 2018). Detailed surveys of many methods can be found in Adadi and Berrada (2018), Arrieta et al. (2020), Bodria et al. (2023), Burkart and Huber (2021), Carvalho et al. (2019), Islam et al. (2022), Guidotti et al. (2019), Li et al. (2022), Rudin (2019), and Rudin et al. (2021).

Most papers considering the convex hull methods study the relationship between location of decision boundaries and convex hulls of a training set. The corresponding methods are presented in Chau et al. (2013), El Mrabti et al. (2024), Gu et al. (2020), Nemirko and Dula (2021a), Nemirko and Dula (2021b), Renwang et al. (2022), Rossignol et al. (2024), Singh and Kumar (2021), Wang et al. (2013), Yousefzadeh (2020), and Zhang X. et al. (2021). Boundary of the dataset's convex hull is studied in Balestriero et al. (2021) to discriminate interpolation and extrapolation occurring for a sample. Efficient algorithms for efficient computation of the convex hull for training data are presented in Khosravani et al. (2016).

The concept of duality was also widely used in machine learning models starting from duality in the support vector machine and its various modifications (Bennett and Bredensteiner, 2000; Zhang, 2002). This concept was successfully applied to some types of neural networks (Ergen and Pilanci, 2020, 2021), including GANs (Farnia and Tse, 2018), to models dealing with the high-dimensional data (Yao et al., 2018).

At the same time, the aforementioned approaches did not apply to explanation models. Concepts of the convex hull and the convex duality may be a way to simplify and to improve the explanation models.

According to Rockafellar (1970), a domain produced by a set of instances as vectors in Euclidean space is convex if a straight line segment that joins every pair of instances belonging to the set contains a vector belonging to the domain. A set S is convex if, for every pair, u,v∈S, and all λ ∈ [0, 1], the vector (1 − λ)u + λv belongs to S.

Moreover, if S is a convex set, then for any x₁, x₂, ..., x_t belonging to S and for any non-negative numbers λ₁, ..., λ_t such that λ₁ +... + λ_t = 1, the sum λ₁x₁ + ... + λ_tx_t is called a convex combination of x₁, ..., x_t. The convex hull or convex envelope of set X of instances in the Euclidean space can be defined in terms of convex sets or convex combinations as the minimal convex set containing X, or the intersection of all convex sets containing X, or the set of all convex combinations of instances in X.

Let us briefly introduce the most popular explanation methods.

LIME (Ribeiro et al., 2016) proposes to approximate a black-box explainable model, denoted as f, with a simple function g in the vicinity of the point of interest x, whose prediction by means of f has to be explained, under condition that the approximation function g belongs to a set of explanation models G, for example, linear models. To construct the function g, a new dataset consisting of generated points around x is constructed with predictions computed be means of the black-box model. Weights w_x are assigned to new instances in accordance with their proximity to point x by using a distance metric, for example, the Euclidean distance. The explanation function g is obtained by solving the following optimization problem:

Here, L is a loss function, for example, mean squared error, which measures how the function g is close to function f at point x; Φ(g) is the model complexity. A local linear model is the result of the original LIME such that its coefficients explain the prediction.

Another approach to explaining the black-box model predictions is SHAP (Lundberg and Lee, 2017; Strumbelj and Kononenko, 2010), which is based on a concept of the Shapley values (Shapley, 1953) estimating contributions of features to the prediction. If we explain prediction f(x₀) from the model at a local point x₀, then the i-th feature contribution is defined by the Shapley value as

where f(S) is the black-box model prediction under condition that a subset S of the instance x₀ features is used as the corresponding input; N is the set of all features.

It can be seen from Equation 2 that the Shapley value ϕ_i can be regarded as the average contribution of the i-th feature across all possible permutations of the feature set. The prediction f(x₀) can be represented by using Shapley values as follows (Lundberg and Lee, 2017; Strumbelj and Kononenko, 2010):

To generalize LIME, NAM was proposed in Agarwal et al. (2021). It is based on the generalized additive model of the form y(x) = g₁(x₁) + ... + g_m(x_m) (Hastie and Tibshirani, 1990) and consists of m neural networks such that a single feature is fed to each subnetwork and each network implements function g_i(x_i), where g_i is a univariate shape function with E(g_i) = 0. All networks are trained jointly using backpropagation and can learn arbitrarily complex shape functions (Agarwal et al., 2021). The loss function for training the whole neural network is of the form:

where xk(i) is the k-th feature of the i-th instance; n is the number of training instances.

The representation of results in NAM in the form of shape functions can be considered in two ways. On the one hand, the functions are more informative, and they show how features contribute into a prediction. On the other hand, we often need to have a single value of the feature contribution which can be obtained by computing an importance measure from the obtained shape function.

NAM significantly extends the flexibility of explanation models due to possibility to implement arbitrary functions of features by means of neural networks.

According to Molnar (2019), an instance or a set of instances are selected in example-based explanation methods to explain the model prediction. In contrast to the feature importance explanation (LIME, SHAP), the example-based methods explain a model by selecting instances from the dataset and do not consider features or their importance for explaining. In the context of obtained results, the example-based methods are represented by influential instances (points from the training set that have the largest impact on the predictions) and by prototypes (representative instances from the training data). It should be noted that instances used for explanation may not belong to a dataset and are combinations of instances from the dataset or some points in the dataset domain. The well-known method of K nearest neighbors can be regarded as an example-based explanation method.

Let us consider the method for dual explanation. Suppose that there is a dataset T={(x1,y1),...,(xt,yt)} of t points (x_i, y_i), where xi=(x1(i),...,xm(i))∈X⊂ℝm is a feature vector consisting of m features, y_i is the observed output for the feature vector x_i such that y_i ∈ ℝ in the regression problem and y_i ∈ {1, 2, ..., C} in the classification problem with C classes. It is assumed that output y of an explained black-box model is a function f(x) of an associated input vector x from X.

To explain an instance x0∈X, an interpretable surrogate model g for the black-box model f is trained in a local region around x₀. It is carried out by generating a new dataset S of n perturbed samples in the vicinity of the point of interest x₀ similarly to LIME. Samples are assigned by weights w_x in accordance with their proximity to the point x. By using the black-box model, output values y are obtained as function f of generated instances. As a result, dataset S consists of n pairs (x_i, f(x_i)), i = 1, ..., n. Interpretable surrogate model g is now trained on S. Many explanation methods such as LIME and SHAP are based on applying the linear regression function

as an interpretable model because each coefficient a_i in g quantifies how the i-th feature impacts on the prediction. Here a = (a₁, ..., a_m). It should be noted that the domain of set S coincides with the domain of set T in the case of the global explanation.

Let us consider the convex hull P of a set of K nearest neighbors of instance x₀ in the Euclidean space. The convex hull P forms a convex polytope with d vertices or extreme points xi*, i = 1, ..., d. Then, each point x∈P is a convex combination of d extreme points:

This implies that we can uniformly generate a vector in the unit simplex of possible vectors λ consisting of d coefficients λ₁, ..., λ_d, denoted Δ^d−1. In other words, we can consider points in the unit simplex Δ^d−1 and construct a new dual dataset D={(λ(1),z1),...,(λ(n),zn)}, which consists of vectors λ(j)=(λ1(j),...,λd(j)), and the corresponding values z_j, j = 1, ..., n, computed by using the black-box model f as follows:

i.e., z_j is a prediction of the black-box model when its input is vector ∑i=1dλi(j)xi*.

In sum, we can train the “dual” linear regression model (the surrogate model) for explanation on dataset D, which is of the form:

where b = (b₁,..., b_d) is the vector of coefficients of the “dual” linear regression model.

The surrogate model can be trained by means of LIME or SHAP with the dual dataset D.

Suppose that we have trained the function h(λ) and computed coefficients b₁, ..., b_d. The next question is how to transform these coefficients to coefficients a₁, ..., a_m which characterize the feature contribution into the prediction. In the case of the linear regression, coefficients of function g(x) = a₁x₁ + ... + a_mx_m can be found from the condition:

which has to be satisfied for all generated λ_j. This obvious condition means that predictions of the “primal” surrogate model with coefficients a₁, ..., a_m has to coincide with predictions of the “dual” model.

Introduce a matrix consisting of extreme points

Note that, λ_i = 1 and λ_j = 0, j ≠ i, for the i-th extreme point. This implies that the condition (Equation 9) can be rewritten as

By using Equation 5, we get

Hence, there holds

It follows from the above that

where X⁻¹ is the pseudoinverse matrix.

Generally, the vector a can be computed by solving the following unconstrained optimization problem:

In the original LIME, perturbed instances are generated around x₀. One of the important advantages of the proposed dual approach is the opportunity to avoid generating instances in accordance with a probability distribution with parameters and to generate only uniformly distributed points λ^(j) in the unit simplex Δ^d−1. Indeed, if we have image data, then it is difficult to perturb pixels or superpixels of images. Moreover, it is difficult to determine parameters of the generation to cover instances from different classes. According to the dual representation, after generating vectors λ^(j), new vectors x_j are computed by using Equation 6. This is similar to the mixup method (Zhang et al., 2018) to some extent that generates new samples by linear interpolation of multiple samples and their labels. However, in contrast to the mixup method, the prediction is obtained as the output of the black-box model (see Equation 7), but not as the convex combination of one-hot label encodings. Another important advantage is that instances corresponding to the generated set D are totally included in the domain of the dataset T. This implies that we do not get anomalous predictions f(x_i) when generated x_i is far from the domain of the dataset T.

Another question is how to choose the convex hull of the predefined size and, hence, how to determine extreme points xi* of the corresponding convex polytope. The problem is that the convex hull has to include some number of points from dataset T and the explained point x₀. Let us consider K nearest neighbors around x₀ from T, where K is a tuning parameter satisfying condition K≥d. The convex hull is constructed on these K + 1 points (K points from T and one point x₀). Then, there are d points among K nearest neighbors which define a convex polytope and can be regarded as its extreme points. It should be noted that d depends on the dataset analyzed. Figure 2 illustrates two cases of the explained point location and the convex polytopes constructed from K = 7 nearest neighbors. The dataset consists of 10 points depicted by circles. A new explained point x₀ is depicted by the red triangle. In Case 1, point x₀ lies in the largest convex polytope with d = 5 extreme points x1*,...,x5* constructed from seven nearest neighbors. The largest polytope is taken in order to envelop as large as possible points from the dataset. In Case 2, point x₀ lies outside the convex polytope constructed from nearest neighbors. Therefore, this point is included into the set of extreme points and d ≤ K+1. As a result, we have d = 6 extreme points x1*,...,x5*,x6*=x0.

To identify whether the newly added point can be expressed as convex combination of the existing vectors, the Farka's lemma (Dinh and Jeyakumar, 2014) can be applied.

Points λ^(j) from the unit simplex Δ^d−1 are randomly selected in accordance with the uniform distribution over the simplex. This procedure can be carried out by means of generating random numbers in accordance with the Dirichlet distribution (Rubinstein and Kroese, 2008). There are also different approaches to generate points from the unit simplex (Smith and Tromble, 2004).

Finally, we write Algorithm 1 implementing the proposed method.

Figure 3 illustrates steps of the algorithm for explanation of a prediction provided by a black-box model at the point depicted by the small triangle. Points of the dataset are depicted by small circles. The training dataset T and the explained point are shown in Figure 3A. Figure 3B shows set TK of K = 13 nearest points such that only two points (0.05, 0.5) and (1.0, 0.1) from training set T do not belong to set TK. The convex hull and the corresponding extreme points are shown in Figure 3C. Points uniformly generated in the unit simplex are depicted by means of small crosses in Figure 3D. It is interesting to point out that the generated points are uniformly distributed in the unit simplex, but not in the convex polytope as it is follows from Figure 3D. We uniformly generate vectors λ, but the corresponding vectors x are not uniformly distributed in the polytope. One can see from Figure 3D that generated points in the initial (primal) feature space tend to be located in the area where the density of extreme points is largest. This is a very interesting property of the dual representation. It means that the method takes into account the concentration of training points and the probability distribution of the instances in the dataset.

The difference between points generated by means of the original LIME and the proposed method is illustrated in Figure 4 where the left picture (Figure 4A) shows a fragment of Figure 1 and the right picture (Figure 4B) illustrates how the proposed method generates instances.

The proposed method requires finding all the extreme points (vertices) of the convex hull of a given point x0∈ℝd and its nearest neighbors x₁, x₂, …, x_n−1. When the dimension d is small, these extreme points can be computed in time O(2^O(dlogd)n²) = O(n²) (Ottmann et al., 2001). In general, determining whether x_i is an extreme point can be done by checking the condition

where conv(P) denotes the convex hull of the set P.

The above condition is equivalent to solving a feasibility problem that can be formulated as a linear program. This linear program involves n variables and n+d constraints and can be solved using the interior-point method described in Vaidya (1989). For each point, the time complexity of this procedure is O((n+d)^3/2nlog(n)), resulting in an overall complexity of

In the extreme case, when d≫1, we can use the AVTA algorithm {xj}j=0n−1.. This algorithm has the time complexityO(n²(t⁻)), wheret ∈ (0, 1). The approximation becomes more precise as t → 0.

The dual approach can work best when applied to analysis of potential outliers. In that regard, the generation procedure proposed in the study is more robust than the one used in LIME. By choosing the generation region as the convex hull of the explained point nearest neighbors, we reduce the likelihood of creating additional samples that fail to align with the original data distribution. As for hyperparameters, the number of nearest neighbors used to construct the convex hull for the explained point largely depends on the user's preferences and the nature of analyzed data. We can stop incorporating additional neighbors when a certain threshold is reached, such as when the next nearest neighbor is considerably more distant compared to the previous ones. Furthermore, we can choose to exclude a new neighbor if its data features clearly indicate that it would not contribute much to the analysis of the explained point. The number of points to generate can be taken as k·n, where k is a real number and n is the number of selected neighbors. By default, k = 3. This implies that we can increment the number of generated points until we observe the convergence of dual coefficients. We can also modify the distribution type employed for creating the dual dataset. For instance, if we take new points to be generated mostly in close proximity to the explained point x = (x₁, ..., x_d), we can sample the points from the Dirichlet distribution with concentration parameters α_i = 1 + t · x_i, where t > 0.

It turns out that the proposed method for the dual explanation inherently leads to the example-based explanation. An example-based explainer justifies the prediction on the explainable instance by returning instances related to it. Let us consider the dual representation (Equation 8). If we normalize coefficients b = (b₁, ..., b_d) as

then new coefficients (v₁, ..., v_d) quantify how extreme points (x1*,...,xd*) associated with (λ₁, ..., λ_d) impact on the prediction. The greater the value of v_i, the greater contribution of xi* into a prediction. Hence, the linear combination of extreme points

allows us to get an instance x explaining x₀.

An outstanding approach considering convex combinations of instances from a dataset as the example-based explanation was proposed in Crabbe et al. (2021). In fact, we came to the similar example-based explanation by using the dual representation and constructing linear regression surrogate model for new variables (λ₁, ..., λ_d).

The example-based explanation may be very useful when we apply NAM (Agarwal et al., 2021) for explaining the black-box prediction. By using dual dataset D={(λ(1),z1),...,(λ(n),zn)}, we train NAM consisting of d subnetworks such that each subnetwork implements the shape function h_i(λ_i). Figure 5 illustrates a scheme of training NAM. Each generated vector λ is fed to NAM such that each its variable λ_i is fed to a separate neural subnetwork. For the same vector λ, the corresponding instance x is computed by using Equation 6, and it is fed to the black-box model. The loss function for training the whole neural network is defined as the difference between the output z of the black-box model and the sum of shape functions h₁, ..., h_d implemented by neural subnetworks for the corresponding vector λ, i.e., the loss function L is of the form:

where λk(i) is the k-th element of vector λ⁽ⁱ⁾; R is a regularization term with the hyperparameter α which controls the strength of the regularization; w is the vector of the neural network training parameters.

The main difficulty of using the NAM results, i.e., shape functions h_k(λ_k), is how to interpret the shape functions for explanation. However, in the context of the example-based explanation, this difficulty can be simply resolved. First, we study how a shape function can be represented by a single value characterizing the importance of each variable λ_k, k = 1, ..., d. The shape function is similar to the partial dependence plot (Friedman, 2001; Molnar, 2019) to some extent. The importance of a variable (λ_k) can be evaluated by studying how rapidly the shape function, corresponding to the variable, is changed. The rapid change of the shape function says that small changes of the variable significantly change the target values (z). The above implies that we can use the importance measure proposed in Greenwell et al. (2018), which is defined as the deviation of each unique variable value from the average curve. In terms of the dual variables, it can be written as:

where r is a number of values of each variable λ_k, which are analyzed to study the corresponding shape function.

Normalized values of the importance measures can be regarded as coefficients v_i, i = 1, ..., d, in Equation 19, i.e., they show how important each extreme point or how each extreme point can be regarded as an instance which explains instance x₀.

An additional important advantage of the dual representation is that shape functions for all variables λ_k, k = 1, ..., d, have the same scale because all variables are in the interval from 0 to 1. This allows us to compare the importance measures I(λ_k) without the preliminary scaling which can make results incorrect.

First, we consider the following simplest example when the black-box model is of the form:

Let us estimate the feature importance by using the proposed dual model. We generate n = 1000 points x_i, i = 1, ..., N, with components uniformly distributed in interval [0, 1], which are explained. For every point x_i, the dual model with K = 10 nearest neighbors is constructed by generating 30 vectors λ⁽ⁱ⁾ ∈ ℝ⁷ in the unit simplex. By applying Algorithm 1, we compute optimal vector a(i)=(a1,...,a7)T for every point x_i. We expect that the mean value a¯ of a⁽ⁱ⁾ over all i = 1, ..., N should be as close as possible to the true vector of coefficients a forming function f(x). The corresponding results are shown in Table 1. It can be seen from Table 1 that the obtained vector a¯ is actually close to vector a.

Let us consider another numerical example where the non-linear black-box model is investigated. It is of the form:

We take N = 400 and generate two sets of points x. The first set contains x whose features are uniformly generated in the interval [0, 1]. The second set consists of x whose features are uniformly generated in the interval [15, 16]. It is interesting to note that the feature x₁ is more important for the case of the second set because x12 rapidly increases whereas x12 decreases when we consider the first set and x₂ is more important in this case.

We take K = 6 and generate 30 vectors λ⁽ⁱ⁾ uniformly distributed in the unit simplex for every x to construct the linear model h(λ⁽ⁱ⁾). Mean values of the normalized importance of features x₁ and x₂ obtained for the first set are −0.3 and 0.86 and for the second set are −0.95 and 0.37. These results completely coincide with the importance of features considered above for two subsets.

A goal of the following numerical example is to consider a case when we try to get predictions for points lying outside bounds of data on which the black-box model was trained as it is depicted in Figure 1. In this case, the predictions of generated instances may be inaccurate and can seriously affect quality of many explanation models, for example, LIME, which uses the perturbation technique.

The initial dataset consists of n = 400 feature vectors x₁, ..., x_n such that there holds

where parameter ρ² is uniformly distributed in interval [0, 2²]; parameter φ is uniformly distributed in interval [0, 2π].

The observed outputs y_i = f(x_i) are defined as

We use two black-box models: the KNN regressor with k = 6 and the random forest consisting of 100 decision trees, implemented by means of the Python Sckit-learn. The above black-box models have default parameters taken from Sckit-learn.

We construct the explanation models at l = 100 testing points x_{1, test}, ..., x_{l, test} of the form Equation 22, but with parameters ρ² uniformly distributed in [1.9², 2²] and φ uniformly distributed in [0, 2π]. It can be seen from the interval of parameter ρ that a part of generated points can be outside bounds of training data x₁, ..., x_n. Figure 6 shows the set of instances for training the black-box model and the set of testing instances for evaluation of the explanation models.

The dual model is constructed in accordance with Algorithm 1 using K = 6 nearest neighbors. We generate 30 dual vectors λ^(j) to train the dual model. We also use LIME and generate 30 points having normal distribution N(xj,test,Σ), where Σ = diag(0.05, 0.05). Every point has a weight generated from the normal distribution with parameter v = 0.01.

To compare the dual model and LIME, we use the mean squared error (MSE) which measures how predictions of the explanation model g(x) are close to predictions of the black-box model f(x) (KNN or the random forest). It is defined as

Values of the MSE measures for the dual explanation model and for the original LIME, when KNN is used as a black-box model, are 0.01 and 0.02, respectively. It can be seen from the results that the dual model provides better results in comparison with LIME because some generated points in LIME are located outside the training domain. Values of the MSE measures for the dual explanation model and for the original LIME, when the random forest is used as a black-box model, are 0.005 and 0.014, respectively.

Let us perform a similar experiment with real data by taking the dataset “Combined Cycle Power Plant Data Set” (https://archive.ics.uci.edu/ml/datasets/combined+cycle+power+plant) consisting of 9568 instances having 4 features. We use Z-score normalization (the mean is 0 and the standard deviation is 1) for feature vectors from the dataset. Two black-box models implemented by using the KNN regressor with K = 10 and the random forest regressor consisting of 100 decision trees. The testing set consisting of l = 200 new instances is produced as follows. The convex hull of the training set in the 4-dimensional feature space is determined. Then, vertices of the obtained polytope are computed. Two adjacent vertices x_j1 and x_j2 are randomly selected. Value λ is generated from the uniform distribution on the unit interval. A new testing instance x_{j, test} is obtained as x_{j, test} = λx_j1+(1−λ)x_j2. Then, we again select adjacent vertices and repeat the procedure for computing testing instances l times. As a result, we get the testing set x_{j, test}, j = 1, ..., l.

The dual model is constructed in accordance with Algorithm 1 using K = 10 nearest neighbors. We again generate 30 dual vectors λ^(j) to train the dual model. We also use LIME and generate 30 points having normal distribution N(xj,test,Σ), where Σ = diag(0.05, 0.05, 0.05, 0.05). Every point has a weight generated from the normal distribution with parameter v = 0.5.

Values of the MSE measures for the dual explanation model and for the original LIME, when KNN is used as a black-box model trained on dataset “Combined Cycle Power Plant Data Set”, are 84 and 0173, respectively. It can be seen from the results that the dual model provides better results in comparison with LIME because some generated points in LIME are located outside the training domain. Values of the MSE measures for the dual explanation model and for the original LIME, when the random forest is used as a black-box model, are 110 and 282, respectively. One can again see from the above results that the dual models outperform LIME.

We start from the synthetic instances illustrating the dual example-based explanation when NAM is used. Suppose that the explained instance x₀ belongs to a polytope with six vertices x₁, ..., x₆ (d = 6). The black-box model is a function f(x) such that

n = 2, 000 vectors λ⁽ⁱ⁾ ∈ ℝ⁶, i = 1, ..., n, are uniformly generated in the unit simplex Δ^{6 − 1}. For each point λ⁽ⁱ⁾, the corresponding prediction z_i is computed by using the black-box function h(λ). NAM is trained with the learning rate 0.0005, with hyperparameter α = 10⁻⁴, the number of epochs is 300, and the batch size is 128.

To determine the normalized values of the importance measures I(λ_i), i = 1, ..., 6, we use three approaches. The first one is to apply the method called accumulated local effect (ALE) (Apley and Zhu, 2020), which describes how features influence the prediction of the black-box model on average. The second approach is to construct the linear regression model (LR) by using the generated points and their predictions obtained by means of the black-box model. The third approach is to use NAM.

The corresponding normalized values of the importance measures for λ₁, ..., λ₆ obtained by means of ALE, LR, and NAM are shown in Table 2. It should be noted that the importance measure I(λ_i) can be obtained only for NAM and ALE. However, normalized coefficients of LR can be interpreted in the same way. Therefore, we consider results of these models jointly in all tables. One can see from Table 2 that all methods provide similar relationships between the importance measures I(λ₁), i = 1, ..., 6. However, LR provides rather large values of I(λ₃), I(λ₅), I(λ₆), which do not correspond to the zero-valued coefficients in Equation 24.

Shape functions illustrating how functions of the generalized additive model depend on λ_i are shown in Figure 7. It can be clearly seen from Figure 7 that the largest importance λ₂ and λ₄ have the highest importance. This implies that the explained instance is interpreted by the fourth and the second nearest instances.

Suppose that the explainable instance x₀ belongs to a polytope with four vertices x1*,...,x4* (d = 4). The black-box model is a function f(x) such that

n = 1000 points λ⁽ⁱ⁾ ∈ ℝ⁴, i = 1, ..., n, are uniformly generated in the unit simplex Δ^{4 − 1}. For each point λ⁽ⁱ⁾, the corresponding prediction z_i is computed by using the black-box function h(λ). NAM is trained with the learning rate 0.0005, with hyperparameter α = 10⁻⁶, the number of epochs is 300, and the batch size is 128.

Normalized values of I(λ_i) obtained by means of ALE, LR, and NAM are shown in Table 3. It can be seen from Table 3 that the obtained importance measures correspond to the intuitive consideration of the expression for h(λ). The corresponding shape functions for all features are shown in Figure 8.

Suppose that the explained instance x₀ belongs to a polytope with three vertices x1*,x2*,x3* (d = 3):

The black-box model has the following function of two features x⁽¹⁾ and x⁽²⁾:

We generate n = 1, 000 points λ⁽ⁱ⁾ ∈ ℝ³, i = 1, ..., n, which are uniformly generated in the unit simplex Δ^{3 − 1}. These points correspond to n vectors xi∈ℝ2 defined as

with the corresponding values of f(x_i) and shown in Figure 9. It can be seen from Figure 9 that this example can be regarded as a classification task with four classes. Parameters of experiments are the same as in the previous examples, but α = 0.

Normalized values of I(λ_i) obtained by means of ALE, LR, and NAM are shown in Table 4. It can be seen from Table 4 that the obtained importance measures correspond to the intuitive consideration of the expression for h(λ). The corresponding shape functions for all features are shown in Figure 10.