Given a set of variables v describing a domain, with causal structure represented by graph G. Counterfactual inference in v requires a structural causal model (SCM) (Galles and Pearl, 1997; Pearl, 2009). An SCM has components u,v,F, where v is a set of endogenous variables, u is a set of exogenous variables, and F is a set of deterministic structural functions, one for each endogenous variable. A structural function f_v(pa_v) = v determines the value of variable v given its parents pa_v in the causal graph G. The exogenous variables u are introduced as root variables in G, such that each u is a parent to exactly one endogenous variable v. Here, causal sufficiency is assumed (u_i⫫u_j for all u_i, u_j ∈ u, i ≠ j). Such SCMs are called Markovian SCMs (Avin et al., 2005) and do not allow exogenous causal confounding. Note that an endogenous variable v may be modeled without an exogenous parent if v is assumed to be deterministically given by its endogenous parents.

The distinction between endogenous and exogenous variables is here assumed to be made based on semantic interpretation. All variables in the environment described by G that represent the concepts of known interpretation are modeled as endogenous variables, regardless of whether it is observed or not. Then, the exogenous variables are included where relevant to account for causal influence of origin outside of the model scope. Exogenous variables are thus always unobserved.

The SCM is further rendered probabilistic by introducing a probability distribution P(u) over the exogenous variables. An SCM where all components 〈u,v,F,P(u)〉 are defined, is referred to as a fully specified SCM (FSCM) (Pearl, 2009; Zaffalon et al., 2024). If the probability distribution P(u) is unknown, the SCM is referred to as a partially specified SCM (PSCM). The most general SCM lets each structural function in F admit every possible function from its endogenous inputs to its output variable and is referred to as canonical.

A counterfactual query combines observed evidence with a hypothetical, retrospective intervention (Pearl, 2009). An intervention, denoted do(v = v), assigns a value for a variable while breaking the dependence between the intervened variable and its causal parents, to simulate action, as opposed to change in observation. A counterfactual query is of the form P(wdo(v=v′)=w′|v,w) ¹, read as the probability that variable w had taken value w′ if variable v had been intervened to take value v′, given that the observed event is v = v, w = w.

Given two binary variables, v and w, the Probability of Sufficiency (PS) is given as P(w_{do(v = 1)} = 1|w = 0, v = 0) (Pearl, 2009), and gives the probability to which variable v intervened to take on value 1 is sufficient for variable w to take on value 1, given that it is observed that v = 0 and w = 0.

For binary variables of certain interpretation, the query may be read as the probability of variable v being a sufficient cause of variable w, implicitly referencing values at 1 as activations. Generalizing the probability of sufficiency to all discrete variables as P(wdo(v=v)=w|w=w′,v=v′) with v ≠ v′ and w ≠ w′, the query is rather read with explicit reference to the values, i.e., the probability of v = v being a sufficient cause of w = w in a context where neither v nor w is observed.

Several types of explanations are defined within the field of XAI. These include feature attributions (Ribeiro et al., 2016; Lundberg and Lee, 2017), which are the scores assigned for each data feature to quantify the contribution of the feature toward the model's outcome. By a generalized interpretation, this type of explanation extends to concept attributions, where attribution is measured for variables not explicit in the data representation.

Another type of explanation is the contrastive explanation (Wachter et al., 2018). For a given sample x to be explained, contrasting data samples that are similar to x but that differ in model prediction are presented as the explanation. This type of explanation is also known as a counterfactual explanation in the XAI field, but the term counterfactual in this context has a different meaning than it does in causality. To avoid confusion, contrastive is used whenever it is the XAI type of explanation that is referred to, while counterfactual explanation refers to explanations generated with counterfactual queries as detailed in Section 2.2.

The scope of explanations for XAI range from local to global (Angelov et al., 2021). Local explanations consider a single data sample x and explain the model's prediction for this sample by considering the model's behavior in a local neighborhood around x only. Global explanations instead explain the model by identifying global trends representative of universal model behavior. Global explanation techniques can be applied to a subgroup of data samples with certain attributes in common, or to infer relationships descriptive of all data samples.

A definition of XAI given by DARPA refers to XAI as “AI systems that can explain their rationale to a human user, characterize their strengths and weaknesses, and convey an understanding of how they will behave in the future” (Gunning and Aha, 2019). In line with this, post-hoc XAI aims to ensure explainability for non-interpretable models, by employing techniques that reveal why the model predicts a certain output. This in turn can permit better evaluation of model performance, increase user trust in the decision making process, or help users understand how a decision can be changed in situations where decisions are not permanent. The latter requires a special case of explanation, where the explanation vocabulary is restricted according to actionability, i.e., whether a variable may be feasibly intervened upon in a current world state. Explaining under actionability constraints with the goal of changing a model decision is referred to as algorithmic recourse. While it requires reliable causal modeling to accurately predict the result of the suggested interventions, it is concerned with the future outcome of actualized interventions only.

The type of post-hoc explanation discussed in this study neither makes assumptions regarding the explainee guiding the process of explanation, nor about the performance or deployment stage of the model. This is done to ensure that explanations are faithful to the model, also when the model cannot be assumed to behave as intended or expected. Counterfactual reasoning about retrospective interventions on all variables is permitted, with no requirement for actionability. While the framework discussed may be applied for algorithmic recourse with some adjustments, the details of this is out of scope of this study.

Concept-based explanations use concepts rather than data features as their vocabulary. While there exist multiple definitions of concepts for XAI, it is the type of concept identified as symbolic concepts by Poeta et al. (2025), that is referred to here. Thus, a concept corresponds to a high-level attribute or characteristic of the data, with semantic meaning as defined by a human.

Because concepts are not explicitly defined in the data features, it is common to use probing to detect neurons, or combinations of neurons, that best represent target concepts. Such probes are often linear, but may also be non-linear in cases where the concept is not linearly separable. Furthermore, probes can be trained in both supervised and unsupervised manners, where the former requires labeled data of the human-specified concept, and the latter identifies concepts that are important to the model, but not necessarily interpretable to humans.

To assess the importance of identified concepts, concept attribution methods such as TCAV (Kim et al., 2018) quantify the importance of a concept z, probed in layer ℓ, for a given output class y. Concepts are learned using linear classifier probes such as logistic regression, resulting in the coefficient-vector v_z, which is often interpreted as the direction in which the concept is represented. Let f_ℓ(x) be the result of forwarding the input x to the intermediate activation in layer ℓ, and g_{ℓ, y}(·) be the remainder of the network, mapping those activations to the logit of class y. Kim et al. (2018) define concept sensitivity as the directional derivative of g_{ℓ, y} wrt. f_ℓ(x) compared to the concept direction v_z. For an input x, the sensitivity score is thus ∇fℓgℓ,y(fℓ(x))⊤vz. These scores are aggregated over a background dataset, here a subset of the test data D*, providing the ratio of positive concept sensitivities as

where I[A] is 1 if A is true and 0 otherwise.

The purpose of the causal framework is to allow explanation of a black-box model h(x)=y^. This model h takes a set of input features x and predicts a corresponding output y^. No assumptions are made about the architecture of h, and importantly, no assumptions are made about the performance of h.

For h to be interpreted as a causal process, it is important to clearly distinguish prediction y^ from the attribute y being predicted. Given an instance x, y = y is considered to represent the ground truth attribute corresponding to x, which generally may not be observed outside of training data. The model prediction y^=h(x) is the model's guess as to what value y takes, such that it is not guaranteed that y = ŷ. This is clearly illustrated if for ground truth y that takes one of two values, h is chosen to be a constant function predicting the same ŷ for all inputs. Even if h perfectly predicts y by y^, the causal nature of these variables is distinct. While the prediction y^ is always caused by x through h, it is rarely meaningful to model x as a cause of the true attribute y. Rather, y can be thought of as a concept that is part of the concept representation z with causal influence on x, or otherwise caused by concepts in z, and thus correlated with x by confounding. This distinction is illustrated further in an example presented in Section 5.

Explaining h is approached by relating change in y^ to systematic change in x. However, the features x themselves will not in general constitute a useful vocabulary for explanation, as atomic features in x may correspond to pixels in an image, words in a text, nodes in a graph etc. Thus, to quantify meaningful change in x, a concept-based vocabulary with known causal structure is considered.

The explanation vocabulary is denoted z, with each variable z_i representing a meaningful, high-level concept relevant to describe the information represented in x. Now z permits explanations in a language understandable for a human explainee.

To describe change in x by change in z, an invertible mapping α between the two sets of variables is required. Ideally, z is sufficient to generate x through a deterministic α, such that all information in x originates from meaningful concepts in z available for explanation. As this is rarely feasible in practice, another set of variables w is introduced, such that α(z, w) = x, with w encoding any unknown factors of variation in x not represented in z. Hence, α is kept a deterministic function. The variables w are not required to be interpretable, and it is assumed that the variable sets z and w are independent.

The independence z⫫w is assumed in order for the effect of interventions in z on x to be correctly modeled, which is not feasible if p(w|do(z)) ≠ p(w). It is noted that w is never to be considered part of the explanation language. For local explanations, w will be considered a context fixed at some value w = w, such that the effect of changes to z may be consistently mapped to x through α.

Another important assumption is thus that z, while not a complete model of all information present in x, constitutes a sufficiently comprehensive explanation language for h's decision logic to be (partially) exposed.

Conceptually, α is considered a decoder α(z, w) = x, and α⁻¹ is a corresponding encoder α⁻¹(x) = z, w. The independence z⫫w is assumed preserved by α⁻¹ ° α. Dependence among variables in w is irrelevant such that independence here may be enforced if convenient. However, the variables of z can be dependent.

Note that while α is modeled as an invertible mapping, it is interpreted as representing a causal process in the data generative direction z, w → x. The causal influence of a world state on a data instance through the ground truth data generative process is modeled as causal influence of the concepts z as a high level representation of this world state, as well as unknown factors in w, on x.

Expressing explanations in terms of causally meaningful change to z further requires a reliable model of z's causal structure, which represents the underlying physical mechanisms governing the relationships of these concepts in the real world. This structure is represented by a causal graph G.

An explicit causal model M is introduced over variables z with structure G. For maximum capacity for reasoning, M is considered an FSCM, and a final set of variables u is included, to act as M's exogenous variables. The model M is assumed to be Markovian as described in Section 2.2. Figure 2 presents an example graph G over variable set z={zi}i=15, with Markovian exogenous variables u={ui}i=15.

The requirement that M be an FSCM can be challenging to meet in practice, as no amount of data, observational or experimental, will reveal the exact nature of the underlying process to be captured by a structural function. Thus in-depth domain knowledge is typically required for the design of reasonable structural functions, with no automated way of verifying the reliability of a given model. Regardless, an FSCM is required for exact calculation of counterfactual queries, and so is the ideal modeling approach to M for causal explanations.

If a single, reasonable choice of FSCM cannot be provided, alternatives may be considered, at some cost to explanation quality. Assuming model complexity is sufficiently low, a PSCM may be considered, for which counterfactual queries may be bounded by intervals. The most general PSCM to be considered is referred to as canonical, which provides intervals guaranteed to contain the true counterfactual probability.

Finally, if no SCM can be provided, M can be modeled as a causal Bayesian network (CBN). In this setting, counterfactual reasoning is not feasible, and the framework may be applied for the calculation of interventional queries only.

With M, α, h as detailed, the causal flow is now summarized. The concept variables z have internal causal structure and are caused by their respective parents among the other concepts, as well as exogenous parents u, exemplified in Figure 2. The concepts z together with unknown factors w are the causes of the data features x through the data generative process represented by α. Finally, the data features x are the causes of the model prediction y^ through the model h. Figure 3 summarizes the complete causal flow for the example in Figure 2. The concepts in z are thus indirect causes of prediction y^, and this is the relationship explored for explanation. While all variables are part of the extended causal model for explanation, only the variables in z are considered for intervention to detect causal relationship. Any intervention to variables x or y^ would remove the link with concept space and the vocabulary chosen for explanation. As u and w are unobserved and uninterpretable, interventions here are not informative.

The framework yields the complete set of variables x∪{y^}∪z∪u∪w. The joint distribution over these variables factorizes as follows:

To explain h, queries of the generalized form p(y^do(z̄=z̄′)|o) are considered, quantifying the effect on the model prediction when intervening on a subset of concepts z̄⊂z, conditioned on some observation o. The observation o is the context of the explanation, and different types of explanations are defined based on varying o.

Notation z̄̄ is used to denote the set of variables in z influenced by an intervention to z̄ but not in z̄, i.e., the causal descendants of variables z̄. Notation z_ denotes variables in z unaffected by an intervention to z̄, such that z=z̄∪z̄̄∪z_.

The general probability of sufficiency presented in Section 2.2 is adapted and applied for XAI explanations. The target variable of interest is the model prediction y^. Hypothetical change in the form of a counterfactual intervention is considered for a subset of concept variables z̄. The context is either a local instance x, ŷ, z, w, or a subgroup defined by common concepts and prediction z_s, ŷ such that z̄∈zs. Then the probability of sufficiency for XAI is given by p(y^do(z̄=z̄′)=ŷ′|x,ŷ) (local) or p(y^do(z̄=z̄′)=ŷ′|zs,ŷ) (subgroup), with ŷ′ ≠ ŷ and z̄′≠z̄. The query quantifies the effect of the intervention do(z̄=z̄′) on y^, relative to observed explanandum ŷ and z̄.

Note that it is the counterfactual outcome that is the target of the sufficiency query, relative to the observation. The query is thus primarily explanatory with regards to the influence of the changed concepts toward the changed outcome. However, information about the causal influence of the observed concepts on the observed outcome is (partially) revealed, by letting sufficiency of change indicate necessity of the original value. Note that if |Ωz̄|>2, a new value z̄′ being sufficient for change to occur in y^ is not always equivalent to the observed value z̄ being necessary for the observed ŷ. Still, the respective scenarios are related.

Given a single data point or a subgroup, the probability of sufficiency can be seen as a measure of attribution for the set of intervened features. The probability of sufficiency is to be read as a measure of attribution of intervened value z̄′ toward alternative outcome ŷ′ in a context (e.g. x, ŷ), and is only indirectly descriptive of observed z̄'s attribution toward observed ŷ. In the special case where |Ωz̄|=2, i.e., it is a single binary concept that is intervened, the probability of sufficiency can be interpreted as a concept importance score as defined for XAI, reflecting the probabilities for the sufficiency of the counterfactual concept value toward the counterfactual outcome and the necessity of the original concept value toward the original outcome.

The presented framework enables the generation of contrastive explanations. The contrastive explanation form considered, based on the explanation form first introduced by Galhotra et al. (2021), is summarized as: For a data instance x that has concept values z, and for which a model h made the decision ŷ, the decision would have been ŷ′ with probability p had the concept(s) z̄ taken value(s) z̄′ with z̄ the subset of the concepts that are changed by (hypothetical) intervention. The probability p is the local probability of sufficiency p(y^z̄=z̄′=ŷ′|x,ŷ).

When selecting the best contrastive explanations from a set of contrastive samples, similarity with the original data instance is typically measured (Wachter et al., 2018). When data features are changed independently, similarity can be calculated as number of changed features. With the contrastive form presented here, similarity is interpreted as the number of variables intervened upon, i.e. |z̄|. Additionally, the probability p must be considered when evaluating a potential explanation. Generally, it is reasonable to expect p to be high in order for the explanation to be useful. In some cases, a limit p^* for this probability may be given, such that it is required that p ≥ p^* for a sample to be considered an explanation. This in turn may allow a simplified explanation form where p is omitted, which is argued helpful in increasing understandability (Miller, 2019).

Searching for optimal contrastive explanations of the form presented, maximizing similarity and probability, is left for future work.

We build an FSCM over a selected subset of the concepts z. The causal structure was learned using an expert in the loop-method from pgmpy (Ankan and Textor, 2024), with a large language model (LLM) acting as the expert deciding on causal direction, and with the forced constraint that concepts Young and Gender should be independent. Conditional independence is also enforced for variables Smiling and Big Nose. The resulting structures are given in Figure 4 for the age classification and Figure 5 for the classification of attractiveness. The parameters of the FSCM defining the probability distribution p(u) were chosen to be sparse and meaningful, while compatible with the observed data D; see Appendix A for details. PSCM intervals are retrieved using the DCCC method for exact interval calculation (Bjøru et al., 2025a). Counterfactual queries are calculated using BCAUSE (Cabañas, 2025).

We use a 50-layer ResNet following the procedure of He et al. (2016). While our framework can explain the behavior of any classifier, irrespectively of merit, we note that both classifiers performed at an acceptable level, with accuracies above 80%.

To have a complete computational pipeline for counterfactual explanations, we train a generative model as a means to represent the mapping between concept-space and data-space. In particular, we provide counterfactual data examples as follows: First, the mapping α⁻¹(x) = z, w is partially provided in the dataset itself, as each image x_i is annotated with concepts z_i. w is intended to capture the remaining information not described by z, e.g., image background. Here we simply set w = x, and will discuss the effects of this simplification in Section 5.3. We aim to evaluate a counterfactual explanation do(z̄=z̄′) by calculating p(y^do(z̄=z̄′)|x,ŷ), cf. Section 4.2. This requires generation of counterfactual data-objects α(z′, w) where z′={z̄′∪z̄̄∪z_} is consistent with the causal model M (see Equation 1). Our implementation adopts the StarGAN architecture (Choi et al., 2018) for image-to-image translation in order to provide the functionality of α(z′, w). StarGAN takes an image x and a binary vector of high-level concepts z′ as inputs and produces an alternation of x consistent with concept description z′={z̄′∪z̄̄∪z_} but otherwise incentivized to be close to x.

We first consider the images in Figure 6. For each row, the original image is given to the left, and we look for explanations of why these people are classified as being young. The explanation engine outlined above provides justification by answering counterfactual questions of the type “Would the model h(·) classify the person as young if 〈counterfactual〉?”, where the counterfactual is defined using the vocabulary shown in Figure 4. An intermediate step of producing explanations is to generate counterfactual images as described above. The counterfactual images shown in columns 2 and 3 are related to the concepts Glasses and Gray Hair. Adding glasses is not sufficient for the classifier to change its evaluation of either person in this example, but changing hair color is sufficient for the male to be classified as old².

Note that even though Young is a concept in the causal model, its value is in general distinct from the predicted value ŷ = h(x) as the classifier is not infallible. Intervening in the causal model M with do(Young = 0) will imply higher probabilities of both Gray Hair and Glasses (consider the causal model in Figure 4), in addition to other visual cues related to this concept, like higher affinity for wrinkles and under-eye bags. To examine counterfactuals with z̄={Young}, we need to marginalize out the uncertainty in the concepts z̄̄={Glasses,Gray Hair}, cf. Equation 1. Two of the counterfactual data-points produced by α to make this calculation are shown in columns 4 and 5 of Figure 6. We eventually find that p(y^do(Young=0)=0|x,ŷ=1)=1 for both subjects, validating the explanation that the model h(·) would indeed consider both persons to look old if they were old.

We next turn to the images in Figure 7. Following from the original image on the left, the figure shows different counterfactual images all consistent with the intervention do(Young = 0). The first counterfactual image, obtained by the intervention do(Young = 0, Glasses = 0, Gray Hair = 0), is erroneously classified as being of a young person, while the other counterfactual images are correctly identified as being of old subjects. Using these counterfactual images to marginalize out the uncertainty over z̄̄, we can calculate p(y^do(Young=0)=0|x=, ŷ=1), the probability that the classifier would classify the subject as old in the counterfactual world where Young = 0. The probabilistic model-class employed to do the calculations will provide different results, as showcased in Table 1.

While XAI systems that solely rely on visual assessment can be misleading (Adebayo et al., 2018), saliency maps are still an often-used technique for attribution-based explanations of image classification systems. We, therefore, compare the results of our causal concept-based explanations with the results of Grad-CAM (Selvaraju et al., 2017) using the implementation by Gildenblat (2021). Consider first the results in Figure 8 which are intended to explain why the two actual (far left) images in Figure 6 were classified as being young. Surprisingly, the top of the head is important for the female's classification while the area around the eyes and one of the ears are of most importance for the classification of the male. Next, Figure 9 investigates the classification of the subject in Figure 7. The first image was correctly classified as young, but according to the Grad-CAM explanation, the classifier has based this decision on elements detected in the image background. The explanation of the erroneously classified image to the right focuses on the forehead and the chin. Grad-CAM provides non-causal explanations, and therefore fails at giving counterfactual information. Instead of considering change, the explanation rather focuses on what part of the input space that are most indicative about the class being chosen the classifier. Furthermore, since the explanations are pixel-based, we do not get insights regarding why the highlighted areas are considered important.

As an example of subgroup-explanations, we provide the probability that changing a single concept would suffice to change h(·)'s prediction of a person from being young (y^=1) to being old in Table 2. We can write this probability compactly as p(y^do(z̄=z̄)=0|z̄=zs,y^=1), cf. Equation 2, where the conditioning on z̄=zs is used to ensure that the value of z is indeed changed by the intervention. For simplicity we restrict ourselves to the most relevant singleton interventions, |z̄|=1. As an example, consider the effect of setting z̄={Gray Hair}, z_s = 0 and z̄=1. Now we are doing an investigation of the subpopulation of subjects that are classified as being young but do not have gray hair, looking for the counterfactual probability that the classifier would consider the subject to be old if their hair color had been changed to gray. The relatively large value for this probability shows that hair color is an important concept to explain the behavior of the classifier, which may be used to spark further investigation into potential bias in the classifier before it can be deployed. We investigate this further in Table 3, where the same calculation is broken down by gender and whether or not the subjects wear glasses. The effect is stronger among males than females, and albeit with a limited dataset, it also seems like the effect is stronger among those wearing glasses.

Since setting z̄={Young} implies z̄̄≠∅, we must again marginalize out the uncertainty over z̄̄. How this is done depends on how the causal model M is represented (cf. Table 4). For this particular query, the canonical interval is narrow, such that the choice of FSCM has limited influence on the calculated probability of sufficiency. Using a model with independent concept variables slightly underestimates the global effect of the intervention, as was also shown to be the case locally for the image in Figure 7.

Global explanations are shown for the attractiveness classifier in Tables 5, 6. Table 5 shows the effect of singleton interventions in young people, grouped by their gender. The numbers are understood as follows, using the value 0.139 in the first row as an example: This is the probability that a female (Gender = 0) currently classified as unattractive (ŷ = 0) not having 5 o'Clock Shadow (z̄=¬z̄=0) would be classified as attractive (ŷ = 1) if she did (z̄=1). The table indicates that females improve their chance of being classified as attractive the most if they remove a possible Big Nose, whereas the best strategy for men is to obtain a Pointy Nose if they do not already have one. For comparison, concept attributions using TCAV scores (Kim et al., 2018) are included in Table 7. These values reflect how concept interventions are moving the prediction toward the subject being seen as more attractive. According to the TCAV scores, obtaining Arched Eyebrows is most beneficial for both genders, followed by the removal of Bushy Eyebrows. The differences in attribution from the two approaches are to be expected. TCAV concept directions are identified with labeled data per concept, with no causal considerations. Sufficiency scores, however, directly reference the probability of crossing a decision boundary, with sound causal interpretation. As an example, the sufficiency attributions for Arched Eyebrows in Table 5 are estimated based on controlled generation of data where only the eyebrows change, thus reflecting the isolated effects of this change. The TCAV scores using observed data (conditioning on a value for Arched Eyebrows) are more prone to influence from correlated concepts. This could be an explanation for the contrasting results obtained for this concept by the two approaches.

Note that the sufficiency scores also better reflect subgroup tendencies, with the effect of interventions to, e.g., Big Nose, Arched Eyebrows and Smiling being more distinct when comparing the male and female subgroups, than what is reflected by the TCAV scores.

Lastly, Table 6 is included as an example of an explanation based on an intervention to a non-leaf variable in the explanation graph for the attractiveness classifier. For young females without makeup, the intervention Makeup = 1 is considered. It is shown that accounting for the causal influence of Makeup on the concepts representing appearance of Pointy Nose and Arched Eyebrows, attributes a higher effect of Makeup on the attractiveness prediction, as measured by the probability of sufficiency, compared to the independent intervention. The probability of sufficiency for Pointy Nose = 1 in particular, shown in Table 5, suggests that Pointy Nose = 1 increases the chance of predicted attractiveness, which is accounted for by the FSCM-based calculation of the probability of sufficiency of Makeup = 1. Independent intervention on Makeup, however, will not influence the value of Pointy Nose, and so the appearance of a pointy nose in the generated image will not change. The PSCM interval included in Table 6 further suggest the independence-model underestimates the true causal effect of Makeup compared to all possible FSCMs compatible with the data, with the independent attribution notably lower than that of the lower PSCM bound.

Section 5.2 presents a set of results obtained by implementing and testing a proof-of-concept explanation model for classifiers trained on the CelebA dataset. As discussed in Section 1, understandability and fidelity of explanations are contingent on the viability of the model assumptions under which explanations are generated. Thus, this section discusses the assumptions made for the proof-of-concept model presented, and their implications regarding explanation interpretation. While the design of this model is intentionally simplistic in order to feasibly illustrate the potential of the general framework, implementation-specific weaknesses are discussed to highlight challenges that may arise when model assumptions are violated.

The level of abstraction of a causal model M determines how the total causal influence of one variable on another is divided between direct and indirect causal influence. If the explanation vocabulary is restricted to be z = {Young}, the causal structure considered is

modeling the total causal effect as direct effect, abstracting away any mediators.

The example presented in Figure 4, considers the structure

such that some of the total effect is mediated through variables gray hair and glasses.

With the data being images of people limited to visual information, it is conceivable that the vocabulary could be extended further such that all causal effect from young to an image x is mediated through some physical feature explicit in z:

In all three models, the total causal influence of young on the image and prediction is the same, illustrating that it is a reliable measure independent of abstraction. However, if an explanation model ignores the causal structure by intervening on variables as if independent, only the direct effect is captured, which varies from equal to the total effect for the first model, to 0 for the last model. Thus, direct influence is not a consistent measure.