In software product development, product teams are frequently interested in learning the best software variant to show or roll out to each subgroup of users. To achieve this, it is often possible to conduct targeting at scale simultaneously, i.e., in one shot, for various combinations of contexts (representing user groups) and actions (representing software variants)—instead of one experiment at a time.³ These targeted experiments can be used to compute relevant metrics (e.g., click-through rate) for each context-action pair. Further, we might have some qualitative domain knowledge of how some context variables are causally related to others; for example, we might know that os has a causal relationship to browser; we would like to exploit this knowledge to learn better policies. This can be naturally modeled as a question of “given the causal graph, what table of targeted experimental data needs to be acquired and how to integrate that data, so as to learn a good policy?” here the table's rows and columns are contexts and actions, and each cell specifies the number of data samples (zero or more) required corresponding to the context-action pair. At the end of the above training phase, the agent moves to an evaluation phase (e.g., when it is deployed), where it observes contexts, and takes actions using the learned policy.

As another example, in online marketing campaign strategy, the objective is to learn the best campaign to show to each group of people.⁴ This is done by conducting experiments where different groups of people are shown different marketing campaigns simultaneously, and learning from the resultant outcomes. Further, letting context variables model the features of groups, we might have some knowledge about how some of the features are causally related to others; for example, we might know that country causally affects income. Our framework provides a natural model for this setting. Further, we might also be interested in ensuring that the campaigns meet certain criteria for fairness. For example, there might be some variables such as race that could be sensitive from a fairness perspective, and we would not want the agent to learn policies that depends on these variables. We discuss fairness implications of our algorithm in Section 5. These are just two of many scenarios where this framework provides a natural model. Two additional examples include experimental design for ads⁵ and recommendation systems.⁶

Our framework captures the various complexities and nuances described in the examples in Section 1.1. We present an overview here; please see Section 2.1 for the mathematical formalism. The agent's interactions consist of a learning phase where the agent incurs no regret, followed by an evaluation phase where regret is measured; this is also called simple regret (Lattimore and Szepesvári, 2020). At the start of the learning phase, the agent is given a (possibly empty) log of offline data—consisting of context-action-reward tuples—generated from some unknown policy. The context variables are partitioned into two sets—the main set and the auxiliary set (possibly empty). The agent observes all context variables during this phase, but learns a policy that only depends on the main set of context variables; this also provides a way to ensure that the learned policy meets certain definitions of fairness (see Section 5 for a more detailed discussion). Further, the agent also has some qualitative⁷ causal side-information available, likely from domain knowledge. This causal side-information is encoded as a causal graph between contextual variables. A key implication of the causal graph is information leakage (Lattimore et al., 2016; Subramanian and Ravindran, 2022)—getting samples for one context-action pair provides information about other context-action pairs because of shared pathways in the causal graph.

Given the logged data and the causal graph, the agent's problem is to decide the set of targeted experimental samples to acquire within a budget, and then integrate the returned samples. More specifically, the agent is allowed to make a one-shot request for data in the form of a table specifying the number of samples it requires for each context-action pair, subject to the total cost being within the budget. The environment then returns the requested samples after conducting the targeted interventions, and the agent integrates those samples to update its internal beliefs and learned policy. After this learning phase, the agent moves to an inference or evaluation phase, where it returns an action (according to the learned policy) for every context it encounters; its regret is measured at this point. The core problem of the agent is to choose these samples in a way that straddles the trade-off between choosing more samples for context-action pairs it knows is likely more valuable (given its beliefs) and choosing more samples to explore less-seen context-action pairs—while taking into account the budget and the information leakage across all obtained samples arising from the causal graph.

Causal bandits have been studied in the last few years (e.g., Lattimore et al., 2016; Yabe et al., 2018; Lu et al., 2020), but they study this in a multi-armed bandit setting where the problem is identification of one best action. There is only one work (Subramanian and Ravindran, 2022) studying causal contextual bandits—where the objective is to learn a policy mapping contexts to actions—and this is the closest related work. While we do leverage some ideas introduced in that work in our methodology and in the design of experiments, our work differs fundamentally from this work in important ways. Subramanian and Ravindran (2022) consider a standard interactive setting where the agent can repeatedly act, observe outcomes and update its beliefs, whereas in our work the agent has a one-shot data collection option for samples from multiple context-action pairs. This fundamentally changes the nature of the optimization problem as we will see in Section 2.2; it also makes it a more natural model in a different set of applications, some of which were discussed in Section 1.1. Further, our work allows for arbitrary costs for collecting those samples, whereas they assume every intervention is of equal cost.

Contextual bandits in purely offline settings, where decision policies are learned from logged data, is a well-studied problem. Most of the work involves inverse propensity weighting based methods (such as Swaminathan and Joachims, 2015b,a; Joachims et al., 2018). Contextual bandits are also well-studied in purely interactive settings (see Lattimore and Szepesvári, 2020 for a discussion on various algorithms). However, in contrast to our work, none of these methods can integrate causal side information or provide a way to actively acquire and integrate new targeted experimental data.

Active learning (Settles, 2012) studies settings where an agent is allowed to query an oracle for ground truth labels for certain data points. This has been studied in supervised learning settings where the agent receives ground truth feedback; in contrast, in our case, the agent receives outcomes only for actions that were taken (“bandit feedback”). However, despite this difference, our approach can be viewed as incorporating some elements of active learning into contextual bandits by enabling the agent to acquire additional samples at a cost. There has been some work that has studied contextual bandits with costs and budget constraints (e.g., Agrawal and Goyal, 2012; Wu et al., 2015). There has also been work that has explored contextual bandit settings where the agent can not immediately integrate feedback from the environment, but can do so only in batches (Zhang et al., 2022; Ren et al., 2022; Han et al., 2020). However, all these works consider settings where the samples are obtained through a standard contextual bandit interaction—observe a context, choose an intervention, receive a reward; in contrast, our work considers targeted interventions where context values to determine the targeted subgroup is specified along with the intervention. Further, importantly, none of these works provide a way to integrate causal side information.

In Theorem 3.1, we are interested in the case where B is finite. This is of interest in practical settings where the budget is usually small. We prove a regret bound under additional assumptions (A2) which we describe below. Define m≜minx,cAπ(x|cA), where π is the (unknown) logging policy that generated DL; and MV≜|val(V)|.

Theorem 3.1 (Regret bound). Under the additional assumptions (A2) mentioned below, for any 0 < δ < 1, with probability ≥ 1 − δ,

where ϵ∈O(Bln(MXMPAYMC/δ)), ignoring terms that are constant in B, m, δ, |C| and the number of possible context-action pairs.

Proof. The proof closely follows the regret bound proof in Subramanian and Ravindran (2022) and adapts it to our setting. The purpose of the proof is to establish an upper bound on performance, and not to provide a tight bound. Bounding regret without these additional assumptions (A2) is left for future work.

First, we define the assumptions (A2) under which the theorem holds:

Section 3.1 looked at the case where B is finite. In contrast, in this section, we consider the case where B → ∞. Theorem 3.2 demonstrates the soundness of our approach by showing that as the budget increases, the learned policy will eventually converge to an optimal policy.

Theorem 3.2 (Soundness). As B → ∞, Regret → 0.

Proof. We would like to show that Regret → 0 as B → ∞. As B → ∞, in the limit, the problem becomes unconstrained minimization of Υ(N). Note that for all N, Υ(N) ≥ 0. Therefore, the smallest possible value of Υ(N) is 0.

First, note that Nx,cA→∞,∀(x,cA)   ⇒   Υ(N)→0. This is because ∀(x, c^A),

which, in turn, makes Q(ℙ[V|pa_V], N) → 0, ∀(V, pa_V). From Equation 3, it is easy to see that this causes Υ(N) → 0.

Also note that Υ(N)→0   ⇒   Nx,cA→∞,∀(x,cA). To see this, let Υ(N) → 0, and consider the case where there exists a (x, c^A) such that Nx,cA is finite. That means that there is at least one term of the form

which occurs in the Q(.) function of at least one CPD ℙ[V|pa_V], causing Q(ℙ[V|pa_V], N) > 0 since Ent^new > 0. This, in turn, causes Υ(N) → ̸0, resulting in a contradiction. Note that this makes an implicit technical assumption that ℙ[c] > 0, ∀c and that min(val(Y)) > 0; these are stronger assumptions than necessary, and could be weakened in the future.

Thus, Nx,cA→∞,∀(x,cA)⇔Υ(N)→0. In other words, each (x, c^A) gets a number of samples tending toward infinity if and only if Υ tends to 0. Thus, since Υ(N) ≥ 0, Algorithm 1 will allocate Nx,cA→∞,∀(x,cA). Each CPD has at least one (x, c^A) whose samples will be used to update its beliefs in Algorithm 1 (due to the technical assumption mentioned above). This means that each CPD will have its beliefs updated a number of times approaching infinity. Thus, for any (V, pa_V), ℙ^[V|paV]→ℙ[V|paV]. As a result, we have that ℙ^→ℙ.

Since the agent's policy constructed using ℙ will necessarily be optimal, we have that Regret → 0. This completes the proof.

Different parameterizations of M can produce a wide range of possible settings. Given this, the first experiment studies the performance of our algorithm over a set of “representative settings.” Each of these settings has a natural interpretation; for example, CA could represent the set of person-level features that we are learning a personalized treatment for, or it could represent the set of customer attributes over which we're learning a marketing policy. The settings capture the intuition that high-value contexts (contexts for which, if the optimal action is learned, high expected rewards accrue to the agent) occur relatively less frequently (say, 20% of the time), but that there can be variation in other aspects. Specifically, the variations come from the number of different values of c^A over which the 20% probability mass is spread, and in how “risky” a particular context is (e.g., difference in rewards between the best and worst actions). For details of the parameterizations, please refer to the Supplementary material. The number of samples in the initial dataset DL is kept at 0.5·|val(CA)|·|val(X)|. We consider a uniformly exploring logging policy for DL; that is, context variables for each sample are realized as per the natural distribution induced by M, but X is chosen randomly. In each run, the agent is presented with a randomly selected setting from the representative set. Results are averaged over 50 independent runs; error bars display ±2 standard errors.

Figure 3 provides the results of this Experiment. It plots the value of regret (normalized to [0, 1] since different settings have different ranges for regret) as budget B increases. We see that our algorithm performs better than all baselines. Our algorithm also retains its relatively lower regret at all values of B, providing empirical evidence of overall better regret performance.

To ensure that the results are not biased due to our choice of the representative set in Experiment 1, this experiment studies the performance of our algorithm when we directly randomize the parameters of the CPDs in each run, subject to realistic constraints. Specifically, in each run, we (1) randomly pick an i ∈ {1, ..., ⌊|val(C₁)|/2⌋}, (2) distribute 20% of the probability mass randomly over the smallest i values of C₁, and (3) distribute the remaining 80% of the mass over the remaining values of C₁. The smallest i values of C₁ have higher value (i.e., the agent obtains higher rewards when the optimal action is chosen) than the other C₁ values. Intuitively, this captures the commonly observed 80–20 pattern (for example, 20% of the customers often contribute to around 80% of the revenue); but we randomize the other aspects. For details of all the parameterizations, please refer to the Supplementary material. Averaging over runs provides an estimate of the performance of the algorithms on expectation. The number of samples in the initial dataset DL is kept at 0.25·|val(CA)|·|val(X)|. The results are averaged over 50 independent runs; error bars display ±2 standard errors.

Figure 4 shows that our algorithm performs better than all baselines in this experiment. Our algorithm also demonstrates overall better regret performance by achieving the lowest regret for every choice of B.

While Experiments 1 and 2 study purely synthetic settings, this experiment seeks to study the performance of our algorithm in realistic scenarios. We use the same causal graph used in the real world-inspired experiment in Section 4.2 of Subramanian and Ravindran (2022) and calibrate the CPDs using the data provided there. For parameterizations, refer to the Supplementary material.

The objective is to learn a policy that can assist salespeople by learning to decide how many outgoing calls to make in an ongoing deal, given just the type of deal and size of customer, so as to maximize a reward metric. The variables are related to each other causally as per the causal graph [Figure 3a in Subramanian and Ravindran (2022)]. The number of samples in the initial dataset DL is kept at 0.125·|val(CA)|·|val(X)|. The results are averaged over 50 independent runs; error bars display ±2 standard errors. Figure 5 shows the results of the experiment. Our algorithm performs better than all other algorithms in this real-world inspired setting as well. Further, it retains its better performance at every value of B.

Experiments 4 through 7 study sensitivity of the results to key aspects that define our settings. To aid this analysis, instead of regret, we consider a more aggregate measure which we call AUC. For any run, AUC is computed for a given algorithm by summing over B the regrets for that algorithm; this provides an approximation of the area under the curve (hence the name). We then study the sensitivity of AUC to various aspects of the setting or environment.

Figures 3, 4 provided regret behavior for small values of B. We are primarily interested in such small-budget behavior since that occurs more commonly in practice; for example, budgets exclusively for experimentation in software teams in often quite low.

However, it is also interesting to look at regret behavior as B becomes large. Specifically, we increase B large enough that all algorithms converge to optimal (or very close to optimal). We do this for Experiments 1 and 2. Figures 10A, B provide the results. Note that the Figures 3, 4 just zoom into these plots for small B (i.e., B between 15 and 30).

PropToValue is the slowest to converge to the optimal policy in both instances, though it demonstrates better low-budget behavior than EqualAlloc. MaxSum has the best low budget behavior among the baselines because it maximizes the total number of samples within that low budget; it, as B gets larger, EqualAlloc catches up (and even outperforms it) as it explores the context-action space better. In both experiments, however, our algorithm converges to an optimal policy faster than all baselines.

As discussed in Section 2.2, our algorithm balances the trade off between allocating more samples to context-action pairs that are higher value according to its beliefs and allocating more samples for exploration, while taking into account information leakage due to the causal graph. To understand this in more detail, we consider Setting 1 of Experiment 1, and zoom into the case where B = 20. We do 50 independent runs and plot the frequency of choosing samples containing different value of C₁. We show this for our algorithm and all baselines (except EqualAlloc since it is obvious how it allocates).

Figure 11 shows the results of this experiment. MaxSum allocates lesser number of samples than our algorithm to the two context values (C₁ ∈ {0, 1}) that are high value. PropToValue over-allocates to these two context values, resulting in poor exploration of other contexts. Our algorithm, in contrast, allocate relatively more to the high-value contexts, while also maintaining good exploration of other contexts.

Counterfactual fairness is a commonly used notion of individual fairness. Intuitively, a counterfactually fair mapping from contexts to actions ensures that the actions mapped to an individual (given by a specific choice of values for the context variables) are the same in a counterfactual world where a subset W⊆C of sensitive contexts is changed.

For example, if a loan decision policy is counterfactually fair with respect to gender, then a a male applicant would have had the same probability of being granted the loan had he been a different gender. Counterfactual fairness requires the knowledge of the causal graph to be able to establish, making it difficult to use in practice; however, when causal graphs are available, it is a powerful notion of individual-level fairness.

In our case, counterfactual fairness can be achieved by setting CB to contain all the sensitive attributes; that is, by letting CB⊇W.

A common criterion for group-level fairness is Demographic Parity (Kusner et al., 2017). Demographic Parity (DP) requires that the distribution over actions remains the same irrespective of the value of the sensitive variables.

Demographic parity is a popular notion of fairness as it aligns well with most people's understanding of fairness and does not require strong assumptions such as the knowledge of the underlying causal graph to compute. As an example, if a loan decision policy has demographic parity with respect to gender, then the probability of granting a loan given a random male is the same as that for a random individual from any other gender group.

In our case, note that

Formally, DP requires that

Our algorithm, however, does not guarantee DP even if CB⊇W:

which may not equal ℙ[ϕ = x|c^B′] since ℙ[c^A|c^B] may not equal ℙ[c^A|c^B′]. And since CB⊇W, we cannot guarantee demographic parity.

However, we discuss one way through which DP can be achieved, but with a reduction in agent's performance. Specifically, we can achieve DP by ensuring that the agent acts according to a fixed policy irrespective of the value of c^A. Intuitively, we construct a fixed policy that maximizes rewards given the agent's learned beliefs.

Specifically, let ψ^(x,cA)≜𝔼^[Y|do(x),cA]. Assume the fixed policy is probabilistic. Therefore, we're interested in a policy q which is a distribution over |val(X)|. Denoting q^(x) ≜ q(x), we solve the following optimization problem:

subject to

It is easy to see that one global optimum to this involves assigning a probability of 1 to an action x that results in the largest value of ∑cAℙ^[cA]·ψ^(x,cA). That is, choose x such that

and let q(x) = 1, and q(x′) = 0, ∀x′ ≠ x. Note that this fixed policy would perform worser on expectation than the context-specific policy learned by the agent in the main part of the paper. This, however, is a cost that can be paid to achieve DP.

As an example of how this might be useful in practice, note that a recruitment agency might be more concerned (perhaps due to legal requirements) about ensuring that it does not discriminate against particular races, genders, etc., even if that leads to suboptimal allocations of jobs. The above approach ensures that such an entity can achieve demographic parity, even though it costs in terms of performance.

As a toy example, suppose there are just two contexts {0, 1} and two actions {0, 1} such that for context 0, the rewards for actions 0 and 1 are 10 and 0, respectively; and for context 1, the corresponding rewards are 0 and 9. Both contexts are equally likely. In this case, the above approach will choose action 0 always and achieve DP with respect to the context variable, but will receive 0 reward for context 1 which is far from optimal. We do not provide a full-fledged characterization of the performance-fairness tradeoffs and reserve that for future work.