Implicit Counterfactual Effect in Partial Feedback Reinforcement Learning: Behavioral and Modeling Approach - Barakchian, Zahra; Vahabie, Abdol-Hossein; Nili Ahmadabadi, Majid (Frontiers in Neuroscience, 2022)
Context remarkably affects learning behavior by adjusting option values according to the distribution of available options. In the presence of a counterfactual factor, displaying the outcomes of both the unchosen and chosen options (i.e., providing complete feedback) would increase the contextual effect by inducing participants to compare the two outcomes during learning. However, when the context only consists of the juxtaposition of several options and there is no such explicit counterfactual factor (i.e., only partial feedback is provided), it is not clear whether and how the contextual effect emerges. In this research, we employ Partial and Complete feedback paradigms in which options are associated with different reward distributions. Our modeling analysis shows that the model that uses the outcome of the chosen option for updating the values of both chosen and unchosen options in opposing directions can better account for the behavioral data. This is also in line with the diffusive effect of dopamine on the striatum. We also observed that the size of this bias depends on the involved systems in the brain: The effect is larger in the transfer phase (in which subcortical systems are more involved) and smaller in the deliberative value estimation phase (in which the cortical system is more involved). Furthermore, our data show that the contextual effect is not limited to probabilistic rewards, but also extends to magnitude rewards. These results suggest that by extending the counterfactual concept to include the effect of the chosen outcome on the unchosen option, we can better explain why there is a contextual effect in situations in which there is no extra information about the unchosen outcome.