The serotonergic system is involved in the modulation of reversal learning, a cognitive task in which behavioral flexibility is assessed, through its role as a modulator of prefrontal cortex (PFC) functioning (Alsio et al., 2021; Clarke et al., 2004; Clarke et al., 2007; Man et al., 2010). Reversal learning requires the learning of a first response to an environmental condition and the inhibition of that response with change in the condition. Thus, behavioral flexibility implies learning of a new response to changes in environmental conditions (Izquierdo and Jentsch, 2012; Schiller and Delgado, 2010).

The contribution of each PFC subregion to behavioral flexibility differs depending on the type of task used for its evaluation (Schiller and Delgado, 2010; Dias and Aggleton, 2000; Izquierdo et al., 2017; Ragozzino, 2007). For example, Chudasama and Robbins (2003) showed that damage to the infralimbic (IL) cortex delayed the reversal of a visual discrimination task, whereas Ashwell and Ito (2014) reported that inactivation of this subregion facilitated the reversal of a spatial learning task. Likewise, Dalton et al. (2016) showed that inactivation of the prelimbic (PL) cortex facilitates the reversal of probabilistic learning. However, simultaneous inactivation of the IL and PL cortices does not affect reversal learning, although it does affect changes in multidimensional or trans-modal attentional sets (Ragozzino et al., 1999; Shaw et al., 2013; Young and Shapiro, 2009). The damage or inactivation of the orbitofrontal cortex (OFC) causes a delay in reversal learning tests but does not affect the change in the attentional setting (Ragozzino, 2007; Dalton et al., 2016; Young and Shapiro, 2009; Boulougouris et al., 2007; Chudasama and Robbins, 2004; Stalnaker et al., 2007).

The PFC is densely innervated by dorsal raphe serotonergic neurons (Goncalves et al., 2009), and depletion of cerebral serotonin (5-HT) in rodents and human beings does not show any effect on initial learning in the reversal learning tasks; however, cognitive behavioral performance showing the retainment of the first learned response delayed reversal learning (Alsio et al., 2021; Kanen et al., 2021; Roberts, 2011; Zhukovsky et al., 2017; Colgin, 2013). Additionally, in non-human primates and rats, the specific depletion of 5-HT in the OFC delays reversal learning of serial discriminations, without affecting the learning of discriminations (Alsio et al., 2021; Clarke et al., 2004; Clarke et al., 2007). In contrast, Alsio et al. (2021) observed that depletion of 5-HT in the medial prefrontal cortex (mPFC) increased the number of sessions required to learn visual discrimination without a subsequent reversal effect. This evidence supports the role of prefrontal 5-HT in modulating reversal learning; however, the neurophysiological processes modulated by 5-HT during reversal learning remain to be investigated. 5-HT has been implicated in the modulation of theta activity (4–12 Hz), a type of oscillatory activity that correlates with learning and memory in different cognitive tasks (Colgin, 2013; Gutierrez-Guzman et al., 2011; Gutierrez-Guzman et al., 2012; Gutierrez-Guzman et al., 2017; Lopez-Vazquez et al., 2014; O'Neill et al., 2013; Paz et al., 2008). In this regard, Gutierrez-Guzman et al. (2011) observed that 5-HT depletion in the hippocampus facilitated the learning of a spatial memory task and increase in high-frequency theta activity recorded during the test. Similarly, 5-HT depletion in the medial septal region facilitated the learning of a working memory task and increased high-frequency theta activity (Lopez-Vazquez et al., 2014). Taken together, the previous evidence suggested that 5-HT modulates theta activity in the PFC during reversal learning. Thus, the aim of this study was to evaluate the effect of 5-HT depletion in the prefrontal subregions (IL, PL, and OFC) on the modulation of reversal learning and their correlation with the characteristics of the underlying theta activity.

Thirty-one male Sprague–Dawley rats weighing between 350 and 450 g were used in this study. All rats were maintained under standard vivarium conditions: regular 12/12 h light–dark cycle, constant room temperature at 23°C, with ad libitum access to water and food. The experiments were performed in accordance with the National Institutes of Health Guide regarding the Care and Use of Laboratory Animals (NIH Publication no. 80-23) and with the “Norma Oficial Mexicana para el uso de animales de laboratorio” (NOM-062-ZOO-1999). Adequate previsions were taken to use minimal number of animals, according to the experimental design, as well as to minimize the animal’s pain or discomfort.

Data of cognitive behavioral performance and power and coherence of brain electrical activity obtained from 14 control rats and from 13 rats with 75% depletion of 5-HT in both the mPFC (5-HT: t = 4.157; df = 11, p = 0.002; 5HIAA: t = 3.685, df = 13, p = 0.003) and OFC (5-HT: t = 2.542, df = 9, p = 0.021; 5HIAA: t = 1.901, df = 14 p = 0.078), as compared to prefrontal 5-HT levels in control animals (100%), were analyzed (Figure 2A, left). No differences were observed in DA and DOPAC levels between the groups (Figure 2A, right).

Several studies have highlighted the role of serotonin in modulating different behavioral components in different prefrontal subregions. For instance, Miyazaki et al. (2020) reported that serotoninergic activation in the OFC successfully promoted waiting with a fixed delay in reinforcement for future rewards, whereas when the delay was uncertain, serotoninergic activation in the mPFC promoted waiting for reinforcement. In addition, a stronger role of serotonin modulation in the extinction of previously learned behaviors, involving the IL cortex, and any influence on the control of conditioned fear, linked to Pl cortex functioning, have been observed (Lopez-Terrones et al., 2022; Bouton et al., 2021). Thus, different studies have shown that the effect of serotonergic manipulation on the prefrontal cortex depends on the targeted cortical subregion and the cognitive demands during behavioral tasks. Evidence indicates that reduction in prefrontal serotonin levels does not impair the acquisition of a response strategy; however, it does impair the change in previously learned strategies (Clarke et al., 2004). Thus, a deficiency occurs when a change in the task context requires the inhibition of a previously learned response and the acquisition of a new response (Alsio et al., 2021; Clarke et al., 2007; Roberts, 2011).

Our results are in accordance with the involvement of serotonin in the inhibition of a previously learned response, as well as with studies by Clarke et al. (2004), Clarke et al. (2007), and Alsio, Lehmann (Alsio et al., 2021), in which the reduction of serotonin levels in the OFC resulted in a delay in reversal learning without any effect on forward learning. However, when the reduction in serotonin level was general to the entire brain, the number of training sessions required to learn both visual discrimination and reversal learning increased (Alsio et al., 2021). Thus, the modulatory action of serotonin on prefrontal processes is prominent when behavioral tasks require strong participation of the PFC, whereas the learning process is affected only by serotonin depletion when it extends to brain regions other than the PFC.

The planning of response strategies to adapt behavior to changes in environmental contingencies is highly dependent on prefrontal function and is hindered by specific serotonergic reduction. In line with this, the current findings suggest that reducing prefrontal serotonin levels leads to a deficit in spatial reversal learning without affecting the learning process per se (evident in the learning of the first rule and in the decrease in the number of correct responses in the reversal task achieved by the serotonin-depleted animals); however, it impairs the inhibition of the first response and consequently the expression of the second response, causing a severe delay in the efficient performance of the second rule. Consistent with this, OFC and mPFC serotonin depletion caused by inescapable stress in rat models produces deficient reversal learning, which is restored by citalopram and vortioxetine treatments (Wallace et al., 2014; Natarajan et al., 2017; Danet et al., 2010; Lapiz-Bluhm et al., 2009). The endophenotype of low serotonergic activity due to low OFC tone predisposes rats to inflexibility and perseveration (Barlow et al., 2015).

Prefrontal serotonin exerts an inhibitory effect on pyramidal neuronal activity and a desynchronizing effect on hippocampal theta activity (Puig et al., 2005; Vertes and Kocsis, 1997); hence, a decrease in prefrontal serotonin would lead to an increase in theta power. According to the test trials, serotonin depletion resulted in a strong increase in theta power in the PL and OFC cortices, both in learning and reversal, and a weak increase in the IL theta power. Lopez-Terrones et al. (2022) indicated a strong inhibitory effect on IL compared to PL for serotonin released by dorsal raphe stimulation. The present results indicate a weak increase in the IL theta power and a strong increase in the PL and OFC theta power after serotonin depletion, possibly indicating stronger disinhibition in the PL and OFC. The three major differences with the present experiment are the behavioral condition (the rats were performing a reversal task), serotonergic manipulation (serotonin was reduced simultaneously in mPFC and OFC), and the type of recording (theta activity, which mainly reflects synaptic activity versus pyramidal neuron firing). These differences could account for the minimal effect observed on IL power and mPFC coherence (IL-PL) and the strong increase in theta power in the OFC and PL, along with the increased OFC-mPFC coherence observed in the present work. In addition, the stronger activation of the OFC linked to a delay in reversal learning observed here is in line with evidence that following a specific infusion of 5,7-DHT, mPFC serotonin depletion did not impair reversal learning, whereas OFC serotonin depletion impaired reversal learning (Alsio et al., 2021), while part of the increase in theta power in the PL would be related to the role of this region in spatial working memory (O'Neill et al., 2013), a component of the task unaffected by serotonin depletion. IL activity is related to emotional processing and stress-generated depression-like responses, whereas no effects on cognition have been observed after serotonergic manipulation in this region (Lopez-Terrones et al., 2022; Lopez-Terrones et al., 2023; Gasull-Camos et al., 2018). Thus, the minor effect on the IL would be related to the weak involvement of this region during cognitive processing compared with the PL, which is related to spatial working memory and the OFC implied in reversal learning.

Lopez-Vazquez et al. (2014) observed that septal serotonergic depletion facilitated spatial working memory assessed in an eight-radial-arm maze and increased the expression of concurrent hippocampal theta activity. In addition, Altman et al. (1990) and Gutierrez-Guzman et al. (2011) have reported that hippocampal serotonin depletion facilitates place learning. Thus, power increases related to the learning and reversal tasks were assessed by intragroup comparisons. The two groups showed an increase in IL power with the number of correct responses in the learning of the first rule, whereas in PL, only control animals had more correct responses that correlated with higher power. No significant correlation was observed in the serotonin-depleted group. This result indicates a strong effect of serotonin modification on PL than IL; however, as learning of the first rule was not different between groups, it is difficult to attribute this correlation to learning.

In reversal learning, the serotonin-depleted group showed a positive correlation between theta power in the OFC and the number of correct responses, in contrast to the control group, which showed no significant negative correlation. Under control conditions, an increase in theta power related to efficient performance did not occur. In fact, increased theta power in the serotonin-depleted group was related to deficient reversal learning. In addition, PL-COF coherence was negatively correlated with the number of correct responses in the reversal task (matching-to-place), possibly indicating that the disengagement of this structure is required for the inhibition of the first response and expression of the second response.

The higher theta power may be a result of the failure in inhibitory control after serotonin depletion, which was dependent on the sub-region of the PFC and task condition. This differential effect depends on the characteristics of serotonergic innervation and the serotonergic receptors expressed in each region, which may result in functional dissociation (Boulougouris et al., 2007; Goncalves et al., 2009; Commons, 2016; Leiser et al., 2015; Linley et al., 2013). Serotonin inhibits mPFC pyramidal neurons directly through 5HT1_A receptors and indirectly through the excitation of interneurons mediated by 5HT₃ receptors. In addition, pyramidal neurons are excited by serotonin via 5HT2_A and 5HT2_C receptors, whereas GABAergic neurons express inhibitory 5HT1_A and excitatory 5HT2_A receptors (Puig and Gulledge, 2011; Pompeiano et al., 1992; Morales and Bloom, 1997; Pompeiano et al., 1994). In addition, the OFC expresses a high density of excitatory 5HT₇ receptors which are related to reversal learning, specifically in the modulation of perseverative responses (Hrnjadovic et al., 2021; Nikiforuk et al., 2015). Although the principal effect of serotonin is complex and depends on the receptors, cortical layer, and type of neuron, its inhibitory effect is prominent (Puig et al., 2005) and is compatible with the increased theta activity observed after serotonin depletion.

Coherence is an index of functional coupling between the cerebral structures involved in cognitive processing (Fell and Axmacher, 2011). Under control conditions, high coherence between the mPFC and OFC occurred during the trials in which the information about the change in the environment was updated; subsequently, the coherence decreased below the initial training values. Increased coherence between the hippocampus and the PFC has been related to successful performance in a spatial alternation task during spatial decision-making (Tavares and Tort, 2022) during successful spatial working memory (O'Neill et al., 2013; de Mooij-van Malsen et al., 2023).

Similar to that observed in theta power, the coherence between the OFC-PL and OFC-IL was higher in serotonin-depleted animals both in learning and reversal (see Figure 6). Moreover, it is evident that groups have distinct coherence patterns during training. The non-significant interaction between block and condition in these cortices was principally due to the similar values in baseline and test trials. Paired comparisons of test trials were significant for the reversal learning, when the OFC-PL and OFC-IL coherences showed a sharp increase from B1 to B6 and reduction from B6 to B8 and B10. These changes match with the time of the training when the animals change their response from the first rule (non-matching-to-position) to the second rule (matching-to-position) to attain the 50% of efficiency. In B8 and B10, the control animals attained 70% efficiency, and the coherence decreased below the values at the start of the training. Similarly, increased cortical–hippocampal theta synchronization has been linked to increased performance efficiency during spatial working memory tasks (O'Neill et al., 2013; Benchenane et al., 2010). This evidence indicates that increasing prefrontal medial–orbitofrontal functional coupling is required for successful performance of prefrontal-dependent reversal tests. Once the animals acquire the new rule, the mPFC-OFC coherence was observed, possibly indicating that communication is not needed to display the adapted behavior. In contrast, the coherence of the serotonin-depleted group was high in the initial block of reversal, and the values were similar on blocks 1, 6, and 8, which finally started to decrease until block 10, when the rats reached 50% efficiency, similar to the control rats in block 6. Thus, increased coherence is necessary to inhibit the first response learned and for the acquisition of the new rule, both in control and 5-HT-depleted rats; the temporal course was slow in serotonin-depleted rats, which started with high coherence values. The increase in coherence in control rats could be related to both the inhibition of the previous response and the acquisition of the second rule, whereas in serotonin-depleted conditions, a state of increased coherence works as the interference, impairing reversal learning, as a type of noise signal that hinders the reversal learning-related increase in coherence.

Although no other studies have assessed the impact of serotonin depletion on prefrontal theta activity related to learning, there has been an increase in coherence reported in other cognitive circuits. After hippocampal or septal serotonin depletion, which leads to an increase in septo-hippocampal and septo-mammillary theta-band EEG coherence during spatial learning in the Morris water maze, place learning facilitation occurred (Gutierrez-Guzman et al., 2011; Gutierrez-Guzman et al., 2017). However, after supramammillary serotonin reduction, coherence was modified in the septo-hippocampal circuit, with higher intrahippocampal coherence and lower septo-hippocampal coherence, resulting in a strong deficit in spatial learning (Hernandez-Perez et al., 2015). Thus, serotonin appears to tune the fine coupling between the circuits that sustain memory and cognitive processing. A failure of the inhibitory action of serotonin on the PFC in serotonin-depleted rats may cause poor inhibition of the previously learned response or the inability to learn the new rule. Although functional coupling among the prefrontal areas occurred in both groups after learning the first rule, only the control group showed a reduction in IL-OFC and PL-OFC coherence in the final phases of reversal learning. This reduction possibly allows the expression of differential processing underlying reversal learning. Accordingly, Cano-Colino et al. (2013) proposed in a computational model that modifications in the prefrontal serotonin underlying working memory tests can change the attentional control of stimuli-guiding behavioral responses. In this view, a reduction in prefrontal serotonin facilitates attention to stimuli that guide behavior and reduces attention to distractors, thereby favoring working memory. Thus, during reversal learning, serotonin depletion favors a perseverant response toward the stimuli that guide initial learning, impairing the ability to shift from cognitive–behavioral responses according to the first rule toward different behavioral–cognitive responses, now dependent on new guiding stimuli, according to the second rule.

Although some studies have focused on the interaction between the prefrontal and hippocampal regions during reversal learning, studies monitoring the coupling between the mPFC and OFC during these types of tasks are lacking. The firing of prefrontal OFC neurons and hippocampal theta activity during acquisition were analyzed by Riceberg et al. (2022) who showed that a decay after the learning criterion was attained. Moreover, Mau et al. identified the formation of neuronal ensembles with stable activation strength when an animal learned a memory task. From this perspective, when reversal learning occurs, a change in active ensembles can occur because of the uncoupling of previously active ensembles and the formation of new ones (Mau et al., 2020). In conjunction with previous studies, the present results suggest that reversal learning, as an expression of behavioral flexibility, requires specific modulation of theta activity by the serotonergic system as serotonin plays a role in mediating behavioral inhibition (Cools et al., 2008; Logue and Gould, 2014). Such modulation involves fine-tuning the power and coherence of prefrontal subregions during different phases of the reversal test, allowing for expression of adaptive responses to changes in environmental contingencies.

Serotonin and dopamine have a close relationship as modulators of the prefrontal cortex, acting as opposite systems in cognitive modulation; thus, serotonin reduction facilitates dopamine-dependent learning (Olvera-Cortes et al., 2008). The present results indicate an impairment in reversal learning after serotonin depletion by the action of the neurotoxin 5,7-DHT; thus, it is possible to assume that dopamine modifications would take place after serotonin depletion. However, Sakaue et al. (2000) injected 5,7-DHT into the lateral ventricles and measured the serotonin content in PFC 14 days later. The findings indicated that prefrontal serotonin level decreased by 96%–97% without any impact on dopamine or noradrenaline (Sakaue et al., 2000). In an earlier work, Reader and Gauthier (1984), also injected 5,7-DHT intraventricular and measured the levels of 5-HT, norepinephrine, and dopamine in different cerebral areas 15 days after the administration of the neurotoxin. The results of radioenzymatic assays showed a reduction in serotonin levels in the PFC without any effect on the other two catecholamines (Reader and Gauthier, 1984). In order to protect noradrenergic terminals, in the first work, desipramine was administered 30 min before the neurotoxin infusion, and in the second work, imipramine was administered 1 h before infusion of the 5,7-DHT. In a more recent work, Alsio et al. (2021) infused 5,7-DHT to the mPFC or OFC 30 min after desipramine administration (ip.) and observed reductions of 90% in mPFC 5-HT and 80% in OFC 5-HT, without effects on dopamine and norepinephrine levels. In the present work, desipramine was administrated 30 min before infusion of 5,7-DHT, and the neurotoxin infusion did not result in significant differences in dopamine and DOPAC levels (Figure 2A, right). Thus, the changes in reversal learning observed here could be attributed to the reduction in serotonin levels. The role of serotonin in regulating impulsivity (Desrochers et al., 2022) suggests that the deficient reversal learning observed here may have been associated with higher impulsivity. However, no differences in the latencies for choosing the arm in the test trials were observed between groups. In fact, a trend for longer latencies was observed in the serotonin-depleted group. This trend was related to a longer time spent in the zone where the decision of which arm to enter is taken. Thus, the failure of reversal learning was not related to impulsivity.

Obsessive compulsive disorders (OCD), post-traumatic stress disorders (PTSD), schizophrenia, gambling disorders, and stimulant use disorder are pathologies that are related to both with impaired behavioral flexibility and serotonergic alterations (Vaghi et al., 2017; Waltz and Gold, 2007; Verdejo-Garcia et al., 2015; Marx et al., 2021; Kish et al., 2009; Kishi et al., 2009; Hasselbalch et al., 2007; Lopez-Figueroa et al., 2004). Our results are in accord with a deficient reversal learning possibly related with the impaired change in the expression of a new response after the learning of a first response in conditions of reduced levels of prefrontal serotonin. Possibly as an expression of impaired behavioral inhibition, a key component of cognitive control (Friedman and Robbins, 2022). Changes in prefrontal neural activity, as observed in the present results, may occur before appearance of behavioral signs in the different pathologies involving alterations in the serotonergic system and impaired behavioral flexibility.