Let us consider the problem of CL, which is the subject of this study. First, an agent endlessly receives the input data xt∈X⊆ℝ|X| (with t = 1, 2, … as the time step) from the environment it interacts with. The corresponding output data yt∈Y⊆ℝ|Y| are predicted. y_t may be provided, as in supervised learning, or estimated by bootstrapping from x_t and other variables, as in RL; however, the former is assumed here for simplicity. Using a function approximator [e.g., deep neural networks (LeCun et al., 2015)] with parameters θ, the following minimization problem is solved:

where hθ:X↦ℝ|Y| denotes the function approximator to be optimized, and g:ℝ|Y|↦Y denotes a fixed mapping function (e.g., a sigmoid function). By setting an appropriate loss function L, y≃g(h_θ(x)) can be obtained.

The difficulty of continual learning stems from the fact that t continues to increase and its maximum cannot be defined. In extreme cases, when t → ∞, the aforementioned minimization problem cannot be numerically optimized. In addition, the data size of {(xτ,yτ)}τ=1t is limited by finite computational resources, particularly for embodied systems such as robots. Therefore, the FIFO buffer DFIFO={(xτ,yτ)}τ=min(0,t-NFIFO)+1t of finite size N^FIFO∈ℕ is often introduced, leading to the following surrogated minimization problem.

This problem can be minimized using the stochastic gradient descent method (e.g., Ilboudo et al., 2023). In other words, a batch of data, the size of which is denoted by B, is randomly extracted from D^FIFO, and θ is then updated using its gradient ∇θ|B|-1∑τ∈BL(g(hθ(xτ)),yτ). Although this solution enables optimization as in general deep learning, it discards past data from the optimization when t>N^FIFO. As t increases, the skills acquired from τ ≤ min(0, t−N^FIFO) are overwritten (unless the data in the FIFO buffer contain equivalent skills). This type of overwriting is known as catastrophic forgetting (McClelland et al., 1995).

Several approaches have been proposed to mitigate catastrophic forgetting. Among them, DER (Buzzega et al., 2020) (more precisely, DER++ in the original paper) is employed as a baseline of this study with slight modifications. DER is a simple yet powerful continual learning method that can be regarded as a combination of rehearsal and functional regularization. DER introduces the RS buffer (Vitter, 1985), D^RS (with size N^RS), to store past data that i) is passed in streaming format (the original implementation); or ii) overflow from the FIFO buffer (this paper's implementation). This RS buffer stochastically stores all the data seen so far with equal probability (see the next section), rather than storing the latest data first as in the FIFO buffer. In addition, feature z_t computed using h_θ is stored in the RS buffer together with (x_t, y_t).

Under this design, the following minimization problem is solved:

where β∈[0, 1] is the coefficient that adjusts the learning priority between the FIFO and RS buffers, and α≥0 is the weight of the regularization term that preserve past features computed using the previous θ. The second and third terms randomly select batches from D^RS. In the original implementation, each batch was selected independently; in this study, however, all data were selected simultaneously and split into two non-overlapping batches to strengthen the regularization of DER. This implementation can be regarded as a generalized version of CLEAR (Rolnick et al., 2019), which was originally designed for RL.

As mentioned previously, the RS buffer used in DER stochastically stores all the data seen so far with equal probability (Vitter, 1985). Once the buffer is full, the following algorithm is applied to select the discarded data d^del based on the new data arriving n-th data point, d′, and the existing entries {di}i=1NRS.

where U(l,u) denotes a discrete uniform distribution over integers in the interval [l, u], l, u∈ℤ and l ≤ u. When data are discarded from the buffer, the new data replace the corresponding index as dk=d′.

In the above algorithm, the probability of accepting the new data d_n on the n-th pass is given as follows:

where P(·) denotes the probability of satisfying the condition in parentheses. In addition, the probability that it remains in the RS buffer after n′ = 1, 2, … additional steps is given by

These two probabilities show that the RS buffer stochastically stores all data seen so far with equal probability, inversely proportional to the total number of data points passed, n. Here, n is sometimes referred to as the reservoir counter (Sun et al., 2022).

In previous research on continual learning, various improvements have been proposed for rehearsal-based approaches, which involve deep engagement with DER and RS. For example, many methods (Buzzega et al., 2021; Boschini et al., 2022; Caccia et al., 2022; Harun et al., 2025) utilize class (or task) information to adjust the balance between data to be rehearsed and stored, or to redefine the loss function. However, such approaches are inappropriate for this study, which assumes a task-agnostic setting.

Without using task information, determining data value often requires significant additional computational cost. For example, Aljundi et al. (2019c),(a) evaluate high-importance data using gradient information or loss changes before and after updates. In either case, gradients must be computed separately from parameter learning, doubling computational cost. Before adding data to RS buffer, Sun et al. (2022) judges a new metric akin to information gain based on data novelty and learnability. However, similarly to the above, this computation also requires substantial redundant computation and models. Bayesian neural networks need to be introduced for estimating data uncertainty, another metric for data importance, taking high cost (Wang et al., 2024). Thus, while several metrics for selecting data to replay and store have been proposed, all incur substantial computational costs solely for value estimation, making them unsuitable for systems with limited computational resources.

Another development involves revising the regularization terms introduced in DER and proposing new regularization terms. For example, Zhuo et al. (2023) proposes the forward consistency loss, although its computation requires retaining past model parameters. In the recent theoretical work (Daxberger et al., 2023), a loss integrating rehearsal, function regularization, and parameter-space regularization has been derived. However, its implementation still requires additional weight adjustment and estimation for parameter-space regularization compared to DER. Thus, regularization terms are not easily added, and in practice, they increase fine-tuning effort. This makes them difficult to handle in this study, which assumes scenarios lacking domain knowledge.

From the above, it is evident that under the task-agnostic setting with limited computational resources as the premise assumed by this study, the development of DER and related methods is not well-suited. Therefore, the subsequent discussion will proceed with the standard DER as the baseline, constructing a methodology that satisfies the aforementioned premise while enabling flexible adjustment of the trade-off between memory consolidation and plasticity.

DER is an effective method that can strongly mitigate catastrophic forgetting despite its simple implementation. However, the hyperparameters β and α in DER must be appropriately tuned, depending on the target problem, to achieve optimal performance. In fact, the original study (Buzzega et al., 2020) used different values for each benchmark. In other words, the balance between memory consolidation and plasticity must be determined manually and often requires considerable effort.

Additionally, the third term in Equation 3, which corresponds to functional regularization, attempts to preserve past features, even if they are inconsistent with the current situation. That is, the features may be valid only for past situations and not after distribution shifts. In that case, plasticity is more preferable than consolidation by the regularization.

To address these issues, this study proposes A2ER, which incorporates three strategies into DER (see Figure 1): adaptation, block, and correction.

Here, the adaptation strategy aims to automatically adjust β and α in DER, inspired by recent machine learning formulations (Haarnoja et al., 2018; Stooke et al., 2020). Specifically, the minimization problem in Equation 3 can be reinterpreted as the minimization of Equation 2 subject to the following equality constraints, which correspond to the second and third terms in DER:

where Δτ=||hθ(xτ)-zτ||22/2 for brevity, and Δ_Q≥0 denotes a threshold value for the average of Δ_τ.

In this study, Δ_Q is heuristically defined as a variable updated at each step using the following quantile function Q:

where ρ∈(0, 1) denotes the quantile level.

These equality constraints can be converted into regularization terms using Lagrange multipliers with β and α. By considering Δ_Q as a constant that is independent of θ, this conversion is consistent with Equation 3. Furthermore, β and α can be optimized to satisfy the equality constraints, yielding the following auto-tuning rules:

These are also solved by stochastic gradient descent, together with the minimization problem in Equation 3. Note that although the Lagrange multipliers (i.e., β and α in this case) are originally real numbers, their respective domains are restricted to β∈[0, 1] and α≥0. These constraints can be enforced using sigmoid and softplus functions, respectively. Even with these transformations, β and α can be efficiently optimized using the mirror descent method (Beck and Teboulle, 2003).

With the above formulation, β related to the first constraint brings about a proper balance between consolidation and plasticity by ensuring that the latest data in the FIFO buffer and the past data in the RS buffer incur similar levels of loss. The second constraint, governed by α, empirically strengthens consolidation by suppressing excessive changes in features, and increases plasticity by reducing functional regularization after sufficient consolidation. Although ρ is added for computing the threshold Δ_Q, its tuning is task-agnostic and robust (see the experimental results later).

In the second equality constraint, past data that exceed Δ_Q are classified as either inconsistent owing to distribution shifts or as having non-optimized features owing to insufficient learning. Therefore, the block strategy restricts the replay probability of such inconsistent data with the current situation, similar to intentional forgetting (Johnson, 1994), whereas the correction strategy updates the features in the buffer if the error is caused by a lack of learning similar to memory engram updating (Josselyn and Tonegawa, 2020). Because these strategies share many underlying processes, they are introduced together in this section.

First, it is assumed that the more Δ_τ deviates from Δ_Q, the more the features are considered inconsistent (or non-optimized). Therefore, the following quantity η_τ∈[0, 1] is designed to be proportional to the degree of deviation.

where clip(x; l, u) is a function that clips x to the interval [l, u], where l, u∈ℝ and l ≤ u. Δ¯Q serves as an approximate empirical estimate of the maximum Δ_τ.

Using the calculated η_τ, the following update of Δ_τ is considered (by correcting z_τ):

With this design, Δ_τ in the range ΔQ≤Δτ≤Δ¯Q forms a quadratic curve with a maximum at η_τ = 1/2, and returns to Δ_Q at η_τ = 0, 1, as illustrated in Figure 2. That is, up to the midpoint, Δ_τ is expected to be minimized through the auto-tuning of α; after that, the behavior shifts to the correction of z_τ.

The correction rate γ_τ∈[0, 1] of z_τ toward h_θ(x_τ) (i.e., the current features) is defined as follows:

The larger γ_τ is, the more inconsistent past data are. The block strategy suppresses the use of past data themselves for replay and learning. Specifically, although the expectation over D^RS in Equation 3 normally replays all data with uniform probability pτ=1/NRS, each probability is instead weighted as follows:

where λ∈(0, 1) denotes the hyperparameter for the exponential moving average of 1−γ_τ. That is, if the error norm does not become sufficiently small even after repeated learning and correction, the data will be gradually blocked for replay. Note that λ would be of low importance because it works well even when set relatively large (i.e., slight smoothing effect).

In the correction strategy z_τ in the buffer is updated using γ_τ as follows:

The minimization problem in Equation 3 can be solved after this correction. In practice, this can be done by appropriately compensating the loss function associated with z_τ. That is, Δ_τ−Δ_Q in the update rule for α in Equation 11 is multiplied by η_τ, and Δ_τ in the update rule for θ in Equation 3 is multiplied by (1-γτ)2. In this way, additional computations can be avoided by reusing values already stored in RAM.

The RS buffer used in DER accepts and stores data with a probability inversely proportional to the reservoir counter n, as expressed in Equations 5 and 6. In other words, when n becomes very large, new data are rarely accepted into the RS buffer; however, if accepted once, they can be stored for a long time. This characteristic is effective in preventing past data from being discarded and in improving consolidation, while it impairs plasticity needed to adapt to distributional shifts.

In addition, data have varying degrees of importance, as suggested by the block strategy, and it is undesirable to pass inconsistent data to the RS buffer, as noted in previous work (Sun et al., 2022). In other words, if inconsistent data are stored in the RS buffer, they may impede learning. Furthermore, simply passing such data to the RS buffer increases n, which reduces the acceptance rate for more important data that may arrive later.

To resolve these issues, this study proposes O2S, which introduces three strategies to enhance RS (see Figure 3), q-logarithm, plural, and omissions.

First, the acceptance rate is generalized to make a balance between consolidation and plasticity adjustable. Specifically, a monotonically nondecreasing transformation f:ℤ↦ℤ is designed to generate random numbers for n>N^RS, such that sampling in Equation 4 becomes k~U(1,f(n)). Note that f is required to be monotonically nondecreasing because it is also interpreted as a counter. The acceptance rate of the n-th new data point d_n is given as follows:

To satisfy the definition of probability and maintain continuity with the case n ≤ N^RS, where the acceptance rate is 1, it must hold that limn→NRSf(n)=NRS.

Considering the probability that d_n remains in the RS buffer at time n+n′-th, one term cannot be canceled out, although the derivation procedure is similar to Equation 6.

Here, the first term corresponds to the conventional case with f(n) = n, whereas the second term—that is, the total product operation—modifies it.

Even with such modifications, the definition of probability must still be satisfied. To this end, the condition for designing f can be derived by focusing on the inner part of the second term.

where Δf(n) = f(n)−f(n−1). If this term ever exceeds 1, then the resulting probability may also exceed 1, violating the definition of probability. Therefore, the sufficient condition Δf(n) ≤ 1 must be satisfied.

Thus, the generalized counter f(n) must satisfy the following conditions:

When Δf(n) = 1, new and past data are equally likely to be included in the buffer after each update, promoting the storage of past data and supporting consolidation. When Δf(n) = 0, the probability of past data remaining in the buffer decays exponentially because of the cumulative product, as in the case where f(n) = c at convergence. This facilitates the acceptance of new data and enhances plasticity. In other words, if Δf(n) can be specified and adjusted to a suitable degree, it becomes possible to balance between consolidation and plasticity.

There are several possible candidates for functions that satisfy these conditions, but this study introduces the following q-logarithmic function (Tsallis, 1988), referred to as the q-logarithm strategy:

where q∈[0, 2] is a hyperparameter that balances consolidation and plasticity.

f_q(n) is shown in Figure 4. As shown in the figure, for q₁>q₂, f_q1(n) ≤ f_q2(n) holds. For q = 0, f_q(n) reduces to the conventional RS with f_{q = 0}(n) = n, which yields the highest consolidation. For 0 ≤ q ≤ 1, limn→∞fq(n)=∞, meaning the buffer eventually stops accepting new data and storing past data, although there is a time lag before convergence. On the other hand, for q>1, an upper bound exists, and the function converges to limn→∞fq(n)=NRS+NRS/(q-1). Therefore, new data are accepted finally with constant probability and past data decay exponentially, increasing plasticity. Note that although q → ∞ is theoretically valid, the acceptance rate of new data converges to 0.5 with q = 2, which offers sufficient plasticity. This study therefore restricts q to the interval [0, 2], as previously defined.

Thus, a small q increases consolidation and a large q increases plasticity, allowing continuous adjustment of the balance between them. Other possible simple designs are examined in Appendix 4.

Although the designed counter f_q can adjust the balance between consolidation and plasticity, the trade-off itself is not eliminated and may lead to suboptimal performance unless it is appropriately tuned for the target problem. To alleviate this limitation, the plural strategy introduces a layered structure with multiple RS buffers configured differently, similar to human short- and long-term memory systems (Cowan, 2008). Furthermore, when passing data between multiple RS buffers, the importance of each data point is calculated in terms of sampling priority. Therefore, the omission strategy uses this information to determine whether the data should be passed to the next buffer.

First, the plural strategy prepares L∈ℕ layers of serially connected RS buffers {DlRS}l=1L with respective sizes NlRS. Each has its own counter n_l and balance q_l for f_q in Equation 22. Here, the FIFO buffer is regarded as a special buffer at l = 0. Data processing is shown in Figure 5. When the past data in the l−1-th buffer are discarded, they are passed to the l-th buffer. When new data passed to the l−1-th buffer are not accepted, they are not passed to the l-th buffer. If q_l is small and consolidation is prioritized in the shallow layers of l≃1, the buffer rarely discards past data, thereby preventing the flow to subsequent layers. Therefore, for l₁<l₂, it is desirable that q_l1≥q_l2. In this way, buffers in deeper layers of l≃L will have a longer time scale because new data will be passed to them less frequently. Note that the batch for training in Equation 3 is constructed as the sum of sub-batches (each of size |B|/L) sampled from all RS buffers.

Second, the omission strategy intercepts the data passing from the l−1-th to the l-th buffers. As previously mentioned, each data point has an importance value represented by the replay priority γ̄. In other words, data that have already been, or will be, blocked from replay tend to waste buffer capacity; therefore, the buffer should be fully utilized by discarding this data before advancing its counter or passing it to the next buffer. By converting γ̄ into rejection probabilities for each buffer pl-1,lrej, a data point at time τ is passed to the l-th buffer as dl′ with the following rejection probability p^rej:

where γ̄l-1,lmax and γ̄l-1,lmin denote the maximum and minimum values in either the l−1- or l-th buffer, and γ̄τ is the replay priority for the current data. ν≥0 adjusts the rejection rate; however, it is difficult to tune intuitively. Instead, this paper specifies the rejection probability ζ∈[0, 1], setting pl-1rej=plrej=0.5ν (for data with intermediate priority), which yields ν=-ln(1-1-ζ)/ln 2. Note that because the FIFO buffer does not have γ̄, the data passed from l = 0 to l = 1 are always accepted, with p0rej=0. In addition, for numerical stability, p·rej=0 if γ̄·max-γ̄·min is less than ϵ (in this study, 10⁻⁵).

First, to validate the effectiveness of A2ER, simple regression and classification problems were conducted, as shown in Figure 6. These detailed conditions are described later, but due to factors such as low learning frequency, they were set such that learning cannot be satisfactorily achieved unless the algorithm possesses sufficient plasticity while keeping consolidation as well. A neural network model consisting of two fully connected layers with 32 neurons each was trained for both toy problems. The model outputs a normal distribution for the regression problem and a categorical distribution for the classification problem. In both cases, the model was optimized by minimizing the negative log-likelihood with respect to the supervised data as the loss function.

The regression problem aimed to predict the one-dimensional output of a composite function of up to seven sine waves with different phases, frequencies, and amplitudes, given a one-dimensional input in the range [−2.5, 2.5]. During training, outputs with white noise (SD = 0.1) were obtained as the input increased in 0.001 increments from the minimum value, and the resulting input-output pairs were passed to the model sequentially. One cycle contained 5,000 data points, exceeding the buffer size, and was repeated five times to achieve sufficient accuracy. Training occurred after every 16 data points were added, and up to 16 batches were replayed from each buffer during a single training session. This problem was evaluated using the sum of Kullback-Leibler divergences (KLD) between the predicted and true distributions.

The classification problem aimed to predict which component of a Gaussian mixture distribution, comprising up to 16 components placed on a two-dimensional input space [−1, 1]², a given input belongs to, or whether it belongs to none. During training, the input space was divided into grids at intervals of 0.02, with added white noise (SD = 0.01). The outputs were the indices of the components with the highest likelihood. A threshold of 0.3² was set; if all likelihoods were below this threshold, the input was assigned an additional index, indicating outliers. This cycle was repeated five times for 10,000 data points to ensure accurate classification. Training was performed after every 32 new data points were added, and up to 16 batches were replayed from each buffer in a single training session. The task was evaluated using classification accuracy (ACC).

Using the above setup, four types of functions (sine waves and Gaussian mixture distributions) were prepared. They were trained under the following methods, each statistically evaluated using 20 different random seeds.

Because the purpose of this section is to evaluate A2ER, the conventional implementation of RS (not O2S) was employed.

The learning results are summarized in Table 2. To evaluate robustness across problems, a weighted average based on the ranks of the evaluation values was used to prioritize the worst cases. The conventional method, DER, was never among the top-2, perhaps because it was not fine-tuned and did not perform sufficiently well. The performance without the correction strategy was significantly worse than that of the conventional method. This may be because the adaptation strategy assigns too much weight to α without the reduction of the threshold Δ_Q by the correction strategy, inhibiting learning.

Under the other conditions where the correction strategy was added, the performances were better than that of the conventional method. In particular, the proposed method, A2ER, achieved the highest number of top-2 entries. However, the performance without the block strategy was similar, and its usefulness was not confirmed by these benchmarks. On the other hand, by focusing on the case without the adaptation strategy, we can find that the automatic adjustment of α is particularly effective. When β was not auto-tuned, the performance degraded mainly in the classification problems. In fact, while β in A2ER stayed around 0.5 for the regression problems, it temporarily increased to over 0.8 for the classification problems. This means that the automatic adjustment of β improves performance by steering it toward an appropriate value, even when the initial setting is not optimal.

The above results indicate that the adaptation and correction strategies of the proposed method are necessary to improve the performance of DER. The block strategy may be activated only when inconsistent data remain in the buffers because of, for example, shifts in the data-generative distributions. The benchmark problems above do not have such characteristics; thus, the effectiveness of the block strategy could not be confirmed.

To verify the effectiveness of the block strategy, additional benchmarks for reinforcement learning (RL) problems, where the data-generative distribution depends on the agent's policy, were conducted. As data generated with past policies would be inconsistent with the current policy, reverting to the inconsistent outputs might cause drastic policy updates, which are prone to make RL unstable (Schulman et al., 2017; Saglam et al., 2023). If their replay can be appropriately blocked, a policy should be smoothly updated, improving control performance steadily.

Specifically, two problems in OpenAI Gym, InvertedDoublePendulum-v4 (DoublePendulum) and Reacher-v4 (Reacher), were solved using the soft actor-critic (SAC) algorithm (Haarnoja et al., 2018). The SAC implementation was adapted from that used in the literature (Kobayashi, 2025). The training occurred after every four interactions, during which up to one batch was replayed for training. DoublePendulum and Reacher were solved with 1,500 and 1,000 episodes, respectively, followed by 100 episodes using the trained policies for evaluation. Note that these tasks were selected as toy problems expected to be solvable even with small buffers, although their performance was lower than when trained with rich computational resources and buffers. In the RL problems, the interquartile mean (IQM) of the 100 episodes' returns (i.e., the sum of rewards in an episode) is computed as the score for each trial, according to the literature (Agarwal et al., 2021).

The following four conditions were compared for the above tasks using 20 random seeds:

To ensure fairness, the buffer size in the case using only the FIFO buffer was set to N^FIFO←N^FIFO+N^RS = 1024, and the batch size was set to B×2 = 64. Note that even with an enlarged FIFO buffer, its size remains much smaller than those typically used in general RL implementations, and generalization performance is expected to decrease because it cannot retain sufficient past information. On the other hand, DER and the proposed method, which include RS buffers and regularization to past outputs, can improve generalization performance. However, excessive consolidation of past outputs may disturb or stop learning because some past data become inconsistent or non-optimized because of distribution shifts or insufficient learning.

The learning results are summarized in Table 3. As well as the above benchmarks, a weighted average based on the ranks of the scores was used to prioritize the worst cases and demonstrate robustness. The first notable point is that DER made little progress in learning. This is because the regularization term to maintain past outputs was too strong with α = 1, which prevented the value function from being updated and led to failure in policy optimization. In addition, FIFO, a common implementation in RL, showed a poor success rate on DoublePendulum. In Reacher, on the other hand, FIFO did not fail significantly, although its score did not reach a satisfactory level. This is probably because, although learning proceeded due to high plasticity, the generalization performance to reach arbitrary target positions degraded due to the lack of consolidation caused by the small buffer size.

However, the proposed method, A2ER, outperformed the other methods on both problems. Although Reacher's score seems not to differ significantly from that of FIFO, both the worst and best cases of A2ER were better than those of FIFO, indicating that A2ER provided more stable generalization performance. This result is largely owing to the block strategy, as a clear performance drop was observed in its absence. In particular, the number of failure cases increased for both problems, with the worst case for Reacher being more than twice as severe. Thus, the block strategy mitigated the influence of inconsistent past data, which causes instability in learning and deterioration of generalization performance.

Based on the above results, we can conclude that all three strategies in the proposed method, A2ER, are essential for enhancing the performance of DER.